Skip to content

feat(warehouse): add Hive as a LEGACY learning lab (Hive → Iceberg migration) - #63

Merged
klagrida merged 1 commit into
mainfrom
add-hive-legacy-lab
Jun 12, 2026
Merged

feat(warehouse): add Hive as a LEGACY learning lab (Hive → Iceberg migration)#63
klagrida merged 1 commit into
mainfrom
add-hive-legacy-lab

Conversation

@klagrida

Copy link
Copy Markdown
Contributor

What

Trino only exposed an Iceberg catalog. This adds an opt-in, clearly-legacy Hive lab so you can see the old table format and practise the Hive → Iceberg migration (a real CDP interview topic) — without diluting the Iceberg-first story. Hive is off by default and flagged legacy in every file.

Why Hive, given Iceberg is the modern path

It's deliberately the old world. In CDP, Hive tables are what you migrate away from. Keeping a tiny HMS around lets you create a Hive table, query it from Trino, and then do the one move that matters — CREATE TABLE iceberg.x AS SELECT * FROM hive.y. That's the lesson.

What's in it

File Role
quickstart/docker-compose.hive.yml Overlay: standalone Hive Metastore (apache/hive:4.0.0, Thrift :9083, embedded Derby) + the hive catalog into Trino. Lean (make hive = minio + trino + hive-metastore) so it fits an 11 GB laptop.
platform/warehouse/hive/catalog/ The lab's own Trino catalog dir, mounted as /etc/trino/catalog — carries hive.properties + a copy of iceberg.properties (the migration target).
platform/warehouse/hive/core-site.xml S3A→MinIO config the metastore needs to resolve the s3a:// locations it stores.
platform/warehouse/hive/hive-legacy.sh Runnable proof: Hive table on S3 → query → migrate to Iceberg → assert counts + totals match.
.github/workflows/hive-ci.yml CI proof, same timeout + Docker Hub retry hardening as the other compose workflows.

Notes / gotchas handled

  • No stub pollution: the overlay replaces Trino's catalog dir rather than nesting a single-file mount inside the base one — nesting needs a writable parent and leaves an empty hive.properties stub in quickstart/trino/etc/catalog, which would break make core.
  • HMS S3A: apache/hive bundles the Hadoop S3A jars but not on the metastore classpath, and has no MinIO config — without core-site.xml + HADOOP_CLASSPATH, CREATE SCHEMA fails with ClassNotFoundException: S3AFileSystem.

Verified

Ran the full slice locally end-to-end: legacy Hive table created (Parquet on s3a://warehouse/hive/...), queried (5 rows), migrated to Iceberg via CTAS, counts (5=5) and totals (256.49=256.49) matched. Base catalog dir confirmed untouched (no stub).

🤖 Generated with Claude Code

…igration)

Trino only exposed an Iceberg catalog. Add an opt-in, clearly-legacy Hive
lab so you can see the old table format and practise the Hive -> Iceberg
migration (a real CDP interview topic) — without diluting the Iceberg-first
story: it's off by default and flagged legacy in every file.

- quickstart/docker-compose.hive.yml: overlay adding a standalone Hive
  Metastore (apache/hive:4.0.0, Thrift :9083, embedded Derby) and a `hive`
  catalog mounted into Trino. Lean by default (make hive = minio + trino +
  hive-metastore only), so it fits an 11GB laptop.
- platform/warehouse/hive/catalog/: the lab's own Trino catalog dir, mounted
  AS /etc/trino/catalog (Compose dedups by target), carrying hive.properties
  + a copy of iceberg.properties (the migration target). Swapping the whole
  dir avoids nesting a file mount inside the base one, which would leave an
  empty stub in quickstart/trino/etc/catalog and break `make core`.
- platform/warehouse/hive/core-site.xml + HADOOP_CLASSPATH: the bundled
  Hadoop S3A jars + MinIO config the metastore needs to resolve the s3a://
  locations it stores (otherwise CREATE SCHEMA fails with ClassNotFound
  S3AFileSystem).
- platform/warehouse/hive/hive-legacy.sh: runnable proof — create a Hive
  table on S3, query it, migrate it into Iceberg via CTAS, assert row counts
  and totals match. Verified locally end-to-end.
- .github/workflows/hive-ci.yml: CI proof with the same timeout + Docker Hub
  retry hardening as the other compose workflows.
- README + Makefile (make hive) wired in, with legacy warnings throughout.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@klagrida
klagrida merged commit c3c6ac3 into main Jun 12, 2026
7 checks passed
@klagrida
klagrida deleted the add-hive-legacy-lab branch June 12, 2026 10:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant