feat(warehouse): add Hive as a LEGACY learning lab (Hive → Iceberg migration) - #63
Merged
Conversation
…igration) Trino only exposed an Iceberg catalog. Add an opt-in, clearly-legacy Hive lab so you can see the old table format and practise the Hive -> Iceberg migration (a real CDP interview topic) — without diluting the Iceberg-first story: it's off by default and flagged legacy in every file. - quickstart/docker-compose.hive.yml: overlay adding a standalone Hive Metastore (apache/hive:4.0.0, Thrift :9083, embedded Derby) and a `hive` catalog mounted into Trino. Lean by default (make hive = minio + trino + hive-metastore only), so it fits an 11GB laptop. - platform/warehouse/hive/catalog/: the lab's own Trino catalog dir, mounted AS /etc/trino/catalog (Compose dedups by target), carrying hive.properties + a copy of iceberg.properties (the migration target). Swapping the whole dir avoids nesting a file mount inside the base one, which would leave an empty stub in quickstart/trino/etc/catalog and break `make core`. - platform/warehouse/hive/core-site.xml + HADOOP_CLASSPATH: the bundled Hadoop S3A jars + MinIO config the metastore needs to resolve the s3a:// locations it stores (otherwise CREATE SCHEMA fails with ClassNotFound S3AFileSystem). - platform/warehouse/hive/hive-legacy.sh: runnable proof — create a Hive table on S3, query it, migrate it into Iceberg via CTAS, assert row counts and totals match. Verified locally end-to-end. - .github/workflows/hive-ci.yml: CI proof with the same timeout + Docker Hub retry hardening as the other compose workflows. - README + Makefile (make hive) wired in, with legacy warnings throughout. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Trino only exposed an Iceberg catalog. This adds an opt-in, clearly-legacy Hive lab so you can see the old table format and practise the Hive → Iceberg migration (a real CDP interview topic) — without diluting the Iceberg-first story. Hive is off by default and flagged legacy in every file.
Why Hive, given Iceberg is the modern path
It's deliberately the old world. In CDP, Hive tables are what you migrate away from. Keeping a tiny HMS around lets you create a Hive table, query it from Trino, and then do the one move that matters —
CREATE TABLE iceberg.x AS SELECT * FROM hive.y. That's the lesson.What's in it
quickstart/docker-compose.hive.ymlapache/hive:4.0.0, Thrift:9083, embedded Derby) + thehivecatalog into Trino. Lean (make hive= minio + trino + hive-metastore) so it fits an 11 GB laptop.platform/warehouse/hive/catalog//etc/trino/catalog— carrieshive.properties+ a copy oficeberg.properties(the migration target).platform/warehouse/hive/core-site.xmls3a://locations it stores.platform/warehouse/hive/hive-legacy.sh.github/workflows/hive-ci.ymlNotes / gotchas handled
hive.propertiesstub inquickstart/trino/etc/catalog, which would breakmake core.apache/hivebundles the Hadoop S3A jars but not on the metastore classpath, and has no MinIO config — withoutcore-site.xml+HADOOP_CLASSPATH,CREATE SCHEMAfails withClassNotFoundException: S3AFileSystem.Verified
Ran the full slice locally end-to-end: legacy Hive table created (Parquet on
s3a://warehouse/hive/...), queried (5 rows), migrated to Iceberg via CTAS, counts (5=5) and totals (256.49=256.49) matched. Base catalog dir confirmed untouched (no stub).🤖 Generated with Claude Code