Fix version drift across Terraform/Docker and remove leftover Hive config - #34
Merged
Merged
Conversation
…nfig The Phase 4-6 upgrade commits bumped the Docker image layer but left the deployed Terraform manifests on the old versions, so the running platform never actually moved to Spark 4.0 / Airflow 3.1. Align everything on one consistent latest set and drop the Hive catalog config left behind after the "Iceberg only" cleanup. Version drift: - Spark master/worker: apache/spark:3.5.0 -> 4.0.1 (matches Docker layer) - Airflow Helm chart: 1.18.0 -> 1.19.0; airflowVersion/defaultAirflowTag 3.0.2 -> 3.1.7 (chart 1.19.0's native appVersion) - Airflow Dockerfile: apache/airflow:3.1.5 -> 3.1.7-python3.13 - Iceberg operator runtime: iceberg-spark-runtime-3.5_2.12 -> 4.0_2.13 (Spark 4.0 uses Scala 2.13) and hadoop-aws 3.3.4 -> 3.4.1 Hive cleanup (Hive was removed in 1143622, Iceberg-only): - Drop spark_catalog=SparkSessionCatalog/type=hive from the Iceberg operator and spark-defaults.conf (needed a Hive metastore that no longer exists); the local Hadoop catalog is unaffected - Remove dead Hive metastore log4j logger filters Verified: terraform fmt -check and terraform validate both pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This was referenced Jun 5, 2026
klagrida
added a commit
that referenced
this pull request
Jun 6, 2026
The #34 version bumps (Spark 3.5->4.0.1, Airflow 3.0.2->3.1.7) increased total image bytes pulled onto the single minikube CI node. With more pull contention, whichever deployment loses the image-pull lottery exceeds its timeout - the failure moved between runs (jupyter, airflow, prometheus, grafana) rather than being one broken component: - platform-tests run: airflow still pulling when the 15m step timeout hit - ci run: airflow deployed fine (6m17s) but prometheus/grafana hit the provider's default 10m rollout wait Give the heavier deploys consistent headroom: - prometheus + grafana: timeouts.create/update=20m, progress_deadline=1200 (matching the jupyter fix in the previous commit) - airflow helm_release: timeout 900 -> 1500 (25m) - Deploy step timeout-minutes: platform-tests 15 -> 30, ci 20 -> 30 Durable fix (smaller/pinned images, pre-pull) tracked in #36. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
klagrida
added a commit
that referenced
this pull request
Jun 6, 2026
The #34 version bumps (Spark 3.5->4.0.1, Airflow 3.0.2->3.1.7) increased total image bytes pulled onto the single minikube CI node. With more pull contention, whichever deployment loses the image-pull lottery exceeds its timeout - the failure moved between runs (jupyter, airflow, prometheus, grafana) rather than being one broken component: - platform-tests run: airflow still pulling when the 15m step timeout hit - ci run: airflow deployed fine (6m17s) but prometheus/grafana hit the provider's default 10m rollout wait Give the heavier deploys consistent headroom: - prometheus + grafana: timeouts.create/update=20m, progress_deadline=1200 (matching the jupyter fix in the previous commit) - airflow helm_release: timeout 900 -> 1500 (25m) - Deploy step timeout-minutes: platform-tests 15 -> 30, ci 20 -> 30 Durable fix (smaller/pinned images, pre-pull) tracked in #36. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The Phase 4–6 upgrade commits bumped the Docker image layer (Spark 4.0.1, Airflow 3.1.x, Iceberg 4.0 runtime) but left the deployed Terraform manifests on the old versions (
apache/spark:3.5.0, Airflow chart 1.18.0 /3.0.2). The running platform therefore never actually moved to Spark 4.0 / Airflow 3.1, and the Iceberg operator was still loading the Spark-3.5 / Scala-2.12 runtime.This PR reconciles the drift — standardizing on a single, internally consistent latest set — and removes Hive catalog config left behind after the "Iceberg only" cleanup (
1143622).Version drift fixes
apache/spark:3.5.0apache/spark:4.0.11.18.01.19.0airflowVersion/defaultAirflowTag3.0.23.1.7apache/airflow:3.1.5-python3.13apache/airflow:3.1.7-python3.13iceberg-spark-runtime-3.5_2.12:1.10.0iceberg-spark-runtime-4.0_2.13:1.10.03.3.43.4.1Notes:
pyspark==4.0.1. Iceberg runtime moves to4.0_2.13because Spark 4.0 uses Scala 2.13 — the old3.5_2.12jar was incompatible with the now-4.0 cluster.Hive cleanup
Hive was removed in
1143622(platform is Iceberg-only), but config kept pointing at a Hive metastore that no longer exists:spark.sql.catalog.spark_catalog = SparkSessionCatalog+type = hivefrom the Iceberg operator andspark-defaults.conf. ThelocalHadoop catalog is unaffected.Verification
terraform fmt -check -recursiveterraform validate— Success! The configuration is valid.3.5.0/3.0.2/3.5_2.12/hadoop-aws:3.3.4/1.18.0/hivereferences in code, config, or.tf.Out of scope (separate follow-ups)
ldp-airflow/ldp-spark/ldp-jupyter) are built in CI but Terraform still deploys upstream images directly — DAGs/jobs aren't delivered via images.jupyter/pyspark-notebook:latest(unpinned + deprecated Docker Hub namespace).🤖 Generated with Claude Code