Custom Docker images are built but never deployed

[klagrida]

Problem

The custom images ldp-airflow, ldp-spark, and ldp-jupyter are built by make build-* and the CI docker-build job, but nothing references them. Terraform deploys vanilla upstream images directly:

• terraform/main.tf:103 — Jupyter uses jupyter/pyspark-notebook:latest
• terraform/modules/spark/main.tf — workers/master use apache/spark:4.0.1
• terraform/modules/airflow/main.tf — Helm defaultAirflowRepository: apache/airflow

Impact

The Dockerfiles' COPY airflow/dags, COPY airflow/plugins, COPY spark/jobs, and COPY spark/lib are effectively dead code — user DAGs, the custom Iceberg operator/plugins, and Spark jobs are not delivered to the running cluster through images. DAGs reach Airflow only via the dags.persistence PVC; Spark jobs have no delivery path at all.

So make build-* and the CI image-build job validate images that production never runs.

Options to fix

1. Wire the custom images into Terraform — point the Spark module, Jupyter deployment, and Airflow defaultAirflowRepository/images.airflow at the locally built ldp-* images (load into Minikube via minikube image load). This makes the COPY steps meaningful.
2. Or drop the custom images if the PVC/mount-based delivery is the intended model, and remove the now-misleading build targets + CI job.

Decide which delivery model is canonical, then make build + deploy consistent.

Context

Surfaced during the version-drift review (#34).