Jupyter uses unpinned, deprecated jupyter/pyspark-notebook:latest

[klagrida]

Problem

Jupyter pulls jupyter/pyspark-notebook:latest in two places:

• terraform/main.tf:103 (deployed image)
• docker/jupyter/Dockerfile:1 (FROM)

Two issues with this tag:

1. Unpinned (:latest) — non-reproducible; the image can change underneath the platform without notice, and won't match the pinned pyspark==4.0.1 in docker/jupyter/requirements.txt.
2. Deprecated namespace — the Docker Hub jupyter/* images are no longer maintained; the project moved to quay.io/jupyter/*. The Docker Hub images will eventually stop receiving updates / may break.

Suggested fix

• Repin to a specific, current tag under the maintained registry, e.g. quay.io/jupyter/pyspark-notebook:<pinned-digest-or-tag>, in both the Dockerfile FROM and terraform/main.tf.
• Choose a tag whose bundled PySpark matches the deployed Spark 4.0.1 cluster (the current image bundles a different Spark version than the cluster, which is its own latent mismatch).

Related

• Tied to Custom Docker images are built but never deployed #35 — if the custom ldp-jupyter image is wired into Terraform, the deployed image and the pinned base should be reconciled together.
• Surfaced during the version-drift review (Fix version drift across Terraform/Docker and remove leftover Hive config #34).

[klagrida]

The :latest weight is now also a CI stability problem, not just reproducibility.

After #34 bumped the Spark images (3.5.0 → 4.0.1), the larger pulls on the constrained minikube CI node pushed jupyter/pyspark-notebook:latest (~4 GB) past Terraform's default 10-minute rollout wait, turning main red on E2E Tests (Linux) / Cluster Integration Test:

Error: Waiting for rollout to finish: 1 replicas wanted; 0 replicas Ready
  with kubernetes_deployment_v1.jupyter

Stop-gap shipped in #38 (bumped the Jupyter rollout timeout to 20m). The durable fix is still this issue: pin a smaller, maintained image so the pull is fast and CI isn't at the mercy of a 4 GB :latest download.