Container validation: no image reaches :latest untested, and the lecture matrix is a trustworthy signal

[quantecon-services]

Goal: no container image reaches :latest without being tested, and the weekly lecture-build matrix (test-containers-lectures.yml) is red only when something is really broken.

Where we stand (verified 2026-09-26)

Next: #102 — PR #144 has a changes-requested review (2026-09-25): a retry over the same _build never re-executes the failed notebook, so it needs a jb clean step between attempts. Waiting on the contributor; it still carries do-not-merge.

• The latest matrix run (36122900556, on 29e148a, the first on the 2026.07 images) failed only in the two lecture-python.myst legs, and only on lectures/bayes_nonconj.md: numpyro 0.22's BinomialProbs distribution got invalid probs parameter, the same failure as the two runs before it. The fix is bayes_nonconj: declare the truncated log-normal prior's (0, 1] support (numpyro 0.22) lecture-python.myst#1065 (open). The other four legs passed. The matrix has been red since 2026-08-03 (alert: 🔴 CI failure: Test Container Builds #159), apart from one green run on 2026-09-14.
• Those two legs took 108 min (quantecon-build) and 123 min (quantecon), inside the 180-minute limit ci(lecture-validation): per-repo timeout (180 min for lecture-python.myst); log the runner CPU #197 gave lecture-python.myst; at the old 120-minute ceiling the quantecon leg would have been cancelled. test-containers-lectures: lecture-python.myst doubled to the 120-minute ceiling #141 stays open for the runtime itself (lecture-python.myst: CPU build time sits at the lecture-validation ceiling #196).
• Both images moved to the anaconda=2026.07 baseline the matrix repos pin (deps(containers): move both images to the Anaconda 2026.07 baseline #199, merged 2026-09-25). The build and the smoke test were green, and the matrix run above shows no new failure. The matrix covers three repos (lecture-python-intro, lecture-python.myst, lecture-python-advanced.myst); lecture-jax is commented out.
• Nothing validates an image before :latest moves (containers: no validation runs before the tag every consumer pins moves #142). Any merge touching containers/** rebuilds and pushes :latest immediately. deps: bump the conda group across 2 directories with 1 update #162, feat(containers): Node 24 LTS from nodejs.org, out of conda #166, test(containers): exercise the real theme and FreeFont path in the smoke fixture (#108) #171 and deps(containers): move both images to the Anaconda 2026.07 baseline #199 all went out that way.

Plan

The work items and their order are the sub-issues below.

Sequencing rationale:

1. test-containers-lectures: absorb transient raw-GitHub 429s when building lecture repos #102 is nearly done.
2. test-containers-lectures: lecture-python.myst doubled to the 120-minute ceiling #141 comes next, because a matrix that times out cannot tell a healthy image from a broken one.
3. containers: no validation runs before the tag every consumer pins moves #142 is the larger change. Its own body says its phase 2 (main-<sha> tag, retag :latest on green) should follow the release-gate fixture work in The release gate needs its own fixture repo — the canary cannot be trusted as one #136.

Related work

• Tracking: release gating — get to a candidate that is verified before v0 moves #138, release gating for the actions' v0 tag. It has the same structural problem for a different artefact. It is kept separate because its goal is the v0 move, not images.
• Optimize preview builds: phased plan for the fastest PR previews (tracking) #92, preview-build speed. Two of its items are container changes:
  • zstd layers for quantecon-build, plus a tag input for test-container.yml #150 pushes a :zstd tag and smoke-tests it before :latest, which overlaps containers: no validation runs before the tag every consumer pins moves #142's phase 2;
  • Ship containers/quantecon-preview: slim TeX-free preview image with pinned deploy CLI #151 adds a new quantecon-preview image that this validation will need to cover.
• Build-time monitoring across lecture repos, with published benchmarks #186, build-time monitoring. test-containers-lectures: lecture-python.myst doubled to the 120-minute ceiling #141's per-leg duration summary is a small first instance of it.

Out of scope

• The bayes_nonconj.md failure itself: that fix belongs in lecture-python.myst.
• Image content changes such as the Anaconda baseline and pins. Those follow the dependency policy in docs/dev/PLAN.md.