From d74739a804e03122ae9c289888f63f4dbe8aa24e Mon Sep 17 00:00:00 2001 From: manzuoni-astera Date: Tue, 11 Aug 2026 17:44:12 -0400 Subject: [PATCH] fix(docker): stop shipping empty pixi environments MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Published images carry all five environments as empty shells: the directories and pixi's own bookkeeping in conda-meta are present, but there are no package records, no bin/, no lib/. /app/.pixi/envs/protenix has no python, and all of /app is 1.6 MB against 12 GB of checkpoints. The failure is silent in both directions. At build time `pixi install --frozen` reports "The environment has been installed" and exits 0 against such a prefix, so nothing fails. At runtime pixi trusts the same state, so it will not repair itself, and because the image sets SAMPLEWORKS_REQUIRE_PREBUILT_PIXI=1 the runner refuses to fall back — the image promises environments it does not carry. Reported by two scientists looking for the prebuilt protenix env. Clearing the prefix first forces a genuine install even when a stale or stub prefix arrives from a cached layer, the base image, or the registry buildcache. The per-environment interpreter check then makes an empty env fail the build rather than ship, which is the part that matters: `pixi install` succeeding is not evidence that anything was installed. Drops the /root/.cache/pixi mount, which carries pixi's own "is this environment current" state — the thing that can disagree with a prefix restored from a different build. The rattler and uv caches stay; they hold the downloads that actually make rebuilds fast, and were verified not to affect what lands in the layer. Verified with minimal Docker builds: against a prefix poisoned to look exactly like the shipped one, the fixed step produces a real 116 MB environment with a working interpreter, and the assertion fails the build when the interpreter is missing. --- Dockerfile | 34 +++++++++++++++++++++++++++++++--- 1 file changed, 31 insertions(+), 3 deletions(-) diff --git a/Dockerfile b/Dockerfile index d75f8b6a..d19ca2d2 100644 --- a/Dockerfile +++ b/Dockerfile @@ -127,14 +127,42 @@ RUN --mount=type=bind,from=checkpoints,target=/ck \ # IMPORTANT: keep these installs in a single RUN. Splitting them into separate # Docker layers duplicates shared conda packages (numpy, CUDA libs, etc.) and can # add tens of GB to the image. -RUN --mount=type=cache,target=/root/.cache/pixi \ - --mount=type=cache,target=/root/.cache/rattler \ +# +# The `rm -rf` and the assertion below are both load-bearing. Published images +# have shipped with all five environments as empty shells: the directories and +# pixi's own bookkeeping in conda-meta present, but no package records, no bin/, +# no lib/. `pixi install --frozen` then reports "The environment has been +# installed" against them and exits 0, so the breakage is invisible at build +# time and only surfaces when a scientist finds /app/.pixi/envs/protenix has no +# python. `SAMPLEWORKS_REQUIRE_PREBUILT_PIXI=1` makes the runner refuse to fall +# back, so the image promises environments it does not carry. +# +# Clearing the prefix first forces a genuine install even when a stale or stub +# prefix arrives from a cached layer, the base image, or the registry +# buildcache. The per-environment check then makes an empty env fail the build +# instead of shipping: a directory that exists but has no interpreter is exactly +# the state that got published, and `pixi install` alone does not catch it. +# +# `/root/.cache/pixi` is deliberately not cached across builds — it carries +# pixi's own "is this environment current" state, which is the thing that can +# disagree with a prefix restored from a different build. The rattler and uv +# caches stay: they hold downloaded packages and wheels, are what actually make +# rebuilds fast, and were verified not to affect what lands in the layer. +RUN --mount=type=cache,target=/root/.cache/rattler \ --mount=type=cache,target=/root/.cache/uv \ + rm -rf /app/.pixi/envs && \ pixi install -e boltz --frozen && \ pixi install -e protenix --frozen && \ pixi install -e rf3 --frozen && \ pixi install -e protpardelle --frozen && \ - pixi install -e analysis --frozen + pixi install -e analysis --frozen && \ + for env in boltz protenix rf3 protpardelle analysis; do \ + test -x "/app/.pixi/envs/${env}/bin/python" || { \ + echo "FATAL: pixi environment '${env}' has no interpreter at /app/.pixi/envs/${env}/bin/python."; \ + echo " pixi reported success but installed nothing — refusing to ship an empty environment."; \ + exit 1; \ + }; \ + done # A GPU is not required to build the image. Pre-compile CUDA extensions only when # the builder exposes NVIDIA devices; if present, failures should stop the build.