fix(docker): stop shipping empty pixi environments - #368
Conversation
Published images carry all five environments as empty shells: the directories and pixi's own bookkeeping in conda-meta are present, but there are no package records, no bin/, no lib/. /app/.pixi/envs/protenix has no python, and all of /app is 1.6 MB against 12 GB of checkpoints. The failure is silent in both directions. At build time `pixi install --frozen` reports "The <env> environment has been installed" and exits 0 against such a prefix, so nothing fails. At runtime pixi trusts the same state, so it will not repair itself, and because the image sets SAMPLEWORKS_REQUIRE_PREBUILT_PIXI=1 the runner refuses to fall back — the image promises environments it does not carry. Reported by two scientists looking for the prebuilt protenix env. Clearing the prefix first forces a genuine install even when a stale or stub prefix arrives from a cached layer, the base image, or the registry buildcache. The per-environment interpreter check then makes an empty env fail the build rather than ship, which is the part that matters: `pixi install` succeeding is not evidence that anything was installed. Drops the /root/.cache/pixi mount, which carries pixi's own "is this environment current" state — the thing that can disagree with a prefix restored from a different build. The rattler and uv caches stay; they hold the downloads that actually make rebuilds fast, and were verified not to affect what lands in the layer. Verified with minimal Docker builds: against a prefix poisoned to look exactly like the shipped one, the fixed step produces a real 116 MB environment with a working interpreter, and the assertion fails the build when the interpreter is missing.
📝 WalkthroughWalkthroughThe Dockerfile now clears existing Pixi environment prefixes, installs with frozen metadata, retains package and wheel caches, and verifies executable Python interpreters in five environments. ChangesPixi installation validation
Estimated code review effort: 2 (Simple) | ~10 minutes Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Pull request overview
This PR fixes a Docker image build issue where pixi environments could be present only as “stub” prefixes (directories + minimal metadata) while pixi install --frozen still reported success, causing published images to ship without usable interpreters for model environments.
Changes:
- Force a real pixi environment installation by clearing
/app/.pixi/envsbefore installing all environments. - Add a post-install assertion that each environment contains an executable
bin/python, failing the build if an env is empty. - Stop using the BuildKit cache mount for
/root/.cache/pixito avoid persisting pixi “current state” across builds.
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| echo " pixi reported success but installed nothing — refusing to ship an empty environment."; \ | ||
| exit 1; \ | ||
| }; \ | ||
| done |
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@Dockerfile`:
- Around line 159-164: Update the environment validation loop in the Dockerfile
to require each `/app/.pixi/envs/${env}/bin/python` path to be a regular
executable file, then invoke it with `-c` to validate its version is at least
3.11 and below 3.14, matching the pyproject.toml constraint; retain the existing
fatal error behavior when validation fails.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
| for env in boltz protenix rf3 protpardelle analysis; do \ | ||
| test -x "/app/.pixi/envs/${env}/bin/python" || { \ | ||
| echo "FATAL: pixi environment '${env}' has no interpreter at /app/.pixi/envs/${env}/bin/python."; \ | ||
| echo " pixi reported success but installed nothing — refusing to ship an empty environment."; \ | ||
| exit 1; \ | ||
| }; \ |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/sh
set -eu
tmp="$(mktemp -d)"
trap 'rm -rf "$tmp"' EXIT
printf '#!/bin/sh\nexit 127\n' > "$tmp/python"
chmod +x "$tmp/python"
test -x "$tmp/python"
if "$tmp/python" -c 'import sys'; then
echo "unexpected: fake interpreter ran"
exit 1
fiRepository: diff-use/sampleworks
Length of output: 158
🏁 Script executed:
#!/bin/sh
set -eu
printf '%s\n' '--- Dockerfile ---'
sed -n '145,170p' Dockerfile
printf '%s\n' '--- Python constraints ---'
rg -n -C 2 'requires-python|python' pyproject.toml
printf '%s\n' '--- Interpreter guard occurrences ---'
rg -n -C 3 'test -x|\.pixi/envs/.*/bin/python|pixi install' DockerfileRepository: diff-use/sampleworks
Length of output: 4818
Run each environment’s Python interpreter during validation.
test -x checks only execute permission. It can accept an executable directory or non-Python file. Add test -f, then run the interpreter with -c and enforce >= 3.11, < 3.14 from pyproject.toml.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@Dockerfile` around lines 159 - 164, Update the environment validation loop in
the Dockerfile to require each `/app/.pixi/envs/${env}/bin/python` path to be a
regular executable file, then invoke it with `-c` to validate its version is at
least 3.11 and below 3.14, matching the pyproject.toml constraint; retain the
existing fatal error behavior when validation fails.
Source: MCP tools
Published images ship all five pixi environments as empty shells. Reported by @feng-yu and @marcuscollins looking for the prebuilt protenix env.
Important: the Dockerfile is not the bug
I built the real thing to check —
pixi install -e protenix --frozenagainst the realpixi.lock, onlinux/amd64:and it survives into the exported image, not just the build. I also poisoned the prefix to look exactly like the shipped one (metadata present, zero packages, matching lock hash) and the current step still repaired it to a full 13 GB env.
So the build instructions are correct, and this PR is not a root-cause fix. I could not reproduce the failure in any local configuration — including three mechanisms I tested and disproved (rattler hardlinking out of a cache mount, a missing
__cudavirtual package, and pixi state cached across builds).What this PR does
Given the above, it is insurance rather than a cure:
pixi installreporting success is demonstrably not evidence that anything was installed — that is exactly how five empty environments shipped. This turns that into a build failure. Verified it fails (exit code: 1) when the interpreter is missing.rm -rf /app/.pixi/envsbefore installing, so a stale or stub prefix arriving from a cached layer can't influence the result./root/.cache/piximount, which carries pixi's own "is this environment current" state — the one cache whose contents can disagree with a prefix restored from a different build. The rattler and uv caches stay; they hold the downloads that make rebuilds fast, and I verified they don't change what lands in the layer.What actually needs to happen
Since the Dockerfile is correct, the published image must have come from a build whose environment differed — and the remaining candidate I can't rule out is the registry buildcache (
cache-from/cache-to: type=registry … mode=max) importing a layer with the metadata but not the package files. That would produce exactly this.Someone with access should rebuild the image with
:buildcachebusted. I don't have dispatch rights on this repo. With this PR merged first, that rebuild either produces working environments or fails loudly instead of silently shipping empties.Meanwhile
SAMPLEWORKS_REQUIRE_PREBUILT_PIXI=1makes the runner refuse to fall back, so the image currently promises environments it doesn't carry. Until a good image exists, building in the synced checkout works and is deterministic against the same lockfile: