Skip to content

fix(docker): stop shipping empty pixi environments - #368

Merged
xraymemory merged 1 commit into
diff-use:mainfrom
manzuoni-astera:michaelanzuoni/fix-empty-pixi-envs
Aug 12, 2026
Merged

fix(docker): stop shipping empty pixi environments#368
xraymemory merged 1 commit into
diff-use:mainfrom
manzuoni-astera:michaelanzuoni/fix-empty-pixi-envs

Conversation

@manzuoni-astera

@manzuoni-astera manzuoni-astera commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

Published images ship all five pixi environments as empty shells. Reported by @feng-yu and @marcuscollins looking for the prebuilt protenix env.

/app/.pixi/envs/protenix   20K   conda-meta: history, pixi, pixi_env_prefix   <- no package records
/app/.pixi/envs/protenix/bin/python                                            <- does not exist
/app total: 1.6 MB          (alongside 12 GB of checkpoints in the same image)

Important: the Dockerfile is not the bug

I built the real thing to check — pixi install -e protenix --frozen against the real pixi.lock, on linux/amd64:

The protenix environment has been installed.        (349 s)
13G  /app/.pixi/envs/protenix
240  conda-meta entries
bin/python -> python3.12    →  runs, 3.12.13

and it survives into the exported image, not just the build. I also poisoned the prefix to look exactly like the shipped one (metadata present, zero packages, matching lock hash) and the current step still repaired it to a full 13 GB env.

So the build instructions are correct, and this PR is not a root-cause fix. I could not reproduce the failure in any local configuration — including three mechanisms I tested and disproved (rattler hardlinking out of a cache mount, a missing __cuda virtual package, and pixi state cached across builds).

What this PR does

Given the above, it is insurance rather than a cure:

  • A per-environment interpreter check. pixi install reporting success is demonstrably not evidence that anything was installed — that is exactly how five empty environments shipped. This turns that into a build failure. Verified it fails (exit code: 1) when the interpreter is missing.
  • rm -rf /app/.pixi/envs before installing, so a stale or stub prefix arriving from a cached layer can't influence the result.
  • Drops the /root/.cache/pixi mount, which carries pixi's own "is this environment current" state — the one cache whose contents can disagree with a prefix restored from a different build. The rattler and uv caches stay; they hold the downloads that make rebuilds fast, and I verified they don't change what lands in the layer.

What actually needs to happen

Since the Dockerfile is correct, the published image must have come from a build whose environment differed — and the remaining candidate I can't rule out is the registry buildcache (cache-from/cache-to: type=registry … mode=max) importing a layer with the metadata but not the package files. That would produce exactly this.

Someone with access should rebuild the image with :buildcache busted. I don't have dispatch rights on this repo. With this PR merged first, that rebuild either produces working environments or fails loudly instead of silently shipping empties.

Meanwhile

SAMPLEWORKS_REQUIRE_PREBUILT_PIXI=1 makes the runner refuse to fall back, so the image currently promises environments it doesn't carry. Until a good image exists, building in the synced checkout works and is deterministic against the same lockfile:

cd ~/workspace && pixi install -e protenix --frozen

Published images carry all five environments as empty shells: the
directories and pixi's own bookkeeping in conda-meta are present, but
there are no package records, no bin/, no lib/. /app/.pixi/envs/protenix
has no python, and all of /app is 1.6 MB against 12 GB of checkpoints.

The failure is silent in both directions. At build time `pixi install
--frozen` reports "The <env> environment has been installed" and exits 0
against such a prefix, so nothing fails. At runtime pixi trusts the same
state, so it will not repair itself, and because the image sets
SAMPLEWORKS_REQUIRE_PREBUILT_PIXI=1 the runner refuses to fall back —
the image promises environments it does not carry. Reported by two
scientists looking for the prebuilt protenix env.

Clearing the prefix first forces a genuine install even when a stale or
stub prefix arrives from a cached layer, the base image, or the registry
buildcache. The per-environment interpreter check then makes an empty
env fail the build rather than ship, which is the part that matters:
`pixi install` succeeding is not evidence that anything was installed.

Drops the /root/.cache/pixi mount, which carries pixi's own "is this
environment current" state — the thing that can disagree with a prefix
restored from a different build. The rattler and uv caches stay; they
hold the downloads that actually make rebuilds fast, and were verified
not to affect what lands in the layer.

Verified with minimal Docker builds: against a prefix poisoned to look
exactly like the shipped one, the fixed step produces a real 116 MB
environment with a working interpreter, and the assertion fails the
build when the interpreter is missing.
Copilot AI lite review requested due to automatic review settings August 11, 2026 21:44
@manzuoni-astera
manzuoni-astera requested review from a team and denis-zaitsev as code owners August 11, 2026 21:44
@coderabbitai

coderabbitai Bot commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

📝 Walkthrough

Walkthrough

The Dockerfile now clears existing Pixi environment prefixes, installs with frozen metadata, retains package and wheel caches, and verifies executable Python interpreters in five environments.

Changes

Pixi installation validation

Layer / File(s) Summary
Install and validate Pixi environments
Dockerfile
The Pixi installation removes existing environment directories, omits the Pixi metadata cache mount, and fails with an explicit diagnostic when any of the five environments lacks an executable Python interpreter.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

Suggested reviewers: mag-astera, abdelsalam-abbas

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely describes the main change: preventing Docker images from shipping empty Pixi environments.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes a Docker image build issue where pixi environments could be present only as “stub” prefixes (directories + minimal metadata) while pixi install --frozen still reported success, causing published images to ship without usable interpreters for model environments.

Changes:

  • Force a real pixi environment installation by clearing /app/.pixi/envs before installing all environments.
  • Add a post-install assertion that each environment contains an executable bin/python, failing the build if an env is empty.
  • Stop using the BuildKit cache mount for /root/.cache/pixi to avoid persisting pixi “current state” across builds.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread Dockerfile
echo " pixi reported success but installed nothing — refusing to ship an empty environment."; \
exit 1; \
}; \
done

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@Dockerfile`:
- Around line 159-164: Update the environment validation loop in the Dockerfile
to require each `/app/.pixi/envs/${env}/bin/python` path to be a regular
executable file, then invoke it with `-c` to validate its version is at least
3.11 and below 3.14, matching the pyproject.toml constraint; retain the existing
fatal error behavior when validation fails.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: a0a940d2-e53d-4250-8b41-799df6913276

📥 Commits

Reviewing files that changed from the base of the PR and between 16070a0 and d74739a.

📒 Files selected for processing (1)
  • Dockerfile

Comment thread Dockerfile
Comment on lines +159 to +164
for env in boltz protenix rf3 protpardelle analysis; do \
test -x "/app/.pixi/envs/${env}/bin/python" || { \
echo "FATAL: pixi environment '${env}' has no interpreter at /app/.pixi/envs/${env}/bin/python."; \
echo " pixi reported success but installed nothing — refusing to ship an empty environment."; \
exit 1; \
}; \

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/sh
set -eu
tmp="$(mktemp -d)"
trap 'rm -rf "$tmp"' EXIT

printf '#!/bin/sh\nexit 127\n' > "$tmp/python"
chmod +x "$tmp/python"

test -x "$tmp/python"

if "$tmp/python" -c 'import sys'; then
  echo "unexpected: fake interpreter ran"
  exit 1
fi

Repository: diff-use/sampleworks

Length of output: 158


🏁 Script executed:

#!/bin/sh
set -eu
printf '%s\n' '--- Dockerfile ---'
sed -n '145,170p' Dockerfile
printf '%s\n' '--- Python constraints ---'
rg -n -C 2 'requires-python|python' pyproject.toml
printf '%s\n' '--- Interpreter guard occurrences ---'
rg -n -C 3 'test -x|\.pixi/envs/.*/bin/python|pixi install' Dockerfile

Repository: diff-use/sampleworks

Length of output: 4818


Run each environment’s Python interpreter during validation.

test -x checks only execute permission. It can accept an executable directory or non-Python file. Add test -f, then run the interpreter with -c and enforce >= 3.11, < 3.14 from pyproject.toml.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@Dockerfile` around lines 159 - 164, Update the environment validation loop in
the Dockerfile to require each `/app/.pixi/envs/${env}/bin/python` path to be a
regular executable file, then invoke it with `-c` to validate its version is at
least 3.11 and below 3.14, matching the pyproject.toml constraint; retain the
existing fatal error behavior when validation fails.

Source: MCP tools

@xraymemory
xraymemory merged commit 40406c7 into diff-use:main Aug 12, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants