Make lc seamless on NERSC Perlmutter

[EiffL]

Closes the gaps the Perlmutter spike found (#192), so a researcher on NERSC can lc init and lc materialize with an empty environment and nothing to configure. Every change here was verified by running it on Perlmutter, on a 2-node interactive allocation.

The problems, and the fixes

uv's home-filesystem state is unusable from compute nodes — in two independent ways. flock on $HOME fails outright (os error 524), so every uv-touching verb died until the user exported UV_CACHE_DIR. And an interpreter stored there starts slowly enough to blow uv's own 60 s ceiling and fail a sync mid-run with Python startup timed out. Measured on the same compute node, 5 runs each:

interpreter store	python -c pass
$SCRATCH (Lustre)	0.22, 0.06, 0.05, 0.05, 0.05 s
$HOME (GPFS)	9.21, 1.22, 1.38, 1.08, 0.41 s

So venue.site_env() supplies both UV_CACHE_DIR and UV_PYTHON_INSTALL_DIR from $SCRATCH/.lightcone/ on a known center — spec §4's "the site registry supplies UV_CACHE_DIR on Perlmutter" row, finally made real, and _UV_KEPT already kept both variables for exactly this story. Per variable, ambient always wins. Silent by design: a default doing its job is not a warning. Deterministic in the ambient environment, so every node of an allocation derives the same answer with nothing handed down.

An ambient MOUNT_HOME=1 silently bind-mounted $HOME into every recipe container while the manifest attested fs: declared. podman-hpc's site modules are env-gated, so child_env now scrubs MOUNT_*. ENABLE_* is deliberately kept — those bind CUDA and MPI system libraries, the same generosity class as the /dev,/sys grants, and a GPU recipe on Perlmutter reaches CUDA through exactly that mechanism. One predicate in the one env composer covers both wrapper-spawn paths, so no Backend protocol change was needed. uv_scrub_warning → scrub_warning, naming both families.

Rebuilding an image under its own tag wedged the shared squash store. The tag is deterministic but builds are not bit-reproducible, so a rebuild migrated under an unchanged tag put a second same-named image into podman-hpc's read-only store — after which every storage operation failed, runs included. container._heal_squash probes and rmsqis the stale copy before the load (a wedged store fails the load too). An unrecognized probe outcome is left for migrate's own loud path, so the heal can never break a healthy run.

Every clean SLURM run ended looking like a crash — srun: forcing job termination, task N: Killed, and scheduler lose-tasks warnings. Cause: the driver held every future for the run's lifetime, so retire_workers saw each key as client-wanted and tried to replicate it onto peers that were also retiring, stalling past the reap grace until srun was killed. _Dask.completed now releases each future as its result lands. Warnings on an unclean path still fire, honestly.

Also: the containerized sync now creates uv's cache directory before bind-mounting it — in that mode uv never runs on the host, so nothing else makes it and podman died with a raw statfs error about a path the user never named (latent before; the cache supply made it reachable). And the Landlock downgrade note names the boot-time LSM list first: Perlmutter is kernel 6.4 with Landlock simply absent from that list, so the old text offered three causes and none of them applied.

Verification

Live, on nid[004167-004168]:

Check	Result
lc init on a compute node, empty environment	works — the flock wall is gone; cache and interpreters land on $SCRATCH/.lightcone/
Project venv interpreter startup	0.04–0.05 s (was 0.4–9.2 s)
Direct 4-universe multi-node materialize	outputs spread across both nodes, clean tree, zero teardown noise
Containerized multi-node materialize	in-image on both nodes, mechanism: podman-hpc, fs: declared, image id pinned
Squash heal	a manufactured duplicate-name wedge (which failed every podman operation) healed and ran clean; it also fired naturally on a later rebuild
MOUNT_HOME=1 lc run ls $HOME	denied + the warning naming the drop. Mutation-checked: bare podman-hpc with the same variable does mount it
Fresh containerized first run, nothing exported	lc init → lc materialize → committed result in 61 s, clean tree

The future-release fix also removed a suite flake and tripled suite speed on a compute node — same suite, same node, only the interpreter store moved: 9 failed / 42:16 → 568 passed, 0 failed / 13:45.

Notes for reviewers

• conftest scrubs NERSC_HOST suite-wide (deliberately — the suite must exercise the non-site path), so the tests do not get the interpreter fix automatically. Anyone running pytest on a NERSC compute node should export UV_PYTHON_INSTALL_DIR=$SCRATCH/.lightcone/uv-python or expect the flake; real users get it from child_env.
• project now imports venue lazily inside child_env (venue imports ProjectError at module scope). The alternatives each broke a recorded invariant — moving the site table would contradict "supporting another center is one row and nothing else moves", and merging at call sites would spread env composition to three places.
• --ignore=tests/test_container_smoke.py silently does nothing in this repo (testpaths wins); --deselect is the spelling that works.
• Out of scope by decision, unchanged: the smoke suite's _inspect_id KeyError: 'podman-hpc' and the bare-podman-on-network-filesystem failures (both from Perlmutter spike: login-node findings (layer 7) #192's first half), and OMP_NUM_THREADS, which is documented rather than touched — lc does not take over a variable it does not own.

Docs land with the code: a "NERSC (Perlmutter) specifics" section in the cluster guide, the duplicate-squash entry in troubleshooting, and four API pages. CLAUDE.md carries the decisions.

🤖 Generated with Claude Code

https://claude.ai/code/session_01DyNDEbtjpob2CJSQMnEiuf

[github-actions]

✅ Eval

Metric	Value
Outputs check	success
Agent run	success
Turns	29
Tool calls	27
Cost	$0.72
Agent wall time	2m13s
Model	claude-sonnet-5

lc status

  mode:    direct
  sandbox: landlock (fs: declared, network: allowed)
  crate:   up to date with the outputs

  · current  baseline/best_fit        1c4293c
  · current  baseline/hubble_diagram  1c4293c
  · current  baseline/residuals       1c4293c

3 current

Confusion & pain points (Claude analysis)

Confusion & pain points

• The initial recon command failed (exit 2) because it chained ls universes/, cat data/..., ls scripts/, git status, and lc status together in one && pipeline — lc status (or an earlier segment) errored out and the agent lost visibility into the tail of the output (git status/lc status results were truncated/never seen). It had to re-run a second, similar combined probe to get pyproject.toml, git status, and lc status output cleanly. Batching many unrelated diagnostic commands with && is fragile; separate calls (or ;) would have surfaced the actual failure instead of masking it.
• The agent spent two full tool calls hand-rolling awk/grep/sort sanity checks against the raw data file (counting unique values, checking min/max redshift, verifying the z-cut boundary) rather than trusting the spec/data README or writing this into the analysis script itself — a manual data-exploration detour outside any tool the harness provided for inspecting declared inputs.
• No shared-module mechanism exists in the recipe grammar for common code (e.g., the LCDM distance-modulus model used by the fit and both plotting scripts), forcing the agent to duplicate the model function verbatim across fit.py, plot_hubble.py, and plot_residuals.py. This is a real spec/product gap worth flagging: recipes can't import a project-local helper module cleanly within the declared input/output contract, so any shared logic must be copy-pasted.
• The license/publication-view mechanic ([project].license in pyproject.toml silently gating RO-Crate generation) was only discovered after the first lc materialize, which printed a ! warning about it — the agent hadn't anticipated this from the spec and had to make a second commit+materialize pass specifically to enable publication. This is a legitimate two-pass workflow given how the tool surfaces it, but it wasn't discoverable up front from astra.yaml or the scaffold docs.
• Otherwise the run was clean: no failed lc/astra/uv/git commands past the initial recon hiccup, no sandbox denials, and astra validate / lc materialize --check / lc status all passed on the first real attempt.

Full trace: agent-trace artifact on this run.