Make lc seamless on NERSC Perlmutter

CLAUDE.md

@@ -788,7 +788,7 @@ annotated: `project.child_env` scrubs ambient `UV_*` outside a plumbing
 allowlist (`_UV_KEPT` — cache dir, link mode, the managed-interpreter
 store and its mirror, timeouts, TLS, air-gap, credentials, uv's own
 recursion guard), and every uv-acting verb names the non-empty
-variables dropped through `project.uv_scrub_warning` — one composer,
+variables dropped through `project.scrub_warning` — one composer,
 one predicate with the scrub: convergence puts it in the report (so
 `lc init` says it), materialize in its warnings, and the probe in its
 outcome's notes, which is why the CLI never composes it (issue #179).
@@ -2141,6 +2141,90 @@ unlinks before writing; a new tampering test should too.
     materialize runs for hours) arrives with the submission-model
     venue.
 
+- **The NERSC seamlessness pass** (2026-08, post-spike — findings in
+  issue #192):
+  - *The site registry supplies uv's plumbing* — spec §4's row, made
+    real: `child_env` fills unset/empty `UV_CACHE_DIR` and
+    `UV_PYTHON_INSTALL_DIR` from `$SCRATCH/.lightcone/` on a known
+    center (`venue.site_env`; the `_Site` row gained `scratch`).
+    Both move state off a home filesystem measurement found unusable
+    from compute nodes, in two independent ways: `flock` fails there
+    outright (os error 524), killing every uv-touching verb, and an
+    interpreter stored there starts in 0.4–9 s rather than 0.05 s —
+    enough, under load, to exceed uv's own 60 s startup ceiling and
+    fail a sync mid-run with "Python startup timed out". Both are
+    disposable, so a scratch purge costs a re-download, never a
+    result; `_UV_KEPT` already kept both for exactly this story.
+    Per variable, ambient always wins; silent by design — the scrub
+    warning is for user configuration that stopped taking effect, not
+    for a default doing its job. Deterministic in the ambient
+    environment, so every node of an allocation derives the same
+    answer with nothing handed down.
+  - *`MOUNT_*` is scrubbed; `ENABLE_*` is kept and **attested**.*
+    podman-hpc's site modules are env-gated. The mount gates bind
+    `$HOME`/`$SCRATCH`/`$CFS` into every container — undeclared inputs
+    under a manifest attesting `fs: declared` (measured) — so they are
+    dropped, and named by `scrub_warning` (the `uv_scrub_warning`
+    rename). The `ENABLE_*` gates are **not** scrubbed: they are how a
+    GPU or MPI recipe reaches its hardware, and removing them would
+    take a working capability away with no replacement.
+    But the first draft kept them on the stated ground that they bind
+    "system libraries only, never a channel undeclared inputs arrive
+    through", and review measured that false: `ENABLE_CVMFS` binds the
+    whole populated `/cvmfs` hierarchy (reference data a recipe can
+    read undeclared) and `ENABLE_MPICH_SS` adds `--privileged` with the
+    host's network, pid and ipc namespaces — the mount table stops
+    being the boundary at all. Only `ENABLE_GPU`/`ENABLE_NCCL*` fit the
+    original description. So the honesty comes from the record instead:
+    `container.site_modules()` names the set gates, `OCIBackend` carries
+    them as a resolved field (the `user_flags` pattern — `wrap` stays
+    pure), and `Attestation.site_modules` puts them in every manifest.
+    The gate names are **asked of the site's module table** (each
+    module declares its own `env:` key; the directory comes from
+    `podman-hpc infohpc`), never matched by prefix: CI caught a prefix
+    match recording GitHub's own `ENABLE_RUNNER_TRACING` as a module
+    the host does not have, which is the exact dishonesty the field
+    exists to prevent. Unreadable table ⇒ empty, never a refusal —
+    attestation must not fail a run, the `uv_version` discipline.
+    This is the one attestation value not derived from lc's own argv,
+    and the docstring says so: the runtime applies them from its own
+    environment. A per-module allowlist was rejected — the module table
+    is site-configurable, so a center adding a data-binding module lc
+    has never heard of would slip through a hardcoded judgement.
+  - *A stale squashed image is healed before the load.* The tag is
+    deterministic, builds are not bit-reproducible, and podman-hpc's
+    read-only squash store refuses two images under one name — after
+    which *every* storage operation fails (measured: a same-tag
+    rebuild wedged the store for runs, listings, everything).
+    `container._heal_squash` probes and `rmsqi`s the stale copies; an
+    unrecognized probe outcome is left for migrate's own loud path,
+    so the heal cannot break a healthy run. Before the *load*, not
+    just the migrate — a wedged store fails the load too on a node
+    that has not loaded yet. Removal is **by id, one call per stale
+    image**: `rmsqi <tag>` resolves a single record (podman-hpc's
+    `remove_image` → `get_img_info` takes the first name match), so
+    against the very state being healed it can take the *current*
+    image and leave the stale one to re-wedge the store on the next
+    migrate. Only the wedged branch spells the tag — a store that
+    cannot be listed cannot be enumerated — and it re-probes, bounded
+    by `_HEAL_ATTEMPTS`, because each blind pass is a real deletion.
+    Recorded hazard, accepted: the squash store is shared across a
+    user's nodes and allocations, so a heal can drop layers an
+    earlier run is still executing from. Leaving the store wedged
+    breaks that run too, and cross-run coordination is not something
+    a heal can offer.
+  - *The driver releases each future as its result is consumed*
+    (`_Dask.completed`). A key still client-held at teardown makes
+    `retire_workers` replicate it onto peers that are also retiring —
+    every clean SLURM run stalled past the reap grace and ended in
+    "srun: forcing job termination" / "task Killed" / lose-tasks
+    warnings (measured, and measured gone: the fix cut a clean 2-node
+    run by the full escalation grace). Warnings on an *unclean* path
+    still fire, honestly.
+  - The downgrade note now names the LSM list first — Perlmutter's
+    kernels are 6.4 with Landlock simply absent from the boot list,
+    so "kernel < 5.13" was three causes none of which applied.
+
 ### Recorded deviations from the spec
 
 - **No `AGENTS.md` scaffolding** (spec §2 calls for an agent notes

docs/api/container.md

@@ -59,6 +59,31 @@ Sources: `src/lightcone/engine/image.py`,
   outside the load branch) and joins `_SHARED_STORE_RUNTIMES`.
   Detection order podman-hpc → podman → docker; docker's daemon is
   probed at detection.
+- **Site container modules are named, not silenced.** `site_modules()`
+  reports the gates set for podman-hpc, which the runtime applies from
+  its own environment. The gate names are **read from the site's own
+  module table** (each module declares its `env:` key; the directory
+  comes from `podman-hpc infohpc`), never matched by prefix — GitHub's
+  `ENABLE_RUNNER_TRACING` was enough to make a prefix match name a
+  module the host does not have. A table that cannot be read reports
+  nothing rather than refusing: attestation must not fail a run.
+  They are recorded because a module widens the container past the
+  mount table — `ENABLE_CVMFS` binds `/cvmfs`, `ENABLE_MPICH_SS` adds
+  `--privileged` and the host namespaces — so `Attestation.site_modules`
+  carries them into every manifest rather than leaving `fs: declared`
+  to overstate the boundary. `MOUNT_*` never appears there:
+  `project.child_env` scrubs it before the runtime sees it.
+- **A stale squashed image is healed before the load.** The tag is
+  deterministic but builds are not bit-reproducible, so a rebuild
+  migrated under an unchanged tag would put a second same-named image
+  into podman-hpc's read-only squash store — after which every storage
+  operation fails. `_heal_squash` probes the store and `rmsqi`s each
+  stale copy **by id** first (before the load, which a wedged store
+  also fails) — `rmsqi <tag>` resolves only one record and could take
+  the current image instead. A store too wedged to list is removed by
+  tag and re-probed, bounded by `_HEAL_ATTEMPTS`; a probe outcome the
+  heal does not recognize is left for migrate's own loud path, so it
+  can never break a healthy run.
 - **The architecture gate refuses before the load** — a wrong-arch
   `load` succeeds and then dies as `exec format error` deep inside a
   recipe. Ignorance passes; a recorded mismatch refuses, naming the

docs/api/materialize.md

@@ -15,7 +15,7 @@ Source: `src/lightcone/engine/materialize.py`.
 | `check(root, targets, *, refresh)` | The same classification without executing, committing, or fetching. Exempt from the dirty refusal. |
 | `status(root)` | The report: every output's state and provenance commit, plus the mode/image/sandbox header facts. |
 | `MaterializeReport` / `StatusReport` | The JSON surfaces; `ok` and `up_to_date` first. |
-| `cluster_for_run()` | The venue ladder, and the two-method scheduler seam (`submit`, `completed`). |
+| `cluster_for_run()` | The venue ladder, and the two-method scheduler seam (`submit`, `completed`). `completed` releases each future as its result lands — a key still held at teardown makes the scheduler fight worker retirement, and every clean SLURM run ends in "srun: forcing job termination". |
 | `run_record(...)` / `datalad_run_subject(...)` | The commit message `datalad rerun` replays, and the one spelling of its subject line — shared with the foreign-write comparator, because two strings here would drift. |
 | `_engine_requirement()` | How a record pins its engine: by version for a release, by source commit (hatch-vcs) for a dev build. |
 

docs/api/project.md

@@ -19,6 +19,7 @@ Source: `src/lightcone/engine/project.py` (+
 | `uv_prefix(root, *, sync)` | The one spelling of the project uv hop. Callers differ only in `sync`: a probe converges the environment, a recipe must not. |
 | `project_name(dir)` | PEP 503-ish name from the directory name. |
 | `_run` / `_check_call` | Every external tool invocation, and the suite's one monkeypatch point. |
+| `child_env()` / `scrub_warning()` | The one composer of the environment external tools run in: drops `VIRTUAL_ENV`, ambient `UV_*` outside the `_UV_KEPT` plumbing allowlist, and `MOUNT_*` (a site container module's mount gates); supplies a known center's uv plumbing (`UV_CACHE_DIR`, `UV_PYTHON_INSTALL_DIR`) from its scratch root where unset (`venue.site_env`). The warning names every non-empty variable dropped. |
 | `ProjectError` | The engine's one exception; the CLI translates it once. |
 
 ## What must stay true

docs/api/venue.md

@@ -17,7 +17,7 @@ Source: `src/lightcone/engine/venue.py` (consumed by
 | `slurm_client()` | The allocation branch: a scheduler in the driver process bound to `SLURMD_NODENAME`, one `srun --overlap` launching a worker per node on `sys.executable`. |
 | `require_compute_node(command)` | The login guard: refuses iff a known center's marker is set and `SLURM_JOB_ID` is not, printing that center's own `salloc`/`sbatch` spellings. |
 | `allocation_nodes()` | How many nodes the allocation holds; 0 outside one. |
-| `_SITES` | One row per known center — name, marker, remedies, **verified against the center's documentation, never guessed**. NERSC is the seeded row. |
+| `_SITES` | One row per known center — name, marker, remedies, and the scratch variable that hosts uv's cache and interpreter store where home cannot (`site_env`, consumed by `project.child_env`), **verified against the center's documentation, never guessed**. NERSC is the seeded row. |
 
 ## What must stay true
 

docs/user/cluster.md

@@ -106,6 +106,70 @@ know:
   refuses a multi-node containerized run unless the runtime is
   `podman-hpc`. Single-node allocations work with any runtime.
 
+## NERSC (Perlmutter) specifics
+
+Facts of the site that shape how `lc` behaves there — all verified on
+Perlmutter itself:
+
+- **uv's cache and interpreters are put on scratch for you.** The
+  home filesystem is unusable from compute nodes in two separate ways:
+  it cannot take uv's cache lock at all (`flock` fails with os error
+  524), and an interpreter stored there starts in 0.4-9 s instead of
+  0.05 s, slow enough under load to exceed uv's own 60 s startup
+  ceiling and fail a sync mid-run. So on NERSC machines `lc` runs uv
+  with `UV_CACHE_DIR=$SCRATCH/.lightcone/uv-cache` and
+  `UV_PYTHON_INSTALL_DIR=$SCRATCH/.lightcone/uv-python` — each only
+  where you have not set that variable yourself:
+
+  ```bash
+  export UV_CACHE_DIR=$SCRATCH/my-uv-cache   # optional; this wins
+  ```
+
+  Both are disposable by design, so scratch purge policies cost a
+  re-download rather than a result.
+
+- **The sandbox does not enforce here.** Perlmutter's kernels boot
+  without Landlock in the LSM list, so direct-mode recipes run
+  unconfined and every run says so:
+
+  ```text
+  not sandboxed on this host — landlock unavailable (not in the kernel's boot-time LSM list, kernel < 5.13, blocked by seccomp, or unsupported arch); recorded as `fs: open`
+  ```
+
+  Manifests record `fs: open` honestly. A containerized project is
+  the real boundary on this site: recipes see only the declared mount
+  table, attested as `fs: declared`.
+
+- **`MOUNT_*` is ignored; `ENABLE_*` passes through.** podman-hpc's
+  site modules are switched by environment variables. The mount gates
+  (`MOUNT_HOME`, `MOUNT_SCRATCH`, `MOUNT_CFS`) would bind-mount those
+  filesystems into every recipe container — undeclared inputs under a
+  manifest attesting `fs: declared` — so `lc` drops them and says so:
+
+  ```text
+  ignored ambient MOUNT_HOME — a site container module would bind-mount undeclared host directories into recipe containers
+  ```
+
+  The `ENABLE_*` gates survive, because they are how a GPU or MPI
+  recipe reaches the hardware it was written for — a GPU recipe gets
+  CUDA by exporting `ENABLE_GPU=1`, the site's own mechanism. They are
+  **recorded in the manifest instead**, under
+  `hermeticity.site_modules`, because a module can widen the container
+  well past the mounts `lc` declared: `ENABLE_CVMFS` binds the whole
+  `/cvmfs` hierarchy, and `ENABLE_MPICH_SS` adds `--privileged` plus
+  the host's network, pid and ipc namespaces. If you enable one, the
+  outputs of that run say so.
+
+- **BLAS may be silently throttled.** NERSC's allocation environment
+  exports `OMP_NUM_THREADS` (often `2`), and `srun` propagates it into
+  every step — so numpy or BLAS inside a recipe uses 2 of the node's
+  256 hardware threads unless you say otherwise. `lc` does not touch
+  the variable; set it yourself before materializing:
+
+  ```bash
+  export OMP_NUM_THREADS=32   # or whatever your recipe's math wants
+  ```
+
 ## Data on parallel filesystems
 
 Keep active projects on the filesystem your center recommends for job

docs/user/troubleshooting.md

@@ -190,6 +190,12 @@ git config --global user.email "ada@example.org"
   refused before the recipe would have died mid-run. Build on a
   matching host (on NERSC, a login node), commit, push, and pull on
   the other side.
+- **"read-only image store assigns the same name to multiple images"**
+  (podman-hpc) — two builds of the same image tag landed in the shared
+  squash store, and podman then refuses every storage operation. `lc`
+  heals this itself before each migrate, so a run clears it; if a
+  store was wedged by something other than `lc`, the manual remedy is
+  `podman-hpc rmsqi <tag>` for the name podman's error names.
 
 ## Filing a bug
 

src/lightcone/_sandbox_exec.py

@@ -305,7 +305,10 @@ def main(argv: list[str] | None = None) -> None:
             raise ValueError("policy must be a JSON object")
         abi_level = abi()
         if abi_level == 0:
-            raise ValueError("landlock unavailable (kernel < 5.13, or blocked by seccomp)")
+            raise ValueError(
+                "landlock unavailable (not in the kernel's boot-time LSM list, "
+                "kernel < 5.13, or blocked by seccomp)"
+            )
         fd = build_ruleset(policy, abi_level)
         restrict_self(fd)
         os.close(fd)