From eee49c616281add3e87e0e507c1b5acea2cd156a Mon Sep 17 00:00:00 2001 From: EiffL Date: Fri, 21 Aug 2026 07:45:24 -0700 Subject: [PATCH 1/3] Make lc seamless on NERSC Perlmutter MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The Perlmutter spike (#192) proved both materialize modes work multi-node, then found four things that made real use rough. Each is fixed here and verified by running it on the machine. The site registry supplies uv's plumbing. On a known center, unset UV_CACHE_DIR and UV_PYTHON_INSTALL_DIR are filled from $SCRATCH/.lightcone/ — spec §4's row, made real. Both move state off a home filesystem that measurement found unusable from compute nodes, in two independent ways: flock fails there outright (os error 524), which killed every uv-touching verb, and an interpreter stored there starts in 0.4–9 s rather than 0.05 s, enough under load to exceed uv's own 60 s ceiling and fail a sync mid-run. Per variable, ambient always wins; silent by design, and deterministic in the ambient environment, so every node of an allocation derives the same answer with nothing handed down. MOUNT_* is scrubbed; ENABLE_* is kept. podman-hpc's site modules are env-gated, and the mount gates bind $HOME/$SCRATCH/$CFS into every recipe container — undeclared inputs under a manifest attesting fs: declared. The library gates stay: CUDA and MPI system libraries are the /dev,/sys generosity class, and the site's own GPU mechanism has to keep working. One predicate in the one env composer covers both wrapper-spawn paths; uv_scrub_warning becomes scrub_warning and names both families. A stale squashed image is healed before the load. The tag is deterministic and builds are not bit-reproducible, so a rebuild migrated under an unchanged tag put a second same-named image into podman-hpc's read-only squash store, after which every storage operation failed. An unrecognized probe outcome is left for migrate's own loud path, so the heal cannot break a healthy run. The driver releases each future as its result is consumed. A key still client-held at teardown makes retire_workers replicate it onto peers that are also retiring, so the run stalled past the reap grace and srun was killed: every clean SLURM run ended in "srun: forcing job termination". Warnings on an unclean path still fire. Also: the containerized sync creates uv's cache directory before bind-mounting it (nothing on the host makes it in that mode, so podman died with a raw statfs error about a path the user never named), and the Landlock downgrade note names the boot-time LSM list first — Perlmutter is kernel 6.4 with Landlock simply absent from it. Verified on a 2-node interactive allocation: a fresh project with an empty environment goes from lc init to a committed result with no setup; direct and containerized multi-node runs spread across both nodes and end silently; a manufactured duplicate-name wedge healed and ran clean. Full suite on a compute node: 568 passed, 0 failed. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01DyNDEbtjpob2CJSQMnEiuf --- CLAUDE.md | 51 ++++++++++- docs/api/container.md | 8 ++ docs/api/materialize.md | 2 +- docs/api/project.md | 1 + docs/api/venue.md | 2 +- docs/user/cluster.md | 58 ++++++++++++ docs/user/troubleshooting.md | 6 ++ src/lightcone/_sandbox_exec.py | 5 +- src/lightcone/engine/container.py | 40 ++++++++ src/lightcone/engine/materialize.py | 13 ++- src/lightcone/engine/project.py | 67 +++++++++++--- src/lightcone/engine/run.py | 4 +- src/lightcone/engine/sandbox/landlock.py | 5 +- src/lightcone/engine/venue.py | 41 +++++++++ tests/conftest.py | 11 ++- tests/test_container.py | 112 ++++++++++++++++++++++- tests/test_materialize.py | 30 ++++++ tests/test_project.py | 82 ++++++++++++++++- 18 files changed, 502 insertions(+), 36 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 7c73800b..459f7079 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -731,7 +731,7 @@ annotated: `project.child_env` scrubs ambient `UV_*` outside a plumbing allowlist (`_UV_KEPT` — cache dir, link mode, the managed-interpreter store and its mirror, timeouts, TLS, air-gap, credentials, uv's own recursion guard), and every uv-acting verb names the non-empty -variables dropped through `project.uv_scrub_warning` — one composer, +variables dropped through `project.scrub_warning` — one composer, one predicate with the scrub: convergence puts it in the report (so `lc init` says it), materialize in its warnings, and the probe in its outcome's notes, which is why the CLI never composes it (issue #179). @@ -2084,6 +2084,55 @@ unlinks before writing; a new tampering test should too. materialize runs for hours) arrives with the submission-model venue. +- **The NERSC seamlessness pass** (2026-08, post-spike — findings in + issue #192): + - *The site registry supplies uv's plumbing* — spec §4's row, made + real: `child_env` fills unset/empty `UV_CACHE_DIR` and + `UV_PYTHON_INSTALL_DIR` from `$SCRATCH/.lightcone/` on a known + center (`venue.site_env`; the `_Site` row gained `scratch`). + Both move state off a home filesystem measurement found unusable + from compute nodes, in two independent ways: `flock` fails there + outright (os error 524), killing every uv-touching verb, and an + interpreter stored there starts in 0.4–9 s rather than 0.05 s — + enough, under load, to exceed uv's own 60 s startup ceiling and + fail a sync mid-run with "Python startup timed out". Both are + disposable, so a scratch purge costs a re-download, never a + result; `_UV_KEPT` already kept both for exactly this story. + Per variable, ambient always wins; silent by design — the scrub + warning is for user configuration that stopped taking effect, not + for a default doing its job. Deterministic in the ambient + environment, so every node of an allocation derives the same + answer with nothing handed down. + - *`MOUNT_*` is scrubbed; `ENABLE_*` is kept.* podman-hpc's site + modules are env-gated; the mount gates bind `$HOME`/`$SCRATCH`/ + `$CFS` into every container — undeclared inputs under a manifest + attesting `fs: declared` (measured; the one item the layer-7 spike + said could add a flag, resolved without one). The library gates + stay: CUDA/MPI system libs are the `/dev`,`/sys` generosity class, + and the site's own GPU mechanism must keep working. Drops are + named by `scrub_warning` (the `uv_scrub_warning` rename). + - *A stale squashed image is healed before the load.* The tag is + deterministic, builds are not bit-reproducible, and podman-hpc's + read-only squash store refuses two images under one name — after + which *every* storage operation fails (measured: a same-tag + rebuild wedged the store for runs, listings, everything). + `container._heal_squash` probes and `rmsqi`s the stale copy; an + unrecognized probe outcome is left for migrate's own loud path, + so the heal cannot break a healthy run. Before the *load*, not + just the migrate — a wedged store fails the load too on a node + that has not loaded yet. + - *The driver releases each future as its result is consumed* + (`_Dask.completed`). A key still client-held at teardown makes + `retire_workers` replicate it onto peers that are also retiring — + every clean SLURM run stalled past the reap grace and ended in + "srun: forcing job termination" / "task Killed" / lose-tasks + warnings (measured, and measured gone: the fix cut a clean 2-node + run by the full escalation grace). Warnings on an *unclean* path + still fire, honestly. + - The downgrade note now names the LSM list first — Perlmutter's + kernels are 6.4 with Landlock simply absent from the boot list, + so "kernel < 5.13" was three causes none of which applied. + ### Recorded deviations from the spec - **No `AGENTS.md` scaffolding** (spec §2 calls for an agent notes diff --git a/docs/api/container.md b/docs/api/container.md index 899f5164..97f76104 100644 --- a/docs/api/container.md +++ b/docs/api/container.md @@ -59,6 +59,14 @@ Sources: `src/lightcone/engine/image.py`, outside the load branch) and joins `_SHARED_STORE_RUNTIMES`. Detection order podman-hpc → podman → docker; docker's daemon is probed at detection. +- **A stale squashed image is healed before the load.** The tag is + deterministic but builds are not bit-reproducible, so a rebuild + migrated under an unchanged tag would put a second same-named image + into podman-hpc's read-only squash store — after which every storage + operation fails. `_heal_squash` probes the store and `rmsqi`s the + stale copy first (before the load, which a wedged store also fails); + a probe outcome it does not recognize is left for migrate's own loud + path, so the heal can never break a healthy run. - **The architecture gate refuses before the load** — a wrong-arch `load` succeeds and then dies as `exec format error` deep inside a recipe. Ignorance passes; a recorded mismatch refuses, naming the diff --git a/docs/api/materialize.md b/docs/api/materialize.md index 54b570de..8428093b 100644 --- a/docs/api/materialize.md +++ b/docs/api/materialize.md @@ -15,7 +15,7 @@ Source: `src/lightcone/engine/materialize.py`. | `check(root, targets, *, refresh)` | The same classification without executing, committing, or fetching. Exempt from the dirty refusal. | | `status(root)` | The report: every output's state and provenance commit, plus the mode/image/sandbox header facts. | | `MaterializeReport` / `StatusReport` | The JSON surfaces; `ok` and `up_to_date` first. | -| `cluster_for_run()` | The venue ladder, and the two-method scheduler seam (`submit`, `completed`). | +| `cluster_for_run()` | The venue ladder, and the two-method scheduler seam (`submit`, `completed`). `completed` releases each future as its result lands — a key still held at teardown makes the scheduler fight worker retirement, and every clean SLURM run ends in "srun: forcing job termination". | | `run_record(...)` / `datalad_run_subject(...)` | The commit message `datalad rerun` replays, and the one spelling of its subject line — shared with the foreign-write comparator, because two strings here would drift. | | `_engine_requirement()` | How a record pins its engine: by version for a release, by source commit (hatch-vcs) for a dev build. | diff --git a/docs/api/project.md b/docs/api/project.md index a8c9f027..ff3b1440 100644 --- a/docs/api/project.md +++ b/docs/api/project.md @@ -19,6 +19,7 @@ Source: `src/lightcone/engine/project.py` (+ | `uv_prefix(root, *, sync)` | The one spelling of the project uv hop. Callers differ only in `sync`: a probe converges the environment, a recipe must not. | | `project_name(dir)` | PEP 503-ish name from the directory name. | | `_run` / `_check_call` | Every external tool invocation, and the suite's one monkeypatch point. | +| `child_env()` / `scrub_warning()` | The one composer of the environment external tools run in: drops `VIRTUAL_ENV`, ambient `UV_*` outside the `_UV_KEPT` plumbing allowlist, and `MOUNT_*` (a site container module's mount gates); supplies a known center's uv plumbing (`UV_CACHE_DIR`, `UV_PYTHON_INSTALL_DIR`) from its scratch root where unset (`venue.site_env`). The warning names every non-empty variable dropped. | | `ProjectError` | The engine's one exception; the CLI translates it once. | ## What must stay true diff --git a/docs/api/venue.md b/docs/api/venue.md index d86735dc..6fca2ef3 100644 --- a/docs/api/venue.md +++ b/docs/api/venue.md @@ -17,7 +17,7 @@ Source: `src/lightcone/engine/venue.py` (consumed by | `slurm_client()` | The allocation branch: a scheduler in the driver process bound to `SLURMD_NODENAME`, one `srun --overlap` launching a worker per node on `sys.executable`. | | `require_compute_node(command)` | The login guard: refuses iff a known center's marker is set and `SLURM_JOB_ID` is not, printing that center's own `salloc`/`sbatch` spellings. | | `allocation_nodes()` | How many nodes the allocation holds; 0 outside one. | -| `_SITES` | One row per known center — name, marker, remedies, **verified against the center's documentation, never guessed**. NERSC is the seeded row. | +| `_SITES` | One row per known center — name, marker, remedies, and the scratch variable that hosts uv's cache and interpreter store where home cannot (`site_env`, consumed by `project.child_env`), **verified against the center's documentation, never guessed**. NERSC is the seeded row. | ## What must stay true diff --git a/docs/user/cluster.md b/docs/user/cluster.md index f422897c..f02e9b64 100644 --- a/docs/user/cluster.md +++ b/docs/user/cluster.md @@ -106,6 +106,64 @@ know: refuses a multi-node containerized run unless the runtime is `podman-hpc`. Single-node allocations work with any runtime. +## NERSC (Perlmutter) specifics + +Facts of the site that shape how `lc` behaves there — all verified on +Perlmutter itself: + +- **uv's cache and interpreters are put on scratch for you.** The + home filesystem is unusable from compute nodes in two separate ways: + it cannot take uv's cache lock at all (`flock` fails with os error + 524), and an interpreter stored there starts in 0.4-9 s instead of + 0.05 s, slow enough under load to exceed uv's own 60 s startup + ceiling and fail a sync mid-run. So on NERSC machines `lc` runs uv + with `UV_CACHE_DIR=$SCRATCH/.lightcone/uv-cache` and + `UV_PYTHON_INSTALL_DIR=$SCRATCH/.lightcone/uv-python` — each only + where you have not set that variable yourself: + + ```bash + export UV_CACHE_DIR=$SCRATCH/my-uv-cache # optional; this wins + ``` + + Both are disposable by design, so scratch purge policies cost a + re-download rather than a result. + +- **The sandbox does not enforce here.** Perlmutter's kernels boot + without Landlock in the LSM list, so direct-mode recipes run + unconfined and every run says so: + + ```text + not sandboxed on this host — landlock unavailable (not in the kernel's boot-time LSM list, kernel < 5.13, blocked by seccomp, or unsupported arch); recorded as `fs: open` + ``` + + Manifests record `fs: open` honestly. A containerized project is + the real boundary on this site: recipes see only the declared mount + table, attested as `fs: declared`. + +- **`MOUNT_*` is ignored; `ENABLE_*` passes through.** podman-hpc's + site modules are switched by environment variables. The mount gates + (`MOUNT_HOME`, `MOUNT_SCRATCH`, `MOUNT_CFS`) would bind-mount those + filesystems into every recipe container — undeclared inputs under a + manifest attesting `fs: declared` — so `lc` drops them and says so: + + ```text + ignored ambient MOUNT_HOME — a site container module would bind-mount undeclared host directories into recipe containers + ``` + + The library gates (`ENABLE_GPU`, `ENABLE_MPICH_SS`, …) survive: a + GPU recipe gets CUDA by exporting `ENABLE_GPU=1`, the site's own + mechanism. + +- **BLAS may be silently throttled.** NERSC's allocation environment + exports `OMP_NUM_THREADS` (often `2`), and `srun` propagates it into + every step — so numpy or BLAS inside a recipe uses 2 of the node's + 256 hardware threads unless you say otherwise. `lc` does not touch + the variable; set it yourself before materializing: + + ```bash + export OMP_NUM_THREADS=32 # or whatever your recipe's math wants + ``` + ## Data on parallel filesystems Keep active projects on the filesystem your center recommends for job diff --git a/docs/user/troubleshooting.md b/docs/user/troubleshooting.md index e13edf62..2f7fbdc5 100644 --- a/docs/user/troubleshooting.md +++ b/docs/user/troubleshooting.md @@ -147,6 +147,12 @@ git config --global user.email "ada@example.org" refused before the recipe would have died mid-run. Build on a matching host (on NERSC, a login node), commit, push, and pull on the other side. +- **"read-only image store assigns the same name to multiple images"** + (podman-hpc) — two builds of the same image tag landed in the shared + squash store, and podman then refuses every storage operation. `lc` + heals this itself before each migrate, so a run clears it; if a + store was wedged by something other than `lc`, the manual remedy is + `podman-hpc rmsqi ` for the name podman's error names. ## Filing a bug diff --git a/src/lightcone/_sandbox_exec.py b/src/lightcone/_sandbox_exec.py index 9ea94cea..b151c21e 100644 --- a/src/lightcone/_sandbox_exec.py +++ b/src/lightcone/_sandbox_exec.py @@ -305,7 +305,10 @@ def main(argv: list[str] | None = None) -> None: raise ValueError("policy must be a JSON object") abi_level = abi() if abi_level == 0: - raise ValueError("landlock unavailable (kernel < 5.13, or blocked by seccomp)") + raise ValueError( + "landlock unavailable (not in the kernel's boot-time LSM list, " + "kernel < 5.13, or blocked by seccomp)" + ) fd = build_ruleset(policy, abi_level) restrict_self(fd) os.close(fd) diff --git a/src/lightcone/engine/container.py b/src/lightcone/engine/container.py index 0a7b8540..f91ad0bc 100644 --- a/src/lightcone/engine/container.py +++ b/src/lightcone/engine/container.py @@ -134,6 +134,11 @@ def runtime_for_run(root: Path, *, build: bool) -> Runtime: _fetch(root, archive) image_id, arch = archive_identity(archive) _require_arch(root, archive, arch) # before the load — see its docstring + if name == "podman-hpc": + # Before the load, not just the migrate: a wedged squash store + # (see _heal_squash) fails the load too, on a node whose local + # store does not hold the image yet. + _heal_squash(root, tag, image_id) if not _loaded(root, name, image_id): _check_call([name, "load", "-i", str(archive)], cwd=root) if name == "podman-hpc": @@ -194,6 +199,36 @@ def _committed(archive: Path) -> bool: return archive.exists() or archive.is_symlink() +def _heal_squash(root: Path, tag: str, image_id: str) -> None: + """Remove a stale squashed image before migrating the current one. + + The tag is deterministic but builds are not bit-reproducible, so a + rebuild migrated under an unchanged tag puts a second image with the + same name into podman-hpc's read-only squash store — after which + *every* storage operation fails ("read-only image store assigns the + same name to multiple images"), runs included (measured on + Perlmutter). ``rmsqi`` removes only the squashed copy, and the + migrate that follows re-squashes the current id. A heal must never + break a run whose store is healthy, so a probe outcome it does not + recognize is left for migrate's own loud path. + """ + probe = project._run( + ["podman-hpc", "images", "--format", "{{.Id}} {{.ReadOnly}}", f"localhost/{tag}"], + cwd=root, + ) + if probe.returncode == 0: + rows = [line.split() for line in probe.stdout.splitlines()] + stale = any( + len(row) == 2 and row[1] == "true" and row[0] != image_id for row in rows + ) + else: + # A wedged store fails the listing too, with the same message + # every other operation gets — the shape a pre-heal lc left. + stale = "assigns the same name to multiple images" in probe.stderr + if stale: + _check_call(["podman-hpc", "rmsqi", tag], cwd=root) + + def runtime_name(root: Path) -> str: """Detect which container runtime this host offers. @@ -377,6 +412,11 @@ def sync(root: Path, runtime: Runtime) -> list[str]: if asked.returncode != 0: raise ProjectError(f"`uv cache dir` failed:\n{asked.stderr.strip()}") cache = asked.stdout.strip() + # A bind mount's source must exist — and in containerized mode uv + # never runs on the host, so nothing else has created it. Otherwise + # the first sync of a fresh project dies as the runtime's `statfs` + # error about a path the user never named. + Path(cache).mkdir(parents=True, exist_ok=True) argv = [ runtime.runtime, "run", "--rm", "--entrypoint", "", # Same reason as the exec boundary's flag: SELinux hosts refuse diff --git a/src/lightcone/engine/materialize.py b/src/lightcone/engine/materialize.py index 331d7b37..85177272 100644 --- a/src/lightcone/engine/materialize.py +++ b/src/lightcone/engine/materialize.py @@ -493,7 +493,7 @@ def materialize( project.require_git_annex() dataset.require_committer(root) report = MaterializeReport() - if warning := project.uv_scrub_warning(): + if warning := project.scrub_warning(): report.warnings.append(warning) # The dirty check comes before anything that writes: the image # converge below *commits*, and `dataset.save` stages scoped but @@ -698,7 +698,16 @@ def completed(self, handles: list[Any]) -> Iterator[worker.TaskResult]: # annotated rather than the module exempted. from distributed import as_completed - for _, result in as_completed(handles, with_results=True): # type: ignore[no-untyped-call] + for fut, result in as_completed(handles, with_results=True): # type: ignore[no-untyped-call] + # Released the moment its result is local: a future still + # held at teardown makes the scheduler preserve its key — + # retiring workers then replicate every key onto each other + # (all retiring), stall past the reap grace, and get killed, + # ending every clean SLURM run in "srun: forcing job + # termination". A dependent's claim on the data is + # scheduler-side and unaffected; an abort mid-loop still + # holds the rest, and warnings on an unclean path are honest. + fut.release() yield result diff --git a/src/lightcone/engine/project.py b/src/lightcone/engine/project.py index 836ca691..0489ab4d 100644 --- a/src/lightcone/engine/project.py +++ b/src/lightcone/engine/project.py @@ -198,7 +198,7 @@ def converge(directory: Path, *, write: bool = True) -> ConvergenceReport: require_git_annex() c = _Converger(write=write) - if warning := uv_scrub_warning(): + if warning := scrub_warning(): c.warn(warning) if write: @@ -744,33 +744,65 @@ def _uv_scrubbed(name: str) -> bool: ) +def _mount_scrubbed(name: str) -> bool: + """Decide whether one ambient variable is dropped by the mount scrub. + + podman-hpc's site modules are gated on ambient environment variables + (``/etc/podman_hpc/modules.d``), and the ``MOUNT_*`` family + (``MOUNT_HOME``, ``MOUNT_SCRATCH``, ``MOUNT_CFS`` at NERSC) + bind-mounts host trees into every container the runtime launches — + undeclared inputs arriving under a manifest that attests + ``fs: declared``. ``ENABLE_*`` is deliberately kept: those modules + bind CUDA and MPI system libraries, the same generosity class as the + ``/dev`` and ``/sys`` grants, and none is a channel undeclared + inputs arrive through. + """ + return name.startswith("MOUNT_") + + def child_env() -> dict[str, str]: """Build the environment external tools run in. - Ours, minus ``VIRTUAL_ENV`` and minus every ``UV_*`` variable outside - the :data:`_UV_KEPT` plumbing allowlist. Every uv invocation names its - project explicitly, so an activated environment elsewhere is never - what we mean — and an ambient install setting would change what a - sync installs without moving ``env_version``, which is the identity - hole the scrub closes. + Ours, minus ``VIRTUAL_ENV``, minus every ``UV_*`` variable outside + the :data:`_UV_KEPT` plumbing allowlist, and minus ``MOUNT_*`` (a + site container module's mount gates — see :func:`_mount_scrubbed`). + Every uv invocation names its project explicitly, so an activated + environment elsewhere is never what we mean — and an ambient install + setting would change what a sync installs without moving + ``env_version``, which is the identity hole the scrub closes. + + On a known center, the uv plumbing whose home-filesystem default is + unusable from compute nodes is supplied from the center's scratch + root (:func:`venue.site_env`) — per variable, and only where the + user set none, so an ambient value always wins. The rule is + deterministic in the ambient environment, so every node of an + allocation derives the same answer with nothing handed down. Returns: - The current environment without ``VIRTUAL_ENV`` or scrubbed ``UV_*``. + The current environment, scrubbed, with the site plumbing + supplied. """ - return { + env = { k: v for k, v in os.environ.items() - if k != "VIRTUAL_ENV" and not _uv_scrubbed(k) + if k != "VIRTUAL_ENV" and not _uv_scrubbed(k) and not _mount_scrubbed(k) } + # Function-local: venue imports ProjectError from this module. + from lightcone.engine import venue + for name, value in venue.site_env().items(): + if not env.get(name): + env[name] = value + return env -def uv_scrub_warning() -> str: + +def scrub_warning() -> str: """Compose the dropped-ambient-variables warning, once for every verb. A user whose ``UV_PYTHON`` or ``UV_INDEX_URL`` stopped steering uv deserves a pointer to why on *whichever* verb they hit first — ``lc init`` resolving against the wrong index fails with uv's raw - error otherwise. One spelling here, and one predicate with + error otherwise. One spelling here, and one predicate per scrub with :func:`child_env`, so the verbs cannot drift from each other or the report from the scrub. Empty variables steer nothing and are not reported. @@ -778,13 +810,20 @@ def uv_scrub_warning() -> str: Returns: The warning, or ``""`` when nothing non-empty was dropped. """ + clauses = [] if dropped := sorted(k for k, v in os.environ.items() if v and _uv_scrubbed(k)): - return ( + clauses.append( f"ignored ambient {', '.join(dropped)} — an install setting is " "the project's to declare (pyproject.toml), and an ambient one " "would steer uv without moving env_version" ) - return "" + if mounts := sorted(k for k, v in os.environ.items() if v and _mount_scrubbed(k)): + clauses.append( + f"ignored ambient {', '.join(mounts)} — a site container module " + "would bind-mount undeclared host directories into recipe " + "containers" + ) + return "; ".join(clauses) def uv_version(directory: Path) -> str: diff --git a/src/lightcone/engine/run.py b/src/lightcone/engine/run.py index 766ec59e..badbb615 100644 --- a/src/lightcone/engine/run.py +++ b/src/lightcone/engine/run.py @@ -24,8 +24,8 @@ SPEC_FILENAME, child_env, require_uv, + scrub_warning, uv_prefix, - uv_scrub_warning, ) @@ -77,7 +77,7 @@ def probe(project: Path, command: Sequence[str]) -> sandbox.Outcome: ) # The probe is what called `child_env`, so the probe's outcome is # where the scrub's fact belongs — the caller prints notes verbatim. - if warning := uv_scrub_warning(): + if warning := scrub_warning(): outcome = replace(outcome, notes=(warning, *outcome.notes)) return outcome diff --git a/src/lightcone/engine/sandbox/landlock.py b/src/lightcone/engine/sandbox/landlock.py index ed439a75..cfcd7bad 100644 --- a/src/lightcone/engine/sandbox/landlock.py +++ b/src/lightcone/engine/sandbox/landlock.py @@ -38,7 +38,10 @@ def capability() -> Capability: return Capability(kind="landlock", landlock_abi=abi) return Capability( kind="none", - detail="landlock unavailable (kernel < 5.13, blocked by seccomp, or unsupported arch)", + detail=( + "landlock unavailable (not in the kernel's boot-time LSM list, " + "kernel < 5.13, blocked by seccomp, or unsupported arch)" + ), ) diff --git a/src/lightcone/engine/venue.py b/src/lightcone/engine/venue.py index 9acdcbda..222e9211 100644 --- a/src/lightcone/engine/venue.py +++ b/src/lightcone/engine/venue.py @@ -66,6 +66,12 @@ class _Site: salloc: str #: The batch form of the same request; the guard appends ``--wrap``. sbatch: str + #: The environment variable naming the center's per-user scratch + #: root, which is where uv's disposable state has to live when the + #: center's home filesystem is unusable from compute nodes (see + #: :func:`site_env`). ``None`` for a center whose home works + #: everywhere. + scratch: str | None = None #: The centers the guard knows. Supporting another is one row here — @@ -77,10 +83,45 @@ class _Site: marker="NERSC_HOST", salloc="salloc --nodes=1 --constraint=cpu --qos=interactive --time=02:00:00", sbatch="sbatch --nodes=1 --constraint=cpu --qos=regular --time=02:00:00", + scratch="SCRATCH", ), ) +def site_env() -> dict[str, str]: + """The uv plumbing a known center's filesystems require. + + Both entries move state off the center's home filesystem, which is + the one thing measurement said is unusable from its compute nodes — + and both are disposable by design, so scratch purge policies cost a + re-download rather than a result: + + * ``UV_CACHE_DIR`` — home cannot take uv's cache lock at all + (measured on Perlmutter: ``flock`` fails with os error 524), so + every uv-touching verb dies without this. + * ``UV_PYTHON_INSTALL_DIR`` — an interpreter on home starts in + 0.4–9 s rather than 0.05 s (measured, same node), which under + load exceeds uv's own 60 s startup ceiling and fails a sync + mid-run with "Python startup timed out". + + Paths are derived, never configured, and the caller applies each + only where the user has not set that variable themselves. + + Returns: + The variables to supply, empty off a known center or where its + scratch variable is unset. + """ + for site in _SITES: + if site.marker in os.environ and site.scratch: + if root := os.environ.get(site.scratch): + base = os.path.join(root, ".lightcone") + return { + "UV_CACHE_DIR": os.path.join(base, "uv-cache"), + "UV_PYTHON_INSTALL_DIR": os.path.join(base, "uv-python"), + } + return {} + + def require_compute_node(command: str = "lc materialize") -> None: """Refuse to execute recipes on a known HPC center's login node. diff --git a/tests/conftest.py b/tests/conftest.py index 51536f3c..abe5927d 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -47,21 +47,22 @@ def venue_env(monkeypatch: pytest.MonkeyPatch) -> None: @pytest.fixture(autouse=True) def ambient_uv(monkeypatch: pytest.MonkeyPatch) -> None: - """Strip scrubbable ``UV_*`` out of the suite's environment. + """Strip scrubbable variables out of the suite's environment. CI pins its matrix interpreter through an ambient ``UV_PYTHON``, which the scrub correctly drops and reports — so without this every converge in the suite carries the warning and every ``warnings == - []`` assertion depends on the host. Derived from the scrub's own - predicate, so a variable the allowlist later admits stops being + []`` assertion depends on the host; a dev shell exporting a site's + ``MOUNT_*`` gates is the same shape. Derived from the scrub's own + predicates, so a variable the allowlist later admits stops being stripped here for free; the scrub tests set their own variables back deliberately. """ import os - from lightcone.engine.project import _uv_scrubbed + from lightcone.engine.project import _mount_scrubbed, _uv_scrubbed - for name in [k for k in os.environ if _uv_scrubbed(k)]: + for name in [k for k in os.environ if _uv_scrubbed(k) or _mount_scrubbed(k)]: monkeypatch.delenv(name) diff --git a/tests/test_container.py b/tests/test_container.py index dc7a8f6a..0eb33199 100644 --- a/tests/test_container.py +++ b/tests/test_container.py @@ -53,7 +53,7 @@ def _write_archive(path: Path, config: bytes = _CONFIG) -> str: @pytest.fixture -def fake(monkeypatch: pytest.MonkeyPatch) -> list[list[str]]: +def fake(monkeypatch: pytest.MonkeyPatch, tmp_path: Path) -> list[list[str]]: """A podman-having host: records argv, models each command's effect. Every tool name resolves except ``podman-hpc`` — a site wrapper no @@ -77,7 +77,10 @@ def run(argv: list[str], *, cwd: Path) -> MagicMock: return MagicMock(returncode=0, stdout="", stderr="") return MagicMock(returncode=0, stdout="", stderr="") if argv[:2] == ["uv", "cache"]: - return MagicMock(returncode=0, stdout="/home/user/.cache/uv\n", stderr="") + # Under tmp_path, not a fictional `/home/user`: the sync + # creates this directory, because a bind mount's source must + # exist and nothing on the host has made it in this mode. + return MagicMock(returncode=0, stdout=f"{tmp_path / 'uv-cache'}\n", stderr="") if argv[:3] == ["git", "diff", "--cached"]: return MagicMock(returncode=1) # something staged: commits proceed if argv[:2] == ["git", "check-attr"]: @@ -399,16 +402,21 @@ def test_lc_build_on_a_direct_project_says_so(tmp_path: Path, fake: list[list[st def test_sync_runs_uv_inside_the_image_with_the_host_cache( - root: Path, fake: list[list[str]] + root: Path, fake: list[list[str]], tmp_path: Path ) -> None: runtime = container.runtime_for_run(root, build=True) fake.clear() container.sync(root, runtime) + cache = tmp_path / "uv-cache" (sync,) = _argvs(fake, "podman", "run") assert f"{root}:{root}:rw" in " ".join(sync) - assert "/home/user/.cache/uv:/home/user/.cache/uv:rw" in " ".join(sync) + assert f"{cache}:{cache}:rw" in " ".join(sync) + assert cache.is_dir(), ( + "a bind mount's source must exist, and in containerized mode uv " + "never runs on the host to create it" + ) assert f"UV_PROJECT_ENVIRONMENT={root / '.lightcone' / 'venv'}" in " ".join(sync) assert "--userns=keep-id" in sync assert runtime.image_id in sync @@ -556,6 +564,102 @@ def test_migrate_runs_even_when_the_store_already_holds_the_image( assert len(_argvs(hpc, "podman-hpc", "migrate")) == 2 +def _squash_probe( + monkeypatch: pytest.MonkeyPatch, hpc: list[list[str]], answer: MagicMock +) -> None: + """Make the store-listing probe answer *answer*; everything else keeps + the `hpc` stub's behavior — the fixture's one-fact-at-a-time rule.""" + inner = project._run + + def run(argv: list[str], *, cwd: Path) -> MagicMock: + if argv[:2] == ["podman-hpc", "images"]: + hpc.append(list(argv)) + return answer + return inner(argv, cwd=cwd) + + monkeypatch.setattr(project, "_run", run) + + +def test_a_stale_squashed_image_is_removed_before_migrate( + root: Path, hpc: list[list[str]], monkeypatch: pytest.MonkeyPatch +) -> None: + """The tag is deterministic but builds are not bit-reproducible, so a + rebuild migrated under an unchanged tag would put a second same-named + image into the read-only squash store — which then refuses every + storage operation. The heal removes the stale copy first.""" + expected = _write_archive(image.archive_path(root, image.tag(root))) + stale = "f" * 64 + _squash_probe( + monkeypatch, + hpc, + MagicMock(returncode=0, stdout=f"{expected} false\n{stale} true\n", stderr=""), + ) + + container.runtime_for_run(root, build=False) + + tag = image.tag(root) + assert _argvs(hpc, "podman-hpc", "rmsqi") == [["podman-hpc", "rmsqi", tag]] + order = [c[1] for c in _argvs(hpc, "podman-hpc") if c[1] in ("images", "rmsqi", "migrate")] + assert order == ["images", "rmsqi", "migrate"] + + +def test_a_current_squashed_image_is_left_alone( + root: Path, hpc: list[list[str]], monkeypatch: pytest.MonkeyPatch +) -> None: + """The healthy pair — the overlay copy and its own squash — is exactly + what a migrate leaves behind; removing it would re-squash every run.""" + expected = _write_archive(image.archive_path(root, image.tag(root))) + _squash_probe( + monkeypatch, + hpc, + MagicMock(returncode=0, stdout=f"{expected} false\n{expected} true\n", stderr=""), + ) + + container.runtime_for_run(root, build=False) + + assert _argvs(hpc, "podman-hpc", "rmsqi") == [] + assert len(_argvs(hpc, "podman-hpc", "migrate")) == 1 + + +def test_a_failing_probe_never_blocks_a_healthy_run( + root: Path, hpc: list[list[str]], monkeypatch: pytest.MonkeyPatch +) -> None: + """A heal must not introduce a failure mode of its own: an + unrecognized probe failure is left for migrate's own loud path.""" + _write_archive(image.archive_path(root, image.tag(root))) + _squash_probe( + monkeypatch, hpc, MagicMock(returncode=125, stdout="", stderr="some other complaint") + ) + + container.runtime_for_run(root, build=False) + + assert _argvs(hpc, "podman-hpc", "rmsqi") == [] + assert len(_argvs(hpc, "podman-hpc", "migrate")) == 1 + + +def test_an_already_wedged_store_is_healed( + root: Path, hpc: list[list[str]], monkeypatch: pytest.MonkeyPatch +) -> None: + """A store wedged before the heal existed fails the listing too, with + the same message every operation gets — that signature is the one + probe failure the heal acts on.""" + _write_archive(image.archive_path(root, image.tag(root))) + _squash_probe( + monkeypatch, + hpc, + MagicMock( + returncode=125, + stdout="", + stderr="Error: configure storage: read-only image store assigns " + "the same name to multiple images", + ), + ) + + container.runtime_for_run(root, build=False) + + assert _argvs(hpc, "podman-hpc", "rmsqi") == [["podman-hpc", "rmsqi", image.tag(root)]] + + def test_podman_hpc_builds_saves_and_migrates(root: Path, hpc: list[list[str]]) -> None: """The wrapper is build-capable — NERSC login nodes are where the matching-arch archive comes from — and a fresh build still migrates.""" diff --git a/tests/test_materialize.py b/tests/test_materialize.py index 5ff96e8d..121e7235 100644 --- a/tests/test_materialize.py +++ b/tests/test_materialize.py @@ -974,6 +974,36 @@ def processes() -> Iterator[engine._Dask]: assert not dataset.status(root) +def test_the_scheduler_adapter_releases_each_future_as_it_yields( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """A future still held at teardown makes the scheduler preserve its + key, so retiring workers replicate every key onto each other — all + retiring — stall past the reap grace, and get killed: every clean + SLURM run ends in "srun: forcing job termination". Releasing as each + result lands is what lets the wind-down be silent.""" + import distributed + + class Fut: + released = False + + def release(self) -> None: + self.released = True + + futs = [Fut(), Fut()] + + def fake_as_completed(handles: list[Fut], with_results: bool) -> Iterator[tuple[Fut, int]]: + return iter((f, i) for i, f in enumerate(handles)) + + monkeypatch.setattr(distributed, "as_completed", fake_as_completed) + + seen = [] + for result in engine._Dask(client=None).completed(list(futs)): + assert futs[result].released, "released before its result is handed over" + seen.append(result) + assert seen == [0, 1] + + # ---- the report ------------------------------------------------------------ diff --git a/tests/test_project.py b/tests/test_project.py index 463af2eb..2454b640 100644 --- a/tests/test_project.py +++ b/tests/test_project.py @@ -683,7 +683,7 @@ def test_ambient_uv_install_settings_are_scrubbed( come from and how fast, never what gets installed.""" import os - from lightcone.engine.project import child_env, uv_scrub_warning + from lightcone.engine.project import child_env, scrub_warning for name in [k for k in os.environ if k.startswith("UV_")]: monkeypatch.delenv(name) # the suite itself may run under `uv run` @@ -709,7 +709,7 @@ def test_ambient_uv_install_settings_are_scrubbed( ) assert env["UV_LINK_MODE"] == "copy", "link-mode is not an audited setting either" assert env["LC_TEST_CANARY"] == "kept" - assert "UV_INDEX_URL, UV_NO_BINARY, UV_PYTHON" in uv_scrub_warning(), ( + assert "UV_INDEX_URL, UV_NO_BINARY, UV_PYTHON" in scrub_warning(), ( "the warning names exactly what the scrub dropped" ) @@ -731,10 +731,84 @@ def test_an_empty_scrubbed_variable_is_not_reported( monkeypatch: pytest.MonkeyPatch, ) -> None: """An empty variable steers nothing, so warning about it is noise.""" - from lightcone.engine.project import uv_scrub_warning + from lightcone.engine.project import scrub_warning monkeypatch.setenv("UV_NO_BUILD", "") - assert "UV_NO_BUILD" not in uv_scrub_warning() + monkeypatch.setenv("MOUNT_X", "") + warning = scrub_warning() + assert "UV_NO_BUILD" not in warning + assert "MOUNT_X" not in warning + + +def test_ambient_site_mount_gates_are_scrubbed( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """A site container module's MOUNT_* gate would bind-mount undeclared + host directories into every recipe container while the manifest + attests `fs: declared`. ENABLE_* survives: those modules bind CUDA + and MPI system libraries — the /dev,/sys generosity class, not a + channel undeclared inputs arrive through.""" + from lightcone.engine.project import child_env, scrub_warning + + monkeypatch.setenv("MOUNT_CFS", "1") + monkeypatch.setenv("ENABLE_GPU", "1") + + env = child_env() + assert "MOUNT_CFS" not in env + assert env["ENABLE_GPU"] == "1", "the GPU module gate is deliberately kept" + assert "MOUNT_CFS" in scrub_warning(), "the warning names the dropped gate" + + +def test_a_known_site_supplies_its_uv_plumbing_when_unset( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """The center's home filesystem is unusable from compute nodes — it + cannot take uv's cache lock at all, and an interpreter there starts + slowly enough to blow uv's own timeout — so both land on scratch, + deterministically in the ambient environment, which is what lets + every node of an allocation derive the same answer.""" + from lightcone.engine.project import child_env + + monkeypatch.setenv("NERSC_HOST", "perlmutter") + monkeypatch.setenv("SCRATCH", "/pscratch/sd/u/user") + monkeypatch.delenv("UV_CACHE_DIR", raising=False) + monkeypatch.delenv("UV_PYTHON_INSTALL_DIR", raising=False) + + env = child_env() + assert env["UV_CACHE_DIR"] == "/pscratch/sd/u/user/.lightcone/uv-cache" + assert env["UV_PYTHON_INSTALL_DIR"] == "/pscratch/sd/u/user/.lightcone/uv-python" + + +def test_ambient_uv_plumbing_wins_over_the_site_supply( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """These locations are the user's plumbing to choose; the site + default fills silence, never overrides a voice — and per variable, + so setting one does not suppress the other.""" + from lightcone.engine.project import child_env + + monkeypatch.setenv("NERSC_HOST", "perlmutter") + monkeypatch.setenv("SCRATCH", "/pscratch/sd/u/user") + monkeypatch.setenv("UV_CACHE_DIR", "/pscratch/sd/u/user/custom") + monkeypatch.delenv("UV_PYTHON_INSTALL_DIR", raising=False) + + env = child_env() + assert env["UV_CACHE_DIR"] == "/pscratch/sd/u/user/custom" + assert env["UV_PYTHON_INSTALL_DIR"] == "/pscratch/sd/u/user/.lightcone/uv-python" + + +def test_no_site_means_no_supply(monkeypatch: pytest.MonkeyPatch) -> None: + """Off a known center — or on one whose scratch variable is absent — + uv's own defaults stand.""" + from lightcone.engine.project import child_env + + monkeypatch.setenv("SCRATCH", "/pscratch/sd/u/user") # scratch alone: no + monkeypatch.delenv("UV_CACHE_DIR", raising=False) + assert "UV_CACHE_DIR" not in child_env() + + monkeypatch.setenv("NERSC_HOST", "perlmutter") # marker without scratch: no + monkeypatch.delenv("SCRATCH") + assert "UV_CACHE_DIR" not in child_env() def test_relays_uv_warnings_into_the_report( From 227439ea764d733234d34b186b1e15d06c5e60c4 Mon Sep 17 00:00:00 2001 From: EiffL Date: Fri, 21 Aug 2026 08:29:41 -0700 Subject: [PATCH 2/3] Attest site container modules; heal squash images by id MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review findings from the NERSC work, both verified against the real podman-hpc on Perlmutter. Site modules are named in the attestation. The MOUNT_*/ENABLE_* split kept ENABLE_* on the stated ground that those modules bind system libraries only, "never a channel undeclared inputs arrive through" — measurably false: ENABLE_CVMFS binds the whole populated /cvmfs hierarchy, which is reference data a recipe can read without declaring it, and ENABLE_MPICH_SS adds --privileged plus the host's network, pid and ipc namespaces, after which the mount table is not the boundary at all. Only ENABLE_GPU and ENABLE_NCCL* fit the original description. They stay working — they are how a GPU or MPI recipe reaches the hardware it was written for — so the honesty comes from the record: container.site_modules() names the gates that are set, OCIBackend carries them as a resolved field (the user_flags pattern, so wrap stays pure), and Attestation.site_modules puts them in every manifest. It is the one attestation value not derived from lc's own argv, because the runtime applies them from its own environment, and the docstring says so. A per-module allowlist was rejected: the module table is site-configurable, so a center adding a data-binding module lc has never heard of would slip through a hardcoded judgement. The squash heal removes by id. `rmsqi ` resolves a single record (remove_image → get_img_info takes the first name match), so against the two-same-name state being healed it could take the current image and leave the stale one to re-wedge the store on the next migrate. Each stale id is now removed by its own call; only the wedged branch spells the tag, because a store that cannot be listed cannot be enumerated, and it re-probes bounded by _HEAL_ATTEMPTS since each blind pass is a real deletion. Also: creating uv's cache directory raises ProjectError rather than a bare OSError, which no caller translates — a full quota or an unwritable UV_CACHE_DIR would have surfaced as a traceback out of lc materialize instead of a refusal naming the path. Recorded rather than fixed: the heal can drop layers a concurrently running allocation is still executing from (leaving the store wedged breaks that run too, and cross-run coordination is not something a heal can offer). Verified: a real containerized run with ENABLE_CVMFS=1 ENABLE_GPU=1 records both in hermeticity.site_modules, and `lc run ls /cvmfs` does list the repositories — the widening is real and the manifest now says so. 573 passed on a compute node. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01DyNDEbtjpob2CJSQMnEiuf --- CLAUDE.md | 48 +++++++++--- docs/api/container.md | 20 ++++- docs/user/cluster.md | 12 ++- src/lightcone/engine/container.py | 103 ++++++++++++++++++++----- src/lightcone/engine/sandbox/model.py | 9 +++ src/lightcone/engine/sandbox/oci.py | 14 ++++ tests/test_container.py | 105 ++++++++++++++++++++++++-- tests/test_sandbox_oci.py | 21 ++++++ 8 files changed, 293 insertions(+), 39 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 459f7079..dcd22aa2 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -2103,24 +2103,52 @@ unlinks before writing; a new tampering test should too. for a default doing its job. Deterministic in the ambient environment, so every node of an allocation derives the same answer with nothing handed down. - - *`MOUNT_*` is scrubbed; `ENABLE_*` is kept.* podman-hpc's site - modules are env-gated; the mount gates bind `$HOME`/`$SCRATCH`/ - `$CFS` into every container — undeclared inputs under a manifest - attesting `fs: declared` (measured; the one item the layer-7 spike - said could add a flag, resolved without one). The library gates - stay: CUDA/MPI system libs are the `/dev`,`/sys` generosity class, - and the site's own GPU mechanism must keep working. Drops are - named by `scrub_warning` (the `uv_scrub_warning` rename). + - *`MOUNT_*` is scrubbed; `ENABLE_*` is kept and **attested**.* + podman-hpc's site modules are env-gated. The mount gates bind + `$HOME`/`$SCRATCH`/`$CFS` into every container — undeclared inputs + under a manifest attesting `fs: declared` (measured) — so they are + dropped, and named by `scrub_warning` (the `uv_scrub_warning` + rename). The `ENABLE_*` gates are **not** scrubbed: they are how a + GPU or MPI recipe reaches its hardware, and removing them would + take a working capability away with no replacement. + But the first draft kept them on the stated ground that they bind + "system libraries only, never a channel undeclared inputs arrive + through", and review measured that false: `ENABLE_CVMFS` binds the + whole populated `/cvmfs` hierarchy (reference data a recipe can + read undeclared) and `ENABLE_MPICH_SS` adds `--privileged` with the + host's network, pid and ipc namespaces — the mount table stops + being the boundary at all. Only `ENABLE_GPU`/`ENABLE_NCCL*` fit the + original description. So the honesty comes from the record instead: + `container.site_modules()` names the set gates, `OCIBackend` carries + them as a resolved field (the `user_flags` pattern — `wrap` stays + pure), and `Attestation.site_modules` puts them in every manifest. + This is the one attestation value not derived from lc's own argv, + and the docstring says so: the runtime applies them from its own + environment. A per-module allowlist was rejected — the module table + is site-configurable, so a center adding a data-binding module lc + has never heard of would slip through a hardcoded judgement. - *A stale squashed image is healed before the load.* The tag is deterministic, builds are not bit-reproducible, and podman-hpc's read-only squash store refuses two images under one name — after which *every* storage operation fails (measured: a same-tag rebuild wedged the store for runs, listings, everything). - `container._heal_squash` probes and `rmsqi`s the stale copy; an + `container._heal_squash` probes and `rmsqi`s the stale copies; an unrecognized probe outcome is left for migrate's own loud path, so the heal cannot break a healthy run. Before the *load*, not just the migrate — a wedged store fails the load too on a node - that has not loaded yet. + that has not loaded yet. Removal is **by id, one call per stale + image**: `rmsqi ` resolves a single record (podman-hpc's + `remove_image` → `get_img_info` takes the first name match), so + against the very state being healed it can take the *current* + image and leave the stale one to re-wedge the store on the next + migrate. Only the wedged branch spells the tag — a store that + cannot be listed cannot be enumerated — and it re-probes, bounded + by `_HEAL_ATTEMPTS`, because each blind pass is a real deletion. + Recorded hazard, accepted: the squash store is shared across a + user's nodes and allocations, so a heal can drop layers an + earlier run is still executing from. Leaving the store wedged + breaks that run too, and cross-run coordination is not something + a heal can offer. - *The driver releases each future as its result is consumed* (`_Dask.completed`). A key still client-held at teardown makes `retire_workers` replicate it onto peers that are also retiring — diff --git a/docs/api/container.md b/docs/api/container.md index 97f76104..1e43afb0 100644 --- a/docs/api/container.md +++ b/docs/api/container.md @@ -59,14 +59,26 @@ Sources: `src/lightcone/engine/image.py`, outside the load branch) and joins `_SHARED_STORE_RUNTIMES`. Detection order podman-hpc → podman → docker; docker's daemon is probed at detection. +- **Site container modules are named, not silenced.** `site_modules()` + reports the `ENABLE_*` gates set for podman-hpc, which the runtime + applies from its own environment: a module can widen the container + past the mount table (`ENABLE_CVMFS` binds `/cvmfs`, + `ENABLE_MPICH_SS` adds `--privileged` and the host namespaces), so + they reach `Attestation.site_modules` and every manifest rather than + leaving `fs: declared` to overstate the boundary. `MOUNT_*` never + appears there — `project.child_env` scrubs it before the runtime + sees it. - **A stale squashed image is healed before the load.** The tag is deterministic but builds are not bit-reproducible, so a rebuild migrated under an unchanged tag would put a second same-named image into podman-hpc's read-only squash store — after which every storage - operation fails. `_heal_squash` probes the store and `rmsqi`s the - stale copy first (before the load, which a wedged store also fails); - a probe outcome it does not recognize is left for migrate's own loud - path, so the heal can never break a healthy run. + operation fails. `_heal_squash` probes the store and `rmsqi`s each + stale copy **by id** first (before the load, which a wedged store + also fails) — `rmsqi ` resolves only one record and could take + the current image instead. A store too wedged to list is removed by + tag and re-probed, bounded by `_HEAL_ATTEMPTS`; a probe outcome the + heal does not recognize is left for migrate's own loud path, so it + can never break a healthy run. - **The architecture gate refuses before the load** — a wrong-arch `load` succeeds and then dies as `exec format error` deep inside a recipe. Ignorance passes; a recorded mismatch refuses, naming the diff --git a/docs/user/cluster.md b/docs/user/cluster.md index f02e9b64..70e4794a 100644 --- a/docs/user/cluster.md +++ b/docs/user/cluster.md @@ -150,9 +150,15 @@ Perlmutter itself: ignored ambient MOUNT_HOME — a site container module would bind-mount undeclared host directories into recipe containers ``` - The library gates (`ENABLE_GPU`, `ENABLE_MPICH_SS`, …) survive: a - GPU recipe gets CUDA by exporting `ENABLE_GPU=1`, the site's own - mechanism. + The `ENABLE_*` gates survive, because they are how a GPU or MPI + recipe reaches the hardware it was written for — a GPU recipe gets + CUDA by exporting `ENABLE_GPU=1`, the site's own mechanism. They are + **recorded in the manifest instead**, under + `hermeticity.site_modules`, because a module can widen the container + well past the mounts `lc` declared: `ENABLE_CVMFS` binds the whole + `/cvmfs` hierarchy, and `ENABLE_MPICH_SS` adds `--privileged` plus + the host's network, pid and ipc namespaces. If you enable one, the + outputs of that run say so. - **BLAS may be silently throttled.** NERSC's allocation environment exports `OMP_NUM_THREADS` (often `2`), and `srun` propagates it into diff --git a/src/lightcone/engine/container.py b/src/lightcone/engine/container.py index f91ad0bc..1bb4d921 100644 --- a/src/lightcone/engine/container.py +++ b/src/lightcone/engine/container.py @@ -48,6 +48,12 @@ #: the multi-node materialize refusal stands on. _SHARED_STORE_RUNTIMES = ("podman-hpc",) +#: How many blind removals the squash heal will make against a store it +#: cannot list. Bounded rather than looped-until-clean: each pass is a +#: real deletion, and a store still unlistable after this many is not +#: one more `rmsqi` away from working. +_HEAL_ATTEMPTS = 3 + @dataclass(frozen=True) class Runtime: @@ -200,32 +206,56 @@ def _committed(archive: Path) -> bool: def _heal_squash(root: Path, tag: str, image_id: str) -> None: - """Remove a stale squashed image before migrating the current one. + """Remove stale squashed images before migrating the current one. The tag is deterministic but builds are not bit-reproducible, so a rebuild migrated under an unchanged tag puts a second image with the same name into podman-hpc's read-only squash store — after which *every* storage operation fails ("read-only image store assigns the same name to multiple images"), runs included (measured on - Perlmutter). ``rmsqi`` removes only the squashed copy, and the - migrate that follows re-squashes the current id. A heal must never - break a run whose store is healthy, so a probe outcome it does not - recognize is left for migrate's own loud path. + Perlmutter). ``rmsqi`` drops only the squashed copy, and the migrate + that follows re-squashes the current id. + + Removal is **by id, one call per stale image**: ``rmsqi `` + resolves a single record and takes that one, so against the very + state this heals — two records sharing a name — it can just as + easily take the current image and leave the stale one, re-wedging + the store on the next migrate. Only the wedged branch has to spell + the tag, because a store that cannot be listed cannot be enumerated; + it re-probes so a second stale record is still reached. + + A heal must never break a run whose store is healthy, so a probe + outcome it does not recognize is left for migrate's own loud path. + + Recorded hazard: the squash store is shared across a user's nodes + and allocations, so removing a stale image drops layers that a run + started earlier — under a tag whose id has since moved — may still + be executing from. The alternative is leaving the store wedged for + every later verb, which breaks that run too; cross-run coordination + is not something a heal can offer. """ - probe = project._run( - ["podman-hpc", "images", "--format", "{{.Id}} {{.ReadOnly}}", f"localhost/{tag}"], - cwd=root, - ) - if probe.returncode == 0: - rows = [line.split() for line in probe.stdout.splitlines()] - stale = any( - len(row) == 2 and row[1] == "true" and row[0] != image_id for row in rows + for _ in range(_HEAL_ATTEMPTS): + probe = project._run( + [ + "podman-hpc", "images", "--format", + "{{.Id}} {{.ReadOnly}}", f"localhost/{tag}", + ], # fmt: skip + cwd=root, ) - else: + if probe.returncode == 0: + rows = [line.split() for line in probe.stdout.splitlines()] + stale = { + row[0] + for row in rows + if len(row) == 2 and row[1] == "true" and row[0] != image_id + } + for victim in sorted(stale): + _check_call(["podman-hpc", "rmsqi", victim], cwd=root) + return # A wedged store fails the listing too, with the same message # every other operation gets — the shape a pre-heal lc left. - stale = "assigns the same name to multiple images" in probe.stderr - if stale: + if "assigns the same name to multiple images" not in probe.stderr: + return _check_call(["podman-hpc", "rmsqi", tag], cwd=root) @@ -299,9 +329,41 @@ def backend(runtime: Runtime) -> sandbox.Backend: image_id=runtime.image_id, root=runtime.root, user_flags=(*uid_flags(runtime.runtime), *pull), + site_modules=site_modules(runtime.runtime), ) +def site_modules(runtime: str) -> tuple[str, ...]: + """Name the site container modules the runtime will apply, if any. + + podman-hpc reads a site's module table + (``/etc/podman_hpc/modules.d``) from its own environment, and a + module widens the container by more than lc declared: NERSC's + ``ENABLE_CVMFS`` binds the whole ``/cvmfs`` hierarchy — reference + data a recipe can read without declaring it — and + ``ENABLE_MPICH_SS`` adds ``--privileged`` plus the host's network, + pid and ipc namespaces. They are left working, because they are how + a GPU or MPI recipe reaches the hardware it was written for, and + named in the attestation instead, so no manifest claims the mount + table was the whole boundary. + + Read from :func:`project.child_env`, which is what the runtime + actually receives — the ``MOUNT_*`` gates it scrubs are absent by + the time this asks. + + Args: + runtime: The resolved runtime's name. + + Returns: + The gate names that are set, sorted; empty for a runtime with no + module system. + """ + if runtime != "podman-hpc": + return () + env = project.child_env() + return tuple(sorted(k for k, v in env.items() if k.startswith("ENABLE_") and v)) + + def build(root: Path) -> tuple[Runtime, str]: """Converge the system-layer image, the whole of ``lc build``. @@ -416,7 +478,14 @@ def sync(root: Path, runtime: Runtime) -> list[str]: # never runs on the host, so nothing else has created it. Otherwise # the first sync of a fresh project dies as the runtime's `statfs` # error about a path the user never named. - Path(cache).mkdir(parents=True, exist_ok=True) + try: + Path(cache).mkdir(parents=True, exist_ok=True) + except OSError as e: + raise ProjectError( + f"uv's cache directory `{cache}` cannot be created ({e}). It is " + "mounted into the image to converge the environment; set " + "UV_CACHE_DIR to a writable path and retry." + ) from e argv = [ runtime.runtime, "run", "--rm", "--entrypoint", "", # Same reason as the exec boundary's flag: SELinux hosts refuse diff --git a/src/lightcone/engine/sandbox/model.py b/src/lightcone/engine/sandbox/model.py index b8ad809b..66aa8b48 100644 --- a/src/lightcone/engine/sandbox/model.py +++ b/src/lightcone/engine/sandbox/model.py @@ -98,6 +98,15 @@ class Attestation: network: Literal["allowed", "denied"] = "allowed" landlock_abi: int | None = None exec_allowlist_version: int | None = None + #: Site container modules the runtime applied on top of the mount + #: table, named by the gates that enabled them. A module widens the + #: world by more than lc declared — NERSC's ``ENABLE_CVMFS`` binds + #: the whole ``/cvmfs`` hierarchy, ``ENABLE_MPICH_SS`` adds + #: ``--privileged`` and the host's network, pid and ipc namespaces — + #: so ``fs: declared`` alone would overstate what was enforced. + #: Naming them is what keeps the record honest while leaving the + #: site's own GPU and MPI mechanisms working. + site_modules: tuple[str, ...] = () class Backend(Protocol): diff --git a/src/lightcone/engine/sandbox/oci.py b/src/lightcone/engine/sandbox/oci.py index 0acbf4e9..87e22531 100644 --- a/src/lightcone/engine/sandbox/oci.py +++ b/src/lightcone/engine/sandbox/oci.py @@ -46,6 +46,13 @@ class OCIBackend: #: resolved by the caller so the wrap stays a pure function of its #: fields (``--userns=keep-id`` / ``--user uid:gid``). user_flags: tuple[str, ...] = () + #: Site container modules this runtime will apply on top of the + #: mount table, named by the gates that enable them. Resolved by the + #: caller, like ``user_flags``, because the runtime reads them from + #: its own environment and the wrap stays a pure function of its + #: fields. Reported by :meth:`attest`, never acted on here — a + #: module is the site's mechanism, applied by the runtime itself. + site_modules: tuple[str, ...] = () contains_prefix: bool = True @property @@ -116,6 +123,12 @@ def attest(self, policy: Policy) -> Attestation: every mechanism gives, because lc does not control the network anywhere and the attestation says only what was enforced. + The exception is :attr:`site_modules`, which the *runtime* + applies from its own environment rather than from this argv. + They are named rather than silently dropped: a module can widen + the world well past the mount table, so a record saying only + ``fs: declared`` would overstate what bounded the run. + Args: policy: The policy being wrapped. @@ -125,4 +138,5 @@ def attest(self, policy: Policy) -> Attestation: return Attestation( mechanism=self.runtime, fs="declared", + site_modules=self.site_modules, ) diff --git a/tests/test_container.py b/tests/test_container.py index 0eb33199..02320f6a 100644 --- a/tests/test_container.py +++ b/tests/test_container.py @@ -17,7 +17,7 @@ import pytest -from lightcone.engine import container, image, project +from lightcone.engine import container, image, project, sandbox from lightcone.engine.project import ProjectError _TABLE = '[tool.lightcone.image]\napt-install = ["bc"]\n' @@ -398,6 +398,39 @@ def test_lc_build_on_a_direct_project_says_so(tmp_path: Path, fake: list[list[st container.build(plain) +def test_the_backend_names_the_site_modules_the_runtime_will_apply( + root: Path, hpc: list[list[str]], monkeypatch: pytest.MonkeyPatch +) -> None: + """ENABLE_* is left working — it is how a GPU or MPI recipe reaches + its hardware — so the honesty has to come from naming it. MOUNT_* is + absent by construction: child_env scrubs it before the runtime sees + it, which is why only one family reaches the attestation.""" + monkeypatch.setenv("ENABLE_GPU", "1") + monkeypatch.setenv("ENABLE_CVMFS", "1") + monkeypatch.setenv("ENABLE_UNSET", "") + monkeypatch.setenv("MOUNT_HOME", "1") + _write_archive(image.archive_path(root, image.tag(root))) + runtime = container.runtime_for_run(root, build=False) + + attested = container.backend(runtime).attest( + sandbox.exec_policy(root, containerized=True) + ) + + assert attested.site_modules == ("ENABLE_CVMFS", "ENABLE_GPU") + assert "MOUNT_HOME" not in attested.site_modules + assert "ENABLE_UNSET" not in attested.site_modules + + +def test_a_runtime_without_a_module_system_names_none(root: Path, fake: list[list[str]], + monkeypatch: pytest.MonkeyPatch) -> None: + """podman and docker read no site module table, so an ENABLE_* in the + environment is inert there and must not imply otherwise.""" + monkeypatch.setenv("ENABLE_GPU", "1") + + assert container.site_modules("podman") == () + assert container.site_modules("docker") == () + + # ---- the containerized converge --------------------------------------------- @@ -597,12 +630,39 @@ def test_a_stale_squashed_image_is_removed_before_migrate( container.runtime_for_run(root, build=False) - tag = image.tag(root) - assert _argvs(hpc, "podman-hpc", "rmsqi") == [["podman-hpc", "rmsqi", tag]] + assert _argvs(hpc, "podman-hpc", "rmsqi") == [["podman-hpc", "rmsqi", stale]], ( + "removal is by id: `rmsqi ` resolves one record and could take " + "the current image, leaving the stale one to re-wedge the store" + ) order = [c[1] for c in _argvs(hpc, "podman-hpc") if c[1] in ("images", "rmsqi", "migrate")] assert order == ["images", "rmsqi", "migrate"] +def test_every_stale_squashed_image_is_removed( + root: Path, hpc: list[list[str]], monkeypatch: pytest.MonkeyPatch +) -> None: + """`rmsqi` takes one record per call, so a store holding two stale + rows needs two — one call would leave the second to re-wedge it.""" + expected = _write_archive(image.archive_path(root, image.tag(root))) + older, newer = "a" * 64, "b" * 64 + _squash_probe( + monkeypatch, + hpc, + MagicMock( + returncode=0, + stdout=f"{expected} false\n{older} true\n{newer} true\n", + stderr="", + ), + ) + + container.runtime_for_run(root, build=False) + + assert _argvs(hpc, "podman-hpc", "rmsqi") == [ + ["podman-hpc", "rmsqi", older], + ["podman-hpc", "rmsqi", newer], + ] + + def test_a_current_squashed_image_is_left_alone( root: Path, hpc: list[list[str]], monkeypatch: pytest.MonkeyPatch ) -> None: @@ -642,7 +702,42 @@ def test_an_already_wedged_store_is_healed( ) -> None: """A store wedged before the heal existed fails the listing too, with the same message every operation gets — that signature is the one - probe failure the heal acts on.""" + probe failure the heal acts on, and it re-probes so the blind + removal stops as soon as the store can be read again.""" + expected = _write_archive(image.archive_path(root, image.tag(root))) + wedged = MagicMock( + returncode=125, + stdout="", + stderr="Error: configure storage: read-only image store assigns " + "the same name to multiple images", + ) + healthy = MagicMock(returncode=0, stdout=f"{expected} false\n{expected} true\n", stderr="") + answers = [wedged, healthy] + inner = project._run + + def run(argv: list[str], *, cwd: Path) -> MagicMock: + if argv[:2] == ["podman-hpc", "images"]: + hpc.append(list(argv)) + return answers.pop(0) if answers else healthy + return inner(argv, cwd=cwd) + + monkeypatch.setattr(project, "_run", run) + + container.runtime_for_run(root, build=False) + + assert _argvs(hpc, "podman-hpc", "rmsqi") == [["podman-hpc", "rmsqi", image.tag(root)]], ( + "a store that cannot be listed cannot be enumerated, so this one " + "branch has to spell the tag — once, because the re-probe then reads" + ) + assert len(_argvs(hpc, "podman-hpc", "images")) == 2 + + +def test_a_store_that_never_recovers_stops_rather_than_looping( + root: Path, hpc: list[list[str]], monkeypatch: pytest.MonkeyPatch +) -> None: + """Each blind pass is a real deletion, so the heal is bounded: a + store still unlistable after that is not one more `rmsqi` away from + working, and migrate's own refusal is the honest end.""" _write_archive(image.archive_path(root, image.tag(root))) _squash_probe( monkeypatch, @@ -657,7 +752,7 @@ def test_an_already_wedged_store_is_healed( container.runtime_for_run(root, build=False) - assert _argvs(hpc, "podman-hpc", "rmsqi") == [["podman-hpc", "rmsqi", image.tag(root)]] + assert len(_argvs(hpc, "podman-hpc", "rmsqi")) == container._HEAL_ATTEMPTS def test_podman_hpc_builds_saves_and_migrates(root: Path, hpc: list[list[str]]) -> None: diff --git a/tests/test_sandbox_oci.py b/tests/test_sandbox_oci.py index cf8cdccd..2842568c 100644 --- a/tests/test_sandbox_oci.py +++ b/tests/test_sandbox_oci.py @@ -215,6 +215,27 @@ def test_the_attestation_is_derived_from_the_flags(root: Path, policy: Policy) - assert attested.fs == "declared" assert attested.network == "allowed" assert attested.landlock_abi is None + assert attested.site_modules == () + + +def test_site_modules_are_named_in_the_attestation(root: Path, policy: Policy) -> None: + """A site module widens the container past the mount table — NERSC's + ENABLE_CVMFS binds the whole /cvmfs hierarchy, ENABLE_MPICH_SS adds + --privileged and the host namespaces — so `fs: declared` alone would + overstate what bounded the run. They stay working and get named.""" + from lightcone.engine.sandbox.oci import OCIBackend + + backend = OCIBackend( + runtime="podman-hpc", + image_id="a" * 64, + root=root, + site_modules=("ENABLE_CVMFS", "ENABLE_GPU"), + ) + + attested = backend.attest(policy) + + assert attested.site_modules == ("ENABLE_CVMFS", "ENABLE_GPU") + assert attested.fs == "declared", "the mount table is still what lc composed" # ---- the seam's composition ------------------------------------------------- From 6a60955ecc8c02ac36d82fb006dc129d8562ca07 Mon Sep 17 00:00:00 2001 From: EiffL Date: Fri, 21 Aug 2026 09:11:47 -0700 Subject: [PATCH 3/3] Read site module gates from the table, never by prefix MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CI caught the attestation naming a module the host does not have: matching `ENABLE_*` picked up GitHub's own ENABLE_RUNNER_TRACING, so a manifest would have recorded a site module that was never applied — the exact dishonesty `site_modules` exists to prevent, pointed the other way. The gates are asked of the mechanism instead of guessed. Each module in podman-hpc's table declares its own `env:` key, and `podman-hpc infohpc` reports where the table lives (a site can move it), so lc reads the real gate names and keeps the ones the environment sets. That is the same probe-don't-assume rule `git check-attr` and `uv cache dir` already follow. A table that cannot be read reports nothing rather than refusing, the uv_version discipline: attestation must not fail a run. Verified against the live table on Perlmutter — ENABLE_CVMFS and ENABLE_GPU recorded, ENABLE_RUNNER_TRACING ignored, MOUNT_HOME absent because child_env scrubbed it. The tests now stub a module table of their own, so they no longer depend on the host's environment: that dependency is why this passed locally and failed on all four CI legs. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01DaQvhUZHHrGCMZxKLdAfgR --- CLAUDE.md | 7 +++ docs/api/container.md | 21 +++++---- src/lightcone/engine/container.py | 43 +++++++++++++----- tests/test_container.py | 74 +++++++++++++++++++++++++++---- 4 files changed, 119 insertions(+), 26 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index dcd22aa2..30cec42f 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -2122,6 +2122,13 @@ unlinks before writing; a new tampering test should too. `container.site_modules()` names the set gates, `OCIBackend` carries them as a resolved field (the `user_flags` pattern — `wrap` stays pure), and `Attestation.site_modules` puts them in every manifest. + The gate names are **asked of the site's module table** (each + module declares its own `env:` key; the directory comes from + `podman-hpc infohpc`), never matched by prefix: CI caught a prefix + match recording GitHub's own `ENABLE_RUNNER_TRACING` as a module + the host does not have, which is the exact dishonesty the field + exists to prevent. Unreadable table ⇒ empty, never a refusal — + attestation must not fail a run, the `uv_version` discipline. This is the one attestation value not derived from lc's own argv, and the docstring says so: the runtime applies them from its own environment. A per-module allowlist was rejected — the module table diff --git a/docs/api/container.md b/docs/api/container.md index 1e43afb0..8e58c9f8 100644 --- a/docs/api/container.md +++ b/docs/api/container.md @@ -60,14 +60,19 @@ Sources: `src/lightcone/engine/image.py`, Detection order podman-hpc → podman → docker; docker's daemon is probed at detection. - **Site container modules are named, not silenced.** `site_modules()` - reports the `ENABLE_*` gates set for podman-hpc, which the runtime - applies from its own environment: a module can widen the container - past the mount table (`ENABLE_CVMFS` binds `/cvmfs`, - `ENABLE_MPICH_SS` adds `--privileged` and the host namespaces), so - they reach `Attestation.site_modules` and every manifest rather than - leaving `fs: declared` to overstate the boundary. `MOUNT_*` never - appears there — `project.child_env` scrubs it before the runtime - sees it. + reports the gates set for podman-hpc, which the runtime applies from + its own environment. The gate names are **read from the site's own + module table** (each module declares its `env:` key; the directory + comes from `podman-hpc infohpc`), never matched by prefix — GitHub's + `ENABLE_RUNNER_TRACING` was enough to make a prefix match name a + module the host does not have. A table that cannot be read reports + nothing rather than refusing: attestation must not fail a run. + They are recorded because a module widens the container past the + mount table — `ENABLE_CVMFS` binds `/cvmfs`, `ENABLE_MPICH_SS` adds + `--privileged` and the host namespaces — so `Attestation.site_modules` + carries them into every manifest rather than leaving `fs: declared` + to overstate the boundary. `MOUNT_*` never appears there: + `project.child_env` scrubs it before the runtime sees it. - **A stale squashed image is healed before the load.** The tag is deterministic but builds are not bit-reproducible, so a rebuild migrated under an unchanged tag would put a second same-named image diff --git a/src/lightcone/engine/container.py b/src/lightcone/engine/container.py index 1bb4d921..60300f69 100644 --- a/src/lightcone/engine/container.py +++ b/src/lightcone/engine/container.py @@ -329,16 +329,15 @@ def backend(runtime: Runtime) -> sandbox.Backend: image_id=runtime.image_id, root=runtime.root, user_flags=(*uid_flags(runtime.runtime), *pull), - site_modules=site_modules(runtime.runtime), + site_modules=site_modules(runtime.runtime, runtime.root), ) -def site_modules(runtime: str) -> tuple[str, ...]: +def site_modules(runtime: str, root: Path) -> tuple[str, ...]: """Name the site container modules the runtime will apply, if any. - podman-hpc reads a site's module table - (``/etc/podman_hpc/modules.d``) from its own environment, and a - module widens the container by more than lc declared: NERSC's + podman-hpc reads a site's module table from its own environment, and + a module widens the container by more than lc declared: NERSC's ``ENABLE_CVMFS`` binds the whole ``/cvmfs`` hierarchy — reference data a recipe can read without declaring it — and ``ENABLE_MPICH_SS`` adds ``--privileged`` plus the host's network, @@ -347,21 +346,45 @@ def site_modules(runtime: str) -> tuple[str, ...]: named in the attestation instead, so no manifest claims the mount table was the whole boundary. - Read from :func:`project.child_env`, which is what the runtime - actually receives — the ``MOUNT_*`` gates it scrubs are absent by - the time this asks. + The table is **asked for, never guessed**: each module declares its + own gate in an ``env:`` key, and the directory holding them comes + from ``podman-hpc infohpc`` (a site can move it). Matching a + ``MOUNT_``/``ENABLE_`` prefix instead would record any passing + variable that happens to share it — GitHub's ``ENABLE_RUNNER_TRACING`` + was enough to make a manifest name a module the host does not + have, which is the opposite of what the field is for. + + The environment consulted is :func:`project.child_env`, what the + runtime actually receives — so the ``MOUNT_*`` gates scrubbed there + are already gone and cannot be reported as applied. Args: runtime: The resolved runtime's name. + root: The project root, for the probe's working directory. Returns: The gate names that are set, sorted; empty for a runtime with no - module system. + module system, and empty rather than a refusal when the table + cannot be read — attestation must not fail a run. """ if runtime != "podman-hpc": return () + info = project._run(["podman-hpc", "infohpc"], cwd=root) + # Greedy up to the *last* colon: the line reads + # `modules_dir (file: modules_dir): /etc/podman_hpc/modules.d`. + found = re.search(r"^modules_dir.*:\s*(\S+)\s*$", info.stdout, re.MULTILINE) + if info.returncode != 0 or not found: + return () + gates = set() + for module in sorted(Path(found.group(1)).glob("*.yaml")): + try: + text = module.read_text() + except OSError: + continue + if gate := re.search(r"^env:\s*(\S+)", text, re.MULTILINE): + gates.add(gate.group(1)) env = project.child_env() - return tuple(sorted(k for k, v in env.items() if k.startswith("ENABLE_") and v)) + return tuple(sorted(g for g in gates if env.get(g))) def build(root: Path) -> tuple[Runtime, str]: diff --git a/tests/test_container.py b/tests/test_container.py index 02320f6a..e5a9dadd 100644 --- a/tests/test_container.py +++ b/tests/test_container.py @@ -398,8 +398,29 @@ def test_lc_build_on_a_direct_project_says_so(tmp_path: Path, fake: list[list[st container.build(plain) +def _module_table(tmp_path: Path, hpc: list[list[str]], monkeypatch: pytest.MonkeyPatch, + gates: dict[str, str]) -> None: + """Stand in for a site's podman-hpc module table: one yaml per module, + each declaring its own gate, and `infohpc` pointing at the directory.""" + modules = tmp_path / "modules.d" + modules.mkdir(exist_ok=True) + for name, gate in gates.items(): + (modules / f"{name}.yaml").write_text(f"name: {name}\nenv: {gate}\nbind:\n - /x:/x\n") + inner = project._run + + def run(argv: list[str], *, cwd: Path) -> MagicMock: + if argv[:2] == ["podman-hpc", "infohpc"]: + hpc.append(list(argv)) + return MagicMock( + returncode=0, stdout=f"modules_dir (file: modules_dir): {modules}\n", stderr="" + ) + return inner(argv, cwd=cwd) + + monkeypatch.setattr(project, "_run", run) + + def test_the_backend_names_the_site_modules_the_runtime_will_apply( - root: Path, hpc: list[list[str]], monkeypatch: pytest.MonkeyPatch + root: Path, tmp_path: Path, hpc: list[list[str]], monkeypatch: pytest.MonkeyPatch ) -> None: """ENABLE_* is left working — it is how a GPU or MPI recipe reaches its hardware — so the honesty has to come from naming it. MOUNT_* is @@ -407,8 +428,13 @@ def test_the_backend_names_the_site_modules_the_runtime_will_apply( it, which is why only one family reaches the attestation.""" monkeypatch.setenv("ENABLE_GPU", "1") monkeypatch.setenv("ENABLE_CVMFS", "1") - monkeypatch.setenv("ENABLE_UNSET", "") + monkeypatch.setenv("ENABLE_MPICH_SS", "") monkeypatch.setenv("MOUNT_HOME", "1") + _module_table( + tmp_path, hpc, monkeypatch, + {"gpu": "ENABLE_GPU", "cvmfs": "ENABLE_CVMFS", + "mpich": "ENABLE_MPICH_SS", "home": "MOUNT_HOME"}, + ) _write_archive(image.archive_path(root, image.tag(root))) runtime = container.runtime_for_run(root, build=False) @@ -417,18 +443,50 @@ def test_the_backend_names_the_site_modules_the_runtime_will_apply( ) assert attested.site_modules == ("ENABLE_CVMFS", "ENABLE_GPU") - assert "MOUNT_HOME" not in attested.site_modules - assert "ENABLE_UNSET" not in attested.site_modules + assert "MOUNT_HOME" not in attested.site_modules, "scrubbed, so never applied" + assert "ENABLE_MPICH_SS" not in attested.site_modules, "set but empty: not enabled" + +def test_only_the_table_s_own_gates_are_named( + root: Path, tmp_path: Path, hpc: list[list[str]], monkeypatch: pytest.MonkeyPatch +) -> None: + """The gates are read from the site's module table, never matched by + prefix: GitHub's own ENABLE_RUNNER_TRACING was enough to make a + manifest name a module the host does not have.""" + monkeypatch.setenv("ENABLE_RUNNER_TRACING", "true") + monkeypatch.setenv("ENABLE_GPU", "1") + _module_table(tmp_path, hpc, monkeypatch, {"gpu": "ENABLE_GPU"}) -def test_a_runtime_without_a_module_system_names_none(root: Path, fake: list[list[str]], - monkeypatch: pytest.MonkeyPatch) -> None: + assert container.site_modules("podman-hpc", root) == ("ENABLE_GPU",) + + +def test_an_unreadable_module_table_names_nothing_rather_than_refusing( + root: Path, hpc: list[list[str]], monkeypatch: pytest.MonkeyPatch +) -> None: + """Attestation must never fail a run: a probe that cannot answer + reports nothing, the same discipline as uv_version.""" + monkeypatch.setenv("ENABLE_GPU", "1") + inner = project._run + monkeypatch.setattr( + project, + "_run", + lambda argv, *, cwd: MagicMock(returncode=1, stdout="", stderr="boom") + if argv[:2] == ["podman-hpc", "infohpc"] + else inner(argv, cwd=cwd), + ) + + assert container.site_modules("podman-hpc", root) == () + + +def test_a_runtime_without_a_module_system_names_none( + root: Path, fake: list[list[str]], monkeypatch: pytest.MonkeyPatch +) -> None: """podman and docker read no site module table, so an ENABLE_* in the environment is inert there and must not imply otherwise.""" monkeypatch.setenv("ENABLE_GPU", "1") - assert container.site_modules("podman") == () - assert container.site_modules("docker") == () + assert container.site_modules("podman", root) == () + assert container.site_modules("docker", root) == () # ---- the containerized converge ---------------------------------------------