diff --git a/CLAUDE.md b/CLAUDE.md index 246ff4b..35cee04 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -788,7 +788,7 @@ annotated: `project.child_env` scrubs ambient `UV_*` outside a plumbing allowlist (`_UV_KEPT` — cache dir, link mode, the managed-interpreter store and its mirror, timeouts, TLS, air-gap, credentials, uv's own recursion guard), and every uv-acting verb names the non-empty -variables dropped through `project.uv_scrub_warning` — one composer, +variables dropped through `project.scrub_warning` — one composer, one predicate with the scrub: convergence puts it in the report (so `lc init` says it), materialize in its warnings, and the probe in its outcome's notes, which is why the CLI never composes it (issue #179). @@ -2141,6 +2141,90 @@ unlinks before writing; a new tampering test should too. materialize runs for hours) arrives with the submission-model venue. +- **The NERSC seamlessness pass** (2026-08, post-spike — findings in + issue #192): + - *The site registry supplies uv's plumbing* — spec §4's row, made + real: `child_env` fills unset/empty `UV_CACHE_DIR` and + `UV_PYTHON_INSTALL_DIR` from `$SCRATCH/.lightcone/` on a known + center (`venue.site_env`; the `_Site` row gained `scratch`). + Both move state off a home filesystem measurement found unusable + from compute nodes, in two independent ways: `flock` fails there + outright (os error 524), killing every uv-touching verb, and an + interpreter stored there starts in 0.4–9 s rather than 0.05 s — + enough, under load, to exceed uv's own 60 s startup ceiling and + fail a sync mid-run with "Python startup timed out". Both are + disposable, so a scratch purge costs a re-download, never a + result; `_UV_KEPT` already kept both for exactly this story. + Per variable, ambient always wins; silent by design — the scrub + warning is for user configuration that stopped taking effect, not + for a default doing its job. Deterministic in the ambient + environment, so every node of an allocation derives the same + answer with nothing handed down. + - *`MOUNT_*` is scrubbed; `ENABLE_*` is kept and **attested**.* + podman-hpc's site modules are env-gated. The mount gates bind + `$HOME`/`$SCRATCH`/`$CFS` into every container — undeclared inputs + under a manifest attesting `fs: declared` (measured) — so they are + dropped, and named by `scrub_warning` (the `uv_scrub_warning` + rename). The `ENABLE_*` gates are **not** scrubbed: they are how a + GPU or MPI recipe reaches its hardware, and removing them would + take a working capability away with no replacement. + But the first draft kept them on the stated ground that they bind + "system libraries only, never a channel undeclared inputs arrive + through", and review measured that false: `ENABLE_CVMFS` binds the + whole populated `/cvmfs` hierarchy (reference data a recipe can + read undeclared) and `ENABLE_MPICH_SS` adds `--privileged` with the + host's network, pid and ipc namespaces — the mount table stops + being the boundary at all. Only `ENABLE_GPU`/`ENABLE_NCCL*` fit the + original description. So the honesty comes from the record instead: + `container.site_modules()` names the set gates, `OCIBackend` carries + them as a resolved field (the `user_flags` pattern — `wrap` stays + pure), and `Attestation.site_modules` puts them in every manifest. + The gate names are **asked of the site's module table** (each + module declares its own `env:` key; the directory comes from + `podman-hpc infohpc`), never matched by prefix: CI caught a prefix + match recording GitHub's own `ENABLE_RUNNER_TRACING` as a module + the host does not have, which is the exact dishonesty the field + exists to prevent. Unreadable table ⇒ empty, never a refusal — + attestation must not fail a run, the `uv_version` discipline. + This is the one attestation value not derived from lc's own argv, + and the docstring says so: the runtime applies them from its own + environment. A per-module allowlist was rejected — the module table + is site-configurable, so a center adding a data-binding module lc + has never heard of would slip through a hardcoded judgement. + - *A stale squashed image is healed before the load.* The tag is + deterministic, builds are not bit-reproducible, and podman-hpc's + read-only squash store refuses two images under one name — after + which *every* storage operation fails (measured: a same-tag + rebuild wedged the store for runs, listings, everything). + `container._heal_squash` probes and `rmsqi`s the stale copies; an + unrecognized probe outcome is left for migrate's own loud path, + so the heal cannot break a healthy run. Before the *load*, not + just the migrate — a wedged store fails the load too on a node + that has not loaded yet. Removal is **by id, one call per stale + image**: `rmsqi ` resolves a single record (podman-hpc's + `remove_image` → `get_img_info` takes the first name match), so + against the very state being healed it can take the *current* + image and leave the stale one to re-wedge the store on the next + migrate. Only the wedged branch spells the tag — a store that + cannot be listed cannot be enumerated — and it re-probes, bounded + by `_HEAL_ATTEMPTS`, because each blind pass is a real deletion. + Recorded hazard, accepted: the squash store is shared across a + user's nodes and allocations, so a heal can drop layers an + earlier run is still executing from. Leaving the store wedged + breaks that run too, and cross-run coordination is not something + a heal can offer. + - *The driver releases each future as its result is consumed* + (`_Dask.completed`). A key still client-held at teardown makes + `retire_workers` replicate it onto peers that are also retiring — + every clean SLURM run stalled past the reap grace and ended in + "srun: forcing job termination" / "task Killed" / lose-tasks + warnings (measured, and measured gone: the fix cut a clean 2-node + run by the full escalation grace). Warnings on an *unclean* path + still fire, honestly. + - The downgrade note now names the LSM list first — Perlmutter's + kernels are 6.4 with Landlock simply absent from the boot list, + so "kernel < 5.13" was three causes none of which applied. + ### Recorded deviations from the spec - **No `AGENTS.md` scaffolding** (spec §2 calls for an agent notes diff --git a/docs/api/container.md b/docs/api/container.md index 899f516..8e58c9f 100644 --- a/docs/api/container.md +++ b/docs/api/container.md @@ -59,6 +59,31 @@ Sources: `src/lightcone/engine/image.py`, outside the load branch) and joins `_SHARED_STORE_RUNTIMES`. Detection order podman-hpc → podman → docker; docker's daemon is probed at detection. +- **Site container modules are named, not silenced.** `site_modules()` + reports the gates set for podman-hpc, which the runtime applies from + its own environment. The gate names are **read from the site's own + module table** (each module declares its `env:` key; the directory + comes from `podman-hpc infohpc`), never matched by prefix — GitHub's + `ENABLE_RUNNER_TRACING` was enough to make a prefix match name a + module the host does not have. A table that cannot be read reports + nothing rather than refusing: attestation must not fail a run. + They are recorded because a module widens the container past the + mount table — `ENABLE_CVMFS` binds `/cvmfs`, `ENABLE_MPICH_SS` adds + `--privileged` and the host namespaces — so `Attestation.site_modules` + carries them into every manifest rather than leaving `fs: declared` + to overstate the boundary. `MOUNT_*` never appears there: + `project.child_env` scrubs it before the runtime sees it. +- **A stale squashed image is healed before the load.** The tag is + deterministic but builds are not bit-reproducible, so a rebuild + migrated under an unchanged tag would put a second same-named image + into podman-hpc's read-only squash store — after which every storage + operation fails. `_heal_squash` probes the store and `rmsqi`s each + stale copy **by id** first (before the load, which a wedged store + also fails) — `rmsqi ` resolves only one record and could take + the current image instead. A store too wedged to list is removed by + tag and re-probed, bounded by `_HEAL_ATTEMPTS`; a probe outcome the + heal does not recognize is left for migrate's own loud path, so it + can never break a healthy run. - **The architecture gate refuses before the load** — a wrong-arch `load` succeeds and then dies as `exec format error` deep inside a recipe. Ignorance passes; a recorded mismatch refuses, naming the diff --git a/docs/api/materialize.md b/docs/api/materialize.md index 54b570d..8428093 100644 --- a/docs/api/materialize.md +++ b/docs/api/materialize.md @@ -15,7 +15,7 @@ Source: `src/lightcone/engine/materialize.py`. | `check(root, targets, *, refresh)` | The same classification without executing, committing, or fetching. Exempt from the dirty refusal. | | `status(root)` | The report: every output's state and provenance commit, plus the mode/image/sandbox header facts. | | `MaterializeReport` / `StatusReport` | The JSON surfaces; `ok` and `up_to_date` first. | -| `cluster_for_run()` | The venue ladder, and the two-method scheduler seam (`submit`, `completed`). | +| `cluster_for_run()` | The venue ladder, and the two-method scheduler seam (`submit`, `completed`). `completed` releases each future as its result lands — a key still held at teardown makes the scheduler fight worker retirement, and every clean SLURM run ends in "srun: forcing job termination". | | `run_record(...)` / `datalad_run_subject(...)` | The commit message `datalad rerun` replays, and the one spelling of its subject line — shared with the foreign-write comparator, because two strings here would drift. | | `_engine_requirement()` | How a record pins its engine: by version for a release, by source commit (hatch-vcs) for a dev build. | diff --git a/docs/api/project.md b/docs/api/project.md index d2d0797..1ca3fdb 100644 --- a/docs/api/project.md +++ b/docs/api/project.md @@ -19,6 +19,7 @@ Source: `src/lightcone/engine/project.py` (+ | `uv_prefix(root, *, sync)` | The one spelling of the project uv hop. Callers differ only in `sync`: a probe converges the environment, a recipe must not. | | `project_name(dir)` | PEP 503-ish name from the directory name. | | `_run` / `_check_call` | Every external tool invocation, and the suite's one monkeypatch point. | +| `child_env()` / `scrub_warning()` | The one composer of the environment external tools run in: drops `VIRTUAL_ENV`, ambient `UV_*` outside the `_UV_KEPT` plumbing allowlist, and `MOUNT_*` (a site container module's mount gates); supplies a known center's uv plumbing (`UV_CACHE_DIR`, `UV_PYTHON_INSTALL_DIR`) from its scratch root where unset (`venue.site_env`). The warning names every non-empty variable dropped. | | `ProjectError` | The engine's one exception; the CLI translates it once. | ## What must stay true diff --git a/docs/api/venue.md b/docs/api/venue.md index d86735d..6fca2ef 100644 --- a/docs/api/venue.md +++ b/docs/api/venue.md @@ -17,7 +17,7 @@ Source: `src/lightcone/engine/venue.py` (consumed by | `slurm_client()` | The allocation branch: a scheduler in the driver process bound to `SLURMD_NODENAME`, one `srun --overlap` launching a worker per node on `sys.executable`. | | `require_compute_node(command)` | The login guard: refuses iff a known center's marker is set and `SLURM_JOB_ID` is not, printing that center's own `salloc`/`sbatch` spellings. | | `allocation_nodes()` | How many nodes the allocation holds; 0 outside one. | -| `_SITES` | One row per known center — name, marker, remedies, **verified against the center's documentation, never guessed**. NERSC is the seeded row. | +| `_SITES` | One row per known center — name, marker, remedies, and the scratch variable that hosts uv's cache and interpreter store where home cannot (`site_env`, consumed by `project.child_env`), **verified against the center's documentation, never guessed**. NERSC is the seeded row. | ## What must stay true diff --git a/docs/user/cluster.md b/docs/user/cluster.md index f422897..70e4794 100644 --- a/docs/user/cluster.md +++ b/docs/user/cluster.md @@ -106,6 +106,70 @@ know: refuses a multi-node containerized run unless the runtime is `podman-hpc`. Single-node allocations work with any runtime. +## NERSC (Perlmutter) specifics + +Facts of the site that shape how `lc` behaves there — all verified on +Perlmutter itself: + +- **uv's cache and interpreters are put on scratch for you.** The + home filesystem is unusable from compute nodes in two separate ways: + it cannot take uv's cache lock at all (`flock` fails with os error + 524), and an interpreter stored there starts in 0.4-9 s instead of + 0.05 s, slow enough under load to exceed uv's own 60 s startup + ceiling and fail a sync mid-run. So on NERSC machines `lc` runs uv + with `UV_CACHE_DIR=$SCRATCH/.lightcone/uv-cache` and + `UV_PYTHON_INSTALL_DIR=$SCRATCH/.lightcone/uv-python` — each only + where you have not set that variable yourself: + + ```bash + export UV_CACHE_DIR=$SCRATCH/my-uv-cache # optional; this wins + ``` + + Both are disposable by design, so scratch purge policies cost a + re-download rather than a result. + +- **The sandbox does not enforce here.** Perlmutter's kernels boot + without Landlock in the LSM list, so direct-mode recipes run + unconfined and every run says so: + + ```text + not sandboxed on this host — landlock unavailable (not in the kernel's boot-time LSM list, kernel < 5.13, blocked by seccomp, or unsupported arch); recorded as `fs: open` + ``` + + Manifests record `fs: open` honestly. A containerized project is + the real boundary on this site: recipes see only the declared mount + table, attested as `fs: declared`. + +- **`MOUNT_*` is ignored; `ENABLE_*` passes through.** podman-hpc's + site modules are switched by environment variables. The mount gates + (`MOUNT_HOME`, `MOUNT_SCRATCH`, `MOUNT_CFS`) would bind-mount those + filesystems into every recipe container — undeclared inputs under a + manifest attesting `fs: declared` — so `lc` drops them and says so: + + ```text + ignored ambient MOUNT_HOME — a site container module would bind-mount undeclared host directories into recipe containers + ``` + + The `ENABLE_*` gates survive, because they are how a GPU or MPI + recipe reaches the hardware it was written for — a GPU recipe gets + CUDA by exporting `ENABLE_GPU=1`, the site's own mechanism. They are + **recorded in the manifest instead**, under + `hermeticity.site_modules`, because a module can widen the container + well past the mounts `lc` declared: `ENABLE_CVMFS` binds the whole + `/cvmfs` hierarchy, and `ENABLE_MPICH_SS` adds `--privileged` plus + the host's network, pid and ipc namespaces. If you enable one, the + outputs of that run say so. + +- **BLAS may be silently throttled.** NERSC's allocation environment + exports `OMP_NUM_THREADS` (often `2`), and `srun` propagates it into + every step — so numpy or BLAS inside a recipe uses 2 of the node's + 256 hardware threads unless you say otherwise. `lc` does not touch + the variable; set it yourself before materializing: + + ```bash + export OMP_NUM_THREADS=32 # or whatever your recipe's math wants + ``` + ## Data on parallel filesystems Keep active projects on the filesystem your center recommends for job diff --git a/docs/user/troubleshooting.md b/docs/user/troubleshooting.md index 73f0b2a..a89c632 100644 --- a/docs/user/troubleshooting.md +++ b/docs/user/troubleshooting.md @@ -190,6 +190,12 @@ git config --global user.email "ada@example.org" refused before the recipe would have died mid-run. Build on a matching host (on NERSC, a login node), commit, push, and pull on the other side. +- **"read-only image store assigns the same name to multiple images"** + (podman-hpc) — two builds of the same image tag landed in the shared + squash store, and podman then refuses every storage operation. `lc` + heals this itself before each migrate, so a run clears it; if a + store was wedged by something other than `lc`, the manual remedy is + `podman-hpc rmsqi ` for the name podman's error names. ## Filing a bug diff --git a/src/lightcone/_sandbox_exec.py b/src/lightcone/_sandbox_exec.py index 9ea94ce..b151c21 100644 --- a/src/lightcone/_sandbox_exec.py +++ b/src/lightcone/_sandbox_exec.py @@ -305,7 +305,10 @@ def main(argv: list[str] | None = None) -> None: raise ValueError("policy must be a JSON object") abi_level = abi() if abi_level == 0: - raise ValueError("landlock unavailable (kernel < 5.13, or blocked by seccomp)") + raise ValueError( + "landlock unavailable (not in the kernel's boot-time LSM list, " + "kernel < 5.13, or blocked by seccomp)" + ) fd = build_ruleset(policy, abi_level) restrict_self(fd) os.close(fd) diff --git a/src/lightcone/engine/container.py b/src/lightcone/engine/container.py index 0a7b854..60300f6 100644 --- a/src/lightcone/engine/container.py +++ b/src/lightcone/engine/container.py @@ -48,6 +48,12 @@ #: the multi-node materialize refusal stands on. _SHARED_STORE_RUNTIMES = ("podman-hpc",) +#: How many blind removals the squash heal will make against a store it +#: cannot list. Bounded rather than looped-until-clean: each pass is a +#: real deletion, and a store still unlistable after this many is not +#: one more `rmsqi` away from working. +_HEAL_ATTEMPTS = 3 + @dataclass(frozen=True) class Runtime: @@ -134,6 +140,11 @@ def runtime_for_run(root: Path, *, build: bool) -> Runtime: _fetch(root, archive) image_id, arch = archive_identity(archive) _require_arch(root, archive, arch) # before the load — see its docstring + if name == "podman-hpc": + # Before the load, not just the migrate: a wedged squash store + # (see _heal_squash) fails the load too, on a node whose local + # store does not hold the image yet. + _heal_squash(root, tag, image_id) if not _loaded(root, name, image_id): _check_call([name, "load", "-i", str(archive)], cwd=root) if name == "podman-hpc": @@ -194,6 +205,60 @@ def _committed(archive: Path) -> bool: return archive.exists() or archive.is_symlink() +def _heal_squash(root: Path, tag: str, image_id: str) -> None: + """Remove stale squashed images before migrating the current one. + + The tag is deterministic but builds are not bit-reproducible, so a + rebuild migrated under an unchanged tag puts a second image with the + same name into podman-hpc's read-only squash store — after which + *every* storage operation fails ("read-only image store assigns the + same name to multiple images"), runs included (measured on + Perlmutter). ``rmsqi`` drops only the squashed copy, and the migrate + that follows re-squashes the current id. + + Removal is **by id, one call per stale image**: ``rmsqi `` + resolves a single record and takes that one, so against the very + state this heals — two records sharing a name — it can just as + easily take the current image and leave the stale one, re-wedging + the store on the next migrate. Only the wedged branch has to spell + the tag, because a store that cannot be listed cannot be enumerated; + it re-probes so a second stale record is still reached. + + A heal must never break a run whose store is healthy, so a probe + outcome it does not recognize is left for migrate's own loud path. + + Recorded hazard: the squash store is shared across a user's nodes + and allocations, so removing a stale image drops layers that a run + started earlier — under a tag whose id has since moved — may still + be executing from. The alternative is leaving the store wedged for + every later verb, which breaks that run too; cross-run coordination + is not something a heal can offer. + """ + for _ in range(_HEAL_ATTEMPTS): + probe = project._run( + [ + "podman-hpc", "images", "--format", + "{{.Id}} {{.ReadOnly}}", f"localhost/{tag}", + ], # fmt: skip + cwd=root, + ) + if probe.returncode == 0: + rows = [line.split() for line in probe.stdout.splitlines()] + stale = { + row[0] + for row in rows + if len(row) == 2 and row[1] == "true" and row[0] != image_id + } + for victim in sorted(stale): + _check_call(["podman-hpc", "rmsqi", victim], cwd=root) + return + # A wedged store fails the listing too, with the same message + # every other operation gets — the shape a pre-heal lc left. + if "assigns the same name to multiple images" not in probe.stderr: + return + _check_call(["podman-hpc", "rmsqi", tag], cwd=root) + + def runtime_name(root: Path) -> str: """Detect which container runtime this host offers. @@ -264,9 +329,64 @@ def backend(runtime: Runtime) -> sandbox.Backend: image_id=runtime.image_id, root=runtime.root, user_flags=(*uid_flags(runtime.runtime), *pull), + site_modules=site_modules(runtime.runtime, runtime.root), ) +def site_modules(runtime: str, root: Path) -> tuple[str, ...]: + """Name the site container modules the runtime will apply, if any. + + podman-hpc reads a site's module table from its own environment, and + a module widens the container by more than lc declared: NERSC's + ``ENABLE_CVMFS`` binds the whole ``/cvmfs`` hierarchy — reference + data a recipe can read without declaring it — and + ``ENABLE_MPICH_SS`` adds ``--privileged`` plus the host's network, + pid and ipc namespaces. They are left working, because they are how + a GPU or MPI recipe reaches the hardware it was written for, and + named in the attestation instead, so no manifest claims the mount + table was the whole boundary. + + The table is **asked for, never guessed**: each module declares its + own gate in an ``env:`` key, and the directory holding them comes + from ``podman-hpc infohpc`` (a site can move it). Matching a + ``MOUNT_``/``ENABLE_`` prefix instead would record any passing + variable that happens to share it — GitHub's ``ENABLE_RUNNER_TRACING`` + was enough to make a manifest name a module the host does not + have, which is the opposite of what the field is for. + + The environment consulted is :func:`project.child_env`, what the + runtime actually receives — so the ``MOUNT_*`` gates scrubbed there + are already gone and cannot be reported as applied. + + Args: + runtime: The resolved runtime's name. + root: The project root, for the probe's working directory. + + Returns: + The gate names that are set, sorted; empty for a runtime with no + module system, and empty rather than a refusal when the table + cannot be read — attestation must not fail a run. + """ + if runtime != "podman-hpc": + return () + info = project._run(["podman-hpc", "infohpc"], cwd=root) + # Greedy up to the *last* colon: the line reads + # `modules_dir (file: modules_dir): /etc/podman_hpc/modules.d`. + found = re.search(r"^modules_dir.*:\s*(\S+)\s*$", info.stdout, re.MULTILINE) + if info.returncode != 0 or not found: + return () + gates = set() + for module in sorted(Path(found.group(1)).glob("*.yaml")): + try: + text = module.read_text() + except OSError: + continue + if gate := re.search(r"^env:\s*(\S+)", text, re.MULTILINE): + gates.add(gate.group(1)) + env = project.child_env() + return tuple(sorted(g for g in gates if env.get(g))) + + def build(root: Path) -> tuple[Runtime, str]: """Converge the system-layer image, the whole of ``lc build``. @@ -377,6 +497,18 @@ def sync(root: Path, runtime: Runtime) -> list[str]: if asked.returncode != 0: raise ProjectError(f"`uv cache dir` failed:\n{asked.stderr.strip()}") cache = asked.stdout.strip() + # A bind mount's source must exist — and in containerized mode uv + # never runs on the host, so nothing else has created it. Otherwise + # the first sync of a fresh project dies as the runtime's `statfs` + # error about a path the user never named. + try: + Path(cache).mkdir(parents=True, exist_ok=True) + except OSError as e: + raise ProjectError( + f"uv's cache directory `{cache}` cannot be created ({e}). It is " + "mounted into the image to converge the environment; set " + "UV_CACHE_DIR to a writable path and retry." + ) from e argv = [ runtime.runtime, "run", "--rm", "--entrypoint", "", # Same reason as the exec boundary's flag: SELinux hosts refuse diff --git a/src/lightcone/engine/materialize.py b/src/lightcone/engine/materialize.py index 331d7b3..8517727 100644 --- a/src/lightcone/engine/materialize.py +++ b/src/lightcone/engine/materialize.py @@ -493,7 +493,7 @@ def materialize( project.require_git_annex() dataset.require_committer(root) report = MaterializeReport() - if warning := project.uv_scrub_warning(): + if warning := project.scrub_warning(): report.warnings.append(warning) # The dirty check comes before anything that writes: the image # converge below *commits*, and `dataset.save` stages scoped but @@ -698,7 +698,16 @@ def completed(self, handles: list[Any]) -> Iterator[worker.TaskResult]: # annotated rather than the module exempted. from distributed import as_completed - for _, result in as_completed(handles, with_results=True): # type: ignore[no-untyped-call] + for fut, result in as_completed(handles, with_results=True): # type: ignore[no-untyped-call] + # Released the moment its result is local: a future still + # held at teardown makes the scheduler preserve its key — + # retiring workers then replicate every key onto each other + # (all retiring), stall past the reap grace, and get killed, + # ending every clean SLURM run in "srun: forcing job + # termination". A dependent's claim on the data is + # scheduler-side and unaffected; an abort mid-loop still + # holds the rest, and warnings on an unclean path are honest. + fut.release() yield result diff --git a/src/lightcone/engine/project.py b/src/lightcone/engine/project.py index c907216..3541bb8 100644 --- a/src/lightcone/engine/project.py +++ b/src/lightcone/engine/project.py @@ -198,7 +198,7 @@ def converge(directory: Path, *, write: bool = True) -> ConvergenceReport: require_git_annex() c = _Converger(write=write) - if warning := uv_scrub_warning(): + if warning := scrub_warning(): c.warn(warning) if write: @@ -752,33 +752,65 @@ def _uv_scrubbed(name: str) -> bool: ) +def _mount_scrubbed(name: str) -> bool: + """Decide whether one ambient variable is dropped by the mount scrub. + + podman-hpc's site modules are gated on ambient environment variables + (``/etc/podman_hpc/modules.d``), and the ``MOUNT_*`` family + (``MOUNT_HOME``, ``MOUNT_SCRATCH``, ``MOUNT_CFS`` at NERSC) + bind-mounts host trees into every container the runtime launches — + undeclared inputs arriving under a manifest that attests + ``fs: declared``. ``ENABLE_*`` is deliberately kept: those modules + bind CUDA and MPI system libraries, the same generosity class as the + ``/dev`` and ``/sys`` grants, and none is a channel undeclared + inputs arrive through. + """ + return name.startswith("MOUNT_") + + def child_env() -> dict[str, str]: """Build the environment external tools run in. - Ours, minus ``VIRTUAL_ENV`` and minus every ``UV_*`` variable outside - the :data:`_UV_KEPT` plumbing allowlist. Every uv invocation names its - project explicitly, so an activated environment elsewhere is never - what we mean — and an ambient install setting would change what a - sync installs without moving ``env_version``, which is the identity - hole the scrub closes. + Ours, minus ``VIRTUAL_ENV``, minus every ``UV_*`` variable outside + the :data:`_UV_KEPT` plumbing allowlist, and minus ``MOUNT_*`` (a + site container module's mount gates — see :func:`_mount_scrubbed`). + Every uv invocation names its project explicitly, so an activated + environment elsewhere is never what we mean — and an ambient install + setting would change what a sync installs without moving + ``env_version``, which is the identity hole the scrub closes. + + On a known center, the uv plumbing whose home-filesystem default is + unusable from compute nodes is supplied from the center's scratch + root (:func:`venue.site_env`) — per variable, and only where the + user set none, so an ambient value always wins. The rule is + deterministic in the ambient environment, so every node of an + allocation derives the same answer with nothing handed down. Returns: - The current environment without ``VIRTUAL_ENV`` or scrubbed ``UV_*``. + The current environment, scrubbed, with the site plumbing + supplied. """ - return { + env = { k: v for k, v in os.environ.items() - if k != "VIRTUAL_ENV" and not _uv_scrubbed(k) + if k != "VIRTUAL_ENV" and not _uv_scrubbed(k) and not _mount_scrubbed(k) } + # Function-local: venue imports ProjectError from this module. + from lightcone.engine import venue + for name, value in venue.site_env().items(): + if not env.get(name): + env[name] = value + return env -def uv_scrub_warning() -> str: + +def scrub_warning() -> str: """Compose the dropped-ambient-variables warning, once for every verb. A user whose ``UV_PYTHON`` or ``UV_INDEX_URL`` stopped steering uv deserves a pointer to why on *whichever* verb they hit first — ``lc init`` resolving against the wrong index fails with uv's raw - error otherwise. One spelling here, and one predicate with + error otherwise. One spelling here, and one predicate per scrub with :func:`child_env`, so the verbs cannot drift from each other or the report from the scrub. Empty variables steer nothing and are not reported. @@ -786,13 +818,20 @@ def uv_scrub_warning() -> str: Returns: The warning, or ``""`` when nothing non-empty was dropped. """ + clauses = [] if dropped := sorted(k for k, v in os.environ.items() if v and _uv_scrubbed(k)): - return ( + clauses.append( f"ignored ambient {', '.join(dropped)} — an install setting is " "the project's to declare (pyproject.toml), and an ambient one " "would steer uv without moving env_version" ) - return "" + if mounts := sorted(k for k, v in os.environ.items() if v and _mount_scrubbed(k)): + clauses.append( + f"ignored ambient {', '.join(mounts)} — a site container module " + "would bind-mount undeclared host directories into recipe " + "containers" + ) + return "; ".join(clauses) def uv_version(directory: Path) -> str: diff --git a/src/lightcone/engine/run.py b/src/lightcone/engine/run.py index 766ec59..badbb61 100644 --- a/src/lightcone/engine/run.py +++ b/src/lightcone/engine/run.py @@ -24,8 +24,8 @@ SPEC_FILENAME, child_env, require_uv, + scrub_warning, uv_prefix, - uv_scrub_warning, ) @@ -77,7 +77,7 @@ def probe(project: Path, command: Sequence[str]) -> sandbox.Outcome: ) # The probe is what called `child_env`, so the probe's outcome is # where the scrub's fact belongs — the caller prints notes verbatim. - if warning := uv_scrub_warning(): + if warning := scrub_warning(): outcome = replace(outcome, notes=(warning, *outcome.notes)) return outcome diff --git a/src/lightcone/engine/sandbox/landlock.py b/src/lightcone/engine/sandbox/landlock.py index ed439a7..cfcd7ba 100644 --- a/src/lightcone/engine/sandbox/landlock.py +++ b/src/lightcone/engine/sandbox/landlock.py @@ -38,7 +38,10 @@ def capability() -> Capability: return Capability(kind="landlock", landlock_abi=abi) return Capability( kind="none", - detail="landlock unavailable (kernel < 5.13, blocked by seccomp, or unsupported arch)", + detail=( + "landlock unavailable (not in the kernel's boot-time LSM list, " + "kernel < 5.13, blocked by seccomp, or unsupported arch)" + ), ) diff --git a/src/lightcone/engine/sandbox/model.py b/src/lightcone/engine/sandbox/model.py index b8ad809..66aa8b4 100644 --- a/src/lightcone/engine/sandbox/model.py +++ b/src/lightcone/engine/sandbox/model.py @@ -98,6 +98,15 @@ class Attestation: network: Literal["allowed", "denied"] = "allowed" landlock_abi: int | None = None exec_allowlist_version: int | None = None + #: Site container modules the runtime applied on top of the mount + #: table, named by the gates that enabled them. A module widens the + #: world by more than lc declared — NERSC's ``ENABLE_CVMFS`` binds + #: the whole ``/cvmfs`` hierarchy, ``ENABLE_MPICH_SS`` adds + #: ``--privileged`` and the host's network, pid and ipc namespaces — + #: so ``fs: declared`` alone would overstate what was enforced. + #: Naming them is what keeps the record honest while leaving the + #: site's own GPU and MPI mechanisms working. + site_modules: tuple[str, ...] = () class Backend(Protocol): diff --git a/src/lightcone/engine/sandbox/oci.py b/src/lightcone/engine/sandbox/oci.py index 0acbf4e..87e2253 100644 --- a/src/lightcone/engine/sandbox/oci.py +++ b/src/lightcone/engine/sandbox/oci.py @@ -46,6 +46,13 @@ class OCIBackend: #: resolved by the caller so the wrap stays a pure function of its #: fields (``--userns=keep-id`` / ``--user uid:gid``). user_flags: tuple[str, ...] = () + #: Site container modules this runtime will apply on top of the + #: mount table, named by the gates that enable them. Resolved by the + #: caller, like ``user_flags``, because the runtime reads them from + #: its own environment and the wrap stays a pure function of its + #: fields. Reported by :meth:`attest`, never acted on here — a + #: module is the site's mechanism, applied by the runtime itself. + site_modules: tuple[str, ...] = () contains_prefix: bool = True @property @@ -116,6 +123,12 @@ def attest(self, policy: Policy) -> Attestation: every mechanism gives, because lc does not control the network anywhere and the attestation says only what was enforced. + The exception is :attr:`site_modules`, which the *runtime* + applies from its own environment rather than from this argv. + They are named rather than silently dropped: a module can widen + the world well past the mount table, so a record saying only + ``fs: declared`` would overstate what bounded the run. + Args: policy: The policy being wrapped. @@ -125,4 +138,5 @@ def attest(self, policy: Policy) -> Attestation: return Attestation( mechanism=self.runtime, fs="declared", + site_modules=self.site_modules, ) diff --git a/src/lightcone/engine/venue.py b/src/lightcone/engine/venue.py index 9acdcbd..222e921 100644 --- a/src/lightcone/engine/venue.py +++ b/src/lightcone/engine/venue.py @@ -66,6 +66,12 @@ class _Site: salloc: str #: The batch form of the same request; the guard appends ``--wrap``. sbatch: str + #: The environment variable naming the center's per-user scratch + #: root, which is where uv's disposable state has to live when the + #: center's home filesystem is unusable from compute nodes (see + #: :func:`site_env`). ``None`` for a center whose home works + #: everywhere. + scratch: str | None = None #: The centers the guard knows. Supporting another is one row here — @@ -77,10 +83,45 @@ class _Site: marker="NERSC_HOST", salloc="salloc --nodes=1 --constraint=cpu --qos=interactive --time=02:00:00", sbatch="sbatch --nodes=1 --constraint=cpu --qos=regular --time=02:00:00", + scratch="SCRATCH", ), ) +def site_env() -> dict[str, str]: + """The uv plumbing a known center's filesystems require. + + Both entries move state off the center's home filesystem, which is + the one thing measurement said is unusable from its compute nodes — + and both are disposable by design, so scratch purge policies cost a + re-download rather than a result: + + * ``UV_CACHE_DIR`` — home cannot take uv's cache lock at all + (measured on Perlmutter: ``flock`` fails with os error 524), so + every uv-touching verb dies without this. + * ``UV_PYTHON_INSTALL_DIR`` — an interpreter on home starts in + 0.4–9 s rather than 0.05 s (measured, same node), which under + load exceeds uv's own 60 s startup ceiling and fails a sync + mid-run with "Python startup timed out". + + Paths are derived, never configured, and the caller applies each + only where the user has not set that variable themselves. + + Returns: + The variables to supply, empty off a known center or where its + scratch variable is unset. + """ + for site in _SITES: + if site.marker in os.environ and site.scratch: + if root := os.environ.get(site.scratch): + base = os.path.join(root, ".lightcone") + return { + "UV_CACHE_DIR": os.path.join(base, "uv-cache"), + "UV_PYTHON_INSTALL_DIR": os.path.join(base, "uv-python"), + } + return {} + + def require_compute_node(command: str = "lc materialize") -> None: """Refuse to execute recipes on a known HPC center's login node. diff --git a/tests/conftest.py b/tests/conftest.py index 04a0188..e66a218 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -47,21 +47,22 @@ def venue_env(monkeypatch: pytest.MonkeyPatch) -> None: @pytest.fixture(autouse=True) def ambient_uv(monkeypatch: pytest.MonkeyPatch) -> None: - """Strip scrubbable ``UV_*`` out of the suite's environment. + """Strip scrubbable variables out of the suite's environment. CI pins its matrix interpreter through an ambient ``UV_PYTHON``, which the scrub correctly drops and reports — so without this every converge in the suite carries the warning and every ``warnings == - []`` assertion depends on the host. Derived from the scrub's own - predicate, so a variable the allowlist later admits stops being + []`` assertion depends on the host; a dev shell exporting a site's + ``MOUNT_*`` gates is the same shape. Derived from the scrub's own + predicates, so a variable the allowlist later admits stops being stripped here for free; the scrub tests set their own variables back deliberately. """ import os - from lightcone.engine.project import _uv_scrubbed + from lightcone.engine.project import _mount_scrubbed, _uv_scrubbed - for name in [k for k in os.environ if _uv_scrubbed(k)]: + for name in [k for k in os.environ if _uv_scrubbed(k) or _mount_scrubbed(k)]: monkeypatch.delenv(name) diff --git a/tests/test_container.py b/tests/test_container.py index dc7a8f6..e5a9dad 100644 --- a/tests/test_container.py +++ b/tests/test_container.py @@ -17,7 +17,7 @@ import pytest -from lightcone.engine import container, image, project +from lightcone.engine import container, image, project, sandbox from lightcone.engine.project import ProjectError _TABLE = '[tool.lightcone.image]\napt-install = ["bc"]\n' @@ -53,7 +53,7 @@ def _write_archive(path: Path, config: bytes = _CONFIG) -> str: @pytest.fixture -def fake(monkeypatch: pytest.MonkeyPatch) -> list[list[str]]: +def fake(monkeypatch: pytest.MonkeyPatch, tmp_path: Path) -> list[list[str]]: """A podman-having host: records argv, models each command's effect. Every tool name resolves except ``podman-hpc`` — a site wrapper no @@ -77,7 +77,10 @@ def run(argv: list[str], *, cwd: Path) -> MagicMock: return MagicMock(returncode=0, stdout="", stderr="") return MagicMock(returncode=0, stdout="", stderr="") if argv[:2] == ["uv", "cache"]: - return MagicMock(returncode=0, stdout="/home/user/.cache/uv\n", stderr="") + # Under tmp_path, not a fictional `/home/user`: the sync + # creates this directory, because a bind mount's source must + # exist and nothing on the host has made it in this mode. + return MagicMock(returncode=0, stdout=f"{tmp_path / 'uv-cache'}\n", stderr="") if argv[:3] == ["git", "diff", "--cached"]: return MagicMock(returncode=1) # something staged: commits proceed if argv[:2] == ["git", "check-attr"]: @@ -395,20 +398,116 @@ def test_lc_build_on_a_direct_project_says_so(tmp_path: Path, fake: list[list[st container.build(plain) +def _module_table(tmp_path: Path, hpc: list[list[str]], monkeypatch: pytest.MonkeyPatch, + gates: dict[str, str]) -> None: + """Stand in for a site's podman-hpc module table: one yaml per module, + each declaring its own gate, and `infohpc` pointing at the directory.""" + modules = tmp_path / "modules.d" + modules.mkdir(exist_ok=True) + for name, gate in gates.items(): + (modules / f"{name}.yaml").write_text(f"name: {name}\nenv: {gate}\nbind:\n - /x:/x\n") + inner = project._run + + def run(argv: list[str], *, cwd: Path) -> MagicMock: + if argv[:2] == ["podman-hpc", "infohpc"]: + hpc.append(list(argv)) + return MagicMock( + returncode=0, stdout=f"modules_dir (file: modules_dir): {modules}\n", stderr="" + ) + return inner(argv, cwd=cwd) + + monkeypatch.setattr(project, "_run", run) + + +def test_the_backend_names_the_site_modules_the_runtime_will_apply( + root: Path, tmp_path: Path, hpc: list[list[str]], monkeypatch: pytest.MonkeyPatch +) -> None: + """ENABLE_* is left working — it is how a GPU or MPI recipe reaches + its hardware — so the honesty has to come from naming it. MOUNT_* is + absent by construction: child_env scrubs it before the runtime sees + it, which is why only one family reaches the attestation.""" + monkeypatch.setenv("ENABLE_GPU", "1") + monkeypatch.setenv("ENABLE_CVMFS", "1") + monkeypatch.setenv("ENABLE_MPICH_SS", "") + monkeypatch.setenv("MOUNT_HOME", "1") + _module_table( + tmp_path, hpc, monkeypatch, + {"gpu": "ENABLE_GPU", "cvmfs": "ENABLE_CVMFS", + "mpich": "ENABLE_MPICH_SS", "home": "MOUNT_HOME"}, + ) + _write_archive(image.archive_path(root, image.tag(root))) + runtime = container.runtime_for_run(root, build=False) + + attested = container.backend(runtime).attest( + sandbox.exec_policy(root, containerized=True) + ) + + assert attested.site_modules == ("ENABLE_CVMFS", "ENABLE_GPU") + assert "MOUNT_HOME" not in attested.site_modules, "scrubbed, so never applied" + assert "ENABLE_MPICH_SS" not in attested.site_modules, "set but empty: not enabled" + + +def test_only_the_table_s_own_gates_are_named( + root: Path, tmp_path: Path, hpc: list[list[str]], monkeypatch: pytest.MonkeyPatch +) -> None: + """The gates are read from the site's module table, never matched by + prefix: GitHub's own ENABLE_RUNNER_TRACING was enough to make a + manifest name a module the host does not have.""" + monkeypatch.setenv("ENABLE_RUNNER_TRACING", "true") + monkeypatch.setenv("ENABLE_GPU", "1") + _module_table(tmp_path, hpc, monkeypatch, {"gpu": "ENABLE_GPU"}) + + assert container.site_modules("podman-hpc", root) == ("ENABLE_GPU",) + + +def test_an_unreadable_module_table_names_nothing_rather_than_refusing( + root: Path, hpc: list[list[str]], monkeypatch: pytest.MonkeyPatch +) -> None: + """Attestation must never fail a run: a probe that cannot answer + reports nothing, the same discipline as uv_version.""" + monkeypatch.setenv("ENABLE_GPU", "1") + inner = project._run + monkeypatch.setattr( + project, + "_run", + lambda argv, *, cwd: MagicMock(returncode=1, stdout="", stderr="boom") + if argv[:2] == ["podman-hpc", "infohpc"] + else inner(argv, cwd=cwd), + ) + + assert container.site_modules("podman-hpc", root) == () + + +def test_a_runtime_without_a_module_system_names_none( + root: Path, fake: list[list[str]], monkeypatch: pytest.MonkeyPatch +) -> None: + """podman and docker read no site module table, so an ENABLE_* in the + environment is inert there and must not imply otherwise.""" + monkeypatch.setenv("ENABLE_GPU", "1") + + assert container.site_modules("podman", root) == () + assert container.site_modules("docker", root) == () + + # ---- the containerized converge --------------------------------------------- def test_sync_runs_uv_inside_the_image_with_the_host_cache( - root: Path, fake: list[list[str]] + root: Path, fake: list[list[str]], tmp_path: Path ) -> None: runtime = container.runtime_for_run(root, build=True) fake.clear() container.sync(root, runtime) + cache = tmp_path / "uv-cache" (sync,) = _argvs(fake, "podman", "run") assert f"{root}:{root}:rw" in " ".join(sync) - assert "/home/user/.cache/uv:/home/user/.cache/uv:rw" in " ".join(sync) + assert f"{cache}:{cache}:rw" in " ".join(sync) + assert cache.is_dir(), ( + "a bind mount's source must exist, and in containerized mode uv " + "never runs on the host to create it" + ) assert f"UV_PROJECT_ENVIRONMENT={root / '.lightcone' / 'venv'}" in " ".join(sync) assert "--userns=keep-id" in sync assert runtime.image_id in sync @@ -556,6 +655,164 @@ def test_migrate_runs_even_when_the_store_already_holds_the_image( assert len(_argvs(hpc, "podman-hpc", "migrate")) == 2 +def _squash_probe( + monkeypatch: pytest.MonkeyPatch, hpc: list[list[str]], answer: MagicMock +) -> None: + """Make the store-listing probe answer *answer*; everything else keeps + the `hpc` stub's behavior — the fixture's one-fact-at-a-time rule.""" + inner = project._run + + def run(argv: list[str], *, cwd: Path) -> MagicMock: + if argv[:2] == ["podman-hpc", "images"]: + hpc.append(list(argv)) + return answer + return inner(argv, cwd=cwd) + + monkeypatch.setattr(project, "_run", run) + + +def test_a_stale_squashed_image_is_removed_before_migrate( + root: Path, hpc: list[list[str]], monkeypatch: pytest.MonkeyPatch +) -> None: + """The tag is deterministic but builds are not bit-reproducible, so a + rebuild migrated under an unchanged tag would put a second same-named + image into the read-only squash store — which then refuses every + storage operation. The heal removes the stale copy first.""" + expected = _write_archive(image.archive_path(root, image.tag(root))) + stale = "f" * 64 + _squash_probe( + monkeypatch, + hpc, + MagicMock(returncode=0, stdout=f"{expected} false\n{stale} true\n", stderr=""), + ) + + container.runtime_for_run(root, build=False) + + assert _argvs(hpc, "podman-hpc", "rmsqi") == [["podman-hpc", "rmsqi", stale]], ( + "removal is by id: `rmsqi ` resolves one record and could take " + "the current image, leaving the stale one to re-wedge the store" + ) + order = [c[1] for c in _argvs(hpc, "podman-hpc") if c[1] in ("images", "rmsqi", "migrate")] + assert order == ["images", "rmsqi", "migrate"] + + +def test_every_stale_squashed_image_is_removed( + root: Path, hpc: list[list[str]], monkeypatch: pytest.MonkeyPatch +) -> None: + """`rmsqi` takes one record per call, so a store holding two stale + rows needs two — one call would leave the second to re-wedge it.""" + expected = _write_archive(image.archive_path(root, image.tag(root))) + older, newer = "a" * 64, "b" * 64 + _squash_probe( + monkeypatch, + hpc, + MagicMock( + returncode=0, + stdout=f"{expected} false\n{older} true\n{newer} true\n", + stderr="", + ), + ) + + container.runtime_for_run(root, build=False) + + assert _argvs(hpc, "podman-hpc", "rmsqi") == [ + ["podman-hpc", "rmsqi", older], + ["podman-hpc", "rmsqi", newer], + ] + + +def test_a_current_squashed_image_is_left_alone( + root: Path, hpc: list[list[str]], monkeypatch: pytest.MonkeyPatch +) -> None: + """The healthy pair — the overlay copy and its own squash — is exactly + what a migrate leaves behind; removing it would re-squash every run.""" + expected = _write_archive(image.archive_path(root, image.tag(root))) + _squash_probe( + monkeypatch, + hpc, + MagicMock(returncode=0, stdout=f"{expected} false\n{expected} true\n", stderr=""), + ) + + container.runtime_for_run(root, build=False) + + assert _argvs(hpc, "podman-hpc", "rmsqi") == [] + assert len(_argvs(hpc, "podman-hpc", "migrate")) == 1 + + +def test_a_failing_probe_never_blocks_a_healthy_run( + root: Path, hpc: list[list[str]], monkeypatch: pytest.MonkeyPatch +) -> None: + """A heal must not introduce a failure mode of its own: an + unrecognized probe failure is left for migrate's own loud path.""" + _write_archive(image.archive_path(root, image.tag(root))) + _squash_probe( + monkeypatch, hpc, MagicMock(returncode=125, stdout="", stderr="some other complaint") + ) + + container.runtime_for_run(root, build=False) + + assert _argvs(hpc, "podman-hpc", "rmsqi") == [] + assert len(_argvs(hpc, "podman-hpc", "migrate")) == 1 + + +def test_an_already_wedged_store_is_healed( + root: Path, hpc: list[list[str]], monkeypatch: pytest.MonkeyPatch +) -> None: + """A store wedged before the heal existed fails the listing too, with + the same message every operation gets — that signature is the one + probe failure the heal acts on, and it re-probes so the blind + removal stops as soon as the store can be read again.""" + expected = _write_archive(image.archive_path(root, image.tag(root))) + wedged = MagicMock( + returncode=125, + stdout="", + stderr="Error: configure storage: read-only image store assigns " + "the same name to multiple images", + ) + healthy = MagicMock(returncode=0, stdout=f"{expected} false\n{expected} true\n", stderr="") + answers = [wedged, healthy] + inner = project._run + + def run(argv: list[str], *, cwd: Path) -> MagicMock: + if argv[:2] == ["podman-hpc", "images"]: + hpc.append(list(argv)) + return answers.pop(0) if answers else healthy + return inner(argv, cwd=cwd) + + monkeypatch.setattr(project, "_run", run) + + container.runtime_for_run(root, build=False) + + assert _argvs(hpc, "podman-hpc", "rmsqi") == [["podman-hpc", "rmsqi", image.tag(root)]], ( + "a store that cannot be listed cannot be enumerated, so this one " + "branch has to spell the tag — once, because the re-probe then reads" + ) + assert len(_argvs(hpc, "podman-hpc", "images")) == 2 + + +def test_a_store_that_never_recovers_stops_rather_than_looping( + root: Path, hpc: list[list[str]], monkeypatch: pytest.MonkeyPatch +) -> None: + """Each blind pass is a real deletion, so the heal is bounded: a + store still unlistable after that is not one more `rmsqi` away from + working, and migrate's own refusal is the honest end.""" + _write_archive(image.archive_path(root, image.tag(root))) + _squash_probe( + monkeypatch, + hpc, + MagicMock( + returncode=125, + stdout="", + stderr="Error: configure storage: read-only image store assigns " + "the same name to multiple images", + ), + ) + + container.runtime_for_run(root, build=False) + + assert len(_argvs(hpc, "podman-hpc", "rmsqi")) == container._HEAL_ATTEMPTS + + def test_podman_hpc_builds_saves_and_migrates(root: Path, hpc: list[list[str]]) -> None: """The wrapper is build-capable — NERSC login nodes are where the matching-arch archive comes from — and a fresh build still migrates.""" diff --git a/tests/test_materialize.py b/tests/test_materialize.py index 5ff96e8..121e723 100644 --- a/tests/test_materialize.py +++ b/tests/test_materialize.py @@ -974,6 +974,36 @@ def processes() -> Iterator[engine._Dask]: assert not dataset.status(root) +def test_the_scheduler_adapter_releases_each_future_as_it_yields( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """A future still held at teardown makes the scheduler preserve its + key, so retiring workers replicate every key onto each other — all + retiring — stall past the reap grace, and get killed: every clean + SLURM run ends in "srun: forcing job termination". Releasing as each + result lands is what lets the wind-down be silent.""" + import distributed + + class Fut: + released = False + + def release(self) -> None: + self.released = True + + futs = [Fut(), Fut()] + + def fake_as_completed(handles: list[Fut], with_results: bool) -> Iterator[tuple[Fut, int]]: + return iter((f, i) for i, f in enumerate(handles)) + + monkeypatch.setattr(distributed, "as_completed", fake_as_completed) + + seen = [] + for result in engine._Dask(client=None).completed(list(futs)): + assert futs[result].released, "released before its result is handed over" + seen.append(result) + assert seen == [0, 1] + + # ---- the report ------------------------------------------------------------ diff --git a/tests/test_project.py b/tests/test_project.py index 1c6d91f..b6b4ba8 100644 --- a/tests/test_project.py +++ b/tests/test_project.py @@ -771,7 +771,7 @@ def test_ambient_uv_install_settings_are_scrubbed( come from and how fast, never what gets installed.""" import os - from lightcone.engine.project import child_env, uv_scrub_warning + from lightcone.engine.project import child_env, scrub_warning for name in [k for k in os.environ if k.startswith("UV_")]: monkeypatch.delenv(name) # the suite itself may run under `uv run` @@ -797,7 +797,7 @@ def test_ambient_uv_install_settings_are_scrubbed( ) assert env["UV_LINK_MODE"] == "copy", "link-mode is not an audited setting either" assert env["LC_TEST_CANARY"] == "kept" - assert "UV_INDEX_URL, UV_NO_BINARY, UV_PYTHON" in uv_scrub_warning(), ( + assert "UV_INDEX_URL, UV_NO_BINARY, UV_PYTHON" in scrub_warning(), ( "the warning names exactly what the scrub dropped" ) @@ -819,10 +819,84 @@ def test_an_empty_scrubbed_variable_is_not_reported( monkeypatch: pytest.MonkeyPatch, ) -> None: """An empty variable steers nothing, so warning about it is noise.""" - from lightcone.engine.project import uv_scrub_warning + from lightcone.engine.project import scrub_warning monkeypatch.setenv("UV_NO_BUILD", "") - assert "UV_NO_BUILD" not in uv_scrub_warning() + monkeypatch.setenv("MOUNT_X", "") + warning = scrub_warning() + assert "UV_NO_BUILD" not in warning + assert "MOUNT_X" not in warning + + +def test_ambient_site_mount_gates_are_scrubbed( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """A site container module's MOUNT_* gate would bind-mount undeclared + host directories into every recipe container while the manifest + attests `fs: declared`. ENABLE_* survives: those modules bind CUDA + and MPI system libraries — the /dev,/sys generosity class, not a + channel undeclared inputs arrive through.""" + from lightcone.engine.project import child_env, scrub_warning + + monkeypatch.setenv("MOUNT_CFS", "1") + monkeypatch.setenv("ENABLE_GPU", "1") + + env = child_env() + assert "MOUNT_CFS" not in env + assert env["ENABLE_GPU"] == "1", "the GPU module gate is deliberately kept" + assert "MOUNT_CFS" in scrub_warning(), "the warning names the dropped gate" + + +def test_a_known_site_supplies_its_uv_plumbing_when_unset( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """The center's home filesystem is unusable from compute nodes — it + cannot take uv's cache lock at all, and an interpreter there starts + slowly enough to blow uv's own timeout — so both land on scratch, + deterministically in the ambient environment, which is what lets + every node of an allocation derive the same answer.""" + from lightcone.engine.project import child_env + + monkeypatch.setenv("NERSC_HOST", "perlmutter") + monkeypatch.setenv("SCRATCH", "/pscratch/sd/u/user") + monkeypatch.delenv("UV_CACHE_DIR", raising=False) + monkeypatch.delenv("UV_PYTHON_INSTALL_DIR", raising=False) + + env = child_env() + assert env["UV_CACHE_DIR"] == "/pscratch/sd/u/user/.lightcone/uv-cache" + assert env["UV_PYTHON_INSTALL_DIR"] == "/pscratch/sd/u/user/.lightcone/uv-python" + + +def test_ambient_uv_plumbing_wins_over_the_site_supply( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """These locations are the user's plumbing to choose; the site + default fills silence, never overrides a voice — and per variable, + so setting one does not suppress the other.""" + from lightcone.engine.project import child_env + + monkeypatch.setenv("NERSC_HOST", "perlmutter") + monkeypatch.setenv("SCRATCH", "/pscratch/sd/u/user") + monkeypatch.setenv("UV_CACHE_DIR", "/pscratch/sd/u/user/custom") + monkeypatch.delenv("UV_PYTHON_INSTALL_DIR", raising=False) + + env = child_env() + assert env["UV_CACHE_DIR"] == "/pscratch/sd/u/user/custom" + assert env["UV_PYTHON_INSTALL_DIR"] == "/pscratch/sd/u/user/.lightcone/uv-python" + + +def test_no_site_means_no_supply(monkeypatch: pytest.MonkeyPatch) -> None: + """Off a known center — or on one whose scratch variable is absent — + uv's own defaults stand.""" + from lightcone.engine.project import child_env + + monkeypatch.setenv("SCRATCH", "/pscratch/sd/u/user") # scratch alone: no + monkeypatch.delenv("UV_CACHE_DIR", raising=False) + assert "UV_CACHE_DIR" not in child_env() + + monkeypatch.setenv("NERSC_HOST", "perlmutter") # marker without scratch: no + monkeypatch.delenv("SCRATCH") + assert "UV_CACHE_DIR" not in child_env() def test_relays_uv_warnings_into_the_report( diff --git a/tests/test_sandbox_oci.py b/tests/test_sandbox_oci.py index cf8cdcc..2842568 100644 --- a/tests/test_sandbox_oci.py +++ b/tests/test_sandbox_oci.py @@ -215,6 +215,27 @@ def test_the_attestation_is_derived_from_the_flags(root: Path, policy: Policy) - assert attested.fs == "declared" assert attested.network == "allowed" assert attested.landlock_abi is None + assert attested.site_modules == () + + +def test_site_modules_are_named_in_the_attestation(root: Path, policy: Policy) -> None: + """A site module widens the container past the mount table — NERSC's + ENABLE_CVMFS binds the whole /cvmfs hierarchy, ENABLE_MPICH_SS adds + --privileged and the host namespaces — so `fs: declared` alone would + overstate what bounded the run. They stay working and get named.""" + from lightcone.engine.sandbox.oci import OCIBackend + + backend = OCIBackend( + runtime="podman-hpc", + image_id="a" * 64, + root=root, + site_modules=("ENABLE_CVMFS", "ENABLE_GPU"), + ) + + attested = backend.attest(policy) + + assert attested.site_modules == ("ENABLE_CVMFS", "ENABLE_GPU") + assert attested.fs == "declared", "the mount table is still what lc composed" # ---- the seam's composition -------------------------------------------------