diff --git a/docs/api/assets.md b/docs/api/assets.md index 39815f8a..77dce6ef 100644 --- a/docs/api/assets.md +++ b/docs/api/assets.md @@ -1,5 +1,52 @@ -# lightcone.engine.assets (removed) +# lightcone.engine.assets -This module was the Dagster asset factory. It no longer exists. The Snakemake -generator that replaced it lives at -[engine/snakefile](snakefile.md). +One output: its directory, its manifest, and whether it is still +current. The classification rule lives here, next to the manifest it +reads and the hashes it compares — and it is the one place in the +engine where a bug is quiet rather than loud, which is why it may not +have two implementations. + +Source: `src/lightcone/engine/assets.py`. + +## Key symbols + +| Symbol | Role | +|---|---| +| `classify(...)` | The one rule: `current` / `behind` / `stale`, with the why. Two callers — the worker and the read-only walk. | +| `Verdict.calls_for_a_remake(refresh=)` | The one place a state becomes an action: `stale` always, `behind` only when asked. | +| `data_version(path)` | Content hash of a directory or file — computed in the worker, before anything is annexed. | +| `Versions` | Per-run memo so a shared declared input hashes once, not once per dependent. | +| `read(dir)` / `write(...)` | The manifest, `.lightcone-manifest.json`. | +| `output_dir(root, u, o)` | The path, guarded: an id that is not one path component is refused — this guard is what lets the worker's reset stay a whole-directory delete. | +| `ContentNotFetchedError` | An annexed file whose content is not in this clone, in either shape it takes. | + +## What must stay true + +- **One `classify`, two callers, one differing value.** The worker + hands live input digests; check mode hands `None` for anything + upstream that will run ("this is going to change"). That value is + the entire difference — never a second body of logic. History (the + foreign-write fact) enters the same way: computed by whoever has + git, handed in as a value. +- **The comparison is fourfold**: `definition_version`, the declared + input *set* (separate on purpose — a dropped dependency moves + neither hash), each recorded input digest, then `env_version`. + `stale` wins over `behind`; `behind` does not propagate and a behind + upstream still feeds its dependents. +- **A skip returns the *recorded* digest, never a recomputed one** — + on a bytes-free clone, rehashing dangling symlinks would quietly + report a different output. +- **Unfetched content refuses loudly, in both shapes.** A pointer file + hashes to a well-formed digest of the wrong thing; a dangling + symlink drops out of an `is_file()` walk without a word. Both raise + `ContentNotFetchedError` naming `git annex get`; only dangling + symlinks are added back to the directory walk. +- **`calls_for_a_remake` has three callers** (worker, check, the + cascade walk) and no inline re-spellings — the third copy is where + they start to disagree. + +## Tests + +`tests/test_assets.py` — pure; nothing on disk beyond `tmp_path`. +The pointer-file and dangling-symlink traps are pinned against real +annex shapes in `tests/test_dataset.py`. diff --git a/docs/api/cli.md b/docs/api/cli.md deleted file mode 100644 index df679953..00000000 --- a/docs/api/cli.md +++ /dev/null @@ -1,88 +0,0 @@ -# lightcone.cli.commands - -The Click surface. Defined in `src/lightcone/cli/commands.py`. Six -public commands: `init`, `run`, `status`, `verify`, `build`, `export`. - -The user-facing reference is in [CLI Overview](../cli/index.md). This -page is a tour of the module internals. - -## Entry point - -```python -@click.group() -@click.version_option(package_name="lightcone-cli") -@click.pass_context -def main(ctx: click.Context) -> None: - ctx.ensure_object(dict) - _ensure_global_config() # auto-create ~/.lightcone/config.yaml with defaults -``` - -`main` is exposed as `lightcone.cli.main` (re-exported from -`lightcone.cli.__init__`) and is the entry point declared in -`pyproject.toml::project.scripts`: - -```toml -[project.scripts] -lc = "lightcone.cli:main" -``` - -## Helpers - -### `_config_path() → Path` - -Returns `~/.lightcone/config.yaml`. Used by `_ensure_global_config()`, -which the `main` group calls to create the file with defaults -(`container: {runtime: auto}`) on first invocation. - -### `_project_root(start: Path | None = None) → Path` - -Walks up from `start` (or `cwd`) looking for `astra.yaml`. Raises -`click.ClickException` if none found. Used by `run`, `status`, `verify`, -`build`. - -### `_target_for(project: Path, output_id: str, universe: str) → str` - -Translate an `output_id` (or qualified `.`) into -the Snakemake target path that materializes it — specifically the -manifest file `results///.lightcone-manifest.json`. -Raises `click.ClickException` if the id is unknown or ambiguous. - -### `_run_snakemake(cmd, *, env, scratch_root, verbose)` - -Spawn `snakemake` and forward the run's narrative output: lines the -executor plugin prefixes with the sentinel -(`lightcone.engine.runner.SENTINEL`) stream to the terminal with the -prefix stripped; everything else (DAG chatter, job stats) is dropped -unless `verbose`. stderr is tailed into a bounded ring buffer and, on -failure, dumped to `snakemake-stderr-.log` under the scratch -root. Returns the exit code. - -### `_status_label(s: str) → str` - -Map a status literal to the Rich-formatted display label: - -| Status | Display | -|--------|---------| -| `ok` | `[green]✓ ok[/green]` | -| `stale` | `[yellow]✸ stale[/yellow]` | -| `missing` | `[red]✗ miss[/red]` | -| `alias` | `[dim]→ alias[/dim]` | - -## Boilerplate text - -`_CONTAINERFILE_TEMPLATE`, `_REQUIREMENTS`, `_GITIGNORE_BASE`, -`_GITIGNORE_APPEND`, `_MYST_YML`, and `_INDEX_MD_BODY` are multi-line -strings written at `lc init` time (the spec boilerplate itself comes -from astra's boilerplate helper). Edit them to change what new -projects look like. - -`init` is a convergence loop, not a one-shot scaffolder: each managed -item is created if missing, offered to an optional -`repair(text) -> str | None` hook otherwise (today only -`_repair_gitignore`, which appends the managed block once), and left -alone when the hook returns `None`. `--check` computes the same report -without writing (exit 1 when not converged); `--json` prints it as -`{converged, created, repaired, unchanged, warnings}`. Warnings carry -problems init can see but must not fix (e.g. a directory `COPY` in the -Containerfile, detected via -`lightcone.engine.container.directory_copy_sources`). diff --git a/docs/api/cloudbuild.md b/docs/api/cloudbuild.md deleted file mode 100644 index 17f0f6a3..00000000 --- a/docs/api/cloudbuild.md +++ /dev/null @@ -1,63 +0,0 @@ -# lightcone.engine.cloudbuild - -Remote image builds through GCP Cloud Build — the build backend for -deployments with no OCI runtime on the host (a JupyterHub user pod on -GKE). Pure `urllib` REST against the metadata server, GCS, the Cloud -Build API, and the Docker Registry v2 API; no SDK dependency, no -stored credentials, no git remote required. - -Source: `src/lightcone/engine/cloudbuild.py`. - -## Deployment contract - -Env vars injected into every user pod by the deployment (see the -hub-deploy `lightcone` hub config): - -| Env var | Meaning | -|---------|---------| -| `LIGHTCONE_REGISTRY` | Artifact Registry prefix (`-docker.pkg.dev//`); also names the GCP project builds run in. Declared in `engine.container` (`REGISTRY_ENV`). | -| `LIGHTCONE_BUILD_BUCKET` | GCS bucket for build sources and logs. Its presence (with the registry) selects this backend — `cloudbuild_available()`. | -| `LIGHTCONE_BUILD_SERVICE_ACCOUNT` | Optional dedicated build SA; the deployment grants it registry-writer rights only. | - -Auth is the pod's Workload Identity, spoken to the GCE metadata server -(`_metadata_access_token()`). The pod's identity needs -`cloudbuild.builds.editor`, `iam.serviceAccountUser` on the build SA, -object create/view on the bucket, and `artifactregistry.reader` for -the freshness probe. - -## `ensure_image(project, containerfile_spec, *, project_name, force=False, on_progress=None) → str` - -Make sure the project's image is in the registry; return its ref. -Content-addressed and git-free: - -1. Compute the ref `$LIGHTCONE_REGISTRY/lc-:` — the - same `image_identity()` digest as the local `lc--` - tag, spelled for a registry. -2. `registry_image_exists(ref)` — one HEAD on the Docker Registry v2 - manifest endpoint. Present → done (no build, no upload). `force` - skips this probe. -3. Tar the **staged build context** (`_populate_build_context` — the - exact file set the tag hashes) and upload it to the bucket under a - content-addressed object name. -4. Submit the build (docker builder step, image push, logs to - `gs:///logs` with `GCS_ONLY` — required for custom SAs and - the source of the failure tail), poll to a terminal status. -5. Non-`SUCCESS` → `CloudBuildError` carrying the build-log tail. - -`on_progress(phase, detail)` phases: `cached`, `staging`, then Cloud -Build statuses lowercased (`queued`, `working`, `success`, …). - -## `registry_image_exists(ref) → bool | None` - -`None` — not `False` — when unknowable (no metadata credentials, -registry unreachable), so callers can distinguish "absent, build it" -from "can't tell". Artifact Registry accepts the OAuth2 access token -directly as a Bearer on `/v2/` endpoints. - -## Tests - -`tests/test_cloudbuild.py` mocks the two HTTP seams -(`_metadata_access_token`, `_request`) and exercises the real control -flow: backend selection, freshness probe, staging, submission -(including the custom-SA payload), polling, failure-tail reporting, -and the staged-tarball ↔ hashed-context equivalence. diff --git a/docs/api/container.md b/docs/api/container.md index dad10864..79d99d46 100644 --- a/docs/api/container.md +++ b/docs/api/container.md @@ -1,192 +1,73 @@ -# lightcone.engine.container - -The container layer. Two surfaces: build-time (`compute_image_tag`, -`build_image`, `pull_image`) and run-time wrap (`wrap_recipe`, -`make_image_tag_resolver`). - -Source: `src/lightcone/engine/container.py`. - -## Constants - -| Constant | Value | -|----------|-------| -| `RUNTIMES` | `("podman", "docker", "podman-hpc")` — detection priority order | -| `DEPENDENCY_FILES` | `("requirements.txt", "requirements-dev.txt", "requirements-test.txt", "pyproject.toml", "setup.py", "setup.cfg", "poetry.lock", "Pipfile.lock")` | - -Detection priority is podman before docker for two reasons: it's -rootless (less surprising on shared machines), and the docker probe -includes `docker info` so a stopped daemon doesn't silently win over a -healthy podman. - -## Runtime detection - -### `detect_runtime() → str | None` - -Returns the first usable runtime in `RUNTIMES`. "Usable" means the -binary is on PATH and (for docker) `docker info` succeeds. Returns -`None` if nothing's available. - -### `load_runtime(*, project_path=None) → RuntimeChoice` - -Resolve the runtime to use. Reads `container.runtime` from -`~/.lightcone/config.yaml`: - -- `auto` (default) → first available, else `"none"` with `explicit=False`. - On a site declaring `container_runtime: kubernetes` (a Dask Gateway - deployment), auto resolves to `kubernetes` with no PATH probing. -- `docker | podman | podman-hpc` → explicit; binary must exist or - raises `ContainerBuildError`. -- `kubernetes` → explicit; no binary involved (the worker pod is the - container). -- `none` → explicit opt-out. -- Anything else → `ContainerBuildError`. - -`project_path` is accepted for future per-project overrides but is not -consulted today. - -### `RuntimeChoice` (dataclass) - -```python -@dataclass(frozen=True) -class RuntimeChoice: - runtime: str # docker | podman | podman-hpc | none - explicit: bool # True if pinned, False if `auto` produced this -``` - -`explicit=False` + `runtime="none"` means auto fell back silently. Callers -should warn — that case mismatches the manifest's recorded -`container_image` against what actually executed. - -## Image tag computation - -### `compute_image_tag(project_name, containerfile, project_path) → str` - -Returns `lc--`. The hash covers the -Containerfile contents plus every dependency file from `DEPENDENCY_FILES` -that exists at the project root. - -Sanitization: lowercase + spaces → hyphens. - -### `find_dependency_files(project_path) → list[Path]` - -Sorted list of dependency files actually present. Used by -`compute_image_tag`. - -### `hash_file_contents(files) → str` - -Concatenated SHA-256 hex digest of the listed files. Internal helper. - -### `is_containerfile(spec, project_path) → bool` - -True if `spec` resolves to an existing file (i.e. it's a Containerfile, -not a registry image). - -## Build - -### `build_image(tag, containerfile, context, *, runtime, build_args=None) → ContainerBuildResult` - -Run ` build -t -f [--build-arg …] `. -For `podman-hpc`, also runs `podman-hpc migrate ` so compute nodes -can read the image. Raises `ContainerBuildError` on any failure. - -### `pull_image(image, *, runtime) → None` - -Run ` pull `, then (for podman-hpc) `migrate`. Used by -`lc build` to pre-stage registry images so `lc run` can pass -`--pull=never`. - -### `image_exists_locally(tag, *, runtime) → bool` - -Check the local image store. Routes to `image_exists_podman_hpc(tag)` -for `podman-hpc`, otherwise runs ` image inspect `. - -### `_podman_hpc_migrate(tag)` (private) - -Wraps `podman-hpc migrate`. Raises `ContainerBuildError` on failure. - -## Run-time wrap - -### `wrap_recipe(recipe, *, image, runtime) → str` - -Wrap `recipe` so it executes inside `image` under `runtime`. Returns a -shell-command string for Snakemake's `shell()`. - -No-op cases (`recipe` returned unchanged): - -- `image is None` -- `runtime == "none"` -- `runtime == "kubernetes"` — the Dask worker pod executing the recipe - was started from `image`; wrapping would containerize twice. The - image still flows into `code_version` and the manifest. - -Otherwise produces: - -```bash - run --rm --pull=never \ - -v "$PWD":"$PWD" -w "$PWD" \ - bash -c '' -``` - -`--pull=never` is critical: it sidesteps podman's -`unqualified-search-registries` resolution, which fails for our -content-addressed `lc--` tags. The cost: registry images -have to be pre-pulled by `lc build`. - -The bind mount and `-w "$PWD"` ensure recipes that write to relative -paths land in the project tree. Snakemake invokes us with `cwd=project`, -so `$PWD` is the project root. - -Snakemake placeholders inside `recipe` (`{output[0]}`, `{input.X}`, -`{wildcards.universe}`) are preserved — they substitute through Python's -`str.format` at execution time, after wrapping. - -### `make_image_tag_resolver(project_path, project_name) → Callable` - -Returns a memoizing wrapper around `resolve_image_for_run`. Multiple -outputs typically share a Containerfile; resolving re-hashes the file -plus all dependency files (lockfiles can be megabytes), so caching by -spec string for the lifetime of the caller's loop matters. - -### `resolve_image_for_run(spec, *, project_path, project_name, registry=None) → str | None` - -Translate an `astra.yaml` `container:` value into the image tag the -runtime will execute: - -- `None` / empty → `None` -- Containerfile path → `lc--` (the tag `lc build` would - produce), or `/lc-:` when `registry` is given - (a deployment with a remote builder — same content-addressed - identity, spelled for a registry) -- Anything else → returned as-is - -## Status - -### `get_container_status(spec, project_path, project_name, *, runtime) → ContainerStatus` - -Without building or pulling, return a `ContainerStatus` describing what -would happen. - -### `ContainerStatus` (dataclass) - -```python -@dataclass -class ContainerStatus: - type: str # "none" | "prebuilt" | "build" - image: str | None = None # the tag (always set for "prebuilt"/"build") - exists: bool | None = None # local-store presence (None for "none" runtime) - containerfile: str | None = None # the spec, only set for "build" -``` - -## Exceptions - -### `ContainerBuildError` - -Raised by `build_image`, `pull_image`, `_podman_hpc_migrate`, and -`load_runtime` (configuration errors). Message carries the failing -runtime and stderr. +# lightcone.engine.image & container + +The container hatch, split down the pure/impure line. `image.py` is +what a containerized project *declares* and how that becomes an +identity — pure, no subprocess anywhere. `container.py` is building, +storing and entering images — impure, every command through +`project._run`. The exec side (the mount table) lives with the other +backends in `sandbox/oci.py`. + +Sources: `src/lightcone/engine/image.py`, +`src/lightcone/engine/container.py`, `src/lightcone/engine/sandbox/oci.py`. + +## Key symbols + +| Symbol | Role | +|---|---| +| `image.declaration(root)` | The `[tool.lightcone.image]` table, validated — a closed key set (`base`, `apt-install`, `run-commands`, `env`), because every key is hashed. | +| `image.tag(root)` | `lc-env-<16 hex>` over the rendered Containerfile *and* the identity document. | +| `image.archive_path(root, tag)` | `.datalad/environments//image` — the `datalad containers-add` layout. | +| `container.build(root)` | Build + save + commit, idempotent; returns `(Runtime, "built" \| "present")`. | +| `container.runtime_for_run(root, *, build)` | One function, two strictnesses: `lc build`/materialize-preflight may build and commit; the probe and worker only ever find, fetch, and load. | +| `container.backend(...)` | The single construction point for the exec backend — the only mode branch. | +| `container.sync(...)` | The in-container environment converge: network on, project `:rw`, host uv cache mounted, into `.lightcone/venv`. | +| `Runtime` | Facts only — root/mode/name/tag/id/arch — never mechanism. | + +## What must stay true + +- **The user never sees a Containerfile.** The render exists only in a + transient build context; the image's `LABEL` carries the identity + document so the archive stays self-describing. There is deliberately + no `pip-install` key — the Python environment is the lock's + business, never the image's. +- **The engine never enters the image.** The container is the + *recipe's* world: driver, git, annex, and classification stay the + host's `lc`; exactly two things run in-image (the sync and each + exec, `--network none`). +- **No project file enters the build context** — that is what makes + "code edits never rebuild" structural rather than incidental. +- **The dataset is the store; runtime stores are caches.** Execution + pins the archive's config-blob **id** (readable with no runtime), + never a tag; a dropped archive never substitutes — a rebuild is a + new archive under a new id. +- **Builds and archive commits happen only on a clean tree, and only + after the graph resolves** — a refusal over a typo must not cost a + minutes-long build, and `dataset.save` commits the whole index. +- **The mount table is the mechanism** (`sandbox/oci.py`): project + `:ro`, `results/` `:rw`, declared inputs `:ro`, private HOME, + `--tmpfs /tmp`, over a `--read-only` rootfs — without that flag a + stray write *succeeds* into the ephemeral layer and vanishes while + the attestation claims `fs: declared`. Mounts are resolved source, + **declared** destination — the one policy shape that keeps its paths + unresolved, because they are addresses the recipe uses. +- **Runtime differences are spellings, never shapes.** One + `OCIBackend`, data-parameterized; the podman family is stated once + (`_PODMAN_FAMILY`) and asked positively, so a new runtime falls + outside it by default. podman-hpc adds exactly one step (`migrate`, + outside the load branch) and joins `_SHARED_STORE_RUNTIMES`. + Detection order podman-hpc → podman → docker; docker's daemon is + probed at detection. +- **The architecture gate refuses before the load** — a wrong-arch + `load` succeeds and then dies as `exec format error` deep inside a + recipe. Ignorance passes; a recorded mismatch refuses, naming the + fix. ## Tests -`tests/test_container.py` covers detection, image tag computation, -build invocation, recipe wrapping, and the `RuntimeChoice` resolution -matrix. +`tests/test_image.py` (pure: structure and ordering, tag sensitivity +both ways, the `env_version` frame), `tests/test_container.py` +(lifecycle against the stubbed `_run` — every refusal on recorded +argv), `tests/test_sandbox_oci.py` (the mount table, pure), and +`tests/test_container_smoke.py` — the runtime's answer, gated by +`LC_CONTAINER_TESTS_REQUIRED=1` in CI, building a real image and +proving the record on a bytes-free clone with a real `datalad rerun`. diff --git a/docs/api/crate.md b/docs/api/crate.md new file mode 100644 index 00000000..af2bf0d3 --- /dev/null +++ b/docs/api/crate.md @@ -0,0 +1,61 @@ +# lightcone.engine.crate + +The publication view: the repository described as a Workflow Run +RO-Crate. The project *is* the crate — `ro-crate-metadata.json` sits at +the root, describes what the repository already holds, and a deposit is +`git archive`, not an export step. lc's manifests stay the canonical +record; the crate is the same facts in schema.org vocabulary for +archives and viewers that will never run `lc`. + +Source: `src/lightcone/engine/crate.py` (converged by +`materialize._converge_crate`). + +## Key symbols + +| Symbol | Role | +|---|---| +| `render(root, graph, *, license, dsid, writer)` | The document, as bytes. A pure function of repository state — git comes in as the `writer` callable, the dataset id as a value. | +| `license_of(root)` | `[project].license` from `pyproject.toml`; empty means no crate is maintained. Presence is publication intent. | +| `CRATE_FILENAME` | `ro-crate-metadata.json`. | + +## What must stay true + +- **The clock never enters the render.** `datePublished` is the newest + manifest `finished_at` (the spec file's last-commit date for a + never-materialized project) and must override rocrate's + construction-time default. Entities build in sorted order, + serialization is `sort_keys` — render-twice-identical is the one + byte-level claim, and it is what makes convergence sound. +- **Maintenance is derived, never configured.** RO-Crate requires a + license; materialize must not refuse to run science over a missing + key, and inventing one asserts terms over someone's data. Absent ⇒ + one report line; removed later ⇒ the file is left, and the line says + it is no longer maintained. +- **Run identity comes free from `git_sha`** — the driver reads HEAD + once per run, so grouping manifests by it *is* grouping by run: one + `OrganizeAction` per materialize, a `ControlAction` per execution, a + `HowToStep` per output id (deduped across universes — a step is spec + structure, an action is one execution). +- **The `Person` is the author of the output's *saving* commit** (via + `writer`), never the manifest's `git_sha` — that is the commit the + run *started* at and can be someone else's. +- **The manifest is not transliterated.** `env_version`, + `definition_version` and `hermeticity` get no invented schema.org + spelling — the manifest itself is in the crate as a `File`, + `subjectOf` its output. Real vocabulary comes from the workflow-run + `@context`, without which `containerImage` and `sha256` are + undefined terms JSON-LD silently drops — the pre-rebuild exporter's + failure mode. +- **The rerun entry point does not regenerate the crate** — it is one + task's executor, so the crate lags until the next materialize. + Recorded residue, not a bug. + +## Tests + +`tests/test_crate.py` — pure: fixture manifests, a hand-built graph, a +stub writer, no git anywhere; structure and ordering assertions plus +the single render-twice byte check. `tests/test_crate_smoke.py` — the +official `rocrate-validator` against Provenance Run Crate 0.5: +REQUIRED clean, RECOMMENDED pinned to the recorded `_FLOOR` set (a new +failure is a regression, a disappearing one is the floor to shrink), +required in CI via `LC_CRATE_TESTS_REQUIRED=1`. diff --git a/docs/api/dask_cluster.md b/docs/api/dask_cluster.md deleted file mode 100644 index be2aebd2..00000000 --- a/docs/api/dask_cluster.md +++ /dev/null @@ -1,104 +0,0 @@ -# lightcone.engine.dask_cluster - -Cluster lifecycle for `lc run`. One context manager (`cluster_for_run`), -four branches, no service to manage. - -Source: `src/lightcone/engine/dask_cluster.py`. - -## `cluster_for_run(*, verbose=False, worker_image=None, max_workers=None) → Iterator[dict[str, str]]` - -Yields the env overlay the child snakemake needs to reach the cluster -(the executor plugin lives in a different process, so connection info -travels via environment variables). Four branches in priority order: - -1. **`DASK_SCHEDULER_ADDRESS` already set** → yield it as-is. We don't - own the cluster, so we don't tear it down. -2. **`DASK_GATEWAY__ADDRESS` set** (a JupyterHub deployment) → - **create** a run-scoped Dask Gateway cluster with `worker_image` as - its `image` cluster option, scale it adaptively `1..max_workers`, - wait (bounded by `LIGHTCONE_GATEWAY_WORKER_TIMEOUT`, default 600 s) - for the first worker, and shut the cluster down on exit — success - or failure. Yields `{LIGHTCONE_GATEWAY_CLUSTER: }`: Gateway - schedulers speak a `gateway://` comm scheme a bare `Client` cannot - dial, so the executor rejoins by name through the Gateway API. -3. **`SLURM_JOB_ID` set** → start an in-process scheduler bound to the - driver's SLURM hostname (`SLURMD_NODENAME` or `gethostname()`), - then `srun` one `dask worker` per node across the allocation. -4. **None of the above** → `LocalCluster()` sized to the local machine. - -Outside the Gateway branch the scheduler is always in-process, so its -lifetime equals the run's lifetime: no orphaned schedulers if the -driver crashes. On the Gateway branch the same contract is enforced -server-side — create per run, cull on exit (the deployment's idle -timeout is the backstop). Create-per-run is also what makes image -updates seamless: a Gateway cluster's image is fixed at creation. - -The Gateway branch self-provisions the worker environment through the -deployment's **standard `environment` cluster option** (no -lightcone-specific injection needed server-side): the -`DASK_DISTRIBUTED__WORKER__RESOURCES__*` scheduling contract mirrored -from the declared `worker_cores`/`worker_memory` option values, the -driver's `HOME`/`USER`/`LOGNAME` (passwd-less uid-1000 images crash -`getpass.getuser()` without them), and `LIGHTCONE_WORKER_IMAGE` as -manifest ground truth. It also fails fast on two silent-hang failure -modes: zero workers within the timeout (unpullable image, -unschedulable pool), and workers that don't advertise the -`cpus`/`memory` resource contract (a deployment that doesn't expose -the `environment` option). - -## Resource keys - -These string constants form a contract with the executor plugin: - -```python -RESOURCE_CPUS = "cpus" -RESOURCE_MEMORY = "memory" -RESOURCE_GPUS = "gpus" -``` - -Workers must advertise every key the executor may request — Dask -matches by exact key presence. The local-cluster path includes all -three even when the executor doesn't ask, so per-rule -`mem_mb`/`gpus_per_task` rules still schedule on a workstation. - -## Node-shape detection - -`_detect_node_shape()` reads SLURM env vars with sane fallbacks: - -| Resource | Env var | Fallback | -|----------|---------|----------| -| CPUs | `SLURM_CPUS_ON_NODE` | `os.cpu_count()` | -| Memory | `SLURM_MEM_PER_NODE` (MB) | `psutil.virtual_memory().total` if installed; otherwise 0 (advisory; workers won't enforce caps) | -| GPUs | `SLURM_GPUS_ON_NODE` | `0` | - -## SLURM-backed cluster details - -```python -srun --ntasks=$SLURM_NNODES --ntasks-per-node=1 \ - dask worker --nthreads $cpus --nworkers 1 \ - --resources "cpus=N memory=B gpus=G" --no-dashboard -``` - -The `--ntasks-per-node=1` is important: we want one worker per node, -not per CPU. The worker uses `--nthreads` to advertise its parallelism -within the node. - -After spawning workers, the manager opens a temporary `Client(addr)` to -`wait_for_workers(n_workers=nnodes, timeout=120)`. If the workers -haven't connected within two minutes, raise. - -On exit, the manager `terminate()`s the worker subprocess group, waits -up to 10s, then `kill()`s anything still alive. - -## Why no `dask-jobqueue`? - -`dask-jobqueue` would `sbatch` workers from inside an existing job — -fine, but adds dependency and indirection. Since we already require the -user to be inside an allocation (`salloc` / `sbatch`), `srun` is enough -and keeps everything in one process tree. - -## Tests - -`tests/test_dask_cluster.py` covers the three branches and the -resource-advertising contract. The SLURM branch is tested with mocked -`subprocess.Popen` plus a stubbed `Client.wait_for_workers`. diff --git a/docs/api/dask_executor.md b/docs/api/dask_executor.md deleted file mode 100644 index 28f2ae5d..00000000 --- a/docs/api/dask_executor.md +++ /dev/null @@ -1,118 +0,0 @@ -# snakemake_executor_plugin_dask - -Snakemake executor plugin that submits each rule as a `client.submit()` -on a `dask.distributed` cluster. Lives at the top of `src/` because -Snakemake discovers executor plugins through the -`snakemake_executor_plugin_*` package-naming convention. - -Source: `src/snakemake_executor_plugin_dask/`. - -## Module shape - -``` -snakemake_executor_plugin_dask/ -├── __init__.py # plugin metadata + Executor re-export -└── executor.py # DaskExecutor class -``` - -## Plugin metadata - -```python -common_settings = CommonSettings( - job_deploy_sources=True, # send Snakefile + sources to workers - non_local_exec=True, # workers may live elsewhere - implies_no_shared_fs=False, # we *do* assume a shared FS -) -``` - -We assume a shared filesystem because all our workers (local threads, -SLURM nodes) see the project tree the same way. If you change this, -you also need to teach `wrap_recipe()` not to use `$PWD` bind mounts. - -## `DaskExecutor` - -Inherits from `snakemake_interface_executor_plugins.executors.remote.RemoteExecutor`. - -### `__init__(workflow, logger)` - -Imports `dask.distributed` lazily; raises `WorkflowError` if missing -("`pip install distributed`"). Connects via `_connect_client()`: if -`LIGHTCONE_GATEWAY_CLUSTER` is set, rejoins that Dask Gateway cluster -through `Gateway().connect(name)` (with `shutdown_on_close=False` — -the executor is a guest; `lc run` owns the cluster lifecycle); -otherwise dials `DASK_SCHEDULER_ADDRESS` with a bare `Client`. Raises -`WorkflowError` if neither is set — `lc run` is responsible for -setting one. - -### `run_job(job)` - -Translate a Snakemake job to a Dask submission: - -```python -client.submit( - _run_shell, job.format_job_exec(), - resources=_build_resources(job), - pure=False, - key=f"snakejob-{job.name}-{job.jobid}", -) -``` - -`_run_shell` runs `subprocess.run(cmd, shell=True, check=False)` on -the worker and returns `(exit_code, output_block)`. The block is the -child snakemake's sentinel-prefixed lines (see -`lightcone.engine.runner.SENTINEL`), kept verbatim; on a failure that -produced no sentinel line at all (the child died before the rule body -— import error, missing package in the worker image) a bounded raw -tail is sentinel-framed instead so bootstrap failures don't vanish -into worker logs. Returning output through the task result is the only -channel that works uniformly across LocalCluster threads, srun-launched -workers, and Gateway worker pods (whose stdout goes to pod logs). The -recipe is already container-wrapped at Snakefile generation time, so -the worker has no runtime logic of its own. - -### `check_active_jobs(active_jobs)` - -Async generator: for each submitted job, check `future.done()`. Yield -back jobs that are still in flight. For finished jobs, unpack the -result via `_unpack_result` (which also accepts the bare-int result of -a worker running an older lightcone-cli release), write the output -block to stdout — `lc run` filters and forwards it — then: - -- `future.exception() is not None` → `report_job_error(...)` -- exit code `!= 0` → `report_job_error(...)` -- otherwise → `report_job_success(...)` - -### `cancel_jobs(active_jobs)` - -Best-effort `future.cancel()` on each in-flight job. **Does not** close -the `Client` — Snakemake calls `cancel_jobs` for partial cancellations -as well as at shutdown, so closing here would break subsequent -submissions. The client is closed in `shutdown()` exclusively. - -### `shutdown()` - -`self._client.close()` then `super().shutdown()`. - -## Resource translation - -```python -def _build_resources(job) -> dict[str, float]: - res = {} - cpus = job.resources.get("cpus_per_task") or job.threads - if cpus: res["cpus"] = float(cpus) - mem_mb = job.resources.get("mem_mb") - if mem_mb: res["memory"] = float(mem_mb) * 1e6 # MB → bytes - gpus = job.resources.get("gpus_per_task") or job.resources.get("gpus") - if gpus: res["gpus"] = float(gpus) - return res -``` - -Returns `None` if the resulting dict is empty (Dask's "no constraints" -sentinel). Resource keys must match those advertised by workers -(`cpus`, `memory`, `gpus` — see [`engine.dask_cluster`](dask_cluster.md)). - -## Tests - -`tests/test_dask_plugin.py` exercises the `_build_resources` mapping -and the executor's lifecycle (init / run / check / shutdown) against a -local `LocalCluster` fixture. diff --git a/docs/api/dataset.md b/docs/api/dataset.md new file mode 100644 index 00000000..89c3dd1e --- /dev/null +++ b/docs/api/dataset.md @@ -0,0 +1,58 @@ +# lightcone.engine.dataset + +The git + git-annex seam: how a project stores what it produced. +Storage follows the DataLad model — git carries the pointers and the +history, git-annex carries the bytes — reached through ordinary `git` +commands. Every command goes through `project._run`, so there is one +monkeypatch point and every invocation is inspectable. + +Source: `src/lightcone/engine/dataset.py` (+ +`templates/files/gitattributes.tmpl` for the routing policy). + +## Key symbols + +| Symbol | Role | +|---|---| +| `save(root, paths, message)` | Stage scoped, commit — with `-c annex.thin=true` and `-c annex.dotfiles=true`, per-add and never written to config. | +| `restore(root, paths)` | `git clean` always; `git checkout HEAD --` only when HEAD has the path. Never `-- .`. | +| `status(root)` | The dirty question, scoped to the project (`-- .`, prefix-stripped) so a project inside a larger repository works. | +| `head(root)` | The commit a run started at — read once per run, by the driver. | +| `last_writer(root, dir)` | Who last touched an output's directory — the foreign-write question. Answers "cannot say" as empty, never an error. | +| `require_committer(root)` | Refuses a repository with no git identity, before any recipe spends time. Asked as `git var`, the question a commit itself asks. | +| `dataset_id(root)` | The DataLad dataset UUID, read via `git config -f`. | + +## What must stay true + +- **Nobody is ever asked to run a git-annex command.** `filter=annex` + plus the `.gitattributes` policy make an ordinary `git add` do the + right thing; `annex.largefiles=nothing` comes first and outputs and + data opt out — last match wins, and + `test_analysis_code_stays_in_git_and_stays_writable` pins it against + a real annex. +- **Manifests stay in git**, exempted back out of the annex, so a + bytes-free clone can classify a whole project. +- **An unfetched file exists, in two shapes** — an unlocked pointer + file (readable, hashes to the wrong thing) and a locked dangling + symlink (drops out of naive walks silently). `assets.data_version` + refuses both with `ContentNotFetchedError`; detection handles both + regardless of which shape lc writes, because `annex.thin` and + `git annex lock` are the researcher's to set. +- **Thin is per-add and only where lc writes.** Thin's hazard is an + in-place write rewriting the annex object under its own key; lc + always resets output directories rather than writing in place, but + `data/` is the researcher's, and their tools (`h5py`, astropy + `mode='update'`) do open files for update — so the flag never + reaches repository config. +- **`restore` is asymmetric on purpose:** a first materialization has + no HEAD version to go back to, and a failed task must not discard + edits made elsewhere while the graph ran. +- **Committing an archive or dot-named file needs `annex.dotfiles`** — + git-annex routes dotfiles to git whatever `largefiles` says, and + without the flag an image archive lands as a git blob, silently. + +## Tests + +`tests/test_dataset.py`, deliberately against **real tools** +(`real_tools` fixture): whether bytes land in the annex or as a blob +in git is not a question a stub can answer, and every bug this seam +has had was invisible to one. diff --git a/docs/api/identity.md b/docs/api/identity.md new file mode 100644 index 00000000..e7f865ca --- /dev/null +++ b/docs/api/identity.md @@ -0,0 +1,53 @@ +# lightcone.engine.identity + +What a materialized output is identified by: two hashes that answer +different questions, and the lock scan that decides whether an +environment can be audited at all. + +Source: `src/lightcone/engine/identity.py`. + +## Key symbols + +| Symbol | Role | +|---|---| +| `definition_version(recipe, decisions)` | What the spec says an output *is* — the rebuild trigger. | +| `env_version(root)` | What it ran under: lock bytes ‖ interpreter pin ‖ install settings ‖ image document. The `behind` trigger. | +| `scan_lock(root)` | Refusals, reports and advisories about what the lock pins. | + +## What must stay true + +- **`env_version` is not part of `definition_version`.** That is the + whole shape of the invalidation model: an environment edit stales + nothing, it makes outputs *behind*. (The original design nested + them; staling every output in every project on an engine upgrade was + the bug, not the cost.) +- **Both hashes are length-framed** — label, length, bytes per field — + so a boundary shift between adjacent fields cannot yield the same + digest from different inputs. Mutation-checked in the suite. +- **The lock is hashed as raw bytes, never parsed.** A comment reflow + moves `env_version`, deliberately: over-invalidation costs a report + line, while a parse of our own can silently disagree with uv. +- **The install-settings list is closed** (`_INSTALL_SETTINGS`), every + key hashed whether or not the project sets it — a setting outside + the list must not move the hash, one merely *matching* today's + default must. Settings are read where uv reads them (`uv.toml` + **replaces** `[tool.uv]`, measured); only values are hashed, never + which file supplied them. User-level uv config is deliberately out + of reach — machine state, not project state — and the residue is + tracked as issue #176. +- **The git commit is recorded, never hashed, and never a signal** — + one sha covers the whole tree, so hashing it stales everything on a + README edit. The honest consequence: editing `src/fit.py` remakes + nothing unless the file is declared as an ASTRA input. Do not add a + heuristic that scans recipes for repo paths. +- **The lock scan refuses only what cannot be audited** — path, + directory, and editable dependencies (two syncs of one lock can + install different code). A registry package with no wheel is a + report; a non-default group is advisory; the project's own package + is exempt. Names compare in PEP 503 form, or a project named + `my_project` fails to recognise itself. + +## Tests + +`tests/test_identity.py` — pure, and written as sensitivity tests in +both directions: what must move each hash, and what must not. diff --git a/docs/api/index.md b/docs/api/index.md index 491843b1..f3126ea8 100644 --- a/docs/api/index.md +++ b/docs/api/index.md @@ -1,61 +1,38 @@ -# Python API Reference - -The interesting public surface lives in `lightcone.engine.*`. The CLI -is a thin Click wrapper around these modules. - -## Module map - -| Module | Role | -|--------|------| -| [`lightcone.cli.commands`](cli.md) | Click CLI: `init`, `run`, `build`, `status`, `verify`, `setup`. | -| [`lightcone.engine.manifest`](manifest.md) | Per-output `.lightcone-manifest.json` write/read; `code_version`, `sha256_dir`. The integrity layer. | -| [`lightcone.engine.snakefile`](snakefile.md) | Generate `.lightcone/Snakefile` and `snakefile-config.json` from `astra.yaml`. | -| [`lightcone.engine.container`](container.md) | Runtime detection, content-addressed image tags, `wrap_recipe`. | -| [`lightcone.engine.dask_cluster`](dask_cluster.md) | Cluster lifecycle for `lc run` (local / SLURM / external). | -| [`lightcone.engine.status`](status.md) | Manifest-driven status walker. | -| [`lightcone.engine.verify`](verify.md) | Recompute hashes; walk the input chain. | -| [`lightcone.engine.tree`](tree.md) | Sub-analysis tree helpers — outputs, decisions, `from:` resolution. | -| [`lightcone.engine.validation`](validation.md) | Post-recipe sanity checks (empty dir, all-NaN columns, …). | -| [`snakemake_executor_plugin_dask`](dask_executor.md) | Snakemake executor plugin → `dask.distributed`. | -| `lightcone.engine.site_registry` | Vestigial — no active code path imports it. See [api/site_registry](site_registry.md). | - -## Common entry points - -```python -from pathlib import Path -from lightcone.engine.snakefile import generate, discover_universes -from lightcone.engine.container import load_runtime - -project = Path("my-analysis") -runtime = load_runtime(project_path=project).runtime -universes = discover_universes(project) # ['baseline', 'experiment'] -snakefile, cfg = generate(project, universes=universes, runtime=runtime) -# Now invoke `snakemake -s snakefile -d project --executor dask ...` -``` - -```python -from lightcone.engine.status import get_output_status - -for s in get_output_status(project, universe_id="baseline"): - print(s.status, s.output_id) # 'ok', 'stale', 'missing', or 'alias' -``` - -```python -from lightcone.engine.verify import verify_outputs - -failed = [r for r in verify_outputs(project, universe_id="baseline") if not r.passed] -for r in failed: - print(r.failure, r.output_id, r.detail) -``` - -```python -from lightcone.engine.container import ( - detect_runtime, - compute_image_tag, - build_image, -) - -runtime = detect_runtime() # 'podman' / 'docker' / 'podman-hpc' / None -tag = compute_image_tag("my-project", Path("Containerfile"), Path(".")) -build_image(tag, Path("Containerfile"), Path("."), runtime=runtime) -``` +# Engine Internals + +The `lightcone.engine.*` modules, one page each: what the module owns, +its key symbols, and the invariants a change must keep. These are +hand-written tours, not generated API dumps — the engine is not a +public API (projects don't depend on lightcone-cli), so what matters +is responsibility and contract, not every signature. + +## The map + +| Module | Owns | Character | +|---|---|---| +| [`project`](project.md) | What a project is: convergence, discovery, mode, the `_run` seam | impure | +| [`dataset`](dataset.md) | How a project stores: git + git-annex, run records, restore | impure | +| [`identity`](identity.md) | `env_version`, `definition_version`, the lock scan | pure | +| [`plan`](plan.md) | The spec, read as a graph of tasks (through ASTRA) | pure | +| [`assets`](assets.md) | One output: its directory, manifest, and state | pure | +| [`worker`](worker.md) | Making one output; the rerun entry point | impure | +| [`materialize`](materialize.md) | The driver: gates, scheduling, the save/restore loop, status | impure | +| [`venue`](venue.md) | Where a run executes: SLURM detection, the login guard | impure | +| [`sandbox`](sandbox.md) | The exec boundary: policy, backends, attestation, denials | mixed | +| [`image` & `container`](container.md) | The container hatch: declaration → image → archive → runtime | pure / impure | +| [`crate`](crate.md) | The publication view: the repo as an RO-Crate | pure | + +"Pure" here is a testing fact: pure modules are tested with nothing on +disk beyond `tmp_path` and nothing spawned; impure ones go through the +one subprocess seam (`project._run`) that the suite stubs — see +[Testing](../contributing/testing.md). + +Two files sit outside the engine on purpose: + +- **`lightcone/_sandbox_exec.py`** — the Landlock shim. Stdlib-only, + zero lightcone imports; it runs on every sandboxed exec, and an + engine import there would put click and the astra stack on that + path. Pinned by tests. +- **`lightcone/cli/commands.py`** — the CLI: flags, rendering, exit + codes. Imports the engine inside callbacks so `lc --help` stays + cheap; never contains logic worth testing beyond rendering. diff --git a/docs/api/io_manager.md b/docs/api/io_manager.md deleted file mode 100644 index ebfe3099..00000000 --- a/docs/api/io_manager.md +++ /dev/null @@ -1,8 +0,0 @@ -# lightcone.engine.io_manager (removed) - -The Dagster IO manager was retired. Output paths are now baked into the -generated Snakefile by [engine/snakefile](snakefile.md), and the canonical -location of every output directory is computed by -[`resolve_output_path`](tree.md) — root outputs land at -`results///`, and path-rooted sub-analyses land at -`/results///`. diff --git a/docs/api/manifest.md b/docs/api/manifest.md deleted file mode 100644 index 345bc451..00000000 --- a/docs/api/manifest.md +++ /dev/null @@ -1,112 +0,0 @@ -# lightcone.engine.manifest - -The integrity layer. Every materialized output gets a sidecar -`.lightcone-manifest.json` written by this module on the host -immediately after the recipe shell exits. - -Source: `src/lightcone/engine/manifest.py`. Schema version: `1` -(`SCHEMA_VERSION = 1` — bump if you change the manifest shape). - -## Public surface - -```python -__all__ = [ - "MANIFEST_FILENAME", # ".lightcone-manifest.json" - "SCHEMA_VERSION", # 1 - "code_version", - "fingerprint_external", - "read_manifest", - "sha256_dir", - "write_manifest", -] -``` - -## `code_version(*, recipe, container_image, decisions) → str` - -Deterministic content hash of everything that defines what the recipe -*does*: the recipe text, the resolved container image identifier, and -the canonicalized decision dict. Returns `"sha256:"`. - -The runtime used to invoke the container (docker / podman / podman-hpc) -is intentionally excluded — the same image produces the same data -regardless of which OCI tool launched it. - -`code_version` is embedded in each rule's `params.cfg` so Snakemake's -`params` rerun-trigger detects drift automatically. - -## `sha256_dir(path) → str` - -Deterministic content hash of a directory tree. Walks recursively, -hashes each file along with its relative path (so renames change the -hash), and excludes: - -- `.lightcone-manifest.json` (chicken-and-egg) -- `.snakemake_timestamp` (touched by Snakemake *after* the rule body - completes — including it would make every hash unstable) - -Raises `FileNotFoundError` if `path` does not exist. - -## `fingerprint_external(path, *, strict=False) → str` - -External (non-manifested) input fingerprint: - -- File: `mtime-size:-` by default; `sha256:` when - `strict=True`. -- Directory: always `sha256:` (via `sha256_dir`). -- Missing path: literal string `"missing"`. - -## `read_manifest(output_dir) → dict | None` - -Read `/.lightcone-manifest.json`. Returns `None` if the -file is missing or unparseable. **Does not** catch `OSError` — a -permission-denied or I/O error is surfaced rather than silently -masquerading as "missing", because it would otherwise hide real -problems from `lc verify` / `lc status`. - -## `write_manifest(*, output_dir, inputs, cfg) → Path` - -Atomically write the manifest for an already-materialized output. -Called from each rule's `run:` block. - -Required keys in `cfg`: - -- `output_id`, `universe_id` -- `recipe`, `container_image`, `decisions` -- `code_version` -- `git_sha`, `lc_version` - -`inputs` is a `dict[str, Path]` mapping declared input id → filesystem -path. For each input, the function reads the upstream manifest if -present and records its `data_version`; otherwise falls back to -`fingerprint_external`. - -Atomicity: writes to `.tmp`, then `os.replace()` rename. -Either both data and manifest exist at the end, or Snakemake reruns -the rule. - -## Manifest shape - -```jsonc -{ - "schema_version": 1, - "output_id": "accuracy", - "universe_id": "baseline", - "code_version": "sha256:…", - "data_version": "sha256:…", - "container_image": "lc-myproject-abc123", - "recipe": "python scripts/eval.py", - "decisions": { "scaling": "standard", "use_pca": "no" }, - "input_versions": { "features": "sha256:…", "labels": "mtime-size:…-…" }, - "git_sha": "...", - "lc_version": "0.4.0", - "host": "saul01", - "slurm_job_id": "1234567", - "finished_at": 1717000000.0 -} -``` - -## Tests - -`tests/test_manifest.py` covers `code_version` determinism, `sha256_dir` -exclusions, `fingerprint_external` modes, and `write_manifest` end-to-end -including the atomic rename. diff --git a/docs/api/materialize.md b/docs/api/materialize.md new file mode 100644 index 00000000..54b570de --- /dev/null +++ b/docs/api/materialize.md @@ -0,0 +1,72 @@ +# lightcone.engine.materialize + +Making a whole analysis: what runs, in what order, and what gets +committed. The driver refuses dirt, hands the graph to Dask, and owns +git alone — plus the read-only halves (`check`, `status`) that share +its classification walk. + +Source: `src/lightcone/engine/materialize.py`. + +## Key symbols + +| Symbol | Role | +|---|---| +| `materialize(root, targets, *, refresh)` | The run: guards → converge → plan → fetch → schedule → save/restore loop → crate converge. | +| `check(root, targets, *, refresh)` | The same classification without executing, committing, or fetching. Exempt from the dirty refusal. | +| `status(root)` | The report: every output's state and provenance commit, plus the mode/image/sandbox header facts. | +| `MaterializeReport` / `StatusReport` | The JSON surfaces; `ok` and `up_to_date` first. | +| `cluster_for_run()` | The venue ladder, and the two-method scheduler seam (`submit`, `completed`). | +| `run_record(...)` / `datalad_run_subject(...)` | The commit message `datalad rerun` replays, and the one spelling of its subject line — shared with the foreign-write comparator, because two strings here would drift. | +| `_engine_requirement()` | How a record pins its engine: by version for a release, by source commit (hatch-vcs) for a dev build. | + +## The run's order, and why + +1. **Login guard first** — the allocation is the remedy with queue + latency, so the user submits it before fixing anything else. +2. **Dirty refusal before the environment converge** — in + containerized mode the converge can commit an image archive, and + `dataset.save` commits the whole index; on a dirty tree the user's + staged edits would be swept in. +3. **Converge before the graph runs** — `uv run --locked --no-sync` + in workers would otherwise execute recipes against a drifted + `.venv` while manifests record the new lock (measured; the state + is made impossible rather than detected). +4. **Graph (validation, lock scan) before the image** — a refusal + over a typo must not cost a minutes-long build. +5. **HEAD, runtime, and foreign-write facts read once, handed down** + — the driver commits as results arrive, so any per-task read could + answer differently mid-run. Nondeterminism in a provenance field is + worse than either answer. +6. **Save on `ok`, restore otherwise, `try/finally` around the loop** + — an interrupt restores whatever is still outstanding; the tree + ends as clean as it started. + +## What must stay true + +- **The driver owns git, alone** — one thread, as results arrive. + A dependent may start while its upstream is being annexed; that is + measured-safe (the clean filter renames over the path, which never + stops existing) and must not be "fixed" by moving the save into the + task. +- **`up_to_date` is `ok and not made and not planned`** — a run where + every recipe failed must not report "nothing to do", and `behind` + never counts against it. +- **A read-only verb never tracebacks.** Anything `check`/`status` + cannot read classifies as "will be remade" and the real error + belongs to the recipe that follows. +- **The run record is genuinely re-runnable**: engine pinned by + requirement, project environment rebuilt by the worker from the + rerun commit's own lock, format tested *through datalad's parser* + and a real `datalad rerun` — a golden test over our own JSON stays + green through a silent break. +- **The crate converge is contained**: it runs after the loop, on the + full graph, and a failure there is a warning — the outputs are + already committed, and the crate is the publication view, not the + run. + +## Tests + +`tests/test_materialize.py` — real repositories, real recipes, a real +`LocalCluster` through the seam exactly once, real `datalad rerun` for +the record's whole claim. `cluster_for_run` is the one monkeypatch +point for venue-free tests. diff --git a/docs/api/plan.md b/docs/api/plan.md new file mode 100644 index 00000000..980eef4a --- /dev/null +++ b/docs/api/plan.md @@ -0,0 +1,57 @@ +# lightcone.engine.plan + +The spec, read as a graph of tasks. `astra.yaml` × `universes/*.yaml` +gives one task per `(universe, output)` pair that has a recipe; a task +carries everything executing it needs — the rendered command, where its +bytes go, what it reads, its decisions, its `definition_version` — and +nothing about *how* it will be executed. + +Source: `src/lightcone/engine/plan.py`. + +## Key symbols + +| Symbol | Role | +|---|---| +| `build(root)` | Validate the spec with ASTRA's own validators, resolve every universe, return the `Graph`. | +| `Graph` | Tasks keyed on `(universe_id, output_id)`; `order()` for the read-only topological walk, `resolve(targets)` for what a user typed, `closure(keys)` to narrow a run. | +| `Task` | One output in one universe, frozen. | +| `declared_path(root, path)` | The one rule that names a path: project-relative inside the tree, absolute outside, never resolved. | + +## What must stay true + +- **What the spec *means* is ASTRA's to say.** `astra.resolve` settles + decisions, resolves inputs, drops `when:`-excluded outputs, and + renders the placeholder grammar. This module holds only what + *execution* adds. A prior in-house interpretation diverged three + ways (couldn't build ASTRA's own nested example, ignored `when:`, + invented an input spelling `astra validate` rejects) — that history + is why re-derivation is banned. Missing semantics → PR to + astra-tools. +- **A spec ASTRA rejects never reaches a recipe.** `build` runs the + schema, file, and universe validators before resolving anything — + resolution answers what a *valid* spec means and does not re-check + that it is one. +- **The layout is flat and path-addressed.** + `results///`, and the path in a + rendered recipe *is* the path on disk — no staging, no relocation. +- **`declared_path` is lexical, never `resolve()`d.** A declared input + under `data/` is an annex symlink; resolving it writes + `.git/annex/objects/…` into the run record — the storage instead of + the input. This shipped once. +- **Two universes cannot share an id** (the id names a directory; + `build` refuses, naming both files), and an out-of-tree absolute + input is **reported, not refused** — its bytes still hash and + cascade, but the repository cannot bring it back, and saying so is + the whole obligation. +- **A target that matches nothing is an error** listing what exists — + quietly making nothing is the least useful thing a build tool can + do. + +## Tests + +`tests/test_plan.py` — pure; tests what lc *adds* (directories, edges, +versions, the validation gate), never what a spec means — that +coverage lives in astra-tools' own suite, and re-asserting it here +would recreate the second implementation this module deleted. Every +fixture must be a spec `astra validate` accepts; the gate enforces it +for free. diff --git a/docs/api/project.md b/docs/api/project.md new file mode 100644 index 00000000..a8c9f027 --- /dev/null +++ b/docs/api/project.md @@ -0,0 +1,54 @@ +# lightcone.engine.project + +What a project is: the convergence engine behind `lc init`, project +discovery, mode detection, and the one subprocess seam the whole +engine shares. + +Source: `src/lightcone/engine/project.py` (+ +`engine/templates/` for the scaffold's file content). + +## Key symbols + +| Symbol | Role | +|---|---| +| `converge(dir, *, write)` | The whole scaffold operation. `write=False` is check mode — the *same* decision path with side effects off. | +| `ConvergenceReport` | `created` / `repaired` / `unchanged` / `blocked` / `warnings`, plus `.converged` and `.as_dict()`. | +| `current_project()` | The cwd as a project: requires `pyproject.toml`, `uv.lock`, `.venv`. | +| `declared_project()` | The weaker question — what the repository carries, without `.venv`. One caller: the worker entry point, which builds the venv a moment later. | +| `mode(root)` | `"direct"` or `"containerized"` — presence of `[tool.lightcone.image]`, nothing else. | +| `uv_prefix(root, *, sync)` | The one spelling of the project uv hop. Callers differ only in `sync`: a probe converges the environment, a recipe must not. | +| `project_name(dir)` | PEP 503-ish name from the directory name. | +| `_run` / `_check_call` | Every external tool invocation, and the suite's one monkeypatch point. | +| `ProjectError` | The engine's one exception; the CLI translates it once. | + +## What must stay true + +- **Everything routes through the converger.** Every scaffold item + goes through `_Converger.item` / `.file` / `.blocked`; nothing + writes or records outside that mechanism. `.file` takes a *thunk*, + so check mode renders no template at all. +- **Derived artifacts converge by correctness, not existence.** + `uv.lock` and `.venv` are probed with uv's own no-write checks + (`uv lock --check`, `uv sync --locked --exact --check`); drift + reports as `repaired`. Check mode may probe but never mutates — + pinned by `test_check_mode_only_probes`. +- **A warning is advisory; a blocked item counts.** Convergence never + claims a project is converged while something it owns is absent or + unfixable — and repairs only ever append (`.gitignore` / + `.gitattributes` are converged entry-wise, order judged against the + template). +- **Only what git can carry is converged.** No `src/`, no empty + directories — a clone must need nothing but `.venv` and + `git annex init`, and + `test_a_clone_of_a_converged_project_is_converged` pins it. +- **There is no discovery.** The invoked directory is the project or + it is a clean error; every uv call carries an explicit `--project`. +- **Templates are files** (`templates/files/*.tmpl`, `string.Template` + with strict substitution), and a template gets a function only when + there is a value to decide or a merge policy to hold. + +## Tests + +`tests/test_project.py` (semantics, against the stubbed `_run`), +`tests/test_templates.py` (content, substitution, repair logic), +`tests/test_cli.py` (the `lc init` surface). diff --git a/docs/api/runner.md b/docs/api/runner.md deleted file mode 100644 index 8efc1cc6..00000000 --- a/docs/api/runner.md +++ /dev/null @@ -1,14 +0,0 @@ -# lightcone.engine.runner (removed) - -The pluggable runner (`docker`, `venv`, `local`, `slurm`) was replaced by -two thinner pieces: - -- The Snakefile generator at [engine/snakefile](snakefile.md) wraps each - recipe in a ` run --rm ...` invocation at generation time - (or leaves it bare when no container is configured). -- The Dask cluster manager at [engine/dask_cluster](dask_cluster.md) - decides whether the run is local, SLURM-backed via `srun`, or attached - to an external scheduler. - -There is no longer a single "backend" abstraction — those two -modules together cover what the runner used to do. diff --git a/docs/api/sandbox.md b/docs/api/sandbox.md new file mode 100644 index 00000000..d1175a35 --- /dev/null +++ b/docs/api/sandbox.md @@ -0,0 +1,70 @@ +# lightcone.engine.sandbox + +The exec boundary: what a command may touch, and how that is enforced. +A `Policy` says *what* in mechanism-free path sets; a `Backend` turns +it into **a different argv that sandboxes itself**; `boundary` picks +one, runs it, and reports what was actually enforced. `run.py` (the +`lc run` engine) and the worker are the two consumers. + +Source: `src/lightcone/engine/sandbox/` — `model.py`, `policy.py`, +`boundary.py`, `landlock.py`, `seatbelt.py`, `oci.py`, `denial.py` — +plus `lightcone/_sandbox_exec.py`, the Landlock shim. + +## Key symbols + +| Symbol | Role | +|---|---| +| `Policy` | What we will enforce: path sets, env overlay, exec allowlist. No mechanism ever appears in it. | +| `Capability` | What this host can do — `detect()`'s answer, the only `sys.platform` branch. | +| `Attestation` | What was actually enforced, derived from the flags applied — never from what the matrix says should have happened. | +| `Backend.wrap(policy, argv)` | The pure rewrite. `contains_prefix` declares whether the uv hop rides inside (a container is a world; a host mechanism trusts host plumbing). | +| `exec_policy(...)` | The one policy: probe and recipe get the same thing. Building it is where the impurity lives (the per-run private `$HOME`); `scope()` owns its cleanup. | +| `Unavailable` | A real backend that wraps to the same argv and attests `fs: open`. Saying so is the caller's job; pretending is nobody's. | +| `denial.explain()` / `denial.trailer()` | Best-guess remedies (allowed to return nothing) and the unconditional trailer on every nonzero sandboxed exit. | + +## What must stay true + +- **`wrap` stays pure** — no temp files, no FDs, no global state + (pinned by `test_wrap_is_pure`). That is what makes every backend + testable on a host that cannot run it, and it is why the Landlock + policy travels as JSON on argv rather than an inherited ruleset FD. +- **The shim stays alone**: stdlib only, zero lightcone imports, setup + failures exit the reserved 97, and it never falls through to running + the command unsandboxed. +- **Never grant EXECUTE on a directory that could be a system + prefix.** Landlock unions rights over ancestors, so one EXECUTE on + `/usr` outranks the whole per-file allowlist — with every test still + green, because the allowlisted binaries are exactly the ones that + were going to work. This shipped once (a venv on a system python); + the rule and its test are the fix. +- **SBPL is last-match-wins; Landlock unions.** The asymmetry decides + where a rule can live: the macOS guard takes back writes the + vendored defaults hand out, and the write tier is restated *after* + the guard — get the order wrong and layer 4 materializes on Linux + and refuses on macOS with the golden test still green. +- **Anything every backend must do belongs to the seam** — the env + overlay is composed in `boundary.env_argv()` once, for every + mechanism, so a mechanism added later cannot forget what it never + had to remember. (While each backend applied its own, `Unavailable` + applied none.) +- **The macOS profiles are vendored, not authored** (codex-derived, + provenance header, single delta) — the read baseline is a list of + things that break, found one production failure at a time. Put our + rules in the generator, keep `diff` against upstream as the re-sync + tool. +- **A denial is never invisible**: `explain()` may find nothing, so + the trailer fires on every nonzero exit, unconditionally. Remedies + name only what exists today. + +## Tests + +The suite splits along the seam: +`test_sandbox_policy/wrap/denial.py` (pure, every OS), +`test_sandbox_shim.py` (the shim as a real subprocess), +`test_sandbox_oci.py` (the mount table, pure), and +`test_sandbox_enforcement.py` — **the kernel's answer**, one suite for +both mechanisms, run against the *real* `exec_policy`, with +`LC_SANDBOX_TESTS_REQUIRED=1` turning "no mechanism, skip" into a hard +failure in CI. Every denial test is mutation-checked through +`Unavailable()` — a denial test that would pass unsandboxed is testing +nothing, silently. diff --git a/docs/api/site_registry.md b/docs/api/site_registry.md deleted file mode 100644 index 6efca3d7..00000000 --- a/docs/api/site_registry.md +++ /dev/null @@ -1,54 +0,0 @@ -# lightcone.engine.site_registry - -Known-site defaults. When lightcone-cli runs on a recognized site -(NERSC Perlmutter, a lightcone JupyterHub deployment), the matching -entry here supplies site-specific defaults — most importantly the -scratch root and the preferred container runtime. - -Source: `src/lightcone/engine/site_registry.py`. - -## What the module exposes - -- `SITE_DEFAULTS` — a dict mapping site keys (`"perlmutter"`, - `"jupyterhub"`, `"local"`) to a structured defaults dict (display - name, hostname patterns or env markers, backend, container runtime, - `scratch_root`, suggested QoS / constraint / time-limit options). -- `detect_current_site() → HostSite` — the high-level entry point. - Single source of truth for "which site are we on?": environment - markers win over hostname patterns (a pod's hostname is noise; the - injected env is the signal). Returns a falsy `HostSite` when nothing - matches. -- `HostSite` — frozen dataclass bundling the matched site key with its - defaults; `site.get("scratch_root")` etc. -- Lower-level pieces: `detect_site(hostname_or_name)`, - `detect_site_from_env()`, `get_site_defaults(site_key)`, - `list_known_sites()`, `get_site_scratch_deny_rules(site_key)`. - -## Who calls it - -- `lc init` (`lightcone.cli.commands`) — detects the site to surface - the resolved scratch root the run layer will use. -- `lightcone.engine.scratch` — `resolve_scratch_root()` falls back to - the site's declared `scratch_root` (e.g. `$SCRATCH` on Perlmutter, - `$HOME` on a JupyterHub pod) when the project config doesn't pin one. -- `lightcone.engine.container` — `auto` runtime resolution prefers the - site's declared `container_runtime` (`podman-hpc` on Perlmutter, - `kubernetes` on a hub). - -Everything should go through `detect_current_site()` rather than -re-deriving `socket.gethostname() + detect_site + get_site_defaults`. - -## Vestigial pieces - -`get_site_scratch_deny_rules()` and `list_known_sites()` currently -have no callers — they are residue from the removed target system. -The `suggested_options` blocks (QoS/constraint/time-limit guidance) -are likewise declared but not consumed yet. - -## Adding a site - -Append an entry to `SITE_DEFAULTS`. HPC sites match by -`hostname_patterns`; deployment-style sites (pods with arbitrary -hostnames) match by `env_markers`. Declare `scratch_root` for any site -where the default tempdir is wrong — see the `jupyterhub` entry's -comment for why a shared filesystem matters there. diff --git a/docs/api/snakefile.md b/docs/api/snakefile.md deleted file mode 100644 index ed8f3b43..00000000 --- a/docs/api/snakefile.md +++ /dev/null @@ -1,106 +0,0 @@ -# lightcone.engine.snakefile - -Generate `.lightcone/Snakefile` and `.lightcone/snakefile-config.json` -from `astra.yaml`. Both are auto-generated on every `lc run` — never -edit by hand. - -Source: `src/lightcone/engine/snakefile.py`. - -## Public surface - -```python -__all__ = ["generate", "discover_universes", "LIGHTCONE_DIR"] -``` - -## `generate(project_path, *, universes, runtime="none") → (Path, Path)` - -Reads `astra.yaml`, resolves the analysis tree, and writes: - -- `.lightcone/Snakefile` — the workflow. -- `.lightcone/snakefile-config.json` — per-`(rule_key, universe)` config. - -Returns the two paths. - -`runtime` is one of `docker | podman | podman-hpc | none` and is used to -wrap each recipe at generation time (see -[engine.container.wrap_recipe](container.md#wrap_recipe)). Resolution is -done once here, not per rule, so all rules use a consistent runtime. - -## `discover_universes(project_path) → list[str]` - -Sorted list of universe ids from `universes/*.yaml`, or `["default"]` if -the directory is empty / missing. - -## Generated Snakefile shape - -For each output with a `recipe:` block: - -```python -rule : - input: - ="/...", # only for sibling outputs - output: - data=directory(""), - manifest="/.lightcone-manifest.json", - params: - cfg=lambda wc: CFG[""][wc.universe], - run: - shell('printf "▶ [%s]\\n" "{wildcards.universe}" >&2') - shell(params.cfg["shell_command"]) - write_manifest( - output_dir=Path(output.data), - inputs={"": Path(input.), ...}, - cfg=params.cfg, - ) - for _w in validate_output(Path(output.data), params.cfg.get("output_type"), params.cfg["output_id"]): - print(f"\033[33m⚠\033[0m {_w}", file=sys.stderr) -``` - -## `cfg` content - -Per-`(rule_key, universe)` entry written into -`snakefile-config.json`: - -| Key | Source | Used by | -|-----|--------|---------| -| `output_id` | `tree_out.output_id` | `write_manifest` | -| `output_type` | `output_def["type"]` | `validate_output` | -| `universe_id` | universe name | `write_manifest` | -| `recipe` | `recipe.command` | `write_manifest` | -| `shell_command` | `wrap_recipe(recipe, image, runtime)` prefixed with `: lc_code_version=…;` | the rule body | -| `container_image` | raw `container:` spec from astra.yaml | `write_manifest` (for provenance) | -| `decisions` | merged universe decisions | `write_manifest`, `code_version` | -| `code_version` | `code_version(recipe, image_tag, decisions)` | drift detection via Snakemake `params` trigger | -| `git_sha`, `lc_version` | runtime metadata | `write_manifest` | -| `inputs` | resolved input paths (with `{universe}` substituted) | informational | - -## Why we own the container wrap - -Snakemake supports container directives (`container:` and -`--sdm apptainer`), but we deliberately don't use them. Two reasons, -both pragmatic: - -- `--sdm apptainer` adds an extra container layer that defeats - podman-hpc's migrate workflow. -- Default registry resolution on podman fails for - `lc--` tags because they trip - `unqualified-search-registries`. We pass `--pull=never` to skip the - lookup; Snakemake's machinery doesn't make this easy to thread - through. - -## Naming details - -- **`_rule_key(tree_out)`** — `output_id` for root outputs, - `.` for sub-analysis outputs. This is the - user-visible name and the cfg key. -- **`_rule_name(tree_out)`** — same as `_rule_key` but with `.` → - `__` because Snakemake rule names must be Python identifiers. -- **`_output_dir_pattern(tree_out)`** — wildcard path. Root and inline - sub-analyses: `results/{universe}/`. Path-rooted - sub-analyses: `/results/{universe}/`. - -## Tests - -`tests/test_snakefile.py` covers rule generation across root + sub-analyses, -input wiring, container wrapping, `code_version` embedding, and (last -test) parses the generated Snakefile via `snakemake -n`. diff --git a/docs/api/status.md b/docs/api/status.md deleted file mode 100644 index 85f4f31c..00000000 --- a/docs/api/status.md +++ /dev/null @@ -1,63 +0,0 @@ -# lightcone.engine.status - -Manifest-driven status walker. Reads only the per-output -`.lightcone-manifest.json` files; does **not** import Snakemake. - -Source: `src/lightcone/engine/status.py`. - -## Public surface - -### `get_output_status(project_path, *, universe_id) → Iterator[OutputStatus]` - -Yield an `OutputStatus` for every declared output in `project_path`'s -`astra.yaml`, against the named universe. Used by `lc status` and by -external tooling that wants a structured view. - -The function: - -1. Loads and resolves the analysis tree. -2. Loads merged universe decisions (tolerates a missing universe file — - returns an empty dict). -3. For each tree output: - - If it has no `recipe:` → `alias`. - - If no manifest at the output dir → `missing`. - - Otherwise recomputes `code_version` against the current spec and - compares to the manifest's recorded value. Match → `ok`, - mismatch → `stale`. - -### `OutputStatus` (dataclass) - -```python -@dataclass -class OutputStatus: - output_id: str - universe_id: str - analysis_id: str | None # None for root-level outputs - output_dir: Path - status: StatusLiteral # "ok" | "stale" | "missing" | "alias" - manifest: dict | None # None for missing/alias -``` - -### `StatusLiteral` - -```python -StatusLiteral = Literal["ok", "stale", "missing", "alias"] -``` - -## Why `code_version` is the staleness signal - -The manifest records the `code_version` that produced the data. Drift -detection just recomputes the current `code_version` from the live -spec and compares. Anything that touches recipe text, container image -tag, or decisions changes `code_version`; everything else is irrelevant -for staleness. - -For staleness against external inputs (e.g. someone edited a CSV under -`inputs/`), `lc status` doesn't catch it — that's outside `code_version`'s -scope. Use `lc verify` or rely on Snakemake's `mtime`/`input` rerun -triggers in `lc run`. - -## Tests - -`tests/test_status.py` covers the four status branches end-to-end -against tmp projects, including the alias and decision-drift paths. diff --git a/docs/api/targets.md b/docs/api/targets.md deleted file mode 100644 index 4f29d61b..00000000 --- a/docs/api/targets.md +++ /dev/null @@ -1,11 +0,0 @@ -# lightcone.engine.targets (removed) - -The target configuration module is gone. The only remaining global config -is `~/.lightcone/config.yaml`, which today carries one key: - -```yaml -container: - runtime: auto # auto | docker | podman | podman-hpc | none -``` - -It is read by [`lightcone.engine.container.load_runtime`](container.md). diff --git a/docs/api/tree.md b/docs/api/tree.md deleted file mode 100644 index 5e5301b3..00000000 --- a/docs/api/tree.md +++ /dev/null @@ -1,104 +0,0 @@ -# lightcone.engine.tree - -Walk the resolved analysis tree. Used by the Snakefile generator, -`status`, and `verify` to enumerate outputs, resolve `from:` -references, and merge universe decisions across nested sub-analyses. - -Source: `src/lightcone/engine/tree.py`. - -## `TreeOutput` (dataclass) - -```python -@dataclass -class TreeOutput: - output_id: str - output_def: dict # the raw dict from astra.yaml - analysis_id: str | None # None for root-level outputs - analysis_path: str | None # e.g. "./analyses/hod_fitting" - analysis_spec: dict # the sub-analysis spec dict (root spec for root outputs) -``` - -## Public functions - -### `collect_tree_outputs(spec) → list[TreeOutput]` - -Walk the resolved tree (root-level outputs first, then each -sub-analysis under `analyses:`) and return one `TreeOutput` per output -declaration. - -### `collect_tree_inputs(spec) → dict[str, dict]` - -Return `{qualified_id: input_def}` where `qualified_id` is `input_id` -for root inputs and `analysis_id.input_id` for sub-analysis inputs. - -### `resolve_universe_decisions(project_path, spec, universe_id) → dict` - -Load and merge universe decisions from root and sub-analyses. Returns -a flat dict using qualified keys for sub-analysis decisions -(`analysis_id.decision_id`) to avoid collisions. - -Behavior: - -- Loads root universe from `universes/.yaml`. -- For each sub-analysis with `path:`, looks for the corresponding - universe file at `/universes/.yaml`. The - sub-universe id comes from `root_universe.analyses..universe` - if present, otherwise the root universe id. -- `from:` references on sub-analysis decisions (`from: ../parent_decision`) - are resolved to the corresponding root decision value. -- Local sub-decisions not referenced via `from:` are still added. - -### `get_decisions_for_analysis(merged_decisions, analysis_id) → dict` - -Extract the decisions relevant to a specific analysis. Root analysis -(`analysis_id=None`): returns all unqualified keys. Sub-analysis: -returns decisions with the matching `analysis_id.` prefix, stripped to -local names. - -### `resolve_output_path(project_path, tree_output, universe_id) → Path` - -Returns the *parent* directory of the output dir (i.e. the -`results//` directory). The actual output dir is this path -joined with `tree_output.output_id`. - -- Root + inline sub-analyses: `/results//` -- Path-rooted sub-analyses: `//results//` - -### `resolve_container_spec(tree_output, root_spec) → str | None` - -Pick the container declaration in priority order: - -``` -recipe-level > sub-analysis-level > root-level -``` - -Returns the raw spec string (Containerfile path or registry image), or -`None` when no container is declared anywhere. - -### `find_upstream_output(consumer, inp_id, all_outputs) → TreeOutput | None` - -Resolve a recipe input id to the producing `TreeOutput`. Mirrors the -lookup the Snakefile generator does for `rule.input`. Handles: - -- Dotted `.` → match by qualified key. -- Inside a sub-analysis, bare `inp_id` → first try - `.`, then bare. -- `inp_id` referencing an analysis-level input with `from:` pointing - at a sibling output → resolved through that. - -Returns `None` for inputs that refer to external files (no producer -rule). - -### `resolve_input_path(project_path, spec, from_ref, universe_id) → str | None` - -Resolve a `from:` reference on an input to a concrete filesystem path. -Handles: - -- `../parent_input` → root input's `source` (only if absolute). -- `../sibling.output_id` → sibling sub-analysis's results path. -- `sibling.output_id` (no `../`) → same as above (root convenience). - -## Tests - -`tests/test_tree.py` covers all four resolvers, including the `from:` -edge cases for sub-analysis decisions and inputs. diff --git a/docs/api/validation.md b/docs/api/validation.md deleted file mode 100644 index ce020198..00000000 --- a/docs/api/validation.md +++ /dev/null @@ -1,50 +0,0 @@ -# lightcone.engine.validation - -Post-recipe sanity checks. Called by every rule's body after -`write_manifest`. **Never raises** — all problems are returned as -warning strings and printed to stderr. - -Source: `src/lightcone/engine/validation.py`. - -## `validate_output(output_dir, output_type, output_id) → list[str]` - -Inspect the output directory after a successful recipe run. Empty list -means no problems. Returned strings are human-readable and prefixed by -the rule body with a `⚠ ` marker. - -The check fires unconditionally on a few "this is almost certainly -wrong" situations: - -- Output directory does not exist after a successful run. -- Output directory exists but is a file rather than a directory. -- Output directory is empty after a successful run. - -Beyond that, behavior depends on the declared `type:` in `astra.yaml`: - -| `output_type` | Check | -|---------------|-------| -| `metric` | At least one `*.json` file present, parseable, not all-null/all-NaN. | -| `table` | At least one `*.csv` file present; parseable; warns on individual all-NaN numeric columns and on tables where every numeric column is all-NaN. | -| `figure` | At least one `*.png/jpg/jpeg/svg/pdf/eps` file present; warns on zero-byte files. | -| anything else | Empty list (no specific check). | - -## What it does *not* do - -This is a smoke test, not a validator. It does not: - -- Check the schema or shape of metric JSON beyond null-detection. -- Compare against expected values. -- Catch silent computational errors. -- Block the run — warnings are printed but the manifest is still - written. - -For deeper validation, layer your own checks in the recipe (`assert`, -`pydantic`, …). The point of `validate_output` is to flag the cheap -common silent failures: empty directories, NaN-only columns, missing -files. - -## Tests - -`tests/test_validation.py` covers each output-type branch including -the malformed-input edge cases (unparseable JSON, missing CSV, zero-byte -figures, …). diff --git a/docs/api/venue.md b/docs/api/venue.md new file mode 100644 index 00000000..d86735dc --- /dev/null +++ b/docs/api/venue.md @@ -0,0 +1,63 @@ +# lightcone.engine.venue + +Where a run executes. A venue is host state, never project state — +nothing here reads the project or enters any identity. The one venue +beyond the local machine is a SLURM allocation, detected rather than +configured: the user already answered every resource question at +`salloc`, so the allocation *is* the declaration and lc's job is to +span it. + +Source: `src/lightcone/engine/venue.py` (consumed by +`materialize.cluster_for_run`). + +## Key symbols + +| Symbol | Role | +|---|---| +| `slurm_client()` | The allocation branch: a scheduler in the driver process bound to `SLURMD_NODENAME`, one `srun --overlap` launching a worker per node on `sys.executable`. | +| `require_compute_node(command)` | The login guard: refuses iff a known center's marker is set and `SLURM_JOB_ID` is not, printing that center's own `salloc`/`sbatch` spellings. | +| `allocation_nodes()` | How many nodes the allocation holds; 0 outside one. | +| `_SITES` | One row per known center — name, marker, remedies, **verified against the center's documentation, never guessed**. NERSC is the seeded row. | + +## What must stay true + +- **The detection ladder lives in `cluster_for_run()` alone.** Nothing + else asks where a run executes; a future submission-model venue is + one more branch there plus only the config it genuinely needs. +- **Workers run the driver's own interpreter** (`sys.executable -m + distributed.cli.dask_worker`) — on HPC that is the tool env on the + shared filesystem, so driver and workers are the identical + installation and version skew is structurally out. Workers need no + git and no annex. +- **The worker flags are each load-bearing**: `--nthreads=` + (tasks block in `subprocess.wait()` with the GIL released), + `--no-nanny` (srun won't relaunch either), `--memory-limit 0` (the + real work is behind the exec boundary; Dask would pause workers over + phantom numbers), `--death-timeout 60` (a worker whose driver died + exits instead of holding the node), `--local-directory /tmp` + **literal** (a site prolog can scope `TMPDIR` per node or step, so a + driver-resolved path can be absent elsewhere). +- **The srun child is the one documented exception to `project._run`** + — it lives as long as the run and its stderr must reach the terminal + live. Teardown retires workers first, then wait → terminate → kill, + bounded; connection is a poll loop so a dead srun reports *its exit + code* now, not a timeout later. +- **A leak refuses loudly, never falls back silently**: `SLURM_JOB_ID` + with no srun on PATH, a non-integer count variable, an unresolvable + `SLURMD_NODENAME` — each is a named refusal. +- **The guard is materialize-scoped** (plus the rerun entry point — + the record's `cmd` is how recipes reach login nodes without `lc` in + the command line). `check`, `status` and `lc run` never call it: a + login node is exactly where "where does this stand" gets asked. +- **A containerized multi-node run requires a shared image store** — + `_SHARED_STORE_RUNTIMES` (podman-hpc), asked positively, checked in + `materialize()` before the runtime resolves so the refusal costs no + build. + +## Tests + +`tests/test_venue.py` — fakes the *host*, never the code: SLURM +variables set deliberately, a bash stub standing in for srun, and the +end-to-end tests run a real graph through the real bind/launch/teardown +on any machine. The `venue_env` autouse fixture scrubs venue variables +suite-wide (derived from `_SITES`, so a new center is one row). diff --git a/docs/api/verify.md b/docs/api/verify.md deleted file mode 100644 index 491f5c7d..00000000 --- a/docs/api/verify.md +++ /dev/null @@ -1,66 +0,0 @@ -# lightcone.engine.verify - -Recompute on-disk hashes and walk the recorded input chain. Catches -tampering, drift, and forged manifests. Like `status`, this module -never imports Snakemake. - -Source: `src/lightcone/engine/verify.py`. - -## Public surface - -### `verify_outputs(project_path, *, universe_id) → Iterator[VerifyResult]` - -Yield a `VerifyResult` for every materialized output (i.e. every output -with a recipe whose directory exists on disk) in the named universe. - -Outputs that aren't materialized at all are silently skipped — that's a -question for `lc status`, not `lc verify`. - -For each materialized output: - -1. Read its manifest. If missing or unparseable → `missing_manifest`. -2. Recompute `sha256_dir(output_dir)`. If it doesn't match the recorded - `data_version` → `tampered_data` (with a `recorded … != actual …` - detail message). -3. Walk recorded `input_versions`: - - For each declared recipe input, look up the upstream output via - `find_upstream_output`. - - If upstream is external (no producer rule) → nothing to chain to. - - If upstream's current manifest is missing → `broken_chain` - ("upstream … missing manifest"). - - If upstream's current `data_version` ≠ recorded → `broken_chain` - ("upstream … data_version drifted"). - - If the input is missing from the manifest entirely → - `broken_chain` ("input … missing from manifest"). -4. Otherwise `passed=True`. - -### `VerifyResult` (dataclass) - -```python -@dataclass -class VerifyResult: - output_id: str - universe_id: str - output_dir: Path - passed: bool - failure: FailureKind | None - detail: str | None = None -``` - -### `FailureKind` - -```python -FailureKind = Literal["missing_manifest", "tampered_data", "broken_chain"] -``` - -## Performance notes - -`sha256_dir` is the dominant cost. Hashing 10 GB of float arrays takes -real wall time. `lc status` is the cheap version that recomputes -`code_version` only — use that for the day-to-day "is this stale?" -question, and `lc verify` for periodic / pre-publication audits. - -## Tests - -`tests/test_verify.py` covers each failure kind end-to-end against tmp -projects, plus the chain-walking through nested sub-analyses. diff --git a/docs/api/worker.md b/docs/api/worker.md new file mode 100644 index 00000000..2dbe05b0 --- /dev/null +++ b/docs/api/worker.md @@ -0,0 +1,59 @@ +# lightcone.engine.worker + +Making one output — the unit of work, and the only thing that runs a +recipe. Also an entry point: + +```text +python -m lightcone.engine.worker / +``` + +which is what the `[DATALAD RUNCMD]` record in every materialization +commit names, behind an engine-pinning `uv run --no-project --with …`. +It is a module rather than an `lc` verb on purpose: it makes the +output unconditionally, commits nothing, and leaves the tree dirty by +design — precisely the state `lc materialize` refuses to start from — +so advertising it would hand people a footgun. + +Source: `src/lightcone/engine/worker.py`. + +## Key symbols + +| Symbol | Role | +|---|---| +| `materialize(task, versions, ...)` | The unit: classify → reset the directory → sandbox → recipe → hash → manifest. Returns a `TaskResult`, always. | +| `TaskResult` | `ok` / `current` / `behind` / `failed` / `blocked`, the output's `data_version`, and the attestation. `.usable` is what dependents check. | +| `main(argv)` | The rerun entry point: guards, converges the project environment from the commit's own lock, resolves its own HEAD and runtime, executes. | +| `lc_version()` | The engine version every manifest records. | + +## What must stay true + +- **The worker never raises** — enforced at the unit boundary, so the + contract holds for failure modes nobody enumerated. Raising would + make Dask abort every task in flight; reporting all independent + failures in one run is most of what owning the loop buys. +- **`data_version` is computed here, before anything is staged** — the + dependent's argument *is* this return value, so the digest must + exist while the files are still unannexed. Deriving it from + `git annex find` records `sha256([])` for everything, silently, with + green tests — and couples the digest to the annex backend, which is + deliberately not pinned. +- **The reset takes the whole directory** — a crashed previous run can + have left anything there, and there is no "expected file list" to + delete by. The `output_dir` guard bounds the blast radius, not a + narrower delete. +- **No git in here.** The driver commits; a worker that asked git + would race the index lock and could read a HEAD this same run moved. +- **`main`'s "no output ``" message covers the task lookup only.** + It once wrapped the whole body, and a `KeyError` from anywhere + inside astra surfaced as "bad target" — a rerun misdiagnosing itself + at the one place nobody is watching. +- **Keep it cheap to import — no click, no rich.** It is on the path + of every task and every rerun; two tests pin the imports and the + absence from `--help`. (Nothing pins the absence of a + `[project.scripts]` entry — treat that as a review item.) + +## Tests + +`tests/test_worker.py` — real recipes through the real boundary +against a real repository (the `analysis` fixture): whether gates +hold and bytes land are not questions a stub can answer. diff --git a/docs/architecture.md b/docs/architecture.md index f8461629..6d28ba8b 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -1,313 +1,190 @@ # Architecture -The whole story in one sentence: **lightcone-cli is a thin shim over -Snakemake that owns provenance.** This page expands that sentence. +How lightcone-cli is put together, for someone about to change it. The +[user-guide concepts page](user/concepts.md) covers what the tool +promises; this page covers how the promises are kept. -## Three subsystems +## The split that everything else follows -1. **Snakefile generation** — translate `astra.yaml` into a - `.lightcone/Snakefile` and a sidecar `snakefile-config.json` keyed by - `(rule, universe)`. Snakemake handles the rest of execution. -2. **Manifest layer** — a per-output sidecar JSON written *by us* on the - host immediately after each rule's recipe shell exits. The integrity - contract lives here. -3. **Cluster management** — `lc run` always dispatches through a Dask - scheduler whose lifetime equals the run's lifetime. The cluster - manager picks the right shape (local / SLURM / external) on the fly. - -Everything the user touches is the `lc` CLI on top of these: the engine -(Snakefile generation + cluster management) and the integrity layer -(manifests, `lc status`, `lc verify`). - ---- - -## 1. Snakefile generation - -Generator: [`lightcone.engine.snakefile.generate`](api/snakefile.md). - -For each output in the resolved analysis tree (root + sub-analyses, -expanded by `astra.helpers.resolve_analysis_tree`), the generator emits -one Snakemake rule per output. The rule body is a `run:` block: - -```python -rule : - input: ... # from upstream outputs (sibling rules) - output: - data=directory("results/{universe}/"), - manifest="results/{universe}//.lightcone-manifest.json", - params: - cfg=lambda wc: CFG[""][wc.universe], - run: - shell('printf "▶ [%s]\\n" "{wildcards.universe}" >&2') - shell(params.cfg["shell_command"]) # the recipe (already container-wrapped) - write_manifest(output_dir=Path(output.data), inputs={...}, cfg=params.cfg) - for w in validate_output(...): print(f"⚠ {w}", file=sys.stderr) -``` - -### What goes in `cfg` - -`snakefile-config.json` is keyed by ` → → cfg` where -the inner dict carries: - -- `shell_command` — the recipe pre-wrapped at generation time. When - containers are configured, this looks like - ` run --rm --pull=never -v "$PWD":"$PWD" -w "$PWD" bash -c ''`. - Snakemake's own `container:` directive and `--sdm apptainer` are - intentionally *not* used — we own the runtime end-to-end. -- `code_version` — `sha256(recipe + container_image + decisions)`. - Embedded as a `: lc_code_version=…;` no-op prefix on the shell command - so it lands in any shell trace. -- `recipe`, `container_image`, `decisions`, `output_id`, `output_type`, - `universe_id`, `git_sha`, `lc_version`, resolved input paths. - -### Why pre-wrap, not Snakemake's `container:`? - -Two reasons. First, `--sdm apptainer` adds an extra container layer that -defeats podman-hpc's migrate workflow. Second, registry image resolution -on podman fails for our content-addressed `lc--` tags -because they trip `unqualified-search-registries` in `registries.conf`. -We pass `--pull=never` to skip the lookup entirely; that requires -images to be present locally, which is what `lc build` does. - -### Staleness detection - -The generator does *not* override Snakemake's rerun logic — it just -makes sure drift is visible to it. We default to -`--rerun-triggers code,input,mtime,params`. The `params` trigger is the -one that fires today: `cfg` is per-universe and contains -`code_version`, so any change to recipe / container image / decisions -flows through. - ---- - -## 2. The manifest layer - -Module: [`lightcone.engine.manifest`](api/manifest.md). Filename: -`.lightcone-manifest.json` (constant; `SCHEMA_VERSION = 1`). - -Every successful rule writes a manifest to its output directory. The -write is atomic (`os.replace` rename); a missing or unparseable manifest -re-runs the rule on the next `lc run`. - -### Fields - -```json -{ - "schema_version": 1, - "output_id": "...", - "universe_id": "baseline", - "code_version": "sha256:…", - "data_version": "sha256:…", - "container_image": "lc-myproject-abc123" , - "recipe": "python scripts/compute.py", - "decisions": {...}, - "input_versions": { "": "sha256:…" }, - "git_sha": "...", - "lc_version": "...", - "host": "...", - "slurm_job_id": "...", - "finished_at": 1700000000.0 -} -``` - -### `data_version` exclusions - -`sha256_dir()` skips two filenames: `.lightcone-manifest.json` (chicken -and egg) and `.snakemake_timestamp` (Snakemake touches the directory -*after* the rule body completes — including it would make every hash -unreproducible). - -### `input_versions` semantics - -For each declared recipe input: -- If the input is a sibling output (has its own manifest) → - `data_version` from that manifest. -- Otherwise treated as external → - `mtime-size:-` for files, `sha256_dir(...)` for - directories, `"missing"` for absent paths. - -### What `lc verify` checks - -- **`tampered_data`** — `sha256_dir()` of the on-disk output no longer - matches the recorded `data_version`. -- **`broken_chain`** — a recorded `input_versions[id]` no longer matches - the upstream output's current `data_version`. -- **`missing_manifest`** — the output directory exists but has no - manifest, or the manifest fails to parse. - -### What `lc status` checks - -- **`ok`** — manifest present, recomputed `code_version` matches. -- **`stale`** — manifest present but `code_version` drifted (recipe, - image, or decisions changed). -- **`missing`** — no manifest. -- **`alias`** — output declared without a recipe; materialized only as a - side effect of an upstream. - -`status` reads only manifests. No Snakemake import, no `.snakemake/` -directory required, works on a fresh clone or frozen archive. - ---- - -## 3. Cluster management - -Module: [`lightcone.engine.dask_cluster`](api/dask_cluster.md). - -`cluster_for_run()` is the only entry point. It is a context manager -that yields a Dask scheduler address valid for the duration of the run, -across three branches: - -1. `DASK_SCHEDULER_ADDRESS` already set → yield as-is. We don't own the - cluster, we don't tear it down. -2. `SLURM_JOB_ID` set → start an in-process scheduler bound to the - driver hostname (`SLURMD_NODENAME` or `gethostname()`), then `srun` - one `dask worker` per node across the allocation. Workers advertise - the node's resources via Dask abstract resources (`cpus`, `memory`, - `gpus`). The Snakemake executor plugin maps per-rule - `cpus_per_task` / `mem_mb` / `gpus_per_task` to per-task constraints. -3. Neither → `LocalCluster()` sized to the local machine. - -The scheduler is always in-process so its lifetime equals the run's -lifetime: no service to manage, no orphaned schedulers. - -### The Snakemake executor - -Module: [`snakemake_executor_plugin_dask`](api/dask_executor.md). - -Snakemake calls `run_job(job)`, we translate it to: - -```python -client.submit( - _run_shell, cmd, - resources=_build_resources(job), - pure=False, - key=f"snakejob-{job.name}-{job.jobid}", -) +```text +lc (CLI) engine ASTRA +───────────── ───────────────────── ───────────── +flags, rendering, ──► what a project is, ──► what a spec +exit codes how outputs are made *means* ``` -The worker shells out to the (already container-wrapped) command. There -is no per-rule "executor logic" to write — recipes are wrapped at -generation time, so the worker just runs them. - ---- - -## Container layer - -Module: [`lightcone.engine.container`](api/container.md). - -Two surfaces: - -- **Build** — `compute_image_tag()` + `build_image()`. Tags are - `lc--` over the Containerfile and dependency - files (`requirements.txt`, `pyproject.toml`, `poetry.lock`, - `Pipfile.lock`, …). Rebuilds happen only when the hash changes. -- **Run-time wrap** — `wrap_recipe()` produces the command string that - the Snakefile generator embeds into each rule. - -Runtime resolution: `~/.lightcone/config.yaml` carries -`container.runtime` (`auto | docker | podman | podman-hpc | none`). -`auto` picks the first usable in `(podman, docker, podman-hpc)`, -skipping docker if its daemon is unreachable. `none` is an explicit -opt-out — recipes run on the host. When `auto` falls back to `none` -silently, `lc run` warns that the manifest's `container_image` field -will misrepresent what actually executed. - -For `podman-hpc`, the build path also runs `podman-hpc migrate ` -so compute nodes can read the image without a registry. - ---- - -## Sub-analysis tree - -`astra.yaml` can declare nested `analyses:` pointing to sub-directories -each with their own `astra.yaml`. The full tree is resolved by -`astra.helpers.resolve_analysis_tree()` before any operation. - -Output paths follow the analysis layout: - -- Root + inline sub-analyses: `results///` -- Path-rooted sub-analyses: `/results///` - -`from:` references on inputs and decisions are resolved by helpers in -[`engine.tree`](api/tree.md). When an output id is ambiguous (the same -name appears in multiple sub-analyses), `lc run` errors and asks for -the qualified `.` form. - ---- - -## Repository at a glance +- **`cli/commands.py`** owns flags, console rendering, and exit codes — + nothing else. It imports the engine *inside* command callbacks, so + `lc --help` stays cheap. The engine never imports click and never + prints. +- **The engine** owns everything about what a project is and how + outputs get made. It raises `ProjectError`; the CLI's group class + translates that into a clean error message, once, for every verb. +- **ASTRA** owns what a spec means. Scoping, `from:` references, + conditional outputs, universe resolution, and the recipe placeholder + grammar are all answered by `astra.resolve` and checked by + `astra.validation` — never re-implemented here. When the spec's + *meaning* looks wrong, the fix is a PR to astra-tools. + +The engine ships as the `lightcone.*` PEP 420 namespace — +`src/lightcone/` has **no `__init__.py`**, so sibling distributions can +share the namespace. The engine is the host's `uv tool`, never a +project dependency: a project's lock carries only what the analysis +imports. + +## One run, end to end ```text -src/lightcone/ # PEP 420 namespace package — NO __init__.py -├── cli/ # Click surface -│ ├── __init__.py # exposes main() -│ └── commands.py # init, run, status, verify, build, export -├── engine/ # execution substrate -│ ├── manifest.py # write_manifest, sha256_dir, code_version -│ ├── snakefile.py # generate .lightcone/Snakefile from astra.yaml -│ ├── container.py # docker/podman/podman-hpc build + recipe wrap -│ ├── cloudbuild.py # GCP Cloud Build backend (kubernetes runtime) -│ ├── dask_cluster.py # cluster lifecycle (local/SLURM/Gateway/external) -│ ├── scratch.py # scratch-root resolution, run dirs, run lock -│ ├── status.py # manifest-driven status walker (no Snakemake) -│ ├── verify.py # recompute hashes, walk the chain -│ ├── tree.py # sub-analysis tree helpers -│ ├── validation.py # post-recipe output sanity checks -│ ├── wrroc.py # Workflow Run RO-Crate export -│ └── site_registry.py # known-site defaults (scratch root, runtime) - -src/snakemake_executor_plugin_dask/ # Snakemake executor → dask.distributed - -tests/ # pytest, mirrors src/ -pyproject.toml # hatchling + hatch-vcs; ASTRA + Snakemake as deps +lc materialize + │ guard: compute node? tools? git identity? + │ refuse: dirty tree + │ converge: uv.lock ⇄ .venv (and the image, containerized) + │ plan: astra validate + resolve → Graph of Tasks + │ fetch: git annex get (declared inputs not in this clone) + │ venue: SLURM allocation? → srun workers · else LocalCluster + ├─► workers: reset output dir → sandbox → recipe → hash → manifest + │ (never raise; return ok/current/behind/failed/blocked) + └─ driver: consume results in one thread + ok → dataset.save (commit + run record) + failed → dataset.restore (tree as clean as it started) + finally → converge ro-crate-metadata.json (if licensed) ``` -The `lightcone.*` namespace is a PEP 420 implicit namespace package. -**Do not add `src/lightcone/__init__.py`** — that would turn it into a -regular package and break coexistence with future sibling distributions -(`lightcone-ui`, etc.). Any new `lightcone-*` package must live under -`src/lightcone//` and ship only its own subpackage. - ---- +The division of labor is strict and load-bearing: + +- **The driver owns git, alone.** Workers execute and return a + `TaskResult`; the driver commits as results arrive, in one thread. + Concurrent git operations race on the index lock — this split is not + a preference. +- **Dask owns the ordering.** Every task is submitted with its + upstream futures as arguments; there is no ready-set loop or + hand-rolled topological sort on the execution path. +- **The worker never raises.** A recipe failure, a gate failure, an + unreadable manifest — all come back as a state, so one failure + doesn't abort every task in flight, and a run reports *all* its + independent failures. +- **Values are resolved once and handed down.** HEAD, the container + runtime, and the foreign-write facts are read by the driver and + passed to workers as values — a worker that asked git itself could + get a different answer mid-run, and workers have no git anyway. + +## Identity: two hashes, three states + +`identity.py` computes two digests that deliberately answer different +questions: + +- **`definition_version`** = hash(rendered recipe ‖ decisions) — what + the spec says the output *is*. When it moves, the artifact + contradicts the spec: **stale**, remade. +- **`env_version`** = hash(lock bytes ‖ interpreter pin ‖ install + settings ‖ image document) — what the output *ran under*. When it + moves, the artifact is merely from another time: **behind**, + reported, left alone. + +`assets.classify` is the one implementation of the rule, with two +callers: the worker (live input digests) and the read-only walk +(`None` for anything upstream that will run — "this is going to +change"). That single value is the entire difference between run and +check, which is what keeps `--check` honest. `behind` does not +propagate; `stale` wins when both apply; and a foreign write (an +output's directory last touched by a commit that is not its own run +record) classifies stale through the same rule, as one more input +value. + +Both hashes are length-framed (label, length, bytes per field), so a +boundary shift between concatenated fields cannot produce a collision. +The lock is hashed as raw bytes, never parsed — over-invalidation +costs a report line; a parse that disagrees with uv costs correctness. + +## Storage: the repository is the record + +`dataset.py` is the whole git + git-annex seam. The model is DataLad's: +git carries pointers and history, the annex carries bytes, and +`.gitattributes` routes content (`annex.largefiles=nothing` by +default; `data/` and `results/` opt out). A researcher only ever types +ordinary `git add` / `git commit`. + +Each output is committed with a **run record** — a `[DATALAD RUNCMD]` +commit message whose `cmd` reconstructs the engine +(`uv run --no-project --with lightcone-cli==`) and re-executes the +worker entry point, so `datalad rerun` replays the making of an output +with the gates, the sandbox, and the manifest intact. Results are +committed *thin* (hard-linked to their annex object), which is safe +precisely because lc never writes an output in place — the worker +resets the directory first. + +## The exec boundary + +Every recipe and every `lc run` command goes through +`engine/sandbox/`: a `Policy` (mechanism-free path sets) is turned +into *a different argv that sandboxes itself* by a `Backend` — +Landlock via the stdlib-only shim `lightcone/_sandbox_exec.py`, +Seatbelt via `sandbox-exec`, the OCI mount table in containerized +mode, and `Unavailable` (wrap = identity) where no mechanism exists. +Because every backend is a pure argv rewrite, all of them are testable +on a host that can't run them, and the manifest's `hermeticity` field +records what was *actually* enforced — never what should have been. + +There is one policy, `exec_policy`: probe and recipe get exactly the +same thing (tree read-only apart from `results/`), so "works under +`lc run`" and "works as a recipe" stay the same fact. + +## The container hatch + +Containerized mode changes the recipe's world and nothing else. +`image.py` (pure) turns the `[tool.lightcone.image]` declaration into +a rendered Containerfile, an identity document, and a content tag; +`container.py` (impure) builds it, saves it as a `docker-archive` +inside the repository (`.datalad/environments//image`, annexed), +and enters it. The engine never enters the image — driver, git, and +classification stay on the host; exactly two things run in-image: the +environment sync and each recipe exec, over a read-only rootfs with +the mount table as the whole policy. Execution pins the archive's +config-blob id, never a tag. + +## Venues + +`materialize.cluster_for_run()` is the one place that decides where a +run executes, and the seam it returns is two methods wide — +`submit(fn, *args, key=…)` and `completed(handles)`. A SLURM +allocation (detected by `SLURM_JOB_ID`) gets one worker per node via a +single `srun`, running the driver's own interpreter so driver and +workers are the identical installation. Anything else is the local +machine. Venues are detected, never configured; the only venue config +that exists is the allocation the user already requested. + +## The publication view + +`crate.py` renders the repository as a Provenance Run Crate — a pure +function of repository state (sorted iteration, no clock, git injected +as a callable), which is what lets `materialize` converge +`ro-crate-metadata.json` byte-for-byte and commit only differences. +Run identity comes free from the manifests' `git_sha` (the driver +reads HEAD once per run), so one materialize maps onto one +`OrganizeAction` with no new manifest field. -## Execution flow +## Repository at a glance ```text -astra.yaml ── snakefile.generate() ──► .lightcone/Snakefile + .lightcone/snakefile-config.json - │ - snakemake -s … -d … --executor dask - │ - ┌──────────────────────┼──────────────────────┐ - │ │ │ - DAG resolution per-rule run: dask scheduler - (Snakemake) shell(recipe) (LocalCluster / - + write_manifest() SLURM-srun / - external) - │ - └─► results///data - results///.lightcone-manifest.json +src/lightcone/ # namespace — NO __init__.py +├── _sandbox_exec.py # the Landlock shim — stdlib only, zero lightcone imports +├── cli/commands.py # flags, rendering, exit codes — nothing else +└── engine/ + ├── project.py # what a project is: convergence, discovery, mode + ├── dataset.py # the git + git-annex seam + ├── identity.py # env_version, definition_version, the lock scan + ├── image.py # the system layer, declared → rendered — pure + ├── container.py # runtimes, the build, the archived image — impure + ├── crate.py # the publication view — pure + ├── assets.py # an output: directory, manifest, state + ├── plan.py # the spec, read as a graph of tasks + ├── worker.py # making one output; the rerun entry point + ├── materialize.py # the driver: gates, Dask, the save/restore loop + ├── run.py # what `lc run` is + ├── venue.py # where a run executes + ├── sandbox/ # the exec boundary + └── templates/ # the scaffold's file content, as real files ``` -What Snakemake owns (we don't write it): DAG construction, topological -execution, parallelism, dry-run, locking, retry, log capture, -per-rule resources, `--rerun-triggers` for staleness detection. - -What we own: a Snakefile generator, the manifest layer (write/read/verify), -a status walker, a verify routine, the Dask cluster manager, the -container-runtime layer, and a Snakemake executor plugin that submits -each rule to a Dask scheduler. - ---- - -## Configuration files - -| File | Scope | Purpose | -|------|-------|---------| -| `astra.yaml` | Project | The spec. Inputs, outputs, recipes, decisions, sub-analyses. | -| `.lightcone/Snakefile` | Project (generated) | Auto-generated by `lc run`. Don't edit. | -| `.lightcone/snakefile-config.json` | Project (generated) | Per-`(rule, universe)` config. | -| `.lightcone/lightcone.yaml` | Project | Tiny scratchpad — currently writes only `target: local`. Not consumed by today's code. | -| `~/.lightcone/config.yaml` | User | `container.runtime`. | - -The `dagster.yaml` and `~/.lightcone/targets/*.yaml` files referenced in -older docs are no longer used — historical residue. +Each module's page in [Engine Internals](api/index.md) carries its +public surface and the invariants that bind it. diff --git a/docs/contributing/backends.md b/docs/contributing/backends.md deleted file mode 100644 index 5c651e2d..00000000 --- a/docs/contributing/backends.md +++ /dev/null @@ -1,42 +0,0 @@ -# Adding an Execution Backend (rewritten) - -The `ASTRAContainerRunner` plugin point is gone. Execution is structured -quite differently now, and "adding a backend" decomposes into one or both -of these: - -## Adding a container runtime - -The supported runtimes are `docker`, `podman`, and `podman-hpc` (plus the -`none` no-op). They are listed in -`src/lightcone/engine/container.py::RUNTIMES`. To add a new one: - -1. Append the binary name to `RUNTIMES` (detection priority is the tuple - order). -2. If detection needs a probe (like the docker-daemon ping), extend - `detect_runtime()`. -3. If `wrap_recipe()` needs different flags for the runtime, branch on - `runtime` there. -4. If post-build migration is required (the `podman-hpc migrate` model), - add a `__migrate(tag)` and call it from `build_image()` / - `pull_image()`. -5. Add tests in `tests/test_container.py`. - -## Adding a Dask cluster shape - -Today the cluster manager has three branches: existing scheduler, SLURM -allocation, local. To add a fourth (for example, a custom GPU farm): - -1. Add a branch to `cluster_for_run()` in - `src/lightcone/engine/dask_cluster.py`. -2. Make sure it advertises the same resource keys (`cpus`, `memory`, - `gpus`) so the [Snakemake executor plugin](../api/dask_executor.md) - can match. -3. Add tests in `tests/test_dask_cluster.py`. - -## Adding a non-Snakemake executor - -In principle Snakemake supports multiple executors and we ship one -(`snakemake_executor_plugin_dask`). If you need a different scheduler, -you can write another Snakemake executor plugin and pass it through -`lc run --executor ` — but that flag does not exist today and would -need to be added to `src/lightcone/cli/commands.py::run`. diff --git a/docs/contributing/extending.md b/docs/contributing/extending.md new file mode 100644 index 00000000..ae3312f9 --- /dev/null +++ b/docs/contributing/extending.md @@ -0,0 +1,55 @@ +# Extending the Codebase + +Where each kind of change belongs, what to read first, and the +invariant it must keep. The engine has one implementation per rule — +most review feedback is some form of "that spelling already exists; +use it". + +## The map + +| To change… | Edit | Keep true | +|---|---|---| +| What a scaffolded file contains | `engine/templates/files/*.tmpl` (+ `test_templates.py`) | A template gets a function only when a value must be decided or a merge policy held. | +| What gets converged | `engine/project.py` (+ `test_project.py`) | Everything through `_Converger.item`/`.file`/`.blocked`; repairs only append; only what git can carry. | +| How a project stores bytes | `engine/dataset.py` + `gitattributes.tmpl` (+ `test_dataset.py`, real annex) | Every command through `project._run`; nobody is asked to run git-annex. | +| How an output is identified | `engine/identity.py` (+ `test_identity.py`) | Sensitivity both ways: what must move the hash, what must not. Length-framing stays. | +| When an output is remade | `engine/assets.py` (+ `test_assets.py`) | One `classify`; callers differ by one input value, never by logic. Ask first: does the change *contradict* the project (stale) or is it *circumstance* (behind)? | +| How the spec becomes a graph | `engine/plan.py` (+ `test_plan.py`) | Ask `astra.resolve`; a missing answer is a PR to astra-tools; ambiguity is a `ProjectError`, never a guess. | +| How a recipe runs | `engine/worker.py` (+ `test_worker.py`) | Never raises; no git; mutation-check every denial test. | +| What a run commits | `engine/materialize.py` (+ `test_materialize.py`) | The driver owns git alone; the tree ends as clean as it started. | +| Where a run executes | `engine/venue.py` + `cluster_for_run` (+ `test_venue.py`) | One detection ladder; venues detected, never configured; test by faking the host. | +| Supporting a new HPC center | `venue._SITES` | One row — marker + the center's own `salloc`/`sbatch` spellings, verified against its documentation, never guessed. | +| What a sandboxed command may touch | `sandbox/policy.py` (+ `test_sandbox_policy.py`) | Path sets only — no mechanism leaks in. | +| Adding a sandbox mechanism | one module in `sandbox/` + one line in `detect()` | `wrap` pure, `attest` honest, `contains_prefix` answered. Nothing above the seam changes. | +| A denial message | `sandbox/denial.py` (+ `test_sandbox_denial.py`) | Remedies copy-pasteable and real *today*; the trailer stays unconditional. | +| What the image is made of | `engine/image.py` (+ `test_image.py`) | Pure; every declaration key hashed; structure tests, never byte goldens. | +| How images are built/stored/entered | `engine/container.py` + `sandbox/oci.py` (+ `test_container.py`) | `runtime_for_run`'s two strictnesses; runtime differences are spellings inside `OCIBackend`, never new shapes. | +| What the crate says | `engine/crate.py` (+ `test_crate.py`) | Pure builder: sorted, no clock, git injected; render-twice-identical. The validator floor lives in `test_crate_smoke._FLOOR`. | +| How a foreign write is detected | `dataset.last_writer` + `materialize._foreign_write` | History, never hashing; `datalad_run_subject` is the one spelling of the record's subject. | +| A CLI verb | `cli/commands.py` (+ `test_cli.py`) | Logic in the engine; raise `ProjectError`; render here; engine imports stay inside callbacks. | + +## Rules that apply everywhere + +- **Land code, tests, and dependencies together.** A dependency enters + `pyproject.toml` with the change that needs it, never speculatively. +- **No dead code, no foreshadowing.** Nothing references a verb, flag, + or feature that doesn't exist yet; `lc --help` advertises only what + works. +- **No escape hatches.** Enforcement ships without a flag to turn it + off; there is deliberately no `--no-sandbox`, no `--force`, no + rebuild-the-world flag. +- **Nothing waits on a human.** No prompt, no interactive shell — + either is a hang for the agents that run these verbs most. +- **Refusals carry remedies, and remedies are verified.** A message + that tells someone to run a command has been run; a center's + spellings come from its documentation. +- **Docstrings are Google-style, comments carry *why*.** A design + decision gets a sentence; its history belongs in the design record, + not the code. + +## Conventions + +Ruff (E, F, I, N, W, UP; line length 100), mypy strict with +`namespace_packages = true`. `src/lightcone/` must never gain an +`__init__.py` — the namespace is shared with future sibling +distributions, and a real package there breaks the contract. diff --git a/docs/contributing/hpc-sites.md b/docs/contributing/hpc-sites.md deleted file mode 100644 index 6e30e06f..00000000 --- a/docs/contributing/hpc-sites.md +++ /dev/null @@ -1,24 +0,0 @@ -# Adding an HPC Site - -The old target system is gone; what remains is the lightweight -[`site_registry`](../api/site_registry.md) module, which supplies -per-site defaults (scratch root, preferred container runtime) via -`detect_current_site()`. - -If you want lightcone-cli to behave well on a new cluster, what you -actually need is: - -1. **A container runtime that works on compute nodes.** `podman-hpc` is - the supported case. Wire it up via `~/.lightcone/config.yaml`, or - declare it as the site's `container_runtime` in `SITE_DEFAULTS`. -2. **Dask workers reachable from the scheduler.** `lc run` already does - the right thing inside an `salloc`/`sbatch` allocation — the cluster - manager binds the scheduler to the SLURM canonical hostname and - launches one worker per node via `srun`. See - [api/dask_cluster](../api/dask_cluster.md). -3. **A sane scratch root.** `lc run` keeps its operational state - (snakemake metadata, dask spill, cross-node run locks) under a - scratch root that must honour `flock` — on Perlmutter that means - `$SCRATCH` (Lustre), not DVS-mounted home/CFS. Declare - `scratch_root` in the site's `SITE_DEFAULTS` entry; users can - override it per-project with `lc init --scratch`. diff --git a/docs/contributing/setup.md b/docs/contributing/setup.md index ed11c9c4..c2348966 100644 --- a/docs/contributing/setup.md +++ b/docs/contributing/setup.md @@ -1,92 +1,77 @@ # Development Setup -You'll need: - -- Python 3.11+ -- [uv](https://docs.astral.sh/uv/) — `curl -LsSf https://astral.sh/uv/install.sh | sh` -- [just](https://github.com/casey/just) — `brew install just` or `cargo install just` -- Git -- One of: docker, podman, podman-hpc (optional — only needed for - container tests and for projects that declare `container:`) +Everything runs through [uv](https://docs.astral.sh/uv/); there is no +task runner and no other build tooling. ## Clone & install ```bash git clone https://github.com/LightconeResearch/lightcone-cli.git cd lightcone-cli -just install # uv sync --all-groups (dev + docs) +uv sync --group dev ``` -`just` (alone, with no recipe) lists everything available — the -recipes that follow are the ones you'll touch most. +That resolves the engine and the dev tools (pytest, ruff, mypy, +datalad, the rocrate validator) into `.venv`. `uv run lc --version` +runs the checkout's `lc`. + +You also need `git` on `PATH` (the one tool uv cannot install); +git-annex arrives as a wheel with the sync. -## Running the test suite +## The loop ```bash -just test # uv run pytest -just test-cov # with coverage report +uv run pytest # the suite +uv run ruff check src/ tests/ # lint (--fix to apply) +uv run mypy src/ # strict mode ``` -The opt-in `slow` marker covers tests that spin up real subsystems -(local Dask cluster, etc.). They are excluded by default; run with -`uv run pytest -m slow` to include them. +These three are exactly what CI runs (`tests.yml`, `lint.yml`) — green +locally means green there, modulo the gated suites below. -## Linting & types +Most of the suite is hermetic: an autouse fixture stubs the engine's +one subprocess seam, so tests spawn nothing and touch no network. The +exceptions opt in explicitly — see [Testing](testing.md). -```bash -just lint # ruff + mypy -just fix # ruff --fix -just fmt # ruff format -``` +### The gated suites -Ruff rules: `E, F, I, N, W, UP`. Line length: 100. Target: Python 3.11. -Mypy is strict, with `namespace_packages = true` and -`explicit_package_bases = true` (we ship a PEP 420 namespace package). +Three suites answer questions only a real mechanism can, and each +skips where its mechanism is missing — with an environment variable CI +sets to turn the skip into a hard failure: -## Building the docs locally +| Variable | Suite | Needs | +|---|---|---| +| `LC_SANDBOX_TESTS_REQUIRED=1` | `test_sandbox_enforcement.py` | Landlock (Linux) or Seatbelt (macOS) | +| `LC_CONTAINER_TESTS_REQUIRED=1` | `test_container_smoke.py` | podman or docker | +| `LC_CRATE_TESTS_REQUIRED=1` | `test_crate_smoke.py` | nothing beyond dev deps | + +## Building the docs ```bash -just docs-serve # syncs docs group + live preview at http://127.0.0.1:8000 -just docs-strict # build with --strict -just docs # one-shot build into site/ +uv sync --group docs +uv run zensical build # renders into site/ +uv run zensical serve # live preview ``` -The docs use [zensical](https://zensical.org). The nav lives in -`zensical.toml`. +The site deploys on release (`docs-deploy.yml`), so docs track the +released CLI, not `main`. ## Building the wheel ```bash -just build # uv build -just version # current version (from git tags via hatch-vcs) -``` - -The wheel ships two packages: - -```toml -[tool.hatch.build.targets.wheel] -packages = ["src/lightcone", "src/snakemake_executor_plugin_dask"] -``` - -## Repo layout - -```text -src/lightcone/ # main namespace (PEP 420; no __init__.py at the package root) -src/snakemake_executor_plugin_dask/ # Snakemake → Dask executor plugin -tests/ # pytest tree, mirrors src/ -evals/ # agentic eval: prompt.md + task seeds (tasks/snae/) -docs/ # docs site +uv build ``` -## Pre-commit checklist +CI runs this only to publish. The version comes from hatch-vcs — the +git tag for a release, tag-plus-commit for a dev build — which is also +what lets a run record pin a dev engine by its source commit. -Quick sequence before pushing a PR: - -```bash -just lint # ruff + mypy -just test # full pytest run -just docs-strict # docs still build cleanly -``` +## Pre-PR checklist -Each line maps to one CI check. CI runs them serially; running locally -catches everything before the PR machinery starts. +1. `uv run pytest` — including, if your change touches the sandbox, + containers, or the crate, the relevant gated suite on a host that + can run it. +2. `uv run ruff check src/ tests/` and `uv run mypy src/`. +3. New behavior lands with its tests, in the same PR. +4. Read [Extending](extending.md) — it says where each kind of change + belongs, and the invariants it must keep. diff --git a/docs/contributing/testing.md b/docs/contributing/testing.md index e25b165a..8143fac4 100644 --- a/docs/contributing/testing.md +++ b/docs/contributing/testing.md @@ -1,85 +1,77 @@ # Testing -## Test layout - -```text -tests/ -├── conftest.py # shared fixtures -├── test_cli.py # Click CliRunner integration tests -├── test_container.py # detection, image tag, build_image, wrap_recipe, RuntimeChoice -├── test_dask_cluster.py # cluster_for_run branches & resource keys -├── test_dask_plugin.py # snakemake_executor_plugin_dask -├── test_eval_tasks.py # eval task seed specs validate against astra -├── test_manifest.py # write_manifest, sha256_dir, code_version -├── test_snakefile.py # generator + final `snakemake -n` parse test -├── test_status.py # OutputStatus across ok/stale/missing/alias -├── test_tree.py # collect_tree_outputs, find_upstream_output, … -├── test_validation.py # validate_output across metric/table/figure types -└── test_verify.py # verify_outputs across all three failure kinds -``` - -Tests mirror `src/` 1:1 — when you add a module, add a test file at the -matching path. - -## Common patterns - -### CLI tests (Click `CliRunner`) - -```python -from click.testing import CliRunner -from lightcone.cli.commands import main - -def test_init_creates_structure(tmp_path): - runner = CliRunner() - result = runner.invoke(main, ["init", str(tmp_path / "myproject"), "--no-git", "--no-venv"]) - assert result.exit_code == 0 - assert (tmp_path / "myproject" / "astra.yaml").exists() -``` - -### End-to-end against a tmp project - -`test_status.py`, `test_verify.py`, and `test_snakefile.py` build a -minimal ASTRA project under `tmp_path` (one `astra.yaml`, one -`universes/baseline.yaml`, optional sub-analyses), then run the -function under test. Helpers: - -- `astra.helpers.load_yaml` / `resolve_analysis_tree` mirror what - production code does. -- `lightcone.engine.snakefile.generate(project, universes=[...], runtime="none")` - for tests that need an actual Snakefile. - -### Snakefile parsing - -`tests/test_snakefile.py` ends with a parse test that runs -`snakemake -n -s ` to confirm the generator -produces a Snakefile the upstream tool actually accepts. Add a similar -assertion when changing rule shape. - -### Slow tests - -```bash -uv run pytest -m slow # opt in to the slow tests -``` - -The `slow` marker is reserved for tests that start a real Dask cluster. -Do not use it for things that are merely a bit chatty — prefer trimming -test scope. - -## Eval harness (separate) - -The agentic eval is a plain GitHub Actions workflow — -`.github/workflows/eval.yml` — with no Python harness behind it. On -each PR it scaffolds a project with `lc init`, overlays the seed files -from `evals/tasks/snae/` (`astra.yaml`, `data/`), runs Claude Code -headlessly with `evals/prompt.md` (the astra skill is installed from -the `LightconeResearch/agent-skills` plugin marketplace), and then -checks the outcome with `astra validate` and `lc status --json` — the -job fails unless every declared output is materialized. Run metrics -(turns, tool calls, cost, wall time) are extracted from the transcript -by `.github/scripts/trace_digest.py` and posted as a sticky PR comment -and job summary. Two artifacts are uploaded: `agent-trace` (the raw -stream-json transcript plus a human-readable markdown digest) and -`eval-project` (the built project with its provenance manifests). - -To reproduce locally, run the same commands the workflow does with -`claude`, `lc`, and `astra` on PATH. +The suite's shape follows the engine's: pure modules get pure tests, +the subprocess seam gets a stub, and the questions only a kernel, a +runtime, or a validator can answer get real ones — gated so they can't +pass by not running. + +## The one seam + +`tests/conftest.py`'s autouse `tools` fixture stubs +`engine.project._run` — the single choke point every external command +goes through — emulating each tool's observable effect (`uv lock` +writes `uv.lock`, `git init` makes `.git`, …) and recording every +argv. Under the stub the suite is hermetic: no network, no resolution, +no subprocesses. + +The `real_tools` fixture opts back out, putting the real `_run` back. +Everything built on it (the `analysis` fixture, the rerun tests) does +spawn and may touch the network — that is the deliberate price of +testing execution. + +## Where a question belongs + +| Question | File | Character | +|---|---|---| +| Convergence semantics | `test_project.py` | stubbed | +| Template content & repair | `test_templates.py` | pure | +| Do bytes land in the annex? | `test_dataset.py` | **real tools** — every bug this seam had was invisible to a stub | +| Identity sensitivity | `test_identity.py` | pure, both directions | +| The graph, the gate | `test_plan.py` | pure — tests what lc *adds*, never what a spec means (that's astra-tools' suite) | +| Classification | `test_assets.py` | pure | +| One output, real recipe | `test_worker.py` | real boundary, real repo | +| The run, the record | `test_materialize.py` | real repos; one real `LocalCluster`; real `datalad rerun` | +| Venue detection & launch | `test_venue.py` | fakes the *host* (env vars, a stub srun), never the code | +| Policy / wrap / denial | `test_sandbox_*.py` | pure, run on every OS | +| The kernel's answer | `test_sandbox_enforcement.py` | gated | +| Image identity | `test_image.py` | pure — structure and ordering, never byte goldens | +| Runtime lifecycle | `test_container.py` | stubbed; refusals asserted on recorded argv | +| The runtime's answer | `test_container_smoke.py` | gated | +| The crate | `test_crate.py` | pure; the one byte claim is render-twice-identical | +| The validator's answer | `test_crate_smoke.py` | gated | +| CLI surface | `test_cli.py` | `CliRunner`; assert short unwrappable fragments | + +## The enforcement suite + +`test_sandbox_enforcement.py` is the only file that can tell you the +sandbox works, and four properties keep it honest: + +1. **One suite, both mechanisms** — parameterized by `detect()` alone; + a leak only Linux catches is a leak, and macOS CI is the sole place + the generated SBPL ever executes. +2. **The real policy** — always `exec_policy`, never one hand-built to + make the point. (`/usr` once sat in the exec set through a fully + green suite built the other way.) +3. **Real leaks, tried literally** — undeclared tools executed, + undeclared libraries `dlopen`ed, undeclared data read. +4. **It cannot pass by not running** — `LC_SANDBOX_TESTS_REQUIRED=1` + in CI turns the skip into a failure, and two tests cover the guard + itself. + +**Mutation-check every denial test**: run the same command through +`Unavailable()` and confirm it *succeeds*. A denial test that would +pass unsandboxed is testing nothing, and the failure mode is silent. +Two related traps: a write-denial must target a path the OS would let +you write (a `/etc` write pins nothing), and enforcement fixtures must +not live under `/tmp`, which is inside the write baseline — the +`outside` fixture roots at `$HOME` for exactly this reason. + +## Conventions + +- Don't add a flag whose only user is a test — stub `project._run` + instead. +- A forged-output test must break the annex hard link before writing + (`test_materialize._forge` shows how) — results are committed thin, + so an in-place write dirties every byte-identical sibling. +- Record formats are tested through their consumer (datalad's parser, + the rocrate validator), never as golden files of our own JSON. diff --git a/docs/hpc/containers.md b/docs/hpc/containers.md deleted file mode 100644 index 786dc86d..00000000 --- a/docs/hpc/containers.md +++ /dev/null @@ -1,52 +0,0 @@ -# Container Builds for HPC - -HPC nodes generally cannot reach a docker daemon, so lightcone-cli ships -support for `podman-hpc` (NERSC Perlmutter and friends). The build/migrate -workflow is owned by `lightcone.engine.container`. - -## podman-hpc workflow - -`podman-hpc` is rootless and HPC-aware. After a build, the image must be -*migrated* into the per-node container cache so compute nodes can read it -without a registry. - -```bash -# On a login node, with podman-hpc on PATH: -lc setup # writes ~/.lightcone/config.yaml -$EDITOR ~/.lightcone/config.yaml # set container.runtime: podman-hpc -lc build # builds + migrates each image -``` - -`lc build` checks for cached tags and skips rebuilds. Use `--force` to -rebuild everything. - -## Tag computation - -Tags are content-addressed: - -``` -lc-- -``` - -The hash covers the Containerfile contents plus any of these dependency -files found at the project root: -`requirements.txt`, `requirements-dev.txt`, `requirements-test.txt`, -`pyproject.toml`, `setup.py`, `setup.cfg`, `poetry.lock`, `Pipfile.lock`. - -## At run time - -`lc run` does **not** re-shell into Snakemake's `container:` directive or -`--sdm apptainer`. The Snakefile generator wraps each rule's recipe in: - -```bash -podman-hpc run --rm --pull=never -v "$PWD":"$PWD" -w "$PWD" \ - bash -c '' -``` - -`--pull=never` is critical: short-name resolution would otherwise try -`unqualified-search-registries` for tags like `lc-myproject-abc123` and -fail. Pre-pulling registry images via `lc build` (or pre-staging -Containerfile images via `lc build`) is therefore mandatory. - -See also: [api/container](../api/container.md) for the implementation, -and [`lc build`](../cli/build.md) for the user-facing command. diff --git a/docs/hpc/index.md b/docs/hpc/index.md deleted file mode 100644 index 672e3fc1..00000000 --- a/docs/hpc/index.md +++ /dev/null @@ -1,17 +0,0 @@ -# HPC & SLURM (consolidated) - -The standalone HPC subsystem (target files, site registry, sbatch -generation) is gone. SLURM execution is now handled by Dask: when `lc run` -is invoked inside an existing SLURM allocation, the cluster manager -launches one `dask worker` per allocated node via `srun` and Snakemake -dispatches each rule across them. - -For the user-facing flow, see [Running on a Cluster](../user/cluster.md). - -For maintainer detail: - -- [api/dask_cluster](../api/dask_cluster.md) — the three-branch decision - (existing scheduler / SLURM allocation / local). -- [api/dask_executor](../api/dask_executor.md) — the Snakemake executor - plugin that turns each rule into a `client.submit(...)` call. -- [api/container](../api/container.md) — `podman-hpc` build & migrate. diff --git a/docs/hpc/site-registry.md b/docs/hpc/site-registry.md deleted file mode 100644 index 49dd39e7..00000000 --- a/docs/hpc/site-registry.md +++ /dev/null @@ -1,6 +0,0 @@ -# Site Registry (orphaned) - -The `lightcone.engine.site_registry` module still exists but is not imported -by any active code path. It carries Perlmutter scheduler defaults that used -to feed the wizard for the (now removed) target system. See -[api/site_registry](../api/site_registry.md) for the current state. diff --git a/docs/hpc/targets.md b/docs/hpc/targets.md deleted file mode 100644 index 2db4548c..00000000 --- a/docs/hpc/targets.md +++ /dev/null @@ -1,3 +0,0 @@ -# Target Configuration (removed) - -The per-machine target system is gone. See [`lc target`](../cli/target.md). diff --git a/docs/maintainer.md b/docs/maintainer.md index 0cbdd3c6..fdfac4a5 100644 --- a/docs/maintainer.md +++ b/docs/maintainer.md @@ -1,24 +1,27 @@ # Developer corner -`lightcone-cli` is a thin shim over Snakemake that owns provenance. This guide -covers everything below the user surface: how the execution and integrity layers -work, what each engine module does, and how to get a working dev loop. +`lightcone-cli` is a small engine with strong opinions: one way to +identify an output, one way to store it, one boundary to execute it +behind. This guide covers everything below the user surface — how the +engine is put together, what each module owns, and how to get a +working dev loop. If you're looking for the user-facing docs, the [user guide](user/index.md) is the other half of this site. ## What this covers -- [Architecture](architecture.md) — the three subsystems (Snakefile generation, - manifest layer, cluster management) and the invariants that hold them together. -- [CLI Reference](cli/index.md) — every `lc` command: flags, options, and the - exact Snakemake invocation each one triggers. -- [Python API](api/index.md) — the `lightcone.engine.*` modules: public - signatures, common entry points, and module responsibilities. -- [HPC & SLURM](hpc/index.md) — how the Dask cluster manager adapts to local, - SLURM, and external schedulers. -- [Contributing](contributing/setup.md) — clone, install, run the test suite, - lint, and build the docs locally. +- [Architecture](architecture.md) — the CLI/engine/ASTRA split, the + run pipeline, identity, storage, the exec boundary, and the + invariants that hold them together. +- [CLI Reference](cli/index.md) — every `lc` command: flags, JSON + report shapes, exit codes. +- [Engine Internals](api/index.md) — the `lightcone.engine.*` + modules: what each owns, its key symbols, and what must stay true + of it. +- [Contributing](contributing/setup.md) — clone, install, run the + test suite; [how the suite is shaped](contributing/testing.md); and + [where a change belongs](contributing/extending.md). ## Get started in three commands @@ -27,18 +30,29 @@ If you're looking for the user-facing docs, the ```bash git clone https://github.com/LightconeResearch/lightcone-cli.git cd lightcone-cli - just install # uv sync --all-groups - just test # pytest + uv sync --group dev && uv run pytest ``` - Run `just` with no arguments to see all available recipes. - -## What lightcone-cli *owns* - -The codebase is intentionally small. Snakemake handles DAG construction, -parallelism, cluster submission, staleness detection, locking, and log capture — -we do not replicate any of that. The parts that are ours: - -- **Snakefile generator** — translates `astra.yaml` into `.lightcone/Snakefile`. -- **Manifest layer** — writes and verifies `.lightcone-manifest.json` per output. -- **Cluster manager** — picks local / SLURM / external Dask shape at runtime. +Test, lint (`uv run ruff check src/ tests/`) and type-check +(`uv run mypy src/`) are the whole loop — there is deliberately no +task runner in between. + +## The house rules + +A few conventions run through every module; changes are reviewed +against them: + +- **No dead code, no foreshadowing.** Nothing lands before the layer + that calls it, and no message names a verb or flag that doesn't + exist yet. `lc --help` advertises only what works. +- **No escape hatches around guarantees.** A feature that enforces + something ships without a flag to turn the enforcement off. +- **Literal behavior over invented convenience.** The current + directory is the project; erroring beats walking up or guessing. + Nothing prompts — a verb is run by an agent more often than a + person, and a prompt is a hang. +- **One implementation per rule.** Classification, path naming, the + run-record subject, tool resolution — each has exactly one spelling, + and a second copy is where the two start to disagree. +- **Honest reporting.** What was enforced, what was skipped, and what + a clone can't see are all recorded or said — never assumed. diff --git a/zensical.toml b/zensical.toml index 8178b33a..cea1338e 100644 --- a/zensical.toml +++ b/zensical.toml @@ -30,23 +30,24 @@ nav = [ {"lc run" = "cli/run.md"}, {"lc build" = "cli/build.md"}, ]}, - {"Python API" = [ + {"Engine Internals" = [ {"Overview" = "api/index.md"}, - {"cli/commands" = "api/cli.md"}, - {"engine/manifest" = "api/manifest.md"}, - {"engine/snakefile" = "api/snakefile.md"}, - {"engine/container" = "api/container.md"}, - {"engine/cloudbuild" = "api/cloudbuild.md"}, - {"engine/status" = "api/status.md"}, - {"engine/verify" = "api/verify.md"}, - {"engine/tree" = "api/tree.md"}, - {"engine/validation" = "api/validation.md"}, - {"engine/dask_cluster" = "api/dask_cluster.md"}, - {"snakemake_executor_plugin_dask" = "api/dask_executor.md"}, + {"project" = "api/project.md"}, + {"dataset" = "api/dataset.md"}, + {"identity" = "api/identity.md"}, + {"plan" = "api/plan.md"}, + {"assets" = "api/assets.md"}, + {"worker" = "api/worker.md"}, + {"materialize" = "api/materialize.md"}, + {"venue" = "api/venue.md"}, + {"sandbox" = "api/sandbox.md"}, + {"image & container" = "api/container.md"}, + {"crate" = "api/crate.md"}, ]}, {"Contributing" = [ {"Development Setup" = "contributing/setup.md"}, {"Testing" = "contributing/testing.md"}, + {"Extending" = "contributing/extending.md"}, ]}, ]}, {"ASTRA docs" = "https://astra-spec.org/latest/"},