diff --git a/docs/cli/build.md b/docs/cli/build.md index d53445cf..f4603a20 100644 --- a/docs/cli/build.md +++ b/docs/cli/build.md @@ -1,7 +1,9 @@ # lc build -Build container images declared in `astra.yaml` (or pre-pull registry -images so `lc run` can use `--pull=never`). +Build the project's system-layer image, and commit it. Containerized +projects only — a project containerizes by declaring a +`[tool.lightcone.image]` table in `pyproject.toml`, and on a direct +project this verb just says so and exits. ## Synopsis @@ -9,82 +11,80 @@ images so `lc run` can use `--pull=never`). lc build [OPTIONS] ``` -## Options +Idempotent: an image that is already built and committed is left +alone. -| Option | Default | Effect | -|--------|---------|--------| -| `--force` | off | Rebuild / re-pull even if the tag already exists locally. | -| `--runtime {docker,podman,podman-hpc,kubernetes}` | resolved from `~/.lightcone/config.yaml` | Override the runtime for this build. | - -## What it does - -For every distinct `container:` value found in the project (root, -sub-analysis, or recipe-level): - -- **Path to a Containerfile** → compute the content-addressed tag - `lc--`, build the image, and (for `podman-hpc`) - migrate it into the per-node container cache. -- **Anything else** (e.g. `python:3.12-slim`, `ghcr.io/foo/bar:tag`) → - pull it into the local image store. This is what lets `lc run` pass - `--pull=never` to the runtime, sidestepping `unqualified-search-registries` - resolution issues with content-addressed tags. - -On the `kubernetes` runtime (a lightcone JupyterHub deployment, where -no local OCI runtime exists) the same command builds through the -deployment's **GCP Cloud Build** service instead: the staged build -context is uploaded to the deployment's build bucket and the resulting -image is pushed as `$LIGHTCONE_REGISTRY/lc-:` — -the same content-addressed identity, so an unchanged environment is a -single registry check and no build at all. Pre-built registry images -are left alone (worker pods pull them directly). Auth is the pod's -Workload Identity; nothing to configure. - -If the runtime is `none` (either by config or because `auto` couldn't -find one), `lc build` prints a friendly note and exits 0. There is -nothing to build. - -## Tag computation +## What the image is -```text -lc-- +The image is the *system layer* only: the declared base (digest-pinned, +or the default), the declared apt packages, and the pinned Python +interpreter. Your analysis environment is not in it — recipes' Python +packages come from the project's lock, synced into the container at run +time — and neither is `lc` itself. That is what makes "editing code +never rebuilds the image" structural: no project file enters the build +context at all. + +The declaration is a closed set of keys, hashed into the image's +identity: + +```toml +[tool.lightcone.image] +base = "docker.io/library/debian@sha256:..." # optional; default pinned by lc +apt-install = ["libfftw3-dev"] # optional +run-commands = ["curl -L ... | tar xz"] # optional, the bounded escape +env = { OMP_NUM_THREADS = "1" } # optional ``` -The hash covers the Containerfile contents plus any of these dependency -files at the project root: +## The archive is the store -- `requirements.txt` -- `requirements-dev.txt` -- `requirements-test.txt` -- `pyproject.toml` -- `setup.py` -- `setup.cfg` -- `poetry.lock` -- `Pipfile.lock` +`lc build` saves the built image into the repository — +`.datalad/environments//image`, a `docker-archive` committed +through git-annex — so the exact bytes travel with the project: a +clone obtains them with a fetch, no registry and no credentials +involved. Execution always pins the image's content *id*, never a tag, +so nothing can substitute a different image under the same name. -Edit any one of those and the tag changes. That, in turn, changes -`code_version` in every recipe that uses the image, which marks all -downstream outputs `stale` in `lc status`. +The archive records the architecture it was built for, and a host that +can't execute that architecture is refused up front — build where the +architecture matches the machines that will run recipes (on NERSC, a +login node). -## Examples +## Requirements -```bash -lc build # build / pull whatever's missing -lc build --force # rebuild / re-pull everything -lc build --runtime podman-hpc # force the HPC runtime -``` +- A clean tree — the image commit must not sweep your staged edits in, + and the tag derives from the committed declaration. +- A build-capable runtime: `podman-hpc`, `podman`, or `docker` + (detected in that order; nothing to configure). -## Pre-staging for HPC +`lc materialize` also builds as a preflight when the committed +declaration has no image yet, announcing it first — `lc build` exists +so you can pay the minutes when *you* choose to. -On a login node: +## Options -```bash -$EDITOR ~/.lightcone/config.yaml # container.runtime: podman-hpc -lc build # builds + migrates everything +| Option | Default | Effect | +|--------|---------|--------| +| `--json` | off | Emit the result as JSON on stdout. | + +## The JSON result + +```json +{ + "mode": "containerized", + "tag": "lc-env-1a2b3c4d5e6f7a8b", + "id": "sha256:...", + "archive": ".datalad/environments/lc-env-1a2b3c4d5e6f7a8b/image", + "action": "built" +} ``` -Then submit a SLURM job for `lc run`. The compute nodes will find every -image already cached. +`action` is `"built"` when this invocation built and committed the +image, `"present"` when it was already there. On a direct project the +result is just `{"mode": "direct"}`. -See [api/container](../api/container.md) for the implementation and -[Architecture](../architecture.md) for why we wrap recipes ourselves -instead of using Snakemake's `container:` directive. +## Examples + +```bash +lc build # build + commit, or confirm it's already there +lc build --json # the machine-readable form +``` diff --git a/docs/cli/export.md b/docs/cli/export.md deleted file mode 100644 index 49ecc00e..00000000 --- a/docs/cli/export.md +++ /dev/null @@ -1,106 +0,0 @@ -# lc export - -Export project artifacts in interoperable formats. Currently the only -exporter is `wrroc` (Workflow Run RO-Crate); the group is shaped to host -future formats without breaking the CLI surface. - -## Synopsis - -```text -lc export wrroc [OPTIONS] -``` - -## lc export wrroc - -Walk the project's per-output `.lightcone-manifest.json` sidecars and -emit a [Workflow Run RO-Crate](https://www.researchobject.org/workflow-run-crate/) -bundle — a JSON-LD package readable by WorkflowHub, Zenodo's RO-Crate -plugin, and any RO-Crate-aware archive. The on-disk manifest format is -unchanged; the bundle is the *publication view*, generated on demand. - -### Options - -| Option | Default | Effect | -|--------|---------|--------| -| `--output`, `-o PATH` | `./wrroc` | Bundle directory, or `.zip` path when `--zip` is set. | -| `--universe`, `-u NAME` | every universe with materialized outputs | Restrict to specific universes. Repeatable. | -| `--author "NAME "` | git `user.name` / `user.email`, then `LIGHTCONE_AUTHOR` env | Override the author recorded in the bundle. | -| `--license URL` | `https://creativecommons.org/licenses/by/4.0/` | License URL or SPDX identifier for the bundle. Required by the WRROC profile. | -| `--zip` / `--no-zip` | `--no-zip` | Package the bundle as a single `.zip` after building. | -| `--metadata-only` | off | Skip data files; bundle manifests + `astra.yaml` + universe files only. | - -### What gets bundled - -- `astra.yaml` → `ComputationalWorkflow` (`programmingLanguage: snakemake`). -- Each materialized output → `Dataset` with `version = data_version`. -- Each recipe execution → `CreateAction` with `object` (inputs, both upstream datasets via stable `@id` and external files), `result` (the produced dataset), and `instrument` (the recipe `SoftwareApplication`). -- Each container → `SoftwareApplication` referenced via `softwareRequirements`. -- Each decision → `FormalParameter` + per-run `PropertyValue`. -- Author → `Person`. - -Provenance metadata (`code_version`, `data_version`, `git_sha`, `lc_version`, host) lands as `PropertyValue` entries on the relevant entities. - -### Examples - -```bash -lc export wrroc # ./wrroc/ directory -lc export wrroc -o run.zip --zip # zip bundle for upload -lc export wrroc --metadata-only # provenance graph only, no data files -lc export wrroc -u baseline -u alt_method # restrict to specific universes -lc export wrroc --author "Ada Lovelace " -lc export wrroc --license https://opensource.org/licenses/MIT -``` - -### Output - -```text -✓ Wrote WRROC directory: ./wrroc - Captured 7 runs across universes: baseline, alt_method -``` - -If no materialized outputs are found, the bundle still writes — but only -contains the workflow definition, and a warning is printed: - -```text -✓ Wrote WRROC directory: ./wrroc -Warning: no materialized outputs were found — the bundle contains only - the workflow definition. - This usually means recipes haven't been run yet (try lc run) or the - .lightcone-manifest.json sidecars are missing. - Workflow-only bundles will not pass strict Provenance Run Crate - validation; that profile requires at least one materialized run. -``` - -### Failure modes - -| Error | Cause | -|---|---| -| `No astra.yaml at ; cannot export.` | The cwd is not inside an ASTRA project. | -| ` is non-empty; refuse to clobber.` | The target directory already has contents. Pass a fresh path or remove the existing one. | -| ` is an existing directory; cannot overwrite with a zip.` | `--zip` was requested but the output path resolves to a directory. Use a file path like `bundle.zip`. | - -Manifests that exist but are unreadable (e.g. cross-user symlinks under -`results/` with permission denied) are warned about and skipped — they -do not abort the export. - -### Validation - -The bundle conforms to the [Provenance Run Crate 0.5](https://w3id.org/ro/wfrun/provenance/0.5) -profile. To validate locally: - -```bash -pip install git+https://github.com/crs4/rocrate-validator.git -rocrate-validator -y validate ./wrroc/ -``` - -### When to run - -- Before submitting a paper or depositing artifacts in Zenodo / WorkflowHub. -- After a clean run (`lc verify` clean) on the final commit you intend to publish. -- For external collaborators who don't have `lc` installed but need to inspect provenance. - -### Related - -- [`lc verify`](verify.md) — confirm the manifest chain is intact before exporting. -- [`lc status`](status.md) — see which outputs will be captured by the export. -- [api/manifest](../api/manifest.md) — the on-disk format the export reads from. diff --git a/docs/cli/index.md b/docs/cli/index.md index 889ebd36..eda2f2ec 100644 --- a/docs/cli/index.md +++ b/docs/cli/index.md @@ -7,21 +7,27 @@ and audit it. ## Global behavior -- `~/.lightcone/config.yaml` is created automatically on first use of - any `lc` command. You do not need to create it manually. -- All commands except `init` walk up from the cwd looking for - `astra.yaml`. If none is found, the command errors out. +- **The current directory is the project.** Every command except + `init` assumes it is invoked from the project root; there is no + walk-up and no global configuration. Outside a project, a command + errors cleanly. +- **Nothing waits on a human.** No command prompts or opens an + interactive shell — every verb runs to completion on its arguments + alone, which is what makes the CLI safe to drive from scripts and + agents. +- **Refusals carry their remedy.** When a command refuses (a dirty + tree, a login node, a missing image), the message names the exact + command that fixes it. ## Commands | Command | Purpose | |---------|---------| -| [`lc init`](init.md) | Scaffold a new ASTRA project (`astra.yaml`, `Containerfile`, `.lightcone/`, MyST report template, optional venv & git). | -| [`lc run`](run.md) | Generate the Snakefile and dispatch through Snakemake + Dask. | -| [`lc build`](build.md) | Build container images declared in `astra.yaml`. | -| [`lc status`](status.md) | Manifest-driven status report. No Snakemake import needed. | -| [`lc verify`](verify.md) | Recompute hashes, walk the input chain, surface tampering. | -| [`lc export`](export.md) | Emit interoperable bundles (Workflow Run RO-Crate) for publication. | +| [`lc init`](init.md) | Converge a directory into a Lightcone project (idempotent). | +| [`lc materialize`](materialize.md) | Make the analysis's outputs; commit each one as it lands. | +| [`lc status`](status.md) | Report the state of every output. Reads only; always exits 0. | +| [`lc run`](run.md) | Run an ad-hoc command in the project environment, under isolation. | +| [`lc build`](build.md) | Containerized projects: build the image and commit it. | ## Global options @@ -33,7 +39,15 @@ Options: --help Show this message and exit. ``` -## Removed commands +## Exit codes -For historical context: `lc dev`, `lc setup`, `lc target`, and `lc update` no -longer exist as explicit commands. See the removal pages for details. +- `0` — the command did what it says. +- `1` — a refusal or a failure, with the reason on stderr. For + `lc materialize --check` and `lc init --check`, exit 1 means "work + would be done" — the gate form scripts branch on. +- `lc run` is a proxy: it exits with the command's own code + (`128 + N` for a signal), so pipelines read it exactly as they would + the bare command. + +Every verb with a report takes `--json` for the machine-readable form; +each verb's page shows its shape. diff --git a/docs/cli/init.md b/docs/cli/init.md index 58feff94..e6429b73 100644 --- a/docs/cli/init.md +++ b/docs/cli/init.md @@ -1,8 +1,8 @@ # lc init -Converge a directory into an ASTRA project. Idempotent — safe to run -at any time, on an empty directory, a half-scaffolded one, or an -existing project. +Converge a directory into a Lightcone project. Idempotent — safe to run +at any time, on an empty directory, a half-scaffolded one, an existing +project, or a fresh clone. ## Synopsis @@ -19,65 +19,66 @@ manages, and never overwrites files you own: - **Created if missing** — every item in the tree below. A directory that already holds an `astra.yaml` is *adopted*: the spec is left - untouched and only the missing lightcone pieces are added. -- **Repaired** — the managed `.gitignore` block (appended exactly once, - keyed on its `# lightcone-cli` marker), and the stored scratch root - when `--scratch` is passed and differs from the project config. -- **Warned about** — problems `lc` can see but must not fix, reported - in the `warnings` list: an unsupported directory `COPY` in your - Containerfile, an unparseable `.lightcone/lightcone.yaml`. Warnings - don't affect the exit code. -- **Never touched** — anything you authored. - -`--check` reports what a run *would* create or repair, writes nothing, -and exits `1` when the project is not converged. `--json` prints the -report as machine-readable JSON: + untouched and only the missing lightcone pieces are added. A + directory inside an existing git repository adopts that repository + rather than nesting a new one. +- **Repaired** — derived artifacts that have drifted: a `uv.lock` that + no longer matches `pyproject.toml`, a `.venv` that no longer matches + the lock, a managed `.gitignore` or `.gitattributes` entry that a + newer `lc` added. Repairs only ever append or rebuild derived state; + hand-written lines are never reordered or removed. +- **Blocked** — something convergence can see but must not fix by + appending: a `.gitignore` rule that would silently swallow + `results/`, a `.gitattributes` whose ordering would misroute storage. + A blocked item names the file and line at fault, counts against + convergence, and is yours to resolve. +- **Warned about** — advisory facts (e.g. uv falling back to file + copies across filesystems). Warnings never affect the exit code. + +`--check` computes the same report without writing anything and exits +`1` when the project is not converged. `--json` prints it +machine-readable: ```json { - "converged": false, - "created": ["Containerfile"], - "repaired": [".gitignore"], - "unchanged": ["astra.yaml", "..."], + "converged": true, + "created": [], + "repaired": [], + "unchanged": ["astra.yaml", "pyproject.toml", "..."], + "blocked": [], "warnings": [] } ``` -Agents driving a project should run `lc init` (or `lc init --check ---json`) at the start of a session to make sure the directory is -workable. +Agents driving a project should run `lc init --check --json` at the +start of a session to make sure the directory is workable. ## What it creates -The spec scaffold follows the `astra init` boilerplate -(`astra.yaml`, `universes/baseline.yaml`), with the -lightcone-specific pieces layered on top. Inside `DIRECTORY` -(creating it if needed): +Inside `DIRECTORY` (creating it if needed): ```text -astra.yaml # tiny boilerplate spec with one example output +astra.yaml # boilerplate spec with one example output universes/ baseline.yaml # the default universe -Containerfile # project image; referenced by `container:` in astra.yaml -requirements.txt # analysis dependencies (numpy, pandas to start) -.gitignore # Python + lightcone state + MyST build output -.lightcone/ - lightcone.yaml # project config: { target: local } (+ scratch_root if --scratch) -results/ - README.md # the materialization contract; outputs land here via `lc run` -myst.yml # MyST report configuration (MySTRA plugin) -index.md # template report referencing astra.yaml elements -.venv/ # Python venv with the analysis dependencies (skipped with --no-venv) +pyproject.toml # the uv project — the environment's source of truth +.python-version # the exact interpreter, pinned +uv.lock # derived: converged by correctness, not existence +.venv/ # derived: built from the lock (local, never committed) +.gitignore # managed entries, converged line-wise +.git/ # a git repository, with git-annex initialized +.gitattributes # the storage policy: what the annex carries +.datalad/config # dataset identity (a DataLad dataset from birth) +data/ + README.md # declared input data lives here +results/ + README.md # outputs land here — lc's to write +myst.yml # MyST report configuration +index.md # template report referencing astra.yaml ``` -The boilerplate `container: python:3.12-slim` from the astra -boilerplate is rewritten to `container: Containerfile`, so the project -builds its own content-addressed image and dependencies can evolve -under `lc build`. - -On a known site (NERSC Perlmutter, a lightcone JupyterHub), `lc init` -also prints the detected site and the scratch root that `lc run` will -use for its operational state. +Two things it deliberately does *not* create: a `src/` directory +(where analysis code lives is your layout, and git doesn't track empty +directories), and any dependency in `pyproject.toml` — the lock +carries only what *your* analysis imports, added with `uv add`. ## Options @@ -85,21 +86,17 @@ use for its operational state. |--------|---------|--------| | `--check` | off | Report drift without writing; exit 1 if not converged. | | `--json` | off | Emit the convergence report as JSON on stdout. | -| `--no-git` | off | Skip `git init`. | -| `--no-venv` | off | Skip venv creation (`uv venv` if available, else `python -m venv`). | -| `--scratch ` | site default | Scratch root for snakemake state, dask spill, and run locks. Shell expressions like `$SCRATCH` are kept verbatim and expanded at run time. | -> The historical `--target`, `--existing-project`, `--sub-analysis`, -> and `--permissions` flags have been removed. +There is deliberately nothing else — no `--no-git`, no template +selection. The project layout is the contract the other verbs rely on. ## Examples ```bash -lc init # converge cwd -lc init my-analysis # scaffold/converge ./my-analysis -lc init my-analysis --no-git --no-venv # bare bones -lc init . --scratch '$SCRATCH' # pin the scratch root explicitly -lc init --check --json # is this directory workable? (for scripts/agents) +lc init # converge cwd +lc init my-analysis # scaffold/converge ./my-analysis +lc init --check --json # is this directory workable? (for scripts/agents) +lc init # in a fresh clone: rebuild .venv + the annex ``` ## Next steps @@ -107,9 +104,9 @@ lc init --check --json # is this directory workable? (for script ```bash cd my-analysis # Describe your analysis in astra.yaml — inputs, outputs, recipes, -# decisions. ASTRA specs are plain YAML; write them by hand or draft -# them with your AI coding assistant of choice. -lc run # materialize the outputs -lc status # check what's ok / stale / missing -myst start # preview the report (requires: npm i -g mystmd) +# decisions — and write the scripts the recipes name. +uv add numpy # declare what the scripts import +git add -A && git commit -m "First analysis" +lc materialize # make the outputs +lc status # see where everything stands ``` diff --git a/docs/cli/materialize.md b/docs/cli/materialize.md new file mode 100644 index 00000000..6a7283c7 --- /dev/null +++ b/docs/cli/materialize.md @@ -0,0 +1,113 @@ +# lc materialize + +Make the analysis's outputs, and commit each one as it lands. This is +the build verb: it validates the spec, converges the environment, runs +every recipe that needs running — in dependency order, in parallel +where the graph allows — and commits each result together with its +manifest, in a commit whose message is a replayable run record. + +## Synopsis + +```text +lc materialize [OPTIONS] [TARGETS]... +``` + +With no targets, everything the spec declares, across every universe. +A target narrows the run to an output and whatever it depends on: + +- `fit` — the output `fit` in every universe that has it. +- `robust/fit` — exactly one universe's output. + +A target that matches nothing is an error listing what exists — +quietly making nothing is the least useful thing a build tool can do. + +## What gets remade + +An output is remade when it is `stale` — the analysis defines it +differently than it was made (a changed recipe or decision), one of +its declared inputs changed content, or it was edited by hand since. +Inputs are compared by content, so a rebuild that comes out +byte-identical stops the cascade there. + +An output that is `behind` — still exactly what the spec asks for, +but made under an earlier environment — is reported and left alone; +`--refresh` widens the run to remake those too. A `current` output is +never touched, under any flag. + +## The run's contract + +- **Starts clean, ends clean.** A dirty tree is a refusal (the message + sorts your uncommitted work from stray files under `results/`); a + failed or interrupted recipe's partial work is rolled back. +- **Fetches what it needs.** Declared inputs whose annexed content is + not in this clone are fetched before anything hashes. +- **Commits as it goes.** Each output lands in its own commit, written + by the driver in one thread while other recipes keep running. +- **Reports every independent failure.** One failing recipe doesn't + abort the rest; its dependents report `blocked` and the run exits 1 + with all of it listed. +- **Maintains the publication view.** With a `[project].license` + declared, the run converges `ro-crate-metadata.json` in a trailing + commit. + +On a containerized project, the run resolves the committed image first +(building it as a preflight if the declaration is committed but the +image never built). Inside a SLURM allocation, the run spans every +allocated node — see [Running on a Cluster](../user/cluster.md). + +## Check mode + +`--check` classifies every output without executing, committing, or +fetching anything, and exits `1` if a run would do work — the gate a +script or CI job branches on. It is exempt from the dirty-tree +refusal: reading the state of a project before deciding what to commit +is what it is for. + +## Options + +| Option | Default | Effect | +|--------|---------|--------| +| `--check` | off | Report what would run and why; exit 1 if anything is out of date. | +| `--refresh` | off | Also remake `behind` outputs. Never touches `current` ones. | +| `--json` | off | Emit the report as JSON on stdout. | + +There is deliberately no `--jobs` (a run takes every core; sizing +belongs to the allocation you run it in), no `--force`, and no flag to +*skip* a stale output — deleting its directory is your own file +operation, and stronger consent than a flag. + +## The JSON report + +```json +{ + "ok": true, + "up_to_date": true, + "made": [], + "current": ["baseline/fit", "robust/fit", "baseline/fit_plot", "robust/fit_plot"], + "behind": {}, + "failed": [], + "blocked": [], + "planned": {}, + "warnings": [], + "notes": [] +} +``` + +The first two keys are the ones to branch on: `ok` — everything +attempted finished; `up_to_date` — nothing needed doing (a failed run +is never up to date, and `behind` outputs don't count against it). +`planned` is check mode's answer, mapping each would-run output to why; +`behind` maps each left-alone output to the commit that can rebuild its +environment. `notes` carries sandbox messages verbatim — denial +remedies are built to be pasted. + +## Examples + +```bash +lc materialize # everything, all universes +lc materialize fit # one output (and upstreams), every universe +lc materialize robust/fit # one universe's output +lc materialize --check # would anything run? (exit 1 = yes) +lc materialize --refresh # also remake behind outputs +lc materialize --check --json # the machine-readable gate +``` diff --git a/docs/cli/run.md b/docs/cli/run.md index 3e311be9..594a18a8 100644 --- a/docs/cli/run.md +++ b/docs/cli/run.md @@ -1,97 +1,58 @@ # lc run -Materialize outputs declared in `astra.yaml`. Generates a Snakefile -and dispatches through Snakemake on a Dask cluster. +Run an ad-hoc command in the project environment, under isolation. +This is the probe verb: it executes exactly one command the way a +recipe would be executed — same environment, same sandbox — so "does +it work under `lc run`?" and "will it work as a recipe?" are the same +question. ## Synopsis ```text -lc run [OPTIONS] [OUTPUTS]... +lc run COMMAND... ``` -`OUTPUTS` is zero or more output ids. With no arguments, materializes -everything (Snakemake's `rule all`). - -## Options - -| Option | Default | Effect | -|--------|---------|--------| -| `--universe`, `-u NAME` | all universes in `universes/*.yaml` (or `["default"]` if none exist) | Restrict to one universe. | -| `--jobs`, `-j N` | `os.cpu_count()` | Parallel jobs / Dask submission concurrency. Passed as both `--cores` and `--jobs` to Snakemake. | -| `--rerun-triggers TRIGGERS` | `code,input,mtime,params` | Comma-separated rerun triggers (forwarded to Snakemake). | -| `--force`, `-f` | off | `--force` when targets are named, `--forceall` otherwise. | -| `--verbose`, `-v` | off | Show the underlying Snakemake / executor chatter and the spawned `snakemake` invocation. | - -## What happens, step by step - -1. Find the project (walk up looking for `astra.yaml`). -2. Discover universes from `universes/*.yaml` (default to `["default"]`). -3. Resolve the container runtime via - `lightcone.engine.container.load_runtime`. If `auto` falls back to - `none` while the spec declares containers, print a loud provenance - warning. -4. Generate `.lightcone/Snakefile` and - `.lightcone/snakefile-config.json` for the selected universes. -5. Translate any explicit `OUTPUTS` into Snakemake target paths - (`/.lightcone-manifest.json`) — this is what tells - Snakemake "build that specific output." -6. Open a Dask cluster context (`local`, `srun`-backed inside - `SLURM_JOB_ID`, or external if `DASK_SCHEDULER_ADDRESS` is set). -7. Spawn `snakemake -s … -d … --cores N --jobs N --executor dask - --rerun-triggers …` with `DASK_SCHEDULER_ADDRESS` in the environment. -8. In the default (non-verbose) path, filter the executor's banner - chatter so the output reads as lightcone's, not Snakemake's. Real - error content always passes through. - -## Output qualification - -When the same `output_id` appears in multiple sub-analyses, you must -qualify it as `.`: +Everything after `run` is the command, verbatim — flags included. +`lc run` takes no options of its own, so nothing needs escaping: ```bash -lc run inference # error if 'inference' is ambiguous -lc run hod_fitting.inference # disambiguated +lc run python -c "import numpy; print(numpy.__version__)" +lc run python src/fit.py --points data/points.csv --outliers keep --output /tmp/probe ``` -Each rule's body wraps the recipe in a ` run --rm --pull=never --v "$PWD":"$PWD" -w "$PWD" bash -c ''` shell when a -container is configured. After the recipe shell exits, the Snakefile -calls `write_manifest()` host-side and the validation snippet emits -warnings for empty / all-NaN / wrong-extension outputs. +## What it does + +- **Converges the environment first.** The probe syncs `.venv` to the + lock before executing, so what you probe is what a recipe gets. +- **Applies the recipe policy.** The project tree is read-only apart + from `results/`, declared inputs are readable, undeclared tools + don't execute. On a containerized project, the command runs inside + the committed image (which must already be built — the probe never + builds). +- **Proxies the exit code.** `lc run` exits with the command's own + code — `128 + N` when a signal killed it — so scripts and pipelines + read it exactly as they would the bare command. +- **Explains denials.** On a nonzero exit, a note on stderr says the + command ran sandboxed; when the failure looks like a denial, the + note names the path and the remedy (`uv add` for a missing package, + an ASTRA input declaration for data, `results/` or + `tempfile.mkdtemp()` for writes). + +A probe has no output and writes no manifest: nothing it does is +recorded anywhere. Any uv project works — `lc run` doesn't require an +`astra.yaml`, only `pyproject.toml`, `uv.lock` and `.venv` in the +current directory. + +## What it is not + +There is no sandbox opt-out and no flag surface — a command that needs +more than the policy grants is a command that would fail as a recipe, +and the fix (declare the dependency) is the same in both places. ## Examples ```bash -lc run # all outputs, all universes -lc run --universe baseline # one universe -lc run accuracy # one output -lc run accuracy precision --universe baseline # several -lc run --jobs 4 --verbose # parallel, with stack noise -lc run --force --universe baseline # rebuild everything -lc run --rerun-triggers params,input # tighter staleness -``` - -## Inside SLURM - -```bash -salloc -N 4 ... -lc run --universe baseline -j 16 +lc run python -c "import scipy" # is the package in the lock? +lc run bash -c 'echo $HOME' # see the private HOME a recipe gets +lc run python src/fit.py --help # exercise a script exactly as a recipe would ``` - -`lc run` detects `SLURM_JOB_ID`, binds the Dask scheduler to the -driver's hostname, and launches one `dask worker` per node via `srun`. -Workers advertise `cpus`, `memory`, and `gpus` resources. Per-rule -resource hints (`cpus_per_task`, `mem_mb`, `gpus_per_task`) constrain -which workers can pick up which jobs. - -## Provenance gotcha - -If `~/.lightcone/config.yaml` says `runtime: auto` and no runtime is -on PATH, `lc run` falls back to running recipes on the host. Because -each manifest still records the *declared* `container_image`, this is a -provenance lie. `lc run` prints a yellow warning telling you to either -install a runtime or set `container.runtime: none` explicitly. - -See [api/dask_cluster](../api/dask_cluster.md) for the cluster-shape -decision and [Architecture](../architecture.md) for the full execution -flow. diff --git a/docs/cli/status.md b/docs/cli/status.md index 7985c3a5..fae5fdb9 100644 --- a/docs/cli/status.md +++ b/docs/cli/status.md @@ -1,7 +1,10 @@ # lc status -Manifest-driven status report for every output declared in -`astra.yaml`. +Report what state each of the analysis's outputs is in. Reads only: it +runs nothing, commits nothing, transfers no data, does not mind an +unclean tree, and always exits `0` — a state is not a failure. The +moment you most need to know where a project stands is when it isn't +clean, so this verb works there. ## Synopsis @@ -9,59 +12,79 @@ Manifest-driven status report for every output declared in lc status [OPTIONS] ``` -## Options - -| Option | Default | Effect | -|--------|---------|--------| -| `--universe`, `-u NAME` | every universe in `universes/*.yaml` | Restrict to one universe. | -| `--json` | off | Emit machine-readable JSON instead of a styled table. | - ## Output -Per universe, one line per declared output: +```text + mode: direct + sandbox: landlock (fs: declared, network: allowed) + + · current baseline/fit a3f1f11 + · current baseline/fit_plot a3f1f11 + · behind robust/fit 00cc14e made under an earlier environment + ! stale robust/fit_plot — no manifest — it has never been materialized -``` -Universe baseline - ✓ ok accuracy - ✸ stale precision - ✗ miss recall - → alias inference +2 current · 1 behind · 1 stale ``` -Statuses (defined in `lightcone.engine.status.StatusLiteral`): +The header is repository facts: which mode the project executes in +(and, for a containerized project, the image's tag and state), and +what enforcement a run on this host would get. No runtime and no +network is needed to answer either. -| Status | Meaning | When you see it | -|--------|---------|-----------------| -| `ok` | Manifest present, recomputed `code_version` matches what the manifest recorded. | The output is up to date. | -| `stale` | Manifest present, but `code_version` drifted. | You changed the recipe, image, or a decision since the last run. `lc run` will re-execute. | -| `missing` | No manifest at the expected output path. | Never built, or the directory was deleted. | -| `alias` | The output has no `recipe:` of its own — it's just a name pointing at a sibling output (typical for ASTRA "promoted" outputs from sub-analyses). | Status is implicitly determined by the upstream. | +Then one line per output the spec declares, in dependency order: its +state, **the commit it was made at**, and — for anything not current — +why. The commit column is the verb's reason to exist: "which code made +this?" has an answer for a current output too, and for a `behind` +output that commit is where the environment that produced it can be +read back. -## Why it doesn't import Snakemake +## States -`lc status` reads only the per-output `.lightcone-manifest.json` files -and recomputes `code_version` against the current spec. It never -imports Snakemake or touches `.snakemake/`. That makes it usable on: +- `current` — exactly what the spec asks for. Nothing to do. +- `behind` — still what the spec asks for; the environment moved since. + Left alone by runs; `--refresh` remakes. +- `stale` — contradicts the project: definition changed, an input's + content changed, or the output was edited by hand since it was made + (a *foreign write* — the offending commit is named). -- A fresh clone before any `lc run`. -- A frozen archive copied off a cluster. -- A read-only workspace. +## Report vs gate -If a manifest is missing, the output reports `missing`. If a manifest is -unparseable, `read_manifest` returns `None` and you also see `missing` -— that is the agent-forged-file scenario; investigate with `lc verify`. +`lc status` reports; **`lc materialize --check` gates.** Two verbs +answering the same question with different exit codes is how a script +comes to depend on the wrong one, so the split is sharp: use status for +eyes, check for exit codes. -## Examples +## Options -```bash -lc status # every output, every universe -lc status --universe baseline # just baseline -lc status --json # machine-readable JSON output +| Option | Default | Effect | +|--------|---------|--------| +| `--json` | off | Emit the report as JSON on stdout. | + +## The JSON report + +```json +{ + "mode": "direct", + "image": null, + "sandbox": "landlock (fs: declared, network: allowed)", + "counts": {"current": 4, "behind": 0, "stale": 0}, + "outputs": [ + { + "output": "baseline/fit", + "status": "current", + "why": "", + "git_sha": "a3f1f11791430d1becbe5548477b5910ab59a94a", + "data_version": "sha256:939e9a55...", + "foreign_write": "" + } + ], + "warnings": [] +} ``` -## Related - -- [`lc verify`](verify.md) — recomputes data hashes too (slower; catches - tampering and broken chains). -- [api/status](../api/status.md) — the Python API. -- [api/manifest](../api/manifest.md) — the manifest schema. +Per output: the state, the reason (empty for `current`), the commit it +was materialized at and its content identity (both empty if it never +was), and `foreign_write` — the sha of a hand-edit's commit when one +was detected, which the prose `why` cannot carry for a machine +consumer. For a containerized project, `image` is +`{"tag": ..., "state": "present" | "absent" | "unfetched"}`. diff --git a/docs/cli/verify.md b/docs/cli/verify.md deleted file mode 100644 index 6e1ae216..00000000 --- a/docs/cli/verify.md +++ /dev/null @@ -1,67 +0,0 @@ -# lc verify - -Recompute hashes for every materialized output and walk the recorded -input chain. Catches tampering, drift, and forged manifests. - -## Synopsis - -```text -lc verify [OPTIONS] -``` - -## Options - -| Option | Default | Effect | -|--------|---------|--------| -| `--universe`, `-u NAME` | every universe | Restrict to one universe. | - -## Output - -``` -Universe baseline - ✓ ok accuracy - ✗ tampered_data precision recorded 'sha256:abc…' != actual 'sha256:def…' - ✗ broken_chain recall upstream 'features' data_version drifted - ✗ missing_manifest f1 No manifest found at output directory -``` - -Exit code is non-zero if any output failed. - -## Failure modes - -| Failure | What it means | -|---------|----------------| -| `missing_manifest` | The output directory exists but `.lightcone-manifest.json` is missing or unparseable. Most innocent cause: someone deleted the manifest. Most concerning: the directory was created by something other than `lc run`. | -| `tampered_data` | The bytes inside the output directory no longer hash to the `data_version` recorded in the manifest. Files were edited, regenerated outside the harness, or the directory contents differ from what was originally written. | -| `broken_chain` | The manifest records a specific upstream `data_version`, but the upstream's current `data_version` doesn't match. Usually means the upstream was rerun without rebuilding the downstream. Fix: `lc run` the downstream. | - -## Outputs without recipes - -Alias outputs (declared in `astra.yaml` without their own `recipe:`) -are skipped — there's no manifest to verify. They are checked -implicitly via the upstream output they reference. - -## Outputs that aren't materialized - -If an output's directory doesn't exist at all, `lc verify` skips it -(no failure to report). Use [`lc status`](status.md) to know what's -missing in the first place. - -## Examples - -```bash -lc verify # every output, every universe — non-zero exit on any failure -lc verify --universe baseline # just baseline -``` - -## When to run - -- Before publishing a result. -- After moving a project between machines. -- Periodically on shared archives. -- Whenever `lc status` shows `ok` but the data feels suspicious. - -## Related - -- [api/verify](../api/verify.md) — implementation and `VerifyResult`. -- [api/manifest](../api/manifest.md) — the manifest schema and what's hashed. diff --git a/zensical.toml b/zensical.toml index 705ce74c..8178b33a 100644 --- a/zensical.toml +++ b/zensical.toml @@ -25,11 +25,10 @@ nav = [ {"CLI Reference" = [ {"Overview" = "cli/index.md"}, {"lc init" = "cli/init.md"}, + {"lc materialize" = "cli/materialize.md"}, + {"lc status" = "cli/status.md"}, {"lc run" = "cli/run.md"}, {"lc build" = "cli/build.md"}, - {"lc status" = "cli/status.md"}, - {"lc verify" = "cli/verify.md"}, - {"lc export" = "cli/export.md"}, ]}, {"Python API" = [ {"Overview" = "api/index.md"},