Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
134 changes: 67 additions & 67 deletions docs/cli/build.md
Original file line number Diff line number Diff line change
@@ -1,90 +1,90 @@
# lc build

Build container images declared in `astra.yaml` (or pre-pull registry
images so `lc run` can use `--pull=never`).
Build the project's system-layer image, and commit it. Containerized
projects only — a project containerizes by declaring a
`[tool.lightcone.image]` table in `pyproject.toml`, and on a direct
project this verb just says so and exits.

## Synopsis

```text
lc build [OPTIONS]
```

## Options
Idempotent: an image that is already built and committed is left
alone.

| Option | Default | Effect |
|--------|---------|--------|
| `--force` | off | Rebuild / re-pull even if the tag already exists locally. |
| `--runtime {docker,podman,podman-hpc,kubernetes}` | resolved from `~/.lightcone/config.yaml` | Override the runtime for this build. |

## What it does

For every distinct `container:` value found in the project (root,
sub-analysis, or recipe-level):

- **Path to a Containerfile** → compute the content-addressed tag
`lc-<project>-<sha256[:12]>`, build the image, and (for `podman-hpc`)
migrate it into the per-node container cache.
- **Anything else** (e.g. `python:3.12-slim`, `ghcr.io/foo/bar:tag`) →
pull it into the local image store. This is what lets `lc run` pass
`--pull=never` to the runtime, sidestepping `unqualified-search-registries`
resolution issues with content-addressed tags.

On the `kubernetes` runtime (a lightcone JupyterHub deployment, where
no local OCI runtime exists) the same command builds through the
deployment's **GCP Cloud Build** service instead: the staged build
context is uploaded to the deployment's build bucket and the resulting
image is pushed as `$LIGHTCONE_REGISTRY/lc-<project>:<sha256[:12]>` —
the same content-addressed identity, so an unchanged environment is a
single registry check and no build at all. Pre-built registry images
are left alone (worker pods pull them directly). Auth is the pod's
Workload Identity; nothing to configure.

If the runtime is `none` (either by config or because `auto` couldn't
find one), `lc build` prints a friendly note and exits 0. There is
nothing to build.

## Tag computation
## What the image is

```text
lc-<sanitized-project-name>-<sha256[:12]>
The image is the *system layer* only: the declared base (digest-pinned,
or the default), the declared apt packages, and the pinned Python
interpreter. Your analysis environment is not in it — recipes' Python
packages come from the project's lock, synced into the container at run
time — and neither is `lc` itself. That is what makes "editing code
never rebuilds the image" structural: no project file enters the build
context at all.

The declaration is a closed set of keys, hashed into the image's
identity:

```toml
[tool.lightcone.image]
base = "docker.io/library/debian@sha256:..." # optional; default pinned by lc
apt-install = ["libfftw3-dev"] # optional
run-commands = ["curl -L ... | tar xz"] # optional, the bounded escape
env = { OMP_NUM_THREADS = "1" } # optional
```

The hash covers the Containerfile contents plus any of these dependency
files at the project root:
## The archive is the store

- `requirements.txt`
- `requirements-dev.txt`
- `requirements-test.txt`
- `pyproject.toml`
- `setup.py`
- `setup.cfg`
- `poetry.lock`
- `Pipfile.lock`
`lc build` saves the built image into the repository —
`.datalad/environments/<tag>/image`, a `docker-archive` committed
through git-annex — so the exact bytes travel with the project: a
clone obtains them with a fetch, no registry and no credentials
involved. Execution always pins the image's content *id*, never a tag,
so nothing can substitute a different image under the same name.

Edit any one of those and the tag changes. That, in turn, changes
`code_version` in every recipe that uses the image, which marks all
downstream outputs `stale` in `lc status`.
The archive records the architecture it was built for, and a host that
can't execute that architecture is refused up front — build where the
architecture matches the machines that will run recipes (on NERSC, a
login node).

## Examples
## Requirements

```bash
lc build # build / pull whatever's missing
lc build --force # rebuild / re-pull everything
lc build --runtime podman-hpc # force the HPC runtime
```
- A clean tree — the image commit must not sweep your staged edits in,
and the tag derives from the committed declaration.
- A build-capable runtime: `podman-hpc`, `podman`, or `docker`
(detected in that order; nothing to configure).

## Pre-staging for HPC
`lc materialize` also builds as a preflight when the committed
declaration has no image yet, announcing it first — `lc build` exists
so you can pay the minutes when *you* choose to.

On a login node:
## Options

```bash
$EDITOR ~/.lightcone/config.yaml # container.runtime: podman-hpc
lc build # builds + migrates everything
| Option | Default | Effect |
|--------|---------|--------|
| `--json` | off | Emit the result as JSON on stdout. |

## The JSON result

```json
{
"mode": "containerized",
"tag": "lc-env-1a2b3c4d5e6f7a8b",
"id": "sha256:...",
"archive": ".datalad/environments/lc-env-1a2b3c4d5e6f7a8b/image",
"action": "built"
}
```

Then submit a SLURM job for `lc run`. The compute nodes will find every
image already cached.
`action` is `"built"` when this invocation built and committed the
image, `"present"` when it was already there. On a direct project the
result is just `{"mode": "direct"}`.

See [api/container](../api/container.md) for the implementation and
[Architecture](../architecture.md) for why we wrap recipes ourselves
instead of using Snakemake's `container:` directive.
## Examples

```bash
lc build # build + commit, or confirm it's already there
lc build --json # the machine-readable form
```
106 changes: 0 additions & 106 deletions docs/cli/export.md

This file was deleted.

40 changes: 27 additions & 13 deletions docs/cli/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,21 +7,27 @@ and audit it.

## Global behavior

- `~/.lightcone/config.yaml` is created automatically on first use of
any `lc` command. You do not need to create it manually.
- All commands except `init` walk up from the cwd looking for
`astra.yaml`. If none is found, the command errors out.
- **The current directory is the project.** Every command except
`init` assumes it is invoked from the project root; there is no
walk-up and no global configuration. Outside a project, a command
errors cleanly.
- **Nothing waits on a human.** No command prompts or opens an
interactive shell — every verb runs to completion on its arguments
alone, which is what makes the CLI safe to drive from scripts and
agents.
- **Refusals carry their remedy.** When a command refuses (a dirty
tree, a login node, a missing image), the message names the exact
command that fixes it.

## Commands

| Command | Purpose |
|---------|---------|
| [`lc init`](init.md) | Scaffold a new ASTRA project (`astra.yaml`, `Containerfile`, `.lightcone/`, MyST report template, optional venv & git). |
| [`lc run`](run.md) | Generate the Snakefile and dispatch through Snakemake + Dask. |
| [`lc build`](build.md) | Build container images declared in `astra.yaml`. |
| [`lc status`](status.md) | Manifest-driven status report. No Snakemake import needed. |
| [`lc verify`](verify.md) | Recompute hashes, walk the input chain, surface tampering. |
| [`lc export`](export.md) | Emit interoperable bundles (Workflow Run RO-Crate) for publication. |
| [`lc init`](init.md) | Converge a directory into a Lightcone project (idempotent). |
| [`lc materialize`](materialize.md) | Make the analysis's outputs; commit each one as it lands. |
| [`lc status`](status.md) | Report the state of every output. Reads only; always exits 0. |
| [`lc run`](run.md) | Run an ad-hoc command in the project environment, under isolation. |
| [`lc build`](build.md) | Containerized projects: build the image and commit it. |

## Global options

Expand All @@ -33,7 +39,15 @@ Options:
--help Show this message and exit.
```

## Removed commands
## Exit codes

For historical context: `lc dev`, `lc setup`, `lc target`, and `lc update` no
longer exist as explicit commands. See the removal pages for details.
- `0` — the command did what it says.
- `1` — a refusal or a failure, with the reason on stderr. For
`lc materialize --check` and `lc init --check`, exit 1 means "work
would be done" — the gate form scripts branch on.
- `lc run` is a proxy: it exits with the command's own code
(`128 + N` for a signal), so pipelines read it exactly as they would
the bare command.

Every verb with a report takes `--json` for the machine-readable form;
each verb's page shows its shape.
Loading
Loading