Skip to content

WIP: execution environment redesign: uv-only substrate, sandboxed recipes, podman container hatch - #172

Draft
EiffL wants to merge 13 commits into
mainfrom
redesign_prototype
Draft

WIP: execution environment redesign: uv-only substrate, sandboxed recipes, podman container hatch#172
EiffL wants to merge 13 commits into
mainfrom
redesign_prototype

Conversation

@EiffL

@EiffL EiffL commented Aug 17, 2026

Copy link
Copy Markdown
Member

Implements the v6.1 execution-environment design (spec) end-to-end, replacing the v3-era authored-Containerfile model with a clean, ground-up execution layer: uv as the only environment substrate, OS-sandboxed recipes, and a podman-only container hatch. Roughly 3,300 lines of legacy venue/builder code deleted; all new layers land behind small protocols (Builder, ExecBoundary, container runtime) so other venues return later without forking call sites.

What changes

Environment model (uv-only)

  • A project is pyproject.toml + uv.lock + .python-version; the engine itself is a locked dependency of the experiment.
  • env_version = sha256(lock ‖ interpreter pin ‖ closed install-settings list ‖ system layer), inside every output's code_version — environment edits stale exactly what they can affect, with the blast radius printed at decision time ("environment changed: N materialized outputs are now stale").
  • New lightcone/launcher.py: lc <verb> discovers the project, detects the mode, scrubs ambient UV_*, converges (uv sync --locked --exact), and delegates execution verbs to the project-locked engine (frozen interface: argv passthrough + LC_DELEGATED=1).
  • Lock scan refuses unauditable path/editable dependencies; sdist-built packages are reported; mid-run relock is caught by a double worker-side gate.

Verb surface

  • lc runlc materialize; the new lc run <cmd> is the probe verb (arbitrary commands in byte-for-byte the recipe environment, tmp-only writes, rename guard for output ids).
  • lc status gains the 3-line header (mode: / image: / sandbox:) and pre_migration states; lc verify gains provenance notes (unsandboxed, dirty_tree, pre_migration).
  • lc init rewritten for the uv scaffold (pyproject with the engine locked in, exact interpreter pin, uv lock+sync, AGENTS.md stanza); refuses an authored root Containerfile.

Hermeticity (spec §7)

  • Vendored Landlock (Linux, incl. in-container) + generated Seatbelt (macOS, new CI smoke workflow): own-output-RW / project+inputs-RO / versioned exec allowlist + ELF-loader tier, fresh per-recipe HOME/XDG, PYTHONPYCACHEPREFIX redirect (approved §7 amendment).
  • The stdlib-only shim lightcone/_sandbox_exec.py restricts between fork and exec; exit 97 is the reserved never-silent setup-failure code.
  • The denial message is the primary UI (re-stat, tool/data classification, two copy-pasteable remedies with costs stated, capped apt-hint table, unconditional trailer on every nonzero sandboxed exit).
  • Every manifest records the applied enforcement in hermeticity; --require-sandbox[=declared-fs] / --no-sandbox recorded honestly.

Container hatch (podman-only)

  • One user surface: [tool.lightcone.image] (+ optional Containerfile.extra). Its presence is containerized mode.
  • Modal-inspired internals (engine/image/): declaration → deterministic Containerfile render (fixed layering, apt before uv sync, offline ENV in the final stage only, golden-tested) → tag lc-env-<hash> as a pure function of rendered text + pyproject.toml + uv.lock (code edits never move it) → podman build with pointed error mapping (base-contract exit codes, apt not-found, arch miss) → build record + dpkg-snapshot attestation.
  • Full-stack execution: one digest-pinned podman run --net=none --userns=keep-id hosts engine, dask workers, child snakemake, and recipes from the baked /opt/venv; in-container Landlock upgrades the record to podman+landlock. lc materialize builds (announced); lc run never builds. macOS podman machine preflight incl. the mount-share check.

Integrity

  • Manifest SCHEMA_VERSION=2 (golden-pinned field list): env_version, git_dirty, hermeticity, worker_runtime, image{tag,digest}, dpkg_snapshot_sha256, sdist_built, runtime attestation (platform/interpreter/uv/GPU/threading knobs). Old manifests read as pre_migration (hashes still verified).
  • Typed RuleJob generator→worker contract; worker sequence = pre-gate → env check (uv sync --check / image identity assert) → boundary exec with the offline overlay → post-gate → manifest.

Deleted (clean break, no compat)

Cloud Build, podman-hpc/docker/kubernetes runtimes, site registry, gateway/SLURM/external dask branches, authored Containerfile + requirements.txt scaffolds, global ~/.lightcone/config.yaml, version-skew shims. Venue docs replaced; API docs collapsed to one maintained module map; eval prompt rewritten for the new verb/environment model.

Recorded deviations from the spec

  1. ASTRA carries no container/sandbox keys — per-output sandbox escalation lives in [tool.lightcone.sandbox] writable-project; BYO per-output containers deferred.
  2. Local-only venue scope, podman-only backend — Perlmutter/hub/Cloud Build return behind the new protocols.
  3. §7 PYTHONPYCACHEPREFIX amendment (noted in the spec in place).

Verification

  • 403 tests green (pytest default + -m "slow or podman"), ruff clean, mypy strict clean.
  • Golden tests pin: env_version fingerprints, rendered Containerfiles, the manifest v2 field list, the frozen delegation interface, the offline-ENV ordering invariant.
  • Real Landlock enforcement suite (unprivileged) incl. the dynamically-linked-exec loader proof and both denial-fallback fixtures; real podman build smoke (8 tests).
  • Manually verified end-to-end on Linux (Landlock ABI 9, podman 6.1): direct-mode init→materialize→status→verify; sandbox denying a recipe that tampers with astra.yaml; full containerized materialize with a declared system package, manifest recording podman+landlock / declared / denied + image digest + dpkg snapshot.
  • Spikes recorded: Landlock ruleset-FD survives uv run's spawn chain (EXEC_VIA_UV=True); landlock_* admitted under podman's default seccomp; --userns=keep-id ownership; --net=none loopback.

🤖 Generated with Claude Code

https://claude.ai/code/session_01DUW9H1vbiwCKUrxQyWRYoj

EiffL and others added 13 commits August 14, 2026 22:28
…e guard

Stage 1 of the v6.1 execution-environment migration (spec §11):
- `lc materialize` takes over pipeline execution (all flags unchanged)
- `lc run CMD` becomes the thin probe verb: uv run --locked --exact
  from the project root (sandbox lands in a later phase)
- rename guard: `lc run <output_id>` or old materialize flags error
  with a pointed 'did you mean lc materialize' redirect before any exec
- user-facing strings (init next-steps, results/README, wrroc hint)
  now say lc materialize

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DUW9H1vbiwCKUrxQyWRYoj
…n-podman venue code

Deletion wave A of the v6.1 migration (local-only scope):
- delete engine/cloudbuild.py and engine/site_registry.py (+ tests)
- dask_cluster.py: LocalCluster only; gateway/SLURM/external-scheduler
  branches, worker-env injection, and resource-arg plumbing removed
- executor plugin: DASK_SCHEDULER_ADDRESS is the only rendezvous;
  version-skew shim (_unpack_result) removed
- container.py: podman is the only OCI runtime — docker, podman-hpc,
  kubernetes, registry refs, and site-aware detection removed
- commands.py: Perlmutter login-node guard, gateway single-image guard,
  provenance-lie warning, Cloud Build path, ~/.lightcone/config.yaml
  auto-creation, and `lc build --runtime` removed
- scratch.py: site scratch defaults removed (env > project > tempdir)
- manifest: slurm_job_id / worker_image fields dropped
- pyproject: dask-gateway dependency removed
- lc run probe: old-flags guard dropped (no muscle-memory accommodation)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DUW9H1vbiwCKUrxQyWRYoj
…tion/render/identity

Additive foundations for the v6.1 execution environment:
- engine/environment.py: Mode (derived from [tool.lightcone.image] /
  Containerfile.extra presence), EnvironmentSpec, env_version (one
  length-framed formula for both modes, golden-pinned), the closed
  install-settings list, [tool.lightcone] closed-surface validation,
  per-output writable-project parsing, lock scan (path-dep refusals,
  sdist_built report, group advisory), packaged×containerized and
  authored-Containerfile refusals
- engine/project.py: shared astra.yaml walk-up discovery
- engine/uv_env.py: closed UV_* scrub list + offline overlay
- engine/image/: Modal-inspired pure core — errors taxonomy, engine
  constants (digest-pinned debian base + uv 0.12.3, manifest-list
  digests for arch-independent tags), declaration parsing with static
  refusals, ImageDefinition, deterministic Containerfile renderer
  (fixed layer order, offline ENV in final stage only), content-
  addressed tag lc-env-<16hex>
- tests: golden Containerfiles (4 cases, --regen-goldens flag), golden
  env_version fingerprints, identity-sensitivity matrix incl. the G5
  code-edits-move-nothing checks; conftest gains make_project

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DUW9H1vbiwCKUrxQyWRYoj
…e lc init

The identity switch (spec §3) and the seams the sandbox/image layers
plug into:
- manifest SCHEMA_VERSION=2: env_version, git_dirty, hermeticity,
  worker_runtime, image{tag,digest}, dpkg_snapshot_sha256, sdist_built,
  runtime attestation (uv_version/platform/python_build/env_snapshot/
  gpu_driver); container_image dies; is_pre_migration reader;
  golden-pinned field list
- code_version = sha256({recipe, decisions, env_version,
  writable_project}) — one shared function for generator and status
- engine/attestation.py: worker-side runtime capture
- engine/boundary.py: ExecBoundary protocol + HostPassthroughBoundary
  (attests mechanism:none until the sandbox lands); run_rule executes
  through it and records hermeticity honestly
- snakefile generator: recipes never wrapped; env loaded once; lock-scan
  refusals block generation; writable-project validated against
  declared outputs; cfg carries env identity + git_dirty
- status: pre_migration label, env_blast_radius, mode/image/sandbox
  header; verify: pre_migration/dirty_tree/unsandboxed notes (v1
  manifests still hash-checked)
- engine/container.py deleted; tree.resolve_container_spec deleted
- lc init rewritten for the uv project model: pyproject scaffold with
  the engine in the lock, .python-version pin, uv lock + sync (loud
  failures), AGENTS.md stanza, authored-Containerfile refusal,
  astra container: line stripped; Containerfile/requirements/venv
  templates die
- containerized mode: explicit interim refusal until the podman
  build/run backends land (never a provenance lie)
- wrroc reads the v2 image field

Verified end-to-end: init → materialize → status (3-line header,
blast radius) → verify (notes) on a real project.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DUW9H1vbiwCKUrxQyWRYoj
lightcone/launcher.py: verbs beyond {init,status,verify,build,export}
delegate to the project-locked engine — astra.yaml walk-up discovery,
mode detection, ambient UV_* scrub (closed list), direct-mode converge
(uv sync --locked --exact --compile-bytecode), then os.execve of
<venv>/bin/lc with the frozen interface (argv passthrough +
LC_DELEGATED=1; never a PATH fallback). The containerized branch falls
through to the interim refusal until the podman runtime lands.
Frozen-interface pin test included.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DUW9H1vbiwCKUrxQyWRYoj
…ine overlay

- engine/job.py: RuleJob value object defines the generator→worker cfg
  contract (JSON via to_cfg/from_cfg); generator constructs it
- run_rule implements spec §6: pre-gate (env_version recomputed from
  the tree vs the job's baked value), env check (direct: uv sync
  --locked --exact --check + venv presence; containerized: baked
  identity.json env_version + LC_IMAGE_DIGEST assert), boundary exec
  with the offline overlay (UV_OFFLINE=1, UV_PYTHON_DOWNLOADS=never)
  and ambient UV_* scrub, post-gate before write_manifest
- --require-sandbox[=declared-fs] / --no-sandbox on lc materialize
  (and probe flags on lc run), traveling to workers via env — run
  flags never perturb the content-addressed job identity; enforcement
  is worker-side against the probed boundary level
- ExecBoundary gains probe(); RuleGateError distinguishes gate refusals
  from recipe failures

Gate tests: pre-gate abort (recipe never runs), post-gate mid-recipe
relock (no manifest), overlay/scrub visibility in recipe env,
require-sandbox refusal before exec.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DUW9H1vbiwCKUrxQyWRYoj
…l UX

The hermeticity layer (spec §7), fully greenfield:
- engine/sandbox/_landlock.py: vendored ctypes bindings (syscalls
  444/445/446, ABI-tiered access bits, file-vs-dir clamping, O_PATH
  parents); abi() doubles as the capability probe (unprivileged;
  seccomp-blocked reads as unavailable)
- engine/sandbox/policy.py: the §7 policy — own-output+scratch+tmp
  write set, project+inputs+OS-baseline read set, two-tier exec set
  (env bin dir + versioned ~35-name utility allowlist + realpath'd ELF
  loaders + symlink-resolved interpreter root), fresh per-recipe
  HOME/XDG/MPLCONFIGDIR/PYTHONPYCACHEPREFIX under the tmp scope,
  writable-project escalation → project-rw
- lightcone/_sandbox_exec.py: stdlib-only shim (namespace-package
  placement keeps its import footprint nil) — fstat FD, prctl
  NO_NEW_PRIVS, landlock_restrict_self, execvp; exit 97 reserved for
  setup failure (the never-silent guard); Seatbelt branch execs
  sandbox-exec with the generated profile
- engine/sandbox/wrap.py: ruleset built pre-fork, FD via pass_fds
  (spike-verified survival; EXEC_VIA_UV=True recorded), no preexec_fn
- engine/sandbox/probe.py: per-job capability probe + the single home
  of the hermeticity enum mapping (landlock⇒network unenforced,
  seatbelt⇒denied, container --net=none⇒denied) + lc status line
- engine/sandbox/exec_boundary.py: the ExecBoundary implementation —
  downgrade console line, setup-failure attribution, denial rendering
- engine/sandbox/denial.py + hints.py: the primary UI — re-stat,
  access-aware classification, two-remedy message with costs stated,
  capped/versioned apt hint table, unconditional trailer on every
  nonzero sandboxed exit
- engine/sandbox/seatbelt.py: generated SBPL (deny default, loopback-
  only network, ipc allowances) + macOS CI smoke workflow
- lc run probe now sandboxed: tmp-only writes, union-of-declared-inputs
  read scope, --sandbox-debug shell, --no-sandbox opt-out; the shim
  rides the uv run hop (FD inheritance verified)

Verified end-to-end on this host (Landlock ABI 9): recipe tampering
with astra.yaml denied + trailer fired + no manifest; hermeticity
recorded {landlock, declared, unenforced, abi 9, allowlist v1}; probe
in-tree write denied, /tmp allowed; full launcher delegation exercised
(and the lock scan refused a path-dep engine, as specified).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DUW9H1vbiwCKUrxQyWRYoj
…uild

- engine/image/record.py: .lightcone/image/record.json (machine-local,
  gitignored) with tag/image_id/digest/platform/env_version + the dpkg
  snapshot text archived beside it (attestation outlives image GC)
- engine/image/builder.py: Builder protocol + the three-file
  BuildContext (rendered Containerfile, pyproject.toml, uv.lock —
  world-readable; the structural G5 guarantee)
- engine/image/builder_podman.py: podman build with pointed error
  mapping — contract exit codes 41/43/44 → BaseContractError with the
  named escapes, apt 'Unable to locate package' → the apt-cache-search
  message, manifest-list arch miss → platform contract, else a bounded
  log tail; dpkg snapshot captured offline with the entrypoint cleared
- ensure_image (tag hit ⇒ no-op; env edit ⇒ rebuild), resolve_pinned
  (lc run never builds — errors with the exact lc build command),
  image_status (offline/local-only)
- lc build wired for containerized mode; lc status image header line
  shows tag + built/needs-build

Real podman smoke (8 tests, opt-in -m podman): minimal image builds;
baked identity.json carries env_version + 3.12.12; /opt/venv python
runs; declared system package works; offline ENV baked in final stage;
snapshot attests the apt layer; unlocatable package surfaces the
pointed error end-to-end.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DUW9H1vbiwCKUrxQyWRYoj
…n runtime, macOS preflight

- engine/image/mounts.py: MountSet — project at its identical abspath
  (RW; RO for probes), declared external inputs RO (deduped, nested
  collapsed, parent-of-project refused), tmpfs /tmp + 1g /dev/shm;
  external_input_paths shared with the probe's read allowlist
- engine/image/runtime_podman.py: the full-stack wrapper — one podman
  run hosts engine + dask workers + child snakemake + recipes from the
  baked /opt/venv; pinned BY image id at the argv; --net=none (loopback
  intact), --userns=keep-id, entrypoint cleared, SELinux label=disable;
  closed env passthrough allowlist; identity env (LC_WORKER_RUNTIME,
  LC_CONTAINER_NETWORK, LC_IMAGE_TAG/DIGEST); the in-container digest
  honesty note documented
- engine/image/machine.py: macOS podman-machine preflight — missing/
  stopped VM and unshared mount sources are refusals naming the exact
  fix (never a silently empty mount)
- launcher: containerized delegation (materialize builds+announces;
  lc run never builds); generator pins every job to the build record
  (tag/image_id/dpkg sha) and refuses a stale record; the interim
  refusal becomes the honest outside-image guard
- sandbox: in containerized mode the image contents are the exec set
  (spec §7) — OS-wide EXECUTE grant, allowlist version recorded as
  absent; read/write fencing unchanged

Verified end-to-end (dev-source overlay inside the built image):
recipe using the declared system package (bc) materialized under
podman+landlock; manifest records worker_runtime container,
image{tag,digest}, hermeticity{podman+landlock, declared, denied,
abi 9}, dpkg_snapshot_sha256; earlier undeclared-tool run was denied
by in-container Landlock with the trailer.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DUW9H1vbiwCKUrxQyWRYoj
… design

- User guide: install (uv-only on-ramp), getting-started (four verbs,
  uv add discipline), NEW environment page (modes, sandbox, denial UX,
  container hatch), troubleshooting, glossary — all rewritten;
  cluster/HPC pages deleted (local-only scope)
- CLI reference: new lc materialize page; run/build/status/verify/init
  rewritten for the probe verb, generated images, the 3-line status
  header, and verify notes
- architecture.md rewritten around the three owned layers + the image
  hatch; Python API section collapsed to one maintained module map
  (hand-written per-module pages deleted)
- README quick start + capabilities updated; CLAUDE.md rewritten
  (architecture, invariants, test patterns, extension points)
- evals/prompt.md rewritten: materialize/probe verbs, one-environment
  model, uv add, sandbox guidance; eval task's legacy container: line
  removed
- spec: implementation record added to the status block (ASTRA carries
  no container/sandbox keys; local-only + podman-only scope) and the
  §7 PYTHONPYCACHEPREFIX amendment noted in place
- nav updated; docs site builds clean

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DUW9H1vbiwCKUrxQyWRYoj
…drop dead code

Applies the 4-angle cleanup review (reuse/simplification/efficiency/
altitude) over the branch:

Single homes for things that must never drift
- engine/contract.py: the launcher↔engine / driver↔worker LC_* env
  contract (DELEGATED, WORKER_RUNTIME + 'container', CONTAINER_NETWORK,
  IMAGE_DIGEST, sandbox flags) + in_container() + recipe_env_prefix();
  the podman wrapper now composes its -e list from it and every
  consumer imports it
- manifest.frame() + manifest.canonical_json(): the one hash-framing
  and canonical-JSON pair behind env_version, the image tag, and
  code_version (previously two/five copies)
- tree.qualified_id + tree.scoped_decisions_for_output/
  load_universe_decisions: rule-key spelling and decision scoping now
  shared by generator, status, and target resolution (was mirrored by
  copy)
- sandbox/wrap.run_wrapped(): the wrapped-spawn lifecycle (FD close +
  tmp-home reclaim) shared by the rule path and the probe path — which
  also fixes the probe's hardcoded .venv env-prefix (wrong inside an
  image)
- shim env names (LC_SANDBOX_MODE/FD/PROFILE) + SETUP_FAILURE_EXIT now
  defined once in the stdlib-only shim and imported by the engine
- engine.lc_version(); image.errors.require_podman(); OFFLINE_ENV
  derived from uv_env.OFFLINE_OVERLAY; runner reads
  constants.IDENTITY_PATH; runtime composes /opt/venv paths from
  constants; MANIFEST_FILENAME interpolated into the generated
  Snakefile
- external_input_paths() now resolves through
  tree.resolve_external_input (follows from: alias hops the raw
  source: walk missed) and feeds both mounts and the probe read scope

Dead code removed
- HostPassthroughBoundary (never instantiated; the sandbox-off branch
  is the single passthrough), DigestMismatchError (structurally
  unnecessary — execution pins by image id; resolve_pinned now
  verifies the recorded id exists), PodmanRuntime.run(),
  _landlock.restrict_self, the unread LC_IMAGE_TAG env,
  SandboxPolicy.unresolved_utilities, ExecScope 'debug' literal,
  cluster_for_run max_workers, explain_failure's mechanism param,
  EnvironmentSpec.packaged/install_settings fields, over-broad
  __all__ re-exports in image/ and sandbox/

Wasted work removed
- lc status: one walk serves display/JSON/blast-radius (was 2×)
- generator: universe decisions resolved once per universe (was
  rules×universes); scan_lock passed through from materialize (was 2×)
- targets resolved from one spec parse (was outputs×universes parses)
- lc run: one spec parse for rename guard + input scope (was 2)
- load_environment passes its pyproject parse into
  load_image_declaration (was 2 TOML parses per call, 4 per rule)
- status/verify accept a spec pass-through across universes
- BuildContext stages the exact EnvInputs bytes the tag hashed (no
  re-read, and the built image can't diverge from the hashed identity)

New pins: TOOL_ENV_VERBS ↔ Click command parity test.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DUW9H1vbiwCKUrxQyWRYoj
@EiffL EiffL changed the title v6.1 execution environment: uv-only substrate, sandboxed recipes, podman container hatch WIP: execution environment: uv-only substrate, sandboxed recipes, podman container hatch Aug 17, 2026
@EiffL EiffL changed the title WIP: execution environment: uv-only substrate, sandboxed recipes, podman container hatch WIP: execution environment redesign: uv-only substrate, sandboxed recipes, podman container hatch Aug 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant