Snakemake orchestration for the real-data pipeline (design for discussion)

[cailmdaley]

Snakemake orchestration for the real-data pipeline

Summary

Replace the shell-script orchestration of real-data ShapePipe runs (run_job_sp_canfar_v2.0.bash → job_sp_canfar_v2.0.bash, plus per-site sbatch reimplementations) with a Snakemake workflow in this repo. Rules call shapepipe_run -c <config> on the existing config chains, at the granularities the bash layer already uses; Snakemake's file-based DAG takes over the state the bash layer hand-rolls — completeness counts, done-logs, flock dedup, retry/force. Module code stays largely untouched. Once the DAG is explicit, "why did this tile fail, what is safe to rerun" become answerable instead of archaeological. Living PRD; the eventual PR just says "implements #848." Please keep poking holes.

Pipeline shape

_{🟦 tile-level 🟧 exposure-level (shared across tiles) 🟪 index 🟩 cleanup}

flowchart TD
    subgraph S1["invocation 1 · PREPARE — one chain per tile (×~7300 at DR6)"]
        Git["tile_get_images<br/>(download)"] --> Uz[uncompress weight] --> Fe["find_exposures<br/>(read header)"]
    end
    subgraph S3["invocation 2 · COMPUTE"]
        IX[("parse time: dedup union of<br/>exposure lists → index.sqlite")]
        subgraph EX["one chain per DEDUPED exposure (×~20k) — each built once, shared by ~4 tiles"]
            Gie[exp_get_images] --> Sp[split into 40 CCDs] --> Ma[exp_mask] --> Psf["exp_psf: SExtractor → setools → PSFEx → PSF interp"]
        end
        subgraph TQ["one chain per tile (×~7300)"]
            Mh[tile_merge_headers] --> Sxt["tile_detect<br/>(SExtractor)"] --> Vi["tile_vignets<br/>(PSF interp + stamps)<br/>🗑 vignettes after ngmix"]
            Vi --> Ng1["ngmix chunk 1"]
            Vi --> Ngd["⋯"]
            Vi --> NgN["ngmix chunk N"]
            Ng1 --> Ms["merge_sep_cats — gather<br/>🗑 chunks after merge"]
            Ngd --> Ms
            NgN --> Ms
            Ms --> Mc["make_cat → final_cat ×1 per tile"]
        end
        Vi --> Cl["🗑 clean_exposure ×1 per exposure:<br/>fires when ALL its tiles' vignettes exist"]
    end
    Fe --> IX
    IX -.->|defines the job set| Gie
    Psf -->|"PSF models, stamps"| Vi
    Sp -->|headers| Mh
    Psf -.-> Cl

    classDef tile fill:#cfe2f3,stroke:#4a86c8,color:#1a3a5c
    classDef exp fill:#fce5cd,stroke:#e69138,color:#5c3a1a
    classDef idx fill:#d9d2e9,stroke:#8e7cc3,color:#3a2a5c
    classDef clean fill:#d9ead3,stroke:#6aa84f,color:#2a4a1a
    class Git,Uz,Fe,Mh,Sxt,Vi,Ng1,Ngd,NgN,Ms,Mc tile
    class Gie,Sp,Ma,Psf exp
    class IX idx
    class Cl clean

Loading

Each box is one Snakemake job: one shapepipe_run on one unit (tile or exposure); modules parallelize over CCDs/objects inside the job. Arrows are declared input: edges; the scheduler orders everything from them, including the tile↔exposure fan and the rolling cleanup. Deletion (🗑) rolls forward with the run — each product is removed when its last reader finishes: the exposure store once its tiles' vignettes exist, the vignettes once ngmix has read them, the chunks at the merge (D5). Only clean_exposure is a rule; the rest is temp() bookkeeping by the scheduler, not jobs. The prepare products and final_cat persist.

Design

D1. Execution model: sp run is the entry point

A run is declared by a tile list. sp run brings the products on disk up to date with it. It is two snakemake invocations, because the exposure job set is determined from tile headers and a Snakemake DAG is fixed at parse time:

sp run [--tiles tiles.txt] [--profile nibi] [-n] [snakemake passthrough…]
│
├─ 1. PREPARE   snakemake prepare_all_tiles
│      Static per-tile DAG, known from the tile list alone:
│      download image, unzip weight, find_exposures.
│      Cheap, wide, idempotent — already-prepared tiles are no-ops.
│
└─ 2. COMPUTE   snakemake all
       At parse time: read each tile's exposure list (paths determined by
       the tile list), dedup the union into index.sqlite; tiles whose list
       is missing land in missing.json and are skipped. Then the DAG:
       exposure chains, tile chains, ngmix scatter/gather, final_cat per tile.

Snakemake flags (--jobs, -n, --forcerun, …) pass through to both invocations.

The two invocations are joined by a tile ↔ exposure index, built at parse time and persisted as index.sqlite. It accumulates across invocations, so the clean_exposure consumer sets span the whole campaign, not just the current tile list.

Each exposure is processed exactly once, and cleaned automatically once its tiles have extracted what they need. In miniature — a run requesting tiles A and B, where A's header lists exposures 101, 102 and B's lists 102, 103:

flowchart LR
    subgraph P["PREPARE"]
        TA["tile A prepared"]
        TB["tile B prepared"]
    end
    TA -->|"101, 102"| IX
    TB -->|"102, 103"| IX
    IX[("index:<br/>101, 102, 103")]
    subgraph C["COMPUTE"]
        E1[exp 101 chain] --> A[tile A chain] --> FA[final_cat A]
        E2["exp 102 chain (shared)"] --> A
        E2 --> B[tile B chain] --> FB[final_cat B]
        E3[exp 103 chain] --> B
        A --> C1[clean exp 101]
        A --> C2[clean exp 102]
        B --> C2
        B --> C3[clean exp 103]
    end
    IX -.-> E1
    IX -.-> E2
    IX -.-> E3

    classDef tile fill:#cfe2f3,stroke:#4a86c8,color:#1a3a5c
    classDef exp fill:#fce5cd,stroke:#e69138,color:#5c3a1a
    classDef idx fill:#d9d2e9,stroke:#8e7cc3,color:#3a2a5c
    classDef clean fill:#d9ead3,stroke:#6aa84f,color:#2a4a1a
    class TA,TB,A,B,FA,FB tile
    class E1,E2,E3 exp
    class IX idx
    class C1,C2,C3 clean

Loading

Two tiles in, three exposure chains out (not four — 102 is deduplicated), built once each, read back by their tiles, cleaned once every consuming tile has extracted its stamps (D5).

D2. Rules call shapepipe_run directly — no wrapper

A rule is one shell line against a committed config, on one unit (tile or exposure), at the existing job-bit boundaries:

SP_UNIT_NUM=-2605805 shapepipe_run -c $SP_CONFIG/config_exp_Ma.ini -b {threads} \
    && completeness.py check exp_mask {output}

Everything the bash layer's per-unit machinery did dissolves into three mechanisms ShapePipe already has (verified against the code):

• Committed configs. Each stage's deterministic output/run_sp_<prefix>/ is the rule's directory() output — Snakemake creates it and deletes it before reruns. Every INPUT_DIR is the literal upstream path (the ngmix configs already work this way), which retires the last: resolver and log_run_sp.txt machinery entirely. NUMBER_LIST = $SP_UNIT_NUM inline.
• Env interpolation. Configs expand $VARS in every path field and fail loudly on unset ones. SP_CONFIG points at the committed config dir (no symlink), SP_RUN at the unit's work dir, SP_UNIT_NUM carries the unit ID.
• Existing CLI. -b {threads} already overrides SMP_BATCH_SIZE.

After shapepipe_run exits, completeness.py check counts outputs against the committed floor table, writes the manifest, and exits by the floor (D3). Hand-running a unit is the same shell line. The two per-unit list files (tile_numbers.txt; the per-exposure list, a find_exposures product) are one-line upstream rule outputs. The on-the-fly mask chain runs with USE_EXT_STAR=False; an external-star variant is one committed INPUT_DIR plus a star-cat download rule.

The one worthwhile module addition, optional: a generic --set SECTION.KEY=VALUE on shapepipe_run (~15 lines at the single config chokepoint) as a uniform override surface. The design does not depend on it. The real migration work is the config sweep: ~13 committed configs rewritten with literal INPUT_DIRs, committed RUN_DATETIME, env'd NUMBER_LIST.

No rule declares a glob. Rows marked (phase A) build during PREPARE; everything else runs in COMPUTE.

Both stores are sharded — tiles/<2-digit-prefix>/<ID>/, exp/<prefix>/<expbase>/ — so no directory exceeds ~1k entries at full-UNIONS scale. Sharding serves humans and the filesystem's metadata server; every path the DAG uses comes from the index. Prefix digits are chosen for uniformity against the real exposure list.

rule	unit	config (example/cfis)	declared output path
tile_get_images+tile_find_exposures (phase A)	tile	config_tile_Git_*, Uz, Fe	run_sp_tile_Git/, run_sp_tile_Uz/, run_sp_tile_Fe/.../exp_numbers-<IDra>-<IDdec>.txt
(index)	run	build_index.py, threshold-gated	index/run_index.sqlite, index/missing.json
exp_get_images	exposure	config_exp_Gie_*	run_sp_exp_Gie/
exp_split	exposure	config_exp_Sp.ini	run_sp_exp_Sp/
exp_mask	exposure	config_exp_Ma_onthefly.ini	run_sp_exp_Ma/
exp_psf	exposure	config_exp_psfex.ini	run_sp_exp_SxSePsfPi/
tile_exp_forest	tile	(symlink curation)	per-tile $SP_EXP view — convenience, not a DAG edge
tile_merge_headers	tile	config_tile_Mh_exp.ini	.../log_exp_headers-<IDra>-<IDdec>.sqlite
tile_detect	tile	uc / sx_nomask / sx+onthefly-mask	run_sp_tile_<Uc|Sx>/
tile_vignets	tile	config_tile_PiViVi_*.ini	run_sp_tile_PiViVi/
tile_ngmix	tile × chunk	config_tile_Ng template, env ID range	run_sp_tile_ngmix_Ng<k>u/
tile_merge_cats	tile	config_merge_sep_cats.ini	run_sp_tile_msc/
tile_make_cat	tile	config_tile_Mc_*.ini	final_cat-<IDra>-<IDdec>.fits

tile_merge_headers output carries the tile suffix because downstream stages consume it via NUMBERING_SCHEME -000-000. tile_detect has three variants (uc, sx_nomask, sx+onthefly-mask); the config selector carries the choice, and DR6 will standardise on one. The data/image_sims axis is kept, but sims parity is a named follow-up; the first pass targets data.

Per-CCD stages declare one manifest file plus the product directory(). Each job writes a small manifest listing sub-units present/missing with failure reasons parsed from the module logs, checked against the committed threshold table (completeness.py). This keeps legitimate CCD attrition (masked, too few stars) below the declared-output layer, where Snakemake would hard-fail it, and keeps declared paths at DR6 in the thousands rather than millions. DES and Rubin landed on the same shape.

NUMBER_LIST is set only for stages whose NUMBERING_SCHEME is the unit ID; the committed configs for *_get_images/exp_mask/exp_psf omit it (a -<ID> entry fails #746 startup validation there, and a 40-entry expansion would turn tolerated per-CCD attrition into a whole-exposure failure).

DAG edges point at real exposure outputs. Rules that read exposure products (tile_merge_headers, tile_vignets, PSF-interp) declare the exposure directory() outputs as input:; tile_exp_forest builds a per-tile symlink view and points $SP_EXP at it purely so ShapePipe configs use one path — a convenience, never an edge.

D3. Failure handling: count floors

completeness.py check — the second half of every rule's shell line — counts outputs against per-runner floors and exits 0 above the floor, nonzero below. Zero output is always fatal. The thresholds are the ported bash complete_check table, committed in completeness.py (stdlib-only, imported by the check and the report); below is its human rendering.

stage	check
tile get_images	get_images_runner:4 (2 for sims)
tile uncompress	uncompress_fits_runner:1
tile find_exposures	find_exposures_runner:1
exp get_images	get_images_runner:6 (3 sims)
exp split	split_exp_runner:121 (40 CCD × 3 + 1 header)
exp mask	mask_runner:40
exp psf	sextractor_runner:80 psfex_runner:80 psfex_interp_runner:40::warn setools_runner:80:rand_split
tile merge_headers	merge_headers_runner:1
tile detect (sx)	sextractor_runner:2
tile detect (uc)	get_images_runner:2 read_ext_sexcat_runner:1
tile vignets (data)	psfex_interp_runner:1 vignetmaker_runner_run_1:1 vignetmaker_runner_run_2:4 (:3 sims)
tile ngmix	ngmix_runner:1
tile make_cat	make_cat_runner:1

:warn means a short count does not fail the unit (expected sparse-CCD PSF loss); :subpath counts a nested dir. So exp_psf tolerates <40 PSF-interp CCDs but fails hard on 0-of-40, a short SExtractor/PSFEx count, or a nonzero exit. Tolerated shortfalls pass, and the manifest records them.

Three mechanisms cover the three real situations:

• Transient (vcp hiccups, node flakiness, OOM): per-rule retries: with attempt-scaled resources (mem_mb = base * attempt). A deterministic failure exhausts retries and fails.
• Hard (real bug): the job exits nonzero. With keep-going: true, everything independent continues; only the failed downstream cone waits.
• Below-floor-but-expected (too few stars on some CCDs): absorbed by :warn, recorded in the manifest. No separate class, no whitelist.

Rerun triggers stay at the v9 default {mtime, params, input, software-env, code}, so a changed config, param, or rule shell line reruns its cone natively.

sp report reads the index and the manifests and emits the success/failure tables — per tile, per exposure, per CCD, with a failure-reason column — the successor to summary_tiles.py. It is a standalone verb, runnable at any time including mid-run, so it can describe failures rather than be blocked by them; onsuccess/onerror handlers call it so every invocation ends with one. Job-level statistics (runtimes, provenance) come from Snakemake's own --report.

D4. Execution: the SLURM executor, one profile

Snakemake submits one SLURM job per rule (--executor slurm), each with its rule's exact resources — cpus_per_task = threads, mem_mb, runtime — and each wrapped in apptainer exec by the SDM. SLURM enforces limits per job, so attempt-scaled retries are real requests, and the cluster backfills around our jobs instead of us renting idle cores. The profile's jobs: cap sits at the cluster's per-user submission limit (queried at P1, not invented); snakemake feeds the queue as jobs finish, so DR6's ~170k total jobs never need to queue at once. Multi-node scaling is inherent. This mode ran end-to-end in a probe on nibi; P0's compute ran in a single-allocation variant, and P1 adopts the executor as the standard mode.

Two consequences to manage:

• Short rules pay queue latency per job. Chunky rules (exp_psf, ngmix) amortize it; the minutes-short ones (uncompress, merges) fuse into their neighbours via group: labels in the profile, tuned at P1/P2. The chains are kept group:-fusable (no mid-chain localrules, no pipe outputs).
• The orchestrator is a light, long-lived process. The full DR6 DAG parses in ~11 min / ~4 GB, so snakemake runs under tmux on the login node or in a 1-core long-walltime shepherd job. Ctrl-C scancels outstanding jobs (executor behaviour); an orphaned run's jobs are one manual scancel.

threads == SMP fork width. A rule's threads must equal ShapePipe's internal fork width, or the SLURM request decouples from the true process count. Enforced mechanically: every rule passes -b {threads}, so the two are one number by construction. mem_mb = threads × measured per-worker footprint (ngmix ≈ 3–4 GB/worker after #843).

ngmix scatter/gather (the fix for the 13-hour straggler chunk):

• Chunking uses Snakemake's native scattergather directive. Chunk count is a parse-time constant, overridable with --set-scatter; N=1 recovers one ngmix job per tile.
• Each chunk's rule computes its ID range at execution time from the tile's own object count: equal shares, closed ranges, no open-ended final chunk (the old root cause: ID_OBJ_MAX=-1 on the last chunk). A params: function cannot do this — params evaluate before the sexcat exists.
• Chunks write nothing shared; each writes its own run_sp_tile_ngmix_Ng<k>u/, and merge_sep_cats — DAG-serialized after all chunks — is the gather.
• Thread/arena hygiene: MALLOC_ARENA_MAX=2, MALLOC_TRIM_THRESHOLD_=0, OMP/OPENBLAS/MKL_NUM_THREADS=1.

RNG. Chunking is only safe with SEED_FROM_POSITION=True (#796): each object's RNG derives from its own (ra, dec, ccd), so results are bit-identical under any chunking — N is a pure throughput knob and every rerun reproduces. It becomes the production mode; the tile-seed mode (one ordered stream across the tile, results dependent on chunk boundaries) is retired. Chunking is on by default: smaller per-job memory and disk high-water mark, better packing, N tuned at P1.

benchmark: on the heavy rules (ngmix, exp_psf) gives per-unit wall/RSS/CPU TSVs — the feed for the measured mem_mb values above.

# profiles/nibi/config.yaml
executor: slurm
jobs: <per-user MaxSubmitJobs>       # query the cluster at P1; do not invent
default-resources: {mem_mb: 2000, runtime: 2h, slurm_account: def-mjhudson}
software-deployment-method: apptainer
apptainer-args: "--cleanenv --env OMP_NUM_THREADS=1 --env MALLOC_ARENA_MAX=2 --env PYTHONPATH=<prod worktree>/src --home /home/cdaley --bind /project --bind /scratch"
keep-going: true
show-failed-logs: true
rerun-incomplete: true
latency-wait: 60
# no rerun-triggers override — full default triggers (D3)

Snakemake wraps every job in apptainer exec itself; the workflow never calls apptainer. Rules point container: at the local /project .sif, never docker:// — no registry pull mid-campaign. This retires the hand-rolled in_container machinery.

The apptainer-args line carries the proven container recipe: $HOME for the CADC cert, thread/arena caps (one openblas pool per worker), and the P0 PYTHONPATH pin at a frozen commit; production rebuilds the sif at the validated commit and drops the pin. The committed launcher (workflow/bin/sp) runs module load apptainer before exec-ing snakemake.

CANFAR is a future profiles/canfar/ wrapping the skaha REST API. The Snakefile is unchanged.

D5. Storage and reclamation

Storage placement. Durable, low-volume products (final_cat, the index, reports, .snakemake/) belong on /project, so scratch purge never resurrects finished-tile reruns. Bulk intermediates live on /scratch, each batch sized to finish inside the ~60-day purge. (/project placement is currently blocked by group quota.)

Disk economy. Un-cleaned intermediates measure ~34 GB/tile — at DR6 (~7300 tiles), beyond any scratch allocation — so intermediates are reclaimed during the campaign. Two tiers, split by who sees each product's full consumer set:

• Intra-tile intermediates → temp(). Their consumer set closes inside one invocation, so native temp() reclaims them when the last reader finishes. --notemp keeps them for debugging.
• Cross-tile exposure store → in-DAG clean_exposure rules. The index lists every consuming tile in the campaign, so one generated clean_exposure rule per exposure takes every consumer's vignette output — the last stage that reads exposure products; everything after it reads tile-level files — as input:, deletes the exposure's intermediates, and leaves a tombstone. Writer, then readers, then cleaner — DAG-ordered, race-free, rolling mid-run while ngmix still grinds; only exposures at the processing frontier stay pinned. It is temp() semantics applied to the campaign's consumer set instead of one invocation's.

A clean: config flag gates the clean rules; flipping it later reclaims retroactively, since the missing tombstones schedule exactly the clean jobs. A tile appended after its exposures were cleaned reschedules those chains — the accepted cost of a late append.

One Snakemake workdir per output root; invocations strictly sequential. All invocations share provenance metadata, and the workdir lock serializes them; the launcher refuses --nolock. Position-seeded RNG (D4) makes every product bit-reproducible, so an unintended recompute costs compute time, not correctness — -n before a real run shows exactly what a code/params drift would rerun.

D6. Repo layout and migration

workflow/
  Snakefile              # parse-time index load; onsuccess/onerror → sp report; includes rule files
  rules/{prepare,exposure,tile}.smk
  scripts/{build_index.py, build_forest.py, run_report.py, completeness.py}
  bin/sp                 # launcher: module load apptainer → snakemake --profile profiles/nibi
profiles/nibi/config.yaml

Pin snakemake>=9,<10 plus snakemake-executor-plugin-slurm. The v8→v9 breaks are minor here: --use-singularity → --sdm, executors became plugins, full rerun-triggers is the default.

The line-count check: four small scripts (index, forest, report, completeness table) plus committed configs replace ~2000 lines across three bash levels, each site's sbatch wrapper, and the run-log machinery. Less code, in one place, testable, with the DAG delegated to a tool whose whole job is DAGs. If an implementation PR does not deliver that, this design has failed.

Phased migration, each phase independently useful, bash layers untouched until Martin retires them (P0/P1 use a separate output tree — no shared exp/ with a bash-driven run during validation):

1. P0 — prepare + exposure stages on a 4-tile P3 subset. Delivered — see the results comment below.
2. P1 — full tile chain through final_cat; retire job_p3_batch1.sh for new runs. Validation is statistical parity against the bash baseline: join on object ID, science columns within tolerances — not bit parity (the RNG mode deliberately changes, D4).
3. P2 — scale via batched invocations sized to scratch purge windows (50-tile → P3 → ~7300 tiles); exercise the count-floor, retries, rerun propagation, and rolling clean_exposure reclamation for real; tune jobs: and group: fusion against measured queue behaviour.
4. P3 — CANFAR profile.

Out of scope for the first pass (each a named follow-up): image_sims parity (the sims im_pipeline step will call these same per-unit rules — the two pipelines converge on one workflow); the CANFAR profile.

Open questions

1. Batch sizing for multi-invocation scaling: how many tile-disjoint sky partitions, how large (bounded above by the 60-day purge, below by exposure-overlap waste at batch borders)? Decided at P2.
2. make_cat over partial tiles: if k of 40 CCDs failed PSF, the tile completes with fewer epochs. Keep that (report records the gaps), or add a quality gate?
3. ngmix chunk configs: one committed template with env-interpolated ID range ($NGMIX_ID_MIN/$NGMIX_ID_MAX, exported by the chunk rule) should replace generated per-chunk configs entirely — confirm the range fields interpolate.
4. Repo layout: workflow/ at top level per Snakemake convention, or under scripts/ next to the bash it replaces?

[cailmdaley]

Round 2 posted — the body above has been rewritten in place (round 1 is preserved in the edit history). What happened between rounds:

1. Cail's feedback on round 1, whose through-line is simplify: one execution mode (the one-big-allocation), one profile, Snakemake owns the containers (--sdm apptainer — nothing in the workflow ever calls apptainer itself), full default rerun-triggers, a single run-level report, snakemake>=9,<10, CANFAR deferred to a future profile.
2. A second research pass that overturned two round-1 assumptions: nibi compute nodes do have outbound internet (verified empirically from a compute node), so downloads run in-DAG; and the failure-handling machinery collapsed to the count-floor that the bash complete_check already encodes — the three-class taxonomy, whitelist, and verdict-classifier are gone. Per-CCD stages now declare deterministic directory() outputs (RUN_DATETIME=False per-unit), with an in-job per-runner count-floor from a committed expected_counts.yaml.
3. A second four-lens adversarial review of the revised design (30 raw findings → 22 verified, 4 blockers), all folded in. The blockers were seam bugs between Snakemake's state model and ShapePipe's: timestamped run dirs vs. declared outputs, the apptainer SDM silently dropping the proven container environment (--home, thread caps), and temp() on shared exposure products destroying the "finished tiles stay finished" invariant on the workflow's most routine operation (appending tiles). That last one is why cleanup is now an explicit index-driven verb instead of temp() on the exposure store.
4. Empirical grounding on nibi: the prepare phase already runs green on a 4-tile P3 subset (deterministic run dirs, run_index.sqlite with 4 tiles / 30 exposures), and both execution paths were probed live (the one-allocation container mechanics, and a slurm-executor probe job that completed end-to-end — kept as a deferred scaling option).

A draft PR with the implementation skeleton follows shortly; this issue stays the living PRD and will be updated as implementation teaches us things. The open questions at the foot are still the discussion agenda.

— Claude (Fable) on behalf of Cail

🤖 Generated with Claude Code

[cailmdaley]

What the first real runs taught (P0 night 1)

The design has now met real data on nibi and delivered. Every compute attempt failed on a genuine defect, each one at the seam between Snakemake's state model and ShapePipe's directory-as-database bookkeeping — never a design error, always a concrete v2.0 mechanism the abstraction hadn't yet been told about. The recovery run then drove the full tile chain through all eight ngmix chunks on every tile: job 17808445 COMPLETED in 3h47, four final_cats on disk, and all four tiles PASS scientific-equivalence validation against the bash baseline. This is what the runs taught, organized by the design section each learning touches.

P0 delivered — four final_cats, validation PASS

• Job 17808445 COMPLETED (3h47): 4/4 final_cats on disk (46/42/47/44 MB, 173 columns). Tile 210.296 = 34,321 rows — exactly its SExtractor object count: the closed-range ngmix chunking covered every object once, no straggler drop, no double-count.
• run_report.json: 4 tiles / 19 exposures, no missing tiles, zero failures at exposure granularity. (Attrition granularity caveat in D3.)
• Append-invariant test PASSED. Appending the 196 quad to the tile list and dry-running scheduled exactly 65 jobs, all for the new tiles (every rule count = 4, ngmix 32 = 4×8) and zero for the finished quad; no exposure jobs until the index is rebuilt (parse-time index doing its job). Run under --rerun-triggers mtime — the first default-trigger invocation would storm the DAG on the script-hash param change (metadata reconciliation, D3).
• Smoke gates green in-job: shapepipe.__file__ = pinned worktree src in both driver logs; OMP pool 32→1; ulimit -u = 3.09 M (no nproc ceiling).
• Validation: all four tiles PASS scientific equivalence to the p3-batch1 bash baseline — table and the PSFEx caveat in the Validation section below.

D1 — Staged execution / the index

The index must carry the exposure's original name, not just the dedup key. The exposure store keeps CFHT's p suffix (2086319p), while the DAG keys exposures by bare base id. The base id is the right dedup key, but get_images needs the original name: the index carries both, and the wrapper writes the original name verbatim into the fabricated per-unit exp_numbers list. This confirms the posted design's own note ("carries both bare exposure ids and original names, 2605805p") — it's now load-bearing in code, not just prose, and the write must be unconditional (an exists-guard had pinned a stale pre-fix copy of the list file across a rerun).

Structural dedup is real and observed. prepare came up green on the 210/211 overlap quad: 34 tile→exposure edges collapsed to 19 unique exposure chains, each scheduled exactly once. The dedup-within-an-invocation contract holds against actual overlap geometry.

D2 — Wrapper fabrication, forest, star-cats, count flavors

Four D2 mechanisms needed correction against the running modules:

1. The exposure forest must be sharded, not flat. exp_utils.get_exp_output_files hardwires the v2.0 sharded layout (forest/<2-digit-prefix>/<base>/output) into its $SP_EXP glob, so a flat forest fails every tile gather stage. The design's flat-store decision remains correct — nothing globs the store, the index drives every path — but the module-facing view the forest presents must carry the shard. This is a refinement of the posted text, not a contradiction: the store is flat, the forest view is sharded.

2. Star cats are consumed as directories, not per-unit files. They're per-CCD directories (~40/exposure) reached via $SP_RUN/star_cat_{exp,tiles} dir symlinks — the v2.0 mechanism the wrapper already promises to reproduce. The per-unit star-cat file rules I had sketched linked names that don't exist and nothing read them; they're deleted. The wrapper's dir-symlink step (finding 11b) is the whole mechanism.

3. get_images completeness counts are config-flavor-dependent. The nibi symlink flavor produces tile 2 / exp 3 files, versus the canfar vos table's 4 / 6. The committed count table now follows the actual configs in use, verified against the p3-batch1 baseline (100 files / 50 tiles). This directly touches the D3 count table (see below).

4. Shell-string placeholders don't reach into params:. {threads}/{output} survive literally into the process if they sit inside a params: value — Snakemake formats the shell: string once. Placeholders must live in shell:. This is a wrapper-plumbing fix, but it validates the D4 insistence that SMP_BATCH_SIZE={threads} be written by the wrapper into the config copy, not passed as a shell placeholder.

D3 — Failure policy, observed live + the quota postmortem

The count-floor works, watched in real job exits. attempt 3's log shows both OK lines (above floor → exit 0) and FATAL breaches (below floor → nonzero) — the P0 failure-policy bullet now has live evidence, not just a ported table.

The report counter needs to be per-CCD, not per-exposure. Validation exposed a granularity gap in run_report: its "zero attrition" is exposure-granularity only. The real signal lives per-CCD — psfex_interp (validation_psf) attrition is 46/760 CCDs = 6.05% (smk) vs 41/760 = 5.39% (bash), ~30× the 0.2% lore figure, on both sides. The run is healthy; the counter is just aggregated one level too coarse to show where the loss happens. This is the DES-blacklist granularity exactly (per-CCD, not per-exposure), so the report should count at that grain. FOLLOW-UP: make the run_report attrition counter per-CCD.

The DES/Rubin precedent is citable. The count-floor + standalone report is the field's converged pattern, and I've verified the citations: DES's per-CCD PSF blacklist (Y3: <25 stars → blacklisted, ~2% of data removed, survey proceeds — Jarvis et al. 2021 §4.2; also 2016, Y6 2025) is our sparse-CCD setools case almost literally; Rubin's Butler quantum registry + pipetask report (Jenness 2022, Lust 2023) is the same "record per-unit, gate on a floor, report and continue" shape at survey scale. Both avoid a workflow engine's per-file targets; Snakemake forces the same choice (#4016, no optional-output primitive). Worth adding these citations to the D3 precedent sentence.

Quota-kill postmortem — two independent failure layers. attempt 3 (job 17798771) computed the full chain of tile 210.296 — all 8 ngmix chunks passed with in-job closed ID ranges (n_obj=34321 → chunk 8 = [30031, 34321], no open-ended last chunk: the straggler fix working) — then died writing Snakemake job metadata, because /project def-mjhudson had gone hard-full (27/27 TiB). Two layers:

• The forest-staleness layer, which is a D3 rerun-trigger finding: three of four tiles FATAL'd at tile_merge_headers on stale flat forests from attempt 2. The forest rule never re-ran, because the code trigger tracks the shell: string but not the external sp_rule.py it invokes — exactly the tradeoff the design accepts to keep wrapper edits from invalidating the whole DAG. The fix folds a wrapper-script content hash into every rule's params:, closing the gap. The one-time cost: a params mismatch on every already-recorded job → a full-DAG storm at the next default-trigger invocation. So the recovery (attempt 4) runs --rerun-triggers mtime deliberately, with the 3 stale forests deleted by hand, and defers metadata reconciliation until after green.
• The quota layer is a real-world instance of D6's storage story, below.

D5 — Profile env, state placement, execution mode

PYTHONPATH pin and container env held up — the smoke gate (attempt 1, job 17794661) confirmed the pinned src resolves and threadpools cap at 1, exactly as D5 specifies.

State placement had to move off /project under duress. The posted design places the Snakemake workdir and index_db on /project (so scratch purge can't break the mtime chain). /project going hard-full forced the opposite tonight: the working repo moved to /home/cdaley/shapepipe-smk (clone @ a273e763, pushed), bin/sp runs Snakemake with --directory <run_dir>-state on /scratch, the Snakefile resolves its configfile relative to itself, and index_db moved to scratch. This is a forced deviation, not a design change — but it means the D6 "durable products + .snakemake/ on /project" placement is currently blocked and should be flagged as such until group-level /project reclamation happens. The design intent stands; the environment doesn't currently permit it.

One-allocation mode delivered — attempt 4 = job 17808445, one deliberate invocation, COMPLETED in 3h47. Confirms the D4 single-allocation model in practice, start to four catalogues.

D6 — Bench numbers, storage reality, precedent

DAG-scale gate cleared, and it reframes P2. Synthetic index (realistic ~2.7 exp/tile overlap), snakemake -n, nibi login, Snakemake 9.23.1, parse-time dict-lookup input functions:

tiles	jobs	DAG build	peak RSS
250	6,689	14.6 s	238 MB
1,000	26,689	77 s	616 MB
2,000	53,357	163 s	1.1 GB
4,000	106,689	351 s	2.1 GB
DR6 (~7,300)	~195k	~11–12 min	~4 GB

Near-linear (~3.3 ms, ~20 KB per job). All-of-DR6 builds in ~11–12 min at ~4 GB — viable on a login node or allocation head. Consequence: the D5/D6 batching-for-scale rationale should say batching is an operational choice (walltime, storage), not a DAG-scale necessity. The group:-fusion escape hatch stays in reserve, unforced.

Storage: ~34 GB/tile of un-cleaned intermediates (smk-p0: 136 G/4 tiles; p3-batch1: 1.6 T/50 tiles). This is the concrete number behind the sp clean-exposures verb — at DR6 the intermediate footprint alone exceeds any single allocation's scratch, so index-driven cleanup isn't optional at scale. And it's exactly what bit us: /project hard-full (27/27 TiB) killed attempt 3's metadata write; scratch is over its soft quota (1693/1024 GiB) even after a 357 G reclaim tonight. The storage story in D6 is now measured, not projected.

---

Validation — scientific equivalence to the bash baseline

All four tiles PASS scientific equivalence to the p3-batch1 bash baseline (verdict: /home/cdaley/scratch-validation/validation_verdict-p0night1.json). Rows are exactly equal on every tile (100% NUMBER crossmatch), column sets are identical, and every deterministic (non-NGMIX) column is bit-identical: SExtractor 760/760 CCD cells exact, setools star counts exact (max delta 0), WCS headers max-diff 0.0 across all 19 exposures. Shape statistics agree — per-object g1/g2 correlation 0.964–0.989, g1/g2/T/SNR/FLUX medians within 1–2%, NGMIX_FLAGS_NOSHEAR==0 at a rate of exactly 1.00000 both sides, all tiles.

tile	rows	g-corr	T rel-Δ	SNR rel-Δ
210.296	34,321	0.988–0.989	0.4%	1.1%
210.297	30,637	0.964	0.2%	0.2%
211.296	35,037	0.983–0.984	0.2%	0.04%
211.297	32,712	0.987–0.988	0.9%	1.7%

Finding — the residual shape differences are PSFEx, not orchestration. The g-correlations sit at 0.96–0.99 rather than 1.00 because PSFEx diverges structurally under the baseline's known mid-run code drift, not because of the Snakemake orchestration. Concretely: bash drops CCD 0 on 6 exposures where smk keeps it; that propagates through NGMIX_N_EPOCH on 0.02–13.1% of objects (tile-dependent) and surfaces as a few 2–2.6σ paired g2/SNR offsets at the ~1e-3 level. Two consequences. (1) FOLLOW-UP: confirm PSFEx config parity between the two code pins before treating the two catalogs as the same data product — parity confirmation is what lets us attribute the whole residual to the drift. (2) Provenance: the smk run's pinned code makes it the better-controlled product of the two — worth one sentence in #848, because it inverts the usual "is the new orchestration trustworthy" framing.

Naming divergence to reconcile

The posted D3 text names the count table two ways: the wrapper's committed table is workflow/scripts/completeness.py (a stdlib-only Python table), while the standalone report is described as reading expected_counts.yaml (D3 final paragraph and the finding-8 report paragraph). Implementation settled on the Python table (completeness.py) as the single source both wrapper and report import in-container. The PRD should pick one name — completeness.py — and drop the expected_counts.yaml references so there's one table, imported, not a YAML the report re-reads.

PRD edits needed (checklist for tightening #848 tomorrow)

• D2, forest bullet — state the store is flat but the module-facing forest view is sharded forest/<2-digit-prefix>/<base> (because get_exp_output_files globs the v2.0 layout). Current text implies the forest is just "convenience"; make the shard explicit.
• D2, exp_numbers fabrication — note the fabricated list must carry the exposure's original p-suffixed name and be written unconditionally (no exists-guard).
• D2, star-cat bullet — the per-exposure/per-tile star-cat outputs are consumed as directories via $SP_RUN symlinks, not per-unit declared files; drop any per-unit star-cat file rule language.
• D2 count table + D3 — the tile get_images:4 (2 sims) / exp get_images:6 (3 sims) rows are the canfar vos flavor; the nibi symlink flavor is 2 / 3. Either parameterize the table by config flavor or state it follows the active configs.
• D3, rerun-triggers section — add the caveat that the code trigger does not cover the external sp_rule.py, so a wrapper-script content hash is carried in each rule's params: to make wrapper edits propagate. (Currently the text implies code on the shell string is sufficient; it isn't for external-script logic.)
• D3 / finding 8 — rename all expected_counts.yaml references to completeness.py; one table, imported by both wrapper and report.
• D3, precedent paragraph — add the DES (arXiv:2011.03409 §4.2) and Rubin (arXiv:2206.14941, 2303.03313) citations.
• D3, attrition counter — make the run_report attrition counter per-CCD, not per-exposure: validation showed exposure-granularity "zero attrition" hides the real per-CCD psfex_interp loss (46/760 smk, 41/760 bash ≈ 6%). Same grain as the DES per-CCD blacklist.
• D3 / metadata reconciliation — record the decision the script-hash param forces: the params mismatch on every already-recorded job means the next default-trigger invocation storms the whole DAG. Pick the reconciliation path (one-time controlled rebuild vs. accepting the storm once) and state it; P0 ran --rerun-triggers mtime to sidestep it, which is not a standing answer.
• Validation, PSFEx parity — before the smk and bash catalogs are called the same data product, confirm PSFEx config parity across the two code pins; the residual shape differences (g-corr 0.96–0.99) trace to the baseline's mid-run PSFEx drift, not orchestration. Note that the smk run's pinned code makes it the better-controlled product.
• D5/D6, batching rationale — reframe batched invocations as an operational choice, citing the bench (DR6 ~195k jobs ≈ 11–12 min / ~4 GB), not a DAG-scale necessity.
• D6, storage — replace projected storage language with the measured ~34 GB/tile.
• D6, state placement — note that /project placement of .snakemake/ and durable products is the intent but is currently blocked by /project being hard-full (27/27 TiB, group-level); state runs off /scratch via --directory until group-level reclamation. This is the storage-placement reality, not the design intent.
• D6 P0 gate list — the DAG-scale/granularity, compute-DAG-green, failure-policy, final_cat delivery, and append-invariant bullets are all now done (job 17808445, 4/4 on disk, validation PASS); the P0 gate is cleared.

Status note for the issue: P0 is delivered — job 17808445 COMPLETED (3h47), four final_cats on disk, all four tiles PASS scientific-equivalence validation against the bash baseline, and the append-invariant regression passed. The open items are the four PRD edits' follow-ups (per-CCD attrition counter, PSFEx config parity, metadata reconciliation, /project reclamation), none of which block the Gate-2 exit. This comment records what the runs proved.

— Claude (Fable) on behalf of Cail

🤖 Generated with Claude Code

[martinkilbinger]

Looks nice!

Comments:

• Level 3 (curl_canfar_local.sh) this is obsolete and already no longer used (was used up to the skaha python canfar update in 2025).
• Setting RUN_DATETIME=False is fine as long as Snakemake handles re-runs of failed modules (shapepipe would exit with an error that output dir already exists)
• exposure index file: Is it flexible that exposures can be added if we run more tiles? E.g. do a test run of a few tiles, then add more for a larger run?
• tiles/exposures/tile_exposures: I am still a bit worried that having all tiles (exposures) in one root dir will overload the file system. Even with index sql files we still want to type ls, find, wc, etc. Are or order 50k subdirectories ok? And what is tiles_exposures?
• star_cat_exp, star catalogue for masking: This will be no longer used once healsparse masks replace ShapePipe-internal masking
• config_merge_sep_cats.ini: running ngmix as parallel jobs on object batches and the merging in the next module has been removed for early v2.0 runs; this was fairly unstable and difficult to manage in v1.4. Make sure this is done properly in the new Snakemake setup.
• I'd like to implement a global statistics of tiles and exposures, how many succeeded and how many failed, and for what reason. See previous script shapepipe/scripts/python/summary_tiles.py.
• When running tiles in parallel, the code has to make sure that not more than one job accesses the same exposure; this should be already implemented in v2.0, but check.

[cailmdaley]

Thanks Martin — great set of questions, several of them sharpened the design. Point by point:

Level 3 (curl_canfar_local.sh) obsolete. Good to know — removed from the PRD; the survey notes keep the history.

RUN_DATETIME=False and re-runs of failed modules. Handled, and more simply than P0's implementation: RUN_DATETIME=False is committed in the configs, so every stage's run dir is a deterministic path, declared as the rule's output — Snakemake itself deletes and recreates it before every re-run, so a retried job never sees "output dir already exists." (No wrapper needed here; more below.) Exercised for real — the P0 run took four attempts and every re-run went through cleanly.

Exposure index — adding tiles later. Yes, and it's tested on real data: after the 4-tile P0 run went green, we appended 196 tiles and re-invoked — Snakemake scheduled exactly the 65 jobs belonging to the new tiles, zero for the finished quad. The index rebuild is incremental by construction, and it's folded into the single sp run entry point, so "add tiles, run again" is the whole workflow.

Flat tiles/ and exp/ directories at ~50k entries. Agreed — the stores get 2-digit-prefix sharding (exp/<prefix>/<expnum>/, tiles/<prefix>/<tile-id>/), same idea as v2.0's layout. Snakemake never globs the stores (every path comes from the index), so the shard is purely for humans and the filesystem's metadata server; at full-UNIONS scale no directory exceeds ~1k entries, comfortably ls/find-able. We'll pick the prefix digits for uniformity against the real exposure list (sequential numbering can cluster; trailing digits if so).

tile_exposures is not a directory — it's the tile↔exposure join table in the sqlite index (each tile → its ~7–10 covering exposures). Renamed nothing on disk; the PRD will say "DB table" explicitly.

star_cat_exp obsolete once healsparse masks land. Turned out to be simpler than that: the committed on-the-fly mask configs (USE_EXT_STAR = False) never read the star-cat rules or symlinks at all — they're unused in the current chain. Dropped from the design outright rather than kept transitional; nothing depended on them. An external-star variant, if it's ever wanted, is a single INPUT_DIR pointing at the shared cat pool — no per-unit plumbing.

ngmix parallel chunks + merge. Reinstated using Snakemake's native scattergather directive, with the v1.4 failure modes removed: chunks compute closed ID ranges in-job from the tile's sexcat (no open-ended last chunk) and write nothing shared — the merge is a plain gather and the single sync point; a failed chunk retries alone. Chunks still parallelize internally via SMP, and chunk count is a runtime knob (--set-scatter; N=1 recovers one ngmix job per tile). Validated row-exact against the unchunked bash baseline on all 4 P0 tiles.

Global tile/exposure statistics (summary_tiles.py). Job-level stats (ran/succeeded/failed, runtimes) come free from Snakemake's metadata and --report. Below job level, each job writes a small manifest file (sub-units present/missing, with reasons parsed from the module logs), and sp report aggregates manifests into the summary_tiles.py-style table — per-tile, per-exposure, and per-CCD, with a failure-reason column.

Same-exposure concurrency. This stops being a locking problem entirely: each exposure's module chain is a single node in the DAG (one job per exposure, no matter how many tiles need it), and tiles consume its products as declared input: — writer-before-readers by construction, no lock needed. The DAG edge is the lock. Observed in P0: 34 tile→exposure needs deduplicated to 19 exposure chains, each built exactly once.

— Claude (Fable) on behalf of Cail