-
-
- 2026 · 06 · 05
- ·sp_validation
- ·restructuring · autonomous checkpoint
-
-
- Restructuring sp_validation.
- One organizing principle — the things you run live at the top — a clean three-way split between analysis, papers, and scratch, and a modular workflow built for more than one person.
-
-
-
-
- § Current state · interactive session · 2026 · 06 · 05
- Decisions settled; the rest of the suite is on a Codex shuttle.
-
-
Cail attached and we did the testing pass together. The decisions below are made (see § Decisions); the four remaining back-pressure guards (②③⑤⑥) are dispatched to an autonomous Codex shuttle — the back-pressure-suite child fiber — so the live session stays free for design (workflow modularity, Phase-2 move sequencing). The Codex worker shares this tree but is constrained to add-only test files; the driver stays hands-off-git while it runs. Phase 2 (the moves) begins once the suite lands and we tidy the branch.
-
-
Foundation folded + back-pressure begun (the autonomous checkpoint that opened this session): branch cleanup/restructuring is pushed; PR #188 is reopened and carries the real implementation (its branch restructuring/proposal was fast-forwarded to it). develop is untouched — propose-never-commit held.
-
# cleanup/restructuring — 5 commits on top of develop (574f4bf)
-e993b9c test: back-pressure guard #1 — imports / first-party resolution
-fdd9008 chore: restore glass-mock cosmosis-config ignores dropped in merge
-1a8b786 chore: gitignore generated apidoc stubs and uv.lock
-d14fc07 fix: add mnu to planck_defaults so bare get_cosmo() resolves
-dd8fc68 Merge sachaguer/develop foundation (PR #192 head, c22f075)
-
Foundation (item 1) — in. Sacha's sachaguer:develop (120 files: paper plots, harmonic configs, the cosmo_val.py/cosmology.py library changes) is merged. The lone blocker — cosmology.py KeyError: 'mnu' (planck_defaults had no mnu key, killing every bare get_cosmo()) — is fixed by one line, and test_cosmology.py passes 26/26 in the container. The .gitignore conflict was resolved toward develop: sacha's broad bans (*.png *.sh *.fits *.out *.err) were rejected, not adopted.
-
Back-pressure guard #1 (your #1) — green. tests/test_imports.py: 42 passed, 1 xfailed in the container. It imports every library module for real and resolves every sp_validation.* import named by the standalone scripts/ (via ast, no execution — several scripts run work at import, one isn't even a valid module name). This is the first thing to go red on a bad git mv.
-
Clean base. Generated apidoc stubs (docs/source/sp_validation.*) and the never-tracked uv.lock are now gitignored; the merge had silently un-ignored ~700 generated glass-mock cosmosis configs (sacha dropped develop's patterns) — restored. Working tree matches develop's baseline.
-
-
-
- § Findings
- What the guards and the survey turned up.
-
- - A dead script, caught immediately.
scripts/plot_leakage.py does from sp_validation.correlation import * — a module that has never existed in this tree. Guard #1 flagged it on first run; marked xfail(strict=True) so the baseline is honest and the marker flips the moment it's fixed or deleted. Triage (fix vs delete) belongs to the scripts/ curation pass.
- - Path breakage is this reorg's dominant risk.
117 hardcoded absolute paths (/home/…, /n17data/…, /automnt/…) live in tracked .py/.yaml/.ini — concentrated in the paper-plot dirs (catalog_paper_plot, 2D_harmonic_space…, 2D_bmodes_paper_workflow/scripts) and scripts/. (Up from the ~35 the plan estimated — Sacha's folded code added more.) Cail's call: we don't rewrite these to repo-relative — they stay messy. The realistic guard is the Candide-local existence checker (③): assert the paths exist where this runs, rather than chasing a clean results/.
- - No console-scripts.
pyproject.toml declares no [project.scripts], so invariant #2 (entry-point resolution) is a no-op here — folded into guard #1's coverage rather than built separately.
-
-
-
-
-
-
- § Back-pressure suite · finalized with Cail
- One guard green; four more on a Codex shuttle, before a file moves.
- The principle (yours): write characterization guards that are green on the pre-reorg tree, then do the move in small git mv steps keeping every guard green. The suite was refined live — simpler and more honest than the first sketch:
-
- - ① Imports / first-party resolution — GREEN, done.
tests/test_imports.py walks the package and scripts/. The model for the rest.
- - ② Snakemake dry-run passes — on the shuttle. Not a golden/identity test — just
snakemake -n exits 0. We don't care about the weather; it's fine if rules change. (Codex probes first — bmodes may be broken or slow; if so that's signal, reported not hidden.)
- - ③ Config-path existence (Candide-local) — on the shuttle. The reframe of the dead path-translation idea: load every config, assert each referenced path exists on this machine. Skips cleanly off-host.
- - ⑤ Symlink integrity — on the shuttle. Every tracked symlink resolves. Cheap; moves silently break these.
- - ⑥ Dangling-reference grep — on the shuttle (harness). Parameterised by a move-map that's empty now (trivially green); activates the moment moves begin.
-
- Dropped, with reasons: entry-point resolution (no [project.scripts]); no-work-lost content manifest (too strong — brutal deletes); package-install (CI already builds the image + runs pytest); output-schema/value invariance (needs cluster data, reference run unreliable, “tests specific code” — deferred until that code changes).
-
-
-
-
-
- § Decisions — settled this session
- The three open calls, made; branch tidy deferred on purpose.
-
- - Branch topology — deferred until the tests land.
cleanup/restructuring and restructuring/proposal are duplicate names for one line of work (#188's head is restructuring/proposal). Cail wants one clean branch = Sacha's merge + restructuring on top, no work lost; names don't matter. Doing it now would break the Codex shuttle's branch, so it waits. Then: collapse to one, close #189 (foundation stub, superseded by folding into #188), keep #190/#191 as tomography stubs.
- - uv.lock — left gitignored. Cail lukewarm; container is canonical. One-line flip later if we want pinned-dep reproducibility.
- - Paths — existence-checker, not translation. The 117-absolute-path sweep is off; they stay messy; guard ③ checks existence on Candide instead.
-
-
-
-
-
-
- § The shape
- Runnable things become top-level siblings.
- Today cosmo_val is buried inside notebooks/ while cosmo_inference/ is top-level — so you constantly hunt for where each one lives. The fix: the things a person actually runs sit side by side at the top, sharing library code in src/ underneath.
-
-sp_validation/
-├── src/sp_validation/ library code (+ glass_mock core)
-├── cosmo_val/ validation: code + config (promoted from notebooks/)
-├── cosmo_inference/ inference: code + config (cosmosis / cosmocov)
-├── workflow/ ALL analysis — modular Snakemake, multi-person → results/
-├── papers/ final-figure assembly only (PDF, colour, layout)
-├── scripts/ real reduction scripts (catalog builders, masking)
-├── scratch/ per-person — ad hoc work + personal workflows (tracked)
-├── notebooks/ curated to official demos / tutorials
-├── results/ analysis products + diagnostic plots (contents gitignored)
-└── docs/ tests/ config/
-
-
- new / restructured
- kept (some curated)
- deleted: defunct/, scratchy notebooks
-
-
-
-
-
-
- § Division of labor
- Analysis, papers, scratch.
-
-
-
-
workflow/
-
All analysis. Generic, reusable, modular, organized for multiple people. Makes diagnostic plots too — they go to results/. The bulk of the work lives here.
-
-
-
papers/
-
Final-figure assembly only — the figure PDF, colours, layout, recombining data for presentation. Tied to one paper, and may never touch Snakemake.
-
-
-
scratch/
-
Personal and ad hoc. Experiments and one-off custom workflows. Tracked, because seeing each other's scratch is useful.
-
-
The line: everything up to the inputs of a paper figure is analysis, and belongs in the workflow. The figure itself is presentation, and belongs to the paper.
-
-
-
-
-
- § How the workflow scales
- Modular, not monolithic.
- Nothing in this analysis is computed once — the catalog changed ~20× in the first release suite, and every paper varies the data vector, covariance, and inference. So the workflow is parameterized: the rules are shared, the config changes each time. Snakemake's module directive imports the rules under your own config and an output prefix, and lets you override any single rule:
-
-module analysis:
- snakefile: "../../workflow/Snakefile"
- config: config # this run's catalog, cuts, blind
- prefix: "results/bmodes" # products land here — no clobbering
-
-use rule * from analysis
-# swap is per-rule: redefine just the data-vector rule to override it
-
- One top-level results/; each run namespaces under results/<name>/ via the prefix, so people don't clobber each other. A --dry-run on each composition is the safety net that lets the structure grow without silent breakage.
-
-
-
-
-
- § Cleanup
- Delete, curate, discipline.
-
- - Delete
defunct/ (quarantined since 2024) and the exploratory 2021–22 notebooks. It all stays in git history.
- - Curate
notebooks/ to official demos and tutorials; personal scratchy ones move to scratch/.
- - Discipline via tooling, not bans:
nbstripout strips notebook outputs on commit (the repo's weight today is committed notebook outputs), plus a pre-commit size hook.
- - Path translation — collecting the paper dirs breaks ~35 hardcoded absolute paths; a mechanical sweep rewrites them (scripts included) to the single repo-relative
results/.
-
-
-
-
-
-
- § The milestone
- A suite of PRs, in sequence.
-
- - Foundation — Sacha's pending code. folded into #188
- - Restructuring — proposal and implementation. this PR · #188 · foundation in, guard #1 green
- - Glass mocks → tomography. #190 · stub
- - Input pipeline → tomography. #191 · stub
-
- #188 is no longer proposal-only: the foundation is folded in and the back-pressure suite has started. The remaining moves (Phase 2) proceed in small git mv steps behind green guards, and were deliberately not forced unattended — see the open questions above.
-
-
-
-
-
diff --git a/.felt/sp-validation-restructuring/restructuring-docs/restructuring-docs.md b/.felt/sp-validation-restructuring/restructuring-docs/restructuring-docs.md
deleted file mode 100644
index 894350ad..00000000
--- a/.felt/sp-validation-restructuring/restructuring-docs/restructuring-docs.md
+++ /dev/null
@@ -1,13 +0,0 @@
----
-id: 01KV8CY7DE98WZ4PCQPFYZZYXM
-name: 'Document the new layout: READMEs + live Sphinx docs'
-status: closed
-tags:
- - reorg
- - sp-validation
- - docs
-created-at: 2026-06-16T14:21:35.022384605Z
-updated-at: 2026-06-16T14:36:25.659158626Z
-closed-at: 2026-06-16T14:36:25.65915811Z
-outcome: 'DONE on cleanup/restructuring. Layout now visible in-tree: structural READMEs for workflow/papers/cosmo_val/scripts/results (matching scratch/), a ''Repository layout'' section in the top-level README, and docs/source/repository_structure.rst wired into the Sphinx toctree so it deploys to gh-pages. Reflects the final shape (notebooks/ gone, glass_mock folded into src/).'
----
diff --git a/.felt/sp_validation.md b/.felt/sp_validation.md
deleted file mode 100644
index 4c709722..00000000
--- a/.felt/sp_validation.md
+++ /dev/null
@@ -1,9 +0,0 @@
----
-id: 01KTS1935KZXP7D7FMH1YPSXFZ
-name: 'sp_validation: shear catalogue validation & cosmology inference'
-tags:
- - sp-validation
- - unions
-created-at: 2026-06-10T17:09:11.731447882+02:00
-outcome: 'Root: CosmoStat/sp_validation — Sacha+Martin+Cail shared repo; restructuring (#197) in flight, tomography extensions (#190/#191) queued behind it.'
----
diff --git a/.felt/sp_validation/reserved-sasha/reserved-sasha.md b/.felt/sp_validation/reserved-sasha/reserved-sasha.md
deleted file mode 100644
index 014de4b0..00000000
--- a/.felt/sp_validation/reserved-sasha/reserved-sasha.md
+++ /dev/null
@@ -1,11 +0,0 @@
----
-id: 01KVHAG0FSYA0Y6F5JDGTT6VNR
-name: 'Reserved: Sasha''s sp_validation work (don''t touch)'
-tags:
- - sp-validation
- - reserved
- - guerrini
-created-at: 2026-06-20T03:32:01.913796095+02:00
-updated-at: 2026-06-20T03:32:01.913796095+02:00
-outcome: 'Hands-off zones owned by Sasha (Sacha Guerrini) on sp_validation: scratch/guerrini/ entirely, and the namaster_utils → source migration + Gaussian-sims de-duplication, which he reserved for HIS NEXT PR in the #197 review. Future workers/agents: do not refactor these; only mechanical import-path updates forced by a library change are OK (and should be flagged).'
----
diff --git a/.github/workflows/lint.yml b/.github/workflows/lint.yml
index c2be73bd..a0f36765 100644
--- a/.github/workflows/lint.yml
+++ b/.github/workflows/lint.yml
@@ -1,21 +1,34 @@
name: Lint
-# Lint gate for `develop` — warn locally, account here.
+# Lint gate for `develop` — warn locally, fix-and-account here.
#
-# The model (settled with Cail + Sacha, 2026-06-23/30):
+# The model (settled with Cail + Sacha, 2026-06-23/30; autofix added 2026-07-10):
# * Locally, ruff auto-applies safe fixes and only WARNS on the rest (see
# .pre-commit-config.yaml) — you commit freely.
# * Getting into `develop` is gated. This job runs the FULL ruff policy
-# (`ruff check .` + `ruff format --check .`, region-aware per pyproject.toml)
-# and on any failure it (1) fails the job RED so the check blocks the merge,
-# and (2) tells the author what to fix — in the surface that fits the event:
-# - On a PR → posts/updates a COMMENT on the PR itself, with the full
-# violation list (and ruff annotations in the run's Checks view). The
-# author sees it where they already are; no disconnected issue. The
-# comment turns green when ruff passes.
-# - On a direct push to develop → there's no PR to comment on, so it opens
-# (or updates) ONE lint-debt issue for the committer, @-mentioning and
-# assigning them. Auto-closes when their next push is clean.
+# (`ruff check .` + `ruff format --check .`, region-aware per pyproject.toml).
+# What happens next depends on the event:
+#
+# - On a same-repo PR (head branch lives in THIS repo, not a fork) → the gate
+# doesn't just report; it FIXES. If the first pass isn't clean it runs
+# `ruff check --fix-only` + `ruff format`, commits the diff as the
+# github-actions bot, and pushes it back to the PR branch. Then it re-runs
+# ruff on the fixed tree IN THE SAME RUN and gates on THAT: if formatting +
+# safe fixes cleaned everything → job GREEN, comment says autofix was
+# pushed; if anything survives (unsafe/judgement lint — undefined names,
+# unused vars) → job RED, comment lists ONLY the residual (the mechanical
+# stuff is already fixed). Contributors mostly never touch ruff by hand.
+#
+# - On a fork PR (read-only token, can't push to the fork's branch) → the old
+# behaviour: run the checks, and on failure (1) fail RED so the check blocks
+# the merge, (2) post/update a COMMENT on the PR with the full violation
+# list. The comment turns green when ruff passes.
+#
+# - On a direct push to develop → there's no PR to comment on, so on failure
+# it opens (or updates) ONE lint-debt issue for the committer, @-mentioning
+# and assigning them. Auto-closes when their next push is clean. (No autofix
+# here — pushing a bot commit onto develop would be its own event.)
+#
# * If ruff itself can't run (network, bad version), the job goes red but says
# nothing — that's infra, not the committer's lint debt.
#
@@ -23,17 +36,25 @@ name: Lint
# branches stay quiet (too noisy otherwise). `workflow_dispatch` is for manual
# testing of the gate itself.
#
-# Why `pull_request_target` for PRs: commenting on a PR needs a write token, and
-# PRs from forks (e.g. sachaguer/) get a read-only token under the plain
-# `pull_request` event. `pull_request_target` runs this workflow from the BASE
-# branch (so the workflow definition is trusted) with a write token, while we
-# check out the PR head ONLY to lint it. Two hardening measures make running
-# tooling over untrusted PR code safe here: ruff is a static analyzer (it parses
-# files, never imports/executes them), and `uvx --no-config` makes uv ignore any
-# `uv.toml`/`[tool.uv]` in the PR tree, so a malicious PR can't redirect ruff's
-# download to a trojaned index. The checkout also drops its git credentials.
-# (A PR can still edit `[tool.ruff]` to weaken its own policy, but that's visible
-# in the diff and reviewed like any other change.)
+# Why `pull_request_target` for PRs: commenting on (and pushing to) a PR needs a
+# write token, and PRs from forks (e.g. sachaguer/) get a read-only token under
+# the plain `pull_request` event. `pull_request_target` runs this workflow from
+# the BASE branch (so the workflow definition is trusted) with a write token,
+# while we check out the PR head ONLY to lint it. Two hardening measures make
+# running tooling over untrusted PR code safe here: ruff is a static analyzer (it
+# parses files, never imports/executes them), and `uvx --no-config` makes uv
+# ignore any `uv.toml`/`[tool.uv]` in the PR tree, so a malicious PR can't
+# redirect ruff's download to a trojaned index. The checkout also drops its git
+# credentials. (A PR can still edit `[tool.ruff]` to weaken its own policy, but
+# that's visible in the diff and reviewed like any other change.)
+#
+# Why the autofix push is safe under pull_request_target: we run ONLY ruff over
+# the untrusted tree (static, never executes PR code), and we push back ONLY the
+# diff ruff itself produced — no PR-authored script runs with our write token.
+# The push uses the workflow token explicitly (the checkout keeps
+# persist-credentials: false), and a GITHUB_TOKEN push does NOT trigger a new
+# workflow run — so no recursion, but also no fresh CI on the bot commit, which
+# is exactly why we re-lint and gate in THIS run rather than waiting for a rerun.
on:
push:
@@ -48,7 +69,7 @@ concurrency:
cancel-in-progress: true
permissions:
- contents: read
+ contents: write # push ruff autofix commit back to a same-repo PR branch
issues: write # develop-push lint-debt issue
pull-requests: write # PR lint comment
@@ -67,6 +88,19 @@ jobs:
- name: Install uv
uses: astral-sh/setup-uv@v3
+ # Is this a PR whose head branch lives in THIS repo (not a fork)? Only then
+ # can we push an autofix commit back to it with the workflow token.
+ - name: Decide whether autofix can push
+ id: mode
+ shell: bash
+ run: |
+ if [ "${{ github.event_name }}" = "pull_request_target" ] && \
+ [ "${{ github.event.pull_request.head.repo.full_name }}" = "${{ github.repository }}" ]; then
+ echo "autofix=true" >> "$GITHUB_OUTPUT"
+ else
+ echo "autofix=false" >> "$GITHUB_OUTPUT"
+ fi
+
# Run the checks WITHOUT failing the step — we post feedback before turning
# the job red. `ruff@