Smokescreen-fork blinding: per-part-at-birth blind, hash-commitment custody, CAMB↔CCL cross-check (PR 6)#253
Draft
cailmdaley wants to merge 7 commits into
Draft
Conversation
This was referenced Jul 10, 2026
cailmdaley
changed the title
cailmdaley
force-pushed
the
feat/sacc-6-blinding
branch
from
July 10, 2026 22:52
971767b to
11b8895
Compare
cailmdaley
changed the base branch from
feat/sacc-5-firecrown-likelihood
to
feat/sacc-2-sacc-io
July 10, 2026 22:52
cailmdaley
changed the title
cailmdaley
added a commit
that referenced
this pull request
Jul 11, 2026
…ll image from lock (#266) * felt: fork-implementation — #243 green, #253 rework under fix; fork PRs reviewed Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KpaRHk3QwN13myduQ3hJyf * felt: fork-implementation — constitution absorbs #241 re-rulings (one-file layout, per-part blinding at birth) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KpaRHk3QwN13myduQ3hJyf * deps: declare cosmo_numba + numba, adopt committed uv.lock, install image from lock Make sp_validation's environment reproducible so an image build can never re-resolve numpy past numba's ceiling — the drift that silently upgraded numpy to 2.5 and broke numba/ngmix. - Declare cosmo-numba (aguinot/cosmo-numba@main; not on PyPI) — the numba B-mode kernels b_modes.py imports. main carries numpy-2 FFT support via its rocket-fft dep and declares its own deps, so numba's numpy window reaches the resolver. Also pin numba directly: the one load-bearing constraint, made visible and resilient to cosmo-numba's dep metadata (which has emptied out between refs). - Declare the other imported-but-undeclared deps the audit found: matplotlib, pandas, pyyaml (core, src/); fitsio (glass extra); a new `workflow` extra for snakemake + mpi4py. cv_runner and unions_wl left undeclared (no resolvable source) with a NOTE. - requires-python and ruff target -> 3.12 (the container's Python; cosmo-numba's floor). - Commit uv.lock (un-ignored) as the SSOT; scope it to Linux via [tool.uv]. numpy resolves to 2.4.6, inside numba 0.66's <2.5 window. - Dockerfile installs from the lock: `uv sync --frozen --inexact` into the base image's /app/.venv (--inexact keeps the ShapePipe stack). Drops the ad-hoc snakemake layer and the cs_util `--upgrade` workaround — the lock pins cs_util 0.2.2 (with cs_util.size), decoupling us from the base image's cs_util. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Vtw1qcgTrQ6YuMvxwYzNup * felt: uv-lock-cosmo-numba — @main resolution, install model, cosmo-numba gotcha finding Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Vtw1qcgTrQ6YuMvxwYzNup * test: don't load base image's stale pytest-pydocstyle/pycodestyle plugins uv sync installs a newer pytest than the base ShapePipe image's pytest-pydocstyle/pytest-pycodestyle (>=2.4) expect; their pytest_collect_file hooks use the removed `path` arg and crash collection. sp_validation lints with ruff, so disable both via addopts (`-p no:`). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Vtw1qcgTrQ6YuMvxwYzNup --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
cailmdaley
force-pushed
the
feat/sacc-2-sacc-io
branch
from
July 16, 2026 00:39
55cb97d to
4489dd4
Compare
…abulary
Rebuild the per-part-at-birth blinding stack on top of feat/sacc-2-sacc-io.
sacc_io.py is now PR-2's canonical module verbatim + a single appended
gather(): the terminal assembly delegates its tracer/point/covariance
assembly to PR-2's merge() and adds only the blind-custody call
(assert_consistent_blind) and shared-stamp write.
The fail-closed load gate lives in sacc_io.load() per the PRD ("sacc_io fails
closed on load"); the blinding tooling uses allow_unblinded=True explicitly
where it must read a not-yet-blinded real vector (blind_part). save() now
carries the required type= (inherited from each part's provenance). Grid
vocabulary swept coarse->reporting, fine->integration across blinding.py,
blinding_theory.py, b_modes.py, blind_data_vector.py, and the blinding tests.
_extract_block/_set_values stay index-based (a code comment says why):
Smokescreen's ConcealDataVector aligns theory_fn output to the sub-SACC mean
by row position, so the block must be carved/written by contiguous integer
index, not by PR-2's tag-matching extract()/update_statistic().
Escrow/custody/CAMB<->CCL machinery is preserved exactly.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WzUt7VbtXwr2SCHUdiQTyt
…nfig Three integration-drift fixes surfaced by the reconciled base: - test_ac6_ac8 / test_ac8_dotted: the end-to-end fixtures now pass type="mock" to sio.save (PR-2 requires the provenance stamp); the parts are mocks, blind_part re-saves inheriting that type. - test_ac13: the CosmoSIS halofit config moved to cosmo_inference/cosmosis_config/templates/ on develop; point the AC13 assertion at the new path (token unchanged: mead2020_feedback). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WzUt7VbtXwr2SCHUdiQTyt
cailmdaley
force-pushed
the
feat/sacc-6-blinding
branch
from
July 16, 2026 09:55
4b058f7 to
c09b063
Compare
cailmdaley
added a commit
that referenced
this pull request
Jul 21, 2026
Fold the integration-grid ξ± into the single terminal {version}.sacc, and
make part loading fail closed on unblinded real data.
Integration ξ± (grid='integration') is no longer its own terminal product.
The xi_highres part is now gathered by rule assemble_sacc into {version}.sacc
as tagged points, next to the reporting ξ± block. It is fiducial-only (the
10k-bin MPI run emits only the fiducial part), so it joins the fiducial
version's terminal file alone. assemble_sacc.py adds xi_integration to
CANONICAL; its own DiagonalCovariance passes straight through.
assemble_sacc.py no longer loads every part with allow_unblinded=True. The
run type (data|mock, from config, default data) gates it: mock runs load
freely, data runs fail closed unless a part carries the concealed=True stamp.
This is the seam for PR #253's blind-at-birth — a concealed data part then
assembles with allow_unblinded=False untouched.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EaL7prmKUHwJQcDyW3LoxD
… feat/sacc-6-blinding
Author the Snakemake rules that realise sp_validation.blinding's three-verb custody surface on the migrated cosmo_val workflow (issues #247/#252). - rule blind_init (once per catalogue version): draws the seed, publishes commitment.json + the encrypted seed bundle. Refuses to overwrite state. - rule blind_part (generic over the three blindable stems — reporting ξ±, integration ξ±, analysis pseudo-Cℓ): conceals the part at birth, escrows the true vector beside the blinded output. The plaintext part is a temp() output of its producer on a data run, so only its blinded sibling persists. - common.py: run-type-aware path helpers (blindable_part / maybe_temp / blind_state_paths / version_of) so a data run binds ξ-derived consumers to the blinded siblings and a mock run bypasses blinding entirely — the whole blinding subgraph appears or vanishes with RUN_TYPE. - assemble_sacc: assert_consistent_blind across parts (assembly-time commitment check of #252), stamping the shared blind on the terminal file. - Constrain the npatch wildcard to \d+ so a producer's ξ± output cannot absorb the _blinded suffix (which made rule xi ambiguous with blind_part). The cosmo_val assemble DAG dry-runs: blind_init x2, blind_part x5, assemble consuming the blinded ξ± + pseudo-Cℓ parts. Born-blinded COSEBIs/pure-E/B and the ρ/τ concealed pass-through follow in the next commit. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EaL7prmKUHwJQcDyW3LoxD
|
Check out this pull request on See visual diffs & provide feedback on Jupyter Notebooks. Powered by ReviewNB |
- common.py path helpers mirror sp_validation.blinding init_paths/part_paths (drift guard), version_of on all three blindable stems, run-type switch. - Data-run fail-closed assembly: assemble_sacc refuses an unblinded type=data part, passes when every part is concealed under one commitment, and refuses a blinded/plaintext mix and divergent commitments. - Candide-only: the blinding subgraph (blind_init/blind_part) resolves in the cosmo_val assemble dry-run and binds the blinded ξ± + pseudo-Cℓ siblings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EaL7prmKUHwJQcDyW3LoxD
cailmdaley
added a commit
that referenced
this pull request
Jul 21, 2026
Keep the integration-grid ξ± as its own per-part intermediate
({version}_xi_integration.sacc) rather than folding it into the terminal
{version}.sacc. Per the #247 ruling (comment 5033716753), the terminal file
carries the analysis vector only; COSEBIs/pure-E/B consume the integration
part directly, and Snakemake provenance covers its traceability. This keeps
the terminal file at tens-of-MB scale.
Removes xi_integration from CANONICAL, the cv_xi_integration_sacc helper, and
the fiducial-gated assemble input. Keeps the fail-closed allow_unblinded gate,
glass A/B/C comments, and dependency restorations from the prior rework. The
integration part stays blinded at birth on data runs (per #253).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EaL7prmKUHwJQcDyW3LoxD
Merge the updated feat/sacc-4-cosmo-val-sacc, which removes the integration-grid
ξ± from the terminal {version}.sacc gather (per the #247 ruling: it persists as
its own per-part intermediate, consumed by COSEBIs/pure-E/B, with Snakemake
provenance covering traceability).
Blinding wiring: the blind_part rule and its BLINDABLE_STEM regex still cover the
integration ξ± (it remains blindable at birth as a per-part file). Only the
assemble subgraph changes — cv_assemble_inputs no longer requests the blinded
integration part, so the dry-run wiring test asserts _xi_integration_blinded.sacc
is NOT pulled into the assemble subgraph.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EaL7prmKUHwJQcDyW3LoxD
cailmdaley
added a commit
that referenced
this pull request
Jul 21, 2026
…-version xi_highres COSEBIs and pure-E/B now derive their E-mode outputs from the born-as-SACC ξ± parts on disk instead of only the raw catalogue recompute. This makes born blinding automatic downstream: whatever the part carries (blinded on data runs in PR #253) flows into the E-modes with no blind-aware consumer code. b_modes: add the values-only seams cosebis_from_xi and pure_eb_from_xi (copied from the sacc-6b spur, de-blinded) — same cosmo_numba kernels as the raw path. cosebis_to_sacc_part / pure_eb_to_sacc_part: add en_override / eb_override so the E-mode En (COSEBIs) and the six pure-mode arrays (pure-E/B) written to the SACC part come from the consumed part; Bn and the covariance stay blind-invariant from the raw estimator run. No concealment machinery (that is PR #253's). cv_cosebis.py / cv_pure_eb.py: keep the raw plot_* run (Bn + jackknife covariance need the catalogue and are blind-invariant), then load the integration part (COSEBIs) / reporting + integration parts (pure-E/B), re-derive the E-modes through the seams, and override both the SACC part and the diagnostic npz. pure-E/B reads the reporting-grid bin edges from the raw reporting gg (SACC stores centers only). Unconditional and version-agnostic. xi_highres: per version (was fiducial-hardcoded); in-container single-process at the config-driven 1000-bin grid (the 10k-bin bare-host MPI path is unnecessary). Grid comes from a dedicated cosmo_val.integration block ([0.08, 300] @ 1000) so the one part serves both consumers (pure-E/B full range, COSEBIs scale-cuts to 0.9); decoupled from covariance.smk's FIDUCIAL grid. Shared twopoint.smk falls back to the fiducial integration grid for configs without a cosmo_val section (e.g. papers/bmodes). The raw .txt byproduct is left undeclared to avoid an AmbiguousRuleException with rule xi; nothing in the DAG consumes it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EaL7prmKUHwJQcDyW3LoxD
…g-aware
Merge the updated feat/sacc-4-cosmo-val-sacc (cv_cosebis/cv_pure_eb consume the
ξ± SACC parts via the b_modes seams; per-version in-container xi_highres on the
dedicated cosmo_val.integration grid; the cosmo_inference zombie purge; the
1000-bin default).
On this blinding branch the consumers bind the *_blinded* parts on data runs and
stamp their derived output concealed, so born blinding flows end to end with no
blind-aware consumer code:
- cv_cosebis/cv_pure_eb inputs wrap the ξ± parts in blindable_part (→ _blinded
sibling on a data run, plaintext on a mock run) and bind commitment.json on a
data run — per version, not fiducial-gated.
- cv_cosebis.py/cv_pure_eb.py thread commitment_path to the *_to_sacc_part
writers; cosebis_to_sacc_part/pure_eb_to_sacc_part re-derive En / the six
pure-mode arrays from the (blinded) part and stamp the emitted part concealed.
- xi_highres keeps its maybe_temp blindable wrapper on {version}_xi_integration.sacc,
so blind_part produces the _blinded sibling the consumers now request.
The dry-run wiring test asserts _xi_integration_blinded.sacc IS in the cosmo_val
subgraph again (the consumers are its explicit consumers on a data run).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EaL7prmKUHwJQcDyW3LoxD
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #252. Stacked on #251 (
feat/sacc-4-cosmo-val-sacc).Wires per-part, at-birth Smokescreen blinding onto the migrated cosmo_val DAG. The three blindable intermediate SACC products — reporting ξ±, integration ξ±, analysis pseudo-Cℓ — are concealed the moment the pipeline computes them, so only blinded parts persist on disk. The terminal
{version}.saccis assembled only from concealed parts and asserts one shared blind commitment. Mock runs bypass blinding entirely.The blinding machinery itself (three-verb custody, three CCL theory backends, CAMB↔CCL cross-check) is #253's existing content, reconciled onto #245's canonical
sacc_io. This description covers the DAG wiring added on top.The custody surface (three verbs + one assertion)
sp_validation/blinding.pycarries the machinery;scripts/blind_data_vector.pyis the CLI.blind-init(once per catalogue version) — draws an OS-entropy seed, writes the repo-committablecommitment.json(sha256(seed)+ a config digest binding envelope + fiducial), and encrypts the seed into a Fernet bundle. The plaintext seed is never written.blind-part(per intermediate, at birth) — reads the encrypted seed (verifying both hashes), conceals the part through its matching CCL theory backend, escrows the true vector into a per-part bundle beside the blinded output, stampsconcealed=True+blind_commitment, stripsseed_smokescreen.unblind— verifies bothsha256(seed)and the config digest before subtracting; fails closed on either mismatch.blind_commitmentis identical across every blindable part (assert_consistent_blind), failing closed on a mixed or divergent-seed assembly.DAG wiring (this PR)
The pre-migration rules on this branch were replaced wholesale by #251's SACC workflow (clean merge — this branch never touched the migrated rules). On top of that:
rule blind_initfires once per catalogue version (data runs only).rule blind_partis generic over the three blindable stems. Its plaintext input is atemp()output of the producing rule (common.maybe_temp), so on a data run only the blinded sibling persists — the true vector is escrowed first.blind_init/blind_partenter the DAG only when a consumer binds to a*_blindedpart.common.blindable_partis the single switch: adatarun binds ξ-derived consumers (the terminal assemble) to the blinded siblings; amockrun binds to the plaintext and the whole blinding subgraph vanishes.assemble_saccasserts the shared commitment across parts and stamps it on{version}.sacc.npatchwildcard is now constrained to\d+so a producer's ξ± output cannot absorb the_blindedsuffix (which maderule xiambiguous withblind_part).The cosmo_val assemble DAG dry-runs:
blind_init×2,blind_part×5, assemble consuming the blinded ξ± + pseudo-Cℓ parts.Born-blinded derived statistics
COSEBIs and pure-E/B are born blinded with no blinding-aware estimator code. The consumers (
cv_cosebis/cv_pure_eb) read the ξ± SACC parts from disk (integration for COSEBIs; reporting + integration for pure-E/B) and re-derive the E-modes through the values-only seamsb_modes.cosebis_from_xi/pure_eb_from_xi. On a data run those inputs bind to the*_blindedsiblings, so the derived E-modes inherit the blind automatically. This holds for every catalogue version:xi_highresruns per version (see the #251 base). The raw estimator still runs once per version for the blind-invariant pieces —Bnand the covariance — and every serialized E-mode (SACC part and.npzsidecar) is part-derived; rawEnnever reaches disk on a data run. The derived parts are stampedconcealedwith the shared commitment (truthfully: their values derive from blinded inputs), so the terminal assembly's commitment assertion covers them.Other flags
.txt. The raw TreeCorr ξ±.txtbyproduct stays unblinded because the CosmoCov covariance seam reads it — an acknowledged plaintext-ξ± tension until the covariance path is migrated. The fine / glass-mock pseudo-Cℓ variants (validation intermediates, not terminal-file parts) are left untouched.Tests
test_blinding_wiring.py(new): the common.py path helpers mirrorblinding.init_paths/part_paths(drift guard),version_ofon all blindable stems, the run-type switch, and the data-run fail-closed assembly (refuses an unblinded part, passes when every part is concealed under one commitment, refuses a blinded/plaintext mix and divergent commitments). A candide-only test asserts the blinding subgraph resolves in the cosmo_val dry-run.test_blinding.py/test_camb_ccl_crosscheck.py/test_sacc_io.pystay green. The-m "not slow"suite passes (268 passed) bar the known environmental candide-path guard.— Claude (opus) on behalf of Cail.
🤖 Generated with Claude Code
https://claude.ai/code/session_01EaL7prmKUHwJQcDyW3LoxD