sacc_io converters: SACC → CosmoSIS 2pt-FITS (byte-verified) + SACC ↔ OneCovariance#249
Open
cailmdaley wants to merge 21 commits into
Open
sacc_io converters: SACC → CosmoSIS 2pt-FITS (byte-verified) + SACC ↔ OneCovariance#249cailmdaley wants to merge 21 commits into
cailmdaley wants to merge 21 commits into
Conversation
Add sp_validation.sacc_io: the writer/reader layer for the two-file SACC
layout that becomes the package's standard data-product format.
{version}.sacc analysis vector — NZ tracers, coarse xi+/-, pseudo-Cl
(EE/BB/EB) with a shared BandpowerWindow, COSEBIs,
pure E/B, rho/tau PSF diagnostics; one FullCovariance
assembled block-diagonally (zero cross-blocks).
{version}_xi_fine COSEBIs/pure-EB integration input — same NZ tracers,
fine-grid xi+/-, DiagonalCovariance from TreeCorr
varxip/varxim.
Covariance order is point-insertion order (SACC preserves it bitwise
through FITS). Writers insert in the canonical order — xi+ then xi-, Cl
(ee, bb, eb), COSEBIs (all En then all Bn), pure E/B in _EB_KEYS order
(xip_E, xim_E, xip_B, xim_B, xip_amb, xim_amb, matching
b_modes.calculate_eb_statistics), rho, then tau — but readers never assume
global order: every getter resolves indices through s.indices(dtype,
tracers, **tags). assemble_covariance validates that blocks are contiguous,
ascending and tile the data vector exactly, failing loud otherwise.
Custom data types (pure E/B, rho, tau) all parse under
sacc.parse_data_type_name. Tag filters are plain kwargs; the tags={...}
form silently selects nothing and is never used.
Test suite (test_sacc_io.py, all synthetic and fast): per-writer
round-trips (arrays/tags/windows/NZ bitwise), covariance block alignment
and zero cross-blocks, assemble_covariance failure modes, DiagonalCovariance
round-trip, extract() sub-covariance alignment, a tomographic multi-pair
case, reader/writer mirroring on a mixed file, and the end-to-end two-file
layout. 20 passed.
Co-Authored-By: Claude Opus <noreply@anthropic.com>
Fresh-eyes review caught a correctness bug: readers re-sorted selections by
theta/ell/n, but covariance blocks and bandpower windows stay in insertion
order. On a non-ascending grid the reader output silently desynchronised
from its covariance, and get_pseudo_cl returned sorted cl arrays against
unsorted window columns — internally inconsistent within one return tuple.
Fix by construction, not by sort:
- Writers validate their grids. add_xi/add_pure_eb/add_rho/add_tau require
strictly ascending theta; add_pseudo_cl requires strictly ascending
ell_eff (add_cosebis is inherently safe — it enumerates the mode index).
Out-of-order grids raise a loud ValueError naming the argument.
- Readers drop the sort entirely and return in s.indices (insertion) order,
so every getter is covariance- and window-aligned for ANY file, and
ascending for canonical files. The _sorted_* helpers are replaced by
plain insertion-order accessors (_mean/_tag).
Also:
- _pair normalises (i, j) -> sorted, so get_xi(s, (1, 0)) addresses the same
symmetric shear-shear pair as (0, 1) instead of a silent empty read.
- Module docstring documents the tomographic ξ covariance ordering:
insertion is pair-major ([pair0 xip; pair0 xim; pair1 xip; …]), supplied
to assemble_covariance as one contiguous block matching add_xi call order;
type-major converters (DES 2pt-FITS) permute explicitly via s.indices.
- extract() docstring states tracers takes SACC names, not integer bins.
New tests (26 total, was 20): writers reject non-ascending theta/ell;
(1, 0) == (0, 1) round-trip; a 3-pair tomographic ξ covariance assembled as
one contiguous pair-major block with per-pair sub-blocks recovered via
extract(); get_pseudo_cl window column j <-> ell_eff[j] via window_ind tags.
Co-Authored-By: Claude Opus <noreply@anthropic.com>
The two-file split (analysis + {version}_xi_fine.sacc) was premised on a
10000-bin fine grid; the production operating point (Paper II B-modes) is
1000 bins, where a dense per-pair fine covariance block is ~32 MB and the
CosmoCov integration-binning covariance — which feeds pure-E/B and COSEBIs
error propagation — has a natural home as a BlockDiagonal block alongside
the analysis blocks. Layout test replaced with the one-file end-to-end
case (dense fine block, extract() sub-covariance alignment, zero
cross-blocks) plus a varxi-diagonal fallback test.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019R5eiy11Lihkgn4MKufXSp
…-forward) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019R5eiy11Lihkgn4MKufXSp
…date_statistic PRD #241 §4 (Mocks vs data): save() requires type='data'|'mock' and stamps it into metadata; load() raises on type='data' files lacking the concealed=True blinding stamp, so skipping the blind can never silently expose real data. Mocks load freely. allow_unblinded=True is the loud escape hatch reserved for the blinding/unblinding tooling. merge() wraps sacc.concatenate_data_sets thinly: per-statistic files combine in order, shared tracers stored once, covariance block-diagonal (library-enforced all-or-none), metadata union with loud conflicts. update_statistic() is the value-only merge-back for the extract -> conceal -> merge blinding flow: matches each sub point by (data_type, tracers, tags) and overwrites the value, leaving order and covariance untouched. Tests: all four data/mock x concealed/unconcealed quadrants, the escape hatch, stamp validation, merge (points, covariance, metadata conflict, mixed-covariance failure) and update_statistic (values-only, unique-match). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QWF72ofwJh6ekgnCt9Xx6C
…comments - grid tag values: coarse -> "reporting", fine -> "integration" (descriptive, not relative); tag kept because both grids share data type and tracer pair, so the tag is the sole disambiguator. Swept module docstrings and tests. - Module docstring now spells out why insertion order is load-bearing: covariance row/column i refers to the i-th inserted data point. - new_sacc: comment that psf_stars rides in the same tracer list as the NZ tracers only because a Sacc has one flat tracer namespace (bookkeeping, not physics). - source_name/_pair helpers kept (4 and 8 call sites) with one-line justifications of the conventions they centralize. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QWF72ofwJh6ekgnCt9Xx6C
- Factor the theta-tagged insertion loop into _add_theta_series (add_rho, add_tau, add_pure_eb); add_pure_eb zips PURE_TYPES.values() against its signature order, and PURE_KEYS is derived from PURE_TYPES instead of restating it. - Factor the (theta, plus, minus) read pattern into _get_pm (get_xi, get_rho, get_tau). - add_xi hoists the optional-tag None-filtering out of the point loop; extract and merge lose their throwaway mutable dicts; get_cosebis builds its scale-cut tags in one expression. - Tests: shared _add_xi default-ξ builder and _xi_block/_cl_block/ _cosebi_block canonical index-block helpers replace ~60 lines of copy-pasted setup; test_readers_on_mixed_file builds on _multi_statistic_sacc. Behaviour unchanged; 41/41 tests green in the container. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QWF72ofwJh6ekgnCt9Xx6C
cailmdaley
force-pushed
the
feat/sacc-2-sacc-io
branch
from
July 16, 2026 00:39
55cb97d to
4489dd4
Compare
Adversarial review (pass 2) findings: - Sacc.indices returns an EMPTY array (warning only) on an unmatched selection; every reader, extract(), and covariance-block selector now funnels through a shared _indices guard that raises instead — a typo'd tag or non-bitwise-identical float scale cut can no longer propagate empty arrays downstream (assemble_covariance previously died with an opaque IndexError on the same path). - get_cosebis(scale_cut=None) on a file carrying several scale cuts silently concatenated them; now raises and asks for an explicit cut. - update_statistic let two sub points claim the same target point (last-write-wins); now raises. - merge: the library keeps the FIRST input's tracer on a name clash with no equality check; shared tracers are now verified identical (z, nz) across inputs before concatenation. 7 regression tests; 48 total. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WzUt7VbtXwr2SCHUdiQTyt
Adds twopoint_convert (SACC → 2pt-FITS, byte-compatible with cosmosis_fitting) and one_covariance_io (SACC ↔ OneCovariance glue), with their test suites. Rebuilt on feat/sacc-2-sacc-io so the diff is converter-only; adapted to that branch's canonical sacc_io — ξ reads use grid='reporting' (was 'coarse'). Optional-block probing (pseudo-Cℓ, τ) stays on raw s.indices behind get_data_types/use_rho_tau guards, so the new fail-loud sacc_io readers don't trip on absent blocks. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WzUt7VbtXwr2SCHUdiQTyt
cailmdaley
force-pushed
the
feat/sacc-3-2pt-converter
branch
from
July 16, 2026 09:34
700a885 to
64b2b5f
Compare
Consistency follows tagging semantics: the grid tag declares which binning a set of points lives on, so all same-length theta arrays under one tag value must be bitwise identical (sacc never validates angles, and grids diverging at floating-point level choke CosmoSIS downstream). Different lengths within a tag group pass (scale-cut subsets); grids under different tag values are unconstrained (reporting vs integration differ by design). The rho/tau/pure-EB writers now tag their points grid="reporting" by default (overridable via grid=), so they join xi's consistency group and no untagged group appears in our own files. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MN9VazXKHUHQg16kiG7Ufk
…seudo-Cl grid="reporting" The theta consistency guard generalizes to both angular domains: theta and ell points are grouped separately by grid tag value, and within a tag value all same-length grids must be bitwise identical. Two ell series sharing a grid must also carry equal bandpower windows (window ells and weight matrix); series without windows skip that check. add_pseudo_cl now stamps grid="reporting" on every point by default (overridable), joining the merge guard's consistency groups; since sacc's add_ell_cl accepts no extra tags, its per-point insertion (ell + shared window + window_ind) is inlined. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MN9VazXKHUHQg16kiG7Ufk
…acc-3-2pt-converter
…ty selections Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MN9VazXKHUHQg16kiG7Ufk
One module whose job is converting between the SACC product and external analysis-tool file formats (CosmoSIS 2pt-FITS, OneCovariance). Public API unchanged; sacc_io untouched. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01766vawzi2XqrgoyHmeHEY9
cailmdaley
changed the title
Cail's ruling: a single sacc_io.py carries the format contract and the external-tool converters (CosmoSIS 2pt-FITS, OneCovariance). Public API unchanged; the sacc_io import name every downstream branch uses is untouched. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01766vawzi2XqrgoyHmeHEY9
cailmdaley
changed the title
assemble_covariance now passes sacc.add_covariance a list of per-block matrices instead of a dense zero-filled N×N array. sacc.BaseCovariance.make turns a list into a BlockDiagonalCovariance (one FITS table per block, Σ block² on disk), so cross-blocks are zero and implicit rather than materialized. Validation (contiguous/ascending indices, no gap/overlap, square blocks matching their index span) is unchanged. Every existing consumer already read through the polymorphic .dense property, so only the type assertions needed updating (FullCovariance -> BlockDiagonalCovariance). Added a test that merging two files that already carry a BlockDiagonalCovariance (assembled via assemble_covariance) stays block-diagonal through merge and a save/load round-trip. Updated the module docstring's storage-cost discussion accordingly. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
pure_eb.py's calculate_pure_eb/plot_pure_eb defaulted nbins_int=100; cosebis.py's calculate_cosebis already defaulted to 1000, and every production config (papers/bmodes, papers/cosmo_val fiducial/pure_eb blocks) already overrides to 1000. This aligns the function-signature default with what every caller actually uses; config plumbing is untouched, so any explicit override still wins. The papers/cosmo_val/config/config.yaml cosebis.nbins_int (currently 2000, production numerics) is deliberately left unchanged — see report. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Writers no longer force components an analysis may not have computed: add_pseudo_cl's BB/EB, add_cosebis's Bn, and add_pure_eb's B/ambiguous blocks now default to None and are simply not written when omitted (add_pure_eb still requires each +/- pair together). Structural arguments (grids, tracer/bin identifiers, the value for a component you ARE adding) remain required with no default. Composite readers (get_pseudo_cl, get_cosebis, get_pure_eb) return None / omit the key for an absent optional component instead of raising, via a new _mean_optional helper; a selection naming a missing component explicitly (s.indices, _mean, extract) still fails loud, per the existing empty-selection guard. merge() and assemble_covariance work unchanged on files with only a subset of components. Documents the optionality contract in the module docstring, and adds partial-file round-trip, merge, and explicit-selection-fails-loud tests. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…fault cosebis.nbins_int was 2000; every other integration-grid entry in this config (pure_eb, the fiducial block) is already 1000, and cosebis.py's own function defaults are 1000. Unify. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Brings the base's four new commits (block-diagonal covariance in assemble_covariance, unified 1000-bin pure-EB integration grid, genuinely-optional statistic components, COSEBIs grid alignment) into the 2pt-converter branch. Sole conflict was in the sacc_io.py module docstring: base added the 'Optionality' paragraph, 249 added the 'Converters' paragraph. Kept both, with 'Optionality' first (matching the file's semantics-then-tail order) and normalized its indentation to the docstring block. No converter-code adaptation needed: sacc_to_twopoint_fits and the OneCovariance glue read covariance through s.covariance.dense, which materializes correctly from the new BlockDiagonalCovariance storage, and 249 already filters covariance selections by grid tag (b007ed1). Full suite green except two environmental failures unrelated to the merge (snakemake absent from the shared venv's python3.12; candide data paths not mounted on the run node). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EaL7prmKUHwJQcDyW3LoxD
…nverter # Conflicts: # src/sp_validation/sacc_io.py
cailmdaley
marked this pull request as ready for review
July 21, 2026 14:28
Member
|
Thank you for this! After this is merged I can also adapt the OneCovariance snakemake workflow in #256 to include the sacc conversion |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #246. PR 3 of the PRD #241 series (base: #245,
sacc_io).What & why
Extends
sp_validation.sacc_iowith converters between the SACC data product and external analysis-tool file formats. SACC stays the standard product; the current CosmoSIS inference chain runs unchanged behind it (PRD #241 §2: "converter first, native likelihood later"). One module now carries the whole SACC surface: format contract, guarded selection, and converters.sacc_to_twopoint_fits(s, path, *, rho_stats_hdu=None, tau_stats_hdu=None, n_bins=1)converts an analysis SACC into the 2pt-FITS read by2pt_likeand the ρ/τ2pt_like_xi_sysfork. Single-bin v1: it raises onn_bins != 1and on any ξ pair beyond(source_0, source_0). Tomographic emission lands with the tomographic round. RHO_STATS is copied verbatim from a sidecar FITS; the 2pt layout carries per-modevarrho_*variances the SACC does not store, and no consumer reads them.[redshift]config stanza, plus acovariance_list_*.datreader that produces dense blocks paired with SACC selectors forassemble_covariance. Coupled to OneCovariance only through its file formats, not as a dependency.Every data and covariance selection goes through the guarded selection with
grid="reporting"(loud on empty). Integration-grid points in the analysis SACC can never leak into the 2pt-FITS. (SACC covariances carry no tags of their own; blocks are addressed through data-point indices, which do — filtering the points filters the covariance with them.)No public API outside
sacc_iochanges; no existing behavior is modified.Validation
Byte equality against the current writer (astropy writes FITS deterministically): build a reference with
cosmosis_fitting.py's own HDU-builders, build a SACC from the same inputs, convert, assert byte-for-byte identity.test_sacc_io_twopoint.py(CI): all three product shapes (plain-ξ; ξ+Cℓ; ξ+ρ/τ), the truncated-but-cross-correlated[τ₀+; τ₂+]covariance block, perturbation teeth, sidecar fail-fast, an encoded-covariance test that fails on any transposition, and byte-identity with integration-grid points present (regression pinning the grid-filtered covariance gather).test_sacc_io_realdata.py(candide-localskipif): byte-equal observed for realSP_v1.4.6_leak_corrandglass_mockproducts. No real data committed.test_sacc_io_one_covariance.py: reshaping, column selection, n(z) round-trip, fail-fast paths.81 tests green, including the 4 real-data byte-compares on candide.
Legacy writer
cosmo_inference/scripts/cosmosis_fitting.pyremains the pipeline's 2pt-FITS writer for now; this converter is its byte-verified equivalent. PR 4 (#251) migrates the pipeline onto the SACC path and retires it. The swap happens downstream, not here.— Claude (Fable, with Opus/Sonnet implementation agents) on behalf of Cail.
🤖 Generated with Claude Code
https://claude.ai/code/session_016Ko89uZF84ez6HEDcTWbJe