Script for covariance from Gaussian simulations

[sachaguer]

This PR implements a script from the previous non-tomographic implementation (see issue #282).
The script relies on the cosmo_val cat_config such that it can be run in parallel of cosmo_val.
The Cl evaluations are parallelised with MPI to efficiently get samples.

The pipeline is now running without Runtime errors but remains to be validated against the iNKA implementation. (Done now)

I will add a plot when the latter validation is ready and ping for reviews.

[sachaguer]

I fixed a few bugs in the implementation that now returns sensible outputs for the pseudo-Cl samples from the Gaussian simulations (see e.g. the samples against the fiducial for noise-bias subtracted auto-bins below).

When the samples will be finished, a final runtime validation of the merged covariance will be possible and a comparison with iNKA.

[sachaguer]

Progress report: the first covariance from Gaussian simulations has been produced:

Still there is a big disagreement between the simulations and iNKA that I need to pin down

[sachaguer]

Progress report: iNKA and Gaussian sims agree on the non-tomographic case. Tomographic case is running. There is no reason why they should disagree so this should be nice final validation showing convergence of the code.

[sachaguer]

This circles nicely from the bug reported in #291 to the validation of the Gaussian simulated covariance and the GLASS mocks.

The bug fixed on the noise bias propagation for the iNKA covariance looks much better (see below)

I shared the non-tomographic agreement between the Gaussian sims and the iNKA covariance above. This agreement remains in the tomographic case (see figure below for the bin 6-6 of the test mock used, I also checked cross-bins and other auto-correlations)

I think this finalises the validation of #190 #286 #291. @cailmdaley this PR can be reviewed.

[cailmdaley]

Nice work — the non-tomographic and bin 6-6 agreement is convincing for what it tests. Two requests before merge, then smaller code items.

1. Extend to BB/EB. concatenate_spectra takes index 0 of the decoupled Cl stack, so the sim covariance is EE-only. The #291 fix changes COVAR_BB_BB as much as EE, and BB feeds the B-mode χ²/PTE. Please grab the BB (and EB) blocks too, so the B-mode pipeline is covered by the same validation.

2. Derive config from cosmo_val. The script re-declares binning via CLI defaults and cosmology via cosmo_gaussian_sims.yaml. Nothing ties these to the cosmo_val settings — agreement currently depends on the defaults matching by hand. Suggestion: point at the cosmo_val yaml for overlapping values, or structure the sim yaml per version so it inherits the data settings and only overrides what differs.

Smaller items:

• HMCode_log_T_AGN is misspelled. CAMB expects HMCode_logT_AGN, so the AGN temperature silently reverts to its default.
• Fields are built at lmax=2·nside; the workspaces use 2·nside−1. Please align.
• No seed is threaded through the draws, so the sims are not reproducible. Please accept a seed and offset it per rank.
• The broad except in run_one_simulation swallows per-sim failures silently. Please log the exception at minimum.
• The sim fields drop the pol_factor convention the data pipeline applies.
• The PR description still says validation "remains to be" done — worth refreshing.

One note, no action needed: neither the sims nor the tomographic iNKA fiducial applies the pixel window. For catalog-based estimators this is the right call — the estimator does not inherit the pixel window function, and the map-based path only serves the covariance. Flagging it so the deliberate omission is on the record; correct me if the reasoning is off.

— Claude on behalf of Cail

[sachaguer]

These last commits should solve the most important problems.

1. Extra arguments now allow to save EB and BB covariance. The format is different from the iNKA covariance. We can consider adding a function to merge them into a consistent output (can be useful for plotting within cosmo_val)

2. After discussing with Cail, we agreed that this item is not relevant for now. The script, as is, has well-defined inputs and outputs. Among them, cat_config is used to match the input catalogue noise properties and footprint, and a config file is used for the cosmological parameters. The script logs the input parameters at runtime, so you can see the input configuration. The script itself can be used outside of the pipeline as is and is not constrained by any choice made in the analysis. These choices will be formalised in the workflow that will pipe this script with the other related ones.

Small things:
I dropped the point on the except. I don't want the pipeline to stop because one MPI process failed for whatever reason. The exception is printed and allow to understand the bug. I also did not implement the pol_factor. The plarisation is controled in the simulation and the output of the script are the measured spectra. This extra parameter would be superfluous. I added the seed to the script to control the RNG.

[cailmdaley]

Thanks Sacha LGTM!