UK dataset sizes: informed L0 and exact-count candidates (#355)

[juaristi22]

The UK rowwise driver currently exports every cloned household. This adds --dataset-households to request a compact candidate after the dense solve, using contribution-informed L0, exact-count sampling, and a frozen-support refit, plus the machinery a licensed 55,000-household run turned out to need and the evaluation that measures such a candidate.

Based on main (after #883). Refs #355 and #665; this does not close #355's certification and default-switch requirements.

Behavior

• Keep pool generation separate from output size: the existing K=15, grain_equal, 1,500-epoch defaults and local gates remain; --epochs is free outside --release-candidate.
• Initialize L0 using target-contribution shares and protect each nonzero target's largest absolute weighted carrier; refuse by name any nonzero target no pool household supports.
• Search the L0 penalty on the gates' open-probability mass and stop only on a probe whose gate probabilities admit the exact-count draw at the requested certainty threshold (calibrate(..., budget_basis="open_probability_mass", feasible_draw_pi_hi=h); exact_k_design_feasibility is the draw's own inequality). Every probe and the stop reason ride the size receipt.
• Draw with the shared exact-count Sampford sampler at --selection-pi-hi (default 1.0; candidate-only) and refit from the normalized Horvitz–Thompson baseline with the stretch multiplier 10 on that inclusion-adjusted reference; the manifest records the reference. A refused draw carries the measured gate mass and the feasible alternatives.
• Checkpoint the dense solve and the search before the draw (size_selection_checkpoint.{npz,json} in --out; --no-size-checkpoint opts out); --resume-size-checkpoint DIR re-derives and verifies the pool and target surface, rebuilds both results through rebuild_calibration_result, and continues at the draw, at any threshold.
• Freeze full-pool target contributions, preserve linked households, benefit units and people, rerun selection inside each holdout fold; --selection-seed re-draws on one pool and one dense reference.
• Export the dense reference the selection was cut from (dense_reference_diagnostics.csv, a summary under solve.dataset_size.dense_reference) and the selection itself (dataset_size_selection.csv).
• Report solver progress (a loss line every hundred epochs, one line per budget probe with its verdict, the search stop) through the calibrator's callback seam.
• Evaluate a size run with one command, tools/evaluate_uk_dataset_size.py (library uk_runtime/size_evaluation.py): run acceptance, size-only deltas against the run's dense reference, comparison with a reference run, incumbent surface re-resolution with the frozen-versus-recomputed check, downstream T3/T4/T5 and the UK scorecard: populace_uk_2023 vs enhanced_frs_2024_25 vs admin benchmarks (2026) #731 scorecard, a summary with the pre-registered outcomes, every step with a receipt.
• Two register changes met on the way: the PIPR private-rent price level is composed into a linear total (12 × mean monthly rent × the A17-uprated private-renter count, receipted per authority), and two zero-support UC payment bands are signed A16 exclusions (tracking WS-C deferred adjudications and the reconciliation layer #736).

Measured

Two licensed 55,000-household candidates from the K=15 pool on spine-p and feed 6fb700e (thresholds 0.5 and 0.95, 2,000 epochs) and the refusal that preceded them are in experiments/355-uk-dataset-size-receipts.md, with the evaluation trees under each run; the summary is in the PR comment below.

Review boundary

These are candidate artifacts. Size requests refuse --release-candidate and remain unreleasable until matched comparisons and the promotion scorecard are adjudicated. No dataset registry, production pointer or downstream default changes. Full builds still require the dense build's peak memory and add selection/refit work; savings apply to the exported dataset and downstream use. The operating instructions are in docs/uk-dataset-size-plan-355.md.

Validation

• Calibrate suite 236; UK rowwise runtime, driver, evaluation and gate battery 100; registers and ledger 84; spec-engine identities and graph parity 58; US multispine pool tool 187; engine re-run on policyengine-uk 2.97.0 120 — all green on the rebased tree.
• Repository-wide Ruff check, changed-file formatting and CI test inventory verification pass; the attested calibrate identities (country bundles, seed protocol and map, loader golden vector, US spec digest, coverage report, H1 calibrate parity fixture, the best-iterate oracle's gates attestation) are re-cut.

[vahid-ahmadi]

Review pass at 37f81933 (Claude Code, high effort; worktree checkout; without the engine: 1,048 passed, 24 skipped, 10 failed — the ten are test_us_multispine_pool_tool needing the US extra, identical on main; ruff check and ruff format clean on all 29 changed Python files; CI green apart from one wheel job unreported).

The default solver paths are byte-identical, and that was checked rather than read

This is the second UK PR in a week to touch the generic solver, so I ran the same synthetic n=300 problem against main's and the PR's microcosm.calibrate in one venv: the count-basis L0 search with target_records, the same under mass="conserve", a fixed l0_lambda, and Sampford select_exact_k plus refit_l0_selection — weights, gate probabilities, support, q, trajectories and losses all hash-identical. Reading confirms why: steer() reduces to the old bisection when the verdict is not_required, and settled() reduces to the old tolerance test without feasible_draw_pi_hi. The US exact-K ladder's tests pass with no fixture regeneration. The parity fixture's direct.csv is unchanged and only the pin moved, asserted bitwise on every platform.

The statistics are right: units with π at or above pi_hi form a certainty stratum, boundary q is m·π/Σπ with a strict cap at one, and exact_k_design_feasibility is the draw's own inequality, property-tested against the draw across all four verdicts. Protected carriers get π exactly one via torch.where on the mask, so forced inclusion lands in the certainty stratum and does not break the design, and the refit asserts none was lost. Refusals carry numbers (the full feasibility scan) rather than errors, the unsupported targets are refused by name — and found exactly the two UC bands this PR then signs — and the resume path verifies arrays, identity keys, household ids, the target surface, initial weights and both closing losses. --release-candidate refuses --dataset-households, the manifest's releasable is false whenever a size is requested, and no registry or pointer moves.

Findings

1. Should-fix — doctrine drift is undetectable on resume. _size_checkpoint_identity (build_uk_rowwise_candidate.py:489-520) carries pins, seeds, epochs and sample settings but no code pin and no doctrine constants; load_uk_size_checkpoint rebuilds with the checkpoint's own target_loss_cap and options, the refit takes max_weight_ratio and mass from the dense solve's options, and the closing-loss check uses the stored cap. If UK_LOCAL_MAX_WEIGHT_RATIO or UK_LOCAL_TARGET_LOSS_CAP change between checkpoint and resume, the refit runs under the old bound while the manifest declares the new one. Put the existing code_pin and the doctrine values into the identity, or assert the restored options against current doctrine after the load.

2. Should-fix — a stale checkpoint in --out is refused only after the dense solve. write_uk_size_checkpoint refuses to overwrite (size_checkpoint.py:135-137), correctly, but it runs after the solve and the search, and _output_paths' pre-solve existence guard does not cover the checkpoint files. A re-run into a directory left by a refused run loses the multi-hour solve at the write. Add the two files to the pre-solve guard.

3. Should-fix — no frozen control covers gates.py, and it sits outside the attested kernel sources. The pre_best_iterate oracle pins the gate module's sha to the current file and imports HardConcrete from it on both sides, so it cannot detect a gate-behaviour change; this PR re-cut that sha. My A/B run is the only evidence of byte identity. Separately, the seed-protocol implementation digest hashes calibrate.solve and calibrate.exact_k only (seeds.py:348-349), so gates.py and the new initialization.py — which now shape L0 behaviour — do not move the attested identity when they change. A frozen-gates control and adding both modules to that list close it.

4. Question — say plainly how deterministic the "draw" was on the licensed runs. The receipts show the protected carriers as certainties at pi_hi = 1 and, at the lower thresholds used for the two 55,000 candidates, a large majority of the selection landing in the certainty stratum with only the tail actually drawn. That is by design — pi_hi is doing the selection — but the body's "exact-count Sampford draw" reads as a random design when it is mostly a threshold on learned π. Reporting the certainty share per run in the body would make the claim honest at a glance. Related: the HT refit baseline is design weights over q projected to design mass, and the stretch multiplier 10 is relative to that baseline, not the dense doctrine's ten-times-design; the plan doc says so and lists it for adjudication, but realized_max_weight_ratio_vs_design is now computed against solve.initial_weights, which on a size run is the HT baseline, so the key name misleads.

5. Question — zero-valued targets in the prior. contribution_initialization uses a unit denominator when the target is zero (initialization.py:327), so raw pound contributions saturate the prior at 0.9. The local parity fixture holds thousands of zero-valued rows (Welsh and Scottish private-rent amounts and renter counts); if any survive onto the compiled surface they dominate the prior. A zero_target_rows count in the size receipt would answer it.

6. Question — two register changes ride along and move the dense surface for every joint build. The PIPR private-rent composition into an annual total (12 × mean monthly rent × the uprated private-renter count, ledger_targets.py:1496) is unconditional, and the two zero-support UC bands become A16 exclusions expiring 2026-12-08 (census counts 49 → 51, 84 → 82). Both are reasonable and both affect comparability with #870's R17 receipts; they deserve their own line in the body, or a split.

Nits

The checkpoint manifest's written_at and absolute directory ride into the candidate manifest (excluded from the identity digest, so harmless, but manifests of identical runs will not byte-match). The parity pin's authoring platform moved from py3.13 to py3.14. Every calibrate() caller's options gains four new keys, which changes the shape of any manifest that embeds them.

Needs the licensed data

Every measured claim (the two 55k candidates, their ESS, the 25% area-support breach, 211 of 650 constituencies below the ESS floor), the frozen_vs_recomputed step — which the receipts say was not measured on either candidate and is the actual test of freezing population-dependent measures from the full pool — any holdout for a size run (all used --skip-holdout), and the incumbent leg of the #731 scorecard, which did not run.

---

No blocking defects; the machinery is careful and the receipts document the failures honestly. 1 through 3 are the ones I would land before this is used to cut a candidate anyone compares: a resume that can silently run under the wrong bound, a checkpoint that can be lost after the expensive step, and a solver module that is neither controlled nor attested.

[juaristi22]

Response at c2ee1342 (review items) + 1bdd6447 (a literal pin the attestation moved in test_spec_engine_field_usage.py, missed in the first push; CI run 34405953724 on the combined head).

Thank you for the A/B run on the default paths; that is the check I could not do from inside the PR. Everything below is on the branch, rebased over #891, with every identity re-cut on the combined tree.

1. Doctrine drift on resume — fixed, two locks. The checkpoint identity now carries the solve doctrine (_doctrine_bounds(): the stretch multiplier, the loss cap, the rule) beside the pins, seeds and sizes, so a resume after a doctrine change is refused by name at the identity check; and independently the doctrine solve asserts the restored options (max_weight_ratio, mass, target_loss_cap) against today's doctrine after the load and refuses if they differ. The writing run's code_pin and build_id ride in the checkpoint's provenance and are reported on resume rather than compared, so a receipts commit does not invalidate a checkpoint. Tests at the solve level (a resume under a doubled stretch bound refuses) and the driver level (identity carries the doctrine, provenance names the run).

2. Stale checkpoint refused after the solve — fixed. The driver now refuses an --out holding size_selection_checkpoint.{npz,json} beside the other pre-solve output guards, naming the files and the --resume-size-checkpoint way out; the writer's refusal stays as the last line. Driver test: a directory holding a stale checkpoint fails before any solve and leaves no manifest.

3. gates.py and initialization.py — attested and controlled. Both join the seed-protocol implementation digest (seeds.py), so the attested identity moves when gate behaviour or the contribution prior change (this moved the seed protocol digest, the pointer inventory, two field counts and the US spec digest; all re-cut in c2ee1342 except the field-usage test's own literal counts, which only run in the rest/engine-shared/wheels lanes and went red there; 1bdd6447 re-pins them from the recomputed ledger). The pre-best-iterate oracle now runs with a frozen copy of the gate module (fixtures/pre_best_iterate/gates_14608a5e.py, loaded under pre_best_iterate_gates) instead of importing the live one; the control asserts the oracle's HardConcrete is the frozen class and not the live one, so a gate-behaviour change surfaces as a numeric mismatch on the gated control path. The oracle module's only change is that import; its digest is re-pinned.

4. How deterministic the draw was — now stated in the receipt. The size receipt carries certainty_share and boundary_draws. On the licensed runs: P50 (threshold 0.5) 54,595 of 55,000 by threshold, 405 drawn (99.3 %); P95b (0.95) 54,844 by threshold, 156 drawn (99.7 %). The body's "exact-count Sampford draw" is therefore a threshold on learned π with a small tail draw, and the receipts say so. On the stretch key: the manifest now carries realized_max_weight_ratio_vs_stretch_reference (against the frame the refit started from, the HT baseline on a size run; 10.0 on both candidates) and realized_max_weight_ratio_vs_design measured against the pool design weights themselves (P50 1,383, P95b 1,679; the share of rows above 100× design is 0.07 % and 0.03 %). The inclusion-adjusted reference stays listed for adjudication in the plan.

5. Zero-valued targets — counted. zero_target_rows in the size receipt. On the compiled 20,794-row surface of the licensed runs it is zero: the zero-valued private-rent rows in the parity fixture are the deferred cells, which never reach the compiled surface.

6. The two register changes. Both are now their own paragraph in the PR body, with the effect on comparability with R17 stated; splitting them would leave this branch unable to run (the composed rent rows are what makes the 314 PIPR cells bindable and the two UC bands are what the size path refuses by name), so they stay, flagged.

Nits. The checkpoint receipt that rides into the candidate manifest no longer carries written_at or the absolute directory (both stay in the checkpoint's own JSON), so manifests of identical runs byte-match again. The parity fixture's authoring platform is py3.14 because that is the interpreter this workspace's venv pins; the linux entries are untouched. The four new options keys are additive (budget_basis, feasible_draw_pi_hi, budget_search, and the selection_receipt main added); manifests that embed options gain keys and lose none.

Measured since your pass (receipts experiments/355-uk-dataset-size-receipts.md): frozen_vs_recomputed ran on both candidates — 340 of 364 national rows matched, maximum divergence 9.3 % (P50) and 11.1 % (P95b) on the council-tax band stock and private-school students, the other 328 rows within 1 % — the pre-registered red flag fires and is reported; the incumbent leg of the #731 scorecard runs under 2.97.0 (households P50 27.2 m, P95b 27.4 m, R17 29.3 m, eFRS 31.3 m; Gini 0.350 / 0.351 / 0.376 / 0.376); the downstream instruments were re-run on one engine for all four files; the T6 replays ran against the enhanced FRS. The holdout is the one item still not run for a size candidate, by ruling; the rule-1 score against the incumbent needs it.

[vahid-ahmadi]

Second pass at 1bdd6447 (Claude Code, high effort; worktree checkout; the six commits since the reviewed head read against the dispositions — four more than the two the reply names; the round-one battery: 901 passed, 3 skipped, 0 failed; on the old head exactly the three new tests fail; ruff check clean; CI run 34405953724 green on all 23 jobs).

1. Verified, two locks. _size_checkpoint_identity now carries "doctrine": _doctrine_bounds() (loss cap, stretch, scale rule, weight rule, epochs, clone count; build_uk_rowwise_candidate.py:555) and the load compares it by key; independently local_rowwise.py:1218-1239 asserts the restored max_weight_ratio, mass and target_loss_cap against today's doctrine. code_pin and build_id ride in provenance and are reported, not compared. Both tests fail on the old head (KeyError: 'doctrine') and pass here.

2. Verified. _refuse_stale_size_checkpoint runs right after the output-path validation and before any ladder load or solve, names the files and the --resume-size-checkpoint way out; the test asserts no manifest is written. Fails on the old head.

3. Verified, with one thing to say plainly. gates.py and initialization.py join _DIRECT_KERNEL_MODULES (seeds.py:349-350), so the attested identity now moves with them. The oracle imports a frozen gate module (gates_14608a5e.py), the control asserts the oracle's HardConcrete is the frozen class and not the live one, and the mechanism works: I mutated the live clamp from 1.0 to 0.999 and all three excluded-path controls failed on weight bytes. One observation: the frozen copy is byte-identical to this PR's gates.py (14608a5e…), not main's (379e4adb…), so the control cannot adjudicate this PR's own gate change — the evidence for that is that the control's numeric values did not move when only the module sha did. Fine going forward; worth one sentence in the receipt.

4. Implemented; the design-ratio numbers are only in the comment. certainty_share, boundary_draws (dataset_size.py:348-349) and both realized_max_weight_ratio_* keys (driver :2496-2513) exist, and the receipts carry the certainty counts (54,595 / 405 and 54,844 / 156) and the >100× design shares. The vs_design values themselves (1,383 and 1,679) and the 10.0 stretch-reference ratio appear nowhere in the tree — the receipts md names the keys at lines 724-725 but not the values, and no manifest with the new keys is committed. Write them into the receipt.

5. Verified. zero_target_rows is counted and the receipt records 0 on the licensed surface, consistent with the zero-valued parity rows being deferred cells.

6. Partial — the disposition says more than the body does. The two register changes are a bullet in the changes list, not a paragraph, and the body contains no comparability statement against R17 (no match for "R17" or "comparab"). The reasoning for keeping them in this PR is sound; the sentence just is not there. Also unnamed in the reply: ad519c26 changes the PIPR census notes from "all 361 remain deferred" to "314 bind, 47 deferred", which is a register-facing edit that deserves a mention.

Nits. written_at and the directory are out of the candidate manifest and stay in the checkpoint's own JSON; the options keys are additive. The parity pin replaces the darwin py3.13 entry with py3.14 rather than keeping both, and there are no linux entries in that file, so "the linux entries are untouched" is vacuous. test_spec_engine_field_usage.py, touched in 1bdd6447, would be reformatted by ruff format; CI does not enforce it, but "changed-file formatting passes" is not true of the head.

---

Approving: the three should-fixes are closed with discriminating tests and a working frozen control. The residuals are all text — the two design-ratio numbers into the receipt, the R17 sentence into the body, one line on the frozen gates being the post-PR module.