Correct UC measurement contracts and calibration iterate selection

[juaristi22]

Partially addresses #882. Builds on merged #879 and #881, rebased onto official main at 923cec2e.

UC calibration should compare administrative claims with the corresponding benefit units and their own children. These changes prevent unrelated people in the same dwelling from contributing to a claim's child/disability categories, align GB coverage and finite payment-band bounds, and expose the approximations between annual model outputs and source statistics.

• Derive claimant/partner roles from retained FRS relationships, preserving older dependent children and cohabiting couples. Use that couple definition in UC reporter and capital-donor predictors, and carry the explicit Boolean is_uc_claimant input through graph storage, export and model loading.
• Count TCL targets at benefit-unit grain using own qualifying children, claimant PIP and own disabled-child elements. Separate statistical post-cutoff child proxies, including exceptions, from the model's element-denial flag. Require the Scottish under-one child and claim to belong to the same benefit unit.
• Apply GB predicates and cap the last included payment band at £2,500 monthly, inclusive. For 111 compiled UC references, require each April–December 2025 observation exactly once and record the nine-month denominator. Official target values, selectors and existing exclusions remain unchanged.
• Retain the best feasible iterate within the requested budget for deterministic, ungated, unregularised free-mass Adam solves. Compare the conserved-mass, L2 and L0 paths with a frozen pre-change optimizer on the same runtime, checking exact weights, complete loss trajectories and gate probabilities. Refresh the affected calibrate H1 identity while retaining its direct-output comparison.
• Add development replay/matrix tools and refresh affected source-coverage and kernel identity receipts. Hash the claimant helper into its three consuming stages and the model-loading adapter into its eight consuming stages, so changes invalidate the relevant cached results.

The UK extras require published PolicyEngine-UK 2.97.0, including the merged claimant-aware formulas in #1850, and the lock selects that release. Supplied FRS roles are authoritative: the model fallback cannot reconstruct a 17-year-old partner or distinguish every older dependent from the claimant's partner. Both export allow-lists permit the new input, and model-loading checks reject a supplied claimant column that the engine cannot consume. Reporter payment landing still selects one row by its established non-child/working-age ordering; this is separate from identifying the one or two members of the claim.

Best-iterate selection reaches the UK national free-mass production solve and the local rowwise default. Their returned weights can change. Epoch defaults, learning rates and weight bounds remain unchanged; the byte controls cover the paths excluded from this selection rule. These checks do not establish the effect on a rebuilt national population or certify new weights.

Current lock, seed, specification and UK gate-verifier mirrors follow their producers. Source comparisons isolate the solver implementation identity and claimant export allow-list addition; seed values, owner assignments, gate scopes and thresholds are unchanged. Historical execution receipts retain their actual runtime identities, including the frozen eFRS candidate's PolicyEngine-UK 2.89.0 provenance.

The rebase preserves #881's survey-year disability rules, survey-year grant/calibration-year DSA split and declared year-rule precedence alongside the claimant checks. Regeneration changes only three current identity artifacts: 13 source-manifest digest leaves, two H2 stage-contract hashes and the UK specification assertion. It does not relabel historical population evidence.

TCL measures remain positive-award and model-birth-year proxies for the administrative April snapshot. They do not reconstruct open/nil claims, exact birth dates or April claim history. The statistical definition includes exceptions; these proxies do not reconstruct administrative exception categories. Missing source crosses remain tracked in Chronicle #251; remaining consumer/support work stays in #882. The diagnostic results and remaining support limits identify the two active zero-support payment cells and the five-plus fit’s dependence on one original source household.

Validation:

• Combined rebase integration — 388 passed across 20 test paths with stock UK 2.97.0/Core 3.31.0; one existing test skipped because the licensed pinned SPI donor is not staged locally. Includes policy-year, claimant-loading, measurement, source, graph and identity checks.
• Current coverage, H2 and specification producers; exact frozen hashes verified after the formatting pass. Other country specification hashes remain unchanged.
• Full Ruff, whitespace, CI inventory and acceptance ratchet against official main at 923cec2e; independent rebase/source/producer verification CLEAN.
• Retained unchanged-source evidence: 225 calibrate tests, H1 direct-byte parity and three installed-wheel portable solver controls. These were not rerun for this rebase; their solver, control and H1 input bytes and locked dependencies are unchanged.

Latest commit 61711ff9 is based on official main at 923cec2e. GitHub CI PASSED all 23 jobs on this exact head and base. A full population rebuild and certification remain separate from code and synthetic checks.

[vahid-ahmadi]

Review pass at 9d13d1dd (Claude Code, high effort; worktree checkout, the PR's own diff against #879's head eb7ca8a5 — 47 files, +3,778/−705 — read alongside the companion policyengine-uk #1850, which I reviewed this morning; without the engine: the in-scope build, calibrate and data suites all pass with only requires_uk and missing-feed skips, but adding microcosm-graph/tests/test_acceptance_h_parity.py gives exactly one failure; ruff check and ruff format clean on all 31 changed Python files; both census tools, the coverage manifest and ci_test_groups --verify current. No hosted CI runs on this PR because the workflow triggers only on PRs to main.)

The measurement work is careful

The shared claimant helper (uc_relationships.py:52-59) derives claimants as head, parent, or every member of a childless unit, refuses a unit with other than one or two claimants, and is now the single couple definition for the redraw predictor (uc_reporter_redraw.py:58, 73, 528) and the capital-donor stage (uc_capital_coherence.py:206), with is_married dropped from the required inputs and the stale contract refused by test — that closes the calibration/redraw divergence I raised on #879. The TCL measures group on person_benunit_id and reindex on benunit ids so nothing leaks across units or drops; the post-cutoff proxy never reads the engine's denial flag; the caveats are carried in the contract and surfaced through materialisation. The band cap is a declared band_upper_bound: 2500 with an inclusive flag, materialised as nextafter(30000, inf) so £30,000.00 is in and £30,000.01 is out, and invalid declarations raise. GB is a declared household region predicate projected to benunit grain with unmatched links raising. The nine-month requirement is a per-target source_months declaration validated post-compile — exactly one fact per declared month, refusing missing, extra, duplicate or wrong-year — with a failure surfacing as unsupported and the national calibration raising; compiled values are unchanged (111 metadata lines added, zero value lines moved). The helper is hashed into the three consuming stages' kernel identities and the mutation test proves it. All of that checks out.

Findings

1. Blocking — the calibrate kernel's parity pin is stale, so test_h1_kernel_parity fails. kernels.py:178-181 hashes solve.py's source; this PR edits solve.py but leaves packages/microcosm-graph/tests/fixtures/parity/kernels/calibrate/pins.json at a7a0330f…, and the assertion at test_acceptance_h_parity.py:177 sees 142dc3bf…. Swapping the base solve.py back in restores the match, so the cause is the solver edit. Because the hash assert fires first, the calibrate kernel's direct.csv byte-parity check never runs, which means the conserve-mass path the PR says is unchanged is currently unguarded. Regenerate the pin and let the parity check run.

2. Blocking — person.is_uc_claimant is exported but on no allow-list, so the release-cut export gate refuses it. The column is added to frs_spine.py's outputs and the graph root, but uk_export_surface.allowed_extra_columns in gates.json and UK_ALLOWED_EXTRA_EXPORT_COLUMNS in terminal_gates.py do not carry it (they carry person.is_parent), and the eFRS reference surface does not either. uk_export_surface_gate computes unexpected = (candidate − reference) − allowed_extras; calling it with the reference plus this column returns passed=False naming person.is_uc_claimant. The body's line about the frozen coverage candidate refers to the coverage manifest, not this gate. Either allow-list the column (it is the whole point of the companion #1850) or explain why it should not ship.

3. Should-fix — the best-iterate solver change reaches the UK national production path with no byte-identity guard. solve.py:842 applies retention iff gates is None and not conserve_mass and l2_lambda == 0, and gates exist only with l0_lambda > 0 or target records. The national doctrine is mass_rule = "free" and l0_lambda = 0.0, and national_calibration.py:128-142 passes neither l2_lambda nor target records — so the national solve is exactly the changed path, and so is the local rowwise default. The three new solver tests are positive cases; nothing pins that conserve, L2 or L0 runs are byte-identical to before, and nothing measures what the change does to certified national weights. This is a generic calibrate-package behaviour change riding in a UK measurement PR; split it out, or at minimum add the byte-identity fixtures for the untouched paths and say in the body that free-mass national weights can move.

4. Should-fix — the helper and #1850's fallback disagree on the ambiguous cases, and results depend on whether the column ships. Microcosm: head, parent, or childless-unit member, no age test. Engine fallback: head, parent, or adult-and-not-qualifying-young-person. Divergences: a 17-year-old cohabiting partner in a childless unit is a claimant here (the PR's own test_uk_frs_spine.py:918-951 asserts it) but not in the fallback; a 19-year-old dependent past the terminal date is a claimant in the fallback but not here. With finding 2 in force the column cannot reach a certified export, so the engine would use its fallback and compute UC on a different family from the one calibration measured. The contract between the two should be stated once, in both PRs; I raised the mirror of this on #1850.

Questions

• The engine-backed checks used a 2.94.0 development runtime with #1850 applied while the lock pins released 2.94.0, which lacks is_uc_claimant; efrs_parity_known_gaps.json's version bump is a one-line hand edit. Nothing here is reproducible against the locked dependency until #1850 releases and the lock moves — worth saying in the body.
• The redraw's landing rule (_claimant_rows, eldest working-age adult by is_SP_age) is not unified with the helper; couple status is. Presumably intentional; state it.

Nits

The changelog fragment holds three entries; the composition error text changed.

---

The contract corrections (A through E, G, H) are the substance and they are sound. 1 and 2 are objective — a failing parity test and a gate that refuses the new column — so request-changes until they land; 3 and 4 are the two design points I would want settled before this reaches a certified build.

[juaristi22]

Thanks, Vahid. Addressed your review through latest commit 61711ff9, now rebased onto official main at 923cec2e after #879 and #881 merged.

• Regenerated the calibrate H1 identity with its producer, preserving the fixture inputs, graph and direct-output bytes.
• Added person.is_uc_claimant to both export allow-lists, with tests for declared/runtime agreement and continued rejection of unrelated columns.
• Kept best-iterate selection with the alternative controls you requested: conserved-mass, L2 and L0 paths compare exact weights, full loss trajectories and gate probabilities with the frozen pre-change optimizer on the same runtime. National and local free-mass weights can move; the description and changelog make that scope explicit.
• Require released UK 2.97.0 across the three UK extras and lock. Explicit FRS roles remain authoritative; adapter and direct-H5 loading reject dropped or altered inputs. Released-engine tests cover a 17-year-old partner, an older nonqualifying dependent and ordinary child/couple cases through final UC.
• Restored the frozen eFRS evidence's original 2.89.0 provenance and clarified that choosing one payment-landing row is separate from claim membership. Current identity mirrors follow their producers; historical receipts remain unchanged.

The new rebase preserves #881's policy-year rules alongside these fixes. Only three generated artifacts required refresh: the source-coverage manifest, UK specification assertion and H2 graph stage-contract hashes. Combined integration passed 388 tests, with one existing licensed-SPI-donor test skipped because its input is not staged locally. Formatting, lint, inventory, the official-main ratchet and independent verification passed. Earlier calibrate (225 passes), H1 and portable-control evidence is retained only for unchanged source/input bytes and locked dependencies; it was not rerun for this rebase.

CI at 61711ff9 PASSED all 23 jobs against official main at 923cec2e. The separate diagnostic comment records the remaining empty payment cells and concentrated five-plus support. These changes do not establish that the population miss is closed or certify a rebuilt dataset.

[juaristi22]

Adding the diagnostic results behind this PR, separately from the implementation/review reply. The repairs address demonstrated relationship, measurement and optimizer defects. They do not yet close the UC population miss. Zero-support cells are already acknowledged in the public #879 comparison, and #882 requests a joint-support audit. They should also be explicit here, alongside what the later checks establish.

The population results below are saved development diagnostics reviewed on September 7: retained 2024 spines, 2025 model outputs, UK 2.94.0/Core 3.31.0 and Chronicle 6fb700e. The later claimant/contract replay used five isolated UC formula corrections on that engine. These are not results from a complete rebuild with the subsequently released UK 2.97.0, its further reviewed claimant/disability fixes and this PR’s final source-stage integration. Current package/synthetic tests validate those implementations separately.

The historical paired SPI experiment improved lone-parent UC from 1.528m to 1.605m after calibration, but still missed its target by 27.92%. Its 39.81% total-loss improvement mainly came from interest, which supplied 91.5% of the net gain. The paired aggregate receipt therefore supports retaining valuable SPI improvements, without interpreting the overall loss reduction as resolution of UC composition.

The subsequent checks separated mechanisms:

Check	Retained diagnostic result	What it establishes
GB scope, unchanged awards/weights	Removing NI removes about 189,638 positive-UC units from the UK estimate.	Comparing UK model mass with GB targets understated the GB shortfall. Correct geography can worsen apparent fit.
Five claimant/child formula corrections, fixed weights	GB positive UC +928; lone parents −121; annual UC −£377.4m.	Real entitlement defects, with offsetting component effects, rather than a large caseload recovery.
Own-benefit-unit/TCL contract repair, equal solver budget	Lone-parent error −26.32%→−26.73%; total UC −10.50%→−10.85%; general five-plus −28.79%→+0.011%.	Child/grain corrections resolve a specific measurement conflict; broad gaps persist.
Positive-weight diagnostic on the repaired matrix, under the same free-mass/cap-10 weight bounds	Lone parents −0.27%; total UC +0.26%; source-household ESS about 2,514→1,764.	Much better headline fits are attainable on existing support, at greater concentration; these are not proposed release weights.

The target-contract comparison held all 371 target names/values, priors and family coefficients fixed: 1,500 updates, learning rate .02, seed 0, free total mass, weight/prior cap 10, loss cap 10, no sparsity penalty and the same best-iterate selection. For 111 compiled references, the guard requires each April–December observation exactly once: nine observed months, not a full calendar year. It reproduced the tested matrix and all solver inputs exactly. Objectives on changed measurement definitions are not directly comparable. The diagnostic weight vector respects those bounds; it does not pass every target/fit gate—the two empty rows still have −100% errors—or certify convergence or the population. Best-iterate selection fixes which result is returned within the requested budget; it cannot create missing support, and default epoch counts are unchanged.

Two active payment cells remain entirely empty in that repaired matrix:

Childless-couple target suffix (@2025)	Published monthly payment	Target claims	Model support/estimate
annual_payment_27_600_to_28_800	£2,300.01–£2,400	746.33	0
annual_payment_28_800_to_30_000	£2,400.01–£2,500	605.78	0

Both use the prefix dwp/uc_payment_dist/COUPLE_NO_CHILDREN_. Together they represent about 1,352 target claims. Every household contributes zero, so changing weights, running longer or copying unchanged records cannot fill them. Neither row was newly excluded. The historical all-band audit also found the already-excluded £2,200.01–£2,300 row empty in both spines; that is separate from the two active rows above. Nearby support was thin: 13 active SPI payment bands had fewer than five original source benefit units. Clones do not count as independent sources.

These are zeros in the captured model support, not proof that those administrative payments are impossible. Earlier apparent childless-couple support came from a dependent being classified as a partner. Correcting the classification exposed the gap. Separately, DWP payment amounts include third-party payments and one-off advances in their issue month. Annual recurring model UC divided by twelve is not an exact cash-payment bridge. This PR fixes the finite £2,500 endpoint; the separately published £2,500.01+ category remains unbound. We should diagnose components/payment timing before generating recurring entitlement to fill those cells.

The five-plus result is a different support warning. Previously, the matrix imposed general 5+ = TCL 5 + TCL 6+, against targets of 77,376.78 versus 55,280. Correct child definitions break that identity, but only through four clone columns from one original source household. The repaired solve assigns them about 22,086 units against a 22,097 target difference. That is a fragile fit, despite almost unchanged global effective sample size; it is not newly adequate independent support. The retained GB own-qualifying-child 5+ sample, from 33 original source benefit units, also contains no older, unaffected cohort under either tested model-birth-year approximation. Retained 2024 ages do not advance in the 2025 replay. Exact birth dates and April open-claim state are unavailable, so this cannot establish real-population absence or source inconsistency. DWP’s TCL population includes exceptions and April open claims, whereas these remain annual positive-award/cohort proxies. These proxies do not reconstruct administrative exception categories.

The remaining work is therefore explicit:

1. Microcosm Align UK UC claim, child and period definitions #882: finish paid/open/nil-claim, payment-amount, own-child and period/age-anchor reconciliation; trace empty and thin joint cells through source, imputation, eligibility and calibration, retaining original-source support and concentration diagnostics.
2. Chronicle: use now-merged #250 in a separately pinned Total/history reconciliation; #251 requests available payment-status × child-entitlement composition crosses. New facts cannot themselves create model support, and separate marginals cannot identify a missing joint.
3. Rebuild the combined claimant/reporter/capital pipeline with released UK 2.97.0, then recalibrate and assess target-specific support, protected outcomes and release gates. Only then choose evidence-led support enrichment or further optimization changes. No target-value, take-up or exclusion change is justified solely by these residuals.

[vahid-ahmadi]

Second pass at 61711ff9 (Claude Code, high effort; worktree checkout against main 923cec2e, the four response commits read against the round-one findings; 27 suites run: 1,045 passed, 35 skipped, 10 failed — the ten are all test_us_multispine_pool_tool.py failing on the missing policyengine-us extra identically on a clean main worktree, so environmental, not the PR; every round-one suite passed or skipped for a stated extras reason; ruff check and ruff format clean on all 45 changed Python files; both census tools, the coverage manifest and ci_test_groups --verify current; CI on the head 23 of 23 green. The UK extra is not installed here, so the requires_uk tests rest on CI.)

1. Fixed, verified. pins.json now carries 142dc3bf…, test_h1_kernel_parity passes with the bitwise comparison executing, and putting main's pin back reproduces the failure. direct.csv, graph.json and inputs.csv are unchanged against main; only the pin moved. Minor: the pin's platform label went py3.14 → py3.13, harmless for a bitwise kernel.

2. Fixed, verified. gates.json:621 and terminal_gates.py:210 both carry person.is_uc_claimant, the declared set equals the runtime tuple, uk_export_surface_gate now passes with the column and still refuses an unrelated one, and main's terminal_gates.py reproduces the round-one refusal text. Tests in test_uk_terminal_gates.py:273 and test_uk_graph.py; the gate-battery digests follow.

3. Addressed, with one receipt gap. The frozen optimizer is a vendored verbatim copy of solve.py from main commit 1f7d676f (fixtures/pre_best_iterate/optimizer_1f7d676f.py), and controls.json pins its module sha, the _optimize source sha, five helper source shas taken from the installed module, the gates.py sha and the prune tolerance; test_best_iterate_preserves_pre_change_excluded_paths runs oracle and current solver from separate seeds and asserts tobytes() equality on weights, the full loss trajectory and the L0 gate probabilities for the conserved-mass, L2 and L0 cases, plus the recorded Mac bytes on the authoring runtime — that branch executed here and passed. The tests live in the shared filter and run in the engine tier's other-shards process. The body and uc-calibration-best-iterate.fixed.md now say plainly that free-mass national and rowwise-local weights can change and that this establishes no national fit stability. Residual: iterate_selection lives only in calibrate()'s options; neither national_calibration.py:199-205 nor local_rowwise.py:1339-1342 copies it into the stage receipt, so a frozen manifest cannot show which iterate a free-mass solve returned. One field.

4. Addressed; the contract is stated, in docstrings. The three UK extras and the lock are at released 2.97.0. validate_uc_claimant_input (adapters/policyengine_uk.py:165-204, called from materialize and the direct-H5 resolver) refuses a definition that is not person/bool/year, a source column that is not complete Boolean, a column the loader did not retain as an input, and loaded values that differ from the supplied ones; six defect cases are tested and the direct-H5 refusal is tested both ways. The released-engine test covers the 24-year-old with a 17-year-old childless partner through final UC. Two notes: the validation returns silently when the column is absent from the frame (the production spine always emits it, so not blocking, but a frame without it gets the engine fallback with no signal); and the post-terminal-date 19-year-old dependent, the other divergent case from round one, is not in the released-engine test — every row there has the terminal-date flag true. The authoritative-roles contract is in uc_relationships.py:24-28, the adapter docstring and the PR body; nothing in the changelog or docs.

5. Verified. efrs_parity_known_gaps.json is unchanged against main at 2.89.0, and the new coverage-manifest test pins the frozen candidate's original engine provenance.

6. Rebase commits are regeneration. 20d29772 carries the actual fixes; 3cb98e02, dcf15048 and 61711ff9 move the approved lock digest, the US spec sha, the UK spec sha, the coverage-manifest shas and the parity fixture. The US surfaces move because microcosm.calibrate.solve is a direct kernel module whose source sha feeds every spec_sha256, so any solver edit ripples there by construction. Nothing out of scope.

7. Partial. The body says these checks do not establish the effect on a rebuilt population or certify new weights, and links the diagnostics comment; only that comment says the UC population miss is not closed and cites the −27.9% lone-parent residual. The changelog is silent on it. Since the diagnostics comment is the honest statement of where this leaves #882, one sentence of it belongs in the body proper.

---

Approving. Both blockers are fixed with the pre-fix failures reproduced, the solver change is now controlled byte-for-byte on the paths it claims not to touch and declared on the paths it does, and the claimant contract is enforced at both load paths. The four residuals — receipting iterate_selection, the silent-absence branch, the 19-year-old case, and the population-miss sentence — are all small and can follow.