UK spine: composition-faithful SPI income draw — bridge predictors and rank-preserving replacement (with-children UC residual)

[juaristi22]

Problem

PR #835 (#832) made the SPI reporter channel truthful — reporter→positive-award conversion is now ≈100% by construction and the 1.24m dead-reporter mass is gone — but the four deferred UC with-children receipts survive it essentially unchanged (v18: single_with_children −34.2%, children_1 −27.9%, children_2 −34.4%, children_5_or_more −29.1%). Decomposing the single_with_children gap on the v18 calibrated weights (target 2,213,760; support 1,460k; gap 754k):

Component	Weighted	Owner
Dead reporters (reported, award-killed)	0	closed by #835
Take-up residue (award > 0, would_claim_uc false)	59k	exhausted — 0.55 stands
Income/eligibility shape: lone parents with no pre-take-up award	400k, of which 195k SPI-drawn and 205k FRS-observed	this issue (the SPI-drawn part)
Structural mass shortfall: only 1.919m lone-parent benefit units exist in the spine	≥295k	separate item (concealed-family representation)

The income-shape signature is stark: SPI-channel lone parents pass the pre-take-up screen (max(0, uc_maximum_amount − uc_income_reduction) > 0) at 35% (0.351m of 1.004m) versus 69.5% on the base FRS channel (0.768m of 1.105m). The SPI channel hands lone-parent benefit units incomes that a lone parent rarely earns, so #835 correctly refuses to draw them as reporters — the defect moved from a hidden contradiction to a sized shortfall.

Why this happens

• Stage 17 (spi_support_channel) samples FRS donor households uniformly within (region, capital-gains-clone, clone) strata; composition is not a stratum, and the donor keeps its benefit-unit structure.
• Stage 18 (hmrc_spi_income_spine) stage 1 then overwrites every SPI-channel adult's 18 income components with a RegimeGatedQRF draw conditioned on age, gender, region only (SPI_SPINE_STAGE1_PREDICTORS; fit_weighted_qrf_stage1 in uk/spec/sources.yaml). Each row's regime is sampled from the gate's probabilities with an RNG uniform (_gate_draw_with_rng, microcosm-fit/qrf.py:950-962) and its magnitude at an RNG quantile (_draw_target_with_rng, qrf.py:965-989: quantiles = rng.random(n)). The donor's own observed income — and with it the composition↔income relationship the FRS carries — is discarded. A lone parent receives the income of a random same-age/gender/region taxpayer, and the taxpayer population is far richer than lone parents are.
• Composition cannot be conditioned from the SPI side: the SPI 2022-23 public-use tape carries no child, dependant, or family-composition variable (Annex A, UKDS SN 9422 — the word "child" appears only in a footnote on the abolished additional personal allowance). The only relationship signals are MAIND (marriage-allowance claimant) and MCAS (married couple's allowance — partner born before 1935). The signal has to come from the FRS side of the pairing.

Proposed fix — one increment, two mechanisms

1. Bridge predictors that exist on both sides. Extend the stage-1 predictor list with fields the SPI tape and the FRS spine share:
   • MAINSRCE (1 pay / 2 occupational pension / 3 sole trader / 4 partnership / 5 other / 6 claims case) ↔ FRS employment_status (+ self_employment_income/PENSION-style precedence) — the mapping is a translate-published-facts adjudication, documented coding by coding;
   • INDUSTRY07 ↔ sic_industry_division (frs_employment);
   • SPA ↔ age vs state pension age;
   • SEINC_NUM ↔ self-employment indicator;
   • MAIND ↔ is_married — a documented approximation (claimants are a subset of married couples with one low-earning partner), included as a weak couple signal, never as a lone-parent signal.
     A part-time retail employee then draws from part-time-retail-employee taxpayers, not from every 35–44 woman in the region; most of what composition does to income rides on these fields.
2. Rank-preserving replacement. Supply the per-row draw quantile from the donor's observed FRS income rank within the predictor cell instead of rng.random(n) (and the gate's uniform likewise), so the FRS lone parent at her cell's 30th percentile receives the SPI cell's 30th-percentile income. This preserves the FRS composition↔income-rank joint while importing SPI's marginal — the fat top tail the channel exists for. Ranks are robust to the FRS's under-reporting of top-tail levels. To adjudicate: rank on the donor's FRS total income vs per component; tie/zero handling within the regime gate; whether the gate keeps its own stream.

Both mechanisms live in stage 18, so they move the pinned stage-1 output surface (_assert_income_stage_parameters), re-draw every SPI income column and, through the stage-2 chain (whose predictors are those incomes), every FRS-only fill on SPI rows — a whole-SPI-surface signed-difference campaign. The twin compare must show the diff confined to SPI-channel rows; the base channel stays byte-unchanged.

Companion (separate, calibration-side, not this increment): a joint target on high-income × has-children from HMRC's High Income Child Benefit Charge / Child Benefit statistics, so the solver cannot leave lone parents rich regardless of how the draw got there.

Explicitly out of scope

• No take-up changes (the residue is 59k; 0.55 stands).
• No composition-aware donor selection at stage 17 — it needs taxpayer-composition evidence the SPI tape does not carry.
• Not the ≥295k structural lone-parent mass shortfall (the ONS lone-parent-household target fits exactly at 1.835m; the missing units are benefit units embedded in larger households) — its own item.
• Not the B-lane (num_uc_children engine variable + numbered-child rebind, plan D5 of UK spine: condition SPI UC receipt on benefit-unit composition and add a UC-specific child count #832).
• No SPI/WAS tail suppression; the 208-fact HMRC replay and its fences stay in force.

Acceptance criteria

• Stage-1 predictors carry the bridge fields with adjudicated, documented mappings from the published SPI codings to the FRS columns.
• Draw quantiles come from the donor's within-cell income rank; declared in the manifest; twin determinism preserved.
• Before/after receipt by family type and channel: screened-eligible share of SPI lone parents (35% → toward the 69.5% base rate) and the no-award SPI lone-parent mass (195k → ?).
• Twin compare: diff confined to SPI-channel person income columns and their stage-2 / reporter / coherence dependents; base-FRS rows byte-unchanged.
• The stage's HMRC income-band replay and distributional-mass gates stay green (the SPI marginals are still the SPI marginals).
• Rebuilt candidate: recalibration + T0–T6; re-measure the four with-children receipts with the childless cells, housing benefit, pension bands and CGT on the sibling watch; solver-pressure movement reported from calibration_diagnostics.json.
• Every moved column signed with receipts before any exclusion changes.

References

• Port UK compact source stages and 2024-25 refresh from uk-data #145 — draw-quality workstream (this is its draw-shape increment; María's condition-the-draw suggestion was routed here from UK spine: preserve FRS benefit-unit capital for UC while retaining WAS household wealth #828's review)
• WS-E spine deferrals: known bugs and signed inconsistencies awaiting their increment #796 — WS-E spine-deferrals tracker: register this as the fix arm for the four UC with-children receipts (renewal-or-fix, window to 2026-09-30)
• UK spine: condition SPI UC receipt on benefit-unit composition and add a UC-specific child count #832 / PR UK spine: benefit-unit-grain SPI UC reporter draw with pre-take-up screen (#832); exact capital sentinel (#833) #835 — the reporter-channel fix whose measurement located this residual (experiments/832-uc-reporter-receipts.md; v17/v18 receipts on uk-publication-stack-835)
• UK spine: preserve FRS benefit-unit capital for UC while retaining WAS household wealth #828 / UK spine: preserve FRS benefit-unit capital for UC, re-anchor take-up after SPI (#828) #829 — capital coherence; UC residual after the pension-wealth split: support 4.24m vs 6.76m — re-draw undershoot, the 0.55 take-up freeze, and composition shape #809 (closed) — M1 draw-quality evidence
• SPI 2022-23 Public Use Tape documentation, UKDS SN 9422 — Annex A variable list

🤖 Generated with Claude Code

[juaristi22]

B-lane hand-off from #832: num_uc_children and the numbered-child rebind

#832 closes with PR #835 (the reporter-channel fix); its second half — the UC-specific child count for the five DWP numbered-child cells — was split off as a separate lane (plan #832 D5 / A4) and is recorded here so it does not live only in a closed issue. It is a target-semantics correction, not a gap-closer: on spine-k about 78k weighted positive-UC benefit units change count band under the UC definition (one-child ≈ +1k support, two-child ≈ −5k, five-plus ≈ −3k).

Why it has to be an engine variable. The target resolver (uk_runtime/measure_simulation.py, UKMeasureResolver.knows/compute) resolves filter variables only through tax_benefit_system.variables, and a person-grain boolean routed to benefit-unit grain collapses to ANY (bool_any_collapse_person_to_benunit), never a count. So there is no binding-layer escape hatch (value_expression covers +-joined same-grain columns on value_variable only, not filters).

The change, in order.

1. policyengine-uk: add num_uc_children — BenUnit, int, YEAR, benunit.sum(benunit.members("is_child_or_qualifying_young_person_for_universal_credit", period)). The flag already excludes looked-after children and is the one the engine's own child element, child-limit and disabled-child logic use (uc_child_index, uc_is_child_limit_affected, …), so the count matches "children on the claim" as DWP counts them, including 16–19 qualifying young persons.
2. microcosm floor bump to the carrying release (packages/microcosm-build/pyproject.toml uk extra, uv.lock), with the UK WAS stage: split private pension wealth out of corporate_wealth (uk-data#452 M2) #750 lesson applied: check whether the new version newly recognizes loader columns (unknown_export_columns_excluded in the parity extraction) and, if so, re-record the licensed coverage/parity evidence from the same pinned artifacts (the ae9457a3 precedent).
3. Rebind in uk/uk_population_targets.json: the five cells dwp.uc.households_children_{1,2,3,4,5_or_more} change their filter {"variable": "num_children", …} → {"variable": "num_uc_children", …} (operators unchanged: == 1/2/3/4, >= 5); update each notes — the "16–19 qualifying young persons … declared residue" sentence is discharged for these cells. Regenerate target_references.json and the membership report with tools/generate_uk_target_references.py --contract … --ledger-facts … --period 2025 --output … --membership-report ….
4. Tests: test_uk_population_targets.py (allowed_filter_variables, currently {universal_credit, num_children, family_type}), the test_uk_ledger_targets.py fixture benunit table.
5. No register re-freeze: filters are not in the frozen-register TargetSpec hash (registry.py), unlike Re-bind the DWP UC composition and TCL disability-children targets to the publisher's grain (#808, #809) #813's entity/measure flip — verify TargetRegistry.version is unmoved; verify_register_coverage.py's family counts and the headline range are unaffected.

Declared residue, on purpose. The four family-type cells keep their binding: family_type derives LONE_PARENT / COUPLE_WITH_CHILDREN from the same age<18 is_child, so they carry the QYP omission this lane fixes for the numbered cells. #832 kept them out of scope; whoever takes this lane should say so in the cell notes rather than widen the change.

Synergy with #835: the reporter-redraw stage already computes its UC child band from the same engine flag (summed in-stage), so once the engine variable exists there is one definition of "children on the claim" across the draw and the targets.

Owner per the #832 plan: María.