Skip to content

Add the E9 net-new UK stages: WAS debt chain segment and UC deduction attributes (#685) - #850

Open
juaristi22 wants to merge 5 commits into
mainfrom
uk-net-new-stages-685
Open

Add the E9 net-new UK stages: WAS debt chain segment and UC deduction attributes (#685)#850
juaristi22 wants to merge 5 commits into
mainfrom
uk-net-new-stages-685

Conversation

@juaristi22

@juaristi22 juaristi22 commented Sep 2, 2026

Copy link
Copy Markdown
Collaborator

Closes #685 (WS-E child E9 of #145, epic #665). Plan: repos/uk-685-net-new-stages-plan.md (approved 2026-09-02 after two review rounds); Codex implemented increments I2–I4 under the approved-plan protocol with zero deviations, followed by an adversarial review whose findings are folded in (listed below). Licensed I5 acceptance attached (Part E of the receipts); ready for review — María decides when to un-draft.

What the issue asked for, re-measured against main

Item State on main This PR
Bus fares Already built by E6 (#730): both columns imputed by QRF, clipped to donor support, bounded, allow-listed. DfT anchors are Chronicle #202 facts owned by #789; the NTS gradient reshape is #790's. No build code. Receipt Part C documents the E6 evidence; Part D (spine vs the BUS05i facts by geography) is licensed evidence handed to #789/#790. PR #153 to be closed with a pointer.
WAS debt Not taken by E5; the two columns were already allow-listed (June rows) with no producer. mortgage_debt (HMortGR8) and consumer_debt (HFINWR8_SUM − HFINWNTR8_exSLC_Sum, clipped at 0) as a fourth sequential segment of the was_wealth chain, conditioned on the base predictors, a declared has_mortgage_tenure flag (WAS Ten1R8 ∈ {2,3} ↔ FRS OWNED_WITH_MORTGAGE) and every prior WAS output. Fourth child of the same seed, so E5's fourteen columns are unchanged by construction (SeedSequence.spawn prefix-stable, pinned by test).
UC deduction attributes Nothing on main. New benunit stage uc_deduction_attributes after uc_capital_coherence, before cgt_incidence_clone.

UC deduction attributes — design (María's rulings A1–A3)

policyengine-uk 2.92.1 computes uc_has_deduction from uc_deduction_random_draw, not from the latent rate, so persisting only the two #450 attribute columns would leave the engine's flag (which every validation statistic uses) disagreeing with the held attributes on about half the units. The stage therefore persists four columns: the two identity-keyed uniforms (blake2b of benunit_id, seed 0, output-name salts, rounded to float32 and clamped to 1 − 2⁻²⁴ exactly as the engine stores them) and the two attributes derived from them by the engine's own recipe — the eleven-band inverse CDF of u / (incidence × region factor) and the normalised seven-way combination searchsorted on v — read from dwp_uc_deduction_distributions.json, a pinned mirror of gov.simulation.uc_deductions (DWP Tables 2/5/6, March–May 2025 pre-FRR, ODS sha 307ec8fa…, Chronicle package dwp-uc-deductions-march-2025-february-2026). Both of the engine's summation orders are reproduced (left-to-right for the flag threshold, NumPy for the rate). Assignment is to all benefit units (latent, award-independent; the engine gates on would_claim_uc and a positive award at run time, and universal_credit is defined_for = would_claim_uc, so a non-claimant's attributes reach nothing) and conditioned on region only, which is what DWP publishes and what keeps the mirror exact. A parallel pe-uk seam PR (flag reads the held rate) is planned; when it releases, the two draw columns retire at that floor bump.

Verification carried in the tree: engine-tree lockstep test for the resource (requires_uk), a golden mirror test that materialises the engine's three variables on held float32 draws, band-edge equality with the engine's BAND_LOWER/UPPER, closed-world operation drift asserts, permutation identity, enum names on export, and an independent I1 re-derivation of Tables 2, 4, 5 and 6 from the ODS (Tables 2 and 5 exact; Table 6 factors within 0.01 at two decimals under a denominator convention noted in the receipt).

Surfaces moved

Two generic operation kinds (map_uniform_to_banded_rate, map_uniform_to_categorical); the WAS chain op declares debt_segment_predictors; gates: uk_uc_deduction_combination_enum_domain, uk_stage_uc_deduction_attributes (stage-health latent_attribute_realization, which recomputes the binomial tolerance and holds each cell to the tighter of producer and recomputed), uk_support +uc_deduction_support_bounds.json, uk_stage_was_wealth_support +2 columns; export allow-list +4 benunit rows (the two household debt rows were the dormant June pre-declaration, reused per A5); six net_new_column register entries; coverage-manifest sha and was_wealth outputs; spec-bundle sha; contract digests; node-graph cells/consumes/kernels; the H2 fixture regenerated (synthetic WAS donor gained the three raw columns; 27-transform oracle); roster pins 28→29. The exact-k fixture also gains uc_capital_coherence, a #828 omission surfaced by the contract test.

Review findings folded in (adversarial round, 2026-09-02)

Engine summation order mirrored for both formulas; has_mortgage_tenure declared in the manifest and drift-asserted (it was executed but undeclared); the boolean predictor cast like is_renting; stage_health.py restored to main's bytes plus the new gate only (Codex had reformatted the whole module); the new gate no longer trusts the producer's tolerance; fail-path tests for every branch of the gate; a hand-edit indentation drift in source_stages.json repaired; receipts and changelog wording.

Verification

UK-specificity note

DWP taxonomies (band edges, combination enum, region factors) and the WAS/FRS column names are UK; both new operation kinds are generic, and the debt segment reuses E5's generic chain machinery.

Not in this PR

Bus targets (#789), scalers (#790), the pe-uk seam PR (I7), closing PR #153 and the pointer comments on #789/#790 (each on María's go), Chronicle facts for Tables 2/5/6.

🤖 Generated with Claude Code

@juaristi22
juaristi22 force-pushed the uk-net-new-stages-685 branch from da8e275 to c7e8505 Compare September 2, 2026 15:54
juaristi22 and others added 4 commits September 2, 2026 18:35
… attributes (#685)

Two of the issue's three items still needed building; bus fares were already
delivered by E6 (#730) and are closed by receipt, not code.

WAS debt: mortgage_debt (HMortGR8) and consumer_debt (HFINWR8_SUM less
HFINWNTR8_exSLC_Sum, clipped at zero) join was_wealth as a fourth sequential
chain segment conditioned on the base predictors, a declared
has_mortgage_tenure flag (WAS Ten1R8 in {2, 3}; FRS OWNED_WITH_MORTGAGE) and
every prior WAS output. The segment draws from the fourth child of the same
declared seed, so the first three children and E5's fourteen columns are
unchanged by construction (SeedSequence.spawn is prefix-stable; pinned).
Bounds regenerated from the licensed tab; two net-new register entries; the
dormant June allow-list rows are the pre-declaration.

UC deduction attributes: a new benunit stage after uc_capital_coherence and
before cgt_incidence_clone persists four columns. policyengine-uk 2.92.1
derives uc_has_deduction from the draw rather than the latent rate, so the
stage writes the two identity-keyed uniforms (seed 0, output-name salts,
float32-rounded and clamped as the engine stores them) together with the
latent rate and type combination derived from them by the engine's own
recipe, read from dwp_uc_deduction_distributions.json, a pinned mirror of
gov.simulation.uc_deductions (DWP Tables 2/5/6, March-May 2025, ODS sha
307ec8fa, Chronicle package dwp-uc-deductions-march-2025-february-2026) held
equal to the engine tree by a lockstep test. Both engine summation orders are
reproduced. Attributes are assigned to every benefit unit and conditioned on
region only; the engine gates on would_claim_uc and a positive award. Two
generic operation kinds, an enum-domain gate, a stage-health check that
recomputes the binomial tolerance, declared support bounds, four allow-list
rows, four net-new register entries.

Derived surfaces: manifest projection (indentation drift repaired), schema
branches, country package, coverage-manifest sha and was_wealth outputs,
spec-bundle sha, contract digests, node-graph cells/consumes/kernels, the H2
fixture regenerated with the three new WAS raw columns and a 27-transform
oracle, roster pins 28 to 29, receipts skeleton with the I1 WAS audit
figures, changelog.

Codex implemented plan increments I2-I4 with zero deviations; an adversarial
review's findings are folded in (engine summation order, declared segment
predictor, producer-tolerance cap, fail-path tests, main-byte restoration of
stage_health.py).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ributes by benunit id

The four benunit columns are pure functions of benunit_id and the household
region under the committed resource, so the licensed harness re-derives them
in original and permuted row order and compares them with the stored
artifact (--check e9). CGT clone and band-donor households are excluded and
counted: those rows carry the completed columns copied onto re-keyed ids,
not ids the stage drew on. Three hermetic tests cover the green path, a
tampered stored rate, and the copy exclusion.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…r sigma

The first licensed twin build ran all 27 stages and was blocked by the new
stage-health gate on one band cell: the receipt compared a population-
weighted share with a binomial band computed on unweighted rows. Identity-
keyed draws give every unit the same probability whatever its weight, so the
unweighted share is the mechanism's own statistic; the weighted share adds
the frame's weight variance (design effect 1.4 among assigned units on the
FRS 2024-25 spine) and belongs to the engine round-trip against the
publisher. The receipt now gates the unweighted realization and reports the
weighted one with its effective sample size. With about thirty cells per
build, the band moves from three to four sigma (familywise false alarm from
roughly one build in twelve to one in five hundred); the gate recomputes it
from target and rows and still takes the tighter of producer and own. On the
control spine's ids every cell passes with a worst ratio of 0.40.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…rity, bus hand-off

Two full builds of the branch are payload-identical, run 27 stages and pass
the spine battery 15 of 15. Against the control build the only new columns
are the six E9 columns, no common column differs and E5's fourteen columns
are byte-equal. The e9 identity receipt passes on all 30,455 unique benefit
units. The engine round-trip at 2024 and 2025 shows the flag, the held
attributes and the fallback recomputed from the held draws agreeing on every
claimant, zero deductions on non-claimants, and per-household statistics
against DWP's (mean monthly 67.05 / 50.89; at-cap 0.148 / 0.265). Recipient
incidence is read on unique rows because CGT copies double-count draws.
Debt shares are donor-faithful with levels 1.2-1.3 times the donor means,
inherited from E5's wealth surface. Strict parity's 15 unsigned columns are
identical on the control (the 2.92.1 recognition re-record, #749's lane).
Part D hands the bus totals by geography and the fare gradient against NTS
to #789 and #790.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@juaristi22
juaristi22 force-pushed the uk-net-new-stages-685 branch from c7e8505 to 132600c Compare September 2, 2026 16:50
@juaristi22
juaristi22 marked this pull request as ready for review September 2, 2026 17:01
The licensed acceptance was re-run on the rebased tree: twins e9r-twin-a and
e9r-twin-b (28 stages) are payload-identical and pass the spine battery, the
control is rebuilt from main 47c7422, attribution again isolates exactly the
six E9 columns with E5 byte-equal, the e9 identity receipt passes, and the
six columns are byte-identical to the pre-rebase twins. The engine round-trip
figures move slightly with the reporter redraw upstream (mean monthly 66.81 /
50.66; at-cap 0.145 / 0.263; unique-row incidence z 1.84 / 2.00).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Net-new UK stages: UC deduction attributes, bus fares, WAS debt

1 participant