UK rowwise candidate: solve local and national targets as one matrix on the cloned spine (#762 PR A) - #852
Draft
juaristi22 wants to merge 8 commits into
Draft
UK rowwise candidate: solve local and national targets as one matrix on the cloned spine (#762 PR A)#852juaristi22 wants to merge 8 commits into
juaristi22 wants to merge 8 commits into
Conversation
…ed spine (#762 PR A) One TargetSet, one weight vector, one calibrate() call: the compiled local surface (both grains, present cells only) and the ladder household rows join the activated national families as rows of a single sparse constraint matrix over the K-cloned spine, with the policyengine-uk engine resolved on every cloned row. Adds the long-format multi-grain surface builder, registry-named local rows with lazy area masks, national rows in the doctrine solve and the rotated holdout, the grain_equal weighting rule as a receipted override, the f001/f010/f100 scale ladder with a below-f100 gate posture, sha and ledger pins, per-clone engine blocks, a reviewed-exclusions channel for the area-support gate (register ships empty), manifest v2 with survey and calibration years, uprating-hold receipts on the local surface, support-limited-miss diagnostics, a scorer that accepts a candidate superset, and the incumbent local-surface extractor. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
#762 PR A) The first licensed dry-run refused at the sampling step: the driver sampled even at the full rung, through the certified-compact policy that keys families on clone_index arithmetic the microcosm spine does not carry. The spine's lineage is explicit (source_household_id, the support-clone index, and the SPI, capital-gains, and band-donor flags), so the ladder now keeps f100 untouched and samples whole source families stratified by the raw row's region below it, through the shared frame sampler with one full-mass renormalization; orphan derivatives and multi-region families are refused. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…al control rows (#762 PR A) The second licensed dry-run refused in the surface assembler: national compiled specs carry their geography under ledger_geography_level and ledger_geography_id (country grain at K02000001 or K03000001), while only the area-authored local specs also carry the local spelling the assembler read. One private resolver now prefers the local spelling, falls back to the Ledger spelling, refuses blanks and disagreements by target name, and checks a national level against the contract's declared levels; the vintage census, diagnostics registry, and the end-to-end stub use the same shape. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… A rebase) The rebase resolution replaced the first digest in the file, which is the Armenia constant, instead of the UK row; both rows now carry the values the merged tree produces. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
juaristi22
force-pushed
the
uk-rowwise-candidate-762
branch
from
September 2, 2026 17:32
a03c2b7 to
9d70c2c
Compare
…ions, with a receipt (#762 PR A) The third licensed dry-run reached the cross-grain pass and was refused: the household-composition partition bridge is partially bound because three of its national cells are microcosm#791's reviewed measure exclusions. A partial bridge whose every missing member carries an in-force reviewed exclusion is now classified unbound and listed under unbound_bridges in the receipt, so the lower side binds as published and the gap stays enumerable; any partial binding with an unreviewed missing member still fails closed by name. The driver passes the measure-exclusion receipt through and surfaces the entry in the dry-run plan and the manifest. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…r geography (#762 PR A) The fourth licensed dry-run reached the engine's control rule and was refused: fifteen activated national targets fan out into several cells at one geography (UC payment-distribution bands, SPI income bands), and the UC family-type partition is the exact-signature control for the by-area UC count. The UK surface assembler now groups national specs by contract target and geography and hands the engine one control row per group, the summed total for fan-out groups, receipted as fanout_controls_summed; every cell still solves as its own row, and non-finite cells or mixed levels refuse. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ands at benefit-unit grain (#762 PR A) The award-band table is the full UC caseload with overlapping cumulative bands, and the compile activates only a subset of those bands, so a fan-out national target can never act as a cross-grain control: the surface assembler now emits no control row for it, drops it from the bound ids handed to the engine, and receipts it as fanout_targets_not_controls (replacing the summed control). Per Maria's A1-4 ruling the four UC child-band local metrics classify each UC benefit unit by its own num_children and sum benefit units into households, matching the national benefit-unit re-bind; names and order are unchanged and the census descriptions are regenerated. The synthetic candidate tests pin an empty support-exclusion register so real micro-LA entries never collide with a synthetic roster. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The local age bands (ons.age.0_10 .. 70_80, half-open) and the national single-cell UK totals (ons.population.age_0_9_by_region .. 70_79, inclusive) describe the same integer-age populations under different edge encodings, so the exact-signature match never fired and the local bands never rescaled to the national totals. Per Maria's ruling each local band now has a declared bridge to its UK control; the national 80-89 band has no local counterpart and stays unbridged. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
First of two PRs for #762 (plan:
repos/uk-762-rowwise-candidate-plan.md, approved 2026-09-02) — the joint-solve machinery. The second PR carries the signed registers, the measured doctrine adjudication, the receipts, and the local-area release contract/assembly after the licensed runs.What this does
One matrix, one weight vector, one solve.
tools/build_uk_rowwise_candidate.pynow clones the spine through the OA ladder at K, runs the policyengine-uk engine on the cloned frame, and solves the national families (compiled + measure-excluded, 364 activated references) and the local families (compiled references over both grains minus signed deferrals, plus the ladder's census-household rows) as rows of a single sparse constraint matrix in onecalibrate()call under the local doctrine. No national-only solve exists on this path; the national line's diagnostics are only a comparison yardstick.local_rowwise.py): a long-format multi-grain builderbuild_uk_rowwise_local_surface_matrix(constituency + local authority, present cells only, signed deferrals are absent rows, canonical grain/metric/area order, fail-closed on unreachable nonzero targets and per-grain roster coverage). The dense single-grain builder delegates to it and keeps its legacy area-major result bit-for-bit. Local rows are named by the compiled register'srow_name, so the scorer joins by identity; area masks are materialized lazily from an int32 index.UKRowwiseNationalRows,solve_uk_rowwise_weights_under_doctrine(..., national_rows=, target_weight_rule=, restore=)): local rows first, national rows after, oneTargetSet; row-name alignment asserted over every compiled row; diagnostics and past-cap censuses reported per row kind; prepared slash-named measure columns are restored away before the frame rebuild and refused if any survive.rotated_uk_local_holdoutkeeps the national rows in every training fold.ledger_targets.uk_local_target_surface): compiled local registry → present cells (family viafamily_for_metric, compiled source family retained), ladder household rows for both grains, thenapply_uk_cross_grain_reconciliationwith the national control rows present (the standing country > constituency > la rule and bridges from Declare cross-grain target reconciliation for the UK local surface: national wins, sub-national rescales (#495 increment 6 companion) #802/UK local target activation: uk-data parity register, masking doctrine, council tax, PIPR measurements (#147 PR 2) #831 fire in the joint surface). A declared bridge whose missing national members are all in-force reviewed measure exclusions is classified unbound and listed undercross_grain.unbound_bridgesin the plan and manifest instead of refusing the surface; a partial binding with any unreviewed missing member still fails closed by name. On the live surface this is the household-composition partition bridge (three cells are UK spine: relationship-to-head frame column, so the ONS household-composition split is measurable #791's exclusions to 2026-11-26), so the ladder's census household counts bind as published until that is adjudicated (plan item A11). A national target that fans out into several cells at one geography (UC payment-distribution bands, SPI income bands: 15 targets, 213 cells on the live compile) is a distribution, not a control: it hands the engine no control row, is dropped from the bound ids the engine sees, and is receipted asfanout_targets_not_controls(the compile activates only a subset of the DWP award bands, so their sum can never reproduce the caseload); every cell still solves as its own row. Eight declared age-band bridges (localons.age.{b}at both grains ↔ the single-cell nationalons.population.age_*_by_regionUK totals; 80–89 has no local counterpart) let the local age bands rescale to the national totals despite the half-open vs inclusive edge encodings that defeat the exact-signature match.local_doctrine.py): weighting vocabulary("uniform", "grain_equal")—grain_equalgives the national rows, the constituency rows, and the LA rows one equal share each, uniform within, derived from row labels through the nationalfamily_equalarithmetic;uk_local_doctrine_with_overridesreceipts the single overridable field.UK_LOCAL_TARGET_LOSS_CAP10.0 andUK_LOCAL_MAX_WEIGHT_RATIO100.0 are unchanged.--sample-fraction {0.01, 0.10, 1.0},--sample-seed): f100 never samples (sampled: false); below it,sample_uk_spine_frame(new beside the certified-compact policy innational_sampling.py) samples whole source families keyed on the spine'ssource_household_id, stratified by the raw row's region, through the shared frame sampler with one full-mass renormalization; orphan derivatives and multi-region families are refused. Found on the first licensed dry-run: the compact policy needs aclone_indexcolumn the spine does not carry. Below f100 the surface restricts to cells whose area has assigned rows (receipted asrung_surface), the battery runs and persists every entry but enforces only the ladder gate, and the manifest saysreleasable: false; at f100 every release-blocking entry blocks as before.UKMeasureResolverover the cloned frame resolves the national measures and feedscompute_household_metricsfor both grains;--engine-blocks Kresolves per clone block and rejoins by entity id (refused unless K equals--n-clones). Dry-run compiles both registries, assembles and reconciles the surface, checks binding adjudications, sizes the matrix analytically, and sweeps--candidate-clone-counts— without the engine, the solve, or any write.--input-sha256,--ladder-sha256,--ledger-facts/-sha256/--ledger-manifest-sha256(the chronicle consumer artifact, same loader as the national seam), optional--measure-exclusionsand--register-json;--release-candidaterequires every pin and refuses overrides,--skip-holdout, sampled rungs, and per-block engine runs.area_support_gate(reviewed_exclusions=)records an excluded area's measured shortfalls instead of failing, fails a stale exclusion (area now meets the floor) and an unknown key; the binding resolvesexclusions_resource= new schema-2 registeruk/local_area_support_exclusions.jsonon the shared exclusion clock (expired/premature entries fail). The register ships empty; entries are adjudications for the second PR. Gate-battery digests, the spec sha, the parameter-key pin, anddocs/gate-battery-contract.mdre-cut accordingly.load_bound_spine_sidecar/spine_provenance_from_sidecar, ladder sha + layer vintages, ledger shas, code pin, runtime),survey_year(2024) andcalibration_year(2025; artifactmicrocosm_uk_2025_local.h5), sampling and rung receipts, geography, weights (kind chain, mass conservation, realized max ratio vs design), solve (rule + override receipt, per-kind censuses, measure resolution, cross-grain, binding adjudications, area-support exclusions), fit by family for local and national rows,support_limited_misses(failing cells vs per-area ESS), holdout, vintages,releasable.uprating_holdslike the national path —local_target_references.jsongainsuprating_from_period/uprating_to_periodon the 17,077 cells whose facts predate the 2025 period (UC 2025-05, council-tax stock 2025 carry none; age 2024, SPI tax-year 2023, tenure 2021/22 do), membership gains the hold census; every other byte of the surface is unchanged.uc_hh_*_childrenmetrics classify each UC benefit unit by its ownnum_childrenand sum benefit units into households, matching the national benefit-unit re-bind (Re-bind the DWP UC composition and TCL disability-children targets to the publisher's grain (#808, #809) #813); names and order unchanged, census descriptions regenerated.tools/score_uk_local_candidate.pypins the merged UK local target activation: uk-data parity register, masking doctrine, council tax, PIPR measurements (#147 PR 2) #831 surface (19,618) and accepts a candidate superset with arows_outside_registerreceipt; newtools/extract_uk_local_incumbent_surface.pyturns the incumbent's wide weight arrays (one2025dataset shaped(650|360, 52846), no ids inside) plus the roster CSVs and the incumbent household table into the sha-pinned CSVs the scorer consumes, pairing by position and refusing length mismatches.Verification (run independently of the implementation pass, on
d2b31496)ruff check .clean;ruff format --checkon the 28 touched files clean (repo-wide format debt untouched).tools/census_uk_local_targets.py --checkandtools/census_uk_data_target_parity.py --checkcurrent;tools/ci_test_groups.py --verifyok (new test files land in the UK lane);git diff | grep -c "policyengine-uk-data#"→ 0.microcosm-datacontract suite: 345 passed, 1 skipped.tools/ci_test_groups.py --list:rest6,098 passed, 63 skipped;spine-uk1,712 passed, 16 skipped, 2 failed;ukengine 1,450 passed, 16 skipped, 2 failed — the 2 failures aretest_cached_reference_regeneration_matches_committed_surfaceandtest_cached_candidate_regeneration_matches_committed_evidence, reproduced identically on clean main (installed policyengine-uk 2.92.1 vs committed 2.89.0 licensed-cache evidence; Make the UK eFRS parity re-pin one reviewed command; hold every identity mirror in lockstep (companion to #747) #749's lane).0733a81e…→91eb9472…; gate policybcbcfd55…→d76f012f…; gates manifest76c861d4…→df8c9292…; composition fingerprintdfbad2aa…→0de7f351….local_target_references.json/ membership: 19,618 rows identical apart from the added uprating-hold fields on 17,077 rows; membership gainsuprating_holds(17,077) andholds_by_target; every other membership field byte-equal.f100,releasable: true) and an f001 run (rung receipt, restricted surface,releasable: false); the licensed spine, ledger artifact, and incumbent weights are not in the PR-CI environment by design.Review notes
origin/main@90293e0a; deviations were verified and are recorded in the plan doc's execution notes (dense-wrapper order compatibility flag;householdsstays the ladder-only metric; compiledspec.familyis the source family, so surface rows classify byfamily_for_metricand keepsource_family; ledger/pin arguments stay optional on the synthetic path and are mandatory under--release-candidate).uk/local_binding_adjudications.jsoncarries in-force entries for the five fenced families (second PR).test_cached_reference_regeneration_matches_committed_surface,test_cached_candidate_regeneration_matches_committed_evidence) are the known policyengine-uk 2.92.1 vs committed 2.89.0 evidence mismatch and fail identically on clean main.Refs #762, #495, #665. Not for merge without María's explicit go.
🤖 Generated with Claude Code