Skip to content

UK rowwise candidate: solve local and national targets as one matrix on the cloned spine (#762 PR A) - #852

Draft
juaristi22 wants to merge 8 commits into
mainfrom
uk-rowwise-candidate-762
Draft

UK rowwise candidate: solve local and national targets as one matrix on the cloned spine (#762 PR A)#852
juaristi22 wants to merge 8 commits into
mainfrom
uk-rowwise-candidate-762

Conversation

@juaristi22

@juaristi22 juaristi22 commented Sep 2, 2026

Copy link
Copy Markdown
Collaborator

First of two PRs for #762 (plan: repos/uk-762-rowwise-candidate-plan.md, approved 2026-09-02) — the joint-solve machinery. The second PR carries the signed registers, the measured doctrine adjudication, the receipts, and the local-area release contract/assembly after the licensed runs.

What this does

One matrix, one weight vector, one solve. tools/build_uk_rowwise_candidate.py now clones the spine through the OA ladder at K, runs the policyengine-uk engine on the cloned frame, and solves the national families (compiled + measure-excluded, 364 activated references) and the local families (compiled references over both grains minus signed deferrals, plus the ladder's census-household rows) as rows of a single sparse constraint matrix in one calibrate() call under the local doctrine. No national-only solve exists on this path; the national line's diagnostics are only a comparison yardstick.

  • Surface (local_rowwise.py): a long-format multi-grain builder build_uk_rowwise_local_surface_matrix (constituency + local authority, present cells only, signed deferrals are absent rows, canonical grain/metric/area order, fail-closed on unreachable nonzero targets and per-grain roster coverage). The dense single-grain builder delegates to it and keeps its legacy area-major result bit-for-bit. Local rows are named by the compiled register's row_name, so the scorer joins by identity; area masks are materialized lazily from an int32 index.
  • National rows (UKRowwiseNationalRows, solve_uk_rowwise_weights_under_doctrine(..., national_rows=, target_weight_rule=, restore=)): local rows first, national rows after, one TargetSet; row-name alignment asserted over every compiled row; diagnostics and past-cap censuses reported per row kind; prepared slash-named measure columns are restored away before the frame rebuild and refused if any survive. rotated_uk_local_holdout keeps the national rows in every training fold.
  • Surface assembly + cross-grain (ledger_targets.uk_local_target_surface): compiled local registry → present cells (family via family_for_metric, compiled source family retained), ladder household rows for both grains, then apply_uk_cross_grain_reconciliation with the national control rows present (the standing country > constituency > la rule and bridges from Declare cross-grain target reconciliation for the UK local surface: national wins, sub-national rescales (#495 increment 6 companion) #802/UK local target activation: uk-data parity register, masking doctrine, council tax, PIPR measurements (#147 PR 2) #831 fire in the joint surface). A declared bridge whose missing national members are all in-force reviewed measure exclusions is classified unbound and listed under cross_grain.unbound_bridges in the plan and manifest instead of refusing the surface; a partial binding with any unreviewed missing member still fails closed by name. On the live surface this is the household-composition partition bridge (three cells are UK spine: relationship-to-head frame column, so the ONS household-composition split is measurable #791's exclusions to 2026-11-26), so the ladder's census household counts bind as published until that is adjudicated (plan item A11). A national target that fans out into several cells at one geography (UC payment-distribution bands, SPI income bands: 15 targets, 213 cells on the live compile) is a distribution, not a control: it hands the engine no control row, is dropped from the bound ids the engine sees, and is receipted as fanout_targets_not_controls (the compile activates only a subset of the DWP award bands, so their sum can never reproduce the caseload); every cell still solves as its own row. Eight declared age-band bridges (local ons.age.{b} at both grains ↔ the single-cell national ons.population.age_*_by_region UK totals; 80–89 has no local counterpart) let the local age bands rescale to the national totals despite the half-open vs inclusive edge encodings that defeat the exact-signature match.
  • Doctrine (local_doctrine.py): weighting vocabulary ("uniform", "grain_equal")grain_equal gives the national rows, the constituency rows, and the LA rows one equal share each, uniform within, derived from row labels through the national family_equal arithmetic; uk_local_doctrine_with_overrides receipts the single overridable field. UK_LOCAL_TARGET_LOSS_CAP 10.0 and UK_LOCAL_MAX_WEIGHT_RATIO 100.0 are unchanged.
  • Scale ladder (--sample-fraction {0.01, 0.10, 1.0}, --sample-seed): f100 never samples (sampled: false); below it, sample_uk_spine_frame (new beside the certified-compact policy in national_sampling.py) samples whole source families keyed on the spine's source_household_id, stratified by the raw row's region, through the shared frame sampler with one full-mass renormalization; orphan derivatives and multi-region families are refused. Found on the first licensed dry-run: the compact policy needs a clone_index column the spine does not carry. Below f100 the surface restricts to cells whose area has assigned rows (receipted as rung_surface), the battery runs and persists every entry but enforces only the ladder gate, and the manifest says releasable: false; at f100 every release-blocking entry blocks as before.
  • Engine on the clone: one scratch-mode UKMeasureResolver over the cloned frame resolves the national measures and feeds compute_household_metrics for both grains; --engine-blocks K resolves per clone block and rejoins by entity id (refused unless K equals --n-clones). Dry-run compiles both registries, assembles and reconciles the surface, checks binding adjudications, sizes the matrix analytically, and sweeps --candidate-clone-counts — without the engine, the solve, or any write.
  • Pins and inputs: --input-sha256, --ladder-sha256, --ledger-facts/-sha256/--ledger-manifest-sha256 (the chronicle consumer artifact, same loader as the national seam), optional --measure-exclusions and --register-json; --release-candidate requires every pin and refuses overrides, --skip-holdout, sampled rungs, and per-block engine runs.
  • Area-support reviewed exclusions: area_support_gate(reviewed_exclusions=) records an excluded area's measured shortfalls instead of failing, fails a stale exclusion (area now meets the floor) and an unknown key; the binding resolves exclusions_resource = new schema-2 register uk/local_area_support_exclusions.json on the shared exclusion clock (expired/premature entries fail). The register ships empty; entries are adjudications for the second PR. Gate-battery digests, the spec sha, the parameter-key pin, and docs/gate-battery-contract.md re-cut accordingly.
  • Manifest v2: identity chain (spine sha + sidecar provenance via the promoted load_bound_spine_sidecar / spine_provenance_from_sidecar, ladder sha + layer vintages, ledger shas, code pin, runtime), survey_year (2024) and calibration_year (2025; artifact microcosm_uk_2025_local.h5), sampling and rung receipts, geography, weights (kind chain, mass conservation, realized max ratio vs design), solve (rule + override receipt, per-kind censuses, measure resolution, cross-grain, binding adjudications, area-support exclusions), fit by family for local and national rows, support_limited_misses (failing cells vs per-area ESS), holdout, vintages, releasable.
  • Vintage receipts: the area authoring path now records uprating_holds like the national path — local_target_references.json gains uprating_from_period/uprating_to_period on the 17,077 cells whose facts predate the 2025 period (UC 2025-05, council-tax stock 2025 carry none; age 2024, SPI tax-year 2023, tenure 2021/22 do), membership gains the hold census; every other byte of the surface is unchanged.
  • UC child bands at benefit-unit grain (A1-4 ruling): the four constituency uc_hh_*_children metrics classify each UC benefit unit by its own num_children and sum benefit units into households, matching the national benefit-unit re-bind (Re-bind the DWP UC composition and TCL disability-children targets to the publisher's grain (#808, #809) #813); names and order unchanged, census descriptions regenerated.
  • Scoring inputs: tools/score_uk_local_candidate.py pins the merged UK local target activation: uk-data parity register, masking doctrine, council tax, PIPR measurements (#147 PR 2) #831 surface (19,618) and accepts a candidate superset with a rows_outside_register receipt; new tools/extract_uk_local_incumbent_surface.py turns the incumbent's wide weight arrays (one 2025 dataset shaped (650|360, 52846), no ids inside) plus the roster CSVs and the incumbent household table into the sha-pinned CSVs the scorer consumes, pairing by position and refusing length mismatches.

Verification (run independently of the implementation pass, on d2b31496)

  • ruff check . clean; ruff format --check on the 28 touched files clean (repo-wide format debt untouched).
  • tools/census_uk_local_targets.py --check and tools/census_uk_data_target_parity.py --check current; tools/ci_test_groups.py --verify ok (new test files land in the UK lane); git diff | grep -c "policyengine-uk-data#" → 0.
  • Targeted battery (rowwise/doctrine/candidate/scorer/gate-battery/bindings/gates/ledger/ladder/national-calibration/diagnostics/authoring/council-tax/spec-bundle/country-spec/holdout/contract-pins/release-certification/extractor): 562 passed on the first commit; after the rebase over UK spine: benefit-unit-grain SPI UC reporter draw with pre-take-up screen (#832); exact capital sentinel (#833) #835 and the three licensed-touch fixes (spine sampler, Ledger geography spelling, reviewed-exclusion bridges): targeted 443 passed, spec/country/contract-pins/contract 511 passed, cross-grain/ledger/candidate/rowwise 90 passed.
  • microcosm-data contract suite: 345 passed, 1 skipped.
  • CI groups run in full from tools/ci_test_groups.py --list: rest 6,098 passed, 63 skipped; spine-uk 1,712 passed, 16 skipped, 2 failed; uk engine 1,450 passed, 16 skipped, 2 failed — the 2 failures are test_cached_reference_regeneration_matches_committed_surface and test_cached_candidate_regeneration_matches_committed_evidence, reproduced identically on clean main (installed policyengine-uk 2.92.1 vs committed 2.89.0 licensed-cache evidence; Make the UK eFRS parity re-pin one reviewed command; hold every identity mirror in lockstep (companion to #747) #749's lane).
  • Re-cut pins (from real payloads on this base): UK spec sha 0733a81e…91eb9472…; gate policy bcbcfd55…d76f012f…; gates manifest 76c861d4…df8c9292…; composition fingerprint dfbad2aa…0de7f351….
  • Regenerated local_target_references.json / membership: 19,618 rows identical apart from the added uprating-hold fields on 17,077 rows; membership gains uprating_holds (17,077) and holds_by_target; every other membership field byte-equal.
  • End-to-end on the synthetic fixture: a K=2 f100 joint build (stub resolver, national + local + ladder rows in one matrix, manifest v2, Logbook rung f100, releasable: true) and an f001 run (rung receipt, restricted surface, releasable: false); the licensed spine, ledger artifact, and incumbent weights are not in the PR-CI environment by design.

Review notes

  • Implemented by Codex under the reviewed plan's protocol on origin/main @ 90293e0a; deviations were verified and are recorded in the plan doc's execution notes (dense-wrapper order compatibility flag; households stays the ladder-only metric; compiled spec.family is the source family, so surface rows classify by family_for_metric and keep source_family; ledger/pin arguments stay optional on the synthetic path and are mandatory under --release-candidate).
  • Nothing here binds a new family in production: the joint path refuses to solve until uk/local_binding_adjudications.json carries in-force entries for the five fenced families (second PR).
  • The two UK-lane failures on this base (test_cached_reference_regeneration_matches_committed_surface, test_cached_candidate_regeneration_matches_committed_evidence) are the known policyengine-uk 2.92.1 vs committed 2.89.0 evidence mismatch and fail identically on clean main.

Refs #762, #495, #665. Not for merge without María's explicit go.

🤖 Generated with Claude Code

juaristi22 and others added 4 commits September 2, 2026 19:26
…ed spine (#762 PR A)

One TargetSet, one weight vector, one calibrate() call: the compiled local
surface (both grains, present cells only) and the ladder household rows join
the activated national families as rows of a single sparse constraint matrix
over the K-cloned spine, with the policyengine-uk engine resolved on every
cloned row. Adds the long-format multi-grain surface builder, registry-named
local rows with lazy area masks, national rows in the doctrine solve and the
rotated holdout, the grain_equal weighting rule as a receipted override, the
f001/f010/f100 scale ladder with a below-f100 gate posture, sha and ledger pins,
per-clone engine blocks, a reviewed-exclusions channel for the area-support
gate (register ships empty), manifest v2 with survey and calibration years,
uprating-hold receipts on the local surface, support-limited-miss diagnostics,
a scorer that accepts a candidate superset, and the incumbent local-surface
extractor.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
#762 PR A)

The first licensed dry-run refused at the sampling step: the driver sampled
even at the full rung, through the certified-compact policy that keys families
on clone_index arithmetic the microcosm spine does not carry. The spine's
lineage is explicit (source_household_id, the support-clone index, and the
SPI, capital-gains, and band-donor flags), so the ladder now keeps f100
untouched and samples whole source families stratified by the raw row's
region below it, through the shared frame sampler with one full-mass
renormalization; orphan derivatives and multi-region families are refused.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…al control rows (#762 PR A)

The second licensed dry-run refused in the surface assembler: national
compiled specs carry their geography under ledger_geography_level and
ledger_geography_id (country grain at K02000001 or K03000001), while only the
area-authored local specs also carry the local spelling the assembler read.
One private resolver now prefers the local spelling, falls back to the Ledger
spelling, refuses blanks and disagreements by target name, and checks a
national level against the contract's declared levels; the vintage census,
diagnostics registry, and the end-to-end stub use the same shape.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… A rebase)

The rebase resolution replaced the first digest in the file, which is the
Armenia constant, instead of the UK row; both rows now carry the values the
merged tree produces.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@juaristi22
juaristi22 force-pushed the uk-rowwise-candidate-762 branch from a03c2b7 to 9d70c2c Compare September 2, 2026 17:32
juaristi22 and others added 4 commits September 2, 2026 19:44
…ions, with a receipt (#762 PR A)

The third licensed dry-run reached the cross-grain pass and was refused: the
household-composition partition bridge is partially bound because three of
its national cells are microcosm#791's reviewed measure exclusions. A partial
bridge whose every missing member carries an in-force reviewed exclusion is
now classified unbound and listed under unbound_bridges in the receipt, so
the lower side binds as published and the gap stays enumerable; any partial
binding with an unreviewed missing member still fails closed by name. The
driver passes the measure-exclusion receipt through and surfaces the entry in
the dry-run plan and the manifest.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…r geography (#762 PR A)

The fourth licensed dry-run reached the engine's control rule and was refused:
fifteen activated national targets fan out into several cells at one
geography (UC payment-distribution bands, SPI income bands), and the UC
family-type partition is the exact-signature control for the by-area UC
count. The UK surface assembler now groups national specs by contract target
and geography and hands the engine one control row per group, the summed
total for fan-out groups, receipted as fanout_controls_summed; every cell
still solves as its own row, and non-finite cells or mixed levels refuse.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ands at benefit-unit grain (#762 PR A)

The award-band table is the full UC caseload with overlapping cumulative bands,
and the compile activates only a subset of those bands, so a fan-out national
target can never act as a cross-grain control: the surface assembler now emits
no control row for it, drops it from the bound ids handed to the engine, and
receipts it as fanout_targets_not_controls (replacing the summed control).
Per Maria's A1-4 ruling the four UC child-band local metrics classify each UC
benefit unit by its own num_children and sum benefit units into households,
matching the national benefit-unit re-bind; names and order are unchanged and
the census descriptions are regenerated. The synthetic candidate tests pin an
empty support-exclusion register so real micro-LA entries never collide with a
synthetic roster.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The local age bands (ons.age.0_10 .. 70_80, half-open) and the national
single-cell UK totals (ons.population.age_0_9_by_region .. 70_79, inclusive)
describe the same integer-age populations under different edge encodings, so
the exact-signature match never fired and the local bands never rescaled to
the national totals. Per Maria's ruling each local band now has a declared
bridge to its UK control; the national 80-89 band has no local counterpart
and stays unbridged.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant