Fix the two production-path breakages the node graph left in the UK spine driver (#836)

[juaristi22]

Problem

Every full licensed UK spine build on main has failed since #836 landed the node graph as the UK driver's only path. Reproduced today at 90293e0a with the production invocation (FRS 2024-25 tabs, WAS R8, LCFS, ETB, SPI donor, HMRC ODS):

UK FRS spine build failed: Node 'spi_support_channel' patch rejected:
Node 'spi_support_channel' mass='conserve' changed stratum 'default': 68251110.0 -> 65436869.60496674.

Diagnosis

• The graph declares the SPI support-channel node mass='conserve' (uk_runtime/graph.py, _STRUCTURAL_MASS).
• The executor checks conservation with Frame.stratum_mass() (microcosm-frame/.../bundle.py), which is weighted person mass per stratum: household weights broadcast through membership, compared per stratum at _MASS_RTOL = 1e-9 (microcosm/graph/population.py::_assert_mass_mapping).
• The stage conserves household mass exactly by construction: allocate_zero_weight_prior_mass declares conservation: exact_total and moves the declared prior-mass share (0.5) onto the stacked synthetic households. Those households have fewer persons per household than the FRS households whose mass they take over, so person mass necessarily falls. The 4.1% drop the executor measured (68.25m → 65.44m) matches the composition difference visible on the last good spine (spine-l: 29.2m households carry 66.1m persons; survey rows alone 34.6m at half mass).
• conserve is therefore unsatisfiable for this node on any real vintage. CGT cloning passes it because a clone is its source household at half weight (composition fixed); the band-donor stack is already free.
• It escaped CI because the H2 parity fixture (packages/microcosm-graph/tests/fixtures/parity/uk_spine/) is synthetic and small and happens to preserve person mass through the stage; Node graph: acceptance charter and frozen interfaces for microcosm-graph #836's history shows no full licensed run.

Consequence until fixed: no spine re-mint, no #835 I5, no #685 twin builds, no recertification.

Fix

• _STRUCTURAL_MASS["spi_support_channel"] → declared, with the rationale written next to the table.
• UKExpandStageKernel emits receipt["mass"] for a declared node: the person-mass ledger (before/after, per stratum) the executor verifies against its own measurement, plus an assertion of the invariant that is actually the stage's contract — the expand weight entity's total mass is unchanged at the ledger's own tolerance (_declared_mass_receipt). A composition-changing expansion that also moved household mass would still fail, loudly and by name.
• The pinned uk_spine.json is regenerated (one node's mass field).
• Regression tests in test_uk_graph.py: a two-household fixture (one person in household 10, two in household 20) whose expansion clones the one-person household and shifts mass onto it at conserved household mass — rejected under conserve with changed stratum (the exact class), accepted under declared with the kernel ledger; and the three structural nodes' policies pinned.
• Changelog fragment.

No stage transform, manifest, gate, digest or spine byte changes: the declaration and the kernel receipt move, the population does not.

Second breakage found by the same reproduction

With the node declared, the licensed run completed all 26 stages and then died in the driver's record projection:

UK FRS spine build failed: graph stage 'frs_spine' exposes 0 artifacts for declared output 'person_id'.

_graph_stage_records (tools/build_uk_frs_spine.py) demands exactly one artifact per declared output. Entity ids and memberships are executor-carried context rather than owned cells (_STRUCTURAL_COLUMNS in uk_runtime/graph.py), so the CREATE node exposes none for the five id/membership columns frs_spine declares. The projection now reads those from the final population (the legacy plan recorded 1.0 for them) and still refuses any other output without exactly one artifact. Second commit; a requires_uk test runs the projection over the H2 fixture for every stage.

Verification

• packages/microcosm-build/tests/test_uk_graph.py, packages/microcosm-graph/tests (incl. test_h2_uk_spine_parity, requires_uk, run locally with the engine present), test_uk_source_stages.py, test_uk_spine_acceptance_receipt.py, test_us_spine_blindness.py: 755 passed, 0 failed, 0 skipped (JUnit-counted; run from the branch's own venv with policyengine-uk 2.92.1 present, so the requires_uk graph tests ran rather than skipped).
• ruff check / ruff format --check clean on the touched files; tools/ci_test_groups.py --verify ok.
• Licensed reproduction on the same inputs, same branch: the build that failed at the SPI node on 90293e0a now runs end to end on this branch (launched 14:48, H5 written 14:55; all 26 stages, 19 QRF fits). The artifact matches the last legacy-path spine (spine-l, built through StagePlan before Node graph: acceptance charter and frozen interfaces for microcosm-graph #836) on every surface the driver records: entity rows 113,649 / 61,211 / 52,846, household weight total 29,247,433.0, 26 stage records, and all 210 final nonzero shares identical to 1e-6; the frs_spine record carries its 86 shares with the id columns at 1.0. Between the two commits the same run reproduced the second failure exactly once, at the end of the build.

Follow-ups (not in this PR)

• The H2 fixture should carry an SPI donor composition that differs from the FRS households so this class is visible hermetically; filed as a note for the node-graph lane.
• Found while preparing Net-new UK stages: UC deduction attributes, bus fares, WAS debt #685 (E9); that work is on a separate branch and will rebase over this.

🤖 Generated with Claude Code

cc @MaxGhenis

[MaxGhenis]

Reviewed as the node-graph owner, with a sol pass over the PR head (fix-spi-support-channel-mass-836). No must-fix defect. The diagnosis matches the executor: population.py measures weighted person mass per stratum through Frame.stratum_mass(), conserve asserts per-stratum equality, and declared requires receipt["mass"] and validates it against the executor's own measurement, so the receipt cannot misstate the change. declared is the right policy here; free would have dropped the check. The new regression tests fail on main without the graph.py/graph_kernels.py hunks and pass with them.

Verified locally on the PR head: test_uk_graph.py plus the graph shard's H parity file (H2 with the engine), ruff, tools/ci_test_groups.py --verify. The full suite is CI's to certify.

Two follow-ups, neither blocking:

1. _graph_stage_records: reading the structural shares from the terminal frame is numerically equivalent today (every root id and membership has share 1.0 and no later stage declares them), but the legacy records were measured immediately after each stage, so the root population is the faithful source. Worth switching when the driver is next touched.
2. The H2 fixture still cannot show this failure class: its SPI draw selects all 135 households and household size is aligned with region, so person mass is 282.24 before and after. The smallest deterministic change the sol pass found: draw 131 households and assign household 52 to region 11; at seed 42 person mass moves 282.24 → 281.48 while household mass stays exactly 226.8. A lane is making that change on top of this branch (it lands as its own PR after this one).

The charter's D2 row now says this in so many words (weighted person mass per stratum; composition-changing expansions declare with a receipt stating the invariant they do hold), in the amendments branch that follows.

🤖 Generated with Claude Code

[vahid-ahmadi]

Review pass at 9312b1e2 (Claude Code, high effort; branch checkout, executor and kernel code read alongside the diff, targeted suites run — without the licensed engine, so the requires_uk paths skipped locally and I lean on the engine-uk lane for those).

The diagnosis is right and I could confirm each link of it from the code: Frame.stratum_mass is household weight broadcast through person membership, _mass_record in population.py applies _assert_mass_mapping per stratum at 1e-9 for conserve, and the stage's allocate_zero_weight_prior_mass declares conservation: exact_total at household grain. An expansion that conserves household mass while changing persons-per-household cannot pass a person-mass conserve, so declared is the correct policy in the executor's vocabulary, and the two-household regression test reproduces exactly that class (household 3 → 3, person 5 → 4.5, refused with changed stratum, accepted under declared). The fixture diff is a single field. Three findings, none blocking.

1. The invariant that replaces conserve has no failing-path test

_declared_mass_receipt (graph_kernels.py:884-921) is where the stage's real contract now lives: household total unchanged at 1e-9, refusing with "declares its person-mass change but must conserve 'household' mass". Nothing exercises that refusal. test_declared_accepts_the_same_expansion_with_the_kernel_ledger hand-builds receipt["mass"] and calls patch() directly, so the kernel function never runs; the only paths through it are the H2 parity test and this PR's own projection test, both requires_uk, both happy-path. So the switch from conserve to declared removes an executor-enforced check and adds a kernel-enforced one that is tested only on the happy path. One test that feeds _declared_mass_receipt a before/after pair whose household totals differ and asserts the ValueError closes it — the same shape as the changed stratum test, on the other side.

2. The executor's "verification" of the ledger is self-consistency, and the description should say so

The PR body says the kernel emits "the person-mass ledger the executor verifies against its own measurement". Mechanically true — _validate_mass_receipt (population.py:1513-1531) checks before/after/stratum_* against _mass_record's own stratum_mass() — but the kernel computes those same numbers from the same two frames (:910-917), so the check can only fail on a coding error, never on a bad population. That is what declared means in this executor and it is fine; the sentence just reads as if the executor independently constrains the change. It does not. The constraints are the kernel assertion and the existing release-blocking uk_stage_spi_support_channel_mass stage-health gate (gates.json:275-287, prior-mass share 0.5 at absolute tolerance 0), and the gate runs at battery time while the kernel assertion is what stops a bad population at build time — which is why 1 matters.

Related and worth one sentence in the rationale: conserve was per-stratum, the replacement asserts the household total only. The error text names a default stratum; whether or not that is the only one on the licensed frame (national_sampling.py does set unit strata), a household-mass move between strata would pass the new assertion where it failed the old one. Cheap to assert per-stratum household mass while you are there, or state the choice.

3. The id-column bypass in the record projection is wider than the case it fixes

_graph_stage_records (build_uk_frs_spine.py:663-665) now takes the final-frame share for any declared output that is an id/membership column and has no artifact, for any stage. Only frs_spine declares those columns today (checked the roster), so behaviour is correct, and the docstring is honest that the legacy plan recorded 1.0 for them. But the guard is "no artifact and structural", not "this is the root node", so a later stage that wrongly lists household_id among its outputs would be projected as 1.0 from the final population instead of tripping the exactly-one-artifact refusal that exists to catch that. Restricting the bypass to the CREATE node (or asserting stage.stage == "frs_spine" alongside) keeps the refusal load-bearing. The requires_uk projection test is a good addition and would still pass.

Checked and clear

• _STRUCTURAL_MASS change is mirrored in the pinned uk_spine.json (one field, verified with a canonical JSON diff) and pinned by test_spi_support_channel_declares_its_mass_change_and_cgt_clones_conserve. Nothing else references the node's graph policy; the separately-named uk_stage_spi_support_channel_mass entry is the stage-health gate, untouched and still correct.
• _structural_columns uses real schema API (EntitySchema.entity_id_column / membership_column, schema.py:138,169), and the final population is what main() passes.
• frame_mass_log_append validation (population.py:1331-1343) checks the log's endpoints against entity totals but not old == new, so the household-conservation assertion in the kernel is not redundant with it; at build time it is the only place that invariant is enforced.
• test_uk_graph.py routes to the engine-uk lane (ci_test_groups.py:engine_group), so the new requires_uk test does run in CI rather than skipping everywhere.
• CGT clone staying conserve is right: a clone is its source household at half weight, composition fixed.

Runs

ruff check and ruff format --check clean on the four touched Python files. test_uk_graph.py, the full microcosm-graph suite, test_uk_source_stages.py and test_uk_spine_acceptance_receipt.py: 233 passed, 3 skipped, 0 failed (the skips are the requires_uk tests; no engine here). CI: 14 lanes green, engine-uk (the lane that runs the requires_uk kernel path), spine-uk, engine-shared and wheels still running when I looked.

---

The fix unblocks the licensed build and the diagnosis is sound; 1 is cheap to add before merge, or as an immediate follow-up if the unblock is urgent — it is the difference between "the household invariant is asserted" and "the household invariant is asserted and known to fire". 2 is wording plus an optional per-stratum tightening, 3 a one-line narrowing.