Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Retire `test_uk_parity_reference.py::test_cached_reference_regeneration_matches_committed_surface`. It regenerated the pinned enhanced-FRS parity reference from the cached licensed artifact under whichever policyengine-uk the lock installs and asserted byte-equality with the committed file, so every engine bump broke it until the reference was re-pinned (the committed reference carries 2.89.0, the lock 2.97.0), and it was skipped in CI, where the licensed artifact is absent. The thirteen other integrity tests on the reference remain, including the check that the recorded sha256 matches the cached artifact's bytes.
1 change: 1 addition & 0 deletions changelog.d/791-frs-relationships-stage.added.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Add the `frs_relationships` UK spine stage (microcosm#791): the FRS household grid (`relhrp`, `R01`-`R14`, `hrpid`, `hrpnum`) becomes four exported frame columns — `person.relationship_to_head`, `person.ons_family_role`, `person.ons_family_index` and `household.ons_household_type`, the last taking Chronicle's `ons.household_type` value ids verbatim — derived per the ONS Families and households definitions (family = couple or lone parent with children; dependent = under 16 or 16–18 in full-time education; one family plus other individuals stays one-family; no family present = two or more unrelated adults). The derivation is declared as a schema-validated `derive_ons_household_composition` manifest operation whose code lists (FRS SN 9563 relationship codes, ONS rules, Chronicle category ids) the runtime lockstep-asserts, and it is gated at the assembled boundary by a stage-health check (one head per household agreeing with `hrpid` and `hrpnum`, contiguous person numbers, no double partners, no unmapped codes, the ten-cell partition covering every household exactly once, a reviewed grid-reciprocity tolerance) and an enum-domain gate over the declared ten values.
1 change: 1 addition & 0 deletions changelog.d/791-graph-placeholder-index-alignment.fixed.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Fix the graph executor's placeholder for a new dense (`int64`/`bool`) column on a filtered population: the zero-filled placeholder was built on a fresh RangeIndex and inserted label-aligned, so on a population whose table index is non-contiguous (a sampled spine rung after the root filter) the gaps were NaN-filled and the column silently widened to `float64`, which the patch then refused. The placeholder now binds to the table's own index. Found by the microcosm#791 sampled build (the first stage to add a new `int64` person column directly after the sampling filter); regression test on a population with a non-contiguous person index.
1 change: 1 addition & 0 deletions changelog.d/791-household-composition-rebind.changed.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Rebind all ten `ons.household_composition.*` calibration rows on `household.ons_household_type` and retire the three microcosm#791 measure exclusions (`multi_family_households`, `unrelated_adult_households`, `lone_parent_non_dependent_children_households`; register 51 → 48 entries). The benefit-unit and person-count proxies inherited from uk-data `compute/households.py` — which read a couple with a non-dependent child or a lodger as multi-family and could not tell unrelated adults from a lone parent with adult children — are replaced by one ONS-family derivation on the frame, so the partition sums to `ons.households_total` and the `national_household_composition_partition_vs_census_households` cross-grain bridge binds (declared factor 1 by closure on the A15-uprated census cells). Contract notes on the ten rows record the translation (ONS dependency vs the engine's `is_child`; "unrelated adults" as no family present); the `uk-data` parity register and the local binding adjudication wording record the bound bridge.
5 changes: 4 additions & 1 deletion docs/uk-local-cross-grain-reconciliation-802.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,10 @@ cannot express on their own:

- The 10-cell `ons.household_composition.*` partition sums to the national
household-count control and bridges to the Chronicle-compiled
`ons.census.households` contract target.
`ons.census.households` contract target. Since microcosm#791 the ten cells
bind on the `frs_relationships` stage's `household.ons_household_type`
column, so the bridge is fully bound (it was reviewed-unbound while three
cells carried measure exclusions).
- `dwp.uc.households` bridges to `dwp.uc.households_by_area`. The four
`dwp.uc.payment_distribution_*` rows also match the by-area target exactly
and form a separate exhaustive national partition.
Expand Down
111 changes: 111 additions & 0 deletions experiments/791-household-composition-receipts.md

Large diffs are not rendered by default.

Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,7 @@
"derive_snap_take_up",
"derive_puf_policyengine_variables",
"derive_mortgage_balance_hints",
"derive_ons_household_composition",
"derive_pregnancy",
"derive_relationship_inputs",
"derive_retirement_distributions",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -1921,6 +1921,225 @@
"kind"
]
},
{
"type": "object",
"additionalProperties": false,
"properties": {
"kind": {
"const": "derive_ons_household_composition"
},
"source_columns": {
"type": "object",
"additionalProperties": false,
"properties": {
"head": {
"type": "string"
},
"head_flag": {
"type": "string"
},
"head_index": {
"type": "string"
},
"grid": {
"type": "array",
"minItems": 14,
"maxItems": 14,
"items": {
"type": "string"
}
},
"full_time_education": {
"type": "array",
"minItems": 1,
"items": {
"type": "string"
}
}
},
"required": [
"head",
"head_flag",
"head_index",
"grid",
"full_time_education"
]
},
"frs_household_grid_relationship_codes": {
"type": "array",
"minItems": 1,
"items": {
"type": "object",
"additionalProperties": false,
"properties": {
"code": {
"type": "integer",
"minimum": 1
},
"label": {
"type": "string",
"enum": [
"SPOUSE",
"COHABITEE",
"CHILD",
"STEP_CHILD",
"FOSTER_CHILD",
"CHILD_IN_LAW",
"PARENT",
"STEP_PARENT",
"FOSTER_PARENT",
"PARENT_IN_LAW",
"SIBLING",
"STEP_SIBLING",
"FOSTER_SIBLING",
"SIBLING_IN_LAW",
"GRANDCHILD",
"GRANDPARENT",
"OTHER_RELATIVE",
"OTHER_NON_RELATIVE",
"CIVIL_PARTNER"
]
}
},
"required": [
"code",
"label"
]
}
},
"head_label": {
"const": "HEAD"
},
"family_link_codes": {
"type": "object",
"additionalProperties": false,
"properties": {
"partner": {
"type": "array",
"minItems": 1,
"items": {
"type": "integer"
}
},
"child_of": {
"type": "array",
"minItems": 1,
"items": {
"type": "integer"
}
},
"parent_of": {
"type": "array",
"minItems": 1,
"items": {
"type": "integer"
}
}
},
"required": [
"partner",
"child_of",
"parent_of"
]
},
"dependent_child": {
"type": "object",
"additionalProperties": false,
"properties": {
"unconditional_below_age": {
"type": "integer",
"minimum": 0
},
"full_time_education_through_age": {
"type": "integer",
"minimum": 0
}
},
"required": [
"unconditional_below_age",
"full_time_education_through_age"
]
},
"one_person_age_split": {
"type": "integer",
"minimum": 0
},
"family_role_values": {
"type": "array",
"minItems": 1,
"items": {
"type": "string",
"enum": [
"COUPLE_PARTNER",
"LONE_PARENT",
"DEPENDENT_CHILD",
"NON_DEPENDENT_CHILD",
"INDIVIDUAL"
]
}
},
"household_type_values": {
"type": "array",
"minItems": 10,
"maxItems": 10,
"items": {
"type": "string",
"enum": [
"lone_households_under_65",
"lone_households_over_65",
"unrelated_adult_households",
"couple_no_children_households",
"couple_under_3_children_households",
"couple_3_plus_children_households",
"couple_non_dependent_children_only_households",
"lone_parent_dependent_children_households",
"lone_parent_non_dependent_children_households",
"multi_family_households"
]
}
},
"reciprocity_mismatch_tolerance": {
"type": "integer",
"minimum": 0
},
"documentation": {
"type": "object",
"additionalProperties": false,
"properties": {
"frs_codes": {
"type": "string"
},
"ons_definitions": {
"type": "string"
},
"category_ids": {
"type": "string"
}
},
"required": [
"frs_codes",
"ons_definitions",
"category_ids"
]
},
"reason": {
"type": "string"
}
},
"required": [
"kind",
"source_columns",
"frs_household_grid_relationship_codes",
"head_label",
"family_link_codes",
"dependent_child",
"one_person_age_split",
"family_role_values",
"household_type_values",
"reciprocity_mismatch_tolerance",
"documentation"
]
},
{
"type": "object",
"additionalProperties": false,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -334,33 +334,6 @@
"approved_on": "2026-08-26",
"expires_on": "2026-11-26"
},
{
"name": "ons.household_composition.multi_family_households",
"reason": "The ONS household-composition split requires a relationship-to-head frame column the spine does not carry; without it multi-family, unrelated-adult, and lone-parent-with-non-dependent-children households are indistinguishable and the categories bleed into each other (multi_family +1618% on the rebind packet).",
"tracking": "microcosm#791",
"approved_by": "juaristi22",
"adjudication": "microcosm#757 (the uk_target_fit dispositions, issue comment 5427936411)",
"approved_on": "2026-08-26",
"expires_on": "2026-11-26"
},
{
"name": "ons.household_composition.unrelated_adult_households",
"reason": "The ONS household-composition split requires a relationship-to-head frame column the spine does not carry; without it multi-family, unrelated-adult, and lone-parent-with-non-dependent-children households are indistinguishable and the categories bleed into each other (multi_family +1618% on the rebind packet).",
"tracking": "microcosm#791",
"approved_by": "juaristi22",
"adjudication": "microcosm#757 (the uk_target_fit dispositions, issue comment 5427936411)",
"approved_on": "2026-08-26",
"expires_on": "2026-11-26"
},
{
"name": "ons.household_composition.lone_parent_non_dependent_children_households",
"reason": "The ONS household-composition split requires a relationship-to-head frame column the spine does not carry; this cell fits numerically (-0.03% on the rebind packet) but measures the wrong concept - the solver holds it by stuffing misclassified households into the sibling categories, so it must unbind together with them.",
"tracking": "microcosm#791",
"approved_by": "juaristi22",
"adjudication": "microcosm#757 (the uk_target_fit dispositions, issue comment 5427936411)",
"approved_on": "2026-08-26",
"expires_on": "2026-11-26"
},
{
"name": "hmrc/dividend_income_income_band_500_000_to_1_000_000",
"reason": "Sparse cell support: 6 in-band carrier records under the SPI assessable-income concept and 10 under the component-sum proxy (6,914 weighted carriers), measured identically on spine-i and spine-j (microcosm#807) - too few for the solver to hit the published value without distorting neighbouring cells. One of the three thin 500k-1m cells retained when the microcosm#807 revision lifted the three mid-band cells whose measured support was in the hundreds; re-measure on the next spine build before renewal.",
Expand Down
30 changes: 30 additions & 0 deletions packages/microcosm-build/src/microcosm/build/uk/gates.json
Original file line number Diff line number Diff line change
Expand Up @@ -404,6 +404,20 @@
},
"notes": "Stage-time achieved-versus-target gate over the age-tail disaggregation receipt. At its post-frs_spine position, achieved_weighted measures raw FRS-grossed 80+ mass before clone or zero-weight support channels exist."
},
{
"id": "uk_stage_frs_relationships_composition",
"gate": "stage_health",
"phase": "assembled",
"criticality": "release_blocking",
"evidence_absent_blocks": true,
"parameters": {
"stage": "frs_relationships",
"check": "household_composition",
"max_grid_reciprocity_mismatches": 3,
"require_partition_closure": true
},
"notes": "Stage-time invariants over the #791 relationship receipt: exactly one HEAD per household agreeing with hrpid and hrpnum, contiguous person numbers, no person with two partners, no unmapped relhrp code, family index and enum domains consistent, and the ten-cell household-type partition covering every household exactly once. The grid reciprocity tolerance is the reviewed 2024-25 tape count; a re-issued tape with a different count refuses for review."
},
{
"id": "uk_release_input_coverage",
"gate": "release_input_coverage",
Expand Down Expand Up @@ -597,6 +611,7 @@
"household.msoa_code",
"household.num_vehicles",
"household.oa_code",
"household.ons_household_type",
"household.private_pension_wealth",
"household.property_purchased",
"household.rail_usage",
Expand All @@ -620,11 +635,14 @@
"person.is_parent",
"person.is_uc_claimant",
"person.legacy_jobseeker_proxy",
"person.ons_family_index",
"person.ons_family_role",
"person.outpatient_visits",
"person.pension_contributions_via_salary_sacrifice",
"person.pip_dl_category",
"person.pip_m_category",
"person.receives_benefits_in_own_right",
"person.relationship_to_head",
"person.salary_sacrifice_asked",
"person.salary_sacrifice_reported",
"person.sic_industry_division",
Expand Down Expand Up @@ -662,6 +680,18 @@
},
"notes": "The household BRMA assignment must remain inside the PolicyEngine-UK brma enum domain. Evaluated at the assembled boundary, right after frs_brma writes the column: enum membership is a property of the values themselves, so checking it there is identical to checking it at the end and refuses fourteen stages earlier. Take-up signal stays terminal by contrast, because its weighted shares are not the same statistic before and after calibration."
},
{
"id": "uk_ons_household_type_enum_domain",
"gate": "enum_domain",
"phase": "assembled",
"criticality": "release_blocking",
"parameters": {
"columns": [
"ons_household_type"
]
},
"notes": "The household ONS household-type value must be one of the ten Chronicle ons.household_type value ids the frs_relationships stage declares (the build supplies the declared domain as evidence; the column is a frame input, not a PolicyEngine-UK enum). Evaluated at the assembled boundary like the BRMA enum: membership is a property of the values themselves."
},
{
"id": "uk_uc_deduction_combination_enum_domain",
"gate": "enum_domain",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
"description": "Reviewed adjudications of the UK local binding fences (uk_local_target_census.json binding_fences). Binding a target family whose fence has no in-force entry here fails in the rowwise doctrine solve before any solve runs. Keys are fence ids; the reason states the ruling (accept the stated basis, or replace it with the named alternative) and cites the evidence.",
"exclusions": {
"census_disclosure_control_noise": {
"reason": "Accept the stated basis: constituency and local-authority household counts bind to the published disclosure-controlled census values compiled from the pinned Chronicle feed (ons.census.households: ONS TS041 PCON24/LAD, NRS UV404 UKPC24 and council all-occupied totals, NISRA PCON24/LGD14). Per-area dispersion of the retired OA-ladder sums against the published cells at review: England mean 7.4 / max 29, Wales 6.4 / 15, Scotland net −557 (NRS applies disclosure control per output level; same shortfall at both grains), Northern Ireland mean 7 / max 16 after the NI constituency leg moved to NISRA's published DZ2021→PARLCON24 lookup (the earlier postcode inference was a mapping defect of 10 Data Zones, max 694, not perturbation). When a national same-concept control is bound, the standing cross-grain rule declared in uk_runtime.ledger_targets rescales both grains; country wins. The census cells count the occupied-household universe and pair only through the household-composition bridge, never with dwelling stock; that cross-grain rescale applies once the bridge is bound. Today it is reviewed-unbound, so the cells bind as published after A15 uprating. Fence evidence: uk_local_target_census.json#/binding_fences/census_disclosure_control_noise; the published cells preserve the source cell-key controls. Re-worded 2026-09-10 (microcosm#887 review round 1).",
"reason": "Accept the stated basis: constituency and local-authority household counts bind to the published disclosure-controlled census values compiled from the pinned Chronicle feed (ons.census.households: ONS TS041 PCON24/LAD, NRS UV404 UKPC24 and council all-occupied totals, NISRA PCON24/LGD14). Per-area dispersion of the retired OA-ladder sums against the published cells at review: England mean 7.4 / max 29, Wales 6.4 / 15, Scotland net −557 (NRS applies disclosure control per output level; same shortfall at both grains), Northern Ireland mean 7 / max 16 after the NI constituency leg moved to NISRA's published DZ2021→PARLCON24 lookup (the earlier postcode inference was a mapping defect of 10 Data Zones, max 694, not perturbation). When a national same-concept control is bound, the standing cross-grain rule declared in uk_runtime.ledger_targets rescales both grains; country wins. The census cells count the occupied-household universe and pair only through the household-composition bridge, never with dwelling stock; that cross-grain rescale applies once the bridge is bound. Bound from microcosm#791 (2026-09-11): the three ONS household-composition cells that held the bridge reviewed-unbound now bind on the frs_relationships household type, so the cross-grain rescale runs on the A15-uprated cells; the ten-cell partition and the A15 control are both ons.households_total, so the declared factor is 1 by closure and the receipt records the bridge as bound. Fence evidence: uk_local_target_census.json#/binding_fences/census_disclosure_control_noise; the published cells preserve the source cell-key controls. Re-worded 2026-09-10 (microcosm#887 review round 1); re-worded 2026-09-11 (microcosm#791: the three composition cells re-bound, bridge bound).",
"approved_by": "juaristi22",
"adjudication": "microcosm#802",
"approved_on": "2026-08-31",
Expand Down
Loading
Loading