Declare survey- and calibration-year rules for UK FRS parameter reads (#862)

changelog.d/862-uk-disability-parameter-year.fixed.md

@@ -0,0 +1 @@
+Declare survey- and calibration-year rules for UK FRS stages, align disability category and flag thresholds on the fiscal-converted DWP tree with one weekly conversion, reuse the disability derivation after SPI redraws, and seed Disabled Students' Allowance eligibility and its maximum at the calibration year while preserving the survey-year frame stamp (#862; uk-data#475, uk-data#476, uk-data#478)

experiments/862-policy-year-rule-receipts.md

@@ -0,0 +1,99 @@
+# #862 policy-year rule receipts
+
+Stage-level receipt for microcosm#862 (uk-data#475, uk-data#476, uk-data#478 inherited
+through parity): the declared FRS stages were run in order through
+`frs_education_grant_split` (frs_spine, frs_employment, frs_council_tax, frs_disability,
+frs_education, frs_legacy_proxies, frs_education_grant_split; the transforms
+`tools/build_uk_frs_spine.py` binds) on the pinned FRS 2024-25 tabs, once on `main` and
+once on this branch, with a scratch script that records the disability category bands,
+the three flag masses and the DSA column. Person weights are the household weights of the
+person's household. Categories are string columns, so their parity share is structurally
+1.0; the band counts below are the real check. No calibration ran: the two two-child-limit
+disability cells (#808 lineage) are not re-measured here, and a full licensed spine +
+calibration run remains available as a follow-up.
+
+## Inputs and code
+
+| item | value |
+|---|---|
+| release | `frs_2024_25` (`uk/frs_release.json`: survey 2024, base 2024, calibration 2025) |
+| tabs | UKDS SN 9563 tab zip sha256 `05dd0069587dbd25e5719d355ce05fc0827d5edd58c24ece9ab85acd954a9aeb`, hash-checked per tab by `frs_spine._read_pinned_tab` |
+| frame | 34,966 persons, 68,251,110 weighted, `time_period` `2024` on both sides |
+| engine | policyengine-uk 2.92.1 (first run) and 2.94.0 (runs repeated after each rebase on `main`), policyengine-core 3.31.0; every figure identical on both |
+| before | `main` `c1b83241` |
+| after | this branch (`uk-policy-year-rule-862`, implementation commit 47d778c8 after the rebase on `main` e07a4735, which carries #874, #879 and #870); the run was repeated at 8ff697cb and reproduced every figure below. Later commits on the branch touch the SPI refresh seam, tests and locks, none of the seven stages measured here |
+| runtime | about 24 s per side, 2026-09-07 |
+
+## Disability categories (records, weighted persons): identical before and after
+
+| column | band | records | weighted |
+|---|---|---|---|
+| aa_category | HIGHER | 635 | 869,690 |
+| aa_category | LOWER | 372 | 523,290 |
+| dla_sc_category | HIGHER | 212 | 367,409 |
+| dla_sc_category | MIDDLE | 292 | 549,247 |
+| dla_sc_category | LOWER | 90 | 136,176 |
+| dla_m_category | HIGHER | 255 | 391,710 |
+| dla_m_category | LOWER | 257 | 511,494 |
+| pip_m_category | ENHANCED | 983 | 1,710,945 |
+| pip_m_category | STANDARD | 500 | 882,913 |
+| pip_dl_category | ENHANCED | 966 | 1,776,373 |
+| pip_dl_category | STANDARD | 793 | 1,373,522 |
+
+Reading: moving the category thresholds from the `baseline` clone (FY 2023-24 rates) to
+the fiscal-converted `gov.dwp` tree at the survey year (FY 2024-25 rates) moves no row.
+Respondents report the statutory rate in force and adjacent bands are about 49% apart, so
+the GBP 6.80 to 7.80 per week threshold shift lands inside every band. This is the same
+result uk-data#480 recorded on the incumbent side with the same tabs.
+
+## Disability flags: identical before and after
+
+| flag | records | weighted |
+|---|---|---|
+| is_disabled_for_benefits | 3,729 | 6,184,013 |
+| is_enhanced_disabled_for_benefits | 1,813 | 3,013,472 |
+| is_severely_disabled_for_benefits | 2,219 | 3,588,886 |
+
+The SPI refresh now calls the same derivation with the same rates, so the SPI-redrawn
+rows cannot diverge from these figures (previously it divided categories by 52 instead of
+365.25/7; a unit test pins the equivalence).
+
+## DSA seed at the calibration year
+
+| measure | before (`main`) | after (this branch) |
+|---|---|---|
+| `disabled_students_allowance_eligible_expenses` records > 0 | 0 | 1 |
+| weighted people with a positive value | 0 | 4,654 |
+| weighted GBP | 0 | 83,772,000 |
+| row value | 0 | 18,000 (DfE maximum at 2025-01-01) |
+| `education_grants` residual, weighted GBP | 2,816,276,654 | 2,732,504,654 |
+| `education_grants` records > 0 | 189 | 188 |
+
+Reading: the DSA capacity is evaluated at the release calibration year (2025) with the
+three eligibility booleans materialised at that year on the 2024 frame, so the seed opens
+to one record whose full reported grant moves out of the residual (delta GBP 83,772,000 on
+both lines). uk-data#480's receipt on the same tabs found the same thin seed (1 record,
+4.7k weighted, GBP 83.8m). The `education_grants` residual level differs from the
+incumbent's (GBP 2,802.5m before its fix) because the two grant splits differ
+upstream of DSA; the delta is what this change owns and it matches. The pinned enhanced-FRS parity reference still lists the column
+as zero-share, so the register signs it as a candidate-only column
+(`dsa-eligible-expenses-seeded-at-calibration-year`) rather than re-minting the reference.
+
+## Relabel versus uprating for the DSA eligibility booleans
+
+uk-data evaluates the DSA seed with `sim.calculate(variable, policy_year)` on a
+2024-stamped dataset, which policyengine-uk uprates on load; microcosm's engine
+adapter builds a `UKSingleYearDataset(fiscal_year=2025)` from the 2024 tables, a
+relabel with no uprating. Measured on the same licensed frame after
+`frs_legacy_proxies` (34,966 persons, policyengine-uk 2.94.0), the three booleans
+are identical under both paths on every person, and identical at 2024:
+
+| variable | true, relabel at 2025 | true, uprated to 2025 | true, at 2024 | rows differing |
+|---|---|---|---|---|
+| maintenance_loan_in_england_system | 25,326 | 25,326 | 25,326 | 0 |
+| disabled_students_allowance_course_eligible | 683 | 683 | 683 | 0 |
+| disabled_students_allowance_has_qualifying_condition | 3,729 | 3,729 | 3,729 | 0 |
+| all three (eligible) | 11 | 11 | 11 | 0 |
+
+The seed is therefore invariant to the relabel; the 11 eligible people carry one
+reported grant record between them, which is the single DSA row above.

packages/microcosm-build/src/microcosm/build/spec_engine/schema/sources.schema.json

@@ -5596,6 +5596,13 @@
       },
       "fillna": {
        "type": "number"
+      },
+      "year_rule": {
+       "type": "string",
+       "enum": [
+        "survey_year",
+        "calibration_year"
+       ]
       }
      },
      "anyOf": [
@@ -5647,6 +5654,13 @@
       },
       "consumed_only": {
        "type": "boolean"
+      },
+      "year_rule": {
+       "type": "string",
+       "enum": [
+        "survey_year",
+        "calibration_year"
+       ]
       }
      }
     },
@@ -7410,7 +7424,11 @@
        "const": "assign_student_loan_plan_cohorts"
       },
       "year_rule": {
-       "type": "string"
+       "type": "string",
+       "enum": [
+        "survey_year",
+        "calibration_year"
+       ]
       },
       "start_year_formula": {
        "type": "string"
@@ -7476,7 +7494,11 @@
        "type": "string"
       },
       "year_rule": {
-       "type": "string"
+       "type": "string",
+       "enum": [
+        "survey_year",
+        "calibration_year"
+       ]
       },
       "age_min": {
        "type": "integer"

packages/microcosm-build/src/microcosm/build/uk/release_input_coverage_manifest.json

@@ -473,7 +473,7 @@
     "capital_gains"
    ],
    "source_manifest": "source_stages.json",
-   "source_manifest_sha256": "e07d148799b5025179038c710634c539a25d979c6b8bdd377eda3f4a6543ad22",
+   "source_manifest_sha256": "d83310aa9ebb309b4a9c6409a35916cc49390e330827cce8d7321c613679cd84",
    "source_vintages": {
     "source": "HMRC Capital Gains Tax statistics, July 2025, Table 2.1a",
     "survey": "HMRC Capital Gains Tax statistics Table 2.1a and Advani-Summers capital-gains incidence"
@@ -496,7 +496,7 @@
     "capital_gains"
    ],
    "source_manifest": "source_stages.json",
-   "source_manifest_sha256": "e07d148799b5025179038c710634c539a25d979c6b8bdd377eda3f4a6543ad22",
+   "source_manifest_sha256": "d83310aa9ebb309b4a9c6409a35916cc49390e330827cce8d7321c613679cd84",
    "source_vintages": {
     "source": "Advani and Summers (2020), Capital Gains and UK Inequality, CAGE Working Paper 465",
     "survey": "Family Resources Survey 2024-25, SPI synthetic support, and Advani-Summers capital-gains incidence"
@@ -525,7 +525,7 @@
    "required_mass_change_reason": "E5 source-stage transform preserves household rows and typed household weights; total household mass is conserved.",
    "rewrites": [],
    "source_manifest": "source_stages.json",
-   "source_manifest_sha256": "e07d148799b5025179038c710634c539a25d979c6b8bdd377eda3f4a6543ad22",
+   "source_manifest_sha256": "d83310aa9ebb309b4a9c6409a35916cc49390e330827cce8d7321c613679cd84",
    "source_vintages": {
     "source": "UK Data Service SN 8856 Effects of Taxes and Benefits household tab, DfT rail fare index, and public NHS activity/cost table.",
     "survey": "Effects of Taxes and Benefits 1977-2024 and NHS age-gender public table"
@@ -545,7 +545,7 @@
    "required_mass_change_reason": "E5 source-stage transform preserves household rows and typed household weights; total household mass is conserved.",
    "rewrites": [],
    "source_manifest": "source_stages.json",
-   "source_manifest_sha256": "e07d148799b5025179038c710634c539a25d979c6b8bdd377eda3f4a6543ad22",
+   "source_manifest_sha256": "d83310aa9ebb309b4a9c6409a35916cc49390e330827cce8d7321c613679cd84",
    "source_vintages": {
     "source": "UK Data Service SN 8856 Effects of Taxes and Benefits household tab and cited VAT anchor resource.",
     "survey": "Effects of Taxes and Benefits 1977-2024"
@@ -576,7 +576,7 @@
    "superseded_by": {
     "reason": "The FRS spine build executes hmrc_cgt_gains_spine, which applies the same HMRC Table 3 amounts redraw directly in source_stages.json before calibration.",
     "source_manifest": "source_stages.json",
-    "source_manifest_sha256": "e07d148799b5025179038c710634c539a25d979c6b8bdd377eda3f4a6543ad22",
+    "source_manifest_sha256": "d83310aa9ebb309b4a9c6409a35916cc49390e330827cce8d7321c613679cd84",
     "stage": "hmrc_cgt_gains_spine"
    }
   },
@@ -596,7 +596,7 @@
     "capital_gains"
    ],
    "source_manifest": "source_stages.json",
-   "source_manifest_sha256": "e07d148799b5025179038c710634c539a25d979c6b8bdd377eda3f4a6543ad22",
+   "source_manifest_sha256": "d83310aa9ebb309b4a9c6409a35916cc49390e330827cce8d7321c613679cd84",
    "source_vintages": {
     "hmrc_surface": "2023-24",
     "mapped_build_period": "2024"
@@ -610,7 +610,7 @@
    "base_candidate_tier": "frs",
    "calibration_permitted": false,
    "canonical_source_manifest": "source_stages.json",
-   "canonical_source_manifest_sha256": "e07d148799b5025179038c710634c539a25d979c6b8bdd377eda3f4a6543ad22",
+   "canonical_source_manifest_sha256": "d83310aa9ebb309b4a9c6409a35916cc49390e330827cce8d7321c613679cd84",
    "effective_mass_requirements": {
     "charitable_investment_gifts": {
      "mass_share_denominator": "all_person_effective_mass",
@@ -693,7 +693,7 @@
    "superseded_by": {
     "reason": "The FRS spine build executes hmrc_spi_income_spine, which supersedes the June retained-leaves/hmrc_spi_income pair inside source_stages.json.",
     "source_manifest": "source_stages.json",
-    "source_manifest_sha256": "e07d148799b5025179038c710634c539a25d979c6b8bdd377eda3f4a6543ad22",
+    "source_manifest_sha256": "d83310aa9ebb309b4a9c6409a35916cc49390e330827cce8d7321c613679cd84",
     "stage": "hmrc_spi_income_spine"
    }
   },
@@ -727,7 +727,7 @@
    "required_mass_change_reason": "E5 source-stage transform preserves household rows and typed household weights; total household mass is conserved.",
    "rewrites": [],
    "source_manifest": "source_stages.json",
-   "source_manifest_sha256": "e07d148799b5025179038c710634c539a25d979c6b8bdd377eda3f4a6543ad22",
+   "source_manifest_sha256": "d83310aa9ebb309b4a9c6409a35916cc49390e330827cce8d7321c613679cd84",
    "source_vintages": {
     "source": "UK Data Service SN 9468 Living Costs and Food Survey 2023-24 household/person tabs, NEED 2023 headline energy tables, Ofgem Q2 2026 unit rates, and WAS round-8 bridge donor.",
     "survey": "Living Costs and Food Survey 2023-24"
@@ -748,7 +748,7 @@
     "property_wealth"
    ],
    "source_manifest": "source_stages.json",
-   "source_manifest_sha256": "e07d148799b5025179038c710634c539a25d979c6b8bdd377eda3f4a6543ad22",
+   "source_manifest_sha256": "d83310aa9ebb309b4a9c6409a35916cc49390e330827cce8d7321c613679cd84",
    "source_vintages": {
     "source": "MHCLG dwellings and ONS UK House Price Index December 2025 regional average prices.",
     "survey": "Public regional property reference"
@@ -772,7 +772,7 @@
     "employee_pension_contributions"
    ],
    "source_manifest": "source_stages.json",
-   "source_manifest_sha256": "e07d148799b5025179038c710634c539a25d979c6b8bdd377eda3f4a6543ad22",
+   "source_manifest_sha256": "d83310aa9ebb309b4a9c6409a35916cc49390e330827cce8d7321c613679cd84",
    "source_vintages": {
     "source": "HMRC, Salary sacrifice reform for pension contributions effective from 6 April 2029",
     "survey": "Family Resources Survey 2024-25 salary-sacrifice respondents and HMRC salary-sacrifice reform analysis"
@@ -794,7 +794,7 @@
     "student_loan_plan"
    ],
    "source_manifest": "source_stages.json",
-   "source_manifest_sha256": "e07d148799b5025179038c710634c539a25d979c6b8bdd377eda3f4a6543ad22",
+   "source_manifest_sha256": "d83310aa9ebb309b4a9c6409a35916cc49390e330827cce8d7321c613679cd84",
    "source_vintages": {
     "source": "Explore Education Statistics Table 6a, Higher education total",
     "survey": "Family Resources Survey 2024-25 and Student Loans Company borrower forecasts for England"
@@ -829,7 +829,7 @@
    "required_mass_change_reason": "E5 source-stage transform preserves household rows and typed household weights; total household mass is conserved.",
    "rewrites": [],
    "source_manifest": "source_stages.json",
-   "source_manifest_sha256": "e07d148799b5025179038c710634c539a25d979c6b8bdd377eda3f4a6543ad22",
+   "source_manifest_sha256": "d83310aa9ebb309b4a9c6409a35916cc49390e330827cce8d7321c613679cd84",
    "source_vintages": {
     "source": "Office for National Statistics Wealth and Assets Survey, UK Data Service SN 7215, DOI 10.5255/UKDA-SN-7215-20; local licensed 2006-22 household tab.",
     "survey": "Wealth and Assets Survey round 8"

packages/microcosm-build/src/microcosm/build/uk/source_stages.json

@@ -511,11 +511,13 @@
    "operations": [
     {
      "kind": "derive",
-     "parameters": "disability category thresholds from baseline.gov.dwp at Jan-1"
+     "parameters": "disability category thresholds from the fiscal-converted gov.dwp tree at the survey year",
+     "year_rule": "survey_year"
     },
     {
      "kind": "derive",
-     "parameters": "disability flags from gov.dwp at Jan-1"
+     "parameters": "disability flags from the fiscal-converted gov.dwp tree at the survey year",
+     "year_rule": "survey_year"
     }
    ],
    "outputs": [
@@ -669,11 +671,24 @@
       "parents_learning_allowance",
       "adult_dependants_grant"
      ],
-     "consumed_only": true
+     "consumed_only": true,
+     "year_rule": "survey_year"
+    },
+    {
+     "kind": "materialize_rules_engine_predictors",
+     "predictors": [
+      "maintenance_loan_in_england_system",
+      "disabled_students_allowance_course_eligible",
+      "disabled_students_allowance_has_qualifying_condition"
+     ],
+     "consumed_only": true,
+     "year_rule": "calibration_year"
     },
     {
      "kind": "derive",
-     "scope": "proportional split of aggregate education_grants and DSA residual capacity"
+     "scope": "proportional split of aggregate education_grants and DSA residual capacity",
+     "parameters": "DSA maximum from gov.dfe.disabled_students_allowance.maximum at the calibration year",
+     "year_rule": "calibration_year"
     }
    ],
    "outputs": [
@@ -685,7 +700,7 @@
    "nonnegative_outputs": [
     "disabled_students_allowance_eligible_expenses"
    ],
-   "notes": "Runs before BRMA and always runs; both are signed inert differences for 2023-24. Pre-2025 DSA capacity is an aligned zero vector rather than an engine shape-sizing read. EDUCQUAL grant-split compatibility is re-cited to the FRS 2024-25 SN 9563 dictionary; codes 1-87 are empirically present and code 87 remains undocumented/defaulted."
+   "notes": "Runs before BRMA and always runs; the DSA seed evaluates at the release calibration year (uk-data#478) and the modelling-start gate compares against that year, not the frame period. EDUCQUAL grant-split compatibility is re-cited to the FRS 2024-25 SN 9563 dictionary; codes 1-87 are empirically present and code 87 remains undocumented/defaulted."
   },
   {
    "stage": "frs_take_up",

packages/microcosm-build/src/microcosm/build/uk/spec/sources.yaml

@@ -434,9 +434,11 @@ stages:
   artifacts: []
   operations:
   - kind: derive
-    parameters: disability category thresholds from baseline.gov.dwp at Jan-1
+    parameters: disability category thresholds from the fiscal-converted gov.dwp tree at the survey year
+    year_rule: survey_year
   - kind: derive
-    parameters: disability flags from gov.dwp at Jan-1
+    parameters: disability flags from the fiscal-converted gov.dwp tree at the survey year
+    year_rule: survey_year
   outputs:
   - aa_category
   - dla_sc_category
@@ -553,15 +555,25 @@ stages:
     - parents_learning_allowance
     - adult_dependants_grant
     consumed_only: true
+    year_rule: survey_year
+  - kind: materialize_rules_engine_predictors
+    predictors:
+    - maintenance_loan_in_england_system
+    - disabled_students_allowance_course_eligible
+    - disabled_students_allowance_has_qualifying_condition
+    consumed_only: true
+    year_rule: calibration_year
   - kind: derive
     scope: proportional split of aggregate education_grants and DSA residual capacity
+    parameters: DSA maximum from gov.dfe.disabled_students_allowance.maximum at the calibration year
+    year_rule: calibration_year
   outputs:
   - disabled_students_allowance_eligible_expenses
   rewrites:
   - education_grants
   nonnegative_outputs:
   - disabled_students_allowance_eligible_expenses
-  notes: Runs before BRMA and always runs; both are signed inert differences for 2023-24. Pre-2025 DSA capacity is an aligned zero vector rather than an engine shape-sizing read. EDUCQUAL grant-split compatibility is re-cited to the FRS 2024-25 SN 9563 dictionary; codes 1-87 are empirically present and code 87 remains undocumented/defaulted.
+  notes: Runs before BRMA and always runs; the DSA seed evaluates at the release calibration year (uk-data#478) and the modelling-start gate compares against that year, not the frame period. EDUCQUAL grant-split compatibility is re-cited to the FRS 2024-25 SN 9563 dictionary; codes 1-87 are empirically present and code 87 remains undocumented/defaulted.
 - stage: frs_take_up
   survey: Family Resources Survey 2024-25
   source: Family Resources Survey 2024-25 reported receipt anchors plus sourced UK take-up contract rates.

packages/microcosm-build/src/microcosm/build/uk/spine_swap_signed_differences.json

@@ -612,6 +612,31 @@
     }
    }
   },
+  {
+   "id": "dsa-eligible-expenses-seeded-at-calibration-year",
+   "class": "defect_fix",
+   "scope": {
+    "surface": "nonzero_shares",
+    "columns": [
+     "disabled_students_allowance_eligible_expenses"
+    ],
+    "entities": [
+     "person"
+    ]
+   },
+   "expectation": "column_missing_in_reference",
+   "magnitude_evidence": "The education-grant split evaluates the DSA eligibility seed and the DfE maximum at the release calibration year (uk-data#478) instead of the frame period, so disabled_students_allowance_eligible_expenses is populated in the candidate while the pinned enhanced-FRS reference, exported before uk-data#480, lists it as a zero-share column. On the pinned FRS 2024-25 tabs the seed is 1 person record, 4,654 weighted people and GBP 83,772,000 at the GBP 18,000 maximum, moved out of the education_grants residual; disability categories and flags are unchanged row for row on the same run.",
+   "evidence": "experiments/862-policy-year-rule-receipts.md#dsa-seed-at-the-calibration-year",
+   "adjudicator": "juaristi22",
+   "adjudicated_on": "2026-09-07",
+   "quantitative": {
+    "structural": {
+     "expected_columns": [
+      "disabled_students_allowance_eligible_expenses"
+     ]
+    }
+   }
+  },
   {
    "id": "uc-deduction-random-draw-net-new-column",
    "class": "net_new_column",