Align UK CGT calibration with FY2024-25 individual observations

changelog.d/875-cgt-observation-year.fixed.md

@@ -0,0 +1,9 @@
+Measure CGT in FY2024–25 while retaining the 2025 calibration index and 2024
+exported data. Pin individual HMRC gains, taxpayer count and liability facts;
+retain the replaced OBR cash forecast as optional diagnostic provenance with
+explicit availability. Retire its obsolete fit exemption, preserve its signed
+history, update live compile-parity receipts and publish aggregate fit evidence.
+
+Retain the merged upstream claimant-role graph prerequisite without duplicate
+commits. Compose the retired fit exemptions and document that the recorded
+national comparison predates main's changed fitting surface and gate disposition.

docs/uk-cgt-observation-year-875.md

@@ -0,0 +1,124 @@
+# CGT observation year and individual scope
+
+The FY2024–25 dataset uses HMRC's three individual observations for that year:
+551,000 taxpayers, £119.258bn gains and £22.503bn liability. Calibration remains
+indexed at 2025 and exported source data remains dated 2024. The contract pins
+exact Chronicle keys, individual scope and `tax_year: 2024`; trust-inclusive
+national totals cannot satisfy these references.
+
+`measurement_period: 2024` forces the CGT outcome calculation to use 2024 rather
+than the default calibration period, and disables the stored-column shortcut.
+The dated aliases bind a variable and year, refuse a mismatched binding, and may
+not be persisted in the source H5. Gains and the taxpayer proxy share the 2024
+annual exempt amount (£3,000) and gains array. The proxy selects gains above the
+allowance; it does not identify every administrative taxpayer below that
+threshold. The gains input is net of losses before the allowance; the aggregate
+binding makes no additional loss or allowance deduction.
+
+## Observed-year fit and cash diagnostic
+
+For this accepted FY2024–25 deliverable, the three published individual
+observations replace the former trust-inclusive gains/count and OBR cash fit.
+This explicitly supersedes the former FY2025–26 solve disposition described in
+[#875](https://github.com/PolicyEngine/microcosm/issues/875) **for this observed-year
+acceptance unit**. The historical issue body and earlier deliverable remain part
+of the record; a scoped decision record does not close the issue or settle the
+general self-assessment liability-to-cash translation.
+The forward-year reconciliation remains separate work.
+
+The unchanged March 2026 OBR FY2025–26 cash forecast (£21.801546197bn) remains a
+separately pinned diagnostic outside fitting. Its exact key, fiscal year,
+source-projection assertion, value and source survive when the fact is available.
+Absent, incompatible or duplicate diagnostic data produces explicit `unavailable`
+metadata, the expected identity and a reason, with no numeric substitute. It does
+not remove any independently valid HMRC observation. A malformed committed
+cash declaration remains a compilation error; so do missing, duplicate or
+incompatible fitted HMRC observations. No cash row returns to the matrix.
+
+The former active `obr.capital_gains_tax@2025` fit exemption is retired because
+that row is no longer fitted. Its exact historical reason, approver,
+adjudication and dates are preserved in the
+[aggregate evidence](evidence/uk-cgt-889/calibration-comparison.json).
+This retirement creates no new human signature, renewal or expiry. Merged #892
+independently retired the five UC/pension fit exemptions. Their composition with
+this PR leaves the active target-fit exemption register empty; gate thresholds
+are unchanged and HMRC liability has no exemption.
+
+The recorded 9b5/cc9 comparison used 366 targets across 21 families. With `family_equal`,
+OBR has 19 rows instead of 20 and `hmrc_cgt` has three instead of two. Each retained
+OBR row changes from weight 1/420 to 1/399; gains/count change from 1/42 to 1/63;
+the replacement tax row changes from 1/420 to 1/63. These 22 changes follow family
+membership. The comparison therefore combines dating, individual scope, liability
+comparator and objective weighting; it is not an isolated causal year effect.
+
+**Applicability, 10 September 2026:** this PR is rebased onto main `45231f9e`,
+which preserves #892 and #877 from the `141bface` feature baseline. The later
+#899 changes only CI run scheduling and its documentation. Main's two additional
+UC payment-tail measure exclusions reduce the expected default fitted roster to
+364 of 415 references.
+Those exclusions apply before solving and are distinct from target-fit exemptions.
+The original 366-row objective, fitted weights and failed gate maps remain
+historical evidence. No fit or gate certification on the rebased head is claimed.
+
+## Dating, revision and materialization
+
+The #881 `year_rule` mechanism resolves release-relative survey/calibration years
+for source construction, policy lookups and predictor materialization. A fixed
+administrative observation must keep its date when the release advances, so its
+`measurement_period` is a separate target contract. A future shared generic
+variable/period resolver may simplify the implementation; automatically replacing
+2024 with the survey year would change the present observation's meaning.
+
+The provisional HMRC observations use exact aggregate keys. A different-key
+revision alongside the old observation leaves the old key selected. Removing the
+pinned key or duplicating it fails compilation. The authoring field
+`matched_fact_count_at_or_before_period` is descriptive metadata, not a runtime
+revision guard. A lone altered value under the same key is accepted by the
+compiler if artifact verification is bypassed; immutable facts and manifest
+hashes prevent that alteration in the authenticated build path.
+
+To adopt a revision, authenticate a new Chronicle artifact, review its year,
+population and values, deliberately update changed keys and feed hashes, and
+regenerate references, membership, relevant public fixtures and parity receipts.
+Rerun strict-selection, real-feed and dated-export checks; refresh numerical
+evidence if fitted values change. Never loosen selectors to choose the latest
+publication silently.
+
+Both national and local paths resolve the dated arrays before materialization.
+The rowwise builder resolves each block, aligns by entity IDs, injects the alias
+columns and then builds the target matrix. Scoring uses the same resolve/inject
+route. The adapter's refusal of an unresolved alias is intentional. The national
+and local parametrized integration tests exercise the real resolver with distinct
+2024/2025 outcomes, fit weights and restore the original base-year HDF columns;
+a separate UK2.97 engine case verifies the calculation date. The source-column
+restoration does not reverse the fitted weights. These tests do not certify a
+full local calibration.
+
+## Source, dependency and evidence boundary
+
+The lock remains PolicyEngine-UK 2.97.0 / Core 3.31.0. National targets use Chronicle
+`ec7169b5db40b9f54117c80f70f14efc1dd0fedd`; local targets retain the separate `6fb700e`
+artifact. The four CGT/OBR facts are identical across those feeds. The #891 paid
+UC source windows and family classification, and #892's explicit monthly-window
+metadata, are preserved.
+
+Source Tables 2/3 remain the 2025 publication's FY2023–24 distribution, mapped into
+the 2024 build. Donor selection, carrier count, gain amounts, ages and geography
+are unchanged. Older source-stage prose describes the preceding two-target or
+trust-inclusive surface; the current target contract controls the three fitted
+individual observations. Distribution fences remain active. No source-vintage,
+tail, age or liability-model repair is implemented here.
+
+Merged [#892](https://github.com/PolicyEngine/microcosm/pull/892) supplies the #883
+graph prerequisite, its serialized fixture and the claimant-role regression.
+The rebase drops this PR's duplicate prerequisite commits and retains the upstream
+coverage. Merged #877's size-selection and checkpoint behavior, corrected local
+private-rent target composition and attested calibration identities are also
+preserved. This PR does not claim a new source build or full local calibration.
+
+The [historical matched comparison](../experiments/889-cgt-observation-year.md)
+records the 9b5/ec7 candidate and cc9/ec7 control on the identical source. Its
+£108.032bn gains remain 9.41% below HMRC, despite liability and taxpayer totals
+being close. Both that comparison and the older 30479731/6fb comparison retain
+their original code and target-surface provenance. Neither historical national
+fit, diagnostic export nor unit tests certify a releasable dataset.

experiments/889-cgt-observation-year.md

@@ -0,0 +1,128 @@
+# Observed FY2024–25 CGT calibration comparison
+
+In the recorded 9b5/cc9 comparison, the observed-year contract brings taxpayer
+count and liability close to the individual HMRC observations, while gains remain
+9.41% low. These are historical national fits on one unchanged source H5. They
+combine the measurement-year correction,
+individual scope, liability comparator and automatic family weighting changes;
+they do not isolate the effect of any one change.
+
+**Applicability, 10 September 2026:** the rebase onto main `45231f9e` preserves
+merged #892 and #877 from the `141bface` feature baseline. The later #899 changes
+only CI run scheduling and its documentation. The original JSON, CSV, 366-row
+objective and gate maps below remain unchanged. Main now declares 51 measure
+exclusions, giving an expected 364-row default fitting surface, and the combined
+target-fit exemption register is empty. The old two empty-band and five
+stale-exemption failures are historical
+run outcomes, not a verdict for the rebased surface. No current-head paired fit,
+unchanged fitted weights, release-gate pass or full local calibration is claimed.
+
+| Common FY2024–25 individual measure | HMRC target | Under control weights | Under candidate weights | Control error | Candidate error |
+| --- | ---: | ---: | ---: | ---: | ---: |
+| Taxpayers | 551,000 | 583,538.709 | 551,037.296 | +5.9054% | +0.0068% |
+| Gains (£bn) | 119.258 | 127.484860 | 108.031570 | +6.8984% | −9.4136% |
+| Liability (£bn) | 22.503 | 26.986389 | 22.653870 | +19.9235% | +0.6704% |
+
+Both columns evaluate the same captured 2024 outcomes. The control's native 2025
+cash-target error is a different comparison and is not substituted into this
+table. Cash receipts have different timing and population scope from disposal-year
+individual liability.
+
+The [JSON receipt](../docs/evidence/uk-cgt-889/calibration-comparison.json) contains
+full-precision values, exact HMRC and OBR identities, options, hashes, original
+gate results, concentration metrics and all 22 changed objective weights. The
+[363-row CSV](../docs/evidence/uk-cgt-889/target-comparison.csv) contains every
+unchanged non-CGT target, both estimates and both errors. Every corresponding
+non-CGT target value and household coefficient row was checked identical.
+
+## Trade-offs and remaining gates
+
+Gains' absolute error worsens from 6.90% to 9.41%; being just inside the 10%
+diagnostic threshold does not establish an adequate gains model. Household ESS
+falls from 14,049.762 to 13,725.441. Among the same 707 gain-above-allowance carriers,
+ESS falls from 490.704 to 384.586. The 692 FRS origin groups do not imply 692
+independent CGT donors. These fits remain sensitive to their limited support.
+
+Among 363 non-CGT rows, the number within 10% rises from 347 to 348, and both fits
+have 361 within 25%; neither threshold gains a new miss. The two empty childless
+couple UC payment bands remain at −100%. Both original runs exit with a failed
+terminal gate and no staged calibrated H5. Five stale UC/pension exemptions also
+appear in both original failure receipts. Their disposition is separate from
+this PR's retirement of the dormant OBR cash exemption. Original failed receipts
+are preserved, including the historical dormant OBR entry in the candidate.
+
+The diagnostic HDF audit verifies all 218 original person/benefit-unit/household
+columns and exact captured fitted weights at base period 2024. That audit is not
+a passing release export. The local integration tests cover the resolver and
+HDF route on synthetic cases; no full local calibration is certified here.
+
+## Scoped supersession and historical record
+
+For the accepted FY2024–25 release, three individual observations are the fitted
+CGT acceptance unit. This supersedes #875's previous FY2025–26 fit disposition
+only for that observed-year deliverable. The OBR March 2026 FY2025–26 forecast
+remains unchanged as optional diagnostic provenance, with explicit unavailable
+status if its exact fact cannot be used. No general payment-lag translation,
+forward-year gains level or replacement distribution prior is inferred.
+
+The JSON receipt archives the exact former `obr.capital_gains_tax@2025` signed
+exemption, including its original reason, approver and approval/expiry dates.
+This PR retires that entry; merged #892 independently retired the other five
+fit exemptions, so the combined active register is empty. HMRC liability has no
+exemption. This records a scoped implementation disposition without a new human
+signature or closure of #875's reconciliation work. The historical issue body and
+signed ruling remain intact alongside any scoped follow-up decision record.
+
+The frozen incumbent parity fixture remains historical evidence. Current signed
+differences identify the new individual values, the added HMRC liability row and
+the cash row absent from the fit; they do not overwrite the incumbent's values.
+The original national calibration battery excludes release-cut-owned compile
+parity. The new real-feed 2023/2025 compile-parity checks are separate validation;
+the recorded national fits do not establish parity or current release readiness.
+
+## Provenance and reproduction
+
+Numerical candidate: `9b5ac7c52f8a3742940674dd2b58a7595ea24db6`.
+Control: `cc9c953c72b003406f4aeec8ae5e82bd211099bc`.
+Source H5 SHA256:
+`03a63ecc1d0b3064d42e8b412fa4a28c587a79bd153ff2ffebc2715f4b316e33`.
+Both use UK 2.97.0, Core 3.31.0 and the same locked environment; the receipt records
+the full lock and installed-version identities. National Chronicle is `ec7169b5`:
+facts `4a50ee9568a01bbb57f73d927084ed6b4b9e52249b51a2338455874ae6e382b5`,
+manifest `a95d0ee9f87f36947eaecdb3de29cf81a91e47ccaa822fed42da677eedca877f`.
+Local Chronicle remains independently pinned to `6fb700e`.
+
+Both solves use 1,500 Adam updates, learning rate 0.02, seed 0, `family_equal`,
+free household mass, cap 10, no sparsity penalty and `best_feasible_loss` iterate
+selection (control epoch 1333; candidate 1422). The defaults are not overridden
+in the library. These recorded runs used 366 targets and 21 families.
+Under this treatment, 19 retained OBR weights change from 1/420 to 1/399;
+gains/count change from 1/42 to 1/63; cash-to-liability changes from 1/420 to 1/63.
+
+A reviewer with authorized access to the exact source can use
+`tools/calibrate_uk_national_dataset.py` in the two pinned checkouts, with the
+recorded H5 and Chronicle SHA arguments, `--epochs 1500 --learning-rate 0.02
+--target-weight-rule family_equal --target-loss-cap 10`, separate output paths
+and non-release identifiers. The two runs must start from identical household
+IDs and initial weights. Preserve the original gate failures; do not bypass
+them to obtain an apparent passing release. Private saved matrices/weights are
+identified by SHA in the receipt and are not published here.
+
+For public contract verification, authenticate the national artifact against its
+committed manifest, run `tools/build_uk_ledger_compile_parity_signed_differences.py
+--surface national --ledger-facts <national-facts.jsonl> --output-dir <scratch>`,
+and compare the two national receipts. The full-feed test also applies the real
+2023 and 2025 compile-parity gates. National and local reference regeneration
+must each use their own authenticated feed. The committed CSV and JSON expose
+full-precision estimates, targets, errors and concentration statistics for audit.
+Recomputing weighted outcomes or ESS from microdata requires the authenticated
+private captures; those records are not included in this public receipt.
+
+Post-run review repairs alter optional diagnostic availability, the active
+exemption register, parity explanations, tests and documentation. They preserve
+the captured source and numerical result arrays. The subsequent main rebase also
+inherits a changed current fitting surface and calibration source identities;
+these historical results do not establish current-head numerical equivalence.
+Both the original 30479731/6fb run and the 9b5/cc9 comparison remain historical.
+Source-vintage, age support, tail allocation and the generic liability mechanism
+remain separate, unimplemented follow-ups.

packages/microcosm-build/src/microcosm/build/target_materialization.py

@@ -152,6 +152,9 @@ def resolve_target_measures(
     source frame rather than on the restored output.
     """
 
+    validate_period = getattr(provider, "validate_period", None)
+    if callable(validate_period):
+        validate_period(period)
     contract = _measure_resolution_contract(
         registry, provider, contract_targets=contract_targets
     )
@@ -484,12 +487,16 @@ def materialize_target_bindings(
     )
     skipped: list[MaterializationSkip] = []
     for spec in registry.specs:
-        if hasattr(adapter, "has_column") and adapter.has_column(
-            spec.entity, spec.measure
-        ):
-            continue
         contract_target_id = spec.metadata.get("contract_target_id")
         target = contract_targets.get(str(contract_target_id))
+        binding = target["bindings"]["policyengine"] if target is not None else {}
+        if (
+            hasattr(adapter, "has_column")
+            and adapter.has_column(spec.entity, spec.measure)
+            and "measurement_period" not in binding
+            and not binding.get("require_matching_fact_period")
+        ):
+            continue
         if target is None:
             skipped.append(
                 MaterializationSkip(
@@ -499,14 +506,23 @@ def materialize_target_bindings(
                 )
             )
             continue
-        binding = target["bindings"]["policyengine"]
+        # A period-constrained binding must validate its fact and prepare again.
+        # A pre-existing column has no period provenance and may be stale.
         kind = binding.get("kind")
         try:
+            measurement_period = binding.get("measurement_period", period)
+            if binding.get("require_matching_fact_period") and str(
+                spec.metadata.get("ledger_fact_period")
+            ) != str(measurement_period):
+                raise ValueError(
+                    f"observation period {spec.metadata.get('ledger_fact_period')!r} "
+                    f"does not match declared measurement period {measurement_period}"
+                )
             if kind:
                 provider = provider_registry.get(str(kind))
                 if provider is None:
                     raise ValueError(f"unsupported binding kind {kind!r}")
-                values = provider(adapter, binding, period)
+                values = provider(adapter, binding, measurement_period)
             else:
                 band = None
                 if binding.get("groupby_variable"):