Add UC family type totals from 2023

[juaristi22]

Summary

• Import DWP Stat-Xplore Universal Credit family-type counts for the April–December windows in 2023, 2024, and 2025.
• Preserve all 27 publisher-emitted Total cells under the distinct total_benefit_units source measure instead of deriving totals from the five family-type rows.
• Replace the 2025-only API response with the exact 4,343-byte publisher CSV, including revision annotations and the disclosure-control notice, and archive the same bytes in R2.
• Re-mint the 45 previously imported 2025 fact keys because the repinned source identity (source_table, source_file, and vintage) changed; their values, dimensions, periods, and source record IDs remain unchanged.
• Add explicit source-row dimension declarations so Chronicle can validate row semantics when a publisher uses an ambiguous axis header.
• Update UK source coverage documentation and merged-bundle count assertions.

Chronicle governance

• Approved Chronicle agent role: ledger-source-ingestor
• Deterministic checks run:
  • Source-package validation: PASS — 54 record sets and 162 source-backed facts with no errors or warnings.
  • Source-cell preservation: PASS — the committed CSV matches the downloaded Stat-Xplore file byte-for-byte; its SHA-256 is pinned in the manifest and R2 metadata.
  • Consumer-contract validation: PASS — 135 detail facts and 27 published Total facts validate successfully.
  • Raw-facts boundary validation: PASS — Chronicle stores the publisher values without aging, imputation, or reconciliation.
• LLM judge verdicts:
  • ledger-source-fidelity: PASS — every emitted value resolves to the archived publisher CSV, and all facts carry source-row and source-cell lineage.
  • ledger-contract: PASS — source_row_dimensions supplies declarative evidence for guarded row headers without changing consumer-fact identity or values.
  • ledger-boundary: PASS — the package preserves DWP's Total independently; the detail-to-total comparison exists only as a test and does not emit a computed fact.

Tests

• .venv/bin/chronicle validate-package dwp-uc-households-family-type-april-december-2025 --year 2025
• .venv/bin/pytest -q tests/test_chronicle_source_package.py -k 'dwp_uc_children_composition or dwp_uc_family_type'
• .venv/bin/pytest -q tests/test_chronicle_source_package.py::test_every_source_package_record_set_declares_provenance_class
• .venv/bin/pytest -q tests/test_chronicle_suite.py
• .venv/bin/pytest -q tests/test_chronicle_bundle.py::test_build_bundle_writes_merged_consumer_contract
• .venv/bin/ruff check

Fixes #247

[vahid-ahmadi]

Review at 024a2d18 (Claude Code, high effort; fresh checkout, full suite run: 870 passed, 1 skipped; validate-package on the package: 54 record sets, 162 rows, 0 errors or warnings, matching the body; committed CSV hashes to the manifest's 3783b64b…5764 at 4,343 bytes and the sha is enforced at build time on both the packaged and fetch paths).

The substance is right: all 45 pre-existing 2025 family-type values are byte-identical to the retired API JSON, the 27 Totals are the publisher's own row (expected_row_header: Total, table_record_kind: total, measure total_benefit_units, with the only sum-of-five living in a test), the Total carries filters: {family_type: all} so a consumer selecting the five named types cannot pick it up while a caseload consumer can select it by source_measure_id, and the bundle delta is exactly 117 (18 months × 6 + 9 Totals) with no other source moving. Total-versus-sum gaps are within ±11 per month, which is Stat-Xplore rounding. Two should-fixes, one question, and nits.

1. Should-fix — the source_table rename re-keys all 45 existing 2025 facts. source_package.yaml:11 changes source_table from "…April to December 2025" to "…April to December 2023 through 2025". build_fact_key hashes _canonical_key_payload, which includes source.source_table, source_file and vintage (chronicle/core.py:630-631); file and vintage churn is inherent to re-pinning, but the table string is a choice, and with all three changed none of the 45 old fact keys survive. By record id everything else is identical — values, dimensions, periods, source record ids — and microcosm's binding by source_concept: dwp.uc_households plus family type and month is unaffected, so this is key churn rather than a value change. Either keep the old source_table string or say in the body that the 2025 keys are re-minted.

2. Should-fix — the machine-readable query is gone and the manifest URL is not an artifact. The retired JSON carried the exact Stat-Xplore query (database, measure, recodes, total: false); the CSV carries only "Family Type by Month" and no geography cell, and the manifest's source_url is the JSF table-view page, which redirects to the catalogue explorer. The SOURCE_ARTIFACT_FETCH fallback would fetch HTML and fail the sha check, so the file cannot be re-materialised from the manifest alone. The sibling packages' REST URL returns 401 too, so unfetchability is parity; losing the query is new. Record the table path and field URIs in extraction_method, or keep the query JSON beside the CSV.

3. Question — the second commit's title overstates. "Declare UC record set provenance explicitly" adds 52 literal provenance_class: administrative lines under the anchor merges and no machinery: the field was already required at load and test_every_source_package_record_set_declares_provenance_class already existed. The commit exists because that test's final assertion counts text lines, which YAML anchor inheritance cannot satisfy. Fine to land; worth a title that says so, and the test is brittle.

4. Question — source_row_dimensions is self-attesting. It is load-bearing: dropping the Total declaration yields 27 row_constraint_not_evidenced errors and dropping all of them 297, and a wrong expected_row_header raises at load. But a declaration, filter and constraint that all say the same wrong value pass. Same limitation as the existing column dimensions; a docstring note is enough.

Nits. The .gitattributes comment says -diff protects the bytes; it only marks the file binary for diff rendering, and the CRLF bytes are intact today because nothing normalises them — the build-time sha is the real protection. The R2 key uses raw/uk/dwp/ where thirteen DWP manifests use raw/dwp/ and six the new form. ruff check reports 231 pre-existing errors on both merge base and head under the local 0.16.4, none introduced.

Could not verify

That the CSV is what Stat-Xplore serves (UI download only, no fetchable URL), the R2 object at the manifest key, and the 2024 (r) revisions against any earlier vintage.

---

Approve once 1 is decided either way; 2 is worth doing in this PR since the query was the one thing the old artifact documented better than the new one.

[vahid-ahmadi]

Round one is in the comment above (#250 (comment)): data and totals verified, two should-fixes (the source_table rename re-keys all 45 existing 2025 facts; the Stat-Xplore query is no longer recorded and the manifest URL is a page, not an artifact). Happy to approve once those are decided either way.

[juaristi22]

@vahid-ahmadi Addressed the two should-fix items from your review.

Should-address fixes

• Updated extraction_method to record the Stat-Xplore table path, database URI, measure URI, and both dimension field URIs from the retired query artifact.
• Updated the PR summary to state that repinning re-mints the 45 existing 2025 fact keys because source_table, source_file, and vintage changed. Their values, dimensions, periods, and source record IDs remain unchanged.

Skipped

• Left the commit-title/test-brittleness question, source_row_dimensions documentation note, .gitattributes wording, and R2 key convention unchanged because the approved scope covered the two should-fix items only.

Verification

• Package validation: 54 record sets, 162 rows, 0 errors, 0 warnings.
• Focused tests: 2 passed, 123 deselected.
• git diff --check: passed.
• Fix-plan verification: 2 of 2 approved items addressed; no unintended local files changed.

[vahid-ahmadi]

Second pass at f5c45f60 (Claude Code, high effort; the one response commit read against the two should-fixes; validate-package 54 record sets / 162 rows / 0 errors; the focused tests, the provenance-class test, the suite tests and the bundle tests: 31 passed; the full suite did not complete locally this round and CI is green on the head).

1. Verified. The PR body now says the 45 existing 2025 fact keys are re-minted because source_table, source_file and vintage changed. I rebuilt the facts on main and on this head: 45 source record ids in common, 0 of 45 keys survive, 45 of 45 identical on value and on every non-source part of the key payload (period, geography, entity, measure, aggregation, domain, filters); only the three named source fields differ. The sentence is accurate.

2. Verified, with one residual. extraction_method (source_package.yaml:16-24) now records the table path, str:database:UC_Households, the measure URI str:count:UC_Households:V_F_UC_HOUSEHOLDS, and both field URIs, all verbatim against the retired query JSON. Not carried: the recode value-URI lists (the nine C_UC_HH_DATE:YYYYMM and five C_UC_FAMILY_TYPE:{1,2,3,4,99} values) and the old total: false. The latter is moot now that the publisher Total is kept; the former follow an obvious pattern, so this is a nit rather than a gap.

The diff since my pass is exactly that one hunk in one file; the skipped items are as she listed.

Approving.