diff --git a/docs/calibration-map-hierarchy.md b/docs/calibration-map-hierarchy.md new file mode 100644 index 0000000..6b37298 --- /dev/null +++ b/docs/calibration-map-hierarchy.md @@ -0,0 +1,130 @@ +# Calibration map hierarchy + +The calibration map displays one producer-authored hierarchy for every target +in Microcosm diagnostics schema 8: + +1. provider +2. category +3. geography +4. zero or more dimensions +5. target + +The first three and final levels occur exactly once. `dimensions` is an ordered +array because different targets can have different numbers of categorical +coordinates. This normalized representation avoids recursive authoring data +while preserving the order in which the dashboard must traverse dimensions. + +## Schema 8 row + +```json +{ + "name": "obr.income_tax@2025", + "target_name": "obr.income_tax", + "hierarchy": { + "provider": { + "id": "obr", + "label": "Office for Budget Responsibility" + }, + "category": { + "id": "obr.efo_receipts", + "label": "Economic and fiscal outlook receipts", + "provider_id": "obr" + }, + "geography": { + "id": "K02000001", + "label": "United Kingdom", + "level": "country" + }, + "dimensions": [ + { + "id": "obr.efo_line", + "label": "Economic and fiscal outlook line", + "value_id": "income_tax", + "value_label": "Income tax (gross of tax credits)" + } + ], + "target": { + "id": "obr.income_tax", + "label": "Income tax receipts" + } + } +} +``` + +Microcosm owns the provider/category relationship. Chronicle supplies +geography, categorical dimensions, categorical value labels, and the preferred +target label when its selected fact can describe the resulting target exactly. +Microcosm supplies deterministic labels when Chronicle omits optional source +labels or when a target combines or restamps multiple facts. The dashboard does +not construct labels for schema 8. + +## Dashboard parsing + +The top-level `schema_version` selects one reader for the entire diagnostics +file. Target row structure is not used to choose a reader. + +| Diagnostics schema | Reader | +| --- | --- | +| 2–6 | Legacy name/filter reader | +| 7 | Structured source/variable/dimensions reader | +| 8 | Normalized hierarchy reader | + +An absent version, version 1, and unsupported future versions produce an +explicit compatibility error. A schema 7 file containing a legacy-shaped row +also produces an error instead of switching readers for that row. + +The schema 8 reader requires every identifier and label, verifies that the +category references the row's provider, rejects duplicate dimension IDs, and +verifies that `hierarchy.target.id` agrees with the diagnostic target ID. + +## Tree construction + +The map first groups targets by provider and category, then by geography. It +traverses each target's `dimensions` array in producer order. The traversal +supports: + +- targets with no dimensions; +- targets with one or many dimensions; +- different dimension sequences within the same category and geography; and +- target leaves alongside deeper dimension paths. + +A dimension with only one categorical value in the current branch is omitted +as an interactive choice. Its complete identifier and label remain on the +target row for details, comparisons, and diagnostics. Navigation uses +`value_id`; map text and breadcrumbs use the corresponding `value_label`. + +Schema 2–7 behavior remains isolated in its historical reader and tree adapter. +Humanization and country-specific parsing in those adapters must not be added +to the schema 8 path. + +## Required verification + +For a producer change, verify these representations in order: + +1. country target declaration; +2. generated scalar reference; +3. Chronicle fact selection; +4. compiled `TargetSpec`; +5. target-registry serialization and reload; +6. compiled calibration `Target`; +7. schema 8 diagnostics row; +8. dashboard hierarchy reader; and +9. calibration-map tree traversal. + +`tools/generate_uk_calibration_hierarchy_fixture.py` in Microcosm performs this +check with synthetic Chronicle-shaped facts for every registered UK target and +writes three representative rows separately for inspection. + +For local inspection, point the dashboard at the generated directory before +starting the frontend: + +```bash +export MICROCOSM_UK_LOCAL_CALIBRATION_DIR=/path/to/generated/artifact +cd frontend +bun run dev +``` + +The override applies only to the UK selection. The loader reads +`calibration_diagnostics.json` and optionally reads `build_manifest.json`, +`release_manifest.json`, and `demographics.json` from the same directory. +Without this variable, the normal repository-backed loader is unchanged. diff --git a/docs/spec-driven-countries.md b/docs/spec-driven-countries.md index 3d97c25..9c247ae 100644 --- a/docs/spec-driven-countries.md +++ b/docs/spec-driven-countries.md @@ -107,9 +107,10 @@ A registration with `fixture: true` (the conformance country `zz`) is a valid country for parsers and builders but is never listed in selectors or the release-alert allowlist. -Schema readers must remain backward-compatible while published releases migrate. -Name/filter parsing is a legacy adapter selected from each row's artifact shape, -never by country. +Schema readers remain backward-compatible for declared diagnostics schemas 2 +through 7 while published releases migrate. The top-level schema version selects +one reader for the complete file, never by country or individual row shape. See +[Calibration map hierarchy](calibration-map-hierarchy.md). ### Presentation @@ -209,39 +210,31 @@ their IDs and raw values are humanized. When an age-band value lacks an artifact label, range values such as `0_17` and `65_plus` become `0–17` and `65+`. -A geography-role dimension sets `row.geography` and uses its declared `level` -or `"region"`. Other dimensions become `target_dimensions` with `key`, +A geography-role dimension sets `row.geography`, `row.geography_id`, +`row.geography_dimension_id`, and `row.geography_rank`, and uses its declared +`level` or `"region"`. A target may contain at most one populated +geography-role dimension. Other dimensions become `target_dimensions` with `key`, `label`, `value`, `source_key`, `raw_value`, and an optional zero-based `rank`. The published dimension ID, not its display label, determines `key`. Simple lowercase IDs retain keys such as `bd_age_band`; IDs containing other characters receive a lossless query-safe encoding. Consequently, two distinct dimensions may share a display label without merging into one facet. -Facet values use rank order only when every displayed value has a rank; -otherwise the legacy facet sorter remains in force. Structured rows are also -excluded from whole-population estimate-scope inference. +Facet values, including geography values, use rank order only when every +displayed value has a rank; otherwise the legacy facet sorter remains in force. +Structured rows are also excluded from whole-population estimate-scope +inference. -### Target representation classification +### Target representation dispatch -The dashboard classifies each target row by structure before normalizing it, -then summarizes the complete `targets` array. Diagnostics schema versions do -not identify the target representation: published schema 5 and schema 6 files -can both contain legacy string fields. The structural classification is: +The top-level diagnostics schema selects exactly one target reader. Schemas 2–6 +use the legacy reader, schema 7 uses the structured reader described above, and +schema 8 uses the normalized hierarchy reader. Missing versions, version 1, and +unsupported future versions are rejected explicitly. A row that does not match +its file's declared representation is rejected; the dashboard does not choose a +different reader from that row's shape. -1. `structured` when every row has a plain-object `source` with a non-empty - `id`, a plain-object `variable` with a non-empty `id`, and a plain-object - `dimensions` field. Use `{}` when a target has no dimensions. -2. `legacy` when no row has object-valued `source`, `variable`, or `dimensions` - fields. -3. `mixed` when complete structured rows and complete legacy rows occur together. -4. `unknown` when there are no target rows. - -Every target row must independently satisfy either the structured or legacy -shape. Partially structured rows are invalid. In a mixed file, each complete -structured row uses the structured reader and each complete legacy row uses the -legacy reader; the file-level `mixed` value is descriptive and does not select a -third normalization strategy. - -Calibration summary and target-diagnostics responses report the classification: +Calibration summary and target-diagnostics responses report the selected +representation: ```json { @@ -254,10 +247,11 @@ Calibration summary and target-diagnostics responses report the classification: ``` `structured_dimensions` reports whether the diagnostics published a plain -dimension dictionary. `target_representation` summarizes the collection. The -existing per-row `dimension_adapter` response field remains for compatibility -and describes only whether that row's dimensions came from a structured object, -a known legacy filter, or legacy name and metadata parsing. +dimension dictionary. `target_representation` identifies the selected reader. +The existing per-row `dimension_adapter` response field remains for +compatibility and describes whether that row's dimensions came from a structured +object, a normalized hierarchy, a known legacy filter, or legacy name and +metadata parsing. ### Structured source and variable identifiers @@ -295,24 +289,18 @@ For a `legacy` file, the isolated legacy reader handles the established dotted, slash, Chronicle metadata, and known filter encodings. It does not interpret an arbitrary underscore as a structural separator. -For a `mixed` file, fully legacy rows retain legacy behavior. Partially -structured rows use the compatibility precedence: Chronicle publisher ID, -then `source.id`, then legacy source parsing; `variable.id`, then legacy -variable parsing; structured dimensions, then known filter dimensions, then -legacy metadata and name dimensions. This prevents one partially migrated row -from changing unrelated legacy rows in the same file. - ### Cross-release target matching -The legacy and structured readers normalize each row independently. Candidate -validation and weighted target-error comparisons then use one collection-level -matcher with this order: +The legacy, structured, and hierarchy readers normalize their rows before +candidate validation and weighted target-error comparison. The collection-level +matcher uses this order: 1. Exact, non-empty `base_name`, which uses `target_name` when supplied and otherwise removes the period suffix from `name`. 2. Exact, non-empty Chronicle `fact_key`. -3. For two structured rows, an exact tuple of source ID, variable ID, measure, - and raw dimension ID/value pairs sorted by dimension ID. +3. For two structured or hierarchy rows, an exact tuple of provider/source ID, + category/variable ID, measure, and raw dimension ID/value pairs. Schema 8 + preserves producer dimension order. At each step, a key is used only when it identifies exactly one still-unmatched row in each release. Duplicate and many-to-one keys remain unmatched unless a @@ -330,12 +318,15 @@ using the structured representation is available. ### Producer follow-up -Microcosm release producers must publish all of the following before the legacy -presentation and normalization readers can be retired: +Microcosm release producers must publish schema 8 hierarchy rows before the +legacy presentation and normalization readers can be retired. Each row must +contain: - `release_manifest.country`; - `release_manifest.presentation`; - `release_manifest.publisher_labels`; -- `calibration_diagnostics.dimensions`; -- a `targets[].dimensions` object on every row, including `{}` where empty; and -- structured `targets[].source` and `targets[].variable` objects on every row. +- a provider ID and label; +- a category ID, label, and provider relationship; +- a geography ID, label, and level; +- an ordered `dimensions` array, including `[]` where empty; and +- a target ID and label. diff --git a/frontend/components/microcosm/calibration-explorer-map.tsx b/frontend/components/microcosm/calibration-explorer-map.tsx index 1a55bfa..6143c74 100644 --- a/frontend/components/microcosm/calibration-explorer-map.tsx +++ b/frontend/components/microcosm/calibration-explorer-map.tsx @@ -632,7 +632,7 @@ export function CalibrationExplorerMap({ const upLabel = expandedView ? `Up to all ${data.currentLevel.label.toLowerCase()}` : explorerUpLabel(state); - const breadcrumbs = explorerBreadcrumbs(state); + const breadcrumbs = explorerBreadcrumbs(state, data.pathLabels); const selectedTarget = data.groups .flatMap((group) => group.nodes) .find((item) => item.kind === "target" && item.id === state.path.target)?.target; diff --git a/frontend/components/microcosm/calibration-explorer-view.test.ts b/frontend/components/microcosm/calibration-explorer-view.test.ts index 05d3ef3..6d5d0e9 100644 --- a/frontend/components/microcosm/calibration-explorer-view.test.ts +++ b/frontend/components/microcosm/calibration-explorer-view.test.ts @@ -50,18 +50,18 @@ describe("calibration explorer presentation model", () => { ); }); - test("resolves program labels at the final presentation boundary", () => { + test("uses the program label resolved by the tree", () => { expect( explorerNodeLabel({ id: "taxable interest income", - label: "taxable interest income", + label: "Taxable interest income", kind: "program", }), ).toBe("Taxable interest income"); expect( explorerNodeLabel({ id: "refundable ctc", - label: "refundable ctc", + label: "Refundable CTC", kind: "program", }), ).toBe("Refundable CTC"); @@ -72,6 +72,14 @@ describe("calibration explorer presentation model", () => { kind: "dimension_value", }), ).toBe("Traditional IRA deduction"); + expect( + explorerNodeLabel({ + id: "income_tax", + label: "Income TAX (GBP) — authored", + kind: "dimension_value", + authored_label: true, + }), + ).toBe("Income TAX (GBP) — authored"); expect( explorerNodeLabel({ id: "target-1", @@ -246,6 +254,80 @@ describe("calibration explorer presentation model", () => { }); }); + test("prefers artifact source and category labels in breadcrumbs", () => { + expect( + explorerBreadcrumbs( + state({ + source: "obr", + program: "efo_receipts", + geography: "United Kingdom — authored", + dimensions: [], + }), + { + source: "Office for Budget Responsibility", + program: "EFO receipts", + geography: "United Kingdom — authored", + }, + ), + ).toEqual([ + { label: "All targets", path: { dimensions: [] } }, + { + label: "Office for Budget Responsibility", + path: { dimensions: [] }, + }, + { + label: "EFO receipts", + path: { + source: "obr", + program: "efo_receipts", + dimensions: [], + }, + }, + { + label: "United Kingdom — authored", + path: { + source: "obr", + program: "efo_receipts", + geography: "United Kingdom — authored", + dimensions: [], + }, + }, + ]); + }); + + test("uses schema 8 value labels instead of formatting navigation ids", () => { + const current = state({ + source: "obr", + program: "obr.efo_receipts", + geography: "United Kingdom", + dimensions: [ + { + key: "obr.efo_line", + label: "Economic and fiscal outlook line", + value: "income_tax", + }, + ], + }); + + expect( + explorerBreadcrumbs(current, { + source: "Office for Budget Responsibility", + program: "Economic and fiscal outlook receipts", + dimensions: ["Income tax (gross of tax credits)"], + }).at(-1)?.label, + ).toBe("Income tax (gross of tax credits)"); + }); + + test("preserves an artifact category label on a program tile", () => { + expect( + explorerNodeLabel({ + id: "efo_receipts", + kind: "program", + label: "EFO receipts", + }), + ).toBe("EFO receipts"); + }); + test("distinguishes an empty filtered result from an invalid hierarchy scope", () => { expect(explorerEmptyMessage(state({ dimensions: [] }, true))).toContain( "filters", diff --git a/frontend/components/microcosm/calibration-explorer-view.ts b/frontend/components/microcosm/calibration-explorer-view.ts index 7d41ee7..9ec585b 100644 --- a/frontend/components/microcosm/calibration-explorer-view.ts +++ b/frontend/components/microcosm/calibration-explorer-view.ts @@ -53,26 +53,34 @@ export interface ExplorerBreadcrumb { path: ExplorerPath; } -export function explorerBreadcrumbs(state: ExplorerState): ExplorerBreadcrumb[] { +export function explorerBreadcrumbs( + state: ExplorerState, + pathLabels: { + source?: string; + program?: string; + geography?: string; + dimensions?: string[]; + } = {}, +): ExplorerBreadcrumb[] { const crumbs: ExplorerBreadcrumb[] = [ { label: "All targets", path: { dimensions: [] } }, ]; if (state.breakdown === "geography" && state.path.geography) { crumbs.push({ - label: geographyLabel(state.path.geography), + label: pathLabels.geography ?? geographyLabel(state.path.geography), path: { geography: state.path.geography, dimensions: [] }, }); } if (state.path.source && state.path.program) { crumbs.push({ - label: sourceLabel(state.path.source), + label: pathLabels.source ?? sourceLabel(state.path.source), path: state.breakdown === "geography" && state.path.geography ? { geography: state.path.geography, dimensions: [] } : { dimensions: [] }, }); crumbs.push({ - label: programLabel(state.path.program), + label: pathLabels.program ?? programLabel(state.path.program), path: { source: state.path.source, program: state.path.program, @@ -90,7 +98,7 @@ export function explorerBreadcrumbs(state: ExplorerState): ExplorerBreadcrumb[] state.path.geography ) { crumbs.push({ - label: geographyLabel(state.path.geography), + label: pathLabels.geography ?? geographyLabel(state.path.geography), path: { source: state.path.source, program: state.path.program, @@ -102,7 +110,7 @@ export function explorerBreadcrumbs(state: ExplorerState): ExplorerBreadcrumb[] if (state.path.source && state.path.program) { state.path.dimensions.forEach((dimension, index) => { crumbs.push({ - label: canonicalLabel(dimension.value), + label: pathLabels.dimensions?.[index] ?? canonicalLabel(dimension.value), path: { source: state.path.source, program: state.path.program, @@ -181,9 +189,10 @@ export function explorerMapHeight(pageIntroHeight: number): string { } export function explorerNodeLabel( - node: Pick, + node: Pick, ): string { - if (node.kind === "program") return programLabel(node.id); + if (node.kind === "program") return node.label || programLabel(node.id); + if (node.kind === "dimension_value" && node.authored_label) return node.label; if (node.kind === "dimension_value") return canonicalLabel(node.label); return node.label; } diff --git a/frontend/components/microcosm/staging-target-change-map.tsx b/frontend/components/microcosm/staging-target-change-map.tsx index a6e8bee..b9eaef9 100644 --- a/frontend/components/microcosm/staging-target-change-map.tsx +++ b/frontend/components/microcosm/staging-target-change-map.tsx @@ -442,7 +442,7 @@ export function StagingTargetChangeMap({ } const directions = layoutDirections(data, size.width, size.height, expanded); - const breadcrumbs = explorerBreadcrumbs(state); + const breadcrumbs = explorerBreadcrumbs(state, data.pathLabels); const upLabel = expanded ? `Up to ${data.currentLevel.label.toLowerCase()}` : explorerUpLabel(state); return ( diff --git a/frontend/lib/api/hooks/use-microcosm.ts b/frontend/lib/api/hooks/use-microcosm.ts index faae5e7..fbbba8e 100644 --- a/frontend/lib/api/hooks/use-microcosm.ts +++ b/frontend/lib/api/hooks/use-microcosm.ts @@ -57,6 +57,7 @@ export interface MicrocosmTargetRow { source_url?: string | null; variable?: string | null; variable_label?: string | null; + target_label?: string | null; measure?: string | null; target_role?: string | null; source_measure_id?: string | null; @@ -79,12 +80,18 @@ export interface MicrocosmTargetRow { key: string; label: string; value: string; + value_id?: string; source_key?: string; raw_value?: string; rank?: number; }[] | null; - dimension_adapter?: "structured" | "legacy_filter" | "legacy_name" | null; - target_representation?: "legacy" | "structured" | null; + dimension_adapter?: + | "hierarchy" + | "structured" + | "legacy_filter" + | "legacy_name" + | null; + target_representation?: "legacy" | "structured" | "hierarchy" | null; variable_key?: string | null; // schema v2 published registry metadata (null on v1). source_citation?: string | null; @@ -212,7 +219,12 @@ export interface MicrocosmArtifactPresentation { export interface MicrocosmTargetSchema { diagnostics_schema_version: number | null; structured_dimensions: boolean; - target_representation: "legacy" | "structured" | "mixed" | "unknown"; + target_representation: + | "legacy" + | "structured" + | "hierarchy" + | "mixed" + | "unknown"; } export interface MicrocosmCalibration { diff --git a/frontend/lib/microcosm/calibration-prefetch.test.ts b/frontend/lib/microcosm/calibration-prefetch.test.ts index bc2f34a..5a15233 100644 --- a/frontend/lib/microcosm/calibration-prefetch.test.ts +++ b/frontend/lib/microcosm/calibration-prefetch.test.ts @@ -28,6 +28,7 @@ function response( return { lossAttributionAvailable: false, path: state.path, + pathLabels: {}, currentLevel: { kind: "overview", label: "Test" }, groups: [{ id: "test", label: "Test", nodes, metrics: EMPTY_METRICS }], dimensionOrder: [], diff --git a/frontend/lib/microcosm/calibration-tree.test.ts b/frontend/lib/microcosm/calibration-tree.test.ts index e2e26d9..9a397d7 100644 --- a/frontend/lib/microcosm/calibration-tree.test.ts +++ b/frontend/lib/microcosm/calibration-tree.test.ts @@ -234,6 +234,31 @@ describe("source, geography, and declared-dimension hierarchy", () => { expect(tree.groups[0].label).toBe("Nova Statistics Agency"); }); + test("retains artifact source and category labels throughout the selected path", () => { + const tree = buildCalibrationTree( + [ + target("obr-income-tax", { + source: "obr", + source_label: "Office for Budget Responsibility", + variable: "efo_receipts", + variable_label: "EFO receipts", + geography: "United Kingdom", + }), + ], + state({ + source: "obr", + program: "efo_receipts", + dimensions: [], + }), + ); + + expect(tree.pathLabels).toEqual({ + source: "Office for Budget Responsibility", + program: "EFO receipts", + }); + expect(tree.groups[0].label).toBe("EFO receipts"); + }); + test("renders geography first, then programs grouped by their source", () => { const overview = buildCalibrationTree( rows, @@ -566,6 +591,119 @@ describe("source, geography, and declared-dimension hierarchy", () => { }); }); +describe("schema 8 ordered hierarchy", () => { + const hierarchyRows: CalibrationTreeTarget[] = [ + target("target-adult@2025", { + source: "provider-a", + source_label: "Provider A — authored", + variable: "provider-a.category", + variable_label: "Category A — authored", + geography: "United Kingdom — authored", + level: "country", + target_label: "Adult target — authored", + target_representation: "hierarchy", + target_dimensions: [ + { + key: "dimension.age", + label: "Age — authored", + value: "Adults — authored", + value_id: "adult", + }, + { + key: "dimension.sex", + label: "Sex — authored", + value: "Female — authored", + value_id: "female", + }, + ], + }), + target("target-child@2025", { + source: "provider-a", + source_label: "Provider A — authored", + variable: "provider-a.category", + variable_label: "Category A — authored", + geography: "United Kingdom — authored", + level: "country", + target_label: "Child target — authored", + target_representation: "hierarchy", + target_dimensions: [ + { + key: "dimension.age", + label: "Age — authored", + value: "Children — authored", + value_id: "child", + }, + { + key: "dimension.sex", + label: "Sex — authored", + value: "Female — authored", + value_id: "female", + }, + ], + }), + target("target-total@2025", { + source: "provider-a", + source_label: "Provider A — authored", + variable: "provider-a.category", + variable_label: "Category A — authored", + geography: "United Kingdom — authored", + level: "country", + target_label: "Total target — authored", + target_representation: "hierarchy", + target_dimensions: [], + }), + ]; + + const hierarchyPath = { + source: "provider-a", + program: "provider-a.category", + geography: "United Kingdom — authored", + dimensions: [], + }; + + test("uses producer order for ragged paths and keeps zero-dimension targets", () => { + const tree = buildCalibrationTree(hierarchyRows, state(hierarchyPath)); + + expect(tree.currentLevel).toEqual({ + kind: "mixed", + label: "Breakdowns and targets", + }); + expect(tree.dimensionOrder).toEqual([ + { key: "dimension.age", label: "Age — authored" }, + { key: "dimension.sex", label: "Sex — authored" }, + ]); + expect(tree.groups.map((group) => group.id)).toEqual([ + "dimension.age", + "targets", + ]); + expect(tree.groups[0].nodes.map((node) => [node.id, node.label])).toEqual([ + ["adult", "Adults — authored"], + ["child", "Children — authored"], + ]); + expect(tree.groups[1].nodes[0].label).toBe("Total target — authored"); + }); + + test("uses stable value ids while preserving labels and collapsing one choice", () => { + const tree = buildCalibrationTree( + hierarchyRows, + state({ + ...hierarchyPath, + dimensions: [ + { key: "dimension.age", label: "Age — authored", value: "adult" }, + ], + }), + ); + + expect(tree.currentLevel).toEqual({ kind: "target", label: "Targets" }); + expect(tree.pathLabels.geography).toBe("United Kingdom — authored"); + expect(tree.pathLabels.dimensions).toEqual(["Adults — authored"]); + expect(tree.groups[0].nodes.map((node) => node.label)).toEqual([ + "Adult target — authored", + ]); + expect(tree.groups[0].nodes[0].target?.target_dimensions).toHaveLength(2); + }); +}); + describe("calibration tree metrics and filters", () => { test("classifies targets without generated model data as skipped", () => { const missingModelData = target("missing-model-data", { diff --git a/frontend/lib/microcosm/calibration-tree.ts b/frontend/lib/microcosm/calibration-tree.ts index cc64c83..0476532 100644 --- a/frontend/lib/microcosm/calibration-tree.ts +++ b/frontend/lib/microcosm/calibration-tree.ts @@ -14,8 +14,10 @@ export interface CalibrationTreeDimension { key: string; label: string; value: string; + value_id?: string; source_key?: string; raw_value?: string; + rank?: number; } export interface CalibrationTreeTarget { @@ -29,6 +31,8 @@ export interface CalibrationTreeTarget { variable_key?: string | null; measure?: string | null; source_measure_id?: string | null; + target_label?: string | null; + target_representation?: string | null; level?: string | null; geography?: string | null; abs_relative_error?: number | null; @@ -72,6 +76,7 @@ export interface CalibrationTreeNode { selection: ExplorerNodeSelection; metrics: CalibrationTreeMetrics; target?: CalibrationTreeTarget; + authored_label?: boolean; } export interface CalibrationTreeGroup { @@ -85,6 +90,12 @@ export interface CalibrationTreeResponse { releaseId?: string; lossAttributionAvailable: boolean; path: ExplorerState["path"]; + pathLabels: { + source?: string; + program?: string; + geography?: string; + dimensions?: string[]; + }; currentLevel: | { kind: "overview"; label: string } | { kind: "geography"; label: string } @@ -313,6 +324,9 @@ function geographyId(row: CalibrationTreeTarget): string { } function targetLabel(row: CalibrationTreeTarget): string { + if (row.target_representation === "hierarchy") { + return String(row.target_label); + } const sourceMeasureId = String(row.source_measure_id ?? "").trim(); if (sourceMeasureId) return humanize(sourceMeasureId); const id = String(row.name ?? row.base_name ?? "Target"); @@ -325,6 +339,125 @@ interface DimensionPartition { rows: CalibrationTreeTarget[]; } +interface HierarchyEntry { + row: CalibrationTreeTarget; + cursor: number; +} + +interface HierarchyDimensionPartition { + dimension: BreakdownDimension; + entries: HierarchyEntry[]; +} + +function hierarchyDimension( + entry: HierarchyEntry, +): CalibrationTreeDimension | null { + return entry.row.target_dimensions?.[entry.cursor] ?? null; +} + +function hierarchyDimensionValueId( + dimension: CalibrationTreeDimension, +): string { + return String( + dimension.value_id ?? dimension.raw_value ?? dimension.value, + ).trim(); +} + +function mergeHierarchyPartitions( + partitions: HierarchyDimensionPartition[], +): HierarchyDimensionPartition[] { + const merged = new Map(); + for (const partition of partitions) { + const existing = merged.get(partition.dimension.key); + if (existing) { + if (existing.dimension.label !== partition.dimension.label) { + throw new Error( + `Schema 8 dimension ${partition.dimension.key} has inconsistent labels.`, + ); + } + existing.entries.push(...partition.entries); + } else { + merged.set(partition.dimension.key, { + dimension: partition.dimension, + entries: [...partition.entries], + }); + } + } + return [...merged.values()]; +} + +function hierarchyPartitions(entries: HierarchyEntry[]): { + dimensions: HierarchyDimensionPartition[]; + targets: HierarchyEntry[]; +} { + const targets: HierarchyEntry[] = []; + const grouped = new Map(); + for (const entry of entries) { + const dimension = hierarchyDimension(entry); + if (!dimension) { + targets.push(entry); + continue; + } + const existing = grouped.get(dimension.key); + if (existing) { + if (existing.dimension.label !== dimension.label) { + throw new Error( + `Schema 8 dimension ${dimension.key} has inconsistent labels.`, + ); + } + existing.entries.push(entry); + } else { + grouped.set(dimension.key, { + dimension: { key: dimension.key, label: dimension.label }, + entries: [entry], + }); + } + } + + const visible: HierarchyDimensionPartition[] = []; + for (const partition of grouped.values()) { + const values = new Set( + partition.entries.map((entry) => + hierarchyDimensionValueId(hierarchyDimension(entry)!), + ), + ); + if (values.size > 1) { + visible.push(partition); + continue; + } + const nested = hierarchyPartitions( + partition.entries.map((entry) => ({ ...entry, cursor: entry.cursor + 1 })), + ); + visible.push(...nested.dimensions); + targets.push(...nested.targets); + } + return { + dimensions: mergeHierarchyPartitions(visible), + targets, + }; +} + +function hierarchyDimensionOrder( + rows: CalibrationTreeTarget[], +): BreakdownDimension[] { + const order = new Map(); + for (const row of rows) { + for (const dimension of row.target_dimensions ?? []) { + const existing = order.get(dimension.key); + if (existing && existing.label !== dimension.label) { + throw new Error( + `Schema 8 dimension ${dimension.key} has inconsistent labels.`, + ); + } + order.set(dimension.key, existing ?? { + key: dimension.key, + label: dimension.label, + }); + } + } + return [...order.values()]; +} + function partitionRowsByDimension( rows: CalibrationTreeTarget[], selectedKeys: Set, @@ -402,8 +535,17 @@ function node( selection: ExplorerNodeSelection, rows: CalibrationTreeTarget[], target?: CalibrationTreeTarget, + authoredLabel = false, ): CalibrationTreeNode { - return { id, label, kind, selection, metrics: calibrationTreeMetrics(rows), target }; + return { + id, + label, + kind, + selection, + metrics: calibrationTreeMetrics(rows), + target, + authored_label: authoredLabel || undefined, + }; } function sortNodes(nodes: CalibrationTreeNode[]): CalibrationTreeNode[] { @@ -484,6 +626,50 @@ function programGroups(rows: CalibrationTreeTarget[]): CalibrationTreeGroup[] { ); } +function pathLabels( + rows: CalibrationTreeTarget[], + path: ExplorerState["path"], +): CalibrationTreeResponse["pathLabels"] { + const labels: CalibrationTreeResponse["pathLabels"] = {}; + if (path.geography) { + const geographyLabels = [ + ...new Set( + rows + .filter( + (row) => + row.target_representation === "hierarchy" && + geographyId(row) === path.geography, + ) + .map((row) => row.geography?.trim()) + .filter((label): label is string => Boolean(label)), + ), + ]; + if (geographyLabels.length === 1) labels.geography = geographyLabels[0]; + } + if (!path.source) return labels; + const sourceRows = rows.filter( + (row) => String(row.source ?? "other") === path.source, + ); + const artifactSourceLabel = sourceRows + .map((row) => row.source_label?.trim()) + .find((label): label is string => Boolean(label)); + labels.source = artifactSourceLabel ?? sourceLabel(path.source); + if (!path.program) return labels; + const programRows = sourceRows.filter((row) => programId(row) === path.program); + const artifactProgramLabels = [ + ...new Set( + programRows + .map((row) => row.variable_label?.trim()) + .filter((label): label is string => Boolean(label)), + ), + ]; + labels.program = programLabel( + path.program, + artifactProgramLabels.length === 1 ? artifactProgramLabels[0] : null, + ); + return labels; +} + function geographyNodes(rows: CalibrationTreeTarget[]): CalibrationTreeNode[] { const byGeography = groupRows(rows, geographyId); return sortNodes( @@ -499,6 +685,151 @@ function geographyNodes(rows: CalibrationTreeTarget[]): CalibrationTreeNode[] { ); } +function hierarchyDimensionTree( + geographyRows: CalibrationTreeTarget[], + state: ExplorerState, + releaseId: string | undefined, + lossAttributionAvailable: boolean, + selectedPathLabels: CalibrationTreeResponse["pathLabels"], + options: CalibrationTreeResponse["filterOptions"], +): CalibrationTreeResponse { + let entries = geographyRows.map((row) => ({ row, cursor: 0 })); + const selectedValueLabels: string[] = []; + for (const selection of state.path.dimensions) { + const partition = hierarchyPartitions(entries).dimensions.find( + ({ dimension }) => dimension.key === selection.key, + ); + if (!partition) { + entries = []; + break; + } + const matching = partition.entries.filter((entry) => { + const dimension = hierarchyDimension(entry); + return ( + dimension != null && + hierarchyDimensionValueId(dimension) === selection.value + ); + }); + const valueLabels = new Set( + matching.map((entry) => hierarchyDimension(entry)!.value), + ); + if (valueLabels.size > 1) { + throw new Error( + `Schema 8 dimension value ${selection.key}/${selection.value} has inconsistent labels.`, + ); + } + selectedValueLabels.push([...valueLabels][0] ?? selection.value); + entries = matching.map((entry) => ({ + ...entry, + cursor: entry.cursor + 1, + })); + } + + const scopedRows = entries.map((entry) => entry.row); + const filteredScopedRows = applyExplorerFilters(scopedRows, state.filters); + const visibleRows = new Set(filteredScopedRows); + const partition = hierarchyPartitions(entries); + const groups: CalibrationTreeGroup[] = partition.dimensions.flatMap( + ({ dimension, entries: dimensionEntries }) => { + const visibleEntries = dimensionEntries.filter((entry) => + visibleRows.has(entry.row), + ); + if (!visibleEntries.length) return []; + const byValue = new Map< + string, + { label: string; rows: CalibrationTreeTarget[] } + >(); + for (const entry of visibleEntries) { + const value = hierarchyDimension(entry)!; + const valueId = hierarchyDimensionValueId(value); + const existing = byValue.get(valueId); + if (existing && existing.label !== value.value) { + throw new Error( + `Schema 8 dimension value ${dimension.key}/${valueId} has inconsistent labels.`, + ); + } + if (existing) existing.rows.push(entry.row); + else byValue.set(valueId, { label: value.value, rows: [entry.row] }); + } + const nodes = sortNodes( + [...byValue.entries()].map(([valueId, value]) => + node( + valueId, + value.label, + "dimension_value", + { + kind: "dimension_value", + key: dimension.key, + label: dimension.label, + value: valueId, + }, + value.rows, + undefined, + true, + ), + ), + ); + return [{ + id: dimension.key, + label: dimension.label, + nodes, + metrics: calibrationTreeMetrics(visibleEntries.map((entry) => entry.row)), + }]; + }, + ); + + const visibleTargets = partition.targets + .map((entry) => entry.row) + .filter((row) => visibleRows.has(row)); + const targetNodes = visibleTargets + .map((row, index) => { + const id = String( + row.comparison_id ?? row.name ?? row.base_name ?? `target-${index}`, + ); + return node( + id, + targetLabel(row), + "target", + { kind: "target", value: id }, + [row], + row, + ); + }) + .sort( + (left, right) => + left.label.localeCompare(right.label) || left.id.localeCompare(right.id), + ); + if (targetNodes.length) { + groups.push({ + id: "targets", + label: "Targets", + nodes: targetNodes, + metrics: calibrationTreeMetrics(visibleTargets), + }); + } + + const currentLevel = + partition.dimensions.length === 1 && partition.targets.length === 0 + ? { kind: "dimension" as const, ...partition.dimensions[0].dimension } + : partition.dimensions.length === 0 + ? { kind: "target" as const, label: "Targets" } + : { kind: "mixed" as const, label: "Breakdowns and targets" }; + return { + releaseId, + lossAttributionAvailable, + path: state.path, + pathLabels: { + ...selectedPathLabels, + dimensions: selectedValueLabels, + }, + currentLevel, + groups, + dimensionOrder: hierarchyDimensionOrder(geographyRows), + filterOptions: options, + filteredMetrics: calibrationTreeMetrics(filteredScopedRows), + }; +} + export function buildCalibrationTree( allRows: CalibrationTreeTarget[], state: ExplorerState, @@ -508,6 +839,7 @@ export function buildCalibrationTree( ): CalibrationTreeResponse { const chartRows = allRows.map(normalizeChartCalibrationStatus); const { path } = state; + const selectedPathLabels = pathLabels(chartRows, path); const options = filterOptions(chartRows); const filteredRows = applyExplorerFilters(chartRows, state.filters); @@ -517,6 +849,7 @@ export function buildCalibrationTree( releaseId, lossAttributionAvailable, path, + pathLabels: selectedPathLabels, currentLevel: { kind: "geography", label: "Geography" }, groups: [{ id: "geography", @@ -542,6 +875,7 @@ export function buildCalibrationTree( releaseId, lossAttributionAvailable, path, + pathLabels: selectedPathLabels, currentLevel: { kind: "overview", label: "Programs" }, groups: programGroups(filteredGeographyRows), dimensionOrder: [], @@ -555,6 +889,7 @@ export function buildCalibrationTree( releaseId, lossAttributionAvailable, path, + pathLabels: selectedPathLabels, currentLevel: { kind: "overview", label: "Programs" }, groups: programGroups(filteredRows), dimensionOrder: [], @@ -575,10 +910,11 @@ export function buildCalibrationTree( releaseId, lossAttributionAvailable, path, + pathLabels: selectedPathLabels, currentLevel: { kind: "geography", label: "Geography" }, groups: [{ id: path.program, - label: path.program, + label: selectedPathLabels.program ?? programLabel(path.program), nodes, metrics: calibrationTreeMetrics(filteredProgramRows), }], @@ -591,6 +927,19 @@ export function buildCalibrationTree( const geographyRows = path.geography ? programRows.filter((row) => geographyId(row) === path.geography) : programRows; + if ( + geographyRows.length > 0 && + geographyRows.every((row) => row.target_representation === "hierarchy") + ) { + return hierarchyDimensionTree( + geographyRows, + state, + releaseId, + lossAttributionAvailable, + selectedPathLabels, + options, + ); + } const dimensionOrder = orderedBreakdownDimensions(geographyRows); let scopedRows = geographyRows; const selectedKeys = new Set(); @@ -677,6 +1026,7 @@ export function buildCalibrationTree( releaseId, lossAttributionAvailable, path, + pathLabels: selectedPathLabels, currentLevel, groups, dimensionOrder, diff --git a/frontend/lib/microcosm/hierarchy-target-reader.test.ts b/frontend/lib/microcosm/hierarchy-target-reader.test.ts new file mode 100644 index 0000000..12b37f2 --- /dev/null +++ b/frontend/lib/microcosm/hierarchy-target-reader.test.ts @@ -0,0 +1,111 @@ +import { describe, expect, test } from "bun:test"; + +import { + readHierarchyTarget, + validateHierarchyTargets, +} from "./hierarchy-target-reader"; + +function row() { + return { + name: "obr.income_tax@2025", + target_name: "obr.income_tax", + hierarchy: { + provider: { id: "obr", label: "Office for Budget Responsibility" }, + category: { + id: "obr.efo_receipts", + label: "Economic and fiscal outlook receipts", + provider_id: "obr", + }, + geography: { + id: "K02000001", + label: "United Kingdom", + level: "country", + }, + dimensions: [ + { + id: "obr.efo_line", + label: "Economic and fiscal outlook line", + value_id: "income_tax", + value_label: "Income tax (gross of tax credits)", + }, + ], + target: { id: "obr.income_tax", label: "Income tax receipts" }, + }, + }; +} + +describe("schema 8 hierarchy target reader", () => { + test("preserves every producer identifier and label", () => { + expect(readHierarchyTarget(row())).toEqual({ + source: "obr", + sourceLabel: "Office for Budget Responsibility", + variable: "obr.efo_receipts", + variableLabel: "Economic and fiscal outlook receipts", + geography: "United Kingdom", + geographyId: "K02000001", + level: "country", + dimensions: [ + { + key: "obr.efo_line", + label: "Economic and fiscal outlook line", + value: "Income tax (gross of tax credits)", + value_id: "income_tax", + source_key: "obr.efo_line", + raw_value: "income_tax", + rank: 0, + }, + ], + targetId: "obr.income_tax", + targetLabel: "Income tax receipts", + breakdown: "Income tax (gross of tax credits)", + }); + }); + + test("rejects missing labels and inconsistent relationships", () => { + expect(() => + readHierarchyTarget({ + ...row(), + hierarchy: { ...row().hierarchy, provider: { id: "obr" } }, + }), + ).toThrow("hierarchy.provider.label"); + expect(() => + readHierarchyTarget({ + ...row(), + hierarchy: { + ...row().hierarchy, + category: { ...row().hierarchy.category, provider_id: "hmrc" }, + }, + }), + ).toThrow("expected obr"); + expect(() => + readHierarchyTarget({ + ...row(), + hierarchy: { ...row().hierarchy, dimensions: {} }, + }), + ).toThrow("dimensions must be an array"); + }); + + test("rejects conflicting labels for repeated ids across a file", () => { + expect(() => + validateHierarchyTargets([ + row(), + { + ...row(), + hierarchy: { + ...row().hierarchy, + provider: { id: "obr", label: "Conflicting OBR label" }, + }, + }, + ]), + ).toThrow("provider obr has inconsistent labels"); + }); + + test("rejects a category reused under a different provider", () => { + const second = row(); + second.hierarchy.provider = { id: "hmrc", label: "HMRC" }; + second.hierarchy.category.provider_id = "hmrc"; + expect(() => validateHierarchyTargets([row(), second])).toThrow( + "category obr.efo_receipts references inconsistent providers", + ); + }); +}); diff --git a/frontend/lib/microcosm/hierarchy-target-reader.ts b/frontend/lib/microcosm/hierarchy-target-reader.ts new file mode 100644 index 0000000..a89780a --- /dev/null +++ b/frontend/lib/microcosm/hierarchy-target-reader.ts @@ -0,0 +1,220 @@ +type JsonObject = Record; + +export interface HierarchyTargetDimension { + key: string; + label: string; + value: string; + value_id: string; + source_key: string; + raw_value: string; + rank: number; +} + +export interface HierarchyTargetIdentity { + source: string; + sourceLabel: string; + variable: string; + variableLabel: string; + geography: string; + geographyId: string; + level: string; + dimensions: HierarchyTargetDimension[]; + targetId: string; + targetLabel: string; + breakdown: string; +} + +function isPlainObject(value: unknown): value is JsonObject { + if (!value || typeof value !== "object" || Array.isArray(value)) return false; + const prototype = Object.getPrototypeOf(value); + return prototype === Object.prototype || prototype === null; +} + +function requiredObject(value: unknown, path: string): JsonObject { + if (!isPlainObject(value)) { + throw new Error(`Schema 8 target ${path} must be an object.`); + } + return value; +} + +function requiredString(value: unknown, path: string): string { + if (typeof value !== "string" || !value.trim()) { + throw new Error(`Schema 8 target ${path} must be a non-empty string.`); + } + return value.trim(); +} + +function node(value: unknown, path: string): { id: string; label: string } { + const object = requiredObject(value, path); + return { + id: requiredString(object.id, `${path}.id`), + label: requiredString(object.label, `${path}.label`), + }; +} + +/** Read the complete, producer-authored hierarchy on one schema-8 row. */ +export function readHierarchyTarget(row: JsonObject): HierarchyTargetIdentity { + const hierarchy = requiredObject(row.hierarchy, "hierarchy"); + const provider = node(hierarchy.provider, "hierarchy.provider"); + const categoryObject = requiredObject( + hierarchy.category, + "hierarchy.category", + ); + const category = node(categoryObject, "hierarchy.category"); + const categoryProviderId = requiredString( + categoryObject.provider_id, + "hierarchy.category.provider_id", + ); + if (categoryProviderId !== provider.id) { + throw new Error( + `Schema 8 category ${category.id} references provider ` + + `${categoryProviderId}, expected ${provider.id}.`, + ); + } + + const geographyObject = requiredObject( + hierarchy.geography, + "hierarchy.geography", + ); + const geography = node(geographyObject, "hierarchy.geography"); + const geographyLevel = requiredString( + geographyObject.level, + "hierarchy.geography.level", + ); + + if (!Array.isArray(hierarchy.dimensions)) { + throw new Error("Schema 8 target hierarchy.dimensions must be an array."); + } + const dimensionIds = new Set(); + const dimensions = hierarchy.dimensions.map((value, rank) => { + const object = requiredObject(value, `hierarchy.dimensions[${rank}]`); + const id = requiredString(object.id, `hierarchy.dimensions[${rank}].id`); + if (dimensionIds.has(id)) { + throw new Error(`Schema 8 target contains duplicate dimension id ${id}.`); + } + dimensionIds.add(id); + const valueId = requiredString( + object.value_id, + `hierarchy.dimensions[${rank}].value_id`, + ); + return { + key: id, + label: requiredString( + object.label, + `hierarchy.dimensions[${rank}].label`, + ), + value: requiredString( + object.value_label, + `hierarchy.dimensions[${rank}].value_label`, + ), + value_id: valueId, + source_key: id, + raw_value: valueId, + rank, + }; + }); + + const target = node(hierarchy.target, "hierarchy.target"); + const rowTargetId = + typeof row.target_name === "string" && row.target_name.trim() + ? row.target_name.trim() + : typeof row.name === "string" + ? row.name.replace(/@[^@]+$/, "").trim() + : ""; + if (rowTargetId && rowTargetId !== target.id) { + throw new Error( + `Schema 8 hierarchy target id ${target.id} does not match row target ` + + `${rowTargetId}.`, + ); + } + + return { + source: provider.id, + sourceLabel: provider.label, + variable: category.id, + variableLabel: category.label, + geography: geography.label, + geographyId: geography.id, + level: geographyLevel, + dimensions, + targetId: target.id, + targetLabel: target.label, + breakdown: dimensions.map((dimension) => dimension.value).join(" · "), + }; +} + +function requireConsistentLabel( + labels: Map, + id: string, + label: string, + kind: string, +): void { + const existing = labels.get(id); + if (existing != null && existing !== label) { + throw new Error( + `Schema 8 ${kind} ${id} has inconsistent labels: ` + + `${existing} and ${label}.`, + ); + } + labels.set(id, label); +} + +/** Require stable labels for every repeated hierarchy identifier in a file. */ +export function validateHierarchyTargets(rows: JsonObject[]): void { + const providers = new Map(); + const categories = new Map(); + const categoryProviders = new Map(); + const geographies = new Map(); + const dimensions = new Map(); + const dimensionValues = new Map(); + const targets = new Map(); + for (const row of rows) { + const identity = readHierarchyTarget(row); + requireConsistentLabel( + providers, + identity.source, + identity.sourceLabel, + "provider", + ); + requireConsistentLabel( + categories, + identity.variable, + identity.variableLabel, + "category", + ); + const categoryProvider = categoryProviders.get(identity.variable); + if (categoryProvider != null && categoryProvider !== identity.source) { + throw new Error( + `Schema 8 category ${identity.variable} references inconsistent ` + + `providers: ${categoryProvider} and ${identity.source}.`, + ); + } + categoryProviders.set(identity.variable, identity.source); + requireConsistentLabel( + geographies, + `${identity.level}\0${identity.geographyId}`, + identity.geography, + "geography", + ); + for (const dimension of identity.dimensions) { + requireConsistentLabel( + dimensions, + dimension.key, + dimension.label, + "dimension", + ); + requireConsistentLabel( + dimensionValues, + `${dimension.key}\0${dimension.value_id}`, + dimension.value, + "dimension value", + ); + } + requireConsistentLabel( + targets, + identity.targetId, + identity.targetLabel, + "target", + ); + } +} diff --git a/frontend/lib/microcosm/latest-artifact.test.ts b/frontend/lib/microcosm/latest-artifact.test.ts index 6b82cd2..e311b09 100644 --- a/frontend/lib/microcosm/latest-artifact.test.ts +++ b/frontend/lib/microcosm/latest-artifact.test.ts @@ -1,4 +1,7 @@ import { expect, test } from "bun:test"; +import { mkdtemp, rm, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; import beDiagnosticsFixture from "./fixtures/be-release/calibration_diagnostics.json"; import beReleaseManifestFixture from "./fixtures/be-release/release_manifest.json"; @@ -20,6 +23,9 @@ import { latestMicrocosmCalibrationHighlights, latestMicrocosmCalibrationSummary, latestMicrocosmTargetDiagnosticsPage, + loadPointerReleaseId, + loadRelease, + loadReleases, microcosmRepo, microcosmRevision, microcosmTargetTreemap, @@ -29,6 +35,7 @@ import { releasePublisherLabels, releasePublishedAtFromTree, releaseRole, + UK_LOCAL_CALIBRATION_DIR_ENV, type ArtifactCountry, type Calibration, } from "./latest-artifact"; @@ -74,12 +81,74 @@ test("uses the private Belgium repository and country revision", () => { ); }); +test("loads an explicit UK local schema 8 artifact without a network request", async () => { + const directory = await mkdtemp(join(tmpdir(), "microcosm-schema-8-")); + const prior = process.env[UK_LOCAL_CALIBRATION_DIR_ENV]; + process.env[UK_LOCAL_CALIBRATION_DIR_ENV] = directory; + try { + await Promise.all([ + writeFile( + join(directory, "calibration_diagnostics.json"), + JSON.stringify({ + schema_version: 8, + release_id: "local-schema-8", + targets: [ + { + name: "provider.target@2025", + target_name: "provider.target", + source: "Citation", + target: 100, + initial_estimate: 90, + final_estimate: 100, + hierarchy: { + provider: { id: "provider", label: "Provider label" }, + category: { + id: "provider.category", + label: "Category label", + provider_id: "provider", + }, + geography: { + id: "K02000001", + label: "United Kingdom", + level: "country", + }, + dimensions: [], + target: { id: "provider.target", label: "Target label" }, + }, + }, + ], + }), + ), + writeFile( + join(directory, "release_manifest.json"), + JSON.stringify({ country: { code: "uk" } }), + ), + ]); + + const pointer = await loadPointerReleaseId(0, "uk"); + const releases = await loadReleases(0, "uk"); + const calibration = await loadRelease("latest", 0, "uk"); + + expect(pointer.release_id).toBe("local-schema-8"); + expect(releases.map((release) => release.release_id)).toEqual([ + "local-schema-8", + ]); + expect(calibration.source).toBe("local_filesystem"); + expect(calibration.target_schema.target_representation).toBe("hierarchy"); + expect(calibration.rows[0].target_label).toBe("Target label"); + } finally { + if (prior == null) delete process.env[UK_LOCAL_CALIBRATION_DIR_ENV]; + else process.env[UK_LOCAL_CALIBRATION_DIR_ENV] = prior; + await rm(directory, { recursive: true, force: true }); + } +}); + test("loads trimmed Belgium diagnostics without optional US artifact fields", () => { expect("loss_trajectory" in beDiagnosticsFixture).toBe(false); expect("past_cap_census" in beDiagnosticsFixture).toBe(false); expect(beDiagnosticsFixture.targets.every((row) => !("registry" in row))).toBe(true); const cal = buildCalibration( - beDiagnosticsFixture, + { ...beDiagnosticsFixture, schema_version: 2 }, "microcosm-be-2026-chronicle-3cef97b-20260823T134247Z", null, {}, @@ -136,6 +205,7 @@ test("derives Belgium population region, sex, and age-band browser facets", () = const cal = buildCalibration( { ...beDiagnosticsFixture, + schema_version: 2, targets: [ target("statbel_population_be1_male_0_17"), target("statbel_population_be2_female_18_64"), @@ -191,6 +261,125 @@ test("diagnosticsDimensions drops malformed entries and normalizes optional meta }); }); +test("schema 8 preserves the complete producer-authored hierarchy", () => { + const cal = buildCalibration( + { + schema_version: 8, + targets: [ + { + name: "obr.income_tax@2025", + target_name: "obr.income_tax", + source: "OBR synthetic citation", + target: 100, + initial_estimate: 90, + final_estimate: 101, + hierarchy: { + provider: { + id: "obr", + label: "Office for Budget Responsibility — authored", + }, + category: { + id: "obr.efo_receipts", + label: "Economic and fiscal outlook receipts — authored", + provider_id: "obr", + }, + geography: { + id: "K02000001", + label: "United Kingdom — authored", + level: "country", + }, + dimensions: [ + { + id: "obr.efo_line", + label: "Economic and fiscal outlook line — authored", + value_id: "income_tax", + value_label: "Income tax receipts — authored", + }, + ], + target: { + id: "obr.income_tax", + label: "Income tax target — authored", + }, + }, + }, + ], + }, + "schema-8", + ); + + expect(cal.target_schema).toEqual({ + diagnostics_schema_version: 8, + structured_dimensions: false, + target_representation: "hierarchy", + }); + expect(cal.rows[0]).toMatchObject({ + family: "obr/obr.efo_receipts", + source: "obr", + source_label: "Office for Budget Responsibility — authored", + variable: "obr.efo_receipts", + variable_label: "Economic and fiscal outlook receipts — authored", + geography: "United Kingdom — authored", + geography_id: "K02000001", + level: "country", + target_label: "Income tax target — authored", + target_representation: "hierarchy", + dimension_adapter: "hierarchy", + target_dimensions: [ + { + key: "obr.efo_line", + label: "Economic and fiscal outlook line — authored", + value: "Income tax receipts — authored", + value_id: "income_tax", + source_key: "obr.efo_line", + raw_value: "income_tax", + rank: 0, + }, + ], + }); +}); + +test("top-level schema version selects one reader and rejects unsupported versions", () => { + expect( + buildCalibration( + { + schema_version: 6, + targets: [{ + name: "legacy.population@2025", + target_name: "legacy.population", + source: "Citation", + target: 100, + initial_estimate: 90, + final_estimate: 100, + }], + }, + "schema-6", + ).target_schema.target_representation, + ).toBe("legacy"); + expect(() => + buildCalibration( + { + schema_version: 7, + targets: [{ name: "legacy-row", source: "Citation" }], + }, + "invalid-schema-7", + ), + ).toThrow("Schema 7 target source.id"); + expect(() => + buildCalibration( + { schema_version: 8, targets: [{ name: "missing-hierarchy" }] }, + "invalid-schema-8", + ), + ).toThrow("hierarchy must be an object"); + for (const schemaVersion of [undefined, 1, 9]) { + expect(() => + buildCalibration( + { schema_version: schemaVersion, targets: [] }, + "unsupported-schema", + ), + ).toThrow("schema_version"); + } +}); + test("structured dimensions shape rows and honor artifact value order", () => { const target = ( suffix: string, @@ -304,6 +493,228 @@ test("structured dimensions shape rows and honor artifact value order", () => { ]); }); +test("legacy UK geography metadata resolves GSS identifiers across target-name formats", () => { + const legacyTarget = ( + name: string, + geographyId: string, + geographyName?: string, + ) => ({ + name: `${name}@2025`, + target_name: name, + period: 2025, + entity: "household", + measure: { kind: "column", name }, + filter: null, + source: "UK source citation", + metadata: { + ledger_geography_level: "country", + ledger_geography_id: geographyId, + ...(geographyName ? { ledger_geography_name: geographyName } : {}), + }, + registry: { family: "uk_source" }, + target: 100, + initial_estimate: 90, + final_estimate: 100, + }); + const cal = buildCalibration( + { + schema_version: 6, + targets: [ + legacyTarget("obr.receipts.income_tax", "K02000001"), + legacyTarget("dwp/uc/payment_band", "K03000001"), + legacyTarget("ons.population.england", "E92000001"), + legacyTarget("scotgov.council_tax.band_a", "S92000003"), + legacyTarget("ons.population.published", "K02000001", "Published area"), + legacyTarget("ons.population.named_uk", "K02000001", "United Kingdom"), + legacyTarget("ons.population.unknown", "X00000000"), + ], + }, + "uk-legacy-geography", + null, + {}, + {}, + {}, + "uk", + ); + + expect(cal.rows.map((row) => [row.geography, row.level])).toEqual([ + ["United Kingdom", "country"], + ["Great Britain", "country"], + ["England", "country"], + ["Scotland", "country"], + ["Published area", "country"], + ["United Kingdom", "country"], + ["United Kingdom", "national"], + ]); +}); + +test("legacy US geography normalization remains unchanged", () => { + const cal = buildCalibration( + { + schema_version: 6, + targets: [ + { + name: "census.population.total@2025", + target_name: "census.population.total", + source: "Census citation", + metadata: { + ledger_geography_level: "country", + ledger_geography_id: "0100000US", + ledger_geography_name: "United States", + }, + target: 100, + initial_estimate: 90, + final_estimate: 100, + }, + ], + }, + "us-legacy-geography", + ); + + expect(cal.rows[0]).toMatchObject({ + geography: "United States", + level: "national", + }); +}); + +test("structured UK geography preserves producer identity and ordering", () => { + const geographyValues = { + K02000001: "United Kingdom", + K03000001: "Great Britain", + E92000001: "England", + S92000003: "Scotland", + }; + const order = ["K02000001", "K03000001", "E92000001", "S92000003"]; + const targets = order.map((geographyId) => ({ + name: `ons.population.${geographyId}@2025`, + target_name: `ons.population.${geographyId}`, + source: { id: "ons", citation: "ONS population table" }, + variable: { id: "population", measure: "count" }, + dimensions: { geography_country: geographyId }, + target: 100, + initial_estimate: 90, + final_estimate: 100, + })); + const cal = buildCalibration( + { + schema_version: 7, + dimensions: { + geography_country: { + label: "Geography", + role: "geography", + level: "country", + values: geographyValues, + order, + }, + }, + targets, + }, + "uk-structured-geography", + null, + {}, + {}, + {}, + "uk", + ); + const page = latestMicrocosmTargetDiagnosticsPage( + "http://x/api/microcosm/target-diagnostics?variable=ons%20%2F%20population%20%C2%B7%20count", + cal, + ); + + expect(cal.rows[3]).toMatchObject({ + geography: "Scotland", + geography_id: "S92000003", + geography_dimension_id: "geography_country", + geography_rank: 3, + level: "country", + }); + expect(page.dimensions[0]).toEqual({ + key: "geography", + label: "Geography", + values: ["United Kingdom", "Great Britain", "England", "Scotland"], + }); +}); + +test("structured geography falls back to deterministic sorting when order is incomplete", () => { + const cal = buildCalibration( + { + schema_version: 7, + dimensions: { + geography_country: { + label: "Geography", + role: "geography", + level: "country", + values: { K02000001: "United Kingdom", E92000001: "England" }, + order: ["K02000001"], + }, + }, + targets: ["K02000001", "E92000001"].map((geographyId) => ({ + name: `ons.population.${geographyId}@2025`, + source: { id: "ons" }, + variable: { id: "population" }, + dimensions: { geography_country: geographyId }, + target: 100, + initial_estimate: 90, + final_estimate: 100, + })), + }, + "uk-structured-geography-incomplete-order", + null, + {}, + {}, + {}, + "uk", + ); + const page = latestMicrocosmTargetDiagnosticsPage( + "http://x/api/microcosm/target-diagnostics?variable=ons%20%2F%20population", + cal, + ); + + expect(page.dimensions[0]?.values).toEqual(["England", "United Kingdom"]); +}); + +test("structured targets reject multiple populated geography dimensions", () => { + expect(() => + buildCalibration( + { + schema_version: 7, + dimensions: { + geography_country: { + label: "Country", + role: "geography", + level: "country", + }, + geography_region: { + label: "Region", + role: "geography", + level: "region", + }, + }, + targets: [ + { + name: "ons.population@2025", + source: { id: "ons" }, + variable: { id: "population" }, + dimensions: { + geography_country: "E92000001", + geography_region: "north_east", + }, + target: 100, + initial_estimate: 90, + final_estimate: 100, + }, + ], + }, + "uk-ambiguous-structured-geography", + null, + {}, + {}, + {}, + "uk", + ) + ).toThrow("at most one populated geography-role dimension"); +}); + test("structured facet ordering falls back when any displayed value lacks a rank", () => { const target = (suffix: string, category: string) => ({ name: `fixture.population.${suffix}@2026`, @@ -321,6 +732,7 @@ test("structured facet ordering falls back when any displayed value lacks a rank }); const cal = buildCalibration( { + schema_version: 7, dimensions: { category: { label: "Category", @@ -342,7 +754,7 @@ test("structured facet ordering falls back when any displayed value lacks a rank ]); }); -test("mixed files select the dimension adapter from each complete row representation", () => { +test("schema 7 rejects a mixture of structured and legacy rows", () => { const base = { source: "ZZ official population table", metadata: { @@ -354,7 +766,7 @@ test("mixed files select the dimension adapter from each complete row representa initial_estimate: 90, final_estimate: 100, }; - const cal = buildCalibration( + expect(() => buildCalibration( { schema_version: 7, dimensions: { @@ -388,21 +800,7 @@ test("mixed files select the dimension adapter from each complete row representa ], }, "mixed-dimension-adapters", - ); - - expect(cal.rows.map((row) => row.dimension_adapter)).toEqual([ - "structured", - "legacy_filter", - "legacy_name", - ]); - expect(cal.rows[0]).toMatchObject({ - geography: "North", - level: "region", - breakdown: "Female · 0–17", - }); - expect(latestMicrocosmCalibrationSummary(cal).target_schema).toEqual( - cal.target_schema, - ); + )).toThrow("Schema 7 target source.id"); }); test("structured rows do not require a dimensions dictionary, including empty objects", () => { @@ -417,6 +815,7 @@ test("structured rows do not require a dimensions dictionary, including empty ob }; const cal = buildCalibration( { + schema_version: 7, targets: [ { ...base, @@ -471,6 +870,7 @@ test("structured dimensions prevent whole-population estimate-scope warnings", ( }); const cal = buildCalibration( { + schema_version: 7, dimensions: { category: { label: "Category" } }, targets: [ target("source.example.slice_a", "a", 10), @@ -486,6 +886,7 @@ test("structured dimensions prevent whole-population estimate-scope warnings", ( test("keeps legacy US dotted target families when Chronicle publisher metadata is present", () => { const cal = buildCalibration( { + schema_version: 2, targets: [ { name: "irs.population.total@2024", @@ -667,7 +1068,15 @@ function agiTarget(band: string, ret: string, filing: string, rel: number) { } function calibration(targets: object[], releaseId = "rel-a"): Calibration { - return buildCalibration({ targets, final_loss: 0.02, fraction_within_10pct: 0.9 }, releaseId); + return buildCalibration( + { + schema_version: 2, + targets, + final_loss: 0.02, + fraction_within_10pct: 0.9, + }, + releaseId, + ); } const SAMPLE = calibration([ @@ -1090,6 +1499,7 @@ test("release highlights split bounded percent fit from absolute miss magnitude" test("calibration inclusion status uses skipped and dropped metadata", () => { const cal = buildCalibration( { + schema_version: 2, targets: [ { name: "included@2024", target: 1, initial_estimate: 1, final_estimate: 1 }, { name: "skipped@2024", target_name: "skipped", target: 1 }, @@ -1290,18 +1700,24 @@ test("comparison matches renamed legacy and structured targets by Chronicle fact final_estimate: 95, }, ], "legacy-current"); - const candidate = calibration([ + const candidate = buildCalibration( { - name: "resident-population@2024", - source: { id: "agency", label: "Statistical agency" }, - variable: { id: "resident_population", measure: "count" }, - dimensions: {}, - metadata: { ledger_fact_key: "agency.population.total" }, - target: 100, - initial_estimate: 90, - final_estimate: 99, + schema_version: 7, + targets: [ + { + name: "resident-population@2024", + source: { id: "agency", label: "Statistical agency" }, + variable: { id: "resident_population", measure: "count" }, + dimensions: {}, + metadata: { ledger_fact_key: "agency.population.total" }, + target: 100, + initial_estimate: 90, + final_estimate: 99, + }, + ], }, - ], "structured-candidate"); + "structured-candidate", + ); const cmp = buildComparison(current, candidate); expect(cmp.summary).toMatchObject({ @@ -1358,6 +1774,7 @@ test("comparison preserves duplicate names and resolves them by unique fallback test("new target loss weighting metadata marks loss as normalized", () => { const normalized = buildCalibration( { + schema_version: 6, targets: [], initial_loss: 0.42, final_loss: 0.39, @@ -1369,7 +1786,12 @@ test("new target loss weighting metadata marks loss as normalized", () => { "normalized-release", ); const raw = buildCalibration( - { targets: [], initial_loss: 752_000_000_000, final_loss: 751_000_000_000 }, + { + schema_version: 5, + targets: [], + initial_loss: 752_000_000_000, + final_loss: 751_000_000_000, + }, "raw-release", ); @@ -1752,6 +2174,7 @@ test("local-area diagnostics (value/estimate schema) render as included targets" test("canonical target/final_estimate are never overwritten by value/estimate aliases", () => { const cal = buildCalibration( { + schema_version: 2, targets: [ { name: "nation/irs/agi/total@2024", @@ -1776,6 +2199,7 @@ test("canonical target/final_estimate are never overwritten by value/estimate al test("unreadable diagnostics rows report an explicit incompatible status, not a silent zero", () => { const cal = buildCalibration( { + schema_version: 2, targets: [ { name: "source.us.total.mystery-a@2024", target_name: "source.us.total.mystery-a", metadata: {}, unknown_metric: 1 }, { name: "source.us.total.mystery-b@2024", target_name: "source.us.total.mystery-b", metadata: {} }, @@ -1796,13 +2220,13 @@ test("unreadable diagnostics rows report an explicit incompatible status, not a }); test("diagnostics with an empty targets list report an explicit empty status", () => { - const cal = buildCalibration({ targets: [] }, "rel"); + const cal = buildCalibration({ schema_version: 2, targets: [] }, "rel"); expect(cal.diagnostics_status).toBe("empty"); expect(latestMicrocosmCalibrationSummary(cal).diagnostics_status).toBe("empty"); }); test("diagnostics missing the targets array report incompatible", () => { - const cal = buildCalibration({ final_loss: 0.1 }, "rel"); + const cal = buildCalibration({ schema_version: 2, final_loss: 0.1 }, "rel"); expect(cal.diagnostics_status).toBe("incompatible"); }); @@ -1862,7 +2286,7 @@ test("presentation flows through calibration, summary, and target responses", () targets_intro: "Artifact target prompt.", }; const cal = buildCalibration( - { targets: [] }, + { schema_version: 2, targets: [] }, "presentation-release", null, {}, @@ -1903,6 +2327,7 @@ test("releasePublisherLabels keeps valid keys and trimmed non-empty labels", () test("publisher labels flow through rows, variables, target responses, and treemaps", () => { const cal = buildCalibration( { + schema_version: 2, targets: [ { name: "fixture_population@2026", @@ -1943,6 +2368,7 @@ test("publisher labels flow through rows, variables, target responses, and treem test("publisher label lookup does not read inherited object properties", () => { const cal = buildCalibration( { + schema_version: 2, targets: [ { name: "constructor.population.total@2026", @@ -1963,6 +2389,7 @@ test("publisher label lookup does not read inherited object properties", () => { test("fully structured targets ignore conflicting legacy identity fields", () => { const cal = buildCalibration( { + schema_version: 7, dimensions: { region: { label: "Region", @@ -2049,6 +2476,73 @@ test("fully structured targets ignore conflicting legacy identity fields", () => }); }); +test("structured categories group count and total rows by variable identity", () => { + const target = (name: string, measure: "count" | "total") => ({ + name: `${name}@2025`, + target_name: name, + source: { id: "hmrc", citation: "HMRC SPI" }, + variable: { id: "spi_employment_income", measure }, + dimensions: { + geography_country: "K02000001", + total_income_lower_bound: "100000", + }, + metadata: {}, + target: 100, + initial_estimate: 90, + final_estimate: 100, + }); + const cal = buildCalibration( + { + schema_version: 7, + dimensions: { + geography_country: { + label: "Country", + role: "geography", + level: "country", + values: { K02000001: "United Kingdom" }, + order: ["K02000001"], + }, + total_income_lower_bound: { + label: "Total Income Lower Bound", + values: { "100000": "100000" }, + order: ["100000"], + }, + }, + targets: [ + target("hmrc/employment_income_amount_band_100000", "total"), + target("hmrc/employment_income_count_band_100000", "count"), + ], + }, + "uk-schema-7-categories", + null, + {}, + {}, + {}, + "uk", + ); + const tree = buildCalibrationTree(cal.rows, { + breakdown: "program", + path: { dimensions: [] }, + filters: { + geographyLevels: [], + geographies: [], + fitBands: [], + calibrationStatuses: [], + }, + }); + + expect(cal.target_schema.target_representation).toBe("structured"); + expect(cal.rows.map((row) => row.measure)).toEqual(["total", "count"]); + expect(cal.rows.every((row) => row.dimension_adapter === "structured")).toBe(true); + expect(tree.groups).toHaveLength(1); + expect(tree.groups[0].nodes).toHaveLength(1); + expect(tree.groups[0].nodes[0]).toMatchObject({ + id: "spi_employment_income", + label: "Spi employment income", + metrics: { nTargets: 2 }, + }); +}); + test("structured dimension ids remain independent when display labels repeat", () => { const structuredTargets = [ ["north", "east"], @@ -2065,6 +2559,7 @@ test("structured dimension ids remain independent when display labels repeat", ( final_estimate: 100, })); const diagnostics = { + schema_version: 7, dimensions: { origin: { label: "Region", @@ -2125,28 +2620,29 @@ test("structured dimension ids remain independent when display labels repeat", ( }); expect(destinationTree.groups[0]?.id).toBe("bd_origin"); - const mixed = buildCalibration( - { - ...diagnostics, - targets: [ - ...structuredTargets, - { - name: "nation/legacy/population", - target: 1, - initial_estimate: 1, - final_estimate: 1, - }, - ], - }, - "mixed-repeated-dimension-labels", - ); - expect(mixed.target_schema.target_representation).toBe("mixed"); - expect(mixed.rows[0].target_dimensions).toEqual(cal.rows[0].target_dimensions); + expect(() => + buildCalibration( + { + ...diagnostics, + targets: [ + ...structuredTargets, + { + name: "nation/legacy/population", + target: 1, + initial_estimate: 1, + final_estimate: 1, + }, + ], + }, + "mixed-repeated-dimension-labels", + ), + ).toThrow("Schema 7 target source.id"); }); -test("mixed diagnostics dispatch complete legacy and structured rows independently", () => { - const cal = buildCalibration( +test("schema 7 refuses row-shape-based fallback to the legacy reader", () => { + expect(() => buildCalibration( { + schema_version: 7, targets: [ { name: "bea_nipa.cy2023.proprietors_income.a041rc.amount@2024", @@ -2184,25 +2680,7 @@ test("mixed diagnostics dispatch complete legacy and structured rows independent ], }, "mixed-identities", - ); - - expect(cal.target_schema.target_representation).toBe("mixed"); - expect(cal.rows[0]).toMatchObject({ - source: "bea", - variable: "amount", - dimension_adapter: "legacy_name", - target_representation: "legacy", - }); - expect(cal.rows[1]).toMatchObject({ - source: "artifact_agency", - source_label: "Artifact agency", - source_citation: "Official population table", - variable: "resident_population", - variable_label: "Resident population", - measure: "count", - dimension_adapter: "structured", - target_representation: "structured", - }); + )).toThrow("Schema 7 target source.id"); }); test("structured source and variable fields remain authoritative", () => { @@ -2230,7 +2708,7 @@ test("structured source and variable fields remain authoritative", () => { final_estimate: 100, }; const cal = buildCalibration( - { targets: [target] }, + { schema_version: 7, targets: [target] }, "structured-identifiers", null, {}, @@ -2259,6 +2737,7 @@ test("structured source and variable fields remain authoritative", () => { const withoutChronicle = buildCalibration( { + schema_version: 7, targets: [ { ...target, @@ -2384,7 +2863,7 @@ test("releaseCountry ignores malformed field values", () => { test("the calibration summary and target page carry the typed country block", () => { const cal = buildCalibration( - beDiagnosticsFixture, + { ...beDiagnosticsFixture, schema_version: 2 }, "be-country", null, {}, @@ -2401,7 +2880,7 @@ test("the calibration summary and target page carry the typed country block", () test("the artifact's national geography label shapes rows without a geography", () => { const cal = buildCalibration( - beDiagnosticsFixture, + { ...beDiagnosticsFixture, schema_version: 2 }, "be-geography", null, {}, diff --git a/frontend/lib/microcosm/latest-artifact.ts b/frontend/lib/microcosm/latest-artifact.ts index 9133bd0..765f700 100644 --- a/frontend/lib/microcosm/latest-artifact.ts +++ b/frontend/lib/microcosm/latest-artifact.ts @@ -1,11 +1,17 @@ -// Pure-HF data layer for the country-selectable Microcosm dashboard. No -// committed snapshot: every release's manifests and per-target calibration -// diagnostics are read live from its country's Hugging Face dataset, resolved -// through latest.json (current release) or by id (version compare). +// Data layer for the country-selectable Microcosm dashboard. Published releases +// are read from each country's Hugging Face dataset; an explicit UK local-run +// directory can replace that source for local diagnostics inspection. + +import { readFile, readdir, stat } from "node:fs/promises"; +import { basename, join } from "node:path"; import { sourceAuthorityLabel } from "@/lib/source-labels"; import { normalizeChronicleMetadata } from "./chronicle-metadata"; +import { + readHierarchyTarget, + validateHierarchyTargets, +} from "./hierarchy-target-reader"; import { chroniclePublisherFromMetadata, qualifyingChildrenFromRecordSet, @@ -31,9 +37,9 @@ import { type TargetLossDiagnosticWarning, } from "./target-loss-attribution"; import { - classifyTargetRow, - classifyTargetRepresentation, + targetRepresentationForSchema, type TargetRepresentation, + type TargetRowRepresentation, } from "./target-representation"; import { readStructuredTarget } from "./structured-target-reader"; import { matchTargetSurfaces } from "./target-surface-matcher"; @@ -849,8 +855,23 @@ function computeDimensions(rows: TargetRow[]): TargetDimension[] { if (values.length <= 1) continue; const label = candidate.label ?? classifyDimension(values); const ranks = new Map(); - let everyValueRanked = candidate.key !== "geography"; + let everyValueRanked = true; for (const value of values) { + if (candidate.key === "geography") { + const matchingRows = rows.filter((row) => row.geography === value); + if ( + !matchingRows.length || + matchingRows.some((row) => typeof row.geography_rank !== "number") + ) { + everyValueRanked = false; + break; + } + ranks.set( + value, + Math.min(...matchingRows.map((row) => row.geography_rank as number)), + ); + continue; + } const matchingDimensions = rows.flatMap((row) => ((row.target_dimensions as TargetBreakdownDimension[] | undefined) ?? []) .filter((dimension) => @@ -978,6 +999,7 @@ function enrichTargetRow( artifactCountry: ArtifactCountry, publisherLabels: Record, dimensionDefinitions: Record, + rowRepresentation: TargetRowRepresentation, ): TargetRow { const nationalGeography = artifactCountry.geography_label; const metadata = normalizeChronicleMetadata(rawRow.metadata); @@ -1017,19 +1039,23 @@ function enrichTargetRow( initialError == null || finalError == null ? null : Math.abs(initialError) - Math.abs(finalError); - const rowRepresentation = classifyTargetRow(row); + const hierarchyIdentity = rowRepresentation === "hierarchy" + ? readHierarchyTarget(row) + : null; const structuredIdentity = rowRepresentation === "structured" ? readStructuredTarget(row, dimensionDefinitions, nationalGeography) : null; - const filterDecomposition = structuredIdentity + const filterDecomposition = hierarchyIdentity || structuredIdentity ? null : decomposeTargetFilter(row.filter); - const dimensionAdapter = structuredIdentity - ? "structured" - : filterDecomposition - ? "legacy_filter" - : "legacy_name"; - const legacyParsed = structuredIdentity + const dimensionAdapter = hierarchyIdentity + ? "hierarchy" + : structuredIdentity + ? "structured" + : filterDecomposition + ? "legacy_filter" + : "legacy_name"; + const legacyParsed = hierarchyIdentity || structuredIdentity ? null : readLegacyTarget( baseName, @@ -1037,7 +1063,15 @@ function enrichTargetRow( nationalGeography, filterDecomposition, ); - const parsed: ParsedTarget = structuredIdentity + const parsed: ParsedTarget = hierarchyIdentity + ? { + geography: hierarchyIdentity.geography, + level: hierarchyIdentity.level, + source: hierarchyIdentity.source, + variable: hierarchyIdentity.variable, + breakdown: hierarchyIdentity.breakdown, + } + : structuredIdentity ? { geography: structuredIdentity.geography, level: structuredIdentity.level, @@ -1051,6 +1085,7 @@ function enrichTargetRow( const level = hasGeography ? parsed.level : DEFAULT_GEOGRAPHY_LEVEL; const measureCol = asObject(row.measure); const metadataTargetDimensions = + hierarchyIdentity?.dimensions ?? structuredIdentity?.dimensions ?? filterDecomposition?.dimensions ?? metadataDimensions(row); @@ -1082,6 +1117,7 @@ function enrichTargetRow( // without filter dimensions retain their legacy family so US/UK releases do // not regroup merely because they also carry Chronicle record IDs. const usesArtifactFamily = + hierarchyIdentity != null || structuredIdentity != null || filterDecomposition != null || (chroniclePublisherFromMetadata(metadata) != null && @@ -1092,20 +1128,29 @@ function enrichTargetRow( name: fullName, base_name: baseName, family: deriveFamily(baseName, parsed, usesArtifactFamily), - state: structuredIdentity + state: hierarchyIdentity || structuredIdentity ? null : stateFromGeoId(stringValue(metadata.ledger_geography_id)) ?? deriveState(baseName), geography, + geography_id: + hierarchyIdentity?.geographyId ?? + structuredIdentity?.geographyId ?? + stringValue(metadata.ledger_geography_id), + geography_dimension_id: structuredIdentity?.geographyDimensionId ?? null, + geography_rank: structuredIdentity?.geographyRank ?? null, level, source: parsed.source, source_label: + hierarchyIdentity?.sourceLabel ?? (Object.hasOwn(publisherLabels, parsed.source) ? publisherLabels[parsed.source] : undefined) ?? structuredIdentity?.sourceLabel ?? sourceAuthorityLabel(parsed.source), variable: parsed.variable, - variable_label: structuredIdentity?.variableLabel ?? null, + variable_label: + hierarchyIdentity?.variableLabel ?? structuredIdentity?.variableLabel ?? null, + target_label: hierarchyIdentity?.targetLabel, measure, target_role: targetRole, source_measure_id: sourceMeasureId, @@ -1687,7 +1732,7 @@ export interface TargetSchema { } export interface Calibration { - source: "huggingface_live"; + source: "huggingface_live" | "local_filesystem"; country: MicrocosmCountry; // Typed `release_manifest.country` merged over the registration. country_info: ArtifactCountry; @@ -1936,11 +1981,15 @@ export function buildCalibration( releaseManifest: JsonObject = {}, demographics: JsonObject = {}, country: MicrocosmCountry = "us", + source: Calibration["source"] = "huggingface_live", ): Calibration { + const targetRepresentation = targetRepresentationForSchema(diag.schema_version); const targets = (Array.isArray(diag.targets) ? (diag.targets as TargetRow[]) : []).map( normalizeDiagnosticsRow, ); - const targetRepresentation = classifyTargetRepresentation(targets); + if (targetRepresentation === "hierarchy") { + validateHierarchyTargets(targets); + } const skipped = Array.isArray(diag.skipped) ? (diag.skipped as JsonObject[]) : []; const targetCompilation = asObject(asObject(buildManifest.gates).target_compilation); const droppedTargetNames = Array.isArray(targetCompilation.dropped_target_names) @@ -1963,6 +2012,7 @@ export function buildCalibration( artifactCountry, publisherLabels, dimensionDefinitions, + targetRepresentation, ), ), ); @@ -1977,7 +2027,7 @@ export function buildCalibration( const rows = normalizedAttribution.rows; const includedTargetCount = rows.filter((row) => row.calibration_status === "included").length; return { - source: "huggingface_live", + source, country, country_info: artifactCountry, presentation, @@ -2103,10 +2153,109 @@ function releaseDate(id: string): string { return m ? m[1] : id; } +export const UK_LOCAL_CALIBRATION_DIR_ENV = + "MICROCOSM_UK_LOCAL_CALIBRATION_DIR"; + +interface LocalCalibrationRelease { + releaseId: string; + updatedAt: string; + files: string[]; + diagnostics: JsonObject; + buildManifest: JsonObject; + releaseManifest: JsonObject; + demographics: JsonObject; +} + +function localCalibrationDirectory(country: MicrocosmCountry): string | null { + if (country !== "uk") return null; + const directory = process.env[UK_LOCAL_CALIBRATION_DIR_ENV]?.trim(); + return directory || null; +} + +async function localJson( + directory: string, + filename: string, +): Promise { + const path = join(directory, filename); + try { + return asObject(JSON.parse(await readFile(path, "utf8"))); + } catch (error) { + throw new Error(`Unable to read local Microcosm artifact ${path}`, { + cause: error, + }); + } +} + +async function optionalLocalJson( + directory: string, + filename: string, +): Promise { + try { + return await localJson(directory, filename); + } catch { + return {}; + } +} + +async function loadLocalCalibrationRelease( + country: MicrocosmCountry, +): Promise { + const directory = localCalibrationDirectory(country); + if (!directory) return null; + + const diagnosticsPath = join(directory, "calibration_diagnostics.json"); + const [ + diagnostics, + diagnosticsStat, + files, + buildManifest, + releaseManifest, + demographics, + ] = await Promise.all([ + localJson(directory, "calibration_diagnostics.json"), + stat(diagnosticsPath), + readdir(directory), + optionalLocalJson(directory, "build_manifest.json"), + optionalLocalJson(directory, "release_manifest.json"), + optionalLocalJson(directory, "demographics.json"), + ]); + const diagnosticsBuild = asObject(diagnostics.build); + const releaseId = assertSafeReleaseId( + String( + diagnostics.release_id ?? diagnosticsBuild.build_id ?? basename(directory), + ), + "local UK calibration release", + ); + return { + releaseId, + updatedAt: diagnosticsStat.mtime.toISOString(), + files: files.sort(), + diagnostics, + buildManifest, + releaseManifest, + demographics, + }; +} + export async function loadReleases( revalidate: number, country: MicrocosmCountry = "us", ): Promise { + const local = await loadLocalCalibrationRelease(country); + if (local) { + return [ + { + release_id: local.releaseId, + date: local.updatedAt, + files: local.files, + has_calibration: true, + dataset_role: null, + is_default: true, + is_local_area: false, + }, + ]; + } + const { repo, revision } = COUNTRY_REPO[country]; const files = new Map>(); // The HF tree endpoint paginates (~1000 entries/page via a Link cursor); @@ -2170,6 +2319,11 @@ export async function loadPointerReleaseId( revalidate: number, country: MicrocosmCountry = "us", ): Promise<{ release_id: string; updated_at: string | null }> { + const local = await loadLocalCalibrationRelease(country); + if (local) { + return { release_id: local.releaseId, updated_at: local.updatedAt }; + } + const pointer = await hfJson(hfResolveUrl("latest.json", country), revalidate); return { release_id: String(pointer.release_id ?? ""), @@ -2207,6 +2361,23 @@ async function loadReleaseUncached( revalidate: number, country: MicrocosmCountry, ): Promise { + const local = await loadLocalCalibrationRelease(country); + if ( + local && + (releaseId === "latest" || !releaseId || releaseId === local.releaseId) + ) { + return buildCalibration( + local.diagnostics, + local.releaseId, + local.updatedAt, + local.buildManifest, + local.releaseManifest, + local.demographics, + country, + "local_filesystem", + ); + } + let id = assertSafeReleaseId(releaseId); let updatedAt: string | null = null; if (releaseId === "latest" || !releaseId) { @@ -2266,6 +2437,7 @@ function targetResponseRow(row: TargetRow): TargetRow { source_label: row.source_label, variable: row.variable, variable_label: row.variable_label, + target_label: row.target_label, measure: row.measure, target_role: row.target_role, source_measure_id: row.source_measure_id, @@ -2287,6 +2459,7 @@ function targetResponseRow(row: TargetRow): TargetRow { target_dimensions: row.target_dimensions, dimension_adapter: row.dimension_adapter, target_representation: row.target_representation, + hierarchy: row.hierarchy, variable_key: row.variable_key, source_citation: row.source_citation, source_url: row.source_url, @@ -2834,12 +3007,17 @@ export function buildComparison(a: Calibration, b: Calibration) { current_representation: match.current_representation, candidate_representation: match.candidate_representation, name, - target_label: [br.geography ?? ar.geography, br.breakdown ?? ar.breakdown] - .filter(Boolean) - .join(" · "), + target_label: + br.target_label ?? + ar.target_label ?? + [br.geography ?? ar.geography, br.breakdown ?? ar.breakdown] + .filter(Boolean) + .join(" · "), source: br.source ?? ar.source, + source_label: br.source_label ?? ar.source_label, variable_key: br.variable_key ?? ar.variable_key, variable: br.variable ?? ar.variable, + variable_label: br.variable_label ?? ar.variable_label, measure: br.measure ?? ar.measure, level: br.level ?? ar.level, breakdown: br.breakdown ?? ar.breakdown, diff --git a/frontend/lib/microcosm/legacy-target-reader.ts b/frontend/lib/microcosm/legacy-target-reader.ts index 8d32441..c7eb9b2 100644 --- a/frontend/lib/microcosm/legacy-target-reader.ts +++ b/frontend/lib/microcosm/legacy-target-reader.ts @@ -33,6 +33,16 @@ const STATE_ABBRS = new Set([ "RI", "SC", "SD", "TN", "TX", "UT", "VT", "VA", "WA", "WV", "WI", "WY", "US", ]); +// Compatibility labels for the country-level GSS identifiers in the current +// UK schema-6 artifact. Future artifacts should publish geography labels in +// structured dimension definitions or `ledger_geography_name`. +const LEGACY_GSS_GEOGRAPHIES: Readonly> = { + K02000001: "United Kingdom", + K03000001: "Great Britain", + E92000001: "England", + S92000003: "Scotland", +}; + const MEASURES = new Set(["total", "count", "mean", "filers", "nonfilers"]); const LEGACY_UNDERSCORE_PUBLISHERS = new Set([ "statbel", @@ -70,6 +80,43 @@ export function districtFromGeoId(value: string | null): string | null { return state ? `${state}-${match[2]}` : null; } +function geographyFromMetadata( + metadata: JsonObject, + nationalGeography: string, +): Pick | null { + const publishedName = stringValue(metadata.ledger_geography_name)?.trim() ?? null; + const geographyId = stringValue(metadata.ledger_geography_id); + const geographyLevel = stringValue(metadata.ledger_geography_level); + const gssGeography = geographyId ? LEGACY_GSS_GEOGRAPHIES[geographyId] : null; + + if (publishedName) { + return { + geography: publishedName, + level: + geographyLevel === "country" && + publishedName === nationalGeography && + !gssGeography + ? "national" + : geographyLevel ?? "", + }; + } + if (gssGeography) { + return { geography: gssGeography, level: geographyLevel ?? "country" }; + } + if (geographyLevel === "congressional_district") { + const geography = districtFromGeoId(geographyId); + return geography ? { geography, level: geographyLevel } : null; + } + if (geographyLevel === "state") { + const geography = stateFromGeoId(geographyId) ?? stringValue(metadata.state); + return geography ? { geography, level: geographyLevel } : null; + } + if (geographyLevel === "country") { + return { geography: nationalGeography, level: "national" }; + } + return null; +} + function variableFromMeasure(value: string | null): string | null { return value ? value.replace(/_(amount|returns|claims|count|total|collections|projected_amount)$/, "") @@ -162,29 +209,13 @@ function legacyVariable( function parseDottedTarget( name: string, row: JsonObject, - nationalGeography: string, + _nationalGeography: string, ): ParsedLegacyTarget | null { if (!name.includes(".")) return null; const metadata = asObject(row.metadata); const registry = asObject(row.registry); const parts = name.split("."); const source = stringValue(registry.family) ?? parts[0] ?? ""; - const geoLevel = stringValue(metadata.ledger_geography_level); - const geoId = stringValue(metadata.ledger_geography_id); - const geography = - geoLevel === "country" - ? nationalGeography - : geoLevel === "congressional_district" - ? districtFromGeoId(geoId) ?? "" - : stateFromGeoId(geoId) ?? stringValue(metadata.state) ?? ""; - const level = - geoLevel === "country" - ? "national" - : geoLevel === "state" - ? "state" - : geoLevel === "congressional_district" - ? "congressional_district" - : ""; const measureId = stringValue(metadata.source_measure_id) ?? parts.at(-1) ?? ""; const variable = readableToken(stringValue(metadata.variable)) ?? @@ -202,7 +233,7 @@ function parseDottedTarget( .filter((value): value is string => Boolean(value && value !== variable)) .join(" · "); - return { geography, level, source, variable, breakdown }; + return { geography: "", level: "", source, variable, breakdown }; } function parseSlashTarget(name: string, nationalGeography: string): ParsedLegacyTarget { @@ -276,10 +307,12 @@ export function readLegacyTarget( parseDottedTarget(name, row, nationalGeography) ?? parseSlashTarget(name, nationalGeography); const publisher = chroniclePublisherFromMetadata(metadata); + const metadataGeography = geographyFromMetadata(metadata, nationalGeography); return { ...parsed, - geography: decomposition?.geography ?? parsed.geography, - level: decomposition?.level ?? parsed.level, + geography: + decomposition?.geography ?? metadataGeography?.geography ?? parsed.geography, + level: decomposition?.level ?? metadataGeography?.level ?? parsed.level, source: publisher ?? parsed.source, variable: legacyVariable(name, row, decomposition) ?? parsed.variable, breakdown: decomposition diff --git a/frontend/lib/microcosm/structured-dimension-reader.ts b/frontend/lib/microcosm/structured-dimension-reader.ts index 6ca60d8..3a6ea3a 100644 --- a/frontend/lib/microcosm/structured-dimension-reader.ts +++ b/frontend/lib/microcosm/structured-dimension-reader.ts @@ -19,6 +19,9 @@ export interface StructuredTargetDimension { export interface StructuredDimensions { geography: string | null; + geographyId: string | null; + geographyDimensionId: string | null; + geographyRank: number | null; level: string | null; dimensions: StructuredTargetDimension[]; } @@ -84,6 +87,9 @@ export function readStructuredDimensions( definitions: Record, ): StructuredDimensions { let geography: string | null = null; + let geographyId: string | null = null; + let geographyDimensionId: string | null = null; + let geographyRank: number | null = null; let level: string | null = null; const dimensions: StructuredTargetDimension[] = []; @@ -98,8 +104,16 @@ export function readStructuredDimensions( (definition?.values ? Object.keys(definition.values) : undefined); const rank = rankOrder?.indexOf(rawValue) ?? -1; if (definition?.role === "geography") { - geography ??= value; - level ??= definition.level ?? "region"; + if (geography != null) { + throw new Error( + "Structured target dimensions must contain at most one populated geography-role dimension.", + ); + } + geography = value; + geographyId = rawValue; + geographyDimensionId = id; + geographyRank = rank >= 0 ? rank : null; + level = definition.level ?? "region"; continue; } dimensions.push({ @@ -112,5 +126,12 @@ export function readStructuredDimensions( }); } - return { geography, level, dimensions }; + return { + geography, + geographyId, + geographyDimensionId, + geographyRank, + level, + dimensions, + }; } diff --git a/frontend/lib/microcosm/structured-target-reader.ts b/frontend/lib/microcosm/structured-target-reader.ts index 3213d78..b849ba4 100644 --- a/frontend/lib/microcosm/structured-target-reader.ts +++ b/frontend/lib/microcosm/structured-target-reader.ts @@ -15,6 +15,9 @@ export interface StructuredTargetIdentity { variableLabel: string | null; measure: string | null; geography: string; + geographyId: string | null; + geographyDimensionId: string | null; + geographyRank: number | null; level: string; dimensions: StructuredTargetDimension[]; breakdown: string; @@ -35,11 +38,20 @@ export function readStructuredTarget( definitions: Record, nationalGeography: string, ): StructuredTargetIdentity { - const source = isPlainObject(row.source) ? row.source : {}; - const variable = isPlainObject(row.variable) ? row.variable : {}; - const values = isPlainObject(row.dimensions) ? row.dimensions : {}; - const sourceId = stringValue(source.id) ?? "other"; - const variableId = stringValue(variable.id) ?? "unknown"; + if (!isPlainObject(row.source) || !stringValue(row.source.id)) { + throw new Error("Schema 7 target source.id must be a non-empty string."); + } + if (!isPlainObject(row.variable) || !stringValue(row.variable.id)) { + throw new Error("Schema 7 target variable.id must be a non-empty string."); + } + if (!isPlainObject(row.dimensions)) { + throw new Error("Schema 7 target dimensions must be an object."); + } + const source = row.source; + const variable = row.variable; + const values = row.dimensions; + const sourceId = stringValue(source.id)!; + const variableId = stringValue(variable.id)!; const structured = readStructuredDimensions(values, definitions); const geography = structured.geography ?? nationalGeography; const level = structured.level ?? "national"; @@ -54,6 +66,9 @@ export function readStructuredTarget( variableLabel: stringValue(variable.label), measure: stringValue(variable.measure), geography, + geographyId: structured.geographyId, + geographyDimensionId: structured.geographyDimensionId, + geographyRank: structured.geographyRank, level, dimensions, breakdown: dimensions.map((dimension) => dimension.value).join(" · "), diff --git a/frontend/lib/microcosm/target-change.ts b/frontend/lib/microcosm/target-change.ts index e004aad..2eac993 100644 --- a/frontend/lib/microcosm/target-change.ts +++ b/frontend/lib/microcosm/target-change.ts @@ -4,6 +4,7 @@ import { type TargetMatchingSummary, type TargetSurfaceStatus, } from "./target-surface-matcher"; +import type { TargetRowRepresentation } from "./target-representation"; type TargetRow = Calibration["rows"][number]; @@ -29,8 +30,8 @@ export interface TargetChangeRow extends Record { match_kind: "base_name" | "chronicle_fact_key" | "structured_identity" | null; current_name: string | null; candidate_name: string | null; - current_representation: "legacy" | "structured" | null; - candidate_representation: "legacy" | "structured" | null; + current_representation: TargetRowRepresentation | null; + candidate_representation: TargetRowRepresentation | null; comparison_status: TargetSurfaceStatus; current: TargetChangeSide | null; candidate: TargetChangeSide | null; @@ -91,6 +92,9 @@ function hierarchyFields(row: TargetRow): Record { source: row.source ?? null, source_label: row.source_label ?? null, variable: row.variable ?? null, + variable_label: row.variable_label ?? null, + target_label: row.target_label ?? null, + target_representation: row.target_representation ?? null, variable_key: row.variable_key ?? null, measure: row.measure ?? null, source_measure_id: row.source_measure_id ?? null, diff --git a/frontend/lib/microcosm/target-matching-presentation.test.ts b/frontend/lib/microcosm/target-matching-presentation.test.ts index b6adbcd..aae26fd 100644 --- a/frontend/lib/microcosm/target-matching-presentation.test.ts +++ b/frontend/lib/microcosm/target-matching-presentation.test.ts @@ -13,9 +13,7 @@ describe("target matching presentation", () => { expect(targetMatchKindExplanation("chronicle_fact_key")).toBe( "Matched by exact Chronicle fact key.", ); - expect(targetMatchKindExplanation("structured_identity")).toContain( - "structured source ID", - ); + expect(targetMatchKindExplanation("structured_identity")).toContain("provider ID"); expect(targetMatchKindExplanation(null)).toBe("Not matched across releases."); }); @@ -24,6 +22,8 @@ describe("target matching presentation", () => { "Legacy format → Structured format", ); expect(targetRepresentationPairLabel(null, "structured")).toBe("Structured format"); + expect(targetRepresentationPairLabel("structured", "hierarchy")).toBe( + "Structured format → Hierarchy format", + ); }); - }); diff --git a/frontend/lib/microcosm/target-matching-presentation.ts b/frontend/lib/microcosm/target-matching-presentation.ts index 0fc728a..614e3b8 100644 --- a/frontend/lib/microcosm/target-matching-presentation.ts +++ b/frontend/lib/microcosm/target-matching-presentation.ts @@ -7,6 +7,7 @@ import type { export function targetRepresentationLabel( representation: TargetRepresentation | TargetRowRepresentation, ): string { + if (representation === "hierarchy") return "Hierarchy format"; if (representation === "structured") return "Structured format"; if (representation === "legacy") return "Legacy format"; if (representation === "mixed") return "Mixed formats"; @@ -27,7 +28,7 @@ export function targetMatchKindExplanation(kind: TargetMatchKind | null): string if (kind === "base_name") return "Matched by period-normalized target name."; if (kind === "chronicle_fact_key") return "Matched by exact Chronicle fact key."; if (kind === "structured_identity") { - return "Matched by structured source ID, statistic ID, measure, and raw dimensions."; + return "Matched by source or provider ID, variable or category ID, measure, and raw dimensions."; } return "Not matched across releases."; } diff --git a/frontend/lib/microcosm/target-representation.test.ts b/frontend/lib/microcosm/target-representation.test.ts index a72c482..0bc5c4d 100644 --- a/frontend/lib/microcosm/target-representation.test.ts +++ b/frontend/lib/microcosm/target-representation.test.ts @@ -1,57 +1,24 @@ import { describe, expect, test } from "bun:test"; import { - classifyTargetRow, - classifyTargetRepresentation, - isCompleteStructuredTarget, - isLegacyTarget, + targetRepresentationForSchema, + UnsupportedCalibrationDiagnosticsSchemaError, } from "./target-representation"; -const structured = { - source: { id: "novastat" }, - variable: { id: "population" }, - dimensions: {}, -}; - describe("calibration target representation", () => { - test("reports unknown when there are no target rows", () => { - expect(classifyTargetRepresentation([])).toBe("unknown"); - }); - - test("classifies rows without structured identity fields as legacy", () => { - const rows = [ - { name: "irs.population.total", source: "IRS SOI" }, - { name: "state/ca/population", variable: "population" }, - ]; - expect(rows.every(isLegacyTarget)).toBe(true); - expect(classifyTargetRepresentation(rows)).toBe("legacy"); - }); - - test("requires complete structured identity on every structured row", () => { - expect(isCompleteStructuredTarget(structured)).toBe(true); - expect(isCompleteStructuredTarget({ ...structured, dimensions: undefined })).toBe(false); - expect(isCompleteStructuredTarget({ ...structured, source: { label: "Nova" } })).toBe(false); - expect(isCompleteStructuredTarget({ ...structured, variable: { id: " " } })).toBe(false); - }); - - test("classifies complete structured rows independently of diagnostics schema", () => { - expect(classifyTargetRepresentation([ - { ...structured, schema_version: 5 }, - { ...structured, schema_version: 6 }, - ])).toBe("structured"); - }); - - test("classifies a collection containing complete structured and legacy rows as mixed", () => { - expect(classifyTargetRepresentation([ - structured, - { name: "irs.population.total", source: "IRS SOI" }, - ])).toBe("mixed"); + test("dispatches historical and current readers only by top-level version", () => { + for (const schemaVersion of [2, 3, 4, 5, 6]) { + expect(targetRepresentationForSchema(schemaVersion)).toBe("legacy"); + } + expect(targetRepresentationForSchema(7)).toBe("structured"); + expect(targetRepresentationForSchema(8)).toBe("hierarchy"); }); - test("rejects partially structured rows", () => { - expect(() => classifyTargetRow({ - source: { id: "novastat" }, - variable: "population", - })).toThrow("complete structured identity fields or legacy fields"); + test("rejects versions without a supported compatibility contract", () => { + for (const schemaVersion of [undefined, null, "8", 1, 9, 7.5]) { + expect(() => targetRepresentationForSchema(schemaVersion)).toThrow( + UnsupportedCalibrationDiagnosticsSchemaError, + ); + } }); }); diff --git a/frontend/lib/microcosm/target-representation.ts b/frontend/lib/microcosm/target-representation.ts index 1bd041d..900eebe 100644 --- a/frontend/lib/microcosm/target-representation.ts +++ b/frontend/lib/microcosm/target-representation.ts @@ -1,47 +1,36 @@ -export type TargetRepresentation = "legacy" | "structured" | "mixed" | "unknown"; -export type TargetRowRepresentation = "legacy" | "structured"; +export type TargetRepresentation = + | "legacy" + | "structured" + | "hierarchy" + | "mixed" + | "unknown"; -type JsonObject = Record; +export type TargetRowRepresentation = "legacy" | "structured" | "hierarchy"; -function isPlainObject(value: unknown): value is JsonObject { - return value != null && typeof value === "object" && !Array.isArray(value); -} - -function nonEmptyString(value: unknown): value is string { - return typeof value === "string" && value.trim().length > 0; -} - -export function isCompleteStructuredTarget(row: JsonObject): boolean { - return ( - isPlainObject(row.source) && - nonEmptyString(row.source.id) && - isPlainObject(row.variable) && - nonEmptyString(row.variable.id) && - isPlainObject(row.dimensions) - ); -} +export class UnsupportedCalibrationDiagnosticsSchemaError extends Error {} -export function isLegacyTarget(row: JsonObject): boolean { - return ( - !isPlainObject(row.source) && - !isPlainObject(row.variable) && - !isPlainObject(row.dimensions) +/** + * Select the only target reader permitted for a diagnostics artifact. + * + * Row shape is deliberately not inspected: the top-level schema version is + * the compatibility contract between Microcosm and this dashboard. + */ +export function targetRepresentationForSchema( + schemaVersion: unknown, +): TargetRowRepresentation { + if ( + typeof schemaVersion !== "number" || + !Number.isInteger(schemaVersion) + ) { + throw new UnsupportedCalibrationDiagnosticsSchemaError( + "Calibration diagnostics must declare an integer schema_version.", + ); + } + if (schemaVersion >= 2 && schemaVersion <= 6) return "legacy"; + if (schemaVersion === 7) return "structured"; + if (schemaVersion === 8) return "hierarchy"; + throw new UnsupportedCalibrationDiagnosticsSchemaError( + `Unsupported calibration diagnostics schema_version ${schemaVersion}; ` + + "this dashboard reads versions 2 through 8.", ); } - -export function classifyTargetRow(row: JsonObject): TargetRowRepresentation { - if (isCompleteStructuredTarget(row)) return "structured"; - if (isLegacyTarget(row)) return "legacy"; - throw new Error( - "Calibration target rows must use either complete structured identity fields or legacy fields.", - ); -} - -export function classifyTargetRepresentation( - rows: readonly JsonObject[], -): TargetRepresentation { - if (rows.length === 0) return "unknown"; - const representations = new Set(rows.map(classifyTargetRow)); - if (representations.size > 1) return "mixed"; - return representations.has("structured") ? "structured" : "legacy"; -} diff --git a/frontend/lib/microcosm/target-surface-matcher.test.ts b/frontend/lib/microcosm/target-surface-matcher.test.ts index 7adf1b7..a0222aa 100644 --- a/frontend/lib/microcosm/target-surface-matcher.test.ts +++ b/frontend/lib/microcosm/target-surface-matcher.test.ts @@ -7,18 +7,40 @@ type Row = Calibration["rows"][number]; function calibration( rows: Row[], - representation: "legacy" | "structured" | "mixed" = "legacy", + representation: "legacy" | "structured" | "hierarchy" | "mixed" = "legacy", ): Calibration { return { rows, target_schema: { - diagnostics_schema_version: 7, + diagnostics_schema_version: representation === "hierarchy" ? 8 : 7, structured_dimensions: representation !== "legacy", target_representation: representation, }, } as unknown as Calibration; } +function hierarchy(name: string, valueId: string): Row { + return { + name, + base_name: name, + target_representation: "hierarchy", + dimension_adapter: "hierarchy", + source: "agency", + variable: "agency.population", + target_dimensions: [ + { + key: "age", + label: "Age", + value: "Published age label", + value_id: valueId, + source_key: "age", + raw_value: valueId, + }, + ], + chronicle: { fact_key: null }, + }; +} + function legacy( name: string, options: { baseName?: string; factKey?: string | null } = {}, @@ -112,6 +134,24 @@ describe("target surface matching", () => { expect(result.matching.matched_by.structured_identity).toBe(1); }); + test("matches schema 8 identities from raw hierarchy dimension values", () => { + const result = matchTargetSurfaces( + calibration([hierarchy("old-name", "adult")], "hierarchy"), + calibration([hierarchy("new-name", "adult")], "hierarchy"), + ); + + expect(result.matches[0]).toMatchObject({ + match_kind: "structured_identity", + current_representation: "hierarchy", + candidate_representation: "hierarchy", + }); + expect(result.matching).toMatchObject({ + current_representation: "hierarchy", + candidate_representation: "hierarchy", + matched_by: { structured_identity: 1 }, + }); + }); + test("lets unique fallback keys resolve a duplicated target name", () => { const result = matchTargetSurfaces( calibration([ diff --git a/frontend/lib/microcosm/target-surface-matcher.ts b/frontend/lib/microcosm/target-surface-matcher.ts index 95391dd..0aa76c7 100644 --- a/frontend/lib/microcosm/target-surface-matcher.ts +++ b/frontend/lib/microcosm/target-surface-matcher.ts @@ -71,6 +71,7 @@ function originalTargetName(row: TargetRow): string | null { } function rowRepresentation(row: TargetRow): TargetRowRepresentation { + if (row.target_representation === "hierarchy") return "hierarchy"; if (row.target_representation === "structured") return "structured"; if (row.target_representation === "legacy") return "legacy"; return row.dimension_adapter === "structured" ? "structured" : "legacy"; @@ -81,6 +82,7 @@ function collectionRepresentation(calibration: Calibration): TargetRepresentatio if ( published === "legacy" || published === "structured" || + published === "hierarchy" || published === "mixed" || published === "unknown" ) { @@ -89,6 +91,7 @@ function collectionRepresentation(calibration: Calibration): TargetRepresentatio if (!calibration.rows.length) return "unknown"; const representations = new Set(calibration.rows.map(rowRepresentation)); if (representations.size > 1) return "mixed"; + if (representations.has("hierarchy")) return "hierarchy"; return representations.has("structured") ? "structured" : "legacy"; } @@ -97,16 +100,28 @@ function chronicleFactKey(row: TargetRow): string | null { } function structuredIdentity(row: TargetRow): string | null { - if (rowRepresentation(row) !== "structured") return null; + const representation = rowRepresentation(row); + if (representation !== "structured" && representation !== "hierarchy") { + return null; + } const source = nonEmptyString(row.source); const variable = nonEmptyString(row.variable); if (!source || !variable) return null; - const dimensions = Object.entries(asObject(row.dimensions)) - .flatMap(([id, value]) => { - const rawValue = nonEmptyString(value); - return id && rawValue ? [[id, rawValue] as const] : []; - }) - .sort(([left], [right]) => left.localeCompare(right)); + const dimensions = representation === "hierarchy" + ? (Array.isArray(row.target_dimensions) ? row.target_dimensions : []) + .flatMap((dimension) => { + const id = nonEmptyString(dimension.source_key ?? dimension.key); + const rawValue = nonEmptyString( + dimension.raw_value ?? dimension.value_id ?? dimension.value, + ); + return id && rawValue ? [[id, rawValue] as const] : []; + }) + : Object.entries(asObject(row.dimensions)) + .flatMap(([id, value]) => { + const rawValue = nonEmptyString(value); + return id && rawValue ? [[id, rawValue] as const] : []; + }) + .sort(([left], [right]) => left.localeCompare(right)); return JSON.stringify({ source, variable, diff --git a/frontend/lib/microcosm/third-country-conformance.test.ts b/frontend/lib/microcosm/third-country-conformance.test.ts index 422b79f..b8c3717 100644 --- a/frontend/lib/microcosm/third-country-conformance.test.ts +++ b/frontend/lib/microcosm/third-country-conformance.test.ts @@ -24,11 +24,15 @@ const RELEASE_ID = "microcosm-zz-2026-conformance"; const DESCRIPTION = "Synthetic ZZ release whose dashboard presentation comes from release artifacts."; const PUBLISHER = "novastat_agency"; +const legacyDiagnosticsFixture = { + ...diagnosticsFixture, + schema_version: 2, +}; // Registration-only contract: production presentation/parsing code must not // gain a `zz` branch or table entry to make these data-shape assertions pass. const calibration = buildCalibration( - diagnosticsFixture, + legacyDiagnosticsFixture, RELEASE_ID, "2026-08-23T18:00:00Z", {}, @@ -67,7 +71,7 @@ describe("synthetic third-country conformance", () => { // A release_manifest.country block overrides the registration's labels and // flows through buildCalibration into the summary and the row geography. const overridden = buildCalibration( - diagnosticsFixture, + legacyDiagnosticsFixture, RELEASE_ID, "2026-08-23T18:00:00Z", {}, @@ -212,7 +216,7 @@ describe("synthetic third-country conformance", () => { targets_intro: "Zedland target-browser copy from the release artifact.", }; const futureCalibration = buildCalibration( - diagnosticsFixture, + legacyDiagnosticsFixture, RELEASE_ID, "2026-08-23T18:00:00Z", {}, @@ -231,7 +235,7 @@ describe("synthetic third-country conformance", () => { "zz can override a publisher display name: treemap shaping ignores release_manifest.publisher_labels", () => { const futureCalibration = buildCalibration( - diagnosticsFixture, + legacyDiagnosticsFixture, RELEASE_ID, "2026-08-23T18:00:00Z", {}, @@ -261,7 +265,8 @@ describe("synthetic third-country conformance", () => { { region: "north", sex: "male", age_band: "65_plus" }, ]; const futureDiagnostics = { - ...diagnosticsFixture, + ...legacyDiagnosticsFixture, + schema_version: 7, dimensions: { region: { label: "Region", @@ -278,17 +283,16 @@ describe("synthetic third-country conformance", () => { values: { "0_17": "0–17", "18_64": "18–64", "65_plus": "65+" }, }, }, - targets: diagnosticsFixture.targets.map((target, index) => - index < facetValues.length - ? { - ...target, - source: { id: PUBLISHER, citation: "ZZ official population table" }, - variable: { id: "population", measure: "count" }, - filter: null, - dimensions: facetValues[index], - } - : target, - ), + targets: diagnosticsFixture.targets.map((target, index) => ({ + ...target, + source: { id: PUBLISHER, citation: "ZZ official population table" }, + variable: { + id: index < facetValues.length ? "population" : "revenue", + measure: "count", + }, + filter: null, + dimensions: facetValues[index] ?? {}, + })), }; const futureCalibration = buildCalibration( futureDiagnostics, @@ -324,7 +328,8 @@ describe("synthetic third-country conformance", () => { measure: "count", }; const futureDiagnostics = { - ...diagnosticsFixture, + ...legacyDiagnosticsFixture, + schema_version: 7, dimensions: { region: { label: "Region",