Artifact presentation, publisher labels, and structured dimensions (spec-driven countries 2/2)

docs/spec-driven-countries.md

@@ -11,23 +11,27 @@ role/default status, and the optional provenance `description`; demographics own
 geography coverage. A description may fill the existing provenance note, but an
 artifact cannot add sections or choose components.
 
-Target sources are derived generically from the first dotted segment of
-`metadata.chronicle_record_ids`, with one shared publisher-label map and a
-humanized fallback for unknown prefixes. Filter-coded target facets may be
-decoded by a country-agnostic pattern spec. Cross-dataset comparisons are not
-part of this release contract: they are served only by `/microcosm/datasets`
-from `cross_dataset.frontend_bundle.v1`.
+Target source IDs are derived generically from the first dotted segment of
+`metadata.chronicle_record_ids`, with structured source IDs and legacy name
+parsing as fallbacks. Artifact publisher labels take precedence over the shared
+publisher-label map and humanized unknown prefixes. Filter-coded target facets
+may be decoded by a country-agnostic pattern spec when a row does not publish
+structured dimensions. Cross-dataset comparisons are not part of this release
+contract: they are served only by `/microcosm/datasets` from
+`cross_dataset.frontend_bundle.v1`.
 
 ## Code-owned metadata that remains
 
 `COUNTRY_REGISTRY` in `frontend/lib/microcosm/countries.ts` is the only required
 registration point: country unions and parsers, selector lists, dataset and
 national-geography labels, public/private link behavior, and page gating read
 the registration or the artifact `country` block. Overview and target-browser
-copy still sit in legacy per-country tables with a generic fallback. Publisher
-display names remain in a shared TypeScript map. Target decomposition still has a
-generic filter-pattern table plus legacy US name grammar (FIPS/state, filing
-status, return type, income band, and qualifying-child rules).
+copy retains marked per-country tables only as fallbacks for releases without
+artifact `presentation`. Publisher display names retain a shared TypeScript map
+as the fallback for publisher keys absent from artifact `publisher_labels`.
+Target decomposition retains a generic filter-pattern table plus legacy US name
+grammar (FIPS/state, filing status, return type, income band, and
+qualifying-child rules) for rows without structured dimensions.
 
 These tables are presentation or parsing metadata, not calibration logic. They
 should not grow another country branch.
@@ -47,9 +51,10 @@ produce:
   unknown-prefix fallback; and
 - filter-derived geography and target dimensions with stable labels and values.
 
-Assertions that expose a current contract gap are `test.todo` with a one-line
-reason. The todos are the migration backlog; conformance is complete when all can
-be enabled without adding `zz` conditionals or tables.
+The conformance suite includes passing assertions for artifact presentation,
+publisher labels, structured dimensions, and structured source and variable
+identifiers. It has no remaining todos and requires no `zz` conditionals or
+production tables.
 
 ## Schema additions that remove the tables
 
@@ -61,7 +66,9 @@ be enabled without adding `zz` conditionals or tables.
 | Filter-pattern decomposition, region/sex/age value maps, and legacy US target-name parsing | A `dimensions` dictionary in `calibration_diagnostics` (label, semantic role, value labels, ordering) plus `targets[].dimensions` values. Geography dimensions also declare their level/id so no country geography fallback is needed. |
 | Source/variable guesses from flat target names | Structured `targets[].source` and `targets[].variable` identifiers, with the publisher still traceable to `metadata.chronicle_record_ids`. |
 
-### Implemented: the `country` block
+## Contract as implemented
+
+### Country
 
 `release_manifest.country` is read by `releaseCountry` in
 `frontend/lib/microcosm/latest-artifact.ts` and served as `country` on the
@@ -101,5 +108,184 @@ country for parsers and builders but is never listed in selectors or the
 release-alert allowlist.
 
 Schema readers must remain backward-compatible while published releases migrate.
-After migration, name/filter parsing is a legacy adapter selected by artifact
-schema version, never by country.
+Name/filter parsing is a legacy adapter selected from each row's artifact shape,
+never by country.
+
+### Presentation
+
+`release_manifest.presentation` fills only the two existing overview and
+target-browser introduction slots:
+
+```json
+{
+  "presentation": {
+    "overview_intro": "Release-owned overview introduction.",
+    "targets_intro": "Release-owned target-browser prompt."
+  }
+}
+```
+
+`releasePresentation` accepts only these keys when their values are non-empty
+strings. Values are trimmed and capped at 600 characters; unknown keys and
+malformed values are dropped. The reader returns `null` when neither slot is
+valid. The typed block is returned as `presentation` on the calibration summary
+and target-diagnostics page.
+
+Each view resolves copy in this order: the artifact slot, the marked legacy
+US/UK/BE copy, then generic copy. The overview's live-source attribution
+sentence remains code-owned. The legacy tables can be deleted after all three
+existing producers publish `presentation`.
+
+### Publisher labels
+
+`release_manifest.publisher_labels` maps the first Chronicle record-ID segment
+to its display name:
+
+```json
+{
+  "publisher_labels": {
+    "novastat_agency": "Nova Statistics Agency"
+  }
+}
+```
+
+The block must be a plain object. Keys must match
+`^[a-z][a-z0-9_]*$` case-insensitively, and values must be non-empty strings;
+values are trimmed and invalid entries are dropped. An absent or malformed
+block becomes `{}`. Every enriched target row carries `source_label`, and the
+field also appears on target responses and variable summaries. Treemap,
+calibration-tree, target-browser, and target-detail presentations use the row
+label when available. Label precedence is the manifest map, then a structured
+row's `source.label`, then the shared authority humanizer.
+
+### Structured dimensions
+
+`calibration_diagnostics.json` may publish a dimension dictionary and a
+dimension-value object on each target:
+
+```json
+{
+  "schema_version": 7,
+  "dimensions": {
+    "region": {
+      "label": "Region",
+      "role": "geography",
+      "level": "region",
+      "values": {
+        "north": "North",
+        "south": "South"
+      },
+      "order": ["north", "south"]
+    },
+    "sex": {
+      "label": "Sex",
+      "values": {
+        "female": "Female",
+        "male": "Male"
+      }
+    },
+    "age_band": {
+      "label": "Age band"
+    }
+  },
+  "targets": [
+    {
+      "dimensions": {
+        "region": "north",
+        "sex": "female",
+        "age_band": "0_17"
+      }
+    }
+  ]
+}
+```
+
+`diagnosticsDimensions` requires a plain-object dictionary and a non-empty
+string `label` on each retained entry. It recognizes only `"geography"` as a
+semantic `role` today. Optional `level` and string value labels are trimmed;
+malformed optional entries are dropped without throwing. An explicit `order`
+array overrides `values` key order. Unknown target dimension IDs remain valid:
+their IDs and raw values are humanized. When an age-band value lacks an
+artifact label, range values such as `0_17` and `65_plus` become `0–17` and
+`65+`.
+
+A geography-role dimension sets `row.geography` and uses its declared `level`
+or `"region"`. Other dimensions become `target_dimensions` with `key`,
+`label`, `value`, `source_key`, `raw_value`, and an optional zero-based `rank`.
+Facet values use rank order only when every displayed value has a rank;
+otherwise the legacy facet sorter remains in force. Structured rows are also
+excluded from whole-population estimate-scope inference.
+
+Adapter selection is per row and never by country:
+
+1. A plain-object `targets[].dimensions` selects `structured`, even when the
+   dictionary is absent or the row object is empty.
+2. Otherwise, a filter matched by the shared filter-decomposition specs selects
+   `legacy_filter`.
+3. Every other row selects `legacy_name`, preserving dotted/slash name parsing
+   and Chronicle metadata dimensions.
+
+Each target response records this choice as `dimension_adapter`. Calibration
+summary and target-diagnostics responses include:
+
+```json
+{
+  "target_schema": {
+    "diagnostics_schema_version": 7,
+    "structured_dimensions": true
+  }
+}
+```
+
+`structured_dimensions` reports whether the diagnostics published a plain
+dimension dictionary; it does not choose every row's adapter.
+
+### Structured source and variable identifiers
+
+Targets accept their legacy strings or the following objects:
+
+```json
+{
+  "targets": [
+    {
+      "source": {
+        "id": "novastat_agency",
+        "citation": "ZZ official population table",
+        "label": "Nova Statistics Agency",
+        "url": "https://stats.example/zz/pop"
+      },
+      "variable": {
+        "id": "population",
+        "label": "Resident population",
+        "measure": "count"
+      }
+    }
+  ]
+}
+```
+
+Publisher-key precedence is Chronicle metadata, then `source.id`, then legacy
+name grammar. `source_citation` is the legacy source string or
+`source.citation`; `source_url` is the structured URL or `null`. The publisher
+label precedence is the manifest map, `source.label`, then the shared
+humanizer.
+
+Variable-ID precedence is `variable.id`, `metadata.variable`, then the existing
+artifact/name fallbacks. A structured ID remains an identifier; its display
+name is separately returned as `variable_label`. Measure precedence is
+`variable.measure`, then the existing first-dimension and Chronicle metadata
+logic. Legacy string sources and variables retain their existing values,
+families, citations, facets, and grouping. The new response fields are
+additive: `source_label`, `source_url`, `variable_label`, and
+`dimension_adapter`.
+
+### Producer follow-up
+
+Microcosm release producers must publish all of the following before the legacy
+presentation and parsing adapters can be retired:
+
+- `release_manifest.country`;
+- `release_manifest.presentation`;
+- `release_manifest.publisher_labels`;
+- `calibration_diagnostics.dimensions` and `targets[].dimensions`; and
+- structured `targets[].source` and `targets[].variable` objects.

frontend/components/microcosm/microcosm-overview-view.tsx

@@ -23,7 +23,7 @@ import {
   useMicrocosm,
   useMicrocosmReleases,
 } from "@/lib/api/hooks/use-microcosm";
-import type { MicrocosmCountry } from "@/lib/microcosm/countries";
+import { microcosmOverviewIntro } from "@/lib/microcosm/presentation";
 import {
   microcosmPublicationUrl,
   microcosmSourceAttribution,
@@ -41,26 +41,6 @@ function formatPublishedAt(value: string | null | undefined): string {
   });
 }
 
-// Legacy per-country intro copy for releases published before a typed
-// `release_manifest.presentation` block; countries without an entry get the
-// generic sentences below.
-const COUNTRY_OVERVIEW_COPY: Partial<
-  Record<MicrocosmCountry, { authorities: string; examples: string }>
-> = {
-  us: {
-    authorities: "the IRS, the Census Bureau, and CMS",
-    examples: "EITC statistics, population, and Medicaid enrollment",
-  },
-  uk: {
-    authorities: "the ONS, OBR, and HMRC",
-    examples: "population by region and age, household types, and tax receipts",
-  },
-  be: {
-    authorities: "Statbel, ONSS, JRC, and SFPD",
-    examples: "population by region, sex, and age band, tax receipts, and benefit totals",
-  },
-};
-
 type LossKind = "normalized_target_loss" | "raw_optimizer_objective" | undefined;
 
 function isNormalizedLoss(kind: LossKind): boolean {
@@ -129,7 +109,7 @@ export function MicrocosmOverviewView() {
     cal.country?.repository_visibility,
   );
   const publicationUrl = microcosmPublicationUrl(data.source_repo, data.release_id);
-  const overviewCopy = COUNTRY_OVERVIEW_COPY[country];
+  const overviewIntro = microcosmOverviewIntro(country, cal.presentation);
 
   return (
     <div className="flex flex-col gap-5">
@@ -139,14 +119,7 @@ export function MicrocosmOverviewView() {
         title="What the data is anchored to"
         description={
           <>
-            Microcosm reweights survey microdata so it matches official statistics
-            from{" "}
-            {overviewCopy
-              ? `agencies like ${overviewCopy.authorities}`
-              : "national statistical agencies and administrative sources"}
-            . Each tile in the Calibration fit explorer below is a category we
-            calibrate to{overviewCopy ? `, including ${overviewCopy.examples}` : ""}.
-            Data is built live from{" "}
+            {overviewIntro} Data is built live from{" "}
             {sourceAttribution.href ? (
               <a
                 className="underline decoration-dotted underline-offset-2"

frontend/components/microcosm/microcosm-target-detail.test.ts

@@ -83,6 +83,23 @@ describe("MicrocosmTargetDetail", () => {
     expect(markup).toContain("Not available");
   });
 
+  test("prefers artifact labels and direct source URLs", () => {
+    const markup = render({
+      ...TARGET,
+      source: "novastat_agency",
+      source_label: "Nova Statistics Agency",
+      source_url: "https://stats.example/zz/pop",
+      variable: "population",
+      variable_label: "Resident population",
+    });
+
+    expect(markup).toContain(">Resident population</h2>");
+    expect(markup).toContain("Nova Statistics Agency");
+    expect(markup).toContain("Source link</dt><dd");
+    expect(markup).toContain('href="https://stats.example/zz/pop"');
+    expect(markup).toContain("View official source");
+  });
+
   test("renders the geography level in sentence case", () => {
     const markup = render({
       ...TARGET,

frontend/components/microcosm/microcosm-target-detail.tsx

@@ -47,6 +47,7 @@ function periodText(row: MicrocosmTargetRow): string {
 function measureText(row: MicrocosmTargetRow): string {
   return (
     titleFromIdentifier(row.chronicle?.measure_concept) ||
+    row.variable_label ||
     canonicalLabel(row.variable as string) ||
     titleFromIdentifier(row.chronicle?.layout_measure_id) ||
     canonicalLabel(row.measure_name)
@@ -427,11 +428,15 @@ export function MicrocosmTargetDetail({
       ? row.abs_relative_error <= 0.1
       : null;
   const chronicle = row.chronicle;
-  const chronicleEntryUrl = chronicleSourceEntryUrl(
-    row.source_citation,
-    chronicle?.layout_record_set_id,
-  );
-  const sourceName = row.source ? sourceLabel(row.source) : "Source not specified";
+  const officialSourceUrl =
+    row.source_url ??
+    chronicleSourceEntryUrl(
+      row.source_citation,
+      chronicle?.layout_record_set_id,
+    );
+  const sourceName =
+    row.source_label ??
+    (row.source ? sourceLabel(row.source) : "Source not specified");
   const measure = measureText(row) || titleFromIdentifier(row.name) || "Calibration target";
 
   const rootRef = useRef<HTMLElement>(null);
@@ -595,16 +600,18 @@ export function MicrocosmTargetDetail({
             <DetailSection title="Official source">
               <DefinitionItem label="Source" value={sourceName} />
               <DefinitionItem
-                label="Chronicle entry"
+                label={row.source_url ? "Source link" : "Chronicle entry"}
                 value={
-                  chronicleEntryUrl ? (
+                  officialSourceUrl ? (
                     <a
-                      href={chronicleEntryUrl}
+                      href={officialSourceUrl}
                       target="_blank"
                       rel="noreferrer"
                       className="text-primary hover:underline"
                     >
-                      View Chronicle entry ↗
+                      {row.source_url
+                        ? "View official source ↗"
+                        : "View Chronicle entry ↗"}
                     </a>
                   ) : (
                     "Not available"