ADR: raw microdata in Chronicle is identity, not content

.github/chronicle-agents.yml

@@ -43,4 +43,4 @@ required_judges:
   ledger-contract:
     verdict: PASS if schema or identity changes preserve source provenance and do not move Microcosm responsibilities into Chronicle.
   ledger-boundary:
-    verdict: PASS if reconciliation, aging, imputation, support-aware activation, and solver construction remain outside Chronicle.
+    verdict: PASS if reconciliation, aging, imputation, support-aware activation, and solver construction remain outside Chronicle; no microdata records, rows, columns, row values, or cells enter any Chronicle parsed-source surface, registry, derived artifact, or journal; no fact derived from raw microdata by Chronicle or by a consumer enters Chronicle, however many intermediate artifacts stand between them, the only exception being an aggregate that the publisher of that microdata computed and published itself; no licensed or restricted microdata bytes enter any Chronicle store, and public microdata bytes enter only with artifact-bound redistribution evidence (docs/adr-chronicle-raw-microdata-identity.md).

AGENTS.md

@@ -19,6 +19,11 @@ Do not put Microcosm work in Chronicle:
 - no solver-ready target construction
 - no target profiles or model-measurement bindings
 - no PolicyEngine-computed values stored as facts
+- no microdata records, rows, columns, row values, or cells, no facts
+  derived from raw microdata by Chronicle or a consumer, and no licensed or
+  restricted microdata bytes in any Chronicle store (registering a release's
+  identity is allowed, and a publisher's own published aggregates remain
+  ordinary facts; see `docs/adr-chronicle-raw-microdata-identity.md`)
 
 Chronicle records every fact's publisher reference period. Consumers own and
 enforce any declaration that aligns those facts to another period; Chronicle

README.md

@@ -12,7 +12,13 @@ Chronicle may normalize structure: parse files, type values, declare units and
 scales, assign geography and period identifiers, preserve lineage back to
 source artifacts, and publish source-backed facts. Chronicle does not own
 selection or measurement contracts, reconcile inconsistent sources, impute
-missing data, store raw survey microdata, or execute simulator-specific calibration.
+missing data, parse survey or administrative microdata into records, rows,
+columns, row values, or cells, derive facts from raw microdata, hold licensed
+or restricted microdata bytes, or execute simulator-specific calibration. Once
+chronicle#221 lands, Chronicle registers the microdata releases its consumers
+build from (publisher, vintage, checksum, licence) and archives the bytes of
+public-use files that carry artifact-bound redistribution evidence; see
+[`docs/adr-chronicle-raw-microdata-identity.md`](docs/adr-chronicle-raw-microdata-identity.md).
 
 Microcosm consumes Chronicle facts, owns the contracts that select and bind
 them, applies declared period alignment, and runs calibration. Thesis can
@@ -49,7 +55,10 @@ The load-bearing rule:
 > consumer or transform it in ways that change its meaning.
 
 The store is facts-only, and the line is who asserted the value. Everything a
-publisher asserted — including the publisher's own projections — is a fact.
+publisher asserted — including the publisher's own projections — is a fact,
+except publisher-authored values at microdata grain, which are content and
+never facts; only the publisher's own published aggregates over its microdata
+are facts (see `docs/adr-chronicle-raw-microdata-identity.md`).
 Everything PolicyEngine computes (aged, uprated, forecast, or reconciled
 levels) is a downstream build artifact and never enters the store; Microcosm
 owns aging as a named, versioned model over Chronicle growth-factor facts. A

chronicle/__init__.py

@@ -1,9 +1,12 @@
 """Chronicle source-data foundation.
 
-Chronicle owns government-statistics releases: source artifacts, source-backed
-facts, constraints, and provenance. Selection contracts, raw microdata storage,
-source reconciliation, aging, imputation, target activation, and calibration
-belong in downstream systems such as Microcosm.
+Chronicle owns government-statistics releases: source artifacts (and, once
+chronicle#221 lands, registrations of raw microdata releases plus custody of
+public-use bytes with redistribution evidence), source-backed facts, constraints, and provenance. Microdata
+content (records, rows, columns, row values, cells, and facts derived from raw
+microdata), licensed or restricted microdata bytes, selection contracts, source reconciliation, aging,
+imputation, target activation, and calibration belong in downstream systems
+such as Microcosm.
 """
 
 __all__ = [

chronicle/boundary.py

@@ -0,0 +1,24 @@
+"""The microdata boundary, stated once so docs, the judge contract, and tests agree.
+
+``docs/adr-chronicle-raw-microdata-identity.md`` decides that Chronicle registers
+raw microdata releases by identity and stores none of their content. These are
+the three negative clauses that decision reduces to. ``.github/chronicle-agents.yml``
+carries them verbatim in the ``ledger-boundary`` judge verdict, and
+``tests/test_chronicle_governance.py`` asserts that it does, so the wording cannot
+regress or invert without failing the suite.
+"""
+
+from __future__ import annotations
+
+MICRODATA_BOUNDARY_CLAUSES: tuple[str, ...] = (
+    "no microdata records, rows, columns, row values, or cells enter any "
+    "Chronicle parsed-source surface, registry, derived artifact, or journal",
+    "no fact derived from raw microdata by Chronicle or by a consumer enters "
+    "Chronicle, however many intermediate artifacts stand between them, the only "
+    "exception being an aggregate that the publisher of that microdata computed "
+    "and published itself",
+    "no licensed or restricted microdata bytes enter any Chronicle store, and "
+    "public microdata bytes enter only with artifact-bound redistribution evidence",
+)
+
+__all__ = ["MICRODATA_BOUNDARY_CLAUSES"]

docs/adr-chronicle-facts-only.md

@@ -9,7 +9,11 @@ Chronicle stores source-published values only. The boundary is **who asserted
 the value**, not level versus projection:
 
 1. Anything a publisher asserted is a fact — including the publisher's own
-   projections. "CBO's January 2026 baseline projects individual income tax
+   projections — with one grain exception added by
+   `docs/adr-chronicle-raw-microdata-identity.md`: a publisher-authored
+   value at microdata grain (a record, row, column, row value, or cell of a
+   microdata release) is content, not a fact; only the aggregates the
+   publisher itself computed and published over that microdata are facts. "CBO's January 2026 baseline projects individual income tax
    receipts of $X in 2027" is a source-backed claim with lineage, exactly
    like an SOI observation. These facts carry
    `assertion: source_projection`; measured or administered outcomes carry

docs/adr-chronicle-raw-microdata-identity.md

@@ -0,0 +1,152 @@
+# ADR: Raw microdata in Chronicle is identity, not content
+
+Status: accepted 2026-09-02. Enforcement lands in two steps and this ADR's
+guarantees are prospective until they merge: chronicle#221 (PR #227: manifest
+classification, the access-aware refusals in `fetch-artifact`, `publish-raw`,
+`register-artifact`, and the source-package byte reader, the redistribution
+allowlist, artifact-bound licence evidence, untracked staging for public
+microdata bytes) and chronicle#238 (asserting principal and root-artifact
+lineage on fact provenance). Until #227 merges, no microdata release may be
+pointed at any Chronicle command: today they materialize, upload, and parse
+every input. Until #238 merges, the derived-fact prohibition is a review
+obligation, not a validator.
+
+## Decision
+
+Chronicle registers every raw microdata release its consumers build from, and
+stores none of their content. "Content" means any parsed representation:
+microdata records, rows, columns, row values, or cells, and any fact derived
+from them by Chronicle or a consumer. Custody of a public-use file's bytes
+under artifact-bound redistribution evidence is not content; custody of
+licensed or restricted bytes is never taken at all.
+
+1. **Classification is explicit.** Every manifest created or modified after
+   this ADR declares `kind`, either `publisher_table` or
+   `microdata_release`. `validate-package`, `fetch-artifact`, `publish-raw`,
+   `register-artifact`, and the source-package byte reader refuse a new or
+   modified manifest without it, at every entry point. The manifests that
+   exist at #227's merge commit are grandfathered as publisher tables by an
+   explicit frozen list checked into the repository; a kindless manifest
+   outside that list is an error, never a publisher table by default.
+   `microdata_release` is what lets every parser refuse the file, so a public
+   microdata release can never be mistaken for a public aggregate workbook.
+2. **Registration.** A microdata release (CPS ASEC, ACS PUMS, SCF, SIPP, FRS,
+   BE-SILC, the IRS PUF, and their successors in every jurisdiction) is a
+   source artifact registered by identity: publisher, source URL or access
+   route, vintage, SHA-256, size, licence, an access class from a closed set
+   (`public`, `licensed`, `restricted`), and the attestation fields in
+   decision 5. One package per publisher release, its files listed under the
+   vintage year; a registration is identified by
+   `{source_id, package_id, year, sha256, filename}`. Keys follow the
+   convention in `docs/storage-architecture.md`:
+   `raw/{country}/{source_id}/{package_id}/{year}/{sha256}/{filename}` for
+   publishers mapped to a country segment (UK and New Zealand today) and the
+   legacy `raw/{source_id}/{package_id}/{year}/{sha256}/{filename}` shape for
+   every unmapped publisher, US sources included. No source package parses
+   the file.
+3. **Bytes only with artifact-bound redistribution evidence.** Being
+   downloadable is not a licence, and an allowlisted licence name is not
+   evidence that this file was issued under it. Chronicle archives a release's
+   bytes only when `access` is `public`, the recorded `licence` is on
+   Chronicle's allowlist of redistributable terms (maintained in code with the
+   evidence for each term: a U.S. Government work under 17 U.S.C. §105, the
+   Open Government Licence v3, CC0, CC BY), and the entry carries
+   `licence_evidence` binding the artifact to the term: issuer, licence
+   identifier and version, the scope statement, a durable evidence URL, and
+   the covered SHA-256. The `ledger-boundary` judge verifies that the evidence
+   names the artifact. Any public-download file without that evidence is
+   classed `licensed`. Licensed or restricted files (FRS under the UKDS
+   agreement, BE-SILC scientific-use files, the IRS PUF) are registered
+   hash-only: no bytes in any Chronicle store, no Chronicle credential grants
+   access to them, and their bytes stay in the licensed environments
+   consumers already operate. Public microdata bytes are staged in an
+   untracked, transient directory outside `db/data/**` during acquisition and
+   uploaded from there; a repository guard refuses tracked microdata bytes.
+   Git custody of a microdata release is always manifest-only.
+4. **No content.** No microdata record, row, column, row value, or cell
+   enters `source_records`, `source_rows`, `source_columns`,
+   `source_row_values`, `source_cells`, the relational registry, the
+   derived-artifact bucket, or the journal, whether the release is public or
+   gated. No fact derived from raw microdata by Chronicle or by a consumer
+   enters Chronicle, however many intermediate artifacts stand between the
+   microdata and the value. The only exception is assertion-based: an
+   aggregate that the publisher of that microdata computed and published
+   itself is an ordinary fact with ordinary provenance. This is enforceable
+   only once facts carry an asserting principal and root-artifact lineage
+   (`asserted_by`, `root_artifacts`; chronicle#238): a fact rooted in a
+   `microdata_release` registration is then rejected unless `asserted_by` is
+   that release's publisher. Until then reviewers enforce it by hand.
+5. **What a registration attests, and who.** Each registration records
+   `hash_source` and its attester:
+   - `chronicle_fetch`: Chronicle fetched the bytes and hashed them;
+     `attested_by: chronicle`, `verified_at` is the fetch date. The
+     registration attests the bytes.
+   - `consumer_attested`: a consumer that holds the bytes recomputed the hash
+     and recorded evidence; `attested_by` is that consumer, `attestation_evidence`
+     points at its record, `verified_at` is its comparison date. The
+     registration attests the bytes on that consumer's word.
+   - `consumer_pin`: transcribed from a consumer's reviewed pin without
+     recomputation; `attested_by` is that consumer, `pinned_from` names the
+     repository path and commit, and there is no `verified_at`. The
+     registration attests the pin, not the bytes.
+   The witnessed journal (the release manifests on `codex/thesis-ledger-facts`,
+   each committing to the journal state that first covers a registration's
+   manifest hash) bounds when the registration existed. It bounds when the
+   bytes existed only for `chronicle_fetch` and `consumer_attested` entries.
+6. **Consumers point at the registration.** A Microcosm source-stage manifest
+   that names a microdata artifact carries the Chronicle artifact reference
+   and the same SHA-256, so every root of a build graph resolves to one
+   witnessed registration and a build fails closed when its local bytes
+   differ from the registered ones.
+
+## Why
+
+- **The publisher record is the thing to witness.** Publishers revise and
+  withdraw microdata files: the IRS withdrew the public-use file in 2026, and
+  Census reissues ASEC files under the same vintage label. A registered hash
+  with a witnessed time is the only durable statement that a given release
+  existed with those bytes. This is the same transparency property Chronicle
+  already provides for published tables, applied to the files calibration
+  actually starts from.
+- **Pins today are scattered and unwitnessed.** Microcosm pins raw inputs in
+  per-country manifests, checkpoint metadata, code constants, and command-line
+  arguments, with no shared registry, no licence record, and no timestamp
+  anyone outside the build can check. A build's root inputs deserve the same
+  declared identity as every other node.
+- **Content would break what makes the store useful.** Row-level microdata
+  would grow the relational registry and journal by orders of magnitude,
+  churn on every reissue, and put access-controlled bytes inside the one
+  system whose value is that anyone can verify it. Thesis resolves forecasts
+  against Chronicle observations; microdata are not observations of anything
+  Thesis scores.
+- **This keeps the 2026-06-30 ruling.** PR #68 removed microdata parsers,
+  adapters, and tracked raw storage from the package. Nothing here brings any
+  of that back. The code the enforcement PRs add refuses to read microdata
+  rather than reading it.
+
+## Consequences
+
+- `docs/adr-chronicle-facts-only.md` is amended: "anything a publisher
+  asserted is a fact" now excludes publisher-authored values at microdata
+  grain, which are content; only the publisher's published aggregates over
+  its microdata are facts. README's boundary block says the same.
+- Manifests gain `kind`, `licence`, `access`, `licence_evidence`,
+  `hash_source`, `attested_by`, and per-source `verified_at` /
+  `attestation_evidence` / `pinned_from`; validators refuse the combinations
+  decisions 1, 3, and 5 forbid (#227).
+- Fact provenance gains `asserted_by` and `root_artifacts` (#238).
+- `docs/storage-architecture.md` narrows its non-goal to the three clauses in
+  `chronicle/boundary.py`; its ownership matrix gains a row for microdata
+  releases and states that Git custody is manifest-only with untracked
+  staging. `docs/target-construction-harness-plan.md` is scoped to publisher
+  aggregate artifacts.
+- `chronicle/boundary.py` states the three negative clauses once;
+  `.github/chronicle-agents.yml` carries them verbatim in the
+  `ledger-boundary` judge verdict; `tests/test_chronicle_governance.py`
+  asserts each complete clause, so the wording cannot regress or invert.
+- Microcosm's raw-input entries reference Chronicle registrations
+  (PolicyEngine/microcosm#848).
+- Flip conditions for revisiting content: a consumer other than Microcosm
+  needs row-level evidence from Chronicle, or a publisher grants
+  redistribution of a currently licensed file and Chronicle has an
+  access-controlled tier designed for it. Neither holds today.

docs/architecture.md

@@ -16,7 +16,7 @@ validation, and build-harness layer. Jurisdiction source packages such as
 
 | Layer | Owns | Does not own |
 |-------|------|--------------|
-| Chronicle | Source artifacts, provenance, aggregate facts, constraints | Selection and measurement contracts, raw microdata storage, source reconciliation, aging, imputation, active target selection |
+| Chronicle | Source artifacts (including microdata release registrations), provenance, aggregate facts, constraints | Selection and measurement contracts, microdata records, rows, columns, row values, or cells; facts derived from raw microdata by Chronicle or a consumer; licensed or restricted microdata bytes; source reconciliation, aging, imputation, active target selection |
 | Microcosm Targets | Selection and measurement contracts, reconciliation, aging, imputation, active target sets | Source artifact storage and provenance |
 | Microcosm | Entity model, weights, calibration interfaces, calibrated output | Source ETL and source provenance |
 | Jurisdiction source packages | Source-specific parsers and specs that emit Chronicle records | Forked fact or constraint schemas |

docs/chronicle-governance.md

@@ -20,6 +20,10 @@ without scoring model output against model output.
 Chronicle may:
 
 - register raw publisher artifacts and checksums
+- register raw microdata releases as source artifacts (publisher, access
+  route, vintage, checksum, licence, hash source, verification date),
+  archiving bytes only when the release is `public` and its licence is on
+  the redistributable allowlist
 - parse source rows and cells
 - emit source-backed aggregate facts, including publisher projections typed
   `assertion: source_projection`
@@ -37,7 +41,10 @@ Chronicle must not:
   levels) as facts or in any other store object
 - compute aligned values; it publishes the source period and value unchanged
 - impute missing values
-- store raw survey or administrative microdata
+- parse survey or administrative microdata into records, rows, columns, row
+  values, or cells, derive facts from raw microdata (by Chronicle or a
+  consumer), or hold licensed or restricted microdata bytes in any Chronicle
+  store
 - own selection, measurement, period-alignment, or model-binding contracts
 - choose a support-aware active target subset
 - build solver-ready calibration targets
@@ -65,5 +72,6 @@ types are:
 
 The overall verdict fails if any required judge fails. A judge must fail if a
 change moves reconciliation, aging, imputation, active target selection, or
-solver construction from Microcosm into Chronicle, or stores a
-PolicyEngine-computed value as a fact.
+solver construction from Microcosm into Chronicle, stores a
+PolicyEngine-computed value as a fact, or breaks any clause of the microdata
+boundary in `chronicle/boundary.py`, quoted here verbatim: (1) no microdata records, rows, columns, row values, or cells enter any Chronicle parsed-source surface, registry, derived artifact, or journal; (2) no fact derived from raw microdata by Chronicle or by a consumer enters Chronicle, however many intermediate artifacts stand between them, the only exception being an aggregate that the publisher of that microdata computed and published itself; (3) no licensed or restricted microdata bytes enter any Chronicle store, and public microdata bytes enter only with artifact-bound redistribution evidence.