From 3eb1f4aa4850eb7b7f97db276e5e01eb557a9f10 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Wed, 2 Sep 2026 10:46:39 -0400 Subject: [PATCH 01/11] ADR: raw microdata in Chronicle is identity, not content Register every raw microdata release consumers build from (publisher, vintage, sha256, licence, access class) as a source artifact; archive bytes only for redistributable public-use files; never parse microdata into rows or facts and never hold licensed bytes. Narrows the storage and governance non-goals accordingly and points README, architecture, and AGENTS at the ADR. Co-Authored-By: Claude Fable 5.1 --- README.md | 7 +- docs/adr-chronicle-raw-microdata-identity.md | 82 ++++++++++++++++++++ 2 files changed, 88 insertions(+), 1 deletion(-) create mode 100644 docs/adr-chronicle-raw-microdata-identity.md diff --git a/README.md b/README.md index 576919c..0f8509a 100644 --- a/README.md +++ b/README.md @@ -12,7 +12,12 @@ Chronicle may normalize structure: parse files, type values, declare units and scales, assign geography and period identifiers, preserve lineage back to source artifacts, and publish source-backed facts. Chronicle does not own selection or measurement contracts, reconcile inconsistent sources, impute -missing data, store raw survey microdata, or execute simulator-specific calibration. +missing data, parse survey or administrative microdata into rows or facts, hold +licensed microdata bytes, or execute simulator-specific calibration. Chronicle +does register the microdata releases its consumers build from (publisher, +vintage, checksum, licence) and archives the bytes of redistributable public-use +files; see +[`docs/adr-chronicle-raw-microdata-identity.md`](docs/adr-chronicle-raw-microdata-identity.md). Microcosm consumes Chronicle facts, owns the contracts that select and bind them, applies declared period alignment, and runs calibration. Thesis can diff --git a/docs/adr-chronicle-raw-microdata-identity.md b/docs/adr-chronicle-raw-microdata-identity.md new file mode 100644 index 0000000..fb8554a --- /dev/null +++ b/docs/adr-chronicle-raw-microdata-identity.md @@ -0,0 +1,82 @@ +# ADR: Raw microdata in Chronicle is identity, not content + +Status: accepted 2026-09-02 (amends the facts-only store; narrows the +"no raw microdata" non-goal in `docs/storage-architecture.md` and +`docs/chronicle-governance.md`) + +## Decision + +Chronicle registers every raw microdata release its consumers build from, and +stores none of their content. + +1. **Registration.** A microdata release (CPS ASEC, ACS PUMS, SCF, SIPP, FRS, + BE-SILC, the IRS PUF, and their successors in every jurisdiction) is a + source artifact like any publisher workbook: publisher, source URL or + access route, vintage, SHA-256, size, fetch time, licence, and an access + class from a closed set (`public`, `licensed`, `restricted`). Registration + is manifest-level. It uses the existing `fetch-artifact` and `publish-raw` + path and the content-addressed key shape + `raw/{source_id}/{package_id}/{year}/{sha256}/{filename}`. No source + package parses the file. +2. **Bytes only where the publisher permits redistribution.** Public-use files + whose terms allow redistribution (Census public-use files are the model + case) are archived in the raw bucket under that key. Licensed or restricted + files (FRS under the UKDS agreement, BE-SILC scientific-use files, the IRS + PUF) are registered hash-only: no bytes in any Chronicle store, and no + Chronicle credential grants access to them. Their bytes stay in the + licensed environments consumers already operate. +3. **No rows, no facts.** No microdata record, column, or cell enters + `source_rows`, `source_cells`, the relational registry, or the journal. No + fact is minted from microdata by Chronicle. An aggregate a publisher + computes from its own microdata and publishes is an ordinary fact; an + aggregate a consumer computes from microdata is that consumer's artifact. +4. **Consumers point at the registration.** A Microcosm source-stage manifest + that names a microdata artifact carries the Chronicle artifact reference + and the same SHA-256, so every root of a build graph resolves to one + witnessed registration and a build fails closed when its local bytes + differ from the registered ones. + +## Why + +- **The publisher record is the thing to witness.** Publishers revise and + withdraw microdata files: the IRS withdrew the public-use file in 2026, and + Census reissues ASEC files under the same vintage label. A registered hash + with a witnessed fetch time is the only durable statement that a given + release existed and had those bytes. This is the same transparency property + Chronicle already provides for published tables, applied to the files + calibration actually starts from. +- **Pins today are scattered and unwitnessed.** Microcosm pins raw inputs in + per-country manifests, checkpoint metadata, code constants, and command-line + arguments, with no shared registry, no licence record, and no timestamp + anyone outside the build can check. A build's root inputs deserve the same + declared identity as every other node. +- **Content would break what makes the store useful.** Row-level microdata + would grow the relational registry and journal by orders of magnitude, + churn on every reissue, and put access-controlled bytes inside the one + system whose value is that anyone can verify it. Thesis resolves forecasts + against Chronicle observations; microdata are not observations of anything + Thesis scores. +- **This keeps the 2026-06-30 ruling.** PR #68 removed microdata parsers, + adapters, and tracked raw storage from the package. Nothing here brings any + of that back. Identity registration adds manifests and two manifest fields, + not code paths that read microdata. + +## Consequences + +- Manifests gain `licence` (publisher terms, as an identifier or URL) and + `access` (`public` | `licensed` | `restricted`). Registration of a + `licensed` or `restricted` artifact records the checksum and access route + and refuses bytes. +- `docs/storage-architecture.md` narrows its non-goal from "no raw survey or + administrative microdata" to "no microdata rows or facts, no licensed or + restricted microdata bytes"; its ownership matrix gains a row for microdata + releases. +- `docs/chronicle-governance.md` allows registering microdata releases and + forbids parsing them or holding gated bytes; the `ledger-boundary` judge + fails a change that does either. +- Microcosm's raw-input entries reference Chronicle registrations; the + consumer side is tracked in PolicyEngine/microcosm. +- Flip conditions for revisiting content: a consumer other than Microcosm + needs row-level evidence from Chronicle, or a publisher grants + redistribution of a currently licensed file and Chronicle has an + access-controlled tier designed for it. Neither holds today. From 7d3fa3651e24afeb8f0e6c2c7b74e3c7e9e5d0cf Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Wed, 2 Sep 2026 10:47:22 -0400 Subject: [PATCH 02/11] Amend architecture, storage, governance, and agent docs for the raw-microdata identity ADR Co-Authored-By: Claude Fable 5.1 --- AGENTS.md | 3 +++ docs/architecture.md | 2 +- docs/chronicle-governance.md | 11 ++++++++--- docs/storage-architecture.md | 18 +++++++++++++++--- 4 files changed, 27 insertions(+), 7 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 0886b2c..5fcce24 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -19,6 +19,9 @@ Do not put Microcosm work in Chronicle: - no solver-ready target construction - no target profiles or model-measurement bindings - no PolicyEngine-computed values stored as facts +- no microdata rows, cells, or microdata-derived facts, and no licensed + microdata bytes (registering a microdata release's identity is allowed; see + `docs/adr-chronicle-raw-microdata-identity.md`) Chronicle records every fact's publisher reference period. Consumers own and enforce any declaration that aligns those facts to another period; Chronicle diff --git a/docs/architecture.md b/docs/architecture.md index 9492c9b..9e28726 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -16,7 +16,7 @@ validation, and build-harness layer. Jurisdiction source packages such as | Layer | Owns | Does not own | |-------|------|--------------| -| Chronicle | Source artifacts, provenance, aggregate facts, constraints | Selection and measurement contracts, raw microdata storage, source reconciliation, aging, imputation, active target selection | +| Chronicle | Source artifacts (including microdata release registrations), provenance, aggregate facts, constraints | Selection and measurement contracts, microdata rows and microdata-derived facts, licensed microdata bytes, source reconciliation, aging, imputation, active target selection | | Microcosm Targets | Selection and measurement contracts, reconciliation, aging, imputation, active target sets | Source artifact storage and provenance | | Microcosm | Entity model, weights, calibration interfaces, calibrated output | Source ETL and source provenance | | Jurisdiction source packages | Source-specific parsers and specs that emit Chronicle records | Forked fact or constraint schemas | diff --git a/docs/chronicle-governance.md b/docs/chronicle-governance.md index f60969f..5933030 100644 --- a/docs/chronicle-governance.md +++ b/docs/chronicle-governance.md @@ -20,6 +20,9 @@ without scoring model output against model output. Chronicle may: - register raw publisher artifacts and checksums +- register raw microdata releases as source artifacts (publisher, access + route, vintage, checksum, licence, fetch time), archiving bytes only where + the publisher's terms permit redistribution - parse source rows and cells - emit source-backed aggregate facts, including publisher projections typed `assertion: source_projection` @@ -37,7 +40,8 @@ Chronicle must not: levels) as facts or in any other store object - compute aligned values; it publishes the source period and value unchanged - impute missing values -- store raw survey or administrative microdata +- parse survey or administrative microdata into source rows, cells, or facts, + or hold licensed or restricted microdata bytes in any Chronicle store - own selection, measurement, period-alignment, or model-binding contracts - choose a support-aware active target subset - build solver-ready calibration targets @@ -65,5 +69,6 @@ types are: The overall verdict fails if any required judge fails. A judge must fail if a change moves reconciliation, aging, imputation, active target selection, or -solver construction from Microcosm into Chronicle, or stores a -PolicyEngine-computed value as a fact. +solver construction from Microcosm into Chronicle, stores a +PolicyEngine-computed value as a fact, parses microdata into rows or facts, or +stores licensed or restricted microdata bytes. diff --git a/docs/storage-architecture.md b/docs/storage-architecture.md index cf89893..77a6c5b 100644 --- a/docs/storage-architecture.md +++ b/docs/storage-architecture.md @@ -14,6 +14,14 @@ artifacts as fetched: workbooks, CSVs, PDFs, ZIPs, HTML snapshots, and similar government-statistics release files. Raw objects are content-addressed by checksum and should never be overwritten in place. +Raw microdata releases follow the same rule at the artifact level only. The +bytes of redistributable public-use files (Census public-use files are the +model case) live in `ledger-raw` under the same content-addressed key shape. +Licensed or restricted microdata (FRS, BE-SILC, the IRS PUF) is registered by +manifest, with checksum, vintage, licence, and access route, and no bytes in +any Chronicle store. No microdata is parsed into source rows, cells, or facts. +See `docs/adr-chronicle-raw-microdata-identity.md`. + `ledger-derived` is the reproducible artifact archive. It stores build outputs that Chronicle can regenerate from raw bytes, package specs, parser code, and build configuration. Examples include parsed-cell or parsed-row Parquet/JSONL files, @@ -36,6 +44,7 @@ Hosted tables mirror accepted build outputs and provide a shared query surface. |------------|-------------------|---------------|-------------------|------------------|-------------------| | Source package specs | Authoritative YAML and parser code | No | Optional packaged snapshot | No | Metadata only | | Raw publisher files | Tiny fixtures only | Authoritative bytes | No | Metadata only | Metadata plus R2 pointer | +| Raw microdata releases | Manifest only, never bytes | Bytes for redistributable public-use files; hash-only for licensed or restricted | No | Metadata only | Metadata, licence, access class, R2 pointer where bytes exist | | Source manifests | Authoritative checked metadata | No | Optional snapshot | Metadata loaded into tables | Queryable artifact registry | | Parsed source rows/cells | Generated local output | No | Snapshot artifact | Queryable table | Queryable mirror | | Source records/facts | Generated local output | No | Snapshot artifact | Queryable table | Queryable mirror | @@ -154,9 +163,12 @@ to it. Use `--dry-run` to verify local JSONL files without writing. Supabase should not store large raw binary artifacts. It should point to R2. -Chronicle should not store raw survey or administrative microdata. It reflects -government statistics releases and the provenance needed to audit those -published facts. +Chronicle should not parse survey or administrative microdata into source rows, +cells, or facts, and should not hold licensed or restricted microdata bytes in +any store. It registers microdata releases as source artifacts and archives +only redistributable public-use bytes (see +`docs/adr-chronicle-raw-microdata-identity.md`). It reflects government +statistics releases and the provenance needed to audit those published facts. R2 should not be the schema authority. It stores bytes and reproducible build files, while Chronicle code and checked specs define semantics. From d101fc259a1b53c5c67a99d51bece29e4d3203c2 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Wed, 2 Sep 2026 12:42:20 -0400 Subject: [PATCH 03/11] Tighten the raw-microdata identity ADR after peer review Require kind: microdata_release; define content as rows, cells, or facts; say licensed or restricted bytes everywhere; narrow the fact prohibition to facts computed directly from raw microdata; reference the country-aware key convention; name the pending fail-closed path (chronicle#221); extend the ledger-boundary judge contract and the package docstring to match. Co-Authored-By: Claude Fable 5.1 --- docs/adr-chronicle-raw-microdata-identity.md | 41 +++++++++++++------- 1 file changed, 28 insertions(+), 13 deletions(-) diff --git a/docs/adr-chronicle-raw-microdata-identity.md b/docs/adr-chronicle-raw-microdata-identity.md index fb8554a..5c35310 100644 --- a/docs/adr-chronicle-raw-microdata-identity.md +++ b/docs/adr-chronicle-raw-microdata-identity.md @@ -2,22 +2,34 @@ Status: accepted 2026-09-02 (amends the facts-only store; narrows the "no raw microdata" non-goal in `docs/storage-architecture.md` and -`docs/chronicle-governance.md`) +`docs/chronicle-governance.md`). The fail-closed registration path this +ADR relies on (`kind: microdata_release`, access-aware refusal in +`fetch-artifact` and `publish-raw`, `register-artifact`) lands with +chronicle#221. Until it merges, do not point the existing commands at a +licensed or restricted file: they materialize and upload bytes. ## Decision Chronicle registers every raw microdata release its consumers build from, and -stores none of their content. +stores none of their content. "Content" means parsed rows, cells, or facts. +Custody of a redistributable public-use file's bytes is not content; custody +of licensed or restricted bytes is never taken at all. 1. **Registration.** A microdata release (CPS ASEC, ACS PUMS, SCF, SIPP, FRS, BE-SILC, the IRS PUF, and their successors in every jurisdiction) is a source artifact like any publisher workbook: publisher, source URL or access route, vintage, SHA-256, size, fetch time, licence, and an access - class from a closed set (`public`, `licensed`, `restricted`). Registration - is manifest-level. It uses the existing `fetch-artifact` and `publish-raw` - path and the content-addressed key shape - `raw/{source_id}/{package_id}/{year}/{sha256}/{filename}`. No source - package parses the file. + class from a closed set (`public`, `licensed`, `restricted`), on a manifest + declared `kind: microdata_release`. That kind is required, not inferred: + it is what lets `validate-package`, the suite builder, and every parser + refuse the file, so a public microdata release can never be mistaken for a + public aggregate workbook. Registration is manifest-level. It uses the + `fetch-artifact` / `publish-raw` path extended with an access-aware + refusal (chronicle#221), and the content-addressed key convention in + `docs/storage-architecture.md`: `raw/{country}/{source_id}/{package_id}/ + {year}/{sha256}/{filename}` for UK and New Zealand sources, the legacy + `raw/{source_id}/...` shape for US sources. No source package parses the + file. 2. **Bytes only where the publisher permits redistribution.** Public-use files whose terms allow redistribution (Census public-use files are the model case) are archived in the raw bucket under that key. Licensed or restricted @@ -25,11 +37,13 @@ stores none of their content. PUF) are registered hash-only: no bytes in any Chronicle store, and no Chronicle credential grants access to them. Their bytes stay in the licensed environments consumers already operate. -3. **No rows, no facts.** No microdata record, column, or cell enters - `source_rows`, `source_cells`, the relational registry, or the journal. No - fact is minted from microdata by Chronicle. An aggregate a publisher - computes from its own microdata and publishes is an ordinary fact; an - aggregate a consumer computes from microdata is that consumer's artifact. +3. **No rows, no cells, no facts.** No microdata record, column, or cell + enters `source_rows`, `source_cells`, the relational registry, or the + journal. No fact is computed directly from raw microdata by Chronicle, and + none computed that way by a consumer enters Chronicle. An aggregate a + publisher computes from its own microdata and publishes is an ordinary + fact with ordinary provenance; an aggregate a consumer computes from + microdata is that consumer's artifact. 4. **Consumers point at the registration.** A Microcosm source-stage manifest that names a microdata artifact carries the Chronicle artifact reference and the same SHA-256, so every root of a build graph resolves to one @@ -73,7 +87,8 @@ stores none of their content. releases. - `docs/chronicle-governance.md` allows registering microdata releases and forbids parsing them or holding gated bytes; the `ledger-boundary` judge - fails a change that does either. + contract in `.github/chronicle-agents.yml` names both refusals, so the + configured judge cannot pass a change that does either. - Microcosm's raw-input entries reference Chronicle registrations; the consumer side is tracked in PolicyEngine/microcosm. - Flip conditions for revisiting content: a consumer other than Microcosm From e3d93896a3c7b8f6a9bc43aa490698c58e23daef Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Wed, 2 Sep 2026 12:43:53 -0400 Subject: [PATCH 04/11] Propagate the tightened microdata boundary to README, architecture, AGENTS, the judge contract, and the package docstring Co-Authored-By: Claude Fable 5.1 --- .github/chronicle-agents.yml | 2 +- AGENTS.md | 6 ++++-- README.md | 5 +++-- chronicle/__init__.py | 11 +++++++---- docs/architecture.md | 2 +- 5 files changed, 16 insertions(+), 10 deletions(-) diff --git a/.github/chronicle-agents.yml b/.github/chronicle-agents.yml index 43f302c..d5d48cb 100644 --- a/.github/chronicle-agents.yml +++ b/.github/chronicle-agents.yml @@ -43,4 +43,4 @@ required_judges: ledger-contract: verdict: PASS if schema or identity changes preserve source provenance and do not move Microcosm responsibilities into Chronicle. ledger-boundary: - verdict: PASS if reconciliation, aging, imputation, support-aware activation, and solver construction remain outside Chronicle. + verdict: PASS if reconciliation, aging, imputation, support-aware activation, and solver construction remain outside Chronicle, no microdata rows, cells, or facts computed directly from raw microdata enter Chronicle, and no licensed or restricted microdata bytes enter any Chronicle store (docs/adr-chronicle-raw-microdata-identity.md). diff --git a/AGENTS.md b/AGENTS.md index 5fcce24..b2fe024 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -19,8 +19,10 @@ Do not put Microcosm work in Chronicle: - no solver-ready target construction - no target profiles or model-measurement bindings - no PolicyEngine-computed values stored as facts -- no microdata rows, cells, or microdata-derived facts, and no licensed - microdata bytes (registering a microdata release's identity is allowed; see +- no microdata rows, cells, or facts computed directly from raw microdata, + and no licensed or restricted microdata bytes in any Chronicle store + (registering a release's identity is allowed, and publisher-computed, + publisher-published aggregates remain ordinary facts; see `docs/adr-chronicle-raw-microdata-identity.md`) Chronicle records every fact's publisher reference period. Consumers own and diff --git a/README.md b/README.md index 0f8509a..d9247c7 100644 --- a/README.md +++ b/README.md @@ -12,8 +12,9 @@ Chronicle may normalize structure: parse files, type values, declare units and scales, assign geography and period identifiers, preserve lineage back to source artifacts, and publish source-backed facts. Chronicle does not own selection or measurement contracts, reconcile inconsistent sources, impute -missing data, parse survey or administrative microdata into rows or facts, hold -licensed microdata bytes, or execute simulator-specific calibration. Chronicle +missing data, parse survey or administrative microdata into rows, cells, or +facts, hold licensed or restricted microdata bytes, or execute +simulator-specific calibration. Chronicle does register the microdata releases its consumers build from (publisher, vintage, checksum, licence) and archives the bytes of redistributable public-use files; see diff --git a/chronicle/__init__.py b/chronicle/__init__.py index 04d5094..acfa663 100644 --- a/chronicle/__init__.py +++ b/chronicle/__init__.py @@ -1,9 +1,12 @@ """Chronicle source-data foundation. -Chronicle owns government-statistics releases: source artifacts, source-backed -facts, constraints, and provenance. Selection contracts, raw microdata storage, -source reconciliation, aging, imputation, target activation, and calibration -belong in downstream systems such as Microcosm. +Chronicle owns government-statistics releases: source artifacts (including +registrations of raw microdata releases and custody of redistributable +public-use bytes), source-backed facts, constraints, and provenance. Microdata +content (rows, cells, facts computed from raw microdata), licensed or +restricted microdata bytes, selection contracts, source reconciliation, aging, +imputation, target activation, and calibration belong in downstream systems +such as Microcosm. """ __all__ = [ diff --git a/docs/architecture.md b/docs/architecture.md index 9e28726..2224f55 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -16,7 +16,7 @@ validation, and build-harness layer. Jurisdiction source packages such as | Layer | Owns | Does not own | |-------|------|--------------| -| Chronicle | Source artifacts (including microdata release registrations), provenance, aggregate facts, constraints | Selection and measurement contracts, microdata rows and microdata-derived facts, licensed microdata bytes, source reconciliation, aging, imputation, active target selection | +| Chronicle | Source artifacts (including microdata release registrations), provenance, aggregate facts, constraints | Selection and measurement contracts, microdata rows, cells, and facts computed directly from raw microdata; licensed or restricted microdata bytes; source reconciliation, aging, imputation, active target selection | | Microcosm Targets | Selection and measurement contracts, reconciliation, aging, imputation, active target sets | Source artifact storage and provenance | | Microcosm | Entity model, weights, calibration interfaces, calibrated output | Source ETL and source provenance | | Jurisdiction source packages | Source-specific parsers and specs that emit Chronicle records | Forked fact or constraint schemas | From 1683e17a6e8e1af4241cc27cb5c1a745d6cba2dc Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Wed, 2 Sep 2026 13:06:11 -0400 Subject: [PATCH 05/11] Answer the round-2 ADR review: redistribution allowlist for byte custody, canonical prohibition vocabulary, hash attestation semantics, scoped harness plan Byte custody now requires access public AND a licence on the redistributable allowlist; the prohibition reads identically in the ADR, governance doc, judge contract, README, architecture table, AGENTS, and package docstring (records, rows, columns, row values, cells; facts derived from raw microdata by Chronicle or a consumer; licensed or restricted bytes) and the governance test pins the judge wording; hash_source and verified_at define what a registration attests; the storage matrix says what exists today; the harness plan is scoped to publisher aggregates; the key convention states the unmapped-publisher fallback. Co-Authored-By: Claude Fable 5.1 --- .github/chronicle-agents.yml | 2 +- AGENTS.md | 10 +- README.md | 10 +- chronicle/__init__.py | 4 +- docs/adr-chronicle-raw-microdata-identity.md | 120 ++++++++++++------- docs/architecture.md | 2 +- docs/chronicle-governance.md | 16 ++- docs/storage-architecture.md | 28 +++-- docs/target-construction-harness-plan.md | 5 + tests/test_chronicle_governance.py | 8 ++ 10 files changed, 130 insertions(+), 75 deletions(-) diff --git a/.github/chronicle-agents.yml b/.github/chronicle-agents.yml index d5d48cb..84aa121 100644 --- a/.github/chronicle-agents.yml +++ b/.github/chronicle-agents.yml @@ -43,4 +43,4 @@ required_judges: ledger-contract: verdict: PASS if schema or identity changes preserve source provenance and do not move Microcosm responsibilities into Chronicle. ledger-boundary: - verdict: PASS if reconciliation, aging, imputation, support-aware activation, and solver construction remain outside Chronicle, no microdata rows, cells, or facts computed directly from raw microdata enter Chronicle, and no licensed or restricted microdata bytes enter any Chronicle store (docs/adr-chronicle-raw-microdata-identity.md). + verdict: PASS if reconciliation, aging, imputation, support-aware activation, and solver construction remain outside Chronicle; no microdata records, rows, columns, row values, or cells and no facts derived from raw microdata by Chronicle or a consumer enter Chronicle; and no licensed or restricted microdata bytes enter any Chronicle store (docs/adr-chronicle-raw-microdata-identity.md). diff --git a/AGENTS.md b/AGENTS.md index b2fe024..3594db6 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -19,11 +19,11 @@ Do not put Microcosm work in Chronicle: - no solver-ready target construction - no target profiles or model-measurement bindings - no PolicyEngine-computed values stored as facts -- no microdata rows, cells, or facts computed directly from raw microdata, - and no licensed or restricted microdata bytes in any Chronicle store - (registering a release's identity is allowed, and publisher-computed, - publisher-published aggregates remain ordinary facts; see - `docs/adr-chronicle-raw-microdata-identity.md`) +- no microdata records, rows, columns, row values, or cells, no facts + derived from raw microdata by Chronicle or a consumer, and no licensed or + restricted microdata bytes in any Chronicle store (registering a release's + identity is allowed, and a publisher's own published aggregates remain + ordinary facts; see `docs/adr-chronicle-raw-microdata-identity.md`) Chronicle records every fact's publisher reference period. Consumers own and enforce any declaration that aligns those facts to another period; Chronicle diff --git a/README.md b/README.md index d9247c7..b88c4f8 100644 --- a/README.md +++ b/README.md @@ -12,12 +12,12 @@ Chronicle may normalize structure: parse files, type values, declare units and scales, assign geography and period identifiers, preserve lineage back to source artifacts, and publish source-backed facts. Chronicle does not own selection or measurement contracts, reconcile inconsistent sources, impute -missing data, parse survey or administrative microdata into rows, cells, or -facts, hold licensed or restricted microdata bytes, or execute -simulator-specific calibration. Chronicle +missing data, parse survey or administrative microdata into records, rows, +columns, row values, or cells, derive facts from raw microdata, hold licensed +or restricted microdata bytes, or execute simulator-specific calibration. Chronicle does register the microdata releases its consumers build from (publisher, -vintage, checksum, licence) and archives the bytes of redistributable public-use -files; see +vintage, checksum, licence) and archives the bytes of public-use files whose +licence is on its redistributable allowlist; see [`docs/adr-chronicle-raw-microdata-identity.md`](docs/adr-chronicle-raw-microdata-identity.md). Microcosm consumes Chronicle facts, owns the contracts that select and bind diff --git a/chronicle/__init__.py b/chronicle/__init__.py index acfa663..9f31fe5 100644 --- a/chronicle/__init__.py +++ b/chronicle/__init__.py @@ -3,8 +3,8 @@ Chronicle owns government-statistics releases: source artifacts (including registrations of raw microdata releases and custody of redistributable public-use bytes), source-backed facts, constraints, and provenance. Microdata -content (rows, cells, facts computed from raw microdata), licensed or -restricted microdata bytes, selection contracts, source reconciliation, aging, +content (records, rows, columns, row values, cells, and facts derived from raw +microdata), licensed or restricted microdata bytes, selection contracts, source reconciliation, aging, imputation, target activation, and calibration belong in downstream systems such as Microcosm. """ diff --git a/docs/adr-chronicle-raw-microdata-identity.md b/docs/adr-chronicle-raw-microdata-identity.md index 5c35310..7f4d329 100644 --- a/docs/adr-chronicle-raw-microdata-identity.md +++ b/docs/adr-chronicle-raw-microdata-identity.md @@ -4,47 +4,74 @@ Status: accepted 2026-09-02 (amends the facts-only store; narrows the "no raw microdata" non-goal in `docs/storage-architecture.md` and `docs/chronicle-governance.md`). The fail-closed registration path this ADR relies on (`kind: microdata_release`, access-aware refusal in -`fetch-artifact` and `publish-raw`, `register-artifact`) lands with -chronicle#221. Until it merges, do not point the existing commands at a -licensed or restricted file: they materialize and upload bytes. +`fetch-artifact` and `publish-raw`, `register-artifact`, the redistributable +licence allowlist) lands with chronicle#221. Until it merges, do not point +the existing commands at any microdata release, public or not: today they +materialize, upload, and parse every input, and a manifest without `kind` +is a publisher table by definition. ## Decision Chronicle registers every raw microdata release its consumers build from, and -stores none of their content. "Content" means parsed rows, cells, or facts. -Custody of a redistributable public-use file's bytes is not content; custody -of licensed or restricted bytes is never taken at all. +stores none of their content. "Content" means any parsed representation: +microdata records, rows, columns, row values, or cells, and any fact derived +from them by Chronicle or a consumer. Custody of a redistributable public-use +file's bytes is not content; custody of licensed or restricted bytes is never +taken at all. 1. **Registration.** A microdata release (CPS ASEC, ACS PUMS, SCF, SIPP, FRS, BE-SILC, the IRS PUF, and their successors in every jurisdiction) is a source artifact like any publisher workbook: publisher, source URL or - access route, vintage, SHA-256, size, fetch time, licence, and an access - class from a closed set (`public`, `licensed`, `restricted`), on a manifest - declared `kind: microdata_release`. That kind is required, not inferred: - it is what lets `validate-package`, the suite builder, and every parser - refuse the file, so a public microdata release can never be mistaken for a - public aggregate workbook. Registration is manifest-level. It uses the - `fetch-artifact` / `publish-raw` path extended with an access-aware - refusal (chronicle#221), and the content-addressed key convention in - `docs/storage-architecture.md`: `raw/{country}/{source_id}/{package_id}/ - {year}/{sha256}/{filename}` for UK and New Zealand sources, the legacy - `raw/{source_id}/...` shape for US sources. No source package parses the - file. -2. **Bytes only where the publisher permits redistribution.** Public-use files - whose terms allow redistribution (Census public-use files are the model - case) are archived in the raw bucket under that key. Licensed or restricted - files (FRS under the UKDS agreement, BE-SILC scientific-use files, the IRS - PUF) are registered hash-only: no bytes in any Chronicle store, and no - Chronicle credential grants access to them. Their bytes stay in the - licensed environments consumers already operate. -3. **No rows, no cells, no facts.** No microdata record, column, or cell - enters `source_rows`, `source_cells`, the relational registry, or the - journal. No fact is computed directly from raw microdata by Chronicle, and - none computed that way by a consumer enters Chronicle. An aggregate a + access route, vintage, SHA-256, size, licence, an access class from a + closed set (`public`, `licensed`, `restricted`), and the hash-source and + verification fields defined below, on a manifest declared + `kind: microdata_release`. That kind is required, not inferred: it is what + lets `validate-package`, the suite builder, the source-package byte reader, + and every parser refuse the file, so a public microdata release can never + be mistaken for a public aggregate workbook. A manifest without `kind` is + a publisher table; every existing manifest is one. Registration is + manifest-level. It uses the `fetch-artifact` / `publish-raw` path extended + with an access-aware refusal (chronicle#221), and the content-addressed key + convention in `docs/storage-architecture.md`: + `raw/{country}/{source_id}/{package_id}/{year}/{sha256}/{filename}` for + publishers mapped to a country segment (UK and New Zealand today), and the + legacy `raw/{source_id}/{package_id}/{year}/{sha256}/{filename}` shape for + every unmapped publisher, US sources included. No source package parses + the file. +2. **Bytes only with an affirmative redistribution permission.** Being + downloadable is not a licence. Chronicle archives a release's bytes in the + raw bucket only when its `access` is `public` **and** its recorded + `licence` matches an entry in Chronicle's allowlist of redistributable + terms, maintained in code with the evidence for each entry (for example a + U.S. Government work under 17 U.S.C. §105, the Open Government Licence, + CC0, CC BY). A file that is publicly downloadable under any other or + unstated terms is classed `licensed`, whatever its access route. Licensed + or restricted files (FRS under the UKDS agreement, BE-SILC scientific-use + files, the IRS PUF) are registered hash-only: no bytes in any Chronicle + store, and no Chronicle credential grants access to them. Their bytes stay + in the licensed environments consumers already operate. +3. **No records, rows, columns, row values, cells, or derived facts.** No + microdata record, row, column, row value, or cell enters `source_records`, + `source_rows`, `source_columns`, `source_row_values`, `source_cells`, the + relational registry, the derived-artifact bucket, or the journal, whether + the release is public or gated. No fact derived from raw microdata by + Chronicle or by a consumer enters Chronicle, however many intermediate + artifacts stand between the microdata and the value. The only exception is + assertion-based, as everywhere else in Chronicle: an aggregate that a publisher computes from its own microdata and publishes is an ordinary - fact with ordinary provenance; an aggregate a consumer computes from - microdata is that consumer's artifact. -4. **Consumers point at the registration.** A Microcosm source-stage manifest + fact with ordinary provenance. +4. **What a registration attests.** `hash_source` records who computed the + registered SHA-256: `chronicle_fetch` (Chronicle fetched the bytes and + hashed them; the registration attests the bytes), `consumer_attested` (a + consumer that holds the bytes recomputed the hash and recorded the + evidence; the registration attests the bytes on that consumer's word), or + `consumer_pin` (transcribed from a consumer's reviewed pin without + recomputation; the registration attests the pin, not the bytes). + `verified_at` is the date the recorded hash was last compared against the + bytes by whoever holds them. The witnessed fetch time in the journal + bounds when the registration existed; it bounds when the bytes existed + only for `chronicle_fetch` and `consumer_attested` entries. +5. **Consumers point at the registration.** A Microcosm source-stage manifest that names a microdata artifact carries the Chronicle artifact reference and the same SHA-256, so every root of a build graph resolves to one witnessed registration and a build fails closed when its local bytes @@ -56,7 +83,7 @@ of licensed or restricted bytes is never taken at all. withdraw microdata files: the IRS withdrew the public-use file in 2026, and Census reissues ASEC files under the same vintage label. A registered hash with a witnessed fetch time is the only durable statement that a given - release existed and had those bytes. This is the same transparency property + release existed with those bytes. This is the same transparency property Chronicle already provides for published tables, applied to the files calibration actually starts from. - **Pins today are scattered and unwitnessed.** Microcosm pins raw inputs in @@ -72,23 +99,28 @@ of licensed or restricted bytes is never taken at all. Thesis scores. - **This keeps the 2026-06-30 ruling.** PR #68 removed microdata parsers, adapters, and tracked raw storage from the package. Nothing here brings any - of that back. Identity registration adds manifests and two manifest fields, - not code paths that read microdata. + of that back. Identity registration adds manifests and manifest fields, and + the code it adds refuses to read microdata rather than reading it. ## Consequences -- Manifests gain `licence` (publisher terms, as an identifier or URL) and - `access` (`public` | `licensed` | `restricted`). Registration of a - `licensed` or `restricted` artifact records the checksum and access route - and refuses bytes. +- Manifests gain `licence` (publisher terms, as an identifier or URL), + `access` (`public` | `licensed` | `restricted`), `hash_source`, and + `verified_at`; a microdata manifest declares `kind: microdata_release`. + Registration of a `licensed` or `restricted` artifact records the checksum + and access route and refuses bytes; registration of a `public` artifact + archives bytes only when the licence is on the redistributable allowlist. - `docs/storage-architecture.md` narrows its non-goal from "no raw survey or - administrative microdata" to "no microdata rows or facts, no licensed or - restricted microdata bytes"; its ownership matrix gains a row for microdata - releases. + administrative microdata" to "no microdata records, rows, columns, row + values, cells, or derived facts; no licensed or restricted microdata bytes"; + its ownership matrix gains a row for microdata releases. + `docs/target-construction-harness-plan.md` is scoped to publisher aggregate + artifacts; microdata releases are identity-only. - `docs/chronicle-governance.md` allows registering microdata releases and forbids parsing them or holding gated bytes; the `ledger-boundary` judge - contract in `.github/chronicle-agents.yml` names both refusals, so the - configured judge cannot pass a change that does either. + contract in `.github/chronicle-agents.yml` names both refusals in the same + canonical words, and `tests/test_chronicle_governance.py` pins that + wording, so the configured judge cannot pass a change that does either. - Microcosm's raw-input entries reference Chronicle registrations; the consumer side is tracked in PolicyEngine/microcosm. - Flip conditions for revisiting content: a consumer other than Microcosm diff --git a/docs/architecture.md b/docs/architecture.md index 2224f55..ae1eb78 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -16,7 +16,7 @@ validation, and build-harness layer. Jurisdiction source packages such as | Layer | Owns | Does not own | |-------|------|--------------| -| Chronicle | Source artifacts (including microdata release registrations), provenance, aggregate facts, constraints | Selection and measurement contracts, microdata rows, cells, and facts computed directly from raw microdata; licensed or restricted microdata bytes; source reconciliation, aging, imputation, active target selection | +| Chronicle | Source artifacts (including microdata release registrations), provenance, aggregate facts, constraints | Selection and measurement contracts, microdata records, rows, columns, row values, or cells; facts derived from raw microdata by Chronicle or a consumer; licensed or restricted microdata bytes; source reconciliation, aging, imputation, active target selection | | Microcosm Targets | Selection and measurement contracts, reconciliation, aging, imputation, active target sets | Source artifact storage and provenance | | Microcosm | Entity model, weights, calibration interfaces, calibrated output | Source ETL and source provenance | | Jurisdiction source packages | Source-specific parsers and specs that emit Chronicle records | Forked fact or constraint schemas | diff --git a/docs/chronicle-governance.md b/docs/chronicle-governance.md index 5933030..6ca5330 100644 --- a/docs/chronicle-governance.md +++ b/docs/chronicle-governance.md @@ -21,8 +21,9 @@ Chronicle may: - register raw publisher artifacts and checksums - register raw microdata releases as source artifacts (publisher, access - route, vintage, checksum, licence, fetch time), archiving bytes only where - the publisher's terms permit redistribution + route, vintage, checksum, licence, hash source, verification date), + archiving bytes only when the release is `public` and its licence is on + the redistributable allowlist - parse source rows and cells - emit source-backed aggregate facts, including publisher projections typed `assertion: source_projection` @@ -40,8 +41,10 @@ Chronicle must not: levels) as facts or in any other store object - compute aligned values; it publishes the source period and value unchanged - impute missing values -- parse survey or administrative microdata into source rows, cells, or facts, - or hold licensed or restricted microdata bytes in any Chronicle store +- parse survey or administrative microdata into records, rows, columns, row + values, or cells, derive facts from raw microdata (by Chronicle or a + consumer), or hold licensed or restricted microdata bytes in any Chronicle + store - own selection, measurement, period-alignment, or model-binding contracts - choose a support-aware active target subset - build solver-ready calibration targets @@ -70,5 +73,6 @@ types are: The overall verdict fails if any required judge fails. A judge must fail if a change moves reconciliation, aging, imputation, active target selection, or solver construction from Microcosm into Chronicle, stores a -PolicyEngine-computed value as a fact, parses microdata into rows or facts, or -stores licensed or restricted microdata bytes. +PolicyEngine-computed value as a fact, parses microdata into records, rows, +columns, row values, or cells, derives facts from raw microdata, or stores +licensed or restricted microdata bytes. diff --git a/docs/storage-architecture.md b/docs/storage-architecture.md index 77a6c5b..7d97515 100644 --- a/docs/storage-architecture.md +++ b/docs/storage-architecture.md @@ -15,12 +15,17 @@ government-statistics release files. Raw objects are content-addressed by checksum and should never be overwritten in place. Raw microdata releases follow the same rule at the artifact level only. The -bytes of redistributable public-use files (Census public-use files are the -model case) live in `ledger-raw` under the same content-addressed key shape. -Licensed or restricted microdata (FRS, BE-SILC, the IRS PUF) is registered by -manifest, with checksum, vintage, licence, and access route, and no bytes in -any Chronicle store. No microdata is parsed into source rows, cells, or facts. -See `docs/adr-chronicle-raw-microdata-identity.md`. +bytes of a `public` release whose licence is on Chronicle's redistributable +allowlist (Census public-use files are the model case) live in the raw bucket +under the same content-addressed key shape. Every other microdata release +(FRS, BE-SILC, the IRS PUF, and any publicly downloadable file under +unstated terms) is registered by manifest, with checksum, vintage, licence, +access route, hash source, and verification date, and no bytes in any +Chronicle store. No microdata is parsed into records, rows, columns, row +values, or cells, and no fact is derived from it. Until chronicle#221 adds +registry columns, a registration exists only in its manifest: `ledger.db` and +Supabase carry nothing for it. See +`docs/adr-chronicle-raw-microdata-identity.md`. `ledger-derived` is the reproducible artifact archive. It stores build outputs that Chronicle can regenerate from raw bytes, package specs, parser code, and build @@ -44,7 +49,7 @@ Hosted tables mirror accepted build outputs and provide a shared query surface. |------------|-------------------|---------------|-------------------|------------------|-------------------| | Source package specs | Authoritative YAML and parser code | No | Optional packaged snapshot | No | Metadata only | | Raw publisher files | Tiny fixtures only | Authoritative bytes | No | Metadata only | Metadata plus R2 pointer | -| Raw microdata releases | Manifest only, never bytes | Bytes for redistributable public-use files; hash-only for licensed or restricted | No | Metadata only | Metadata, licence, access class, R2 pointer where bytes exist | +| Raw microdata releases | Manifest only (`kind: microdata_release`), never bytes | Bytes only for `public` releases on the redistributable allowlist; hash-only otherwise | No | Not represented until chronicle#221 | Not represented until chronicle#221 | | Source manifests | Authoritative checked metadata | No | Optional snapshot | Metadata loaded into tables | Queryable artifact registry | | Parsed source rows/cells | Generated local output | No | Snapshot artifact | Queryable table | Queryable mirror | | Source records/facts | Generated local output | No | Snapshot artifact | Queryable table | Queryable mirror | @@ -163,10 +168,11 @@ to it. Use `--dry-run` to verify local JSONL files without writing. Supabase should not store large raw binary artifacts. It should point to R2. -Chronicle should not parse survey or administrative microdata into source rows, -cells, or facts, and should not hold licensed or restricted microdata bytes in -any store. It registers microdata releases as source artifacts and archives -only redistributable public-use bytes (see +Chronicle should not parse survey or administrative microdata into records, +rows, columns, row values, or cells, derive facts from it, or hold licensed or +restricted microdata bytes in any store. It registers microdata releases as +source artifacts and archives bytes only for public releases on its +redistributable allowlist (see `docs/adr-chronicle-raw-microdata-identity.md`). It reflects government statistics releases and the provenance needed to audit those published facts. diff --git a/docs/target-construction-harness-plan.md b/docs/target-construction-harness-plan.md index 65514f1..6bcf30b 100644 --- a/docs/target-construction-harness-plan.md +++ b/docs/target-construction-harness-plan.md @@ -1,5 +1,10 @@ # Fact Construction Harness Plan +Scope: this plan covers publisher aggregate artifacts (workbooks, CSVs, API +extracts of published statistics). Raw microdata releases are out of its +scope: Chronicle registers them by identity only and never preserves or +parses their content (`docs/adr-chronicle-raw-microdata-identity.md`). + ## Goal Build a harness that verifies source fact construction end to end: diff --git a/tests/test_chronicle_governance.py b/tests/test_chronicle_governance.py index f914237..56c0f99 100644 --- a/tests/test_chronicle_governance.py +++ b/tests/test_chronicle_governance.py @@ -50,6 +50,14 @@ def test_chronicle_governance_files_define_required_review_surface(): for judge_id in required_judges: assert judge_id in pr_template + boundary_verdict = agents["required_judges"]["ledger-boundary"]["verdict"] + for phrase in ( + "microdata records, rows, columns, row values, or cells", + "facts derived from raw microdata", + "licensed or restricted microdata bytes", + ): + assert phrase in boundary_verdict + def test_ci_runs_boundary_and_governance_tests(): ci = (ROOT / ".github" / "workflows" / "ci.yml").read_text() From 92a44bfec871fdbca384a2bc579b59a6f081dce1 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 3 Sep 2026 09:29:45 -0400 Subject: [PATCH 06/11] Fourth review round on the raw-microdata identity ADR: explicit classification, artifact-bound licence evidence, provenance lineage as the enforcement path, one shared boundary contract Every new or modified manifest declares kind (frozen grandfather list for the legacy set); byte custody needs licence_evidence binding the artifact, not just an allowlisted licence name; the derived-fact prohibition names its enforcement (asserting principal + root_artifacts, chronicle#238) and is a review obligation until then; attestation records who attested and with what evidence; README and the package docstring are prospective until #221 lands; public microdata bytes are staged untracked; the facts-only ADR and README boundary carry the microdata-grain exception; the three negative clauses live once in chronicle/boundary.py and are asserted verbatim in the judge contract and the governance doc. Co-Authored-By: Claude Fable 5.1 --- .github/chronicle-agents.yml | 2 +- README.md | 13 +- chronicle/__init__.py | 6 +- chronicle/boundary.py | 24 +++ docs/adr-chronicle-facts-only.md | 6 +- docs/adr-chronicle-raw-microdata-identity.md | 191 +++++++++++-------- docs/chronicle-governance.md | 5 +- docs/storage-architecture.md | 2 +- tests/test_chronicle_governance.py | 13 +- 9 files changed, 158 insertions(+), 104 deletions(-) create mode 100644 chronicle/boundary.py diff --git a/.github/chronicle-agents.yml b/.github/chronicle-agents.yml index 84aa121..f16217d 100644 --- a/.github/chronicle-agents.yml +++ b/.github/chronicle-agents.yml @@ -43,4 +43,4 @@ required_judges: ledger-contract: verdict: PASS if schema or identity changes preserve source provenance and do not move Microcosm responsibilities into Chronicle. ledger-boundary: - verdict: PASS if reconciliation, aging, imputation, support-aware activation, and solver construction remain outside Chronicle; no microdata records, rows, columns, row values, or cells and no facts derived from raw microdata by Chronicle or a consumer enter Chronicle; and no licensed or restricted microdata bytes enter any Chronicle store (docs/adr-chronicle-raw-microdata-identity.md). + verdict: PASS if reconciliation, aging, imputation, support-aware activation, and solver construction remain outside Chronicle; no microdata records, rows, columns, row values, or cells enter any Chronicle parsed-source surface, registry, derived artifact, or journal; no fact derived from raw microdata by Chronicle or by a consumer enters Chronicle, however many intermediate artifacts stand between them, the only exception being an aggregate that the publisher of that microdata computed and published itself; no licensed or restricted microdata bytes enter any Chronicle store, and public microdata bytes enter only with artifact-bound redistribution evidence (docs/adr-chronicle-raw-microdata-identity.md). diff --git a/README.md b/README.md index b88c4f8..84eb09d 100644 --- a/README.md +++ b/README.md @@ -14,10 +14,10 @@ source artifacts, and publish source-backed facts. Chronicle does not own selection or measurement contracts, reconcile inconsistent sources, impute missing data, parse survey or administrative microdata into records, rows, columns, row values, or cells, derive facts from raw microdata, hold licensed -or restricted microdata bytes, or execute simulator-specific calibration. Chronicle -does register the microdata releases its consumers build from (publisher, -vintage, checksum, licence) and archives the bytes of public-use files whose -licence is on its redistributable allowlist; see +or restricted microdata bytes, or execute simulator-specific calibration. Once +chronicle#221 lands, Chronicle registers the microdata releases its consumers +build from (publisher, vintage, checksum, licence) and archives the bytes of +public-use files that carry artifact-bound redistribution evidence; see [`docs/adr-chronicle-raw-microdata-identity.md`](docs/adr-chronicle-raw-microdata-identity.md). Microcosm consumes Chronicle facts, owns the contracts that select and bind @@ -55,7 +55,10 @@ The load-bearing rule: > consumer or transform it in ways that change its meaning. The store is facts-only, and the line is who asserted the value. Everything a -publisher asserted — including the publisher's own projections — is a fact. +publisher asserted — including the publisher's own projections — is a fact, +except publisher-authored values at microdata grain, which are content and +never facts; only the publisher's own published aggregates over its microdata +are facts (see `docs/adr-chronicle-raw-microdata-identity.md`). Everything PolicyEngine computes (aged, uprated, forecast, or reconciled levels) is a downstream build artifact and never enters the store; Microcosm owns aging as a named, versioned model over Chronicle growth-factor facts. A diff --git a/chronicle/__init__.py b/chronicle/__init__.py index 9f31fe5..a944c94 100644 --- a/chronicle/__init__.py +++ b/chronicle/__init__.py @@ -1,8 +1,8 @@ """Chronicle source-data foundation. -Chronicle owns government-statistics releases: source artifacts (including -registrations of raw microdata releases and custody of redistributable -public-use bytes), source-backed facts, constraints, and provenance. Microdata +Chronicle owns government-statistics releases: source artifacts (and, once +chronicle#221 lands, registrations of raw microdata releases plus custody of +public-use bytes with redistribution evidence), source-backed facts, constraints, and provenance. Microdata content (records, rows, columns, row values, cells, and facts derived from raw microdata), licensed or restricted microdata bytes, selection contracts, source reconciliation, aging, imputation, target activation, and calibration belong in downstream systems diff --git a/chronicle/boundary.py b/chronicle/boundary.py new file mode 100644 index 0000000..6a46edb --- /dev/null +++ b/chronicle/boundary.py @@ -0,0 +1,24 @@ +"""The microdata boundary, stated once so docs, the judge contract, and tests agree. + +``docs/adr-chronicle-raw-microdata-identity.md`` decides that Chronicle registers +raw microdata releases by identity and stores none of their content. These are +the three negative clauses that decision reduces to. ``.github/chronicle-agents.yml`` +carries them verbatim in the ``ledger-boundary`` judge verdict, and +``tests/test_chronicle_governance.py`` asserts that it does, so the wording cannot +regress or invert without failing the suite. +""" + +from __future__ import annotations + +MICRODATA_BOUNDARY_CLAUSES: tuple[str, ...] = ( + "no microdata records, rows, columns, row values, or cells enter any " + "Chronicle parsed-source surface, registry, derived artifact, or journal", + "no fact derived from raw microdata by Chronicle or by a consumer enters " + "Chronicle, however many intermediate artifacts stand between them, the only " + "exception being an aggregate that the publisher of that microdata computed " + "and published itself", + "no licensed or restricted microdata bytes enter any Chronicle store, and " + "public microdata bytes enter only with artifact-bound redistribution evidence", +) + +__all__ = ["MICRODATA_BOUNDARY_CLAUSES"] diff --git a/docs/adr-chronicle-facts-only.md b/docs/adr-chronicle-facts-only.md index 8bfa88d..4ffdb96 100644 --- a/docs/adr-chronicle-facts-only.md +++ b/docs/adr-chronicle-facts-only.md @@ -9,7 +9,11 @@ Chronicle stores source-published values only. The boundary is **who asserted the value**, not level versus projection: 1. Anything a publisher asserted is a fact — including the publisher's own - projections. "CBO's January 2026 baseline projects individual income tax + projections — with one grain exception added by + `docs/adr-chronicle-raw-microdata-identity.md`: a publisher-authored + value at microdata grain (a record, row, column, row value, or cell of a + microdata release) is content, not a fact; only the aggregates the + publisher itself computed and published over that microdata are facts. "CBO's January 2026 baseline projects individual income tax receipts of $X in 2027" is a source-backed claim with lineage, exactly like an SOI observation. These facts carry `assertion: source_projection`; measured or administered outcomes carry diff --git a/docs/adr-chronicle-raw-microdata-identity.md b/docs/adr-chronicle-raw-microdata-identity.md index 7f4d329..f0b46ad 100644 --- a/docs/adr-chronicle-raw-microdata-identity.md +++ b/docs/adr-chronicle-raw-microdata-identity.md @@ -1,77 +1,99 @@ # ADR: Raw microdata in Chronicle is identity, not content -Status: accepted 2026-09-02 (amends the facts-only store; narrows the -"no raw microdata" non-goal in `docs/storage-architecture.md` and -`docs/chronicle-governance.md`). The fail-closed registration path this -ADR relies on (`kind: microdata_release`, access-aware refusal in -`fetch-artifact` and `publish-raw`, `register-artifact`, the redistributable -licence allowlist) lands with chronicle#221. Until it merges, do not point -the existing commands at any microdata release, public or not: today they -materialize, upload, and parse every input, and a manifest without `kind` -is a publisher table by definition. +Status: accepted 2026-09-02. Enforcement lands in two steps and this ADR's +guarantees are prospective until they merge: chronicle#221 (PR #227: manifest +classification, the access-aware refusals in `fetch-artifact`, `publish-raw`, +`register-artifact`, and the source-package byte reader, the redistribution +allowlist, artifact-bound licence evidence, untracked staging for public +microdata bytes) and chronicle#238 (asserting principal and root-artifact +lineage on fact provenance). Until #227 merges, no microdata release may be +pointed at any Chronicle command: today they materialize, upload, and parse +every input. Until #238 merges, the derived-fact prohibition is a review +obligation, not a validator. ## Decision Chronicle registers every raw microdata release its consumers build from, and stores none of their content. "Content" means any parsed representation: microdata records, rows, columns, row values, or cells, and any fact derived -from them by Chronicle or a consumer. Custody of a redistributable public-use -file's bytes is not content; custody of licensed or restricted bytes is never -taken at all. +from them by Chronicle or a consumer. Custody of a public-use file's bytes +under artifact-bound redistribution evidence is not content; custody of +licensed or restricted bytes is never taken at all. -1. **Registration.** A microdata release (CPS ASEC, ACS PUMS, SCF, SIPP, FRS, +1. **Classification is explicit.** Every manifest created or modified after + this ADR declares `kind`, either `publisher_table` or + `microdata_release`. `validate-package`, `fetch-artifact`, `publish-raw`, + `register-artifact`, and the source-package byte reader refuse a new or + modified manifest without it, at every entry point. The manifests that + exist at #227's merge commit are grandfathered as publisher tables by an + explicit frozen list checked into the repository; a kindless manifest + outside that list is an error, never a publisher table by default. + `microdata_release` is what lets every parser refuse the file, so a public + microdata release can never be mistaken for a public aggregate workbook. +2. **Registration.** A microdata release (CPS ASEC, ACS PUMS, SCF, SIPP, FRS, BE-SILC, the IRS PUF, and their successors in every jurisdiction) is a - source artifact like any publisher workbook: publisher, source URL or - access route, vintage, SHA-256, size, licence, an access class from a - closed set (`public`, `licensed`, `restricted`), and the hash-source and - verification fields defined below, on a manifest declared - `kind: microdata_release`. That kind is required, not inferred: it is what - lets `validate-package`, the suite builder, the source-package byte reader, - and every parser refuse the file, so a public microdata release can never - be mistaken for a public aggregate workbook. A manifest without `kind` is - a publisher table; every existing manifest is one. Registration is - manifest-level. It uses the `fetch-artifact` / `publish-raw` path extended - with an access-aware refusal (chronicle#221), and the content-addressed key + source artifact registered by identity: publisher, source URL or access + route, vintage, SHA-256, size, licence, an access class from a closed set + (`public`, `licensed`, `restricted`), and the attestation fields in + decision 5. One package per publisher release, its files listed under the + vintage year; a registration is identified by + `{source_id, package_id, year, sha256, filename}`. Keys follow the convention in `docs/storage-architecture.md`: `raw/{country}/{source_id}/{package_id}/{year}/{sha256}/{filename}` for - publishers mapped to a country segment (UK and New Zealand today), and the + publishers mapped to a country segment (UK and New Zealand today) and the legacy `raw/{source_id}/{package_id}/{year}/{sha256}/{filename}` shape for every unmapped publisher, US sources included. No source package parses the file. -2. **Bytes only with an affirmative redistribution permission.** Being - downloadable is not a licence. Chronicle archives a release's bytes in the - raw bucket only when its `access` is `public` **and** its recorded - `licence` matches an entry in Chronicle's allowlist of redistributable - terms, maintained in code with the evidence for each entry (for example a - U.S. Government work under 17 U.S.C. §105, the Open Government Licence, - CC0, CC BY). A file that is publicly downloadable under any other or - unstated terms is classed `licensed`, whatever its access route. Licensed - or restricted files (FRS under the UKDS agreement, BE-SILC scientific-use - files, the IRS PUF) are registered hash-only: no bytes in any Chronicle - store, and no Chronicle credential grants access to them. Their bytes stay - in the licensed environments consumers already operate. -3. **No records, rows, columns, row values, cells, or derived facts.** No - microdata record, row, column, row value, or cell enters `source_records`, - `source_rows`, `source_columns`, `source_row_values`, `source_cells`, the - relational registry, the derived-artifact bucket, or the journal, whether - the release is public or gated. No fact derived from raw microdata by - Chronicle or by a consumer enters Chronicle, however many intermediate - artifacts stand between the microdata and the value. The only exception is - assertion-based, as everywhere else in Chronicle: an aggregate that a - publisher computes from its own microdata and publishes is an ordinary - fact with ordinary provenance. -4. **What a registration attests.** `hash_source` records who computed the - registered SHA-256: `chronicle_fetch` (Chronicle fetched the bytes and - hashed them; the registration attests the bytes), `consumer_attested` (a - consumer that holds the bytes recomputed the hash and recorded the - evidence; the registration attests the bytes on that consumer's word), or - `consumer_pin` (transcribed from a consumer's reviewed pin without - recomputation; the registration attests the pin, not the bytes). - `verified_at` is the date the recorded hash was last compared against the - bytes by whoever holds them. The witnessed fetch time in the journal - bounds when the registration existed; it bounds when the bytes existed - only for `chronicle_fetch` and `consumer_attested` entries. -5. **Consumers point at the registration.** A Microcosm source-stage manifest +3. **Bytes only with artifact-bound redistribution evidence.** Being + downloadable is not a licence, and an allowlisted licence name is not + evidence that this file was issued under it. Chronicle archives a release's + bytes only when `access` is `public`, the recorded `licence` is on + Chronicle's allowlist of redistributable terms (maintained in code with the + evidence for each term: a U.S. Government work under 17 U.S.C. §105, the + Open Government Licence v3, CC0, CC BY), and the entry carries + `licence_evidence` binding the artifact to the term: issuer, licence + identifier and version, the scope statement, a durable evidence URL, and + the covered SHA-256. The `ledger-boundary` judge verifies that the evidence + names the artifact. Any public-download file without that evidence is + classed `licensed`. Licensed or restricted files (FRS under the UKDS + agreement, BE-SILC scientific-use files, the IRS PUF) are registered + hash-only: no bytes in any Chronicle store, no Chronicle credential grants + access to them, and their bytes stay in the licensed environments + consumers already operate. Public microdata bytes are staged in an + untracked, transient directory outside `db/data/**` during acquisition and + uploaded from there; a repository guard refuses tracked microdata bytes. + Git custody of a microdata release is always manifest-only. +4. **No content.** No microdata record, row, column, row value, or cell + enters `source_records`, `source_rows`, `source_columns`, + `source_row_values`, `source_cells`, the relational registry, the + derived-artifact bucket, or the journal, whether the release is public or + gated. No fact derived from raw microdata by Chronicle or by a consumer + enters Chronicle, however many intermediate artifacts stand between the + microdata and the value. The only exception is assertion-based: an + aggregate that the publisher of that microdata computed and published + itself is an ordinary fact with ordinary provenance. This is enforceable + only once facts carry an asserting principal and root-artifact lineage + (`asserted_by`, `root_artifacts`; chronicle#238): a fact rooted in a + `microdata_release` registration is then rejected unless `asserted_by` is + that release's publisher. Until then reviewers enforce it by hand. +5. **What a registration attests, and who.** Each registration records + `hash_source` and its attester: + - `chronicle_fetch`: Chronicle fetched the bytes and hashed them; + `attested_by: chronicle`, `verified_at` is the fetch date. The + registration attests the bytes. + - `consumer_attested`: a consumer that holds the bytes recomputed the hash + and recorded evidence; `attested_by` is that consumer, `attestation_evidence` + points at its record, `verified_at` is its comparison date. The + registration attests the bytes on that consumer's word. + - `consumer_pin`: transcribed from a consumer's reviewed pin without + recomputation; `attested_by` is that consumer, `pinned_from` names the + repository path and commit, and there is no `verified_at`. The + registration attests the pin, not the bytes. + The witnessed journal (the release manifests on `codex/thesis-ledger-facts`, + each committing to the journal state that first covers a registration's + manifest hash) bounds when the registration existed. It bounds when the + bytes existed only for `chronicle_fetch` and `consumer_attested` entries. +6. **Consumers point at the registration.** A Microcosm source-stage manifest that names a microdata artifact carries the Chronicle artifact reference and the same SHA-256, so every root of a build graph resolves to one witnessed registration and a build fails closed when its local bytes @@ -82,10 +104,10 @@ taken at all. - **The publisher record is the thing to witness.** Publishers revise and withdraw microdata files: the IRS withdrew the public-use file in 2026, and Census reissues ASEC files under the same vintage label. A registered hash - with a witnessed fetch time is the only durable statement that a given - release existed with those bytes. This is the same transparency property - Chronicle already provides for published tables, applied to the files - calibration actually starts from. + with a witnessed time is the only durable statement that a given release + existed with those bytes. This is the same transparency property Chronicle + already provides for published tables, applied to the files calibration + actually starts from. - **Pins today are scattered and unwitnessed.** Microcosm pins raw inputs in per-country manifests, checkpoint metadata, code constants, and command-line arguments, with no shared registry, no licence record, and no timestamp @@ -99,30 +121,31 @@ taken at all. Thesis scores. - **This keeps the 2026-06-30 ruling.** PR #68 removed microdata parsers, adapters, and tracked raw storage from the package. Nothing here brings any - of that back. Identity registration adds manifests and manifest fields, and - the code it adds refuses to read microdata rather than reading it. + of that back. The code the enforcement PRs add refuses to read microdata + rather than reading it. ## Consequences -- Manifests gain `licence` (publisher terms, as an identifier or URL), - `access` (`public` | `licensed` | `restricted`), `hash_source`, and - `verified_at`; a microdata manifest declares `kind: microdata_release`. - Registration of a `licensed` or `restricted` artifact records the checksum - and access route and refuses bytes; registration of a `public` artifact - archives bytes only when the licence is on the redistributable allowlist. -- `docs/storage-architecture.md` narrows its non-goal from "no raw survey or - administrative microdata" to "no microdata records, rows, columns, row - values, cells, or derived facts; no licensed or restricted microdata bytes"; - its ownership matrix gains a row for microdata releases. - `docs/target-construction-harness-plan.md` is scoped to publisher aggregate - artifacts; microdata releases are identity-only. -- `docs/chronicle-governance.md` allows registering microdata releases and - forbids parsing them or holding gated bytes; the `ledger-boundary` judge - contract in `.github/chronicle-agents.yml` names both refusals in the same - canonical words, and `tests/test_chronicle_governance.py` pins that - wording, so the configured judge cannot pass a change that does either. -- Microcosm's raw-input entries reference Chronicle registrations; the - consumer side is tracked in PolicyEngine/microcosm. +- `docs/adr-chronicle-facts-only.md` is amended: "anything a publisher + asserted is a fact" now excludes publisher-authored values at microdata + grain, which are content; only the publisher's published aggregates over + its microdata are facts. README's boundary block says the same. +- Manifests gain `kind`, `licence`, `access`, `licence_evidence`, + `hash_source`, `attested_by`, and per-source `verified_at` / + `attestation_evidence` / `pinned_from`; validators refuse the combinations + decisions 1, 3, and 5 forbid (#227). +- Fact provenance gains `asserted_by` and `root_artifacts` (#238). +- `docs/storage-architecture.md` narrows its non-goal to the three clauses in + `chronicle/boundary.py`; its ownership matrix gains a row for microdata + releases and states that Git custody is manifest-only with untracked + staging. `docs/target-construction-harness-plan.md` is scoped to publisher + aggregate artifacts. +- `chronicle/boundary.py` states the three negative clauses once; + `.github/chronicle-agents.yml` carries them verbatim in the + `ledger-boundary` judge verdict; `tests/test_chronicle_governance.py` + asserts each complete clause, so the wording cannot regress or invert. +- Microcosm's raw-input entries reference Chronicle registrations + (PolicyEngine/microcosm#848). - Flip conditions for revisiting content: a consumer other than Microcosm needs row-level evidence from Chronicle, or a publisher grants redistribution of a currently licensed file and Chronicle has an diff --git a/docs/chronicle-governance.md b/docs/chronicle-governance.md index 6ca5330..bed2c41 100644 --- a/docs/chronicle-governance.md +++ b/docs/chronicle-governance.md @@ -73,6 +73,5 @@ types are: The overall verdict fails if any required judge fails. A judge must fail if a change moves reconciliation, aging, imputation, active target selection, or solver construction from Microcosm into Chronicle, stores a -PolicyEngine-computed value as a fact, parses microdata into records, rows, -columns, row values, or cells, derives facts from raw microdata, or stores -licensed or restricted microdata bytes. +PolicyEngine-computed value as a fact, or breaks any clause of the microdata +boundary in `chronicle/boundary.py`, quoted here verbatim: (1) no microdata records, rows, columns, row values, or cells enter any Chronicle parsed-source surface, registry, derived artifact, or journal; (2) no fact derived from raw microdata by Chronicle or by a consumer enters Chronicle, however many intermediate artifacts stand between them, the only exception being an aggregate that the publisher of that microdata computed and published itself; (3) no licensed or restricted microdata bytes enter any Chronicle store, and public microdata bytes enter only with artifact-bound redistribution evidence. diff --git a/docs/storage-architecture.md b/docs/storage-architecture.md index 7d97515..a62059d 100644 --- a/docs/storage-architecture.md +++ b/docs/storage-architecture.md @@ -49,7 +49,7 @@ Hosted tables mirror accepted build outputs and provide a shared query surface. |------------|-------------------|---------------|-------------------|------------------|-------------------| | Source package specs | Authoritative YAML and parser code | No | Optional packaged snapshot | No | Metadata only | | Raw publisher files | Tiny fixtures only | Authoritative bytes | No | Metadata only | Metadata plus R2 pointer | -| Raw microdata releases | Manifest only (`kind: microdata_release`), never bytes | Bytes only for `public` releases on the redistributable allowlist; hash-only otherwise | No | Not represented until chronicle#221 | Not represented until chronicle#221 | +| Raw microdata releases | Manifest only (`kind: microdata_release`); public bytes staged in an untracked transient directory, never tracked | Bytes only for `public` releases with artifact-bound redistribution evidence; hash-only otherwise | No | Not represented until chronicle#221 | Not represented until chronicle#221 | | Source manifests | Authoritative checked metadata | No | Optional snapshot | Metadata loaded into tables | Queryable artifact registry | | Parsed source rows/cells | Generated local output | No | Snapshot artifact | Queryable table | Queryable mirror | | Source records/facts | Generated local output | No | Snapshot artifact | Queryable table | Queryable mirror | diff --git a/tests/test_chronicle_governance.py b/tests/test_chronicle_governance.py index 56c0f99..2d233b5 100644 --- a/tests/test_chronicle_governance.py +++ b/tests/test_chronicle_governance.py @@ -51,12 +51,13 @@ def test_chronicle_governance_files_define_required_review_surface(): assert judge_id in pr_template boundary_verdict = agents["required_judges"]["ledger-boundary"]["verdict"] - for phrase in ( - "microdata records, rows, columns, row values, or cells", - "facts derived from raw microdata", - "licensed or restricted microdata bytes", - ): - assert phrase in boundary_verdict + from chronicle.boundary import MICRODATA_BOUNDARY_CLAUSES + + for clause in MICRODATA_BOUNDARY_CLAUSES: + assert clause in boundary_verdict + governance_doc = (REPO_ROOT / "docs" / "chronicle-governance.md").read_text() + for clause in MICRODATA_BOUNDARY_CLAUSES: + assert clause in governance_doc def test_ci_runs_boundary_and_governance_tests(): From 53e2ddb95b90a52292bac1d02c23502e2f009dbc Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 3 Sep 2026 09:31:54 -0400 Subject: [PATCH 07/11] Use the test module's ROOT constant in the boundary-clause assertion Co-Authored-By: Claude Fable 5.1 --- tests/test_chronicle_governance.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/test_chronicle_governance.py b/tests/test_chronicle_governance.py index 2d233b5..682dbb3 100644 --- a/tests/test_chronicle_governance.py +++ b/tests/test_chronicle_governance.py @@ -55,7 +55,7 @@ def test_chronicle_governance_files_define_required_review_surface(): for clause in MICRODATA_BOUNDARY_CLAUSES: assert clause in boundary_verdict - governance_doc = (REPO_ROOT / "docs" / "chronicle-governance.md").read_text() + governance_doc = (ROOT / "docs" / "chronicle-governance.md").read_text() for clause in MICRODATA_BOUNDARY_CLAUSES: assert clause in governance_doc From f148bfc0616402c291b7644233bad7ce1b77f208 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 3 Sep 2026 11:17:32 -0400 Subject: [PATCH 08/11] Address the gate's round 1: define consumer, state the exception as who asserted the value, align byte-custody wording, drop the phantom command, rewrap --- .github/chronicle-agents.yml | 2 +- AGENTS.md | 9 +++--- chronicle/__init__.py | 11 ++++---- chronicle/boundary.py | 10 ++++--- docs/adr-chronicle-facts-only.md | 5 ++-- docs/adr-chronicle-raw-microdata-identity.md | 29 ++++++++++++-------- docs/chronicle-governance.md | 13 +++++---- docs/storage-architecture.md | 7 +++-- 8 files changed, 51 insertions(+), 35 deletions(-) diff --git a/.github/chronicle-agents.yml b/.github/chronicle-agents.yml index f16217d..e318433 100644 --- a/.github/chronicle-agents.yml +++ b/.github/chronicle-agents.yml @@ -43,4 +43,4 @@ required_judges: ledger-contract: verdict: PASS if schema or identity changes preserve source provenance and do not move Microcosm responsibilities into Chronicle. ledger-boundary: - verdict: PASS if reconciliation, aging, imputation, support-aware activation, and solver construction remain outside Chronicle; no microdata records, rows, columns, row values, or cells enter any Chronicle parsed-source surface, registry, derived artifact, or journal; no fact derived from raw microdata by Chronicle or by a consumer enters Chronicle, however many intermediate artifacts stand between them, the only exception being an aggregate that the publisher of that microdata computed and published itself; no licensed or restricted microdata bytes enter any Chronicle store, and public microdata bytes enter only with artifact-bound redistribution evidence (docs/adr-chronicle-raw-microdata-identity.md). + verdict: PASS if reconciliation, aging, imputation, support-aware activation, and solver construction remain outside Chronicle; no microdata records, rows, columns, row values, or cells enter any Chronicle parsed-source surface, registry, derived artifact, or journal; no fact computed from raw microdata by Chronicle or by a PolicyEngine-side consumer (Microcosm, PolicyEngine, Thesis, or any system that builds from a Chronicle registration) enters Chronicle, however many intermediate artifacts stand between them, while a value that a third party asserted and published, whether the microdata's own publisher or another, is an ordinary fact with ordinary provenance; no licensed or restricted microdata bytes enter any Chronicle store, and public microdata bytes enter only with artifact-bound redistribution evidence (docs/adr-chronicle-raw-microdata-identity.md). diff --git a/AGENTS.md b/AGENTS.md index 3594db6..da075c7 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -20,10 +20,11 @@ Do not put Microcosm work in Chronicle: - no target profiles or model-measurement bindings - no PolicyEngine-computed values stored as facts - no microdata records, rows, columns, row values, or cells, no facts - derived from raw microdata by Chronicle or a consumer, and no licensed or - restricted microdata bytes in any Chronicle store (registering a release's - identity is allowed, and a publisher's own published aggregates remain - ordinary facts; see `docs/adr-chronicle-raw-microdata-identity.md`) + computed from raw microdata by Chronicle or by a PolicyEngine-side consumer + (Microcosm, PolicyEngine, Thesis), and no licensed or restricted microdata + bytes in any Chronicle store (registering a release's identity is allowed, + and any value a third party asserted and published remains an ordinary + fact; see `docs/adr-chronicle-raw-microdata-identity.md`) Chronicle records every fact's publisher reference period. Consumers own and enforce any declaration that aligns those facts to another period; Chronicle diff --git a/chronicle/__init__.py b/chronicle/__init__.py index a944c94..d4a8e56 100644 --- a/chronicle/__init__.py +++ b/chronicle/__init__.py @@ -2,11 +2,12 @@ Chronicle owns government-statistics releases: source artifacts (and, once chronicle#221 lands, registrations of raw microdata releases plus custody of -public-use bytes with redistribution evidence), source-backed facts, constraints, and provenance. Microdata -content (records, rows, columns, row values, cells, and facts derived from raw -microdata), licensed or restricted microdata bytes, selection contracts, source reconciliation, aging, -imputation, target activation, and calibration belong in downstream systems -such as Microcosm. +public-use bytes with redistribution evidence), source-backed facts, +constraints, and provenance. Microdata content (records, rows, columns, row +values, cells, and facts computed from raw microdata by Chronicle or a +PolicyEngine-side consumer), licensed or restricted microdata bytes, selection +contracts, source reconciliation, aging, imputation, target activation, and +calibration belong in downstream systems such as Microcosm. """ __all__ = [ diff --git a/chronicle/boundary.py b/chronicle/boundary.py index 6a46edb..459db71 100644 --- a/chronicle/boundary.py +++ b/chronicle/boundary.py @@ -13,10 +13,12 @@ MICRODATA_BOUNDARY_CLAUSES: tuple[str, ...] = ( "no microdata records, rows, columns, row values, or cells enter any " "Chronicle parsed-source surface, registry, derived artifact, or journal", - "no fact derived from raw microdata by Chronicle or by a consumer enters " - "Chronicle, however many intermediate artifacts stand between them, the only " - "exception being an aggregate that the publisher of that microdata computed " - "and published itself", + "no fact computed from raw microdata by Chronicle or by a PolicyEngine-side " + "consumer (Microcosm, PolicyEngine, Thesis, or any system that builds from a " + "Chronicle registration) enters Chronicle, however many intermediate artifacts " + "stand between them, while a value that a third party asserted and published, " + "whether the microdata's own publisher or another, is an ordinary fact with " + "ordinary provenance", "no licensed or restricted microdata bytes enter any Chronicle store, and " "public microdata bytes enter only with artifact-bound redistribution evidence", ) diff --git a/docs/adr-chronicle-facts-only.md b/docs/adr-chronicle-facts-only.md index 4ffdb96..e959a84 100644 --- a/docs/adr-chronicle-facts-only.md +++ b/docs/adr-chronicle-facts-only.md @@ -12,8 +12,9 @@ the value**, not level versus projection: projections — with one grain exception added by `docs/adr-chronicle-raw-microdata-identity.md`: a publisher-authored value at microdata grain (a record, row, column, row value, or cell of a - microdata release) is content, not a fact; only the aggregates the - publisher itself computed and published over that microdata are facts. "CBO's January 2026 baseline projects individual income tax + microdata release) is content, not a fact; only the values + a publisher asserted and published over that microdata are facts. "CBO's + January 2026 baseline projects individual income tax receipts of $X in 2027" is a source-backed claim with lineage, exactly like an SOI observation. These facts carry `assertion: source_projection`; measured or administered outcomes carry diff --git a/docs/adr-chronicle-raw-microdata-identity.md b/docs/adr-chronicle-raw-microdata-identity.md index f0b46ad..1d3750a 100644 --- a/docs/adr-chronicle-raw-microdata-identity.md +++ b/docs/adr-chronicle-raw-microdata-identity.md @@ -2,8 +2,8 @@ Status: accepted 2026-09-02. Enforcement lands in two steps and this ADR's guarantees are prospective until they merge: chronicle#221 (PR #227: manifest -classification, the access-aware refusals in `fetch-artifact`, `publish-raw`, -`register-artifact`, and the source-package byte reader, the redistribution +classification, the access-aware refusals in `validate-package`, +`fetch-artifact`, `publish-raw`, and the source-package byte reader, the redistribution allowlist, artifact-bound licence evidence, untracked staging for public microdata bytes) and chronicle#238 (asserting principal and root-artifact lineage on fact provenance). Until #227 merges, no microdata release may be @@ -15,15 +15,18 @@ obligation, not a validator. Chronicle registers every raw microdata release its consumers build from, and stores none of their content. "Content" means any parsed representation: -microdata records, rows, columns, row values, or cells, and any fact derived -from them by Chronicle or a consumer. Custody of a public-use file's bytes +microdata records, rows, columns, row values, or cells, and any fact computed +from them by Chronicle or by a PolicyEngine-side consumer (Microcosm, +PolicyEngine, Thesis, or any system that builds from a Chronicle +registration). A value that a third party asserted and published, whether the +microdata's own publisher or another, is an ordinary fact. Custody of a public-use file's bytes under artifact-bound redistribution evidence is not content; custody of licensed or restricted bytes is never taken at all. 1. **Classification is explicit.** Every manifest created or modified after this ADR declares `kind`, either `publisher_table` or `microdata_release`. `validate-package`, `fetch-artifact`, `publish-raw`, - `register-artifact`, and the source-package byte reader refuse a new or + and the source-package byte reader refuse a new or modified manifest without it, at every entry point. The manifests that exist at #227's merge commit are grandfathered as publisher tables by an explicit frozen list checked into the repository; a kindless manifest @@ -67,15 +70,19 @@ licensed or restricted bytes is never taken at all. enters `source_records`, `source_rows`, `source_columns`, `source_row_values`, `source_cells`, the relational registry, the derived-artifact bucket, or the journal, whether the release is public or - gated. No fact derived from raw microdata by Chronicle or by a consumer - enters Chronicle, however many intermediate artifacts stand between the - microdata and the value. The only exception is assertion-based: an - aggregate that the publisher of that microdata computed and published - itself is an ordinary fact with ordinary provenance. This is enforceable + gated. No fact computed from raw microdata by Chronicle or by a + PolicyEngine-side consumer enters Chronicle, however many intermediate + artifacts stand between the microdata and the value. The test is who + asserted the value: a value that a third party asserted and published, + whether the microdata's own publisher (Census over the ASEC) or another + publisher (JCT or TPC over the IRS PUF, JRC over EU-SILC), is an ordinary + fact with ordinary provenance; a value Chronicle or a PolicyEngine-side + consumer computed is not. This is enforceable only once facts carry an asserting principal and root-artifact lineage (`asserted_by`, `root_artifacts`; chronicle#238): a fact rooted in a `microdata_release` registration is then rejected unless `asserted_by` is - that release's publisher. Until then reviewers enforce it by hand. + a third-party publisher, never Chronicle or a PolicyEngine-side consumer. + Until then reviewers enforce it by hand. 5. **What a registration attests, and who.** Each registration records `hash_source` and its attester: - `chronicle_fetch`: Chronicle fetched the bytes and hashed them; diff --git a/docs/chronicle-governance.md b/docs/chronicle-governance.md index bed2c41..56dc0b6 100644 --- a/docs/chronicle-governance.md +++ b/docs/chronicle-governance.md @@ -22,8 +22,9 @@ Chronicle may: - register raw publisher artifacts and checksums - register raw microdata releases as source artifacts (publisher, access route, vintage, checksum, licence, hash source, verification date), - archiving bytes only when the release is `public` and its licence is on - the redistributable allowlist + archiving bytes only when the release is `public`, its licence is on + the redistributable allowlist, and the entry carries artifact-bound + redistribution evidence naming the file - parse source rows and cells - emit source-backed aggregate facts, including publisher projections typed `assertion: source_projection` @@ -42,9 +43,9 @@ Chronicle must not: - compute aligned values; it publishes the source period and value unchanged - impute missing values - parse survey or administrative microdata into records, rows, columns, row - values, or cells, derive facts from raw microdata (by Chronicle or a - consumer), or hold licensed or restricted microdata bytes in any Chronicle - store + values, or cells, compute facts from raw microdata (by Chronicle or a + PolicyEngine-side consumer), or hold licensed or restricted microdata bytes + in any Chronicle store - own selection, measurement, period-alignment, or model-binding contracts - choose a support-aware active target subset - build solver-ready calibration targets @@ -74,4 +75,4 @@ The overall verdict fails if any required judge fails. A judge must fail if a change moves reconciliation, aging, imputation, active target selection, or solver construction from Microcosm into Chronicle, stores a PolicyEngine-computed value as a fact, or breaks any clause of the microdata -boundary in `chronicle/boundary.py`, quoted here verbatim: (1) no microdata records, rows, columns, row values, or cells enter any Chronicle parsed-source surface, registry, derived artifact, or journal; (2) no fact derived from raw microdata by Chronicle or by a consumer enters Chronicle, however many intermediate artifacts stand between them, the only exception being an aggregate that the publisher of that microdata computed and published itself; (3) no licensed or restricted microdata bytes enter any Chronicle store, and public microdata bytes enter only with artifact-bound redistribution evidence. +boundary in `chronicle/boundary.py`, quoted here verbatim: (1) no microdata records, rows, columns, row values, or cells enter any Chronicle parsed-source surface, registry, derived artifact, or journal; (2) no fact computed from raw microdata by Chronicle or by a PolicyEngine-side consumer (Microcosm, PolicyEngine, Thesis, or any system that builds from a Chronicle registration) enters Chronicle, however many intermediate artifacts stand between them, while a value that a third party asserted and published, whether the microdata's own publisher or another, is an ordinary fact with ordinary provenance; (3) no licensed or restricted microdata bytes enter any Chronicle store, and public microdata bytes enter only with artifact-bound redistribution evidence. diff --git a/docs/storage-architecture.md b/docs/storage-architecture.md index a62059d..d817b95 100644 --- a/docs/storage-architecture.md +++ b/docs/storage-architecture.md @@ -16,7 +16,9 @@ content-addressed by checksum and should never be overwritten in place. Raw microdata releases follow the same rule at the artifact level only. The bytes of a `public` release whose licence is on Chronicle's redistributable -allowlist (Census public-use files are the model case) live in the raw bucket +allowlist, and whose registration carries artifact-bound redistribution +evidence naming the file (Census public-use files are the model case), live in +the raw bucket under the same content-addressed key shape. Every other microdata release (FRS, BE-SILC, the IRS PUF, and any publicly downloadable file under unstated terms) is registered by manifest, with checksum, vintage, licence, @@ -172,7 +174,8 @@ Chronicle should not parse survey or administrative microdata into records, rows, columns, row values, or cells, derive facts from it, or hold licensed or restricted microdata bytes in any store. It registers microdata releases as source artifacts and archives bytes only for public releases on its -redistributable allowlist (see +redistributable allowlist that carry artifact-bound redistribution evidence +(see `docs/adr-chronicle-raw-microdata-identity.md`). It reflects government statistics releases and the provenance needed to audit those published facts. From af43d371878dabdef97f78c632ec06b1b239df8b Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 3 Sep 2026 11:23:56 -0400 Subject: [PATCH 09/11] Round 2: mirror the who-asserted-the-value wording in README, the Consequences bullet, and the architecture table; rewrap --- README.md | 5 +++-- docs/adr-chronicle-raw-microdata-identity.md | 20 +++++++++++--------- docs/architecture.md | 2 +- 3 files changed, 15 insertions(+), 12 deletions(-) diff --git a/README.md b/README.md index 84eb09d..dea6764 100644 --- a/README.md +++ b/README.md @@ -57,8 +57,9 @@ The load-bearing rule: The store is facts-only, and the line is who asserted the value. Everything a publisher asserted — including the publisher's own projections — is a fact, except publisher-authored values at microdata grain, which are content and -never facts; only the publisher's own published aggregates over its microdata -are facts (see `docs/adr-chronicle-raw-microdata-identity.md`). +never facts; only the values a publisher asserted and published over that +microdata are facts, whether the microdata's own publisher or another (see +`docs/adr-chronicle-raw-microdata-identity.md`). Everything PolicyEngine computes (aged, uprated, forecast, or reconciled levels) is a downstream build artifact and never enters the store; Microcosm owns aging as a named, versioned model over Chronicle growth-factor facts. A diff --git a/docs/adr-chronicle-raw-microdata-identity.md b/docs/adr-chronicle-raw-microdata-identity.md index 1d3750a..c1c7c42 100644 --- a/docs/adr-chronicle-raw-microdata-identity.md +++ b/docs/adr-chronicle-raw-microdata-identity.md @@ -3,9 +3,10 @@ Status: accepted 2026-09-02. Enforcement lands in two steps and this ADR's guarantees are prospective until they merge: chronicle#221 (PR #227: manifest classification, the access-aware refusals in `validate-package`, -`fetch-artifact`, `publish-raw`, and the source-package byte reader, the redistribution -allowlist, artifact-bound licence evidence, untracked staging for public -microdata bytes) and chronicle#238 (asserting principal and root-artifact +`fetch-artifact`, `publish-raw`, and the source-package byte reader, the +redistribution allowlist, artifact-bound licence evidence, untracked staging +for public microdata bytes) and chronicle#238 (asserting principal and +root-artifact lineage on fact provenance). Until #227 merges, no microdata release may be pointed at any Chronicle command: today they materialize, upload, and parse every input. Until #238 merges, the derived-fact prohibition is a review @@ -18,10 +19,10 @@ stores none of their content. "Content" means any parsed representation: microdata records, rows, columns, row values, or cells, and any fact computed from them by Chronicle or by a PolicyEngine-side consumer (Microcosm, PolicyEngine, Thesis, or any system that builds from a Chronicle -registration). A value that a third party asserted and published, whether the -microdata's own publisher or another, is an ordinary fact. Custody of a public-use file's bytes -under artifact-bound redistribution evidence is not content; custody of -licensed or restricted bytes is never taken at all. +registration). A value that a third party asserted and published, whether +the microdata's own publisher or another, is an ordinary fact. Custody of a +public-use file's bytes under artifact-bound redistribution evidence is not +content; custody of licensed or restricted bytes is never taken at all. 1. **Classification is explicit.** Every manifest created or modified after this ADR declares `kind`, either `publisher_table` or @@ -135,8 +136,9 @@ licensed or restricted bytes is never taken at all. - `docs/adr-chronicle-facts-only.md` is amended: "anything a publisher asserted is a fact" now excludes publisher-authored values at microdata - grain, which are content; only the publisher's published aggregates over - its microdata are facts. README's boundary block says the same. + grain, which are content; only the values a publisher asserted and + published over that microdata are facts, whether the microdata's own + publisher or another. README's boundary block says the same. - Manifests gain `kind`, `licence`, `access`, `licence_evidence`, `hash_source`, `attested_by`, and per-source `verified_at` / `attestation_evidence` / `pinned_from`; validators refuse the combinations diff --git a/docs/architecture.md b/docs/architecture.md index ae1eb78..56453ad 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -16,7 +16,7 @@ validation, and build-harness layer. Jurisdiction source packages such as | Layer | Owns | Does not own | |-------|------|--------------| -| Chronicle | Source artifacts (including microdata release registrations), provenance, aggregate facts, constraints | Selection and measurement contracts, microdata records, rows, columns, row values, or cells; facts derived from raw microdata by Chronicle or a consumer; licensed or restricted microdata bytes; source reconciliation, aging, imputation, active target selection | +| Chronicle | Source artifacts (including microdata release registrations), provenance, aggregate facts, constraints | Selection and measurement contracts, microdata records, rows, columns, row values, or cells; facts computed from raw microdata by Chronicle or a PolicyEngine-side consumer; licensed or restricted microdata bytes; source reconciliation, aging, imputation, active target selection | | Microcosm Targets | Selection and measurement contracts, reconciliation, aging, imputation, active target sets | Source artifact storage and provenance | | Microcosm | Entity model, weights, calibration interfaces, calibrated output | Source ETL and source provenance | | Jurisdiction source packages | Source-specific parsers and specs that emit Chronicle records | Forked fact or constraint schemas | From f1c1b6a643595dcd1b976164748b44edf5ee8d68 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 3 Sep 2026 11:31:23 -0400 Subject: [PATCH 10/11] Round 3: reflow every touched paragraph to the files' column --- README.md | 13 ++-- docs/adr-chronicle-facts-only.md | 17 ++--- docs/adr-chronicle-raw-microdata-identity.md | 77 ++++++++++---------- docs/storage-architecture.md | 12 ++- 4 files changed, 57 insertions(+), 62 deletions(-) diff --git a/README.md b/README.md index dea6764..81c2d12 100644 --- a/README.md +++ b/README.md @@ -59,13 +59,12 @@ publisher asserted — including the publisher's own projections — is a fact, except publisher-authored values at microdata grain, which are content and never facts; only the values a publisher asserted and published over that microdata are facts, whether the microdata's own publisher or another (see -`docs/adr-chronicle-raw-microdata-identity.md`). -Everything PolicyEngine computes (aged, uprated, forecast, or reconciled -levels) is a downstream build artifact and never enters the store; Microcosm -owns aging as a named, versioned model over Chronicle growth-factor facts. A -fact's `period` is the period its value refers to. Consumers must enforce any -contract that aligns it to another period (see -[`docs/adr-chronicle-facts-only.md`](docs/adr-chronicle-facts-only.md)). +`docs/adr-chronicle-raw-microdata-identity.md`). Everything PolicyEngine +computes (aged, uprated, forecast, or reconciled levels) is a downstream build +artifact and never enters the store; Microcosm owns aging as a named, versioned +model over Chronicle growth-factor facts. A fact's `period` is the period its +value refers to. Consumers must enforce any contract that aligns it to another +period (see [`docs/adr-chronicle-facts-only.md`](docs/adr-chronicle-facts-only.md)). | Layer | Owns | Examples | |-------|------|----------| diff --git a/docs/adr-chronicle-facts-only.md b/docs/adr-chronicle-facts-only.md index e959a84..7992fa5 100644 --- a/docs/adr-chronicle-facts-only.md +++ b/docs/adr-chronicle-facts-only.md @@ -10,15 +10,14 @@ the value**, not level versus projection: 1. Anything a publisher asserted is a fact — including the publisher's own projections — with one grain exception added by - `docs/adr-chronicle-raw-microdata-identity.md`: a publisher-authored - value at microdata grain (a record, row, column, row value, or cell of a - microdata release) is content, not a fact; only the values - a publisher asserted and published over that microdata are facts. "CBO's - January 2026 baseline projects individual income tax - receipts of $X in 2027" is a source-backed claim with lineage, exactly - like an SOI observation. These facts carry - `assertion: source_projection`; measured or administered outcomes carry - the default `assertion: observation`. + `docs/adr-chronicle-raw-microdata-identity.md`: a publisher-authored value + at microdata grain (a record, row, column, row value, or cell of a microdata + release) is content, not a fact; only the values a publisher asserted and + published over that microdata are facts. "CBO's January 2026 baseline + projects individual income tax receipts of $X in 2027" is a source-backed + claim with lineage, exactly like an SOI observation. These facts carry + `assertion: source_projection`; measured or administered outcomes carry the + default `assertion: observation`. 2. Anything PolicyEngine computed — an aged, uprated, forecast, or reconciled level — is never a Chronicle object. Such values are regenerable build artifacts and live in the consumer (Microcosm calibration owns diff --git a/docs/adr-chronicle-raw-microdata-identity.md b/docs/adr-chronicle-raw-microdata-identity.md index c1c7c42..7e955e6 100644 --- a/docs/adr-chronicle-raw-microdata-identity.md +++ b/docs/adr-chronicle-raw-microdata-identity.md @@ -6,11 +6,10 @@ classification, the access-aware refusals in `validate-package`, `fetch-artifact`, `publish-raw`, and the source-package byte reader, the redistribution allowlist, artifact-bound licence evidence, untracked staging for public microdata bytes) and chronicle#238 (asserting principal and -root-artifact -lineage on fact provenance). Until #227 merges, no microdata release may be -pointed at any Chronicle command: today they materialize, upload, and parse -every input. Until #238 merges, the derived-fact prohibition is a review -obligation, not a validator. +root-artifact lineage on fact provenance). Until #227 merges, no microdata +release may be pointed at any Chronicle command: today they materialize, +upload, and parse every input. Until #238 merges, the derived-fact prohibition +is a review obligation, not a validator. ## Decision @@ -18,22 +17,22 @@ Chronicle registers every raw microdata release its consumers build from, and stores none of their content. "Content" means any parsed representation: microdata records, rows, columns, row values, or cells, and any fact computed from them by Chronicle or by a PolicyEngine-side consumer (Microcosm, -PolicyEngine, Thesis, or any system that builds from a Chronicle -registration). A value that a third party asserted and published, whether -the microdata's own publisher or another, is an ordinary fact. Custody of a -public-use file's bytes under artifact-bound redistribution evidence is not -content; custody of licensed or restricted bytes is never taken at all. +PolicyEngine, Thesis, or any system that builds from a Chronicle registration). +A value that a third party asserted and published, whether the microdata's own +publisher or another, is an ordinary fact. Custody of a public-use file's bytes +under artifact-bound redistribution evidence is not content; custody of +licensed or restricted bytes is never taken at all. 1. **Classification is explicit.** Every manifest created or modified after - this ADR declares `kind`, either `publisher_table` or - `microdata_release`. `validate-package`, `fetch-artifact`, `publish-raw`, - and the source-package byte reader refuse a new or - modified manifest without it, at every entry point. The manifests that - exist at #227's merge commit are grandfathered as publisher tables by an - explicit frozen list checked into the repository; a kindless manifest - outside that list is an error, never a publisher table by default. - `microdata_release` is what lets every parser refuse the file, so a public - microdata release can never be mistaken for a public aggregate workbook. + this ADR declares `kind`, either `publisher_table` or `microdata_release`. + `validate-package`, `fetch-artifact`, `publish-raw`, and the source-package + byte reader refuse a new or modified manifest without it, at every entry + point. The manifests that exist at #227's merge commit are grandfathered as + publisher tables by an explicit frozen list checked into the repository; a + kindless manifest outside that list is an error, never a publisher table by + default. `microdata_release` is what lets every parser refuse the file, so a + public microdata release can never be mistaken for a public aggregate + workbook. 2. **Registration.** A microdata release (CPS ASEC, ACS PUMS, SCF, SIPP, FRS, BE-SILC, the IRS PUF, and their successors in every jurisdiction) is a source artifact registered by identity: publisher, source URL or access @@ -67,32 +66,32 @@ content; custody of licensed or restricted bytes is never taken at all. untracked, transient directory outside `db/data/**` during acquisition and uploaded from there; a repository guard refuses tracked microdata bytes. Git custody of a microdata release is always manifest-only. -4. **No content.** No microdata record, row, column, row value, or cell - enters `source_records`, `source_rows`, `source_columns`, - `source_row_values`, `source_cells`, the relational registry, the - derived-artifact bucket, or the journal, whether the release is public or - gated. No fact computed from raw microdata by Chronicle or by a - PolicyEngine-side consumer enters Chronicle, however many intermediate - artifacts stand between the microdata and the value. The test is who - asserted the value: a value that a third party asserted and published, - whether the microdata's own publisher (Census over the ASEC) or another - publisher (JCT or TPC over the IRS PUF, JRC over EU-SILC), is an ordinary - fact with ordinary provenance; a value Chronicle or a PolicyEngine-side - consumer computed is not. This is enforceable - only once facts carry an asserting principal and root-artifact lineage - (`asserted_by`, `root_artifacts`; chronicle#238): a fact rooted in a - `microdata_release` registration is then rejected unless `asserted_by` is - a third-party publisher, never Chronicle or a PolicyEngine-side consumer. - Until then reviewers enforce it by hand. +4. **No content.** No microdata record, row, column, row value, or cell enters + `source_records`, `source_rows`, `source_columns`, `source_row_values`, + `source_cells`, the relational registry, the derived-artifact bucket, or the + journal, whether the release is public or gated. No fact computed from raw + microdata by Chronicle or by a PolicyEngine-side consumer enters Chronicle, + however many intermediate artifacts stand between the microdata and the + value. The test is who asserted the value: a value that a third party + asserted and published, whether the microdata's own publisher (Census over + the ASEC) or another publisher (JCT or TPC over the IRS PUF, JRC over + EU-SILC), is an ordinary fact with ordinary provenance; a value Chronicle or + a PolicyEngine-side consumer computed is not. This is enforceable only once + facts carry an asserting principal and root-artifact lineage (`asserted_by`, + `root_artifacts`; chronicle#238): a fact rooted in a `microdata_release` + registration is then rejected unless `asserted_by` is a third-party + publisher, never Chronicle or a PolicyEngine-side consumer. Until then + reviewers enforce it by hand. 5. **What a registration attests, and who.** Each registration records `hash_source` and its attester: - `chronicle_fetch`: Chronicle fetched the bytes and hashed them; `attested_by: chronicle`, `verified_at` is the fetch date. The registration attests the bytes. - `consumer_attested`: a consumer that holds the bytes recomputed the hash - and recorded evidence; `attested_by` is that consumer, `attestation_evidence` - points at its record, `verified_at` is its comparison date. The - registration attests the bytes on that consumer's word. + and recorded evidence; `attested_by` is that consumer, + `attestation_evidence` points at its record, `verified_at` is its + comparison date. The registration attests the bytes on that consumer's + word. - `consumer_pin`: transcribed from a consumer's reviewed pin without recomputation; `attested_by` is that consumer, `pinned_from` names the repository path and commit, and there is no `verified_at`. The diff --git a/docs/storage-architecture.md b/docs/storage-architecture.md index d817b95..4ff6d91 100644 --- a/docs/storage-architecture.md +++ b/docs/storage-architecture.md @@ -18,11 +18,10 @@ Raw microdata releases follow the same rule at the artifact level only. The bytes of a `public` release whose licence is on Chronicle's redistributable allowlist, and whose registration carries artifact-bound redistribution evidence naming the file (Census public-use files are the model case), live in -the raw bucket -under the same content-addressed key shape. Every other microdata release -(FRS, BE-SILC, the IRS PUF, and any publicly downloadable file under -unstated terms) is registered by manifest, with checksum, vintage, licence, -access route, hash source, and verification date, and no bytes in any +the raw bucket under the same content-addressed key shape. Every other +microdata release (FRS, BE-SILC, the IRS PUF, and any publicly downloadable +file under unstated terms) is registered by manifest, with checksum, vintage, +licence, access route, hash source, and verification date, and no bytes in any Chronicle store. No microdata is parsed into records, rows, columns, row values, or cells, and no fact is derived from it. Until chronicle#221 adds registry columns, a registration exists only in its manifest: `ledger.db` and @@ -175,8 +174,7 @@ rows, columns, row values, or cells, derive facts from it, or hold licensed or restricted microdata bytes in any store. It registers microdata releases as source artifacts and archives bytes only for public releases on its redistributable allowlist that carry artifact-bound redistribution evidence -(see -`docs/adr-chronicle-raw-microdata-identity.md`). It reflects government +(see `docs/adr-chronicle-raw-microdata-identity.md`). It reflects government statistics releases and the provenance needed to audit those published facts. R2 should not be the schema authority. It stores bytes and reproducible build From 5604f9dd777a2bb38394305f185d7db89f3cb279 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 3 Sep 2026 11:37:49 -0400 Subject: [PATCH 11/11] Round 4: qualify registration on chronicle#221 landing; the clause-pinning test asserts the clause count --- AGENTS.md | 8 +++++--- docs/chronicle-governance.md | 10 ++++++---- tests/test_chronicle_governance.py | 2 ++ 3 files changed, 13 insertions(+), 7 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index da075c7..86eb518 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -22,9 +22,11 @@ Do not put Microcosm work in Chronicle: - no microdata records, rows, columns, row values, or cells, no facts computed from raw microdata by Chronicle or by a PolicyEngine-side consumer (Microcosm, PolicyEngine, Thesis), and no licensed or restricted microdata - bytes in any Chronicle store (registering a release's identity is allowed, - and any value a third party asserted and published remains an ordinary - fact; see `docs/adr-chronicle-raw-microdata-identity.md`) + bytes in any Chronicle store (registering a release's identity is allowed + once chronicle#221 lands its access-aware refusals, and until then no + microdata release may be pointed at any Chronicle command; any value a + third party asserted and published remains an ordinary fact; see + `docs/adr-chronicle-raw-microdata-identity.md`) Chronicle records every fact's publisher reference period. Consumers own and enforce any declaration that aligns those facts to another period; Chronicle diff --git a/docs/chronicle-governance.md b/docs/chronicle-governance.md index 56dc0b6..b9bb42b 100644 --- a/docs/chronicle-governance.md +++ b/docs/chronicle-governance.md @@ -21,10 +21,12 @@ Chronicle may: - register raw publisher artifacts and checksums - register raw microdata releases as source artifacts (publisher, access - route, vintage, checksum, licence, hash source, verification date), - archiving bytes only when the release is `public`, its licence is on - the redistributable allowlist, and the entry carries artifact-bound - redistribution evidence naming the file + route, vintage, checksum, licence, hash source, verification date) once + chronicle#221 lands its access-aware refusals, archiving bytes only when + the release is `public`, its licence is on the redistributable allowlist, + and the entry carries artifact-bound redistribution evidence naming the + file; until then no microdata release may be pointed at any Chronicle + command - parse source rows and cells - emit source-backed aggregate facts, including publisher projections typed `assertion: source_projection` diff --git a/tests/test_chronicle_governance.py b/tests/test_chronicle_governance.py index 682dbb3..de2e299 100644 --- a/tests/test_chronicle_governance.py +++ b/tests/test_chronicle_governance.py @@ -53,6 +53,8 @@ def test_chronicle_governance_files_define_required_review_surface(): boundary_verdict = agents["required_judges"]["ledger-boundary"]["verdict"] from chronicle.boundary import MICRODATA_BOUNDARY_CLAUSES + assert len(MICRODATA_BOUNDARY_CLAUSES) == 3 + assert all(len(clause) > 60 for clause in MICRODATA_BOUNDARY_CLAUSES) for clause in MICRODATA_BOUNDARY_CLAUSES: assert clause in boundary_verdict governance_doc = (ROOT / "docs" / "chronicle-governance.md").read_text()