You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Implements the raw-microdata identity ADR (docs/adr-chronicle-raw-microdata-identity.md, PR opened alongside this issue): Chronicle registers every raw microdata release its consumers build from, stores bytes only for redistributable public-use files, and never parses microdata into rows or facts.
Why now
Microcosm's raw inputs are pinned in four different places with no shared registry, licence record, or witnessed fetch time: per-country source_stages.json (UK: 19 licensed_microdata + 2 private_microdata entries carry sha256; US: 8 of 37 public_microdata entries carry sha256, 29 do not; BE: the restricted_microdata entry carries none), the ASEC raw-stage checkpoint metadata, code constants (e.g. weeks_unemployed.py), and build_us_multispine_pool.py --*-sha256 arguments. None references a Chronicle artifact.
Publishers revise and withdraw microdata (IRS PUF withdrawn 2026; Census reissues under the same vintage label). A registered hash with a witnessed fetch time is the only durable statement that a release existed with those bytes.
The node-graph rebuild (microcosm#836) declares identity per node; the roots of the graph should resolve to one witnessed registration.
Tasks
Manifest schema: add licence (publisher terms id or URL) and access (public | licensed | restricted, closed set) to manifest.yaml; validate in validate-package / artifact inventory. Today 1 of 159 manifests carries a license field.
Hash-only registration: fetch-artifact / publish-raw (or a sibling register-artifact) must accept a licensed/restricted entry with checksum, size, vintage, access route and refuse bytes; public entries archive bytes under raw/{source_id}/{package_id}/{year}/{sha256}/{filename} as today.
Registration is manifest-level: confirm no harness gate demands parsed source rows/cells for a microdata package, or add the explicit carve-out.
First registrations, from Microcosm's manifests: US public-use files (CPS ASEC pooled vintages, CPS basic monthly 2024, ACS PUMS household/person, SCF 2022, SIPP 2023) with bytes; IRS PUF 2015, FRS 2023-24 tabs, SPI donor, BE-SILC 2023 hash-only. Exact list = every *_microdata artifact kind in packages/microcosm-build/src/microcosm/build/*/source_stages.json.
Docs: storage-architecture "Object Key Conventions" example for a microdata release; harness doc section on hash-only entries.
Consumer side: PolicyEngine/microcosm issue (linked below) adds chronicle_artifact references and sha256 to every microdata artifact entry and fails closed on mismatch.
Boundary
Unchanged in substance: no reconciliation, aging, imputation, target activation, or microdata-derived facts. The June 30 removal (#68) of microdata parsers, adapters, and tracked raw storage stands; this adds manifests and two manifest fields only. Licensed bytes never enter a Chronicle store or credential scope.
Related: #143 (ledger→chronicle surface migration; the ledger-raw bucket name is migrating under mechanism 3, registrations should target whichever bucket is current at landing time), microcosm#530 (pin raw PUF), microcosm#665 (UK raw ingest, workstream E), microcosm#204.
Implements the raw-microdata identity ADR (
docs/adr-chronicle-raw-microdata-identity.md, PR opened alongside this issue): Chronicle registers every raw microdata release its consumers build from, stores bytes only for redistributable public-use files, and never parses microdata into rows or facts.Why now
source_stages.json(UK: 19licensed_microdata+ 2private_microdataentries carry sha256; US: 8 of 37public_microdataentries carry sha256, 29 do not; BE: therestricted_microdataentry carries none), the ASEC raw-stage checkpoint metadata, code constants (e.g.weeks_unemployed.py), andbuild_us_multispine_pool.py --*-sha256arguments. None references a Chronicle artifact.Tasks
licence(publisher terms id or URL) andaccess(public|licensed|restricted, closed set) tomanifest.yaml; validate invalidate-package/ artifact inventory. Today 1 of 159 manifests carries alicensefield.fetch-artifact/publish-raw(or a siblingregister-artifact) must accept alicensed/restrictedentry with checksum, size, vintage, access route and refuse bytes;publicentries archive bytes underraw/{source_id}/{package_id}/{year}/{sha256}/{filename}as today.*_microdataartifact kind inpackages/microcosm-build/src/microcosm/build/*/source_stages.json.chronicle_artifactreferences and sha256 to every microdata artifact entry and fails closed on mismatch.Boundary
Unchanged in substance: no reconciliation, aging, imputation, target activation, or microdata-derived facts. The June 30 removal (#68) of microdata parsers, adapters, and tracked raw storage stands; this adds manifests and two manifest fields only. Licensed bytes never enter a Chronicle store or credential scope.
Related: #143 (ledger→chronicle surface migration; the
ledger-rawbucket name is migrating under mechanism 3, registrations should target whichever bucket is current at landing time), microcosm#530 (pin raw PUF), microcosm#665 (UK raw ingest, workstream E), microcosm#204.