Skip to content

Register raw microdata releases as source artifacts: licence/access manifest fields, hash-only registration, first registrations #221

Description

@MaxGhenis

Implements the raw-microdata identity ADR (docs/adr-chronicle-raw-microdata-identity.md, PR opened alongside this issue): Chronicle registers every raw microdata release its consumers build from, stores bytes only for redistributable public-use files, and never parses microdata into rows or facts.

Why now

  • Microcosm's raw inputs are pinned in four different places with no shared registry, licence record, or witnessed fetch time: per-country source_stages.json (UK: 19 licensed_microdata + 2 private_microdata entries carry sha256; US: 8 of 37 public_microdata entries carry sha256, 29 do not; BE: the restricted_microdata entry carries none), the ASEC raw-stage checkpoint metadata, code constants (e.g. weeks_unemployed.py), and build_us_multispine_pool.py --*-sha256 arguments. None references a Chronicle artifact.
  • Publishers revise and withdraw microdata (IRS PUF withdrawn 2026; Census reissues under the same vintage label). A registered hash with a witnessed fetch time is the only durable statement that a release existed with those bytes.
  • The node-graph rebuild (microcosm#836) declares identity per node; the roots of the graph should resolve to one witnessed registration.

Tasks

  • Manifest schema: add licence (publisher terms id or URL) and access (public | licensed | restricted, closed set) to manifest.yaml; validate in validate-package / artifact inventory. Today 1 of 159 manifests carries a license field.
  • Hash-only registration: fetch-artifact / publish-raw (or a sibling register-artifact) must accept a licensed/restricted entry with checksum, size, vintage, access route and refuse bytes; public entries archive bytes under raw/{source_id}/{package_id}/{year}/{sha256}/{filename} as today.
  • Registration is manifest-level: confirm no harness gate demands parsed source rows/cells for a microdata package, or add the explicit carve-out.
  • First registrations, from Microcosm's manifests: US public-use files (CPS ASEC pooled vintages, CPS basic monthly 2024, ACS PUMS household/person, SCF 2022, SIPP 2023) with bytes; IRS PUF 2015, FRS 2023-24 tabs, SPI donor, BE-SILC 2023 hash-only. Exact list = every *_microdata artifact kind in packages/microcosm-build/src/microcosm/build/*/source_stages.json.
  • Docs: storage-architecture "Object Key Conventions" example for a microdata release; harness doc section on hash-only entries.
  • Consumer side: PolicyEngine/microcosm issue (linked below) adds chronicle_artifact references and sha256 to every microdata artifact entry and fails closed on mismatch.

Boundary

Unchanged in substance: no reconciliation, aging, imputation, target activation, or microdata-derived facts. The June 30 removal (#68) of microdata parsers, adapters, and tracked raw storage stands; this adds manifests and two manifest fields only. Licensed bytes never enter a Chronicle store or credential scope.

Related: #143 (ledger→chronicle surface migration; the ledger-raw bucket name is migrating under mechanism 3, registrations should target whichever bucket is current at landing time), microcosm#530 (pin raw PUF), microcosm#665 (UK raw ingest, workstream E), microcosm#204.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions