Register raw microdata releases as source artifacts: licence/access fields, hash-only registration, first UK registrations (#221)

[MaxGhenis]

Summary

Implements chronicle#221: Chronicle registers every raw microdata release its
consumers build from, and stores the content of none of them.

Depends on #222 (the raw-microdata-identity ADR and the governance,
architecture, AGENTS.md, and non-goal amendments). This PR deliberately makes
none of those edits, so the two do not conflict — but until #222 merges, the
governance doc still forbids what this branch does, and the ADR file that four
files cite does not exist on main. Merge #222 first.

Contract is shared verbatim with microcosm#848.

Schema and validation

Manifest file entries gain two fields:

• licence — the publisher's terms, as an identifier or URL.
• access — a closed set: public | licensed | restricted.

public is inferred where absent; fetch-artifact now writes the class
explicitly onto every entry it touches. Both fields are required on a
kind: microdata_release manifest, the new manifest-level carve-out.

Only public bytes enter a Chronicle store, so the R2 key
raw/{source_id}/{package_id}/{year}/{sha256}/{filename} exists only for that
class. A licensed or restricted release is registered hash-only: the
manifest carries the checksum, size, vintage, licence, access route, and
verification timestamp, and no bytes are fetched, written, or uploaded.

A registration is identified by {source_id, package_id, year, sha256, filename} — the tuple consumers reference under chronicle_artifact.

New command

chronicle register-artifact registers a licensed or restricted release from
CLI arguments alone. It refuses --access public, a missing or malformed
checksum, a missing licence, vintage, access route, or verification timestamp,
bytes sitting beside the manifest, a publisher_table manifest, a foreign
source_id, and an unrequested reissue. Public releases keep the existing
fetch-artifact path, which gains --access, --licence, and --kind.

First registrations

15 hash-only registrations, every value read verbatim from Microcosm's reviewed
pins and cross-checked field by field:

• DWP Family Resources Survey 2023-24 — 14 tabs, access: licensed, UK Data
  Service End User Licence, study SN 9367, DOI 10.5255/UKDA-SN-9367-2.
  Microcosm carries 19 licensed_microdata references across five stages;
  they resolve to 14 distinct files (identical SHA-256 where they repeat).
• HMRC Survey of Personal Incomes Public Use Tape 2022-23 — put2223uk.tab,
  access: restricted, study SN 9422, DOI 10.5255/UKDA-SN-9422-1.

chronicle inventory-artifacts --root db/data is valid with 195 artifacts: 180
with R2 links, 15 hash-only with no local bytes.

BE-SILC 2023 is not registered. Microcosm pins it with no sha256, no
size_bytes, and no per-file filename — the locator names four file roles.
No hash is invented; the blocker and its unblock steps are recorded in
docs/data-sources/be-silc-2023-registration-blocker.md.

Tooling

scripts/register_microdata_releases.py resolves a declarative catalogue
against a read-only microcosm checkout. emit writes the hash-only manifests
byte-stably; plan prints the exact fetch-artifact --upload-r2 commands to
run from a networked machine for the public US releases, copying publisher URLs
verbatim and printing TODO_PUBLISHER_URL where Microcosm records none (CPS
basic monthly 2024, SIPP 2023). The catalogue is the only authored content —
every checksum, size, filename, and vintage is read from Microcosm.

Audit fixes

An adversarial pass over the first implementation found the byte boundary was
escapable. All six are fixed here with regression tests:

1. Licensed bytes could reach the package tree and R2. The hash-only guard
   only inspected files[year] for the requested year and only entries that
   survived a kind-gated list expansion, while the write target and R2 key are
   directory-scoped. A release registered under one vintage could be fetched
   under another, and a list under a manifest with an absent or misspelled
   kind was invisible to the guard. Both were reproduced writing licensed
   bytes into the tree. The guard now walks every vintage and expands lists
   unconditionally.
2. A fetch silently deleted an existing registration by replacing the whole
   files[year] value. It now replaces only the entry for its own filename.
3. emit stopped being re-runnable after a reissue — registration raised on
   the first filename match with a different checksum before reaching the exact
   match. Now two passes.
4. --skip-hash-only reported a contract violation as valid, disagreeing
   with inventory-artifacts about the same tree. Hash-only entries are now
   always validated; the flag turns off the refusal, not the check.
5. No way to register a public microdata release as kind: microdata_release, though the ADR puts redistributable public-use files in
   scope. fetch-artifact gains --kind, and a release manifest keeps the list
   shape so several files share one vintage (ACS household + person).
6. Every generated fetch command died at argparse — plan emitted
   --source-url, which fetch-artifact does not accept. Fixed to --url, and
   a test now parses each generated command against the real CLI parser.

Also hardened _year_mapping, which returned a list and let the caller raise an
uncaught TypeError that validate-package does not catch.

No content, anywhere

No microdata row, cell, or fact is added. No source package parses a release:
validate-package fails with microdata_release_not_parseable, and the FRS and
SPI packages hold nothing but manifest.yaml. Bucket names stay configurable
(ledger-raw default, unchanged).

Chronicle Governance

For any source package, consumer contract, schema, or source-data
boundary change:

• Approved Chronicle agent role:

  • ledger-source-ingestor — the db/data/** microdata-release manifests
  • ledger-contract-maintainer — the manifest schema (licence, access,
    kind), its validation, and the registration command

  Note both roles' allowed_paths in .github/chronicle-agents.yml predate
  this work and cover none of chronicle/registration.py,
  chronicle/artifacts.py, chronicle/harness.py, chronicle/cli.py,
  db/data/**, scripts/**, docs/**, or README.md. Flagging rather than
  quietly widening the manifest; reviewers may want a follow-up that extends
  the globs.

• Deterministic checks run:

  • uv run pytest -q — full suite
  • uv run ruff check . — passes
  • uv run ruff format --check . — 14 pre-existing failures, all in files this
    branch does not touch (verified unchanged against origin/main); the
    branch's own files pass
  • uv run chronicle inventory-artifacts --root db/data — valid, 195
    artifacts, 15 hash-only
  • scripts/register_microdata_releases.py emit — re-run is byte-stable
  • every registered checksum cross-checked verbatim against Microcosm's pins

• LLM judge verdicts:

  • ledger-source-fidelity: PASS — no fact is added. Every registered
    value (sha256, size, filename, vintage) is transcribed verbatim from the
    consumer's reviewed pin, machine-verified against Microcosm, and each entry
    records hash_source plus a note that Chronicle did not recompute the
    checksum. BE-SILC is refused rather than guessed.
  • ledger-contract: PASS — the change adds two manifest fields and one
    manifest kind, preserves provenance, and moves no Microcosm responsibility
    into Chronicle. Registration is identity only; selection, reconciliation,
    aging, and imputation are untouched.
  • ledger-boundary: PASS — no microdata row, cell, or fact enters
    Chronicle; no source package parses a release; no licensed or restricted
    bytes enter any Chronicle store, and the escapes that previously allowed
    that are closed and regression-tested.

Tests

tests/test_chronicle_microdata_registration.py — 69 tests covering the access
and kind vocabularies, every manifest-entry validation code, each command's
refusals, the six audit regressions, public microdata releases, the generated
fetch plan, and the committed FRS/SPI registrations as identity-only.

Two existing assertions in tests/test_chronicle_artifacts.py gain the new
hash_only_count and hash_only_refused_count report keys.

🤖 Generated with Claude Code

[MaxGhenis]

Review (Fable, main): approving, with the Sol gate running. Cross-checked every registered hash against Microcosm's uk/source_stages.json: 15 of 15 match, none extra, BE-SILC correctly refused rather than guessed. The refusal surface is the right shape: bytes for licensed/restricted can enter no store by any command, and the six escapes the lane found in its own first cut are the kind of thing this boundary exists for.

Sequencing: #222 merges first (its gate is running; merge-on-agreement), then this. No file overlap between the two.

Notes, none blocking:

• private_microdata → restricted is a safe over-classification (SPI PUT is a UKDS licensed study; both classes are hash-only, so nothing changes byte-wise). Fine as is; say so in the catalogue comment if it is not already.
• dataset: dwp_dwp-frs-2023-24 is a mechanical id; harmless, could read dwp_frs_2023_24 in a follow-up.
• The agent-role allowed_paths gap the body flags is real and predates this PR; follow-up to widen the globs rather than silently editing the roles file here.
• The Microcosm side (microcosm#848, in flight) must reference exactly {source_id: dwp, package_id: dwp-frs-2023-24} and the SPI tuple committed here; I will align that PR to these ids.
• The public-release fetch commands from plan will be run from a networked session after merge (two need publisher URLs Microcosm does not record: CPS basic monthly 2024, SIPP 2023).

[MaxGhenis]

Gate round 1: changes requested. The 15 registrations match the consumer pins and add no bytes or fact surfaces, but Sol found real escapes around the boundary, all of which stand: filename aliases (./adult.tab vs adult.tab) bypass the hash-only guard; the source-package byte reader checks manifest kind but never the selected entry's access, so a mapping-shaped manifest with access: licensed can still be read, cached, and parsed; register-artifact can reclassify a public identity that already has an R2 object as licensed/restricted, leaving the object in R2 while inventory looks clean; kind conflicts and missing microdata access are not rejected before I/O; with --filename omitted the artifact is downloaded before the inferred filename is refused; _upsert_manifest is destructive and shape-unsafe; year keys are looked up as int only while readers accept '2023'; the generated public-release commands drop publisher/vintage and carry the expected sha256 only as a comment (fetch-artifact needs --expected-sha256 and the metadata params); the catalogue tests hardcode a home-directory checkout and skip on failure.

Two additions from the ADR review (#222 round 2) fold into the same fix: byte custody requires an affirmative, evidence-backed redistribution signal, not merely access: public (an allowlist of redistributable licence terms enforced in code, everything else is licensed); and the prohibition vocabulary is canonicalized to records, rows, columns, row values, cells, and facts.

Fix lane queues behind #226's round-2 fixes because both rewrite chronicle/artifacts.py; it rebases onto that branch first.

[MaxGhenis]

Cross-PR convention ruling (chronicle#227 ↔ microcosm#853), so both sides land on one shape:

1. One Chronicle package per publisher release, files listed under the vintage. Chronicle's kind: microdata_release manifests hold a list under files[year] precisely so several distributed files share one registration (14 FRS tabs = one UKDS study). The consumer reference {source_id, package_id, year, sha256, filename, access} already disambiguates files inside a package by filename + sha256. So the ids are dwp / dwp-frs-2024-25 (year 2024, 14 tabs) and hmrc / hmrc-spi-public-use-tape-2022-23 (year 2022, put2223uk.tab), not one package per tab.
2. Vintage: FRS 2024-25 (UKDS SN 9563). Microcosm main moved to the 2024-25 tabs today (#832); chronicle#227 registered the 2023-24 hashes from a stale checkout, so only 1 of its 15 hashes still matches main. The Chronicle side regenerates from microcosm origin/main (emit against a fresh checkout) and adds the other licensed/private roots main pins (WAS round 8, LCFS 2023-24 and 2018-20, ETB 1977-2024, SPI PUT): 19 hashes today.
3. Access class derived from the Microcosm kind as #853 does (licensed_microdata → licensed; private_microdata/restricted_microdata → restricted; public_microdata → public), with byte custody additionally requiring the licence allowlist (ADR round 2). versioned_derived_microdata is never registered.

Chronicle's #227 fix round and a small #853 follow-up apply this.