diff --git a/PROGRESS.md b/PROGRESS.md index 24773009..79bafcd9 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -1,51 +1,190 @@ -# Lane C5 progress +# Operational rename, slice 1 (chronicle#143, mechanism 3) + +Lane C5's handoff notes previously lived here; its durable record is +`LANE_C5_REPORT.md`. This file now tracks the active lane on this branch. ## State -- Branch: `be-2025-vintages` from `origin/main` at `5c15bfd`. -- Worktree inputs are staged under `.lane-raw/` and must remain uncommitted. -- Lane C5 is complete, validated, independently reviewed, and ready for handoff. -- The requested staged C2 report is absent, but root `LANE_C2_REPORT.md` is byte-identical - to the sibling lane's staged copy (SHA-256 `4590e0dc...50f06e7`) and is the pattern used. +- Branch: `ops-rename-slice1`, cut from `origin/main` at `ff3efd3`. +- Scope: env names, R2 bucket configurability, `ledger.db` -> `chronicle.db`, + and the docs for all three. Code and docs only; no infrastructure changes. +- Out of scope and deliberately untouched: the `ledger` console-script alias, + the Supabase `"ledger"` schema and mirror table names, governance role ids and + concept authorities, hash domains and schema ids, anything under `releases/`. +- This PR does not touch the source-data boundary. No package spec, parser, + selector, manifest, or fact value changes. ## Done -- Read the repository Chronicle boundary rules in `AGENTS.md`. -- Read `.lane-raw/SOURCES.md` and confirmed all five named publisher artifacts are present. -- Confirmed the worktree is otherwise clean apart from `.lane-raw/` and the shared `.venv` link. -- Verified all five staged artifact SHA-256 pins exactly. -- Mapped FPB workbook cells: 990 facts across T01/T06/T07/T11/T17/T24, with - 2022–2025 observations and 2026–2031 `source_projection` facts. -- Confirmed PDF boundary evidence: printed page 19 calls 2026 the first projection year; - annex table units appear on printed pages 45, 48, 49, 53, 58, and 65. -- Chosen Eurostat layout: two vintage-specific source-package aliases share new manifest - entries, preserving the prior package YAMLs, raw bytes, and fact outputs unchanged. -- Reproduced the Statbel curator logic: 18 NUTS1 × sex × age-band cells totaling 11,825,551. -- Added the hash-pinned FPB workbook and publication PDF plus the - `fpb-economic-outlook-2026-2031-june-2026` package alias. -- Built 990 line-specific publisher facts (99 per year): 396 observations for - 2022–2025 and 594 `source_projection` facts for 2026–2031. -- Passed FPB `validate-package` and `build-suite`: 990 facts, full cell lineage, - zero acceptance errors, and pinned 2025 cells 320578 / 77771 / 5602 million euro. -- Re-ran the Statbel 2026 curator logic on the 2025 ZIP and added the hash-pinned - raw capture plus its deterministic 18-row curated CSV. -- Passed Statbel 2025 `validate-package` and `build-suite`: 18 facts totaling - 11,825,551, 66 constraints, full lineage, and zero acceptance errors. -- Added the Eurostat `gov_10a_taxag` 2025 and `spr_exp_func` 2024 manifest - entries plus vintage-specific package aliases, without modifying either - prior artifact or prior package specification. -- Passed both new Eurostat package validations and suite builds: 12 tax facts - and 9 ESSPROS facts, full lineage, and zero acceptance errors. -- Extended Belgium and Eurostat regressions for FPB table counts/cells and - assertion boundary, vintage non-overlap, prior-output digests, Statbel pins, - and the declared 0.25% Statbel/FPB population comparison tolerance. -- Passed 43 focused tests and the full merged-bundle regression: 157,177 facts, - 148 packages, zero aggregate-key duplicates, and expected goldens throughout. -- Recorded pins, counts, boundary evidence, curator commands, validation tails, - and consumer fact families in `LANE_C5_REPORT.md`. -- Passed independent `ledger-source-fidelity` and `ledger-boundary` reviews with - no required corrections. +- Read `AGENTS.md`, `docs/storage-architecture.md`, + `docs/agent-source-package-harness.md`, and the mechanism-3 migration spec in + the first comment of PolicyEngine/chronicle#143. +- Enumerated every ledger-named env read in tracked Python: the four real + variables (`LEDGER_SOURCE_ARTIFACT_CACHE_DIR`, `LEDGER_SOURCE_ARTIFACT_FETCH`, + `LEDGER_PE_US_DATA_ROOT`, `LEDGER_PE_UK_DATA_ROOT`) plus + `POLICYENGINE_LEDGER_SCHEMA`. `LEDGER_MIRROR_TABLES`, + `LEDGER_MIRROR_PRIMARY_KEYS`, and `LEDGER_DB_SCHEMA_VERSION` are module + constants, not env reads, and name out-of-scope surfaces. +- Added `chronicle/env.py`: one shared `env_value`/`env_flag`/`env_names` + helper reading `CHRONICLE_` first, then `POLICYENGINE_LEDGER_` and + `LEDGER_` with a once-per-process `ChronicleEnvDeprecationWarning` naming + the preferred variable. +- Replaced all three ad-hoc helpers (`db/supabase_client._env`, + `chronicle/source_package._env_value`/`_truthy_env`, + `db/pe_source_inventory._env_value`) with the shared helper. +- Made the R2 bucket names configurable via `CHRONICLE_R2_RAW_BUCKET` and + `CHRONICLE_R2_DERIVED_BUCKET`, plumbed through fetch-artifact, publish-raw, + publish-derived and bootstrap-r2. Defaults unchanged at `ledger-raw` and + `ledger-derived`. Both manifest write paths now preserve a recorded + `storage.r2` block instead of restating it under a renamed bucket. +- Emitted `chronicle.db` for new suite outputs, with `ledger.db` still accepted + on read and on derived-artifact kind inference. +- Added `tests/test_chronicle_env.py` plus artifact tests: 75 hermetic tests + covering the lookup ladder, precedence, the once-per-process warning, and every + real call site. +- Swept the docs. `docs/storage-architecture.md` gained an "Environment Variable + Rename Window" section (the old text stated the fallback direction backwards) + and a "Bucket Cutover" section; `docs/agent-source-package-harness.md` and + `README.md` follow. Verified 186 distinct `ledger-raw` objects across 154 + tracked manifest files, every key content-addressed by sha256. + +## Review fixes (gate round 1) + +The Fable+Sol gate requested changes; both findings are applied on this branch. + +- **[high] `fetch-artifact` could attach a recorded R2 URI to new bytes.** The + preserve rule keyed on the bucket, so a repeated fetch that did not re-upload + into the same bucket kept the recorded `storage.r2` block while rewriting the + entry's `sha256`/`size_bytes`. Reproduced against this branch's parent by + serving two different bodies from one URL: the entry ends up declaring the + fetched bytes' `sha256` under a key addressed by the superseded bytes' one, + both when the fetch only registers the bytes and when the bucket default has + moved. + The rule now keys on identity — the recorded key's `{sha256}/{filename}` tail + against the fetched bytes. Identical preserves the block exactly; different + raises `SourceArtifactRevisionError` before the cached artifact or its + manifest entry is touched, naming recorded and fetched `sha256`/`size_bytes` + and the ADR rule that same vintage plus new bytes is a new release revision. + `--record-revision` opts in: the new bytes get their own content-addressed key + under the configured bucket, never the old key, and the superseded block moves + to `storage.previous_r2`. `publish-raw` applies the same check before treating + a recorded block as history (`recorded_r2_identity_mismatch`, nothing + uploaded). +- **[low] Env isolation was scoped to one module.** The autouse fixture moved to + `tests/conftest.py` and now clears all three prefixes for every test. + `db.supabase_client` resolves `LEDGER_SCHEMA` at import — during collection, + before any fixture — so `tests/test_chronicle_namespace.py` re-imports it + under the cleared environment instead of asserting the constant it bound at + collection time. + +`storage.previous_r2` is a sibling key, chosen because every reader +(`inventory-artifacts`, `publish-raw`, `source_package._artifact_content`, the +suite's raw-R2-link acceptance check) reads `storage.r2` alone, and +`publish-raw` already spreads the rest of the `storage` block when it writes +back, so a revision survives publication untouched. All 180 tracked manifest +entries that carry a `storage.r2` block are content-addressed and agree with +their declared `sha256` and `filename`, so the identity check never fires on +tracked data. + +## Review fixes (gate round 2) + +The second Fable+Sol gate requested changes again. Seven findings, each fixed +with a regression test on this branch. Plan, in dependency order: + +1. **[high] `CHRONICLE_SCHEMA` does not reach the Supabase mirror writer.** + `chronicle/harness.py` and `chronicle/mirror.py` default the schema to the + literal `"ledger"`; only `db.supabase_client` reads the renamed variable. + Resolve through the shared helper whenever no explicit `--schema` is given. +2. **[high] The derived-fact boundary check is not rename-safe.** + `chronicle/consumer_contract.py` matches the `.ledger_derived` suffix only. +3. **[high] `fetch-artifact` cannot address a package's non-default manifest.** + Seven tracked packages keep a `manifest_*_source_package.yaml`; three + directories keep two. A fetch into one of them writes a third manifest and + never sees the recorded block. +4. **[high] Revision protection vanishes when the entry has no `storage.r2`.** +5. **[medium] Recorded-R2 locator fields must be cross-checked**, not read as + key-or-URI, before a block is preserved or published. +6. **[medium] `_read_manifest` must reject a malformed document**, not treat a + non-mapping YAML payload as an absent manifest. +7. **[low] Schema resolution must be lazy** so no legacy variable is read at + collection, before the autouse isolation fixture runs. + +## State (round 2) + +- Read both gate rounds on PolicyEngine/chronicle#226 and the code each finding + names. +- Scanned all 154 tracked manifest files (187 `files` entries, every one + carrying `storage.r2`): every recorded block supplies provider, bucket, key + and uri; every key is content-addressed; every declared `sha256`/`filename` + agrees with its key tail; no `uri` contradicts its `key`. Strict locator + validation therefore refuses nothing that is tracked today. +- All seven findings are applied, each with a regression test, and each + reproduced against this branch's previous head (`34d1d0f`) first. + +### What each fix does + +1. `chronicle/env.py` gains `default_chronicle_schema()`: one home for the + `CHRONICLE_SCHEMA` -> `POLICYENGINE_LEDGER_SCHEMA` -> `LEDGER_SCHEMA` -> + `"ledger"` ladder. `load_supabase_mirror`, its harness wrapper and the + `--schema` CLI default all resolve through it when no schema is supplied; + an explicit `--schema` still wins. Defaults unchanged. +2. `chronicle/consumer_contract.py` matches the whole final dot-segment of a + `source_record_id` against both `ledger_derived` and `chronicle_derived`. +3. `fetch-artifact --manifest ` selects which of a package's + manifests the entry belongs to (default `manifest.yaml`); the name must be + a filename inside `--out-dir`. +4. Revision protection now compares against the entry's recorded identity -- + the recorded key's `{sha256}/{filename}` once published, the declared + `sha256` before that -- so a registered-but-unpublished entry, or one whose + upload failed, is protected exactly like a published one. +5. `_validated_recorded_r2` cross-checks every supplied locator field against + every other and against the content-addressed key shape. A contradiction is + `RecordedR2LocatorError` at fetch time and `recorded_r2_locator_invalid` at + publish time, never a preserved block. +6. `_read_manifest` refuses a non-mapping or unparseable document + (`MalformedManifestError`) before the publisher is read at all; + `inventory-artifacts` and `publish-raw` report it instead of crashing. +7. `db.supabase_client` resolves both schemas per call rather than at import, + and `tests/conftest.py` strips the rename window in `pytest_configure`, so + no module can read or warn from an operator's shell during collection. + +All four refusals share a `SourceArtifactManifestError` base, so the +`fetch-artifact` CLI reports every one as exit 1 with nothing written. + +### Reproduced against `34d1d0f` (the round-1 head) + +Running the same operations against a checkout of the previous head: + +1. `load_supabase_mirror` default `schema='ledger'`; with + `CHRONICLE_SCHEMA=chronicle_probe` the load still reports `schema='ledger'`. +2. `'.chronicle_derived'.endswith('.ledger_derived')` is False: the boundary + never fired for the chronicle spelling. +3. `fetch_source_artifact()` rejects `manifest_filename` as an unexpected + keyword; a fetch into `ira_contributions/` writes `manifest.yaml`. +4. A fetch of different bytes over a registered (unpublished) entry was + accepted silently: the entry's `sha256` was rewritten with no refusal. +5. A block whose `key` and `uri` named different objects was preserved + verbatim, key sha `c63744a4...` beside uri sha `1e9b3fdb...`. +6. A list-valued `manifest.yaml` was overwritten by the fetch. +7. Importing `db.supabase_client` under `LEDGER_SCHEMA=zzz` bound + `LEDGER_SCHEMA='zzz'` and emitted a `FutureWarning` at collection. + +## Verification + +- `uv run pytest -q`: green. +- `uv run ruff check .`: clean. +- `uv run ruff format --check .`: clean for every file this branch touches. 13 + files are unformatted on `main` already and are byte-identical here; CI runs + `ruff check` only, so they are pre-existing and out of scope. +- CI's db CLI gate (`chronicle init` / `load all` / `stats`): passes. +- `CHRONICLE_R2_RAW_BUCKET=zzz CHRONICLE_SCHEMA=zzz uv run pytest -q`: green. + Before the shared fixture it failed five tests — four bucket-default + assertions in `tests/test_chronicle_artifacts.py` and the collection-time + schema constant in `tests/test_chronicle_namespace.py`. ## Next -- None; ready for handoff. No push was performed. +- Push and open the PR against `main`. Do not merge. +- Follow-up PR, after Max creates and backfills the new buckets: flip + `DEFAULT_R2_RAW_BUCKET` / `DEFAULT_R2_DERIVED_BUCKET` to `chronicle-raw` / + `chronicle-derived`. diff --git a/README.md b/README.md index f75fe697..237bee9e 100644 --- a/README.md +++ b/README.md @@ -65,10 +65,12 @@ contract that aligns it to another period (see | Microcosm Target Contracts | Selection, measurement bindings, and active subset | Period alignment, support-aware activation, solver inputs, diagnostics | The storage split is documented in -[`docs/storage-architecture.md`](docs/storage-architecture.md): `ledger-raw` -stores immutable source bytes, `ledger-derived` stores reproducible build +[`docs/storage-architecture.md`](docs/storage-architecture.md): a raw R2 archive +stores immutable source bytes, a derived R2 archive stores reproducible build artifacts, and Supabase/Postgres hosts the queryable relational Chronicle registry -mirrored from accepted builds. +mirrored from accepted builds. The bucket names are configuration +(`$CHRONICLE_R2_RAW_BUCKET` and `$CHRONICLE_R2_DERIVED_BUCKET`), still defaulting +to the ledger-era `ledger-raw` and `ledger-derived`. ## Repository Model @@ -247,7 +249,7 @@ This writes: source_regions.jsonl facts.jsonl consumer_facts.jsonl - ledger.db + chronicle.db reports/ source_rows.json source_cells.json @@ -349,11 +351,14 @@ needed, even when your Cloudflare user belongs to several accounts: # One-time per machine (opens a browser consent page): bunx wrangler login -# One-time per account (already done for the PolicyEngine account): -uv run chronicle bootstrap-r2 --raw-bucket ledger-raw --derived-bucket ledger-derived +# One-time per account (already done for the PolicyEngine account). The bucket +# flags default to $CHRONICLE_R2_RAW_BUCKET / $CHRONICLE_R2_DERIVED_BUCKET: +uv run chronicle bootstrap-r2 # Fetch/register a source artifact, write db/data/.../manifest.yaml, and upload -# the exact bytes to R2 when Wrangler is authenticated: +# the exact bytes to R2 when Wrangler is authenticated. Pass --manifest when the +# package directory keeps more than one manifest (ira_contributions keeps a +# traditional and a Roth one): uv run chronicle fetch-artifact \ --url https://www.irs.gov/pub/irs-soi/23in12ms.xls \ --source-id irs_soi \ @@ -364,12 +369,47 @@ uv run chronicle fetch-artifact \ --table "Publication 1304 Table 1.2" \ --upload-r2 +# Re-fetching is safe: identical bytes keep the recorded storage.r2 block, and +# bytes that disagree with what the entry identifies -- its declared sha256, or +# its recorded content-addressed key once published -- are refused. When a +# publisher has re-published +# under the same URL and vintage, register the revision explicitly — the new +# bytes get their own content-addressed key and the superseded object is kept +# in storage.previous_r2: +uv run chronicle fetch-artifact ... --record-revision + # Audit local manifests and checksums: uv run chronicle inventory-artifacts --root db/data -# Upload all existing manifest-declared local artifacts to ledger-raw and write -# storage.r2 metadata back into the manifests: +# Upload all existing manifest-declared local artifacts to the raw archive and +# write storage.r2 metadata back into the manifests: uv run chronicle publish-raw --root db/data + +# Register a licensed or restricted release by identity alone. No bytes are +# fetched, stored, or uploaded, and no R2 key is recorded. The checksum is the +# consumer's: name the attester and where the pin was read from: +uv run chronicle register-artifact \ + --source-id dwp \ + --package-id dwp-frs-2023-24 \ + --year 2023 \ + --out-dir db/data/dwp/frs_2023_24 \ + --filename adult.tab \ + --sha256 e09f9647d03585c81a528636028b2ed495f8f1fbcf64c5e7b4fe521b67367e06 \ + --size-bytes 35323384 \ + --vintage 2023_24 \ + --licence "UK Data Service End User Licence" \ + --access licensed \ + --doi 10.5255/UKDA-SN-9367-2 \ + --hash-source consumer_pin \ + --attested-by PolicyEngine/microcosm \ + --pinned-from-repository PolicyEngine/microcosm \ + --pinned-from-path packages/microcosm-build/src/microcosm/build/uk/source_stages.json \ + --pinned-from-commit 2fb2e2f8a99c37725bd6e7a15ff4c2595c912b77 + +# A public microdata release is archived only against a reviewed checksum and +# licence evidence binding the file to an allowlisted term; its bytes are +# staged outside the repository and uploaded from there. See +# docs/agent-source-package-harness.md, "Hash-Only Registrations". ``` To coordinate broad PE source migration without jumping straight to semantic @@ -406,10 +446,10 @@ To prepare the deterministic SQLite artifact for a hosted Supabase/Postgres mirror, export each relational table to JSONL plus a manifest: ```bash -uv run chronicle export-db-tables --db /tmp/chronicle-suite/ledger.db --out /tmp/chronicle-mirror --replace +uv run chronicle export-db-tables --db /tmp/chronicle-suite/chronicle.db --out /tmp/chronicle-mirror --replace ``` -To publish the deterministic build outputs to the `ledger-derived` R2 bucket: +To publish the deterministic build outputs to the derived R2 archive: ```bash uv run chronicle publish-derived \ @@ -437,6 +477,14 @@ uv run chronicle load-supabase-mirror \ Use `--dry-run` first to validate JSONL row counts and file coverage without writing to Supabase. +Chronicle settings are read chronicle-first: `CHRONICLE_X` wins, and the +ledger-era `POLICYENGINE_LEDGER_X` and `LEDGER_X` spellings still work behind a +one-time deprecation warning naming the variable to move to. +[`docs/storage-architecture.md`](docs/storage-architecture.md#environment-variable-rename-window) +lists every variable in that window, and +[Bucket Cutover](docs/storage-architecture.md#bucket-cutover) covers the R2 +bucket rename. + Chronicle facts keep source concepts and canonical concepts separately. For example, the SOI Table 1.1 adjusted gross income column is preserved as `irs_soi.adjusted_gross_income`, while the canonical concept is diff --git a/chronicle/__init__.py b/chronicle/__init__.py index 04d50943..e3ddfb06 100644 --- a/chronicle/__init__.py +++ b/chronicle/__init__.py @@ -13,6 +13,7 @@ "consumer_contract", "core", "database", + "env", "facts", "harness", "jurisdictions", diff --git a/chronicle/artifacts.py b/chronicle/artifacts.py index fa56d620..27fd5d4a 100644 --- a/chronicle/artifacts.py +++ b/chronicle/artifacts.py @@ -5,10 +5,12 @@ import hashlib import json import mimetypes +from collections.abc import Mapping from dataclasses import dataclass from datetime import UTC, datetime from pathlib import Path import posixpath +import re import shlex import sqlite3 import subprocess @@ -18,14 +20,258 @@ import httpx import yaml +from chronicle.database import ( + CHRONICLE_DB_FILENAME, + CHRONICLE_DB_FILENAMES, + LEGACY_CHRONICLE_DB_FILENAME, +) +from chronicle.env import env_value from chronicle.epoch import EMIT_EPOCH, Epoch, canonicalize_key, hash_domain - - +from chronicle.licences import ( + REDISTRIBUTABLE_LICENCES, + is_redistributable_licence, + licence_evidence_errors, +) +from chronicle.registration import ( + ACCESS_CLASSES, + ACCESS_PUBLIC, + CHRONICLE_ATTESTER, + HASH_SOURCE_CHRONICLE_FETCH, + MANIFEST_KINDS, + MICRODATA_RELEASE_KIND, + AmbiguousVintageKeyError, + ListSpecRejected, + ManifestAccessError, + ManifestKindError, + bare_filename, + filename_key, + has_file_entries, + is_bare_filename, + is_hash_only, + iter_file_specs, + iter_manifest_entries, + manifest_kind as normalize_manifest_kind, + normalize_access, + recorded_r2, + resolve_vintage_key, + safe_entry_access, + safe_manifest_kind, + validate_file_entry, + validate_manifest_files, +) + + +R2_RAW_BUCKET_ENV = "CHRONICLE_R2_RAW_BUCKET" +R2_DERIVED_BUCKET_ENV = "CHRONICLE_R2_DERIVED_BUCKET" + +# The bucket defaults stay at their ledger-era names. Archived witness records +# pin raw R2 URLs by hash, so ledger-raw and ledger-derived are preserved +# read-only forever and no recorded manifest URI is ever rewritten. The env +# vars exist so the cutover in docs/storage-architecture.md can be rehearsed, +# and so flipping to chronicle-raw/chronicle-derived is a default change rather +# than a code change (PolicyEngine/chronicle#143, mechanism 3). DEFAULT_R2_RAW_BUCKET = "ledger-raw" DEFAULT_R2_DERIVED_BUCKET = "ledger-derived" DEFAULT_R2_PREFIX = "raw" DEFAULT_R2_DERIVED_PREFIX = "derived" +#: ``RawArtifactPublishEntry.skipped`` prefix for a licensed or restricted +#: registration: nothing to upload, because no Chronicle store holds its bytes. +HASH_ONLY_SKIP_PREFIX = "hash_only_access:" + +# Most packages keep one manifest.yaml. Publisher directories that feed several +# source packages keep one manifest each -- db/data/irs_soi/ira_contributions +# holds manifest_traditional_source_package.yaml beside the Roth one -- so the +# name is an input, not a constant, wherever a caller addresses a package. +DEFAULT_MANIFEST_FILENAME = "manifest.yaml" + +#: Wrangler invocation used for R2 uploads unless a caller overrides it. +DEFAULT_WRANGLER_COMMAND = "npx wrangler" + +MICRODATA_STAGING_DIR_ENV = "CHRONICLE_MICRODATA_STAGING_DIR" +#: Where a public microdata release's bytes are staged before upload. Outside +#: the repository by construction: public microdata is archived in R2, never +#: committed beside its manifest (docs/adr-chronicle-raw-microdata-identity.md). +DEFAULT_MICRODATA_STAGING_DIR = ( + Path.home() / ".cache" / "policyengine-chronicle" / "microdata-staging" +) + + +def default_microdata_staging_dir() -> Path: + """Resolve the staging root: ``$CHRONICLE_MICRODATA_STAGING_DIR`` or default.""" + return Path( + env_value(MICRODATA_STAGING_DIR_ENV, default=DEFAULT_MICRODATA_STAGING_DIR) + ) + + +def microdata_staging_path( + *, + staging_dir: str | Path | None, + source_id: str, + package_id: str, + year: Any, + sha256: str, + filename: str, +) -> Path: + """Return the transient, content-addressed staging path for release bytes. + + Mirrors the raw R2 key shape so a staged file is addressed by the same + identity as the object it becomes. + """ + root = ( + Path(staging_dir) + if staging_dir is not None + else (default_microdata_staging_dir()) + ) + return root / source_id / package_id / str(year) / sha256 / Path(filename).name + + +def _manifest_path(output: Path, manifest_filename: str) -> Path: + """Return the named manifest inside ``output``. + + The name is a filename, not a path: it selects among the manifests a + package directory keeps, and must not reach outside it. + """ + name = manifest_filename.strip() + if not name or name in (".", "..") or name != Path(name).name: + raise ManifestNameError( + "Manifest must name a file inside the package directory, not " + f"{manifest_filename!r}." + ) + return output / name + + +def _sibling_manifests(output: Path) -> list[str]: + """Return the ``manifest_*.yaml`` files a package directory keeps.""" + if not output.is_dir(): + return [] + return sorted( + path.name + for pattern in ("manifest_*.yaml", "manifest_*.yml") + for path in output.glob(pattern) + if path.is_file() + ) + + +def _refuse_a_stray_default_manifest(output: Path, manifest_path: Path) -> None: + """Refuse to create ``manifest.yaml`` beside a package's named manifests. + + A publisher directory that feeds several source packages keeps one + ``manifest_.yaml`` per package and no ``manifest.yaml``. A fetch + that omits ``--manifest`` there would create a third manifest none of the + packages read, and would bypass the revision guard of the one it should + have addressed (PolicyEngine/chronicle#225). + """ + if manifest_path.name != DEFAULT_MANIFEST_FILENAME or manifest_path.exists(): + return + siblings = _sibling_manifests(output) + if not siblings: + return + raise AmbiguousManifestError( + f"{output} keeps {', '.join(siblings)} and no {DEFAULT_MANIFEST_FILENAME}; " + "pass --manifest to name the manifest this fetch records into rather " + f"than creating {DEFAULT_MANIFEST_FILENAME} beside them." + ) + + +def _manifest_files(payload: dict[str, Any], manifest_path: Path) -> dict[str, Any]: + """Return a manifest's ``files`` block, refusing one that is not a mapping. + + ``inventory-artifacts`` and ``publish-raw`` report the same document as + ``files must be a mapping``; a fetch must refuse it before reading the + publisher, or the write fails only after the local artifact has been + overwritten and any upload has run. + """ + files = payload.get("files") + if files is None: + # A bare ``files:`` line parses as None: no entries, like an absent + # block. The writer normalizes it to a mapping before recording into it. + return {} + if not isinstance(files, dict): + raise MalformedManifestError( + f"{manifest_path} files must be a mapping; it parses as a " + f"{type(files).__name__}. Chronicle will not overwrite a manifest " + "it cannot read." + ) + return files + + +def default_r2_raw_bucket() -> str: + """Resolve the raw bucket: ``$CHRONICLE_R2_RAW_BUCKET`` or the default.""" + return env_value(R2_RAW_BUCKET_ENV, default=DEFAULT_R2_RAW_BUCKET) + + +def default_r2_derived_bucket() -> str: + """Resolve the derived bucket: ``$CHRONICLE_R2_DERIVED_BUCKET`` or default.""" + return env_value(R2_DERIVED_BUCKET_ENV, default=DEFAULT_R2_DERIVED_BUCKET) + + +class SourceArtifactManifestError(RuntimeError): + """A manifest refuses the write a fetch is about to make. + + The checks that raise these run before the publisher is read, so an + ordinary refusal costs nothing and leaves the package exactly as it was. + They are repeated immediately before the manifest is rewritten, so no + caller can reach a false-provenance write by another route. + """ + + +class SourceArtifactRevisionError(SourceArtifactManifestError): + """Fetched bytes are not the bytes the manifest entry identifies. + + A manifest entry identifies specific bytes: by its declared ``sha256``, and + -- once published -- by a content-addressed R2 key that repeats them. When + a publisher re-publishes under the same URL and vintage, rewriting that + entry would attach its provenance, and any recorded URI, to bytes it never + described. Chronicle refuses instead: same vintage plus new bytes is a new + release revision (docs/adr-chronicle-fact-identity-v2.md), registered with + ``fetch-artifact --record-revision``. + """ + + +class ManifestNameError(SourceArtifactManifestError, ValueError): + """A manifest name is not a bare filename inside the package directory. + + Also a :class:`ValueError` for callers that validated the name that way + before the CLI learned to report it as an ordinary manifest refusal. + """ + + +class AmbiguousManifestError(SourceArtifactManifestError): + """The default manifest name would create a manifest beside the ones a + package already keeps (PolicyEngine/chronicle#225).""" + + +class MalformedManifestError(SourceArtifactManifestError): + """A manifest document, or a block inside one, is not a mapping. + + Reading such a file as an absent manifest would let a fetch replace it with + a single entry, dropping whatever the unreadable document recorded. + """ + + +class RecordedR2LocatorError(SourceArtifactManifestError): + """A recorded ``storage.r2`` block does not locate exactly one object. + + ``provider``, ``bucket``, ``key`` and ``uri`` all describe the same object, + so any that are supplied have to agree, and the key has to carry the + ``{sha256}/{filename}`` tail that says which bytes it holds. A block whose + fields contradict each other has no single answer to "which bytes does this + entry claim R2 holds", and preserving or publishing under it would ship + whichever field the reader happened to consult. + """ + + +class ExpectedArtifactIdentityError(SourceArtifactManifestError): + """Fetched bytes are not the bytes a reviewed pin said to expect. + + Distinct from :class:`SourceArtifactRevisionError`: a revision has an + opt-in (``--record-revision`` supersedes what the manifest records), an + expectation has none. The operator re-reviews the release and changes + ``--expected-sha256``; Chronicle never archives an unreviewed reissue. + """ + + # New UK and New Zealand uploads are namespaced by country. US objects predate # the country segment and deliberately keep their legacy ``raw/{source_id}`` # and ``derived/{source_id}`` shapes. Publisher directories are the stable @@ -75,6 +321,60 @@ def to_dict(self) -> dict[str, str]: } +# A raw key ends in {sha256}/{filename} (see build_r2_key), so the segment +# before the filename is what says which bytes the object holds. +_SHA256_KEY_SEGMENT = re.compile(r"[0-9a-f]{64}") + + +@dataclass(frozen=True) +class RecordedR2Object: + """The R2 object a manifest entry's ``storage.r2`` block claims exists. + + Built only by :func:`_validated_recorded_r2`, so every instance names one + object whose locator fields agree with each other. + """ + + provider: str + bucket: str + key: str + sha256: str + filename: str + + @property + def uri(self) -> str: + """Return the storage URI the recorded fields spell out.""" + return f"{self.provider}://{self.bucket}/{self.key}" + + +@dataclass(frozen=True) +class RecordedIdentity: + """The bytes a manifest entry says its vintage currently holds. + + From the recorded object's content-addressed key once the entry has been + published, and from the entry's own declared ``sha256``/``filename`` before + that. ``r2`` is None in the second case: protection does not wait for an + upload to have happened. + """ + + sha256: str + filename: str + size_bytes: int | None + declared_sha256: str | None + r2: RecordedR2Object | None + + def holds(self, *, sha256: str, filename: str) -> bool: + """Whether this identity is exactly the given bytes under that name. + + The filename participates only when the entry records one: a published + key always carries it, an entry that declares bytes and no name does + not, and inventing a mismatch there would refuse a re-fetch of the very + bytes the entry describes. + """ + if self.sha256 != sha256: + return False + return not self.filename or self.filename == Path(filename).name + + @dataclass(frozen=True) class ArtifactCommandResult: """Result from a storage command.""" @@ -157,10 +457,13 @@ class ArtifactInventoryEntry: source_url: str | None r2: dict[str, Any] | None errors: tuple[str, ...] + access: str = ACCESS_PUBLIC + licence: str | None = None + hash_only: bool = False @property def valid(self) -> bool: - """Whether this artifact is locally available and checksum-valid.""" + """Whether this registration is complete and, if public, available.""" return not self.errors def to_dict(self) -> dict[str, Any]: @@ -172,6 +475,9 @@ def to_dict(self) -> dict[str, Any]: "filename": self.filename, "local_path": self.local_path, "exists": self.exists, + "access": self.access, + "licence": self.licence, + "hash_only": self.hash_only, "sha256_expected": self.sha256_expected, "sha256_actual": self.sha256_actual, "size_bytes": self.size_bytes, @@ -221,16 +527,32 @@ class RawArtifactPublishEntry: r2_location: ArtifactStorageLocation | None upload: ArtifactCommandResult | None errors: tuple[str, ...] = () + skipped: str | None = None + + @property + def uploaded(self) -> bool: + """Whether this run uploaded the artifact.""" + return self.upload is not None and self.upload.ok + + @property + def hash_only_refused(self) -> bool: + """Whether this entry was left alone because its access is hash-only.""" + return self.skipped is not None and self.skipped.startswith( + HASH_ONLY_SKIP_PREFIX + ) @property def valid(self) -> bool: - """Whether this raw artifact uploaded and was registered.""" - return not self.errors and self.upload is not None and self.upload.ok + """Whether this raw artifact is published or correctly not uploaded: + uploaded now, already held by the recorded object in a preserved + bucket, or skipped as a hash-only registration (``skipped``).""" + return not self.errors and (self.skipped is not None or self.uploaded) def to_dict(self) -> dict[str, Any]: """Return a JSON-serializable entry.""" return { "valid": self.valid, + "skipped": self.skipped, "manifest_path": self.manifest_path, "source_id": self.source_id, "package_id": self.package_id, @@ -265,8 +587,14 @@ def counts(self) -> dict[str, int]: return { "manifest_count": len(manifest_paths), "artifact_count": len(self.entries), - "uploaded_count": sum(1 for entry in self.entries if entry.valid), + "uploaded_count": sum(1 for entry in self.entries if entry.uploaded), + "skipped_count": sum( + 1 for entry in self.entries if entry.skipped is not None + ), "failed_count": sum(1 for entry in self.entries if not entry.valid), + "hash_only_refused_count": sum( + 1 for entry in self.entries if entry.hash_only_refused + ), "r2_link_count": sum( 1 for entry in self.entries if entry.r2_location is not None ), @@ -391,32 +719,206 @@ def fetch_source_artifact( source_page: str | None = None, table: str | None = None, filename: str | None = None, + manifest_filename: str = DEFAULT_MANIFEST_FILENAME, + access: str = ACCESS_PUBLIC, + licence: str | None = None, + kind: str | None = None, + publisher: str | None = None, + vintage: str | None = None, + expected_sha256: str | None = None, + expected_size_bytes: int | None = None, + licence_evidence: Mapping[str, Any] | None = None, + staging_dir: str | Path | None = None, upload_r2: bool = False, - r2_bucket: str = DEFAULT_R2_RAW_BUCKET, + record_revision: bool = False, + r2_bucket: str | None = None, r2_prefix: str | None = None, - wrangler_command: str = "npx wrangler", + wrangler_command: str = DEFAULT_WRANGLER_COMMAND, ) -> ArtifactFetchReport: - """Fetch/register a source artifact and optionally upload it to R2.""" + """Fetch/register a source artifact and optionally upload it to R2. + + ``manifest_filename`` names the manifest inside ``output_dir`` the entry + belongs to. Packages that split one publisher directory across several + source packages keep one manifest each, so a fetch that always wrote + ``manifest.yaml`` would write a fresh manifest beside the real ones and + never see the entry it is revising. + + ``record_revision`` opts into registering a publisher revision: the fetched + bytes get their own content-addressed key under the configured bucket and + the superseded object moves to ``storage.previous_r2``. Without it, bytes + that disagree with the entry's recorded identity raise + :class:`SourceArtifactRevisionError` before anything is overwritten. + + ``expected_sha256`` (and ``expected_size_bytes``) pin the bytes a reviewed + identity says the publisher serves: bytes that hash differently raise + :class:`ExpectedArtifactIdentityError` before anything is written or + uploaded, and ``record_revision`` does not override that. + + Only ``public`` artifacts travel this path: it writes bytes and can upload + them to the raw bucket. A publisher table's bytes land beside its + manifest; a public microdata release (``kind: microdata_release``) is + archived only under an allowlisted ``licence`` with ``licence_evidence`` + bound to ``expected_sha256``, and its bytes are staged in a transient + directory outside the package tree and uploaded from there. Licensed and + restricted artifacts are registered hash-only with + :func:`chronicle.registration.register_hash_only_artifact`. + + Every refusal happens before the publisher is read; the checks that need + the bytes (expected identity, recorded identity) run before anything is + written. + """ + r2_bucket = r2_bucket or default_r2_raw_bucket() output = Path(output_dir) + manifest_path = _manifest_path(output, manifest_filename) + access_class = normalize_access(access) + if is_hash_only(access_class): + raise ManifestAccessError( + f"fetch-artifact stores bytes and refuses access={access_class!r}. " + "Register a licensed or restricted artifact by identity with " + "`chronicle register-artifact`." + ) + # The destination name is a pure function of the arguments, so it is + # resolved -- and any alias of a registered name refused -- before the + # publisher is read. Reading first and refusing afterwards would pull + # gated bytes to decide their name. + artifact_filename = bare_filename( + filename if filename is not None else _infer_artifact_filename(source_url), + what=( + "--filename" + if filename is not None + else "The filename inferred from the URL" + ), + ) + expected = _expected_identity(expected_sha256, expected_size_bytes) resolved_r2_prefix = resolve_r2_prefix( prefix=r2_prefix, default_prefix=DEFAULT_R2_PREFIX, source_id=source_id, package_path=output, ) - fetched_at = datetime.now(UTC).replace(microsecond=0).isoformat() - content, inferred_filename = _read_artifact(source_url) - artifact_filename = filename or inferred_filename - if not artifact_filename: - raise ValueError("Could not infer artifact filename; pass --filename.") - output.mkdir(parents=True, exist_ok=True) - local_path = output / artifact_filename - local_path.write_bytes(content) + # Read and validate the manifest being written before anything is fetched: + # a manifest Chronicle cannot read, a manifest name that would sit beside + # the ones a package keeps, a recorded block that names two different + # objects, or a registration the fetch would overwrite are all refusals + # that need not touch the publisher. + _refuse_a_stray_default_manifest(output, manifest_path) + existing_manifest = _read_manifest(manifest_path) + _manifest_files(existing_manifest, manifest_path) + # The byte boundary is checked first: overwriting a hash-only registration + # with bytes is the more serious refusal, and its message is the one the + # caller needs, not a prompt about the manifest's kind or licence. + _assert_no_hash_only_entry(existing_manifest, manifest_path, artifact_filename) + manifest_kind_value = _resolve_manifest_kind( + existing_manifest, + manifest_path=manifest_path, + requested_kind=kind, + ) + _assert_manifest_valid_for_fetch( + existing_manifest, + manifest_path, + kind=manifest_kind_value, + package_dir=output, + ) + licence_text = licence.strip() if isinstance(licence, str) else None + release = manifest_kind_value == MICRODATA_RELEASE_KIND + evidence: dict[str, str] | None = None + if release: + evidence = _release_fetch_evidence( + manifest_path, + existing_manifest, + filename=artifact_filename, + package_dir=output, + licence=licence_text, + vintage=vintage, + publisher=publisher, + expected=expected, + licence_evidence=licence_evidence, + ) + + vintage_key, existing_value, selected_spec, _index = _select_vintage_entry( + existing_manifest, + manifest_path=manifest_path, + year=year, + filename=artifact_filename, + kind=manifest_kind_value, + ) + recorded_identity = _recorded_identity( + selected_spec, + manifest_path=manifest_path, + year=vintage_key, + ) + _assert_table_vintage_is_revisable( + existing_value, + recorded_identity, + manifest_path=manifest_path, + year=vintage_key, + filename=artifact_filename, + release=release, + ) + if ( + expected.sha256 + and recorded_identity is not None + and not record_revision + and not recorded_identity.holds( + sha256=expected.sha256, filename=artifact_filename + ) + ): + # The manifest already identifies other bytes: no download can satisfy + # both the recorded identity and the expectation, so refuse before it. + raise ExpectedArtifactIdentityError( + f"{manifest_path} entry {vintage_key!r} already records " + f"sha256={recorded_identity.sha256} " + f"filename={recorded_identity.filename or 'unknown'}, and the " + f"reviewed pin expects sha256={expected.sha256}. Re-review the pin, " + "or pass --record-revision together with the reviewed " + "--expected-sha256 to register the publisher revision." + ) + + fetched_at = datetime.now(UTC).replace(microsecond=0).isoformat() + content, _inferred_filename = _read_artifact(source_url) sha256 = hashlib.sha256(content).hexdigest() size_bytes = len(content) - manifest_path = output / "manifest.yaml" + + # Guard before the cached artifact is touched. A rejected fetch must leave + # the recorded bytes and their manifest entry exactly as they were. + _assert_expected_identity( + expected, + manifest_path=manifest_path, + year=vintage_key, + filename=artifact_filename, + source_url=source_url, + sha256=sha256, + size_bytes=size_bytes, + ) + _assert_recorded_identity_holds_these_bytes( + recorded_identity, + manifest_path=manifest_path, + year=vintage_key, + filename=artifact_filename, + sha256=sha256, + size_bytes=size_bytes, + r2_bucket=r2_bucket, + record_revision=record_revision, + ) + + if release: + # Public microdata never lands in the package tree: it is staged in an + # untracked, transient directory and uploaded from there. + local_path = microdata_staging_path( + staging_dir=staging_dir, + source_id=source_id, + package_id=package_id, + year=year, + sha256=sha256, + filename=artifact_filename, + ) + local_path.parent.mkdir(parents=True, exist_ok=True) + else: + output.mkdir(parents=True, exist_ok=True) + local_path = output / artifact_filename + local_path.write_bytes(content) r2_location = ArtifactStorageLocation( provider="r2", @@ -447,15 +949,23 @@ def fetch_source_artifact( source_id=source_id, package_id=package_id, dataset=dataset or f"{source_id}_{package_id}", - source_page=source_page or source_url, - table=table or package_id, + source_page=source_page, + table=table, + publisher=publisher, year=year, filename=artifact_filename, source_url=source_url, sha256=sha256, size_bytes=size_bytes, fetched_at=fetched_at, + access=access_class, + licence=licence_text, + kind=manifest_kind_value, + vintage=vintage, + licence_evidence=evidence, + expected=expected, r2_location=(r2_location if upload_r2 and r2_upload and r2_upload.ok else None), + record_revision=record_revision, ) return ArtifactFetchReport( @@ -469,7 +979,7 @@ def fetch_source_artifact( sha256=sha256, size_bytes=size_bytes, fetched_at=fetched_at, - r2_location=r2_location if upload_r2 and r2_upload and r2_upload.ok else None, + r2_location=r2_location if upload_r2 else None, r2_upload=r2_upload, errors=tuple(errors), ) @@ -482,12 +992,13 @@ def publish_derived_artifacts( package_id: str, year: int, build_id: str | None = None, - r2_bucket: str = DEFAULT_R2_DERIVED_BUCKET, + r2_bucket: str | None = None, r2_prefix: str | None = None, wrangler_command: str = "npx wrangler", build_artifacts_output: str | Path | None = None, ) -> DerivedArtifactPublishReport: """Upload a deterministic build output directory to the derived R2 bucket.""" + r2_bucket = r2_bucket or default_r2_derived_bucket() input_path = Path(input_dir) if not input_path.exists(): return DerivedArtifactPublishReport( @@ -614,14 +1125,26 @@ def publish_derived_artifacts( def publish_source_artifacts( root: str | Path, *, - manifest_filename: str = "manifest.yaml", + manifest_filename: str = DEFAULT_MANIFEST_FILENAME, source_id: str | None = None, package_id: str | None = None, - r2_bucket: str = DEFAULT_R2_RAW_BUCKET, + r2_bucket: str | None = None, r2_prefix: str | None = None, - wrangler_command: str = "npx wrangler", + wrangler_command: str = DEFAULT_WRANGLER_COMMAND, + skip_hash_only: bool = False, + staging_dir: str | Path | None = None, ) -> RawArtifactPublishReport: - """Upload manifest-declared raw source artifacts and record R2 locations.""" + """Upload manifest-declared raw source artifacts and record R2 locations. + + Only ``public`` artifacts are uploaded. A licensed or restricted entry is + refused: no bytes are read or sent, and the entry carries a + ``hash_only_access_refuses_bytes`` error unless ``skip_hash_only`` marks the + scan as deliberately mixed. A manifest that declares no kind (and is not + frozen kindless), or whose entries collide, is reported and skipped whole: + nothing under it is uploaded. A public microdata release's bytes are read + from the staging directory, never from beside the manifest. + """ + r2_bucket = r2_bucket or default_r2_raw_bucket() root_path = Path(root) if not root_path.exists(): return RawArtifactPublishReport( @@ -634,8 +1157,8 @@ def publish_source_artifacts( errors: list[str] = [] for manifest_path in sorted(root_path.rglob(manifest_filename)): try: - manifest = yaml.safe_load(manifest_path.read_text(encoding="utf-8")) or {} - except (OSError, yaml.YAMLError) as exc: + manifest = _read_manifest(manifest_path) + except (OSError, MalformedManifestError) as exc: errors.append(f"Could not read {manifest_path}: {exc}") continue @@ -663,22 +1186,36 @@ def publish_source_artifacts( errors.append(f"Could not resolve R2 prefix for {manifest_path}: {exc}") continue + kind, kind_error = safe_manifest_kind(manifest, manifest_path=manifest_path) + manifest_errors = [kind_error] if kind_error else [] + manifest_errors.extend(validate_manifest_files(manifest)) + if manifest_errors: + # Validate, then touch: a manifest Chronicle cannot classify or + # whose entries collide is reported and left alone; publishing any + # entry under it could ship bytes through the wrong record. + errors.extend(f"{code}: {manifest_path}" for code in manifest_errors) + continue updated = False for year, spec in files.items(): - entry, updated_spec = _publish_raw_manifest_entry( - manifest_path, - manifest_source_id, - manifest_package_id, - year, - spec, - r2_bucket=r2_bucket, - r2_prefix=resolved_r2_prefix, - wrangler_command=wrangler_command, - ) - entries.append(entry) - if updated_spec is not None and isinstance(spec, dict): - spec.update(updated_spec) - updated = True + for file_spec in iter_file_specs(spec, kind=kind): + entry, updated_spec = _publish_raw_manifest_entry( + manifest_path, + manifest_source_id, + manifest_package_id, + year, + file_spec, + manifest=manifest, + kind=kind, + r2_bucket=r2_bucket, + r2_prefix=resolved_r2_prefix, + wrangler_command=wrangler_command, + skip_hash_only=skip_hash_only, + staging_dir=staging_dir, + ) + entries.append(entry) + if updated_spec is not None and isinstance(file_spec, dict): + file_spec.update(updated_spec) + updated = True if updated: manifest.setdefault("source_id", manifest_source_id) manifest.setdefault("package_id", manifest_package_id) @@ -738,9 +1275,16 @@ def write_build_artifacts_jsonl( def inventory_source_artifacts( root: str | Path, *, - manifest_filename: str = "manifest.yaml", + manifest_filename: str = DEFAULT_MANIFEST_FILENAME, + staging_dir: str | Path | None = None, ) -> ArtifactInventoryReport: - """Inventory manifest-declared source artifacts under a root directory.""" + """Inventory manifest-declared source artifacts under a root directory. + + Reports every manifest-level defect (a missing or unknown kind, a vintage + under two key spellings, filename collisions) alongside the per-entry + codes, and treats a public microdata release's bytes as staged outside + the tree: a copy beside the manifest is an error, not an artifact. + """ root_path = Path(root) errors: list[str] = [] entries: list[ArtifactInventoryEntry] = [] @@ -752,6 +1296,7 @@ def inventory_source_artifacts( "artifact_count": 0, "missing_count": 0, "checksum_mismatch_count": 0, + "hash_only_count": 0, "r2_link_count": 0, }, entries=(), @@ -761,24 +1306,41 @@ def inventory_source_artifacts( manifests = sorted(root_path.rglob(manifest_filename)) for manifest_path in manifests: try: - manifest = yaml.safe_load(manifest_path.read_text(encoding="utf-8")) or {} - files = manifest.get("files") or {} - except (OSError, yaml.YAMLError) as exc: + manifest = _read_manifest(manifest_path) + except (OSError, MalformedManifestError) as exc: errors.append(f"Could not read {manifest_path}: {exc}") continue + files = manifest.get("files") or {} if not isinstance(files, dict): errors.append(f"Manifest files must be a mapping: {manifest_path}") continue + kind, kind_error = safe_manifest_kind(manifest, manifest_path=manifest_path) + if kind_error: + errors.append(f"{kind_error}: {manifest_path}") + errors.extend( + f"{code}: {manifest_path}" for code in validate_manifest_files(manifest) + ) for year, spec in files.items(): - entries.append(_inventory_entry(manifest_path, year, spec)) + for file_spec in iter_file_specs(spec, kind=kind): + entries.append( + _inventory_entry( + manifest_path, + year, + file_spec, + manifest=manifest, + kind=kind, + staging_dir=staging_dir, + ) + ) counts = { "manifest_count": len(manifests), "artifact_count": len(entries), - "missing_count": sum(1 for entry in entries if not entry.exists), + "missing_count": sum(1 for entry in entries if "missing_file" in entry.errors), "checksum_mismatch_count": sum( 1 for entry in entries if "checksum_mismatch" in entry.errors ), + "hash_only_count": sum(1 for entry in entries if entry.hash_only), "r2_link_count": sum(1 for entry in entries if entry.r2 is not None), } return ArtifactInventoryReport( @@ -791,12 +1353,15 @@ def inventory_source_artifacts( def bootstrap_r2_buckets( *, - raw_bucket: str = DEFAULT_R2_RAW_BUCKET, - derived_bucket: str = DEFAULT_R2_DERIVED_BUCKET, + raw_bucket: str | None = None, + derived_bucket: str | None = None, wrangler_command: str = "npx wrangler", ) -> R2BootstrapReport: """Create the R2 buckets Chronicle expects, if Wrangler is authenticated.""" - buckets = (raw_bucket, derived_bucket) + buckets = ( + raw_bucket or default_r2_raw_bucket(), + derived_bucket or default_r2_derived_bucket(), + ) commands: list[ArtifactCommandResult] = [] errors: list[str] = [] @@ -1010,9 +1575,9 @@ def infer_build_id(input_dir: str | Path) -> str | None: if build_id: return str(build_id) - db_path = input_path / "ledger.db" + db_path = input_path / CHRONICLE_DB_FILENAME if not db_path.exists(): - db_path = input_path / "ledger.db" + db_path = input_path / LEGACY_CHRONICLE_DB_FILENAME if db_path.exists(): with sqlite3.connect(db_path) as connection: row = connection.execute( @@ -1043,48 +1608,946 @@ def _filename_from_url(source_url: str) -> str: return Path(unquote(parsed.path)).name +def _read_manifest(manifest_path: Path) -> dict[str, Any]: + """Return a manifest's parsed payload, refusing a document it cannot read. + + An absent or empty manifest reads as an empty mapping: ``fetch-artifact`` + writes the first entry into a package that has none. A document that parses + as anything else -- a list, a scalar, a truncated or half-merged file -- is + not an absent manifest, and treating it as one would let the fetch replace + it with a single entry and drop everything it recorded. + """ + if not manifest_path.exists(): + return {} + try: + payload = yaml.safe_load(manifest_path.read_text(encoding="utf-8")) + except yaml.YAMLError as exc: + raise MalformedManifestError( + f"{manifest_path} is not valid YAML: {exc}" + ) from exc + if payload is None: + return {} + if not isinstance(payload, dict): + raise MalformedManifestError( + f"{manifest_path} must be a YAML mapping; it parses as a " + f"{type(payload).__name__}. Chronicle will not overwrite a manifest " + "it cannot read." + ) + return payload + + +def _infer_artifact_filename(source_url: str) -> str: + """Return the filename :func:`_read_artifact` would report, without I/O. + + The name is a pure function of the URL: the last path segment for http(s) + and ``file://`` URLs, the basename for a bare path. Resolving it before + the read lets every filename guard run before the publisher is touched. + """ + parsed = urlparse(source_url) + if parsed.scheme in ("http", "https"): + return _filename_from_url(source_url) + if parsed.scheme == "file": + return Path(unquote(parsed.path)).name + if not parsed.scheme: + return Path(source_url).name + raise ValueError(f"Unsupported source URL scheme: {parsed.scheme}") + + +@dataclass(frozen=True) +class ExpectedIdentity: + """The bytes a reviewed pin says the publisher serves, if any.""" + + sha256: str | None + size_bytes: int | None + + +def _expected_identity( + expected_sha256: str | None, + expected_size_bytes: int | None, +) -> ExpectedIdentity: + """Validate the expected-identity arguments before any I/O.""" + sha256 = expected_sha256.strip() if isinstance(expected_sha256, str) else None + if expected_sha256 is not None and ( + not sha256 or not _SHA256_KEY_SEGMENT.fullmatch(sha256) + ): + raise ExpectedArtifactIdentityError( + "--expected-sha256 must be a lowercase 64-character SHA-256 taken " + f"from a reviewed pin, not {expected_sha256!r}. Never invent a hash." + ) + if expected_size_bytes is not None and ( + isinstance(expected_size_bytes, bool) + or not isinstance(expected_size_bytes, int) + or expected_size_bytes <= 0 + ): + raise ExpectedArtifactIdentityError( + "--expected-size-bytes must be a positive integer, not " + f"{expected_size_bytes!r}." + ) + return ExpectedIdentity(sha256=sha256, size_bytes=expected_size_bytes) + + +def _assert_expected_identity( + expected: ExpectedIdentity, + *, + manifest_path: Path, + year: Any, + filename: str, + source_url: str, + sha256: str, + size_bytes: int, +) -> None: + """Refuse fetched bytes the reviewed pin does not cover. + + ``--record-revision`` does not override this: it governs what the manifest + records about bytes Chronicle chose to archive, and an expectation is the + statement that only reviewed bytes are archived at all. + """ + if expected.sha256 is None and expected.size_bytes is None: + return + if (expected.sha256 is None or expected.sha256 == sha256) and ( + expected.size_bytes is None or expected.size_bytes == size_bytes + ): + return + raise ExpectedArtifactIdentityError( + f"{manifest_path} entry {year!r} {filename}: the bytes served by " + f"{source_url} are not the bytes the reviewed pin covers. Expected " + f"sha256={expected.sha256 or 'unspecified'} " + f"size_bytes={expected.size_bytes or 'unspecified'}; fetched " + f"sha256={sha256} size_bytes={size_bytes}. The publisher is serving " + "bytes the pin does not describe. Chronicle will not archive an " + "unreviewed reissue; re-review the release and re-run with the pin you " + "reviewed." + ) + + +def _resolve_manifest_kind( + existing_manifest: dict[str, Any], + *, + manifest_path: Path, + requested_kind: str | None, +) -> str: + """Return the kind a fetch writes, refusing a conflict with the manifest. + + A manifest's kind is fixed once declared: an explicit ``--kind`` that + differs from it would reclassify every entry the manifest holds as a side + effect of one fetch. A kindless manifest with content is refused unless it + is frozen kindless byte for byte, and a stored kind Chronicle does not + recognise is never masked by the command line. + """ + requested = ( + normalize_manifest_kind({"kind": requested_kind}) + if requested_kind is not None + else None + ) + try: + stored = normalize_manifest_kind(existing_manifest, manifest_path=manifest_path) + except ManifestKindError as exc: + raise ManifestAccessError( + f"{exc} fetch-artifact will not add to a manifest whose kind it " + "cannot read; declare the kind by editing the manifest deliberately." + ) from exc + except ManifestAccessError as exc: + raise ManifestAccessError( + f"{manifest_path} declares an unknown manifest kind " + f"{existing_manifest.get('kind')!r}; expected one of " + f"{list(MANIFEST_KINDS)}. Fix the manifest before fetching into it." + ) from exc + if requested is None: + return stored + if requested != stored and ( + existing_manifest.get("kind") is not None or has_file_entries(existing_manifest) + ): + # A declared kind is fixed, and a frozen kindless manifest with + # entries is a publisher table. A manifest with neither is declared + # by this fetch. + raise ManifestAccessError( + f"{manifest_path} is a {stored} manifest; refusing to fetch into it " + f"as a {requested}. A manifest's kind is fixed once declared: " + f"register a {requested} in its own package directory, or migrate " + "this manifest deliberately." + ) + return requested + + +def _assert_manifest_valid_for_fetch( + manifest: dict[str, Any], + manifest_path: Path, + *, + kind: str, + package_dir: Path, +) -> None: + """Refuse to fetch into a manifest inventory would report as invalid. + + Uses the exact vocabulary ``inventory-artifacts`` and ``publish-raw`` + report, so a fetch never carries an invalid registration forward -- or + conceals one under a rewrite. + """ + codes: list[str] = list(validate_manifest_files(manifest)) + files = manifest.get("files") or {} + if isinstance(files, dict): + for key, spec in files.items(): + for file_spec in iter_file_specs(spec, kind=kind): + name = ( + file_spec.get("filename") if isinstance(file_spec, dict) else None + ) + exists = ( + bool(name) + and is_bare_filename(name) + and (package_dir / str(name)).exists() + ) + for code in validate_file_entry( + file_spec, + kind=kind, + manifest=manifest, + local_file_exists=exists, + ): + codes.append(f"{key!r}/{name}: {code}") + if codes: + raise ManifestAccessError( + f"{manifest_path} is not a valid {kind} manifest: " + f"{'; '.join(codes)}. fetch-artifact rewrites this manifest and " + "will not carry an invalid registration forward; " + "inventory-artifacts reports the same codes." + ) + + +def _release_fetch_evidence( + manifest_path: Path, + existing_manifest: dict[str, Any], + *, + filename: str, + package_dir: Path, + licence: str | None, + vintage: str | None, + publisher: str | None, + expected: ExpectedIdentity, + licence_evidence: Mapping[str, Any] | None, +) -> dict[str, str]: + """Return the ``licence_evidence`` block a public release fetch records. + + Bytes only with artifact-bound redistribution evidence: the licence must + be on the allowlist, the evidence must bind this artifact (by the reviewed + SHA-256) to that term, and the release must carry its publisher and + vintage. Public microdata is staged outside the package tree, so a file of + that name beside the manifest is refused as tracked microdata bytes. + """ + if not licence: + raise ManifestAccessError( + f"{manifest_path} registers a microdata release, so every entry " + "must record its publisher licence; pass --licence." + ) + if not (vintage and vintage.strip()): + raise ManifestAccessError( + f"{manifest_path} registers a microdata release, so every entry " + "must record its publisher vintage; pass --vintage." + ) + if not (publisher or existing_manifest.get("publisher")): + raise ManifestAccessError( + f"{manifest_path} registers a microdata release, so it must name " + "the publisher; pass --publisher." + ) + if expected.sha256 is None: + raise ManifestAccessError( + "A public microdata release is archived only against a reviewed " + "checksum that its licence evidence covers; pass --expected-sha256." + ) + if not is_redistributable_licence(licence): + raise ManifestAccessError( + f"licence {licence!r} is not on Chronicle's allowlist of " + f"redistributable terms {sorted(REDISTRIBUTABLE_LICENCES)}. A " + "public-download file without redistribution evidence is classed " + "licensed: register it hash-only with `chronicle register-artifact`." + ) + supplied = dict(licence_evidence or {}) + evidence = { + "issuer": str(supplied.get("issuer") or "").strip(), + "licence": licence, + "scope": str(supplied.get("scope") or "").strip(), + "url": str(supplied.get("url") or "").strip(), + "sha256": expected.sha256, + } + codes = licence_evidence_errors(evidence, licence=licence, sha256=expected.sha256) + if codes: + raise ManifestAccessError( + f"{manifest_path}: archiving {filename!r} needs licence evidence " + f"binding it to {licence!r}: {', '.join(codes)}. Pass " + "--licence-evidence-issuer, --licence-evidence-scope and a durable " + "--licence-evidence-url; the evidence covers --expected-sha256." + ) + if (package_dir / filename).exists(): + raise ManifestAccessError( + f"{package_dir / filename} exists beside the manifest. Public " + "microdata bytes are staged outside the package tree and uploaded " + "from there; a repository never holds them. Remove the file first." + ) + return evidence + + +def _select_vintage_entry( + payload: dict[str, Any], + *, + manifest_path: Path, + year: Any, + filename: str, + kind: str, +) -> tuple[Any, Any, dict[str, Any], int | None]: + """Locate the entry a fetch revises: ``(key, files[key], entry, index)``. + + The vintage key is whichever spelling the manifest already uses for + ``year`` (``2023`` and ``'2023'`` are one vintage). A publisher table's + vintage is one mapping, and that entry is its identity whatever filename + it records. A microdata release lists several files under one vintage, and + the entry is the one whose bare filename matches. ``index`` is the entry's + position in that list, or None. + """ + files = payload.get("files") if isinstance(payload, dict) else None + if files is None: + return year, None, {}, None + if not isinstance(files, dict): + raise MalformedManifestError( + f"{manifest_path} files must be a mapping; it is a " + f"{type(files).__name__}. Chronicle will not write into a manifest " + "it cannot read." + ) + try: + key = resolve_vintage_key(files, year) + except AmbiguousVintageKeyError as exc: + raise MalformedManifestError( + f"{manifest_path}: {exc} Chronicle will not choose which entry is " + "the record." + ) from exc + if key is None: + return year, None, {}, None + existing = files[key] + if isinstance(existing, dict): + return key, existing, existing, None + if isinstance(existing, list): + if kind != MICRODATA_RELEASE_KIND: + raise MalformedManifestError( + f"{manifest_path} entry {key!r} lists {len(existing)} files, but " + f"the manifest is a {kind} manifest. Only a kind: " + "microdata_release manifest may list several files under one " + "vintage (list_file_spec_requires_microdata_release_kind); " + "Chronicle will not add to or revise a vintage it cannot " + "validate." + ) + wanted = filename_key(filename) + matches = [ + (index, entry) + for index, entry in enumerate(existing) + if isinstance(entry, dict) and filename_key(entry.get("filename")) == wanted + ] + if len(matches) > 1: + raise MalformedManifestError( + f"{manifest_path} entry {key!r} lists {len(matches)} entries " + f"named {filename!r}; a fetch can revise exactly one, and " + "Chronicle will not guess which." + ) + if matches: + return key, existing, matches[0][1], matches[0][0] + return key, existing, {}, None + raise MalformedManifestError( + f"{manifest_path} entry {key!r} must be a mapping or a list of " + f"mappings; it is a {type(existing).__name__}." + ) + + +def _assert_table_vintage_is_revisable( + existing_value: Any, + recorded_identity: RecordedIdentity | None, + *, + manifest_path: Path, + year: Any, + filename: str, + release: bool, +) -> None: + """Refuse a second filename in a publisher-table vintage with no identity. + + A publisher table holds one file per vintage. When its entry identifies + bytes, a fetch under another name is judged by the recorded identity (and + ``--record-revision`` supersedes the entry). An entry that identifies + nothing -- no ``sha256``, no recorded object -- cannot be superseded, and + adding a second file would turn the mapping into a list every reader + refuses. + """ + if release or not isinstance(existing_value, dict) or recorded_identity: + return + recorded_name = existing_value.get("filename") + if recorded_name is None or filename_key(recorded_name) == filename_key(filename): + return + raise MalformedManifestError( + f"{manifest_path} entry {year!r} records {recorded_name!r} without a " + f"sha256; a publisher table holds one file per vintage, so {filename!r} " + "cannot be added and there is no recorded identity to supersede. " + "Register the vintage's bytes first or fix the entry by hand." + ) + + +def _recorded_storage(spec: Any) -> dict[str, Any]: + """Return a manifest file spec's recorded ``storage`` block, if any.""" + if not isinstance(spec, dict): + return {} + storage = spec.get("storage") + return storage if isinstance(storage, dict) else {} + + +def _recorded_r2(spec: Any) -> dict[str, Any]: + """Return the recorded ``storage.r2`` block verbatim, if any. + + Raw access, for callers that carry the block forward as history. Callers + that reason about which object it names go through + :func:`_validated_recorded_r2` instead. + """ + recorded = _recorded_storage(spec).get("r2") + return recorded if isinstance(recorded, dict) else {} + + +def _split_r2_uri(uri: str) -> tuple[str, str, str] | None: + """Split ``provider://bucket/key`` into its three parts, or None.""" + provider, separator, remainder = uri.partition("://") + if not separator or not provider: + return None + bucket, separator, key = remainder.partition("/") + if not separator or not bucket or not key: + return None + return (provider, bucket, key) + + +def _validated_recorded_storage( + spec: Any, + *, + manifest_path: Path, + year: Any, +) -> dict[str, Any]: + """Return the entry's ``storage`` mapping, refusing a malformed one.""" + if not isinstance(spec, dict) or "storage" not in spec: + return {} + storage = spec["storage"] + if not isinstance(storage, dict): + raise MalformedManifestError( + f"{manifest_path} entry {year!r} storage must be a mapping; it is a " + f"{type(storage).__name__}." + ) + return storage + + +def _validated_recorded_r2( + spec: Any, + *, + manifest_path: Path, + year: Any, +) -> RecordedR2Object | None: + """Return the object a recorded ``storage.r2`` block names, or None. + + Every locator field the block supplies is cross-checked against every + other: ``key`` against the URI's path, ``bucket`` against its authority, + ``provider`` against its scheme, and the resulting key against the + canonical content-addressed shape :func:`build_r2_key` writes. Reading one + field and trusting the rest is what lets a block that says two different + things survive a preserve or a publish. + """ + storage = _validated_recorded_storage(spec, manifest_path=manifest_path, year=year) + if "r2" not in storage: + return None + block = storage["r2"] + where = f"{manifest_path} entry {year!r} storage.r2" + if not isinstance(block, dict): + raise MalformedManifestError( + f"{where} must be a mapping; it is a {type(block).__name__}." + ) + + supplied: dict[str, str] = {} + for field in ("provider", "bucket", "key", "uri"): + value = block.get(field) + if value is None: + continue + if not isinstance(value, str) or not value.strip(): + raise RecordedR2LocatorError( + f"{where}: {field} must be a non-empty string, not {value!r}." + ) + supplied[field] = value + + provider = supplied.get("provider") + bucket = supplied.get("bucket") + key = supplied.get("key") + uri = supplied.get("uri") + if uri is not None: + parts = _split_r2_uri(uri) + if parts is None: + raise RecordedR2LocatorError( + f"{where}: uri {uri!r} is not provider://bucket/key." + ) + for field, value, from_uri in zip( + ("provider", "bucket", "key"), (provider, bucket, key), parts + ): + if value is not None and value != from_uri: + raise RecordedR2LocatorError( + f"{where}: {field}={value!r} contradicts uri {uri!r}, which " + f"names {from_uri!r}. The block records two different " + "objects, so Chronicle cannot say which bytes it claims." + ) + provider, bucket, key = ( + provider or parts[0], + bucket or parts[1], + key or parts[2], + ) + + missing = [ + field + for field, value in ( + ("provider", provider), + ("bucket", bucket), + ("key", key), + ) + if not value + ] + if missing: + raise RecordedR2LocatorError( + f"{where}: records no {', '.join(missing)}. A recorded block has to " + "locate its object: provider, bucket and key, or a uri that " + "supplies them." + ) + + segments = key.split("/") + if ( + len(segments) < 2 + or not all(segments) + or not _SHA256_KEY_SEGMENT.fullmatch(segments[-2]) + ): + raise RecordedR2LocatorError( + f"{where}: key {key!r} is not content-addressed. A raw key ends in " + "{sha256}/{filename}, which is what says the object holds the " + "entry's bytes; Chronicle will not guess for a key that does not." + ) + return RecordedR2Object( + provider=provider, + bucket=bucket, + key=key, + sha256=segments[-2], + filename=segments[-1], + ) + + +def _recorded_identity( + spec: Any, + *, + manifest_path: Path, + year: Any, +) -> RecordedIdentity | None: + """Return what a manifest entry says its vintage holds, if anything. + + A published entry is identified by its recorded object's content-addressed + key. An entry that has not been published yet -- registered without an + upload, or left behind by a failed one -- is identified by its own declared + ``sha256`` and ``filename``. Both are recorded identities, and a fetch of + different bytes over either one is a publisher revision. + """ + recorded_r2 = _validated_recorded_r2(spec, manifest_path=manifest_path, year=year) + declared_sha256 = spec.get("sha256") if isinstance(spec, dict) else None + declared_sha256 = declared_sha256 if isinstance(declared_sha256, str) else None + declared_filename = spec.get("filename") if isinstance(spec, dict) else None + declared_filename = ( + declared_filename if isinstance(declared_filename, str) else None + ) + size_bytes = spec.get("size_bytes") if isinstance(spec, dict) else None + size_bytes = size_bytes if isinstance(size_bytes, int) else None + if recorded_r2 is not None: + return RecordedIdentity( + sha256=recorded_r2.sha256, + filename=recorded_r2.filename, + # Only report a size the recorded key agrees with: an entry can + # arrive here already describing the new bytes. + size_bytes=size_bytes if declared_sha256 == recorded_r2.sha256 else None, + declared_sha256=declared_sha256, + r2=recorded_r2, + ) + if not declared_sha256: + return None + return RecordedIdentity( + sha256=declared_sha256, + filename=Path(declared_filename).name if declared_filename else "", + size_bytes=size_bytes, + declared_sha256=declared_sha256, + r2=None, + ) + + +def _revision_error_message( + *, + manifest_path: Path, + year: Any, + filename: str, + identity: RecordedIdentity, + sha256: str, + size_bytes: int, + r2_bucket: str, +) -> str: + """Explain a refused fetch: recorded identity, fetched identity, next step.""" + records = ( + f"already records the R2 object {identity.r2.uri}, which holds" + if identity.r2 is not None + else "already records" + ) + message = ( + f"{manifest_path} entry {year!r} {records} " + f"sha256={identity.sha256} " + f"filename={identity.filename or 'unknown'} " + f"size_bytes=" + f"{identity.size_bytes if identity.size_bytes is not None else 'unknown'}. " + f"The fetched bytes are sha256={sha256} filename={Path(filename).name} " + f"size_bytes={size_bytes}. Chronicle will not rewrite a vintage that " + "identifies specific bytes to describe bytes it never identified." + ) + if identity.r2 is not None and identity.declared_sha256 not in ( + None, + identity.sha256, + ): + message += ( + f" (The entry also declares sha256={identity.declared_sha256}, which " + "its own R2 key contradicts: an earlier fetch rewrote the hash " + "without moving the object.)" + ) + return message + ( + " The same vintage with new bytes is a new release revision " + "(docs/adr-chronicle-fact-identity-v2.md). Re-run with " + "--record-revision to store the fetched bytes under their own " + f"content-addressed key in {r2_bucket} and keep the superseded object " + "in storage.previous_r2." + ) + + +def _assert_recorded_identity_holds_these_bytes( + identity: RecordedIdentity | None, + *, + manifest_path: Path, + year: Any, + filename: str, + sha256: str, + size_bytes: int, + r2_bucket: str, + record_revision: bool, +) -> None: + """Refuse a publisher revision that has not been opted into.""" + if identity is None or identity.holds(sha256=sha256, filename=filename): + return + if identity.sha256 == sha256: + # The recorded object holds exactly these bytes under another name. A + # rename is not a publisher revision, so --record-revision does not + # apply, and silently adopting the new name would leave the entry's + # filename disagreeing with the key its own storage block records. + raise SourceArtifactRevisionError( + f"{manifest_path} entry {year!r} already records these exact bytes " + f"(sha256={sha256}) as filename={identity.filename}; this fetch " + f"names them {Path(filename).name}. A rename is not a release " + "revision, so --record-revision does not apply. Re-run with " + f"--filename {identity.filename} to keep the recorded identity." + ) + if record_revision: + return + raise SourceArtifactRevisionError( + _revision_error_message( + manifest_path=manifest_path, + year=year, + filename=filename, + identity=identity, + sha256=sha256, + size_bytes=size_bytes, + r2_bucket=r2_bucket, + ) + ) + + +def _superseding_storage( + recorded_spec: dict[str, Any], + *, + recorded_r2: RecordedR2Object | None, + new_r2: dict[str, Any] | None, + superseded_at: str, +) -> dict[str, Any]: + """Return a storage block in which the recorded object becomes history. + + ``storage.r2`` only ever names the object that holds the entry's current + bytes. The superseded block is appended, oldest first, to + ``storage.previous_r2`` so the earlier bytes stay addressable by the URI + archived witness records already pin. An entry that was never published has + no object to supersede, and gets no ``previous_r2`` key. + """ + storage = dict(_recorded_storage(recorded_spec)) + previous = storage.get("previous_r2") + entries = list(previous) if isinstance(previous, list) else [] + if recorded_r2 is not None: + entry = dict(_recorded_r2(recorded_spec)) + entry["sha256"] = recorded_r2.sha256 + if recorded_spec.get("sha256") == recorded_r2.sha256: + # Only carry metadata the superseded key agrees with: a manifest + # can arrive here already describing the new bytes. + for field in ("size_bytes", "fetched_at", "source_url"): + value = recorded_spec.get(field) + if value is not None: + entry[field] = value + entry["superseded_at"] = superseded_at + entries.append(entry) + if entries: + storage["previous_r2"] = entries + if new_r2 is None: + storage.pop("r2", None) + else: + storage["r2"] = new_r2 + return storage + + +#: Entry fields a fetch owns. Everything else an entry already records -- +#: notes, doi, study, access_route, a vintage the fetch did not restate -- is +#: carried forward when the fetch replaces that entry, so a re-fetch is never a +#: silent de-registration. +_FETCH_OWNED_FIELDS: frozenset[str] = frozenset( + { + "filename", + "source_url", + "access", + "licence", + "licence_evidence", + "sha256", + "size_bytes", + "fetched_at", + "verified_at", + "hash_source", + "attested_by", + "storage", + } +) + + def _upsert_manifest( manifest_path: Path, *, source_id: str, package_id: str, dataset: str, - source_page: str, - table: str, + source_page: str | None, + table: str | None, + publisher: str | None, year: int, filename: str, source_url: str, sha256: str, size_bytes: int, fetched_at: str, + access: str, + licence: str | None, + kind: str, + vintage: str | None, + licence_evidence: Mapping[str, Any] | None, + expected: ExpectedIdentity, r2_location: ArtifactStorageLocation | None, + record_revision: bool = False, ) -> None: - if manifest_path.exists(): - payload = yaml.safe_load(manifest_path.read_text(encoding="utf-8")) or {} - else: - payload = {} + """Write one fetched entry into its manifest, in place. + + The guards fetch_source_artifact ran are repeated against the freshly + re-read manifest, so no caller can reach a false-provenance write by + another route. The entry the fetch revises is located by vintage key and + bare filename and replaced where it sits; a publisher-table vintage stays + one mapping and a release vintage stays a list, and every manifest this + command touches declares its kind. + """ + payload = _read_manifest(manifest_path) + kind = _resolve_manifest_kind( + payload, manifest_path=manifest_path, requested_kind=kind + ) + _assert_no_hash_only_entry(payload, manifest_path, filename) payload.setdefault("source_id", source_id) payload.setdefault("package_id", package_id) + payload = _with_declared_kind(payload, kind) payload.setdefault("dataset", dataset) + if publisher: + payload.setdefault("publisher", publisher) payload.setdefault("source_page", source_page) payload.setdefault("table", table) - payload.setdefault("files", {}) - file_entry: dict[str, Any] = { - "filename": filename, - "source_url": source_url, - "sha256": sha256, - "size_bytes": size_bytes, - "fetched_at": fetched_at, - } - if r2_location is not None: - file_entry["storage"] = {"r2": r2_location.to_dict()} - payload["files"][year] = file_entry + if payload.get("files") is None: + # setdefault keeps an explicit null (a bare ``files:`` line); the + # entry below needs a mapping to record into. + payload["files"] = {} + release = kind == MICRODATA_RELEASE_KIND + + key, existing_value, recorded_spec, index = _select_vintage_entry( + payload, + manifest_path=manifest_path, + year=year, + filename=filename, + kind=kind, + ) + recorded_storage = _recorded_storage(recorded_spec) + identity = _recorded_identity(recorded_spec, manifest_path=manifest_path, year=key) + _assert_expected_identity( + expected, + manifest_path=manifest_path, + year=key, + filename=filename, + source_url=source_url, + sha256=sha256, + size_bytes=size_bytes, + ) + new_r2 = r2_location.to_dict() if r2_location is not None else None + # Different bytes under the same vintage, or the same bytes under another + # name: fetch_source_artifact refuses both before the read; the guard is + # repeated here so no caller can reach a false-provenance write. + _assert_recorded_identity_holds_these_bytes( + identity, + manifest_path=manifest_path, + year=key, + filename=filename, + sha256=sha256, + size_bytes=size_bytes, + r2_bucket=(new_r2 or {}).get("bucket") or default_r2_raw_bucket(), + record_revision=record_revision, + ) + holds = identity is not None and identity.holds(sha256=sha256, filename=filename) + if holds and identity.r2 is not None: + # A recorded storage.r2 block for these exact bytes is historical + # truth: archived witness records pin raw R2 URLs by hash. Re-fetching + # under a renamed bucket copies bytes; it does not restate where the + # bytes were first published (PolicyEngine/chronicle#143, mechanism 3). + storage = {**recorded_storage, "r2": _recorded_r2(recorded_spec)} + elif identity is not None and not holds: + storage = _superseding_storage( + recorded_spec, + recorded_r2=identity.r2, + new_r2=new_r2, + superseded_at=fetched_at, + ) + elif new_r2 is not None: + storage = {**recorded_storage, "r2": new_r2} + else: + storage = dict(recorded_storage) + + # Access is written explicitly on every entry this command touches, so a + # manifest never relies on the inferred ``public`` default once rewritten. + file_entry: dict[str, Any] = {"filename": filename} + if release: + file_entry["access"] = access + file_entry["licence"] = licence + if licence_evidence: + file_entry["licence_evidence"] = dict(licence_evidence) + file_entry["vintage"] = vintage or recorded_spec.get("vintage") + file_entry["sha256"] = sha256 + file_entry["size_bytes"] = size_bytes + file_entry["source_url"] = source_url + file_entry["fetched_at"] = fetched_at + # Chronicle fetched and hashed these bytes itself. + file_entry["verified_at"] = fetched_at[:10] + file_entry["hash_source"] = HASH_SOURCE_CHRONICLE_FETCH + file_entry["attested_by"] = CHRONICLE_ATTESTER + else: + file_entry["source_url"] = source_url + file_entry["access"] = access + if licence: + file_entry["licence"] = licence + if vintage: + file_entry["vintage"] = vintage + file_entry["sha256"] = sha256 + file_entry["size_bytes"] = size_bytes + file_entry["fetched_at"] = fetched_at + for field, value in recorded_spec.items(): + if field not in _FETCH_OWNED_FIELDS and field not in file_entry: + file_entry[field] = value + # An entry that has no storage to record carries no empty block: a + # revision over a never-published entry supersedes nothing. + if storage: + file_entry["storage"] = storage + + if existing_value is None: + payload["files"][key] = [file_entry] if release else file_entry + elif isinstance(existing_value, dict): + if not release: + # A publisher table holds one file per vintage; a rename under + # --record-revision supersedes that one entry, and the superseded + # key in storage.previous_r2 keeps the old name. + payload["files"][key] = file_entry + elif filename_key(existing_value.get("filename")) == filename_key(filename): + payload["files"][key] = [file_entry] + else: + payload["files"][key] = [existing_value, file_entry] + else: + entries = list(existing_value) + if index is not None: + entries[index] = file_entry + else: + entries.append(file_entry) + payload["files"][key] = entries + # A release's bytes never enter the package directory, so the directory + # may not exist yet when its manifest is first written. + manifest_path.parent.mkdir(parents=True, exist_ok=True) manifest_path.write_text( - yaml.safe_dump(payload, sort_keys=False), + yaml.safe_dump(payload, sort_keys=False, allow_unicode=True), encoding="utf-8", ) +def _with_declared_kind(payload: dict[str, Any], kind: str) -> dict[str, Any]: + """Return ``payload`` declaring ``kind``, placed after its identity keys.""" + if "kind" in payload: + payload["kind"] = kind + return payload + ordered: dict[str, Any] = {} + inserted = False + for field, value in payload.items(): + ordered[field] = value + if field == "package_id" and not inserted: + ordered["kind"] = kind + inserted = True + if not inserted: + ordered["kind"] = kind + return ordered + + +def _assert_no_hash_only_entry( + manifest: dict[str, Any], + manifest_path: Path, + filename: str, +) -> None: + """Refuse to fetch bytes over an existing hash-only registration. + + The write target is a path in the package directory, so the search spans + every vintage rather than the requested one: a licensed release registered + under one year must not be fetched into the tree under another. Names are + compared as resolved, case-folded bare filenames, so no alias of a + registered name -- ``./adult.tab``, ``ADULT.TAB`` -- slips past. + """ + wanted = filename_key(filename) + for key, _index, spec in iter_manifest_entries(manifest): + if not isinstance(spec, dict) or spec.get("filename") is None: + continue + if filename_key(spec.get("filename")) != wanted: + continue + declared = spec.get("access") + if declared is None: + # A release entry without an access class is reported by the + # strict manifest validation that follows; it is never read as + # public here. + continue + try: + access = normalize_access(declared) + except ManifestAccessError: + raise ManifestAccessError( + f"{manifest_path} registers {spec.get('filename')!r} for " + f"{key!r} with access={declared!r}, which is not one of " + f"{list(ACCESS_CLASSES)}. Its bytes must not enter a Chronicle " + "store until the registration is fixed." + ) from None + if is_hash_only(access): + requested = ( + f" (requested as {filename!r})" + if spec.get("filename") != filename + else "" + ) + raise ManifestAccessError( + f"{manifest_path} registers {spec.get('filename')!r}{requested} " + f"for {key!r} as access={access!r}. Its bytes must not enter a " + "Chronicle store; keep the hash-only registration." + ) + + def _upload_r2_object( location: ArtifactStorageLocation, local_path: Path, @@ -1114,23 +2577,124 @@ def _publish_raw_manifest_entry( year: Any, spec: Any, *, + manifest: dict[str, Any] | None = None, + kind: str | None = None, r2_bucket: str, r2_prefix: str, wrangler_command: str, + skip_hash_only: bool = False, + staging_dir: str | Path | None = None, ) -> tuple[RawArtifactPublishEntry, dict[str, Any] | None]: errors: list[str] = [] + if isinstance(spec, ListSpecRejected): + return ( + RawArtifactPublishEntry( + manifest_path=str(manifest_path), + source_id=source_id, + package_id=package_id, + year=str(year), + filename="", + local_path=str(manifest_path.parent), + sha256=None, + size_bytes=None, + r2_location=None, + upload=None, + errors=("list_file_spec_requires_microdata_release_kind",), + ), + None, + ) if not isinstance(spec, dict): spec = {} errors.append("malformed_file_spec") filename = str(spec.get("filename") or "") - artifact_path = manifest_path.parent / filename + if filename and not is_bare_filename(filename): + # Refuse before resolving the path: a name that is not bare could + # address a file outside the package directory, or one another entry + # already governs. + return ( + RawArtifactPublishEntry( + manifest_path=str(manifest_path), + source_id=source_id, + package_id=package_id, + year=str(year), + filename=filename, + local_path=str(manifest_path.parent), + sha256=None, + size_bytes=None, + r2_location=None, + upload=None, + errors=(f"non_canonical_filename:{filename}",), + ), + None, + ) + kind = kind or safe_manifest_kind(manifest, manifest_path=manifest_path)[0] + access = safe_entry_access(spec) + if is_hash_only(access): + # Refuse before touching bytes: no Chronicle store holds a licensed or + # restricted artifact, so there is nothing here to upload. The entry is + # still validated, because bytes on disk or a recorded R2 key are + # contract violations that --skip-hash-only must not hide. + hash_only_errors = list( + validate_file_entry( + spec, + kind=kind, + manifest=manifest, + local_file_exists=(manifest_path.parent / filename).exists() + if filename + else False, + ) + ) + if not skip_hash_only: + hash_only_errors.insert(0, f"hash_only_access_refuses_bytes:{access}") + return ( + RawArtifactPublishEntry( + manifest_path=str(manifest_path), + source_id=source_id, + package_id=package_id, + year=str(year), + filename=filename, + local_path=str(manifest_path.parent / filename), + sha256=spec.get("sha256"), + size_bytes=spec.get("size_bytes"), + r2_location=None, + upload=None, + errors=tuple(dict.fromkeys(hash_only_errors)), + skipped=f"{HASH_ONLY_SKIP_PREFIX}{access}", + ), + None, + ) + errors.extend( + validate_file_entry( + spec, + kind=kind, + manifest=manifest, + local_file_exists=(manifest_path.parent / filename).exists() + if filename + else False, + ) + ) + release = kind == MICRODATA_RELEASE_KIND sha256_expected = spec.get("sha256") + if release and filename and sha256_expected: + # Public microdata is never read from beside its manifest: its bytes + # are staged outside the tree (validate_file_entry reports a copy in + # the tree as bytes_present_for_microdata_release_entry). + artifact_path = microdata_staging_path( + staging_dir=staging_dir, + source_id=source_id, + package_id=package_id, + year=year, + sha256=str(sha256_expected), + filename=filename, + ) + else: + artifact_path = manifest_path.parent / filename sha256_actual = None size_bytes = None if not filename: errors.append("missing_filename") elif not artifact_path.exists(): - errors.append("missing_file") + errors.append("staged_bytes_missing" if release else "missing_file") else: content = artifact_path.read_bytes() sha256_actual = hashlib.sha256(content).hexdigest() @@ -1138,7 +2702,10 @@ def _publish_raw_manifest_entry( if sha256_expected and sha256_actual != sha256_expected: errors.append("checksum_mismatch") - if errors: + def refuse(reason: str | None = None) -> tuple[RawArtifactPublishEntry, None]: + """Report the entry unpublished, with nothing uploaded or rewritten.""" + if reason is not None: + errors.append(reason) return ( RawArtifactPublishEntry( manifest_path=str(manifest_path), @@ -1156,6 +2723,34 @@ def _publish_raw_manifest_entry( None, ) + if errors: + return refuse() + + try: + recorded_r2 = _validated_recorded_r2( + spec, manifest_path=manifest_path, year=year + ) + except SourceArtifactManifestError as error: + # A block that does not name one object cannot be treated as history, + # and publishing under it would ship whichever field was read. + return refuse(f"recorded_r2_locator_invalid:{error}") + if recorded_r2 is not None and (recorded_r2.sha256, recorded_r2.filename) != ( + sha256_actual or "", + Path(filename).name, + ): + # The recorded object is addressed by different bytes, so it is not + # this file's history. Uploading anyway would either publish under a + # key that misdescribes its content or restate a URI that belongs to + # the superseded bytes. Registering a publisher revision is + # `fetch-artifact --record-revision`, not a publish-time rewrite. + return refuse( + "recorded_r2_identity_mismatch:" + f"recorded_sha256={recorded_r2.sha256}:" + f"recorded_filename={recorded_r2.filename}:" + f"local_sha256={sha256_actual}:" + f"local_filename={Path(filename).name}" + ) + location = ArtifactStorageLocation( provider="r2", bucket=r2_bucket, @@ -1169,14 +2764,15 @@ def _publish_raw_manifest_entry( package_path=manifest_path, ), ) - storage = spec.get("storage") if isinstance(spec.get("storage"), dict) else {} - recorded_r2 = storage.get("r2") if isinstance(storage.get("r2"), dict) else {} - recorded_key = recorded_r2.get("key") - if recorded_key and recorded_key != location.key: - errors.append( - "recorded_r2_key_disagrees_with_country_prefix:" - f"recorded={recorded_key}:expected={location.key}" - ) + recorded_bucket = recorded_r2.bucket if recorded_r2 is not None else None + if recorded_r2 is not None and recorded_bucket != location.bucket: + # The recorded bucket is preserved history and, per the identity check + # above, its object holds exactly these bytes: the artifact is already + # published. Restating it under the configured bucket would rewrite + # where the bytes were first published (a backfill copy is not a + # restatement), so the entry is reported as skipped with nothing + # uploaded or rewritten. After the bucket-default flip every entry + # published before it takes this path, and the sweep stays green. return ( RawArtifactPublishEntry( manifest_path=str(manifest_path), @@ -1187,12 +2783,26 @@ def _publish_raw_manifest_entry( local_path=str(artifact_path), sha256=sha256_actual, size_bytes=size_bytes, - r2_location=None, + r2_location=ArtifactStorageLocation( + provider="r2", + bucket=recorded_r2.bucket, + key=recorded_r2.key, + ), upload=None, - errors=tuple(errors), + errors=(), + skipped=( + "recorded_r2_bucket_is_preserved_history:" + f"recorded={recorded_bucket}:requested={location.bucket}" + ), ), None, ) + recorded_key = recorded_r2.key if recorded_r2 is not None else None + if recorded_key and recorded_key != location.key: + return refuse( + "recorded_r2_key_disagrees_with_country_prefix:" + f"recorded={recorded_key}:expected={location.key}" + ) upload = _upload_r2_object( location, artifact_path, @@ -1236,19 +2846,85 @@ def _inventory_entry( manifest_path: Path, year: Any, spec: Any, + *, + manifest: dict[str, Any] | None = None, + kind: str | None = None, + staging_dir: str | Path | None = None, ) -> ArtifactInventoryEntry: errors: list[str] = [] + original_spec = spec + if isinstance(spec, ListSpecRejected): + return ArtifactInventoryEntry( + manifest_path=str(manifest_path), + year=str(year), + filename="", + local_path=str(manifest_path.parent), + exists=False, + sha256_expected=None, + sha256_actual=None, + size_bytes=None, + source_url=None, + r2=None, + errors=("list_file_spec_requires_microdata_release_kind",), + ) if not isinstance(spec, dict): spec = {} errors.append("malformed_file_spec") filename = str(spec.get("filename") or "") - artifact_path = manifest_path.parent / filename - exists = bool(filename) and artifact_path.exists() + kind = kind or safe_manifest_kind(manifest, manifest_path=manifest_path)[0] + bare = bool(filename) and is_bare_filename(filename) + # A name that is not bare is reported by validate_file_entry and never + # resolved to a path, which could lie outside the package directory. + in_tree = bare and (manifest_path.parent / filename).exists() + access = safe_entry_access(spec) + hash_only = is_hash_only(access) + release = kind == MICRODATA_RELEASE_KIND + errors.extend( + validate_file_entry( + original_spec, + kind=kind, + manifest=manifest, + local_file_exists=in_tree, + ) + ) sha256_expected = spec.get("sha256") + if release and not hash_only and bare and sha256_expected: + artifact_path = microdata_staging_path( + staging_dir=staging_dir, + source_id=str((manifest or {}).get("source_id") or ""), + package_id=str((manifest or {}).get("package_id") or ""), + year=year, + sha256=str(sha256_expected), + filename=filename, + ) + exists = artifact_path.exists() + else: + artifact_path = ( + manifest_path.parent / filename if bare else manifest_path.parent + ) + exists = in_tree sha256_actual = None - size_bytes = None + size_bytes = spec.get("size_bytes") if hash_only or release else None if not filename: errors.append("missing_filename") + elif not bare: + pass + elif hash_only: + # A licensed or restricted registration is identity only: Chronicle + # never holds the bytes, so a missing local file is the correct state. + pass + elif release: + # A public release is archived, not committed: its registration is + # complete once the raw bucket records the object. Staged bytes are + # transient and checked when present. + if recorded_r2(spec) is None: + errors.append("r2_object_not_recorded") + if exists: + content = artifact_path.read_bytes() + sha256_actual = hashlib.sha256(content).hexdigest() + size_bytes = len(content) + if sha256_expected and sha256_actual != sha256_expected: + errors.append("checksum_mismatch") elif not exists: errors.append("missing_file") else: @@ -1257,8 +2933,7 @@ def _inventory_entry( size_bytes = len(content) if sha256_expected and sha256_actual != sha256_expected: errors.append("checksum_mismatch") - storage = spec.get("storage") if isinstance(spec, dict) else None - r2 = storage.get("r2") if isinstance(storage, dict) else None + r2 = recorded_r2(spec) return ArtifactInventoryEntry( manifest_path=str(manifest_path), year=str(year), @@ -1269,13 +2944,16 @@ def _inventory_entry( sha256_actual=sha256_actual, size_bytes=size_bytes, source_url=spec.get("source_url"), - r2=r2, - errors=tuple(errors), + r2=dict(r2) if r2 is not None else None, + errors=tuple(dict.fromkeys(errors)), + access=access, + licence=spec.get("licence"), + hash_only=hash_only, ) def _derived_artifact_kind(artifact_name: str) -> str: - if artifact_name in {"ledger.db", "ledger.db"}: + if artifact_name in CHRONICLE_DB_FILENAMES: return "sqlite_database" if artifact_name.endswith(".jsonl"): return "jsonl" diff --git a/chronicle/cli.py b/chronicle/cli.py index b9098d7f..bf032e5f 100644 --- a/chronicle/cli.py +++ b/chronicle/cli.py @@ -26,6 +26,7 @@ def main() -> None: ["plan-pe-sources"], ["publish-derived"], ["publish-raw"], + ["register-artifact"], ["scaffold-package"], ["validate-concept-alignments"], ["validate-package"], diff --git a/chronicle/consumer_contract.py b/chronicle/consumer_contract.py index a1c78325..17315a13 100644 --- a/chronicle/consumer_contract.py +++ b/chronicle/consumer_contract.py @@ -468,6 +468,38 @@ def validate_consumer_fact_contract( ) +def _r2_uri_parts(uri: str) -> tuple[str, str]: + """Split an ``r2://bucket/key`` URI into its bucket and key.""" + if not uri.startswith("r2://"): + return "", "" + bucket, _, key = uri[len("r2://") :].partition("/") + return bucket, key + + +# The marker a downstream target row carries in its source_record_id. It moves +# with the rename window: producers write `ledger_derived` today and +# `chronicle_derived` once they migrate (PolicyEngine/chronicle#143, mechanism +# 3), so the boundary has to reject both spellings identically or the guard +# stops firing the moment a producer renames. +DERIVED_SOURCE_RECORD_SUFFIXES = frozenset({"ledger_derived", "chronicle_derived"}) + + +def _is_derived_source_record_id(source_record_id: str) -> bool: + """Whether a source_record_id marks a downstream derived target row.""" + _, separator, suffix = source_record_id.rpartition(".") + return bool(separator) and suffix in DERIVED_SOURCE_RECORD_SUFFIXES + + +def _points_at_derived(bucket: str, key: str) -> bool: + """Whether an R2 bucket/key pair addresses derived build output. + + Matched on shape rather than on the ledger-era bucket names, so the guard + keeps firing once the buckets are renamed (PolicyEngine/chronicle#143, + mechanism 3). + """ + return bucket.endswith("-derived") or key.startswith("derived/") + + def _derived_source_provenance_issue(fact: AggregateFact) -> str | None: """Return a boundary error if a fact is a downstream target derivation.""" source = fact.source @@ -489,26 +521,20 @@ def _derived_source_provenance_issue(fact: AggregateFact) -> str | None: "itself. Target construction, aging, and reconciliation belong in " "Microcosm." ) - if source_file.startswith("ledger-derived:"): + source_file_bucket, bucket_separator, _ = source_file.partition(":") + if bucket_separator and source_file_bucket.endswith("-derived"): return ( "Chronicle consumer facts must cite raw publisher artifacts. Derived " "target-construction artifacts belong in Microcosm." ) - if ( - raw_r2_bucket.endswith("-derived") - or raw_r2_key.startswith("derived/") - or raw_r2_uri.startswith( - ( - "r2://ledger-derived/", - "r2://ledger-raw/derived/", - ) - ) + if _points_at_derived(raw_r2_bucket, raw_r2_key) or _points_at_derived( + *_r2_uri_parts(raw_r2_uri) ): return ( "Chronicle consumer facts must point at raw source artifacts, not " "derived build artifacts." ) - if source_record_id.endswith(".ledger_derived"): + if _is_derived_source_record_id(source_record_id): return ( "Chronicle source_record_id must identify a publisher-backed row, not " "a downstream derived target row." diff --git a/chronicle/database.py b/chronicle/database.py index 3f18ef78..5297db16 100644 --- a/chronicle/database.py +++ b/chronicle/database.py @@ -42,6 +42,15 @@ LEDGER_DB_SCHEMA_VERSION = schema_id("relational", Epoch.LEDGER) +# New suite outputs write chronicle.db. Existing builds wrote ledger.db and are +# still read and published unchanged, so the legacy name stays accepted for +# inference and artifact classification (PolicyEngine/chronicle#143, +# mechanism 3). The relational schema id above is a frozen machine surface that +# migrates with the epoch lane, not with this rename. +CHRONICLE_DB_FILENAME = "chronicle.db" +LEGACY_CHRONICLE_DB_FILENAME = "ledger.db" +CHRONICLE_DB_FILENAMES = (CHRONICLE_DB_FILENAME, LEGACY_CHRONICLE_DB_FILENAME) + @dataclass(frozen=True) class ChronicleDbBuildReport: diff --git a/chronicle/env.py b/chronicle/env.py new file mode 100644 index 00000000..0d4bbbad --- /dev/null +++ b/chronicle/env.py @@ -0,0 +1,161 @@ +"""Environment configuration for the Chronicle rename window. + +Chronicle's operational stores migrate by dual-run (PolicyEngine/chronicle#143, +mechanism 3): every configuration variable gets a ``CHRONICLE_``-prefixed name +that is read first, while the ledger-era ``LEDGER_`` and +``POLICYENGINE_LEDGER_`` names keep working behind a deprecation warning. That +window lets downstream publish flows migrate on their own schedule instead of +breaking the moment Chronicle ships a rename. + +Names that carry none of those three prefixes are read literally: this helper +renames the ledger-era surface, not every PolicyEngine variable. +""" + +from __future__ import annotations + +import os +from typing import TypeVar +import warnings + +__all__ = [ + "CHRONICLE_ENV_PREFIX", + "CHRONICLE_SCHEMA_ENV", + "ChronicleEnvDeprecationWarning", + "DEFAULT_CHRONICLE_SCHEMA", + "LEGACY_ENV_PREFIXES", + "default_chronicle_schema", + "env_flag", + "env_names", + "env_value", + "reset_env_deprecation_state", +] + +CHRONICLE_ENV_PREFIX = "CHRONICLE_" + +# Ordered most specific first so prefix stripping is unambiguous. +LEGACY_ENV_PREFIXES = ("POLICYENGINE_LEDGER_", "LEDGER_") + +TRUTHY_ENV_VALUES = frozenset({"1", "true", "yes", "on"}) + +CHRONICLE_SCHEMA_ENV = "CHRONICLE_SCHEMA" + +# The hosted Postgres schema is still named "ledger". Renaming the schema value +# is a later slice of PolicyEngine/chronicle#143, coordinated with the CI +# writers that already target it; only the variable that overrides the name has +# moved to the chronicle prefix. +DEFAULT_CHRONICLE_SCHEMA = "ledger" + + +class ChronicleEnvDeprecationWarning(FutureWarning): + """A ledger-era environment variable supplied a Chronicle setting. + + Subclasses :class:`FutureWarning` rather than :class:`DeprecationWarning` + so the notice reaches operators running the CLI, who are the people who + have to move the variable. ``DeprecationWarning`` is silenced by default + outside ``__main__``. + """ + + +_Default = TypeVar("_Default") + +_WARNED_LEGACY_NAMES: set[str] = set() + + +def _env_suffix(name: str) -> str | None: + """Return the rename-window suffix of ``name``, or None if it has none.""" + for prefix in (CHRONICLE_ENV_PREFIX, *LEGACY_ENV_PREFIXES): + if name.startswith(prefix) and len(name) > len(prefix): + return name[len(prefix) :] + return None + + +def env_names(name: str) -> tuple[str, ...]: + """Return the lookup order for ``name``. + + The chronicle-preferred name comes first, then the ledger-era names that + remain accepted during the migration window. A name outside the rename + window is returned unchanged, as its own single-element lookup order. + """ + suffix = _env_suffix(name) + if suffix is None: + return (name,) + return ( + f"{CHRONICLE_ENV_PREFIX}{suffix}", + *(f"{prefix}{suffix}" for prefix in LEGACY_ENV_PREFIXES), + ) + + +def _warn_legacy(found: str, preferred: str) -> None: + """Warn once per process that a ledger-era variable supplied a value. + + ``stacklevel=4`` walks out through :func:`_first_set` and its public + wrapper so the notice points at the code that asked for the setting. + """ + if found in _WARNED_LEGACY_NAMES: + return + _WARNED_LEGACY_NAMES.add(found) + warnings.warn( + f"{found} is a ledger-era Chronicle environment variable; " + f"set {preferred} instead. The old name is still honored during the " + "Chronicle rename window and will be removed once consumers migrate.", + ChronicleEnvDeprecationWarning, + stacklevel=4, + ) + + +def reset_env_deprecation_state() -> None: + """Forget which legacy names have already warned. Test-support hook.""" + _WARNED_LEGACY_NAMES.clear() + + +def _first_set(names: tuple[str, ...]) -> str | None: + """Return the first set value across ``names``, warning on a legacy hit. + + Both public readers call this at the same stack depth so the deprecation + warning is always attributed to their caller, not to this module. + """ + for name in names: + candidates = env_names(name) + preferred = candidates[0] + for candidate in candidates: + value = os.environ.get(candidate) + if value: + if candidate != preferred: + _warn_legacy(candidate, preferred) + return value + return None + + +def env_value(*names: str, default: _Default = None) -> str | _Default: + """Read the first set value across ``names``, chronicle-preferred first. + + Each name is expanded through :func:`env_names`, so a caller can pass the + chronicle name and still pick up a value set under a ledger-era name. + Empty values are treated as unset, matching the helpers this replaces. + """ + value = _first_set(names) + return default if value is None else value + + +def default_chronicle_schema() -> str: + """Resolve the Chronicle schema: ``$CHRONICLE_SCHEMA``, else the default. + + Every reader of the setting goes through this function so the lookup ladder + and the default have one home. It resolves at call time rather than at + import: a module-level constant binds whatever the shell held when the + module was first imported, which for a library means an arbitrary moment + the caller cannot control, and for the test suite means collection. + """ + return env_value(CHRONICLE_SCHEMA_ENV, default=DEFAULT_CHRONICLE_SCHEMA) + + +def env_flag(*names: str) -> bool: + """Return whether the first set value across ``names`` reads as true. + + The chronicle-preferred name wins even when it reads false, so an operator + who has migrated can turn a flag off without unsetting the legacy name. + """ + value = _first_set(names) + if value is None: + return False + return value.strip().lower() in TRUTHY_ENV_VALUES diff --git a/chronicle/grandfathered_manifests.py b/chronicle/grandfathered_manifests.py new file mode 100644 index 00000000..1e9de9ee --- /dev/null +++ b/chronicle/grandfathered_manifests.py @@ -0,0 +1,554 @@ +"""Manifests that predate the explicit-kind rule, frozen at their pre-rule bytes. + +Every manifest created or modified after ``docs/adr-chronicle-raw-microdata- +identity.md`` declares ``kind``: ``publisher_table`` or ``microdata_release``. +The manifests listed here existed before that rule and declare none. They read +as ``publisher_table`` only while their bytes still match the digest frozen +here: a grandfathered manifest that is modified in any way -- by +``fetch-artifact``, which always writes ``kind``, or by hand -- leaves the +freeze and must declare its kind. A kindless manifest that is not on this list +is an error, never a publisher table by default. + +The list is frozen at the freeze commit: entries are removed once a manifest +declares its kind, and never added. ``tests/test_chronicle_manifest_kind.py`` +checks that every kindless manifest in the tree is listed here with its frozen +digest, so a new kindless manifest cannot land. +""" + +from __future__ import annotations + +import hashlib +from collections.abc import Mapping +from pathlib import PurePosixPath +from types import MappingProxyType +from typing import Any + +__all__ = [ + "GRANDFATHERED_KINDLESS_MANIFESTS", + "grandfathered_manifest_key", + "is_grandfathered_manifest", + "manifest_digest", +] + +#: Repository-relative manifest path -> SHA-256 of the manifest bytes at the +#: freeze. +GRANDFATHERED_KINDLESS_MANIFESTS: Mapping[str, str] = MappingProxyType( + { + "db/data/bea/nipa_total_wages_salaries/manifest.yaml": ( + "765f391487a698506647a2bb60823f6742046abae0b8bed62874c6f98bd50d93" + ), + "db/data/bea/regional_personal_income_state/manifest.yaml": ( + "a1283e1ccc574fbeb9710abd409d41c4b735b75c674a34bcc58e6df755278e56" + ), + "db/data/bfp/economic_outlook_2026_06/manifest.yaml": ( + "0a46d70e52cb8ea3a51af5e104e378e257ee1bc8e7c03aabf5a2286b09024e7c" + ), + "db/data/cbo/individual_income_tax_receipts_2026_02/manifest.yaml": ( + "c24431df9284fef70334e51b0d9eaa03a47553f777de0f78d6430db6d7e79954" + ), + "db/data/cbo/revenue_projections_income_by_source_2026_02/manifest.yaml": ( + "acbd1ff5577b638cea9dcbceba6a3fce7911169ffce661410b4e72adcc0d5b76" + ), + "db/data/census/acs_s0101_district_2024/manifest.yaml": ( + "41f1df4f9e5377993ef8bc6be27f26a690d6d9bee2bb73a4c0b5212bd9e9fdb1" + ), + "db/data/census/acs_s0101_national_2024/manifest.yaml": ( + "a5bce23e75ec58ca86247a52ac03ff793924a66513b1c8f04cec45da001e1037" + ), + "db/data/census/acs_s0101_state_2024/manifest.yaml": ( + "d38f64b5ad62b05ed6c3e3c6b4d803f03a4543611d0a5f9bb4459f17785713e7" + ), + "db/data/census/acs_s2201_district_2024/manifest.yaml": ( + "ce5062818e45345a95a5d87da5415f426a3e114b128cab5c62cd00f8f0c999f6" + ), + "db/data/census/b01001_female_15_44_2023/manifest.yaml": ( + "d5bf52beefdeea84998391c6ea2905b6fd89bcd46dcbd9e7e7d539183d45f8d4" + ), + "db/data/census/pep_2024_age_sex/manifest_national_source_package.yaml": ( + "4392052b88202e9799712b46fa291ec3f396c3b0c84f25e1f6f8bbc6029b4678" + ), + "db/data/census/pep_2024_age_sex/manifest_state_source_package.yaml": ( + "2e0e173301250dfd7b339b2d6a6b4a367c1536c42a84567acb705dfa98a6bb5f" + ), + "db/data/census/pep_county_2024/manifest.yaml": ( + "7622c3dbaab7a1eafce5682c072cce9a79d0d60edc80936ce0f76b127c8f9d55" + ), + "db/data/census/population_projections_2023/manifest.yaml": ( + "fbc68ea1bd58126e5122de51ce1470eaec1001b69328fb959a1d84d859d71ca1" + ), + "db/data/census/stc_individual_income_tax/manifest.yaml": ( + "b6146e6ce891dc26306f0ad550e608e63118ae33029285974158ee44fdd5085f" + ), + "db/data/cms_aca/effectuated_enrollment_2022/manifest.yaml": ( + "d6fb650034b2d1ef4b783f92fae9f31422534ed2c391add978652d9f4455e322" + ), + "db/data/cms_aca/oep_state_level/manifest.yaml": ( + "671bb0ae03eec9317ba17fd2adcdba6b289cb12f1f2fdc776bb3274adb234a3c" + ), + "db/data/cms_aca/oep_state_level_2022/manifest.yaml": ( + "0b0a9c41400b77504a60ab8e07d1559360f93b1af946e1fbcf4c6418646aab1a" + ), + "db/data/cms_aca/oep_state_level_2025/manifest.yaml": ( + "9de293a350360e0d1804ad1d1fe9fca73f71ca613fd9094db50530e7e3f19d4b" + ), + "db/data/cms_medicaid/chip_monthly_enrollment_dataset/manifest.yaml": ( + "8fa93cf69e7c4466d7c73680ca0a287245550c9ffdfbcfde631aa98fdf38455a" + ), + "db/data/cms_medicare/medicare_trustees_report_2025/manifest.yaml": ( + "8fa057bcf1bcb630bb4e7ce8c32a199378d57886ea7be85f0846beb1cebac1e3" + ), + "db/data/cms_nhe/historical_service_source/manifest.yaml": ( + "2fe570abe9079b013c03045b5714cb81af68c4d4b747494ba457f5097bc368d5" + ), + "db/data/cms_nhe/historical_service_source/manifest_source_package.yaml": ( + "319e4e352ebbfd7af32b6c9f67811854eeb7615c23efa02c14cd2d284538face" + ), + "db/data/cms_nhe/table_24/manifest.yaml": ( + "d6c70a307040d5fd4e99f4ecec6a97a167ac71d17bc2d42cf3ce27bd8ae648d8" + ), + "db/data/dfc_ni/uc_statistics_may_2026/manifest.yaml": ( + "b8ed0c6a85339195940f75148152f0f924f9a9500bbcb2b26b35c90d712d1591" + ), + "db/data/dfe/funded_early_education_childcare_2026/manifest.yaml": ( + "6cc43ea3d3e16ec4845e42c2972b415ae2db7c07d035ed93b1db3ef4562eb67c" + ), + "db/data/dft/bus0415_fares_index_2026/manifest.yaml": ( + "6453b9a65651239eddc97a0b41114667aafb39c2e9c183773261a870a42dccf0" + ), + "db/data/dft/bus05i_revenue_support_2025/manifest.yaml": ( + "4cea963c6ed0177ea4846279bf32242cc96c0e67530a3bdee4a4970ff6826d96" + ), + "db/data/dft/nts0705_local_bus_trips_2024/manifest.yaml": ( + "6bd4ecba81b1e0482356a1bc9e79842743d5f0a07e55476822f3ff946615a7fb" + ), + "db/data/dft/nts_vehicle_ownership_2024/manifest.yaml": ( + "46c35b4ac7117f41fc1524a70a7712709904f265e5c27e1ab6778b3b18923d26" + ), + "db/data/dwp/benefit_cap_november_2025/manifest.yaml": ( + "92e48e42d94d2dc7da18c39cb061c17c0a15f935de8f9f552f56cee4aac54323" + ), + "db/data/dwp/benefit_statistics_february_2026/manifest.yaml": ( + "dd07920ba85e98726da226366d6d93854ea0b06338f587ed3962db353fc78a8b" + ), + "db/data/dwp/pip_daily_living_foi_2025/manifest.yaml": ( + "4ee8292dbc7c9d539c9f86e4c1926c7cc044b071006252622f8716f79418839a" + ), + "db/data/dwp/uc_childcare_element_march_2021_august_2025/manifest.yaml": ( + "eac3f1eeb946b68b9bcd2460fc621c4f60956808b332b04743c78071beed8132" + ), + "db/data/dwp/uc_deductions_march_2025_february_2026/manifest.yaml": ( + "2c148e2981347334c537f12a8e6caea11ccae543678271f76308af70594b2081" + ), + "db/data/dwp/uc_households_by_constituency_children_may_2025/manifest.yaml": ( + "073f66903927591d0e3ad5f36eb5cc9ccc573a28b379c600c9d74ac1a189a15d" + ), + "db/data/dwp/uc_households_by_constituency_may_2025/manifest.yaml": ( + "a846fc14a57e82b3eb71d2d72afa21e57ed05e9d922bb4dd04fe73022859fa4a" + ), + "db/data/dwp/uc_households_by_local_authority_may_2025/manifest.yaml": ( + "ff95bda95cc1e56f4993b75c3f218373c41c77097f5e804da28f1e16d5cfbe81" + ), + "db/data/dwp/uc_households_carer_entitlement_april_december_2025/manifest.yaml": ( + "67729d45580419b84b85dc679e45869dbc2f3dafda46cec53d5a531543f0a4d3" + ), + "db/data/dwp/uc_households_children_april_december_2025/manifest.yaml": ( + "9b44c346294f84fcfb5e199f8b555d052dae828cecb7de2a633a46c34d215b62" + ), + "db/data/dwp/uc_households_family_type_april_december_2025/manifest.yaml": ( + "6a55f6c3219c1f8bbea56800bfa9a324f61bec446008bc412949c4f66f0ea812" + ), + "db/data/dwp/uc_households_housing_entitlement_april_december_2025/manifest.yaml": ( + "867b596a4224ab1e73a03c6951106f49e0eb1d5821aa397a292e8bafaf9c3dba" + ), + "db/data/dwp/uc_households_lcwra_entitlement_april_december_2025/manifest.yaml": ( + "6a0164ef778d320dbbe61e414a68066bc1b64fe42ebf5bbb6571a4d47fd7f1af" + ), + "db/data/dwp/uc_payment_distribution_may_2025/manifest.yaml": ( + "fb54f10c9829ab08b1304cf50644dff46d5241cc156edb91b081becec34558cb" + ), + "db/data/dwp/uc_scotland_youngest_child_may_2025/manifest.yaml": ( + "0c0d3dcb13bdb4ad391dd7198ee3b75f4e185b0000f149092ea92b767c5cb110" + ), + "db/data/dwp/uc_two_child_limit_2025/manifest.yaml": ( + "068959db08a1a970caaafed86caa1cb48e5aafc1cbd800af98fed09c309a03d4" + ), + "db/data/eurostat/gov_10a_taxag/manifest.yaml": ( + "d023d782a0d34ab0823fdd75394b234f9d23754060f334726fb936c333170293" + ), + "db/data/eurostat/ilc_di01/manifest.yaml": ( + "7da09c7bcb936fc34672f88dffc6ce4e16b67254823b8dc2ccf76956858d0992" + ), + "db/data/eurostat/ilc_li02/manifest.yaml": ( + "2cbcdb9face89a8e66716c4aaa6167091a07fadf8aba1f12f3975e92c00d4e7c" + ), + "db/data/eurostat/nasa_10_nf_tr/manifest.yaml": ( + "627cb9b76778e8cd85062c7afe3ce4d0581497d15f0fdd048a1e2aec7910f62f" + ), + "db/data/eurostat/spr_exp_func/manifest.yaml": ( + "618e21e482f571fc882d49b848f0c57c01f01d6b9c70faa2c39baa0402750496" + ), + "db/data/federal_reserve/z1_household_net_worth_2026/manifest.yaml": ( + "ba43416d74a2d30b727625e9ce80825511b955b7a3507a5e8b6d5f786b437c5f" + ), + "db/data/fpb/economic_outlook_2026_2031_june_2026/manifest.yaml": ( + "2251eeb3287c72e558f94811122962b56a30a8071d8d10508e025ef2fb98c27e" + ), + "db/data/hhs_acf/tanf_caseload_2024/manifest.yaml": ( + "013fad141428e15fc4d29d0f61501522e20e21f278765716aec5635559bf8108" + ), + "db/data/hhs_acf/tanf_financial_2024/manifest.yaml": ( + "c8d9a2129b177ddfc33ffb7b0f4cd745b83bb18b3d195ef4327f0224dd604d54" + ), + "db/data/hhs_acf_liheap/fy2023_national_profile/manifest.yaml": ( + "53eca5fd455593231bfe15aa79584b665e413665f51ce4bbc46701a1815fea87" + ), + "db/data/hhs_acf_liheap/fy2024_national_profile/manifest.yaml": ( + "5d44f82f41f911cc213cbc545b70627612fa328ee03f7552d774af644e95d75e" + ), + "db/data/hmrc/cgt_age_2026/manifest.yaml": ( + "19ad2f017bc99c2d3dd48f638055119bc0d77621baf184ca269d045f5f1a00cd" + ), + "db/data/hmrc/cgt_country_region_2026/manifest.yaml": ( + "fe9354f7751797eefcc2a8c35d123e77b936e861db88a9d8205deec7f87076e5" + ), + "db/data/hmrc/cgt_gain_by_income_2026/manifest.yaml": ( + "10a802ead7dd1f81fd3740babc4adf292d0389ac8efed0d83bf78bc1270f47e0" + ), + "db/data/hmrc/cgt_size_of_gain_2026/manifest.yaml": ( + "e77c9f68e40ef7878918605e6348eb8f70472bdb3ac30322d8fc0380e4629b5b" + ), + "db/data/hmrc/cgt_statistics_2026/manifest.yaml": ( + "939a93e7e9f7a437ea58331e0f804408b1a4481e6229bc91f41e5685d09fa4b0" + ), + "db/data/hmrc/salary_sacrifice_reform_2029_headcounts/manifest.yaml": ( + "00e2c4d343532b946ce0c6959468554e0a9bd4059a1c7f171d0a9888f2516464" + ), + "db/data/hmrc/salary_sacrifice_relief_2024_25/manifest.yaml": ( + "ae65160f1da3bbb56681d334091c5a137cb8cd9a3066dc9e20a64fb01de3847b" + ), + "db/data/hmrc/spi_income_bands_2023_24/manifest.yaml": ( + "972496558bc9cadd1ea4607c635a75279460aab160578cff63cd2f11740c1774" + ), + "db/data/hmrc/spi_income_by_area_2023_24/manifest.yaml": ( + "55a57302bf60a2d654a2dd9511416d7654e1dd9d99fa8c9541479799f5eb4c3f" + ), + "db/data/hmrc/tax_free_childcare_march_2026/manifest.yaml": ( + "49ff7bf6aa28f21275f263c378a0619f2026bf2e3356274fe7362ad3f0a997b9" + ), + "db/data/hmrc/vat_firm_sector_targets_2024_25/manifest.yaml": ( + "a775ff5af9834b5deb0f80cf9941efd63db3f97c319969ff246fc32cac6eb881" + ), + "db/data/hmrc/vat_firm_targets_2024_25/manifest.yaml": ( + "b557e65210cfd8bd55180075af09e15a1a8b6cd8fcca693d96805e653af8936b" + ), + "db/data/ici/fact_book_table_30/manifest.yaml": ( + "7fd06be33ec2af5658f8ffea95b8d9aa8276f2cf2d569d15ff7dade754f5f307" + ), + "db/data/irs_soi/congressional_district_2022/manifest.yaml": ( + "62e1d4c8d00b0f3e3d2c57ae6accd064d8b4118788e2524369604e7cbb1a39a9" + ), + "db/data/irs_soi/county_2022/manifest.yaml": ( + "84dd9f95b7a17478cdf852ca85202be68484181b855e5c6d2a39ce0abd67cf0b" + ), + "db/data/irs_soi/filing_season_week47_2024/manifest.yaml": ( + "a58b5e8e998a0273d37c7b073ffce30e2d56723c30048a4bfb52b360462a2f12" + ), + "db/data/irs_soi/historic_table_2/manifest.yaml": ( + "941ce7849e7834e188436bec8c880f0917e0509a6fccf2565d7d6dba69ccb5bc" + ), + "db/data/irs_soi/ira_contributions/manifest_roth_source_package.yaml": ( + "2c2f5031e81358222e35ed1ccdf0d285fd49aac7f8ba90d5ef40e38856d453fa" + ), + "db/data/irs_soi/ira_contributions/manifest_traditional_source_package.yaml": ( + "e7f447dc67770be73a00c88ded5dff2b2e8569b876f244c7db3667b44ee7df61" + ), + "db/data/irs_soi/state_2022/manifest.yaml": ( + "1c8ab10d77167adce93cdc695fb2a41afb997b81001962462c96cacaa37e6575" + ), + "db/data/irs_soi/table_1_1/manifest.yaml": ( + "3877ef9629ad30fcaf19aa0b2cf75ce7797aea7598d02a7cc3af7ae945c43a51" + ), + "db/data/irs_soi/table_1_2/manifest.yaml": ( + "6e3db0b2809cced841f0acfa073dfa0304389f08753c8bdfae21c55ca9845b80" + ), + "db/data/irs_soi/table_1_4/manifest.yaml": ( + "d9b365c0e9fa17874af0eb5aa4014fe8d7edca21fa54889ad3c7a5d533a9d469" + ), + "db/data/irs_soi/table_2_1/manifest.yaml": ( + "02cff4531ddd4ab456e6b0cefd95d711f2370d13c48b1c5b1736a51bc0f5f6f3" + ), + "db/data/irs_soi/table_2_5/manifest.yaml": ( + "51246f286233be241d0fb3243efe8dfc86d3390233d56c1e49931dab063b6a10" + ), + "db/data/irs_soi/table_4_3/manifest.yaml": ( + "f0a0836f9856b2a767309577ee0f61a9c8adcb2b31a4ca9f4f45d2436818f9f6" + ), + "db/data/irs_soi/w2_statistics/manifest_2020_source_package.yaml": ( + "9f61439825fa7ea82325310e0f04d5292842c468b471632385ecd1a60133c3d3" + ), + "db/data/isc/annual_census_2023/manifest.yaml": ( + "178582171c6636a1f8b2bff88213ea1583db6849abb19556a7b088cbced8c26f" + ), + "db/data/isc/annual_census_2024/manifest.yaml": ( + "a9c03be443bc0c684a6f8be08c9cc3ae51e6d379db17a04ecccf050c34dced88" + ), + "db/data/jct/obbba_revenue_estimates_2025/manifest.yaml": ( + "12e1ae416138e6171f3f043328859de9aef4ebcfd5a1af822198859ad823b667" + ), + "db/data/jct/tax_expenditures_2024/manifest.yaml": ( + "e3216c3e1b782691ed9da6c45b9a853149a7a863a5dd4a0efe72845f9a7f1320" + ), + "db/data/jrc/euromod_be_baseline_statistics_2025/manifest.yaml": ( + "7f02b5f98607f7b69fca12e28e92e8c8cf2a091712e8b34f81a55164740b4efb" + ), + "db/data/kff/marketplace_effectuated_enrollment/manifest.yaml": ( + "0227988dea8c69843fed2d3c4c179de5e6f2c242ccb1e9eb8b6574ed12420748" + ), + "db/data/mhclg/council_tax_collection_england_2025_26/manifest.yaml": ( + "ceed04b0de8fc684be2f8856868def92d3c98ad953aa79ad7e3c44ec64ce0c6e" + ), + "db/data/mhclg/council_tax_levels_england_2026_27/manifest.yaml": ( + "f0af6b945807a6da7cefcba4eee33840ecf5ffaa189a6e6d0df24ed7d3e4d68e" + ), + "db/data/mhclg/council_tax_levels_england_summary_2025_26/manifest.yaml": ( + "0fc78b37cac572e48a1453226f012596e61a3aa690b1f6bad3c0270e4875ab44" + ), + "db/data/mhclg/ehs_weekly_housing_costs_2023_24/manifest.yaml": ( + "9420c57fec384921fea8f948f0fb84a20f7c634072aac92bae23a32982c7cd1f" + ), + "db/data/nbb/national_accounts_household_disposable_income_2024/manifest.yaml": ( + "699eb899809e669017d7cd6fdf0e19eb5ca1561228ff01145981883fcae02684" + ), + "db/data/nisra/census2021_households_lgd/manifest.yaml": ( + "6457db483b85c04378432f4f56311e5fe77852c27c35155f59435cbd5a3c2bc6" + ), + "db/data/nisra/census2021_households_pcon24/manifest.yaml": ( + "15fc9bf9ddbda812a0a961244316f601f918d66c4d0876e4de264b57f97edbd1" + ), + "db/data/nisra/census2021_tenure_lgd/manifest.yaml": ( + "c696267fdb598b43c713eb626eae3bbd20b32c23522ce87aff65ab7e10cb40da" + ), + "db/data/nisra/pcon24_population_by_age_2024/manifest.yaml": ( + "973db94fa8a7200a98eb78d642e311b2090dd1d5d1ac54c273ee8e081e4d41a5" + ), + "db/data/nrs/census2022_households_ukpc24/manifest.yaml": ( + "c6bbbc5e23dcd3e3fe61152844debd416f91111e7c8c4e9cea4b5fe9fc54c7e9" + ), + "db/data/nrs/census2022_uv404_tenure_council_area/manifest.yaml": ( + "cbe62456a64cd1b2404fd79b97a730b7400fdf7ee1eef31077cc5d1fbd7bdfea" + ), + "db/data/nrs/pcon24_population_by_age_2024/manifest.yaml": ( + "69e3a274aa93623eff741cb62a718c08c507f78cf4d67357f87d8596ede18ef0" + ), + "db/data/obr/efo_aggregates_march_2026/manifest.yaml": ( + "64dab83a0d63684a771fa05624931d6fff1b71370021e24aeeaa4db8c4a1bb87" + ), + "db/data/obr/efo_economy_march_2026/manifest.yaml": ( + "33e3f30422ea8170ab13b8828e0c7b7f9dbc26165c7dd216abffece53ae90eae" + ), + "db/data/obr/efo_expenditure_march_2026/manifest.yaml": ( + "9ae17f2ed29fab5091bfe6eea3bc3eda16f4614fd7572206041e7a4f39785d86" + ), + "db/data/obr/efo_receipts_march_2026/manifest.yaml": ( + "a0794755ea8de46cf98456b965127087e9512cd636a50ea882397f946abbfd47" + ), + "db/data/onem_rva/unemployment_2024/manifest.yaml": ( + "bcc298a74823509123de5cb4ace444c22f67516f009b2d9ade4adbae9b74fe2e" + ), + "db/data/ons/census2021_ts041_households_lad/manifest.yaml": ( + "08b60f70f9548def4f821b0326bf1a4ae1a05fafcc703900a8221cb9689d4e7b" + ), + "db/data/ons/census2021_ts041_households_pcon24/manifest.yaml": ( + "9393cf5f5c6e2ce29d40938fff52c843236a958115cba465b8d83c8d414cfa91" + ), + "db/data/ons/census2021_ts054_tenure_lad/manifest.yaml": ( + "67be94cefa57786109e29793d76bb21af374aa8325ac98c9fdbd493bd2cb7c26" + ), + "db/data/ons/families_households_2025/manifest.yaml": ( + "d9ca199ec60bcf584757065e918d20d25baa010499b7ba779814a7a79c4c2d3b" + ), + "db/data/ons/households_by_type_country_2025/manifest.yaml": ( + "11cb033bd19846de333c20f025773019db105fd3ca2b9bbc9cfffb9cb6def7c6" + ), + "db/data/ons/lad_population_by_age_2024/manifest.yaml": ( + "96d3dd78b53cf1c3b9de6e1556f535cfa491a2728ee214b6ea89ebd0aa0f73b7" + ), + "db/data/ons/mye_2023_england_regions/manifest.yaml": ( + "f7bfe957544c7b498a005c825f9448f253935ae641a3bdac604319260609e949" + ), + "db/data/ons/mye_2023_uk_countries/manifest.yaml": ( + "9a553c91a801bc1544c538e22761f4eb226eb2209a770667e9d1c902f74751b9" + ), + "db/data/ons/mye_2024_uk/manifest.yaml": ( + "d23afe667223bad9c9fa1f954ccab65d21a8af1e221a9eb50fb9e2debf0c097c" + ), + "db/data/ons/national_balance_sheet_land_2025/manifest.yaml": ( + "091412976df9d01cac8de486b380f2e7ae129a3209a09d59acf2471ad81909e5" + ), + "db/data/ons/npp_2024_uk/manifest.yaml": ( + "36bc156c49939ad023d3e57489f4d6ac94d5cc8b8b6cc81ca782ed4197ad1d62" + ), + "db/data/ons/pcon24_population_by_age_2024/manifest.yaml": ( + "f3042a9f4764d34d17ecbb6b539a2f0b736ac36e64e3328a9c980060112a84bf" + ), + "db/data/ons/pipr_private_rent_march_2026/manifest.yaml": ( + "9c8548f433803ea081fcc3529905f6a8fcd6499fe7e5887e5f674c39a90ac593" + ), + "db/data/ons/pipr_rents_by_area_june_2026/manifest.yaml": ( + "3d6d8c202aa8c40f79d79cef735aa53167f2e6383a78f4d07227ea039dbe87f5" + ), + "db/data/ons/public_sector_employment_2026/manifest.yaml": ( + "a126604b78b10e5dc6bf2019fdb118862bc23402a3f6c5b49d37ec5f4e3c4a2f" + ), + "db/data/ons/savings_interest_income/manifest.yaml": ( + "f85200aaafa452dc3f2b7e7940253c552d47f481a068f6adff63ef9a511e8de7" + ), + "db/data/ons/small_area_income_msoa_fye2023/manifest.yaml": ( + "f5b0813fdbd8ab84ed651e3442c93b034c7b1f5995aaf65b0f4b9215c53c2800" + ), + "db/data/ons/subnational_dwellings_by_tenure_2024/manifest.yaml": ( + "66926aa574b44a9b2db66fdd65f1a1d4b5a36cc976ac04c3568c7baa0bf6e423" + ), + "db/data/ons/uk_business_firm_sector_targets_2025/manifest.yaml": ( + "b5c1101e89b6c47835fb22addfae764edc2c728d35def77c701ca95343ea9f6b" + ), + "db/data/ons/uk_business_firm_targets_2025/manifest.yaml": ( + "407e08fb557132d8a9f5d7f0ce2552b1023c2b86c9da1bbb9dc1f7932a6a3c34" + ), + "db/data/onss/contributions_2024/manifest.yaml": ( + "cd8dac264b4aae0e9257f4d3f1dae8cbc61d42cb58ee4449439ef01e0c1f3c66" + ), + "db/data/opgroeien/groeipakket_caseload_2025/manifest.yaml": ( + "d74a0b9c70eee26dc349985cc231ef81e5f2684bb5e7d63987cde2abd62fd8eb" + ), + "db/data/scotgov/band_d_council_tax_rates_2026_27/manifest.yaml": ( + "290d477aa64e951e34fb560a7892f1a4c77aebf2ce71a6e1652d95669b2d57eb" + ), + "db/data/scotgov/band_d_equivalents_2025/manifest.yaml": ( + "22e6ac2a13339d50f7ee2c86d886b32d5242330290687f2c8875ad11e0495e3f" + ), + "db/data/scotgov/council_tax_bands_2025/manifest.yaml": ( + "67918754625b035094c79f30d62cc638e96bc4925faf6bc78e981ae6275759cc" + ), + "db/data/scotgov/council_tax_collection_2024_25/manifest.yaml": ( + "3cc00a7c7762ed68ea7c9145b429a26360f061ac89518d510dacb123f846cd20" + ), + "db/data/scotgov/council_tax_collection_2025_26/manifest.yaml": ( + "bde33b4e442769aa93b215667ce4c87e5747f86b056306ada61af8862b5f5220" + ), + "db/data/scotgov/scottish_budget_social_security_assistance_2026/manifest.yaml": ( + "a46059c0c5cde32d90a7291ec1b497d65d3f11e93cccf532f04b7684b52fa0e9" + ), + "db/data/scotgov/slgfs_council_tax_2024_25/manifest.yaml": ( + "13bed33835cf6829cb0a34f78f9a59c03ee1fd5c8f18c7589c2b3549b8b15317" + ), + "db/data/sfpd/legal_pension_caseload_2025/manifest.yaml": ( + "6aee6347bdbd98534cfadf0620a63bf875eb61722845a2efb902a605e0292885" + ), + "db/data/slc/student_loan_borrower_forecasts_england_2025/manifest.yaml": ( + "5bcb4476cd10707cb12c5583207f96858b7f99af58cf02bb230719e96083da24" + ), + "db/data/slc/student_loan_repayments_england_2025/manifest.yaml": ( + "cdabb4ef47f3e31d7bdc6c03cc5104e7311a2944ef3fef889f6848e65ce49f4e" + ), + "db/data/slc/student_loan_repayments_northern_ireland_2025/manifest.yaml": ( + "e6c9012ee553b3c2b745b3519b8eef57bc265009841b27fa9624330e6ad67cc1" + ), + "db/data/slc/student_loan_repayments_scotland_2025/manifest.yaml": ( + "1653c9411a3aa209c4246d06411a41a0c9ca7c35335e22ba14edeef6773f04df" + ), + "db/data/slc/student_loan_repayments_wales_2025/manifest.yaml": ( + "d78d6ecbd553c88f6fdb37f5230e8ce1a659e028c912eb41ad037779d2565187" + ), + "db/data/slc/student_support_england_2025/manifest.yaml": ( + "be20a9f99f9b670e1066256a055e9bd36cd3f3438ec1ff5b84ebd34fb1e970ca" + ), + "db/data/spf_finances/pit_2023/manifest.yaml": ( + "3fb455eece095ec8178067c560eb8f125bcc3e849af569494142d1770914fedb" + ), + "db/data/ssa/annual_statistical_supplement_2025/manifest.yaml": ( + "04028ca0dd26e94acdfddda08a36998f440ba226403b85e54d0cfa5a23c9b43f" + ), + "db/data/ssa/ssi_monthly_statistics_2024_12/manifest.yaml": ( + "8dc1e91c4f49b7db3ecb30eaa7e9ade885e4c4c96125373551163286fcb05903" + ), + "db/data/ssa/ssi_table_7b1_2024/manifest.yaml": ( + "11e11f4ba5ed569cae2a0a4494c149ea9fc2919e0cb8a255cd591c6f7e1891e1" + ), + "db/data/statbel/fiscal_income_commune_2023_nis_2025/manifest.yaml": ( + "c0d93d91b73201deb31d8234bf5ff6b79b67e6600cab897214ce5b135a57839f" + ), + "db/data/statbel/fiscal_income_distribution_2023/manifest.yaml": ( + "612a5debce569635dfabc7df48747c938388ad4e624f32eb02de6f2b0d7ac1e5" + ), + "db/data/statbel/nis_2025_commune_crosswalk/manifest.yaml": ( + "d38cc3252396c9f343b5acca1f1537d8b128a05d8c6acb4846e49a25ae680cef" + ), + "db/data/statbel/population_structure_nuts1_2025/manifest.yaml": ( + "1ee3c24db3c24feb887225ef7565a3e8443d53666fbcbf8328fae57c902cb8fb" + ), + "db/data/statbel/population_structure_nuts1_2026/manifest.yaml": ( + "e21a51518842e591c535145bbbe8e6e2ff7cb33fa53513d8d17a1443a405139a" + ), + "db/data/usda_snap/fy69_to_current/manifest.yaml": ( + "e735895977bfb23c2a2a7d36b4b255be4fcd7b2712b06468de7d936dcf841830" + ), + "db/data/usda_snap/fy69_to_current/manifest_fy2025_monthly_source_package.yaml": ( + "ed366f4d02e86356abce8c30ee54b341154b48deb615819c5e71e538df3851c8" + ), + "db/data/voa/council_tax_bands_2025/manifest.yaml": ( + "1e5d497ad917eb0c0c17003d915eb37b1a8def303c065378e785e59afbaf3ca4" + ), + "db/data/voa/council_tax_stock_by_lad_2025/manifest.yaml": ( + "4b2da68c429bb75ea915f753dc8c66d401d9858a92f4cdd403a9107cc3ececfd" + ), + "db/data/welshgov/council_tax_collection_2024_25/manifest.yaml": ( + "2b6bee77282acc6bee952161d432684e0fde6f66a4dd4d59e0fe8aa8d964fcff" + ), + "db/data/welshgov/council_tax_collection_2025_26/manifest.yaml": ( + "769dd2f8f7178f4df2290316eb2034ffbdb4850fc25e71b4ebf5c319d8be94d2" + ), + "db/data/welshgov/council_tax_levels_2026_27/manifest.yaml": ( + "36021ed92e559233d7442685df4e04331b11fd320fe13f40ac9b1e9afc0f2850" + ), + "db/data/welshgov/ctrs_annual_report_2024_25/manifest.yaml": ( + "e3836e875eaf9656e296e66e7017086161a3f76303697ff0d132640bd99dfd70" + ), + "db/data/welshgov/ctrs_annual_report_2025_26/manifest.yaml": ( + "dcbcd60d7ff2775827205dcbde03a3479cc8c53f030dc1943b9a2ba5e9609455" + ), + } +) + + +def manifest_digest(manifest_path: Any) -> str: + """Return the SHA-256 of a manifest file's bytes.""" + return hashlib.sha256(manifest_path.read_bytes()).hexdigest() + + +def grandfathered_manifest_key(manifest_path: Any) -> str | None: + """Return the frozen-list key ``manifest_path`` addresses, if any. + + Manifests are addressed by their repository-relative path, so the lookup + matches the longest trailing run of path segments that is a key. A path + outside the repository can only match by carrying the same segments, and + then only counts once its bytes match too. + """ + parts = PurePosixPath(str(manifest_path).replace("\\", "/")).parts + for start in range(len(parts)): + candidate = "/".join(parts[start:]) + if candidate in GRANDFATHERED_KINDLESS_MANIFESTS: + return candidate + return None + + +def is_grandfathered_manifest(manifest_path: Any) -> bool: + """Whether the file at ``manifest_path`` is frozen kindless, byte for byte.""" + key = grandfathered_manifest_key(manifest_path) + if key is None: + return False + try: + digest = manifest_digest(manifest_path) + except (OSError, AttributeError): + return False + return digest == GRANDFATHERED_KINDLESS_MANIFESTS[key] diff --git a/chronicle/harness.py b/chronicle/harness.py index 298a1380..b450bfa0 100644 --- a/chronicle/harness.py +++ b/chronicle/harness.py @@ -5,14 +5,19 @@ import argparse import json import shlex +import sys from pathlib import Path from chronicle.artifacts import ( + DEFAULT_MANIFEST_FILENAME, + DEFAULT_R2_DERIVED_BUCKET, + DEFAULT_R2_RAW_BUCKET, ArtifactFetchReport, ArtifactInventoryReport, DerivedArtifactPublishReport, R2BootstrapReport, RawArtifactPublishReport, + SourceArtifactManifestError, bootstrap_r2_buckets, fetch_source_artifact, inventory_source_artifacts, @@ -33,6 +38,7 @@ ) from chronicle.core import AggregateFact, ValidationReport, validate_facts from chronicle.database import ChronicleDbBuildReport, build_chronicle_db +from chronicle.env import DEFAULT_CHRONICLE_SCHEMA from chronicle.mirror import ( ChronicleMirrorExportReport, SupabaseMirrorLoadReport, @@ -45,6 +51,16 @@ write_pe_source_plan_json, write_pe_source_plan_markdown, ) +from chronicle.registration import ( + ACCESS_CLASSES, + HASH_ONLY_HASH_SOURCES, + MANIFEST_KINDS, + ArtifactRegistrationReport, + HashOnlyRegistrationError, + ManifestAccessError, + MicrodataReleaseNotParseableError, + register_hash_only_artifact, +) from chronicle.sources.cells import ( SourceCell, SourceCellReport, @@ -334,12 +350,29 @@ def fetch_artifact_file( source_page: str | None = None, table: str | None = None, filename: str | None = None, + manifest_filename: str = DEFAULT_MANIFEST_FILENAME, + access: str = "public", + licence: str | None = None, + kind: str | None = None, + publisher: str | None = None, + vintage: str | None = None, + expected_sha256: str | None = None, + expected_size_bytes: int | None = None, + licence_evidence: dict[str, str] | None = None, + staging_dir: str | Path | None = None, upload_r2: bool = False, - r2_bucket: str = "ledger-raw", + record_revision: bool = False, + r2_bucket: str | None = None, r2_prefix: str | None = None, wrangler_command: str = "npx wrangler", ) -> ArtifactFetchReport: - """Fetch/register a raw source artifact and optionally upload it to R2.""" + """Fetch/register a raw source artifact and optionally upload it to R2. + + ``manifest_filename`` selects which of the package directory's manifests + the entry belongs to. Raises :class:`SourceArtifactRevisionError` when the + fetched bytes are not the bytes that manifest's entry identifies, unless + ``record_revision`` opts into registering the publisher revision. + """ return fetch_source_artifact( source_url, source_id=source_id, @@ -350,20 +383,96 @@ def fetch_artifact_file( source_page=source_page, table=table, filename=filename, + manifest_filename=manifest_filename, + access=access, + licence=licence, + kind=kind, + publisher=publisher, + vintage=vintage, + expected_sha256=expected_sha256, + expected_size_bytes=expected_size_bytes, + licence_evidence=licence_evidence, + staging_dir=staging_dir, upload_r2=upload_r2, + record_revision=record_revision, r2_bucket=r2_bucket, r2_prefix=r2_prefix, wrangler_command=wrangler_command, ) +def register_artifact_file( + *, + source_id: str, + package_id: str, + year: int, + output_dir: str | Path, + filename: str, + sha256: str, + licence: str, + access: str, + vintage: str, + hash_source: str, + attested_by: str, + attestation_evidence: str | None = None, + pinned_from: dict[str, str] | None = None, + verified_at: str | None = None, + size_bytes: int | None = None, + source_page: str | None = None, + source_url: str | None = None, + access_route: str | None = None, + doi: str | None = None, + study: str | None = None, + dataset: str | None = None, + table: str | None = None, + publisher: str | None = None, + fetched_at: str | None = None, + notes: str | None = None, + allow_reissue: bool = False, +) -> ArtifactRegistrationReport: + """Register a licensed or restricted artifact by identity, without bytes.""" + return register_hash_only_artifact( + source_id=source_id, + package_id=package_id, + year=year, + output_dir=output_dir, + filename=filename, + sha256=sha256, + licence=licence, + access=access, + vintage=vintage, + hash_source=hash_source, + attested_by=attested_by, + attestation_evidence=attestation_evidence, + pinned_from=pinned_from, + verified_at=verified_at, + size_bytes=size_bytes, + source_page=source_page, + source_url=source_url, + access_route=access_route, + doi=doi, + study=study, + dataset=dataset, + table=table, + publisher=publisher, + fetched_at=fetched_at, + notes=notes, + allow_reissue=allow_reissue, + ) + + def inventory_artifact_files( root: str | Path, *, manifest_filename: str = "manifest.yaml", + staging_dir: str | Path | None = None, ) -> ArtifactInventoryReport: """Inventory local manifest-declared source artifacts.""" - return inventory_source_artifacts(root, manifest_filename=manifest_filename) + return inventory_source_artifacts( + root, + manifest_filename=manifest_filename, + staging_dir=staging_dir, + ) def publish_raw_artifact_files( @@ -372,9 +481,11 @@ def publish_raw_artifact_files( manifest_filename: str = "manifest.yaml", source_id: str | None = None, package_id: str | None = None, - r2_bucket: str = "ledger-raw", + r2_bucket: str | None = None, r2_prefix: str | None = None, wrangler_command: str = "npx wrangler", + skip_hash_only: bool = False, + staging_dir: str | Path | None = None, ) -> RawArtifactPublishReport: """Publish manifest-declared raw source artifacts to R2.""" return publish_source_artifacts( @@ -385,13 +496,15 @@ def publish_raw_artifact_files( r2_bucket=r2_bucket, r2_prefix=r2_prefix, wrangler_command=wrangler_command, + skip_hash_only=skip_hash_only, + staging_dir=staging_dir, ) def bootstrap_r2_storage( *, - raw_bucket: str = "ledger-raw", - derived_bucket: str = "ledger-derived", + raw_bucket: str | None = None, + derived_bucket: str | None = None, wrangler_command: str = "npx wrangler", ) -> R2BootstrapReport: """Create Chronicle R2 buckets when Wrangler is authenticated.""" @@ -409,7 +522,7 @@ def publish_derived_artifact_files( package_id: str, year: int, build_id: str | None = None, - r2_bucket: str = "ledger-derived", + r2_bucket: str | None = None, r2_prefix: str | None = None, wrangler_command: str = "npx wrangler", build_artifacts_output: str | Path | None = None, @@ -441,12 +554,16 @@ def export_chronicle_db_table_files( def load_supabase_mirror_files( input_dir: str | Path, *, - schema: str = "ledger", + schema: str | None = None, batch_size: int = 500, dry_run: bool = False, build_artifacts_path: str | Path | None = None, ) -> SupabaseMirrorLoadReport: - """Load exported Chronicle JSONL mirror files into Supabase/Postgres.""" + """Load exported Chronicle JSONL mirror files into Supabase/Postgres. + + ``schema`` of None resolves to ``$CHRONICLE_SCHEMA``, else the default + schema, so the hosted mirror writer answers to the renamed variable. + """ table_paths = ( {"build_artifacts": Path(build_artifacts_path)} if build_artifacts_path is not None @@ -484,6 +601,30 @@ def plan_pe_source_files( return report +def _licence_evidence_arguments(args: argparse.Namespace) -> dict[str, str] | None: + """Collect the licence-evidence flags, or None when none was passed.""" + evidence = { + "issuer": args.licence_evidence_issuer, + "scope": args.licence_evidence_scope, + "url": args.licence_evidence_url, + } + if all(value is None for value in evidence.values()): + return None + return {key: value for key, value in evidence.items() if value is not None} + + +def _pinned_from_arguments(args: argparse.Namespace) -> dict[str, str] | None: + """Collect the pinned-from flags, or None when none was passed.""" + pinned = { + "repository": args.pinned_from_repository, + "path": args.pinned_from_path, + "commit": args.pinned_from_commit, + } + if all(value is None for value in pinned.values()): + return None + return {key: value for key, value in pinned.items() if value is not None} + + def main(argv: list[str] | None = None) -> int: """Run the harness CLI.""" parser = argparse.ArgumentParser(description="Chronicle fact validation harness") @@ -836,7 +977,7 @@ def main(argv: list[str] | None = None) -> int: artifact_parser = subparsers.add_parser( "fetch-artifact", - help="Fetch/register a raw source artifact and update manifest.yaml", + help="Fetch/register a raw source artifact and update its manifest", ) artifact_parser.add_argument( "--url", @@ -857,13 +998,23 @@ def main(argv: list[str] | None = None) -> int: "--year", type=int, required=True, - help="Artifact vintage year to record in manifest.yaml", + help="Artifact vintage year to record in the manifest", ) artifact_parser.add_argument( "--out-dir", type=Path, required=True, - help="Directory where the raw artifact and manifest.yaml should live", + help="Directory where the raw artifact and its manifest should live", + ) + artifact_parser.add_argument( + "--manifest", + default=DEFAULT_MANIFEST_FILENAME, + help=( + "Manifest filename inside --out-dir. A publisher directory that " + "feeds several source packages keeps one manifest each, and the " + "entry being revised lives in exactly one of them. Defaults to " + f"{DEFAULT_MANIFEST_FILENAME}." + ), ) artifact_parser.add_argument( "--dataset", @@ -879,17 +1030,118 @@ def main(argv: list[str] | None = None) -> int: ) artifact_parser.add_argument( "--filename", - help="Override artifact filename inferred from URL/path.", + help=( + "Override the artifact filename inferred from the URL. Must be a " + "bare filename; the artifact always lands under that name." + ), + ) + artifact_parser.add_argument( + "--access", + default="public", + choices=list(ACCESS_CLASSES), + help=( + "Publisher access class. Only public artifacts may be fetched; " + "licensed and restricted artifacts are registered hash-only with " + "`chronicle register-artifact`." + ), + ) + artifact_parser.add_argument( + "--licence", + help=( + "Publisher terms identifier or URL. Required for entries in a " + "microdata-release manifest." + ), + ) + artifact_parser.add_argument( + "--kind", + choices=list(MANIFEST_KINDS), + help=( + "Manifest kind to declare. Pass microdata_release to archive a " + "public-use microdata release, which may hold several files under " + "one vintage and is never parsed by a source package. Must match " + "the existing manifest's kind; a conflicting kind is refused. " + "Omit to inherit it (publisher_table for a new manifest)." + ), + ) + artifact_parser.add_argument( + "--publisher", + help="Publishing body. Required for a microdata release.", + ) + artifact_parser.add_argument( + "--vintage", + help="Publisher vintage label. Required for a microdata release.", + ) + artifact_parser.add_argument( + "--expected-sha256", + help=( + "Lowercase 64-character SHA-256 the fetched bytes must have, from " + "a reviewed pin. The fetch refuses, before writing or uploading, " + "bytes that hash differently; --record-revision does not override " + "it. Required for a microdata release: the licence evidence covers " + "this checksum. Never invent one." + ), + ) + artifact_parser.add_argument( + "--expected-size-bytes", + type=int, + help="Size the fetched bytes must have, from the same reviewed pin.", + ) + artifact_parser.add_argument( + "--licence-evidence-issuer", + help=( + "Who issued the file under the allowlisted --licence. Required " + "for a microdata release." + ), + ) + artifact_parser.add_argument( + "--licence-evidence-scope", + help=( + "Statement of what the evidence covers, such as 'public-use file " + "of a federal agency'. Required for a microdata release." + ), + ) + artifact_parser.add_argument( + "--licence-evidence-url", + help=( + "Durable http(s) URL of the publisher's evidence that this file is " + "issued under --licence. Required for a microdata release." + ), + ) + artifact_parser.add_argument( + "--staging-dir", + type=Path, + default=None, + help=( + "Transient directory where a microdata release's bytes are staged " + "before upload, outside the repository. Defaults to " + "$CHRONICLE_MICRODATA_STAGING_DIR, else " + "~/.cache/policyengine-chronicle/microdata-staging." + ), ) artifact_parser.add_argument( "--upload-r2", action="store_true", help="Upload the artifact to R2 after local checksum capture.", ) + artifact_parser.add_argument( + "--record-revision", + action="store_true", + help=( + "Register a publisher revision: the fetched bytes get their own " + "content-addressed key under the configured bucket and the " + "superseded object moves to storage.previous_r2. Without this " + "flag, bytes that disagree with what the entry identifies -- its " + "declared sha256, or its recorded content-addressed key once " + "published -- are refused." + ), + ) artifact_parser.add_argument( "--r2-bucket", - default="ledger-raw", - help="R2 bucket for raw artifacts when --upload-r2 is set.", + default=None, + help=( + "R2 bucket for raw artifacts when --upload-r2 is set. Defaults to " + f"$CHRONICLE_R2_RAW_BUCKET, else {DEFAULT_R2_RAW_BUCKET}." + ), ) artifact_parser.add_argument( "--r2-prefix", @@ -905,6 +1157,159 @@ def main(argv: list[str] | None = None) -> int: help="Wrangler command prefix to use for R2 uploads.", ) + registration_parser = subparsers.add_parser( + "register-artifact", + help="Register a licensed or restricted artifact by identity, no bytes", + description=( + "Register a raw artifact Chronicle may not redistribute. The " + "manifest records the checksum, size, vintage, licence, and access " + "route; no bytes are fetched, stored, or uploaded, and no R2 key " + "is recorded." + ), + ) + registration_parser.add_argument( + "--source-id", + required=True, + help="Stable source ID, such as dwp.", + ) + registration_parser.add_argument( + "--package-id", + required=True, + help="Stable package ID, such as dwp-frs-2023-24.", + ) + registration_parser.add_argument( + "--year", + type=int, + required=True, + help="Artifact vintage year to record in manifest.yaml", + ) + registration_parser.add_argument( + "--out-dir", + type=Path, + required=True, + help="Directory where manifest.yaml should live. No bytes are written.", + ) + registration_parser.add_argument( + "--filename", + required=True, + help="Publisher filename this registration identifies.", + ) + registration_parser.add_argument( + "--sha256", + required=True, + help="Lowercase 64-character SHA-256 of the publisher bytes.", + ) + registration_parser.add_argument( + "--size-bytes", + type=int, + help="Size of the publisher bytes, when known.", + ) + registration_parser.add_argument( + "--vintage", + required=True, + help="Publisher vintage label, such as 2023_24.", + ) + registration_parser.add_argument( + "--licence", + required=True, + help="Publisher terms identifier or URL, such as a UKDS licence.", + ) + registration_parser.add_argument( + "--access", + required=True, + choices=["licensed", "restricted"], + help="Access class. Public artifacts are registered with fetch-artifact.", + ) + registration_parser.add_argument( + "--source-page", + help="Publisher or archive landing page for the release.", + ) + registration_parser.add_argument( + "--source-url", + help="Direct publisher URL, when the release has one behind its licence.", + ) + registration_parser.add_argument( + "--access-route", + help="How an authorized consumer obtains the bytes.", + ) + registration_parser.add_argument( + "--doi", + help="Persistent identifier for the study, such as a UKDS DOI.", + ) + registration_parser.add_argument( + "--study", + help="Archive study reference, such as a UK Data Service study number.", + ) + registration_parser.add_argument( + "--dataset", + help="Manifest dataset ID. Defaults to _.", + ) + registration_parser.add_argument( + "--table", + help="Human-readable release title.", + ) + registration_parser.add_argument( + "--publisher", + help="Publishing body, when it differs from the source ID.", + ) + registration_parser.add_argument( + "--fetched-at", + help="When the authorized environment fetched the bytes, if known.", + ) + registration_parser.add_argument( + "--hash-source", + required=True, + choices=list(HASH_ONLY_HASH_SOURCES), + help=( + "How the checksum is known: consumer_attested (the consumer " + "verified it against bytes it holds; pass --attested-by, " + "--attestation-evidence and --verified-at) or consumer_pin (it is " + "transcribed from a reviewed pin in the consumer's repository; pass " + "--attested-by and the three --pinned-from-* fields, and no " + "--verified-at)." + ), + ) + registration_parser.add_argument( + "--attested-by", + required=True, + help="The consumer that attests the checksum, such as PolicyEngine/microcosm.", + ) + registration_parser.add_argument( + "--attestation-evidence", + help="The consumer's evidence for a consumer_attested checksum.", + ) + registration_parser.add_argument( + "--verified-at", + help=( + "When the consumer verified the checksum against the bytes " + "(consumer_attested only)." + ), + ) + registration_parser.add_argument( + "--pinned-from-repository", + help="Repository holding the consumer's pin, such as PolicyEngine/microcosm.", + ) + registration_parser.add_argument( + "--pinned-from-path", + help="Path of the pin inside that repository.", + ) + registration_parser.add_argument( + "--pinned-from-commit", + help="40-hex commit the pin was read from.", + ) + registration_parser.add_argument( + "--notes", + help="Free-text provenance notes for the registration.", + ) + registration_parser.add_argument( + "--allow-reissue", + action="store_true", + help=( + "Register different bytes for a filename already registered in " + "this year, keeping both registrations." + ), + ) + artifact_inventory_parser = subparsers.add_parser( "inventory-artifacts", help="Inventory local manifest-declared source artifacts", @@ -920,10 +1325,20 @@ def main(argv: list[str] | None = None) -> int: default="manifest.yaml", help="Manifest filename to scan for.", ) + artifact_inventory_parser.add_argument( + "--staging-dir", + type=Path, + default=None, + help=( + "Where public microdata releases are staged. Defaults to " + "$CHRONICLE_MICRODATA_STAGING_DIR, else " + "~/.cache/policyengine-chronicle/microdata-staging." + ), + ) raw_publish_parser = subparsers.add_parser( "publish-raw", - help="Upload manifest-declared raw source artifacts to ledger-raw R2", + help="Upload manifest-declared raw source artifacts to the raw R2 bucket", ) raw_publish_parser.add_argument( "--root", @@ -936,6 +1351,25 @@ def main(argv: list[str] | None = None) -> int: default="manifest.yaml", help="Manifest filename to scan for.", ) + raw_publish_parser.add_argument( + "--skip-hash-only", + action="store_true", + help=( + "Treat licensed and restricted registrations as deliberately " + "skipped rather than refused, so a mixed tree can be published." + ), + ) + raw_publish_parser.add_argument( + "--staging-dir", + type=Path, + default=None, + help=( + "Where public microdata releases are staged; their bytes are read " + "from there, never from beside the manifest. Defaults to " + "$CHRONICLE_MICRODATA_STAGING_DIR, else " + "~/.cache/policyengine-chronicle/microdata-staging." + ), + ) raw_publish_parser.add_argument( "--source-id", help="Override manifest source_id for scanned artifacts.", @@ -946,8 +1380,11 @@ def main(argv: list[str] | None = None) -> int: ) raw_publish_parser.add_argument( "--r2-bucket", - default="ledger-raw", - help="R2 bucket for immutable raw artifacts.", + default=None, + help=( + "R2 bucket for immutable raw artifacts. Defaults to " + f"$CHRONICLE_R2_RAW_BUCKET, else {DEFAULT_R2_RAW_BUCKET}." + ), ) raw_publish_parser.add_argument( "--r2-prefix", @@ -969,13 +1406,19 @@ def main(argv: list[str] | None = None) -> int: ) r2_parser.add_argument( "--raw-bucket", - default="ledger-raw", - help="R2 bucket name for immutable raw source artifacts.", + default=None, + help=( + "R2 bucket name for immutable raw source artifacts. Defaults to " + f"$CHRONICLE_R2_RAW_BUCKET, else {DEFAULT_R2_RAW_BUCKET}." + ), ) r2_parser.add_argument( "--derived-bucket", - default="ledger-derived", - help="R2 bucket name for derived Chronicle build artifacts.", + default=None, + help=( + "R2 bucket name for derived Chronicle build artifacts. Defaults to " + f"$CHRONICLE_R2_DERIVED_BUCKET, else {DEFAULT_R2_DERIVED_BUCKET}." + ), ) r2_parser.add_argument( "--wrangler-command", @@ -985,7 +1428,7 @@ def main(argv: list[str] | None = None) -> int: derived_publish_parser = subparsers.add_parser( "publish-derived", - help="Upload deterministic Chronicle build outputs to ledger-derived R2", + help="Upload deterministic Chronicle build outputs to the derived R2 bucket", ) derived_publish_parser.add_argument( "--dir", @@ -1014,13 +1457,16 @@ def main(argv: list[str] | None = None) -> int: help=( "Build ID under an accepted epoch prefix, ledger.build.v1: or " "chronicle.build.v2:; any other form is refused. Defaults to " - "the ID inferred from reports or ledger.db." + "the ID inferred from reports, chronicle.db, or a legacy ledger.db." ), ) derived_publish_parser.add_argument( "--r2-bucket", - default="ledger-derived", - help="R2 bucket for derived build artifacts.", + default=None, + help=( + "R2 bucket for derived build artifacts. Defaults to " + f"$CHRONICLE_R2_DERIVED_BUCKET, else {DEFAULT_R2_DERIVED_BUCKET}." + ), ) derived_publish_parser.add_argument( "--r2-prefix", @@ -1075,8 +1521,11 @@ def main(argv: list[str] | None = None) -> int: ) mirror_load_parser.add_argument( "--schema", - default="ledger", - help="Supabase/Postgres schema to load into.", + default=None, + help=( + "Supabase/Postgres schema to load into. Defaults to " + f"$CHRONICLE_SCHEMA, else {DEFAULT_CHRONICLE_SCHEMA}." + ), ) mirror_load_parser.add_argument( "--batch-size", @@ -1252,15 +1701,21 @@ def main(argv: list[str] | None = None) -> int: return 0 if args.command == "build-suite": axiom_command = shlex.split(args.axiom_cli) if args.axiom_cli else None - report = build_source_suite_dir( - args.source, - args.out, - year=args.year, - axiom_command=axiom_command, - axiom_roots=args.axiom_root, - require_axiom_validation=args.require_axiom_validation, - replace=args.replace, - ) + try: + report = build_source_suite_dir( + args.source, + args.out, + year=args.year, + axiom_command=axiom_command, + axiom_roots=args.axiom_root, + require_axiom_validation=args.require_axiom_validation, + replace=args.replace, + ) + except (ManifestAccessError, MicrodataReleaseNotParseableError) as error: + # Refused before the output directory was touched: no source + # package parses a microdata release or a hash-only entry. + print(f"error: {error}", file=sys.stderr) + return 1 print(json.dumps(report.to_dict(), indent=2, sort_keys=True)) return 0 if report.valid else 1 if args.command == "build-bundle": @@ -1305,27 +1760,78 @@ def main(argv: list[str] | None = None) -> int: print(json.dumps(report.to_dict(), indent=2, sort_keys=True)) return 0 if args.command == "fetch-artifact": - report = fetch_artifact_file( - args.url, - source_id=args.source_id, - package_id=args.package_id, - year=args.year, - output_dir=args.out_dir, - dataset=args.dataset, - source_page=args.source_page, - table=args.table, - filename=args.filename, - upload_r2=args.upload_r2, - r2_bucket=args.r2_bucket, - r2_prefix=args.r2_prefix, - wrangler_command=args.wrangler_command, - ) + try: + report = fetch_artifact_file( + args.url, + source_id=args.source_id, + package_id=args.package_id, + year=args.year, + output_dir=args.out_dir, + dataset=args.dataset, + source_page=args.source_page, + table=args.table, + filename=args.filename, + manifest_filename=args.manifest, + access=args.access, + licence=args.licence, + kind=args.kind, + publisher=args.publisher, + vintage=args.vintage, + expected_sha256=args.expected_sha256, + expected_size_bytes=args.expected_size_bytes, + licence_evidence=_licence_evidence_arguments(args), + staging_dir=args.staging_dir, + upload_r2=args.upload_r2, + record_revision=args.record_revision, + r2_bucket=args.r2_bucket, + r2_prefix=args.r2_prefix, + wrangler_command=args.wrangler_command, + ) + except (SourceArtifactManifestError, ManifestAccessError, ValueError) as error: + print(f"error: {error}", file=sys.stderr) + return 1 print(json.dumps(report.to_dict(), indent=2, sort_keys=True)) return 0 if report.valid else 1 + if args.command == "register-artifact": + try: + registration = register_artifact_file( + source_id=args.source_id, + package_id=args.package_id, + year=args.year, + output_dir=args.out_dir, + filename=args.filename, + sha256=args.sha256, + licence=args.licence, + access=args.access, + vintage=args.vintage, + hash_source=args.hash_source, + attested_by=args.attested_by, + attestation_evidence=args.attestation_evidence, + pinned_from=_pinned_from_arguments(args), + verified_at=args.verified_at, + size_bytes=args.size_bytes, + source_page=args.source_page, + source_url=args.source_url, + access_route=args.access_route, + doi=args.doi, + study=args.study, + dataset=args.dataset, + table=args.table, + publisher=args.publisher, + fetched_at=args.fetched_at, + notes=args.notes, + allow_reissue=args.allow_reissue, + ) + except (HashOnlyRegistrationError, ManifestAccessError) as error: + print(f"error: {error}", file=sys.stderr) + return 1 + print(json.dumps(registration.to_dict(), indent=2, sort_keys=True)) + return 0 if registration.valid else 1 if args.command == "inventory-artifacts": report = inventory_artifact_files( args.root, manifest_filename=args.manifest, + staging_dir=args.staging_dir, ) print(json.dumps(report.to_dict(), indent=2, sort_keys=True)) return 0 if report.valid else 1 @@ -1338,6 +1844,8 @@ def main(argv: list[str] | None = None) -> int: r2_bucket=args.r2_bucket, r2_prefix=args.r2_prefix, wrangler_command=args.wrangler_command, + skip_hash_only=args.skip_hash_only, + staging_dir=args.staging_dir, ) print(json.dumps(report.to_dict(), indent=2, sort_keys=True)) return 0 if report.valid else 1 diff --git a/chronicle/licences.py b/chronicle/licences.py new file mode 100644 index 00000000..a3679ba7 --- /dev/null +++ b/chronicle/licences.py @@ -0,0 +1,137 @@ +"""Redistributable licence terms Chronicle may archive microdata bytes under. + +Being downloadable is not a licence, and a licence *name* on a manifest entry +is not evidence that a particular file was issued under it. Chronicle archives +a microdata release's bytes only when the entry's ``licence`` is one of the +terms below and the entry carries ``licence_evidence`` binding the artifact to +that term (``docs/adr-chronicle-raw-microdata-identity.md``). This module is +the allowlist, kept in code so every term carries the evidence for why +redistribution is permitted. + +Adding a term is a reviewed code change: give it a stable identifier (SPDX +where one exists), the legal basis, and a durable URL to the terms. +""" + +from __future__ import annotations + +from collections.abc import Mapping +from dataclasses import dataclass +from types import MappingProxyType +from typing import Any + +__all__ = [ + "LICENCE_EVIDENCE_FIELDS", + "REDISTRIBUTABLE_LICENCES", + "RedistributableLicence", + "is_redistributable_licence", + "licence_evidence_errors", +] + + +@dataclass(frozen=True) +class RedistributableLicence: + """One term under which Chronicle may hold and re-serve publisher bytes.""" + + identifier: str + name: str + basis: str + evidence_url: str + + +REDISTRIBUTABLE_LICENCES: Mapping[str, RedistributableLicence] = MappingProxyType( + { + "US-Government-Work": RedistributableLicence( + identifier="US-Government-Work", + name="Work of the United States Government", + basis=( + "17 U.S.C. §105: copyright protection is not available for any " + "work of the United States Government, so a federal statistical " + "agency's public-use file may be copied and redistributed." + ), + evidence_url=( + "https://uscode.house.gov/view.xhtml?req=granuleid:USC-prelim-" + "title17-section105" + ), + ), + "OGL-UK-3.0": RedistributableLicence( + identifier="OGL-UK-3.0", + name="Open Government Licence v3.0", + basis=( + "The licence grants a worldwide, royalty-free, perpetual, " + "non-exclusive licence to copy, publish, distribute and transmit " + "the information, subject to attribution." + ), + evidence_url=( + "https://www.nationalarchives.gov.uk/doc/open-government-licence/" + "version/3/" + ), + ), + "CC0-1.0": RedistributableLicence( + identifier="CC0-1.0", + name="Creative Commons CC0 1.0 Universal", + basis=( + "The affirmer waives all copyright and related rights, so the " + "work may be copied and redistributed without restriction." + ), + evidence_url="https://creativecommons.org/publicdomain/zero/1.0/legalcode", + ), + "CC-BY-4.0": RedistributableLicence( + identifier="CC-BY-4.0", + name="Creative Commons Attribution 4.0 International", + basis=( + "Section 2(a)(1) grants a worldwide, royalty-free, non-exclusive " + "licence to reproduce and share the licensed material, subject to " + "attribution." + ), + evidence_url="https://creativecommons.org/licenses/by/4.0/legalcode", + ), + } +) + +#: Fields a ``licence_evidence`` block must carry to bind an artifact to a term. +LICENCE_EVIDENCE_FIELDS: tuple[str, ...] = ( + "issuer", + "licence", + "scope", + "url", + "sha256", +) + + +def is_redistributable_licence(licence: Any) -> bool: + """Whether ``licence`` names a term on the allowlist.""" + return isinstance(licence, str) and licence.strip() in REDISTRIBUTABLE_LICENCES + + +def licence_evidence_errors( + evidence: Any, + *, + licence: Any, + sha256: Any, +) -> list[str]: + """Return error codes for a ``licence_evidence`` block. + + The block binds one artifact to one allowlisted term: its ``licence`` must + be the entry's own (allowlisted) licence, its ``sha256`` the entry's own + checksum, and its ``url`` a durable http(s) location of the evidence. + """ + if evidence is None: + return ["missing_licence_evidence"] + if not isinstance(evidence, Mapping): + return ["malformed_licence_evidence"] + errors: list[str] = [] + for field in LICENCE_EVIDENCE_FIELDS: + value = evidence.get(field) + if not isinstance(value, str) or not value.strip(): + errors.append(f"licence_evidence_missing_field:{field}") + if errors: + return errors + if evidence["licence"].strip() != str(licence).strip(): + errors.append("licence_evidence_licence_mismatch") + if not is_redistributable_licence(evidence["licence"]): + errors.append(f"licence_not_redistributable:{evidence['licence'].strip()}") + if evidence["sha256"].strip() != str(sha256 or "").strip(): + errors.append("licence_evidence_sha256_mismatch") + if not evidence["url"].strip().startswith(("http://", "https://")): + errors.append("licence_evidence_url_not_durable") + return errors diff --git a/chronicle/mirror.py b/chronicle/mirror.py index 7a3ccaf3..f9c95dc2 100644 --- a/chronicle/mirror.py +++ b/chronicle/mirror.py @@ -11,6 +11,8 @@ from pathlib import Path from typing import Any +from chronicle.env import default_chronicle_schema + LEDGER_MIRROR_TABLES = ( "ledger_builds", "build_artifacts", @@ -200,15 +202,22 @@ def export_chronicle_db_tables( def load_supabase_mirror( input_dir: str | Path, *, - schema: str = "ledger", + schema: str | None = None, batch_size: int = 500, dry_run: bool = False, table_paths: dict[str, str | Path] | None = None, client: Any | None = None, ) -> SupabaseMirrorLoadReport: - """Load exported Chronicle JSONL mirror files into Supabase/Postgres.""" + """Load exported Chronicle JSONL mirror files into Supabase/Postgres. + + ``schema`` defaults to :func:`chronicle.env.default_chronicle_schema`, so + the writer that owns the hosted mirror answers to ``CHRONICLE_SCHEMA`` (and + the ledger-era names behind it) exactly like every other reader of the + setting. The resolved name is reported back in the load report. + """ if batch_size < 1: raise ValueError("batch_size must be at least 1.") + schema = schema or default_chronicle_schema() input_path = Path(input_dir) tables: list[SupabaseTableLoad] = [] errors: list[str] = [] diff --git a/chronicle/registration.py b/chronicle/registration.py new file mode 100644 index 00000000..a698c243 --- /dev/null +++ b/chronicle/registration.py @@ -0,0 +1,1253 @@ +"""Access classification and hash-only registration for Chronicle artifacts. + +Chronicle registers every raw artifact its consumers build from and stores the +bytes of only those the publisher permits it to redistribute. Manifest fields +carry that split: + +``kind`` + Manifest-level: ``publisher_table`` or ``microdata_release``. Every + manifest created or modified after + ``docs/adr-chronicle-raw-microdata-identity.md`` declares it. Manifests + that predate the rule are read as publisher tables only while they match + the frozen list in :mod:`chronicle.grandfathered_manifests`; any other + kindless manifest is an error, never a publisher table by default. + +``licence`` + The publisher's terms. For a public microdata release this is an + identifier from the allowlist in :mod:`chronicle.licences`, and the entry + also carries ``licence_evidence`` binding this artifact to that term. + +``access`` + A closed class: ``public``, ``licensed``, or ``restricted``. + +``hash_source`` and its attester + Who asserts the checksum: ``chronicle_fetch`` (``attested_by: chronicle``, + ``verified_at`` = fetch date), ``consumer_attested`` (``attested_by`` = the + consumer, ``attestation_evidence``, ``verified_at``), or ``consumer_pin`` + (``attested_by`` = the consumer, ``pinned_from`` = repository, path and + commit, no ``verified_at``). + +``public`` artifacts keep the fetch/publish path: bytes are archived in the raw +R2 bucket under the content-addressed key +``raw/{source_id}/{package_id}/{year}/{sha256}/{filename}``. ``licensed`` and +``restricted`` artifacts are registered *hash-only*: the manifest records the +checksum, vintage, licence, access route and attestation, and no Chronicle +store ever holds the bytes. That key exists only for ``public`` artifacts. + +A registration is identified by ``{source_id, package_id, year, sha256, +filename}``. Consumers reference a registration by exactly that tuple. The +filename is a bare name inside the package directory, compared case-folded, +and ``2023`` and ``'2023'`` are one vintage key. +""" + +from __future__ import annotations + +from collections.abc import Iterable, Iterator, Mapping +from dataclasses import dataclass +from pathlib import Path +import re +from typing import Any + +import yaml + +from chronicle.grandfathered_manifests import is_grandfathered_manifest +from chronicle.licences import is_redistributable_licence, licence_evidence_errors + + +ACCESS_PUBLIC = "public" +ACCESS_LICENSED = "licensed" +ACCESS_RESTRICTED = "restricted" +#: The closed set of access classes a manifest file entry may declare. +ACCESS_CLASSES: tuple[str, ...] = (ACCESS_PUBLIC, ACCESS_LICENSED, ACCESS_RESTRICTED) +#: Access class inferred for a publisher-table entry that does not declare one. +DEFAULT_ACCESS = ACCESS_PUBLIC + +PUBLISHER_TABLE_KIND = "publisher_table" +MICRODATA_RELEASE_KIND = "microdata_release" +#: The closed set of manifest kinds. +MANIFEST_KINDS: tuple[str, ...] = (PUBLISHER_TABLE_KIND, MICRODATA_RELEASE_KIND) +#: Kind of a manifest that does not exist yet, and of a frozen kindless one. +DEFAULT_MANIFEST_KIND = PUBLISHER_TABLE_KIND + +HASH_SOURCE_CHRONICLE_FETCH = "chronicle_fetch" +HASH_SOURCE_CONSUMER_ATTESTED = "consumer_attested" +HASH_SOURCE_CONSUMER_PIN = "consumer_pin" +#: The closed set of checksum provenances a registration may declare. +HASH_SOURCES: tuple[str, ...] = ( + HASH_SOURCE_CHRONICLE_FETCH, + HASH_SOURCE_CONSUMER_ATTESTED, + HASH_SOURCE_CONSUMER_PIN, +) +#: Provenances a hash-only registration may declare: Chronicle never fetched +#: the bytes, so the checksum is always the consumer's. +HASH_ONLY_HASH_SOURCES: tuple[str, ...] = ( + HASH_SOURCE_CONSUMER_ATTESTED, + HASH_SOURCE_CONSUMER_PIN, +) +#: The attester of a ``chronicle_fetch`` checksum. +CHRONICLE_ATTESTER = "chronicle" +#: Fields a ``pinned_from`` block carries: where the consumer's pin was read. +PINNED_FROM_FIELDS: tuple[str, ...] = ("repository", "path", "commit") + +_SHA256_RE = re.compile(r"^[0-9a-f]{64}$") +_COMMIT_RE = re.compile(r"^[0-9a-f]{40}$") + +# Registration entry key order, so emitted manifests are byte-stable. +_REGISTRATION_FIELD_ORDER: tuple[str, ...] = ( + "filename", + "access", + "licence", + "licence_evidence", + "vintage", + "sha256", + "size_bytes", + "source_url", + "access_route", + "doi", + "study", + "fetched_at", + "verified_at", + "hash_source", + "attested_by", + "attestation_evidence", + "pinned_from", + "notes", +) + + +class ManifestAccessError(ValueError): + """Raised when a manifest declares an unusable access class or kind.""" + + +class ManifestKindError(ManifestAccessError): + """Raised when a manifest declares no kind and is not frozen kindless.""" + + +class ArtifactFilenameError(ManifestAccessError): + """Raised when a filename is not a bare name inside the package directory.""" + + +class AmbiguousVintageKeyError(ManifestAccessError): + """Raised when a manifest records one vintage under both key spellings.""" + + +class HashOnlyRegistrationError(ValueError): + """Raised when a hash-only registration is malformed or would store bytes.""" + + +class MicrodataReleaseNotParseableError(ValueError): + """Raised when a source package points at a registered microdata release. + + Registration is manifest-level: no source package parses a microdata + release, and no microdata row, cell, or fact enters Chronicle. + """ + + +@dataclass(frozen=True) +class ListSpecRejected: + """Marker for a list ``files[year]`` value outside a microdata release.""" + + spec: Any + + +# -------------------------------------------------------------------------- +# Manifest kind +# -------------------------------------------------------------------------- + + +def manifest_kind( + manifest: Mapping[str, Any] | None, + *, + manifest_path: Any = None, +) -> str: + """Return a manifest's declared kind. + + An absent or empty manifest has the default kind: the command creating it + declares one. So does a manifest that declares no file entry (a bare + ``files:`` line or an empty mapping): there is nothing in it that could be + read as a publisher table, and the command writing its first entry + declares the kind. A manifest with entries must declare ``kind`` itself, + unless ``manifest_path`` names a file frozen kindless before the rule and + its bytes still match the freeze. + """ + if not isinstance(manifest, Mapping) or not manifest: + return DEFAULT_MANIFEST_KIND + declared = manifest.get("kind") + if declared is None: + if not has_file_entries(manifest): + return DEFAULT_MANIFEST_KIND + if manifest_path is not None and is_grandfathered_manifest(manifest_path): + return PUBLISHER_TABLE_KIND + where = str(manifest_path) if manifest_path is not None else "Manifest" + raise ManifestKindError( + f"{where} declares no kind. Every manifest created or modified " + f"after the microdata-identity ADR declares kind: one of " + f"{list(MANIFEST_KINDS)}; a kindless manifest is read as a " + "publisher table only while it matches the frozen list in " + "chronicle/grandfathered_manifests.py byte for byte." + ) + kind = str(declared) + if kind not in MANIFEST_KINDS: + raise ManifestAccessError( + f"Unknown manifest kind {kind!r}; expected one of {list(MANIFEST_KINDS)}." + ) + return kind + + +def safe_manifest_kind( + manifest: Mapping[str, Any] | None, + *, + manifest_path: Any = None, +) -> tuple[str, str | None]: + """Return ``(kind, error_code)`` without raising. + + The reporting commands use this so a manifest with a missing or unknown + kind is still walked and reported. The returned kind is only what the + entries are read *as* for that report; the error code says the manifest + itself is invalid. + """ + try: + return manifest_kind(manifest, manifest_path=manifest_path), None + except ManifestKindError: + return DEFAULT_MANIFEST_KIND, "manifest_kind_missing" + except ManifestAccessError: + declared = manifest.get("kind") if isinstance(manifest, Mapping) else None + return DEFAULT_MANIFEST_KIND, f"unknown_manifest_kind:{declared}" + + +def is_microdata_release( + manifest: Mapping[str, Any] | None, + *, + manifest_path: Any = None, +) -> bool: + """Whether a manifest registers a microdata release rather than a table.""" + return manifest_kind(manifest, manifest_path=manifest_path) == ( + MICRODATA_RELEASE_KIND + ) + + +# -------------------------------------------------------------------------- +# Access +# -------------------------------------------------------------------------- + + +def normalize_access(access: str | None) -> str: + """Return a validated access class, defaulting to ``public``.""" + if access is None: + return DEFAULT_ACCESS + value = str(access) + if value not in ACCESS_CLASSES: + raise ManifestAccessError( + f"Unknown access class {value!r}; expected one of {list(ACCESS_CLASSES)}." + ) + return value + + +def entry_access(spec: Any) -> str: + """Return the access class a manifest file entry declares or inherits.""" + if not isinstance(spec, Mapping): + return DEFAULT_ACCESS + return normalize_access(spec.get("access")) + + +def safe_entry_access(spec: Any) -> str: + """Return an entry's access class, falling back to ``public`` if unknown. + + An unparseable class is reported by :func:`validate_file_entry`; treating it + as ``public`` here would be unsafe, so it is treated as ``restricted`` and + therefore never uploaded. + """ + if not isinstance(spec, Mapping): + return DEFAULT_ACCESS + try: + return normalize_access(spec.get("access")) + except ManifestAccessError: + return ACCESS_RESTRICTED + + +def strict_entry_access(spec: Any, *, kind: str) -> str: + """Return an entry's access class, refusing to infer one for a release. + + A publisher-table entry that omits ``access`` is public; a microdata + release entry must say what it is, and an unknown class is refused with + the value the manifest actually declares. + """ + if not isinstance(spec, Mapping): + return DEFAULT_ACCESS + declared = spec.get("access") + if declared is None: + if kind == MICRODATA_RELEASE_KIND: + raise ManifestAccessError( + f"Entry {spec.get('filename')!r} declares no access class. A " + "microdata release entry must declare access; Chronicle will " + "not infer public for it." + ) + return DEFAULT_ACCESS + return normalize_access(declared) + + +def stores_bytes(access: str) -> bool: + """Whether Chronicle may hold this access class's bytes.""" + return normalize_access(access) == ACCESS_PUBLIC + + +def is_hash_only(access: str) -> bool: + """Whether this access class must be registered without bytes.""" + return not stores_bytes(access) + + +# -------------------------------------------------------------------------- +# Filenames and vintage keys +# -------------------------------------------------------------------------- + + +def is_bare_filename(value: Any) -> bool: + """Whether ``value`` names a file inside a directory, with no path.""" + text = _text(value) + if text is None or text != str(value) or text in (".", ".."): + return False + if "/" in text or "\\" in text or "\x00" in text: + return False + return Path(text).name == text + + +def bare_filename(value: Any, *, what: str = "filename") -> str: + """Return ``value`` as a bare filename, refusing any other spelling. + + ``./adult.tab``, ``sub/../adult.tab``, ``adult.tab/`` and an absolute path + all resolve to the same file as ``adult.tab`` once joined under the package + directory, so the manifest and every guard use one spelling. + """ + if not is_bare_filename(value): + raise ArtifactFilenameError( + f"{what} must be a bare filename inside the package directory, not " + f"{value!r}; it may not carry a directory, '.', '..', a trailing " + "slash, surrounding whitespace, or an absolute path." + ) + return str(value) + + +def filename_key(value: Any) -> str: + """Return the comparison key for a filename. + + Case-folded, because the package directories these commands run in are as + often as not on a case-insensitive filesystem, where ``ADULT.TAB`` and + ``adult.tab`` are one file. Treating them as one artifact path is the safe + rule everywhere. + """ + return Path(str(value)).name.casefold() + + +def vintage_key_forms(year: Any) -> tuple[Any, ...]: + """Return the key spellings that address the same vintage as ``year``. + + ``2023`` and ``'2023'`` are one vintage: every identity Chronicle derives + from a year (registration ids, R2 keys) renders it as text, and the + source-package reader accepts both. Label keys such as ``'A_1'`` have no + other spelling. + """ + if isinstance(year, bool): + return (year,) + if isinstance(year, int): + return (year, str(year)) + text = str(year) + if text.isdecimal() and (text == "0" or not text.startswith("0")): + return (text, int(text)) + return (year,) + + +def resolve_vintage_key(files: Mapping[Any, Any], year: Any) -> Any | None: + """Return the key ``files`` already uses for ``year``'s vintage, or None. + + Refuses a mapping that records the vintage under both spellings: one + vintage has one key, and Chronicle will not choose which entry is the + record. + """ + present = [form for form in vintage_key_forms(year) if form in files] + if len(present) > 1: + raise AmbiguousVintageKeyError( + f"Vintage {year!r} is recorded under both keys {present!r}; one " + "vintage has one key. Merge the entries by hand first." + ) + return present[0] if present else None + + +def iter_manifest_entries( + manifest: Mapping[str, Any] | None, +) -> Iterator[tuple[Any, int | None, Any]]: + """Yield every ``(key, index, entry)`` a manifest declares, whatever shape. + + Deliberately not gated on the manifest ``kind``: a guard must see the + entries a manifest actually holds, including a list under a manifest whose + kind is absent or misspelled. ``index`` is the entry's position in a list + value and None for a single mapping. + """ + if not isinstance(manifest, Mapping): + return + files = manifest.get("files") + if not isinstance(files, Mapping): + return + for key, spec in files.items(): + if isinstance(spec, list): + for index, entry in enumerate(spec): + yield key, index, entry + else: + yield key, None, spec + + +def has_file_entries(manifest: Mapping[str, Any] | None) -> bool: + """Whether a manifest declares any file entry at all. + + A ``files`` block that is absent, an explicit null (a bare ``files:`` + line) or an empty mapping declares nothing; a vintage key holding an empty + list declares nothing either. A ``files`` value that is not a mapping is + content Chronicle cannot read, and counts as entries so that the kind rule + and the ``files_not_a_mapping`` refusal both fire on it. + """ + if not isinstance(manifest, Mapping): + return False + files = manifest.get("files") + if files is None: + return False + if not isinstance(files, Mapping): + return True + return any(True for _entry in iter_manifest_entries(manifest)) + + +def registration_id( + *, + source_id: str, + package_id: str, + year: Any, + sha256: str, + filename: str, +) -> str: + """Return the registration identity tuple as a stable string.""" + return f"{source_id}/{package_id}/{year}/{sha256}/{filename}" + + +def iter_file_specs(spec: Any, *, kind: str) -> tuple[Any, ...]: + """Expand one ``files[year]`` value into individual file entries. + + A microdata release registers many files under one vintage — the 14 FRS + 2023-24 tabs share ``{source_id, package_id, year}`` and differ only by + ``filename`` and ``sha256`` — so its ``files[year]`` value may be a list. + Publisher-table manifests keep the single-mapping shape, and a list there is + surfaced as a rejected entry rather than silently expanded. + """ + if isinstance(spec, list): + if kind != MICRODATA_RELEASE_KIND: + return (ListSpecRejected(spec),) + return tuple(spec) + return (spec,) + + +# -------------------------------------------------------------------------- +# Validation vocabulary +# -------------------------------------------------------------------------- + + +def validate_manifest_files(manifest: Mapping[str, Any] | None) -> tuple[str, ...]: + """Return manifest-level error codes: shape, key and filename collisions. + + These are properties of the ``files`` mapping as a whole, which no single + entry can see: a vintage recorded under two key spellings, a filename that + is not a bare name, and two entries that resolve to one file in the + package directory while disagreeing about what it holds. + """ + if not isinstance(manifest, Mapping): + return () + files = manifest.get("files") + if files is None: + return () + if not isinstance(files, Mapping): + return ("files_not_a_mapping",) + + errors: list[str] = [] + for key in files: + for other in vintage_key_forms(key): + if other != key and other in files: + errors.append(f"duplicate_vintage_key:{key}") + + # Same file, one package directory: group every entry by its resolved name. + by_name: dict[str, list[tuple[Any, Mapping[str, Any]]]] = {} + for key, _index, entry in iter_manifest_entries(manifest): + if not isinstance(entry, Mapping): + continue + filename = entry.get("filename") + if filename is None: + continue + if not is_bare_filename(filename): + errors.append(f"non_canonical_filename:{filename}") + by_name.setdefault(filename_key(filename), []).append((key, entry)) + + for name, entries in by_name.items(): + if len(entries) < 2: + continue + classes = {is_hash_only(safe_entry_access(entry)) for _key, entry in entries} + if len(classes) > 1: + # One path cannot be both bytes Chronicle holds and bytes it must + # never hold. + errors.append(f"filename_collision:{name}") + continue + hash_only = classes.pop() + seen: dict[Any, set[str]] = {} + for key, entry in entries: + digest = _text(entry.get("sha256")) or "" + vintage = seen.setdefault(key, set()) + if digest in vintage: + errors.append(f"duplicate_filename_in_vintage:{name}") + vintage.add(digest) + if hash_only: + # Several vintages, or an explicit reissue, may register the same + # filename with different bytes: no file exists to collide. + continue + digests = {_text(entry.get("sha256")) or "" for _key, entry in entries} + if len(digests) > 1: + # Public entries share one path in the tree and one current object + # per name; a revision is recorded in storage.previous_r2, not as a + # second entry. + errors.append(f"filename_collision:{name}") + return tuple(_dedupe(errors)) + + +def validate_file_entry( + spec: Any, + *, + kind: str, + manifest: Mapping[str, Any] | None, + local_file_exists: bool, +) -> tuple[str, ...]: + """Return stable error codes for one manifest file entry. + + The codes are the refusal vocabulary shared by ``inventory-artifacts``, + ``publish-raw``, ``fetch-artifact`` and ``register-artifact``. + ``local_file_exists`` says whether the entry's filename exists beside the + manifest, in the package directory. + """ + if isinstance(spec, ListSpecRejected): + return ("list_file_spec_requires_microdata_release_kind",) + if not isinstance(spec, Mapping): + return () + + errors: list[str] = [] + filename = spec.get("filename") + if filename is not None and not is_bare_filename(filename): + errors.append(f"non_canonical_filename:{filename}") + + declared_access = spec.get("access") + if declared_access is None: + if kind == MICRODATA_RELEASE_KIND: + errors.append("missing_access") + access = DEFAULT_ACCESS + else: + try: + access = normalize_access(declared_access) + except ManifestAccessError: + return (*errors, f"unknown_access_class:{declared_access}") + + if kind == MICRODATA_RELEASE_KIND and not _text(spec.get("licence")): + errors.append("missing_licence") + + if is_hash_only(access): + errors.extend( + _hash_only_entry_errors( + spec, + manifest=manifest, + local_file_exists=local_file_exists, + ) + ) + elif kind == MICRODATA_RELEASE_KIND: + errors.extend( + _public_release_entry_errors(spec, local_file_exists=local_file_exists) + ) + if is_hash_only(access) or kind == MICRODATA_RELEASE_KIND: + errors.extend(_attestation_errors(spec)) + return tuple(_dedupe(errors)) + + +def _checksum_errors(spec: Mapping[str, Any]) -> list[str]: + sha256 = _text(spec.get("sha256")) + if not sha256: + return ["missing_sha256"] + if not _SHA256_RE.match(sha256): + return ["malformed_sha256"] + return [] + + +def _hash_only_entry_errors( + spec: Mapping[str, Any], + *, + manifest: Mapping[str, Any] | None, + local_file_exists: bool, +) -> list[str]: + """Return refusal codes for a licensed or restricted registration.""" + errors: list[str] = [] + if not _text(spec.get("licence")): + errors.append("missing_licence") + errors.extend(_checksum_errors(spec)) + if not _text(spec.get("vintage")): + errors.append("missing_vintage") + if not _access_route(spec, manifest): + errors.append("missing_access_route") + if local_file_exists: + errors.append("bytes_present_for_hash_only_entry") + if recorded_r2(spec): + errors.append("r2_location_for_hash_only_entry") + if recorded_previous_r2(spec): + errors.append("r2_history_for_hash_only_entry") + return errors + + +def _public_release_entry_errors( + spec: Mapping[str, Any], + *, + local_file_exists: bool, +) -> list[str]: + """Return refusal codes for a public microdata release entry. + + Bytes are archived only under an allowlisted licence with evidence that + binds this artifact to it, and never inside the package directory: public + microdata is staged outside ``db/data`` and uploaded from there. + """ + errors: list[str] = [] + errors.extend(_checksum_errors(spec)) + if not _text(spec.get("vintage")): + errors.append("missing_vintage") + licence = _text(spec.get("licence")) + if licence and not is_redistributable_licence(licence): + errors.append(f"licence_not_redistributable:{licence}") + errors.extend( + licence_evidence_errors( + spec.get("licence_evidence"), + licence=licence, + sha256=_text(spec.get("sha256")), + ) + ) + if local_file_exists: + errors.append("bytes_present_for_microdata_release_entry") + return errors + + +def _attestation_errors(spec: Mapping[str, Any]) -> list[str]: + """Return refusal codes for an entry's ``hash_source`` and attester.""" + errors: list[str] = [] + hash_source = _text(spec.get("hash_source")) + if not hash_source: + return ["missing_hash_source"] + if hash_source not in HASH_SOURCES: + return [f"unknown_hash_source:{hash_source}"] + attested_by = _text(spec.get("attested_by")) + if not attested_by: + errors.append("missing_attested_by") + verified_at = _text(spec.get("verified_at")) + if hash_source == HASH_SOURCE_CHRONICLE_FETCH: + if attested_by and attested_by != CHRONICLE_ATTESTER: + errors.append("attested_by_not_chronicle") + if not verified_at: + errors.append("missing_verified_at") + elif hash_source == HASH_SOURCE_CONSUMER_ATTESTED: + if not _text(spec.get("attestation_evidence")): + errors.append("missing_attestation_evidence") + if not verified_at: + errors.append("missing_verified_at") + else: + errors.extend(_pinned_from_errors(spec.get("pinned_from"))) + if verified_at: + errors.append("verified_at_forbidden_for_consumer_pin") + return errors + + +def _pinned_from_errors(pinned_from: Any) -> list[str]: + if pinned_from is None: + return ["missing_pinned_from"] + if not isinstance(pinned_from, Mapping): + return ["malformed_pinned_from"] + errors = [ + f"pinned_from_missing_field:{field}" + for field in PINNED_FROM_FIELDS + if not _text(pinned_from.get(field)) + ] + commit = _text(pinned_from.get("commit")) + if commit and not _COMMIT_RE.match(commit): + errors.append("malformed_pinned_from_commit") + return errors + + +def _access_route( + spec: Mapping[str, Any], + manifest: Mapping[str, Any] | None, +) -> str | None: + """Return the recorded route to the bytes, from the entry or the manifest.""" + for key in ("access_route", "source_url", "source_page", "doi"): + value = _text(spec.get(key)) + if value: + return value + if isinstance(manifest, Mapping): + for key in ("source_page", "access_route"): + value = _text(manifest.get(key)) + if value: + return value + return None + + +def recorded_r2(spec: Any) -> Mapping[str, Any] | None: + """Return a recorded ``storage.r2`` mapping, if the entry carries one.""" + if not isinstance(spec, Mapping): + return None + storage = spec.get("storage") + if not isinstance(storage, Mapping): + return None + r2 = storage.get("r2") + return r2 if isinstance(r2, Mapping) else None + + +def recorded_previous_r2(spec: Any) -> tuple[Any, ...]: + """Return the entry's ``storage.previous_r2`` history, if it carries one.""" + if not isinstance(spec, Mapping): + return () + storage = spec.get("storage") + if not isinstance(storage, Mapping): + return () + previous = storage.get("previous_r2") + if isinstance(previous, list): + return tuple(previous) + return (previous,) if previous else () + + +def records_r2_object(spec: Any) -> bool: + """Whether an entry names any object in the raw bucket, current or past.""" + return recorded_r2(spec) is not None or bool(recorded_previous_r2(spec)) + + +def normalize_hash_source(value: Any, *, allowed: Iterable[str] = HASH_SOURCES) -> str: + """Return a validated ``hash_source`` value.""" + text = _text(value) + allowed = tuple(allowed) + if text is None or text not in allowed: + raise ManifestAccessError( + f"Unknown hash_source {value!r}; expected one of {list(allowed)}." + ) + return text + + +# -------------------------------------------------------------------------- +# Hash-only registration +# -------------------------------------------------------------------------- + + +@dataclass(frozen=True) +class ArtifactRegistrationReport: + """Report from registering one hash-only source artifact.""" + + manifest_path: str + source_id: str + package_id: str + year: int + filename: str + sha256: str + size_bytes: int | None + vintage: str + licence: str + access: str + registration: str + replaced: bool + hash_source: str + attested_by: str + errors: tuple[str, ...] = () + + @property + def valid(self) -> bool: + """Whether the registration was written without refusals.""" + return not self.errors + + def to_dict(self) -> dict[str, Any]: + """Return a JSON-serializable report.""" + return { + "valid": self.valid, + "manifest_path": self.manifest_path, + "source_id": self.source_id, + "package_id": self.package_id, + "year": self.year, + "filename": self.filename, + "sha256": self.sha256, + "size_bytes": self.size_bytes, + "vintage": self.vintage, + "licence": self.licence, + "access": self.access, + "registration": self.registration, + "replaced": self.replaced, + "hash_source": self.hash_source, + "attested_by": self.attested_by, + "r2_location": None, + "errors": list(self.errors), + } + + +def register_hash_only_artifact( + *, + source_id: str, + package_id: str, + year: int, + output_dir: str | Path, + filename: str, + sha256: str, + licence: str, + access: str, + vintage: str, + hash_source: str, + attested_by: str, + attestation_evidence: str | None = None, + pinned_from: Mapping[str, Any] | None = None, + verified_at: str | None = None, + size_bytes: int | None = None, + source_page: str | None = None, + source_url: str | None = None, + access_route: str | None = None, + doi: str | None = None, + study: str | None = None, + dataset: str | None = None, + table: str | None = None, + publisher: str | None = None, + fetched_at: str | None = None, + notes: str | None = None, + allow_reissue: bool = False, +) -> ArtifactRegistrationReport: + """Register a licensed or restricted artifact by identity, without bytes. + + Writes (or updates) a ``kind: microdata_release`` manifest entry carrying the + checksum, size, vintage, licence, access route, and the attestation of who + asserts the checksum. No bytes are read, written, or uploaded, and no R2 + key is recorded. Every refusal below happens before the manifest is + touched. + """ + access_class = normalize_access(access) + if stores_bytes(access_class): + raise HashOnlyRegistrationError( + "register-artifact records identity without bytes and refuses " + f"access={ACCESS_PUBLIC!r}. Register a public artifact with its " + "bytes using fetch-artifact." + ) + checksum = _text(sha256) + if not checksum or not _SHA256_RE.match(checksum): + raise HashOnlyRegistrationError( + "A registration needs a lowercase 64-character SHA-256; refusing to " + f"register {filename!r} with sha256={sha256!r}. Never invent a hash." + ) + if not _text(licence): + raise HashOnlyRegistrationError( + f"A {access_class} registration must record the publisher licence." + ) + if not _text(vintage): + raise HashOnlyRegistrationError( + f"A {access_class} registration must record the artifact vintage." + ) + if not is_bare_filename(filename): + raise HashOnlyRegistrationError( + f"Registration filename must be a bare filename; got {filename!r}." + ) + artifact_name = str(filename) + provenance = _hash_only_attestation( + hash_source=hash_source, + attested_by=attested_by, + attestation_evidence=attestation_evidence, + pinned_from=pinned_from, + verified_at=verified_at, + access_class=access_class, + ) + + output = Path(output_dir) + local_path = output / artifact_name + if local_path.exists(): + raise HashOnlyRegistrationError( + f"Refusing to register {artifact_name!r} hash-only while its bytes " + f"are present at {local_path}. A {access_class} artifact's bytes " + "must not live in a Chronicle store." + ) + + manifest_path = output / "manifest.yaml" + payload = _load_manifest(manifest_path) + try: + existing_kind = manifest_kind(payload, manifest_path=manifest_path) + except ManifestAccessError as exc: + raise HashOnlyRegistrationError(str(exc)) from exc + if existing_kind != MICRODATA_RELEASE_KIND and ( + payload.get("kind") is not None or has_file_entries(payload) + ): + # A declared kind is fixed, and a frozen kindless manifest with + # entries is a publisher table. A manifest with neither is declared + # by this write. + raise HashOnlyRegistrationError( + f"{manifest_path} is a {existing_kind} manifest; hash-only " + "registrations belong in a kind: microdata_release manifest." + ) + + entry = _registration_entry( + filename=artifact_name, + access=access_class, + licence=str(licence), + vintage=str(vintage), + sha256=checksum, + size_bytes=size_bytes, + source_url=source_url, + access_route=access_route, + doi=doi, + study=study, + fetched_at=fetched_at, + notes=notes, + **provenance, + ) + route_context = dict(payload) + if source_page: + route_context["source_page"] = source_page + if not _access_route(entry, route_context): + raise HashOnlyRegistrationError( + "A hash-only registration must record how the bytes are reached; " + "pass --access-route, --source-url, --doi, or --source-page." + ) + + _assert_manifest_identity(payload, manifest_path, "source_id", source_id) + _assert_manifest_identity(payload, manifest_path, "package_id", package_id) + files = payload.get("files") + if files is not None and not isinstance(files, dict): + raise HashOnlyRegistrationError( + f"{manifest_path} files must be a mapping; it is a " + f"{type(files).__name__}. Chronicle will not write into a manifest " + "it cannot read." + ) + manifest_errors = validate_manifest_files(payload) + if manifest_errors: + raise HashOnlyRegistrationError( + f"{manifest_path} is not a valid manifest: " + f"{', '.join(manifest_errors)}. Fix it by hand before registering " + "into it." + ) + _assert_no_archived_identity(payload, manifest_path, artifact_name, access_class) + + try: + vintage_key = resolve_vintage_key(files or {}, year) + except AmbiguousVintageKeyError as exc: + raise HashOnlyRegistrationError(f"{manifest_path}: {exc}") from exc + key = vintage_key if vintage_key is not None else year + + payload.setdefault("source_id", source_id) + payload.setdefault("package_id", package_id) + payload["kind"] = MICRODATA_RELEASE_KIND + payload.setdefault("dataset", dataset or f"{source_id}_{package_id}") + if publisher: + payload.setdefault("publisher", publisher) + if source_page: + payload.setdefault("source_page", source_page) + if table: + payload.setdefault("table", table) + if payload.get("files") is None: + # setdefault keeps an explicit null (a bare ``files:`` line); the + # entry below needs a mapping to record into. + payload["files"] = {} + + entries = _existing_entries(payload["files"], key) + wanted = filename_key(artifact_name) + # Two passes, so re-registering an existing pin stays idempotent even after + # a reissue has added a second entry for the same filename. A single pass + # would raise on the first filename match with a different checksum before + # it could reach the exact match further down the list. + replaced = False + for index, existing in enumerate(entries): + if not isinstance(existing, Mapping): + continue + if filename_key(existing.get("filename")) != wanted: + continue + if _text(existing.get("sha256")) == checksum: + entries[index] = entry + replaced = True + break + if not replaced: + superseded = [ + existing + for existing in entries + if isinstance(existing, Mapping) + and filename_key(existing.get("filename")) == wanted + ] + if superseded and not allow_reissue: + raise HashOnlyRegistrationError( + f"{manifest_path} already registers {artifact_name!r} for " + f"{key!r} with sha256={superseded[0].get('sha256')!r}. Different " + "bytes are a new publisher release, not a pin replacement; " + "pass --allow-reissue to register both." + ) + # A reissue sits alongside the pin it supersedes. + entries.append(entry) + + payload["files"][key] = sorted(entries, key=_entry_sort_key) + output.mkdir(parents=True, exist_ok=True) + manifest_path.write_text( + yaml.safe_dump(payload, sort_keys=False, allow_unicode=True), + encoding="utf-8", + ) + + return ArtifactRegistrationReport( + manifest_path=str(manifest_path), + source_id=str(payload["source_id"]), + package_id=str(payload["package_id"]), + year=year, + filename=artifact_name, + sha256=checksum, + size_bytes=size_bytes, + vintage=str(vintage), + licence=str(licence), + access=access_class, + registration=registration_id( + source_id=str(payload["source_id"]), + package_id=str(payload["package_id"]), + year=year, + sha256=checksum, + filename=artifact_name, + ), + replaced=replaced, + hash_source=provenance["hash_source"], + attested_by=provenance["attested_by"], + ) + + +def _hash_only_attestation( + *, + hash_source: Any, + attested_by: Any, + attestation_evidence: Any, + pinned_from: Any, + verified_at: Any, + access_class: str, +) -> dict[str, Any]: + """Validate and return the attestation fields of a hash-only registration. + + Chronicle never fetched the bytes, so the checksum is always the + consumer's: attested against bytes it holds (``consumer_attested``, with + evidence and a date) or transcribed from a reviewed pin in its repository + (``consumer_pin``, with the repository, path and commit and no + verification date). + """ + try: + source = normalize_hash_source(hash_source, allowed=HASH_ONLY_HASH_SOURCES) + except ManifestAccessError as exc: + raise HashOnlyRegistrationError( + f"A {access_class} registration must record how its checksum is " + f"known: {exc} Chronicle holds no bytes for it, so " + f"{HASH_SOURCE_CHRONICLE_FETCH!r} does not apply." + ) from exc + attester = _text(attested_by) + if not attester: + raise HashOnlyRegistrationError( + f"A {source} registration must name the consumer that attests the " + "checksum; pass --attested-by." + ) + fields: dict[str, Any] = {"hash_source": source, "attested_by": attester} + if source == HASH_SOURCE_CONSUMER_ATTESTED: + if not _text(attestation_evidence): + raise HashOnlyRegistrationError( + "A consumer_attested registration must record the consumer's " + "attestation evidence; pass --attestation-evidence." + ) + if not _text(verified_at): + raise HashOnlyRegistrationError( + "A consumer_attested registration must record when the checksum " + "was verified against the bytes; pass --verified-at." + ) + fields["attestation_evidence"] = str(attestation_evidence) + fields["verified_at"] = str(verified_at) + return fields + if pinned_from is None or not isinstance(pinned_from, Mapping): + raise HashOnlyRegistrationError( + "A consumer_pin registration must record where the pin was read: " + "pass --pinned-from-repository, --pinned-from-path and " + "--pinned-from-commit." + ) + errors = _pinned_from_errors(pinned_from) + if errors: + raise HashOnlyRegistrationError( + "A consumer_pin registration's pinned_from must carry the " + f"repository, path and a 40-hex commit: {', '.join(errors)}." + ) + if _text(verified_at): + raise HashOnlyRegistrationError( + "A consumer_pin registration carries no verified_at: Chronicle did " + "not verify the checksum against bytes, it transcribed the " + "consumer's pin. Record the pin's commit instead." + ) + fields["pinned_from"] = { + field: str(pinned_from[field]).strip() for field in PINNED_FROM_FIELDS + } + return fields + + +def _assert_no_archived_identity( + payload: Mapping[str, Any], + manifest_path: Path, + artifact_name: str, + access_class: str, +) -> None: + """Refuse to register hash-only a filename the manifest holds as public. + + A public entry may have been archived: its object sits in the raw bucket + under ``storage.r2`` (or in ``storage.previous_r2`` once revised). + Replacing that entry with a hash-only one would leave the bytes in a + Chronicle store with nothing recording them, and inventory would report + the tree clean. The transition is refused until the public entry, and the + object it names, have been explicitly removed. + """ + wanted = filename_key(artifact_name) + for key, _index, existing in iter_manifest_entries(payload): + if not isinstance(existing, Mapping): + continue + if filename_key(existing.get("filename")) != wanted: + continue + recorded = [ + str(block.get("uri") or block.get("key") or block) + for block in (recorded_r2(existing), *recorded_previous_r2(existing)) + if isinstance(block, Mapping) + ] + if recorded: + raise HashOnlyRegistrationError( + f"{manifest_path} records the R2 object(s) {recorded} for " + f"{existing.get('filename')!r} ({key!r}, " + f"access={safe_entry_access(existing)!r}). Registering it " + f"{access_class} would leave those bytes in a Chronicle store " + "with nothing recording them. Remove the object and its " + "storage record explicitly first; Chronicle will not reclassify " + "an archived release in place." + ) + if not is_hash_only(safe_entry_access(existing)): + raise HashOnlyRegistrationError( + f"{manifest_path} already registers {existing.get('filename')!r} " + f"({key!r}) as access={safe_entry_access(existing)!r}. A change " + f"of access class to {access_class!r} is an explicit decision: " + "remove the public entry by hand, then register the release." + ) + + +def _assert_manifest_identity( + payload: Mapping[str, Any], + manifest_path: Path, + key: str, + value: str, +) -> None: + """Refuse to register into a manifest that identifies a different source.""" + existing = _text(payload.get(key)) + if existing is not None and existing != value: + raise HashOnlyRegistrationError( + f"{manifest_path} declares {key}={existing!r}; refusing to register " + f"{key}={value!r} into it." + ) + + +def _entry_sort_key(entry: Any) -> tuple[str, str]: + """Return a deterministic sort key for a registration entry.""" + if not isinstance(entry, Mapping): + return ("", "") + return (_text(entry.get("filename")) or "", _text(entry.get("sha256")) or "") + + +def _registration_entry(**values: Any) -> dict[str, Any]: + """Build a deterministic, field-ordered registration entry.""" + entry: dict[str, Any] = {} + for key in _REGISTRATION_FIELD_ORDER: + value = values.get(key) + if value is None: + continue + if isinstance(value, str) and not value.strip(): + continue + entry[key] = value + return entry + + +def _existing_entries(files: Any, key: Any) -> list[Any]: + """Return the existing file entries under a vintage key as a mutable list.""" + if not isinstance(files, dict): + return [] + spec = files.get(key) + if spec is None: + return [] + if isinstance(spec, list): + return list(spec) + return [spec] + + +def _load_manifest(manifest_path: Path) -> dict[str, Any]: + """Load a manifest mapping, or an empty mapping when absent.""" + if not manifest_path.exists(): + return {} + payload = yaml.safe_load(manifest_path.read_text(encoding="utf-8")) or {} + if not isinstance(payload, dict): + raise HashOnlyRegistrationError(f"Manifest must be a mapping: {manifest_path}") + return payload + + +def _text(value: Any) -> str | None: + """Return a non-empty stripped string, or None.""" + if value is None: + return None + text = str(value).strip() + return text or None + + +def _dedupe(values: Iterable[str]) -> list[str]: + """Return values with duplicates removed, preserving order.""" + seen: set[str] = set() + ordered: list[str] = [] + for value in values: + if value not in seen: + seen.add(value) + ordered.append(value) + return ordered + + +__all__ = [ + "ACCESS_CLASSES", + "ACCESS_LICENSED", + "ACCESS_PUBLIC", + "ACCESS_RESTRICTED", + "AmbiguousVintageKeyError", + "ArtifactFilenameError", + "ArtifactRegistrationReport", + "CHRONICLE_ATTESTER", + "DEFAULT_ACCESS", + "DEFAULT_MANIFEST_KIND", + "HASH_ONLY_HASH_SOURCES", + "HASH_SOURCES", + "HASH_SOURCE_CHRONICLE_FETCH", + "HASH_SOURCE_CONSUMER_ATTESTED", + "HASH_SOURCE_CONSUMER_PIN", + "HashOnlyRegistrationError", + "ListSpecRejected", + "MANIFEST_KINDS", + "MICRODATA_RELEASE_KIND", + "ManifestAccessError", + "ManifestKindError", + "MicrodataReleaseNotParseableError", + "PINNED_FROM_FIELDS", + "PUBLISHER_TABLE_KIND", + "bare_filename", + "entry_access", + "filename_key", + "has_file_entries", + "is_bare_filename", + "is_hash_only", + "is_microdata_release", + "iter_file_specs", + "iter_manifest_entries", + "manifest_kind", + "normalize_access", + "normalize_hash_source", + "recorded_previous_r2", + "recorded_r2", + "records_r2_object", + "register_hash_only_artifact", + "registration_id", + "resolve_vintage_key", + "safe_entry_access", + "safe_manifest_kind", + "stores_bytes", + "strict_entry_access", + "validate_file_entry", + "validate_manifest_files", + "vintage_key_forms", +] diff --git a/chronicle/source_package.py b/chronicle/source_package.py index 2a6320ff..390fa9e7 100644 --- a/chronicle/source_package.py +++ b/chronicle/source_package.py @@ -3,7 +3,6 @@ from __future__ import annotations import hashlib -import os from dataclasses import dataclass, replace from importlib.resources import files from io import BytesIO @@ -32,7 +31,19 @@ AggregateFact, build_label, ) +from chronicle.env import env_flag, env_value from chronicle.epoch import SCHEMA_IDS, schema_id +from chronicle.registration import ( + AmbiguousVintageKeyError, + ManifestAccessError, + ManifestKindError, + MicrodataReleaseNotParseableError, + entry_access, + is_bare_filename, + is_hash_only, + is_microdata_release, + resolve_vintage_key, +) from chronicle.sources.cells import ( SourceArtifactMetadata, SourceCell, @@ -373,8 +384,8 @@ "usda_snap/fy2025_monthly_state_caseloads" ), } -SOURCE_ARTIFACT_CACHE_ENV = "LEDGER_SOURCE_ARTIFACT_CACHE_DIR" -SOURCE_ARTIFACT_FETCH_ENV = "LEDGER_SOURCE_ARTIFACT_FETCH" +SOURCE_ARTIFACT_CACHE_ENV = "CHRONICLE_SOURCE_ARTIFACT_CACHE_DIR" +SOURCE_ARTIFACT_FETCH_ENV = "CHRONICLE_SOURCE_ARTIFACT_FETCH" DEFAULT_SOURCE_ARTIFACT_CACHE_DIR = ( Path.home() / ".cache" / "policyengine-chronicle" / "source-artifacts" ) @@ -849,17 +860,60 @@ def _source_artifact_metadata( raw_r2_uri=raw_r2.get("uri"), ) - def _artifact_content( - self, - year: int, - ) -> tuple[bytes, str, str, dict[str, str]]: - manifest_path = files(self.resource_package).joinpath( + def manifest_resource(self) -> Any: + """Return the manifest file this package spec points at.""" + return files(self.resource_package).joinpath( self.resource_directory, self.manifest, ) - with manifest_path.open("r", encoding="utf-8") as file: - manifest = yaml.safe_load(file) + + def manifest_payload(self) -> dict[str, Any]: + """Load the artifact manifest this package spec points at.""" + with self.manifest_resource().open("r", encoding="utf-8") as file: + return yaml.safe_load(file) or {} + + def assert_parseable_manifest(self) -> None: + """Refuse to parse a manifest that registers a microdata release. + + Microdata registration is manifest-level identity: no source package + parses a release, and no microdata row, cell, or fact enters Chronicle + (``docs/adr-chronicle-raw-microdata-identity.md``). A manifest that + declares no kind and is not frozen kindless is refused too: the reader + never assumes a publisher table. + """ + manifest = self.manifest_payload() + if is_microdata_release(manifest, manifest_path=self.manifest_resource()): + raise MicrodataReleaseNotParseableError( + f"{self.resource_directory}/{self.manifest} registers a " + "microdata release. Registration is identity only: no source " + "package parses a microdata release and no microdata rows, " + "cells, or facts enter Chronicle." + ) + + def assert_parseable(self, year: int) -> dict[str, Any]: + """Return the entry a parse would read, refusing any it must not. + + Two carve-outs, decided from the manifest alone before any byte is + read: the manifest-level microdata-release kind, and the selected + entry's own access class -- a licensed or restricted entry is identity + only whatever manifest it sits in. + """ + self.assert_parseable_manifest() + manifest = self.manifest_payload() spec = _year_mapping(manifest["files"], self.artifact_year or year) + _assert_entry_bytes_readable(spec) + return spec + + def _artifact_content( + self, + year: int, + ) -> tuple[bytes, str, str, dict[str, str]]: + spec = self.assert_parseable(year) + if not is_bare_filename(spec.get("filename")): + raise ValueError( + f"Source artifact filename must be a bare filename inside " + f"{self.resource_directory}, not {spec.get('filename')!r}." + ) artifact_path = files(self.resource_package).joinpath( self.resource_directory, spec["filename"], @@ -1211,6 +1265,58 @@ def validate_source_package( errors=tuple(errors), ) + try: + package.artifact.assert_parseable_manifest() + except (MicrodataReleaseNotParseableError, ManifestKindError) as exc: + errors.append( + SourcePackageIssue( + code=( + "microdata_release_not_parseable" + if isinstance(exc, MicrodataReleaseNotParseableError) + else "manifest_kind_missing" + ), + message=str(exc), + ) + ) + return SourcePackageValidationReport( + package_id=package.package_id, + package_path=str(package.package_path), + year=year, + counts=counts, + errors=tuple(errors), + warnings=tuple(warnings), + ) + except (FileNotFoundError, OSError, ValueError) as exc: + errors.append( + SourcePackageIssue( + code="source_artifact_manifest_unreadable", + message=str(exc), + ) + ) + else: + try: + package.artifact.assert_parseable(year) + except ManifestAccessError as exc: + # Decided from the manifest alone: no package tree, cache, or + # publisher is consulted for an entry a parser must never read. + errors.append( + SourcePackageIssue( + code="hash_only_artifact_not_parseable", + message=str(exc), + ) + ) + return SourcePackageValidationReport( + package_id=package.package_id, + package_path=str(package.package_path), + year=year, + counts=counts, + errors=tuple(errors), + warnings=tuple(warnings), + ) + except (FileNotFoundError, KeyError, OSError, ValueError): + # Reported below by the artifact read itself. + pass + try: package.artifact._artifact_content(year) except (FileNotFoundError, KeyError, OSError, ValueError) as exc: @@ -2217,18 +2323,70 @@ def _required(payload: dict[str, Any], key: str, context: str) -> Any: def _year_mapping(files_by_year: dict[Any, Any], year: int) -> dict[str, str]: - if year in files_by_year: - return files_by_year[year] - if str(year) in files_by_year: - return files_by_year[str(year)] - raise ValueError(f"No source artifact for year {year}") + """Return the file spec for ``year``, whichever key spelling records it. + + ``2023`` and ``'2023'`` are one vintage; a manifest that records both is + refused rather than silently read through the integer key, which would + hide whichever entry -- often the one carrying the R2 history -- a writer + left under the other spelling. + """ + try: + key = resolve_vintage_key(files_by_year, year) + except AmbiguousVintageKeyError as exc: + raise ValueError(f"Source artifact for year {year}: {exc}") from exc + if key is None: + raise ValueError(f"No source artifact for year {year}") + return _single_year_spec(files_by_year[key], year) + + +def _assert_entry_bytes_readable(spec: Any) -> None: + """Refuse to read a hash-only entry's bytes from any store. + + This is the lowest byte-reader boundary: it runs before the package tree, + the content-addressed cache, or the publisher is consulted, so a licensed + or restricted entry is never read, cached, fetched, or parsed -- whatever + manifest kind it sits under. An access class Chronicle cannot parse is + refused too, never read as public. + """ + if not isinstance(spec, dict): + return + access = entry_access(spec) + if is_hash_only(access): + raise ManifestAccessError( + f"Source artifact {spec.get('filename')!r} is registered as " + f"access={access!r}. Its bytes must not enter a Chronicle store or " + "a parser: a licensed or restricted artifact is identity only, so " + "no source package reads, caches, fetches, or parses it " + "(docs/adr-chronicle-raw-microdata-identity.md)." + ) + + +def _single_year_spec(spec: Any, year: int) -> dict[str, str]: + """Return one file spec, refusing the multi-file microdata-release shape. + + Only a ``kind: microdata_release`` manifest may list several files under one + vintage, and no source package parses one of those, so a list here is a + malformed publisher-table manifest rather than something to index into. + """ + if isinstance(spec, list): + raise ValueError( + f"Source artifact for year {year} is a list of {len(spec)} entries. " + "Only a kind: microdata_release manifest may list several files " + "under one vintage, and no source package parses one." + ) + return spec def _read_source_artifact_content( artifact_path: Any, spec: dict[str, Any], ) -> bytes: - """Read a source artifact from package data, cache, or explicit fetch.""" + """Read a source artifact from package data, cache, or explicit fetch. + + Refuses a hash-only entry before touching any of the three: none of them + may hold its bytes, and the fetch branch would write them into the cache. + """ + _assert_entry_bytes_readable(spec) try: return artifact_path.read_bytes() except FileNotFoundError: @@ -2238,7 +2396,7 @@ def _read_source_artifact_content( if cache_path.exists(): return cache_path.read_bytes() - if not _truthy_env(SOURCE_ARTIFACT_FETCH_ENV): + if not env_flag(SOURCE_ARTIFACT_FETCH_ENV): raise FileNotFoundError( f"Source artifact {spec['filename']} is not packaged and was not " f"found in {cache_path}. Set {SOURCE_ARTIFACT_FETCH_ENV}=1 to fetch " @@ -2260,7 +2418,7 @@ def _read_source_artifact_content( def _source_artifact_cache_path(spec: dict[str, Any]) -> Path: cache_root = Path( - _env_value( + env_value( SOURCE_ARTIFACT_CACHE_ENV, default=DEFAULT_SOURCE_ARTIFACT_CACHE_DIR, ) @@ -2297,21 +2455,6 @@ def _validate_source_artifact_sha( ) -def _env_value(*names: str, default: str | Path) -> str | Path: - for name in names: - value = os.environ.get(name) - if value: - return value - return default - - -def _truthy_env(*names: str) -> bool: - return any( - os.environ.get(name, "").strip().lower() in {"1", "true", "yes", "on"} - for name in names - ) - - def _single_archive_member(archive: ZipFile, *, suffixes: tuple[str, ...]) -> str: members = [ name diff --git a/chronicle/suite.py b/chronicle/suite.py index 157b7420..731b553d 100644 --- a/chronicle/suite.py +++ b/chronicle/suite.py @@ -22,7 +22,11 @@ build_fact_key, validate_facts, ) -from chronicle.database import ChronicleDbBuildReport, build_chronicle_db +from chronicle.database import ( + CHRONICLE_DB_FILENAME, + ChronicleDbBuildReport, + build_chronicle_db, +) from chronicle.epoch import canonicalize_key from chronicle.sources.cells import ( SourceCell, @@ -285,6 +289,10 @@ def build_source_suite( source_package = try_load_source_package(source) source_id = source_package.package_id if source_package else source output_path = Path(output_dir) + if source_package is not None: + # Validate, then touch: a package whose artifact must not be parsed is + # refused before the output directory is created or replaced. + source_package.artifact.assert_parseable(year) _prepare_output_dir(output_path, replace=replace) reports_path = output_path / "reports" reports_path.mkdir(parents=True, exist_ok=True) @@ -372,7 +380,7 @@ def build_source_suite( concept_report.to_dict(), ) - db_path = output_path / "ledger.db" + db_path = output_path / CHRONICLE_DB_FILENAME db_report = build_chronicle_db( facts, db_path, @@ -1675,7 +1683,7 @@ def _write_package_sidecars(output_path: Path, *, source: str, year: int) -> Non output_path / "source_regions.jsonl", output_path / "facts.jsonl", output_path / "consumer_facts.jsonl", - output_path / "ledger.db", + output_path / CHRONICLE_DB_FILENAME, output_path / "reports" / "source_rows.json", output_path / "reports" / "source_cells.json", output_path / "reports" / "source_regions.json", diff --git a/db/cli.py b/db/cli.py index 52257240..66538149 100644 --- a/db/cli.py +++ b/db/cli.py @@ -276,14 +276,15 @@ def cmd_query(args): def _pe_source_root_env_default(jurisdiction: str) -> str | None: + from chronicle.env import env_value + from .pe_source_inventory import ( PE_UK_DATA_ROOT_ENV, PE_US_DATA_ROOT_ENV, - _env_value, ) env_var = PE_US_DATA_ROOT_ENV if jurisdiction == "us" else PE_UK_DATA_ROOT_ENV - return _env_value(env_var) + return env_value(env_var) def main(): diff --git a/db/data/dwp/frs_2023_24/manifest.yaml b/db/data/dwp/frs_2023_24/manifest.yaml new file mode 100644 index 00000000..604adda3 --- /dev/null +++ b/db/data/dwp/frs_2023_24/manifest.yaml @@ -0,0 +1,330 @@ +source_id: dwp +package_id: dwp-frs-2023-24 +kind: microdata_release +dataset: dwp_dwp-frs-2023-24 +publisher: Department for Work and Pensions +table: Family Resources Survey 2023-24 +files: + 2023: + - filename: accounts.tab + access: licensed + licence: UK Data Service End User Licence + vintage: '2023_24' + sha256: c5e31932bfd06087f835d2c83c0984c85a93409bf5ef85b699cb0958abcba1ea + size_bytes: 1807921 + access_route: UK Data Service study SN 9367 under its End User Licence. Bytes + stay in the licensed environment the consumer already operates; no Chronicle + credential grants access to them. + doi: 10.5255/UKDA-SN-9367-2 + study: UK Data Service SN 9367 + hash_source: consumer_pin + attested_by: PolicyEngine/microcosm + pinned_from: + repository: PolicyEngine/microcosm + path: packages/microcosm-build/src/microcosm/build/uk/source_stages.json + commit: 2fb2e2f8a99c37725bd6e7a15ff4c2595c912b77 + notes: Microcosm's frs_spine stage cites 'UK Data Service SN 9367, DOI 10.5255/UKDA-SN-9367-2'; + its frs_employment, frs_council_tax, frs_education, and frs_legacy_proxies stages + cite 'SN 9252' for the same 2023_24 tabs. The tabs are the same bytes across + all five stages (identical SHA-256), so the registration carries the study reference + that also carries a DOI. SHA-256 transcribed verbatim from the consumer's reviewed + pin; Chronicle holds no bytes for this release and did not recompute the checksum. + - filename: adult.tab + access: licensed + licence: UK Data Service End User Licence + vintage: '2023_24' + sha256: e09f9647d03585c81a528636028b2ed495f8f1fbcf64c5e7b4fe521b67367e06 + size_bytes: 35323384 + access_route: UK Data Service study SN 9367 under its End User Licence. Bytes + stay in the licensed environment the consumer already operates; no Chronicle + credential grants access to them. + doi: 10.5255/UKDA-SN-9367-2 + study: UK Data Service SN 9367 + hash_source: consumer_pin + attested_by: PolicyEngine/microcosm + pinned_from: + repository: PolicyEngine/microcosm + path: packages/microcosm-build/src/microcosm/build/uk/source_stages.json + commit: 2fb2e2f8a99c37725bd6e7a15ff4c2595c912b77 + notes: Microcosm's frs_spine stage cites 'UK Data Service SN 9367, DOI 10.5255/UKDA-SN-9367-2'; + its frs_employment, frs_council_tax, frs_education, and frs_legacy_proxies stages + cite 'SN 9252' for the same 2023_24 tabs. The tabs are the same bytes across + all five stages (identical SHA-256), so the registration carries the study reference + that also carries a DOI. SHA-256 transcribed verbatim from the consumer's reviewed + pin; Chronicle holds no bytes for this release and did not recompute the checksum. + - filename: benefits.tab + access: licensed + licence: UK Data Service End User Licence + vintage: '2023_24' + sha256: ff30d054cc659bcf23b44c492d98cfd701c0bfdb63e8e9aa9769b490ba9d636b + size_bytes: 4460292 + access_route: UK Data Service study SN 9367 under its End User Licence. Bytes + stay in the licensed environment the consumer already operates; no Chronicle + credential grants access to them. + doi: 10.5255/UKDA-SN-9367-2 + study: UK Data Service SN 9367 + hash_source: consumer_pin + attested_by: PolicyEngine/microcosm + pinned_from: + repository: PolicyEngine/microcosm + path: packages/microcosm-build/src/microcosm/build/uk/source_stages.json + commit: 2fb2e2f8a99c37725bd6e7a15ff4c2595c912b77 + notes: Microcosm's frs_spine stage cites 'UK Data Service SN 9367, DOI 10.5255/UKDA-SN-9367-2'; + its frs_employment, frs_council_tax, frs_education, and frs_legacy_proxies stages + cite 'SN 9252' for the same 2023_24 tabs. The tabs are the same bytes across + all five stages (identical SHA-256), so the registration carries the study reference + that also carries a DOI. SHA-256 transcribed verbatim from the consumer's reviewed + pin; Chronicle holds no bytes for this release and did not recompute the checksum. + - filename: benunit.tab + access: licensed + licence: UK Data Service End User Licence + vintage: '2023_24' + sha256: 88946815eace8561516d5cbb442c27e319c1e90abc381fb2338f0126e3b9e05b + size_bytes: 21213867 + access_route: UK Data Service study SN 9367 under its End User Licence. Bytes + stay in the licensed environment the consumer already operates; no Chronicle + credential grants access to them. + doi: 10.5255/UKDA-SN-9367-2 + study: UK Data Service SN 9367 + hash_source: consumer_pin + attested_by: PolicyEngine/microcosm + pinned_from: + repository: PolicyEngine/microcosm + path: packages/microcosm-build/src/microcosm/build/uk/source_stages.json + commit: 2fb2e2f8a99c37725bd6e7a15ff4c2595c912b77 + notes: Microcosm's frs_spine stage cites 'UK Data Service SN 9367, DOI 10.5255/UKDA-SN-9367-2'; + its frs_employment, frs_council_tax, frs_education, and frs_legacy_proxies stages + cite 'SN 9252' for the same 2023_24 tabs. The tabs are the same bytes across + all five stages (identical SHA-256), so the registration carries the study reference + that also carries a DOI. SHA-256 transcribed verbatim from the consumer's reviewed + pin; Chronicle holds no bytes for this release and did not recompute the checksum. + - filename: child.tab + access: licensed + licence: UK Data Service End User Licence + vintage: '2023_24' + sha256: b5dc84fe8b002ee925e61fae23fed27b11537af9fb174f1d07d9cc1748b9702e + size_bytes: 2913156 + access_route: UK Data Service study SN 9367 under its End User Licence. Bytes + stay in the licensed environment the consumer already operates; no Chronicle + credential grants access to them. + doi: 10.5255/UKDA-SN-9367-2 + study: UK Data Service SN 9367 + hash_source: consumer_pin + attested_by: PolicyEngine/microcosm + pinned_from: + repository: PolicyEngine/microcosm + path: packages/microcosm-build/src/microcosm/build/uk/source_stages.json + commit: 2fb2e2f8a99c37725bd6e7a15ff4c2595c912b77 + notes: Microcosm's frs_spine stage cites 'UK Data Service SN 9367, DOI 10.5255/UKDA-SN-9367-2'; + its frs_employment, frs_council_tax, frs_education, and frs_legacy_proxies stages + cite 'SN 9252' for the same 2023_24 tabs. The tabs are the same bytes across + all five stages (identical SHA-256), so the registration carries the study reference + that also carries a DOI. SHA-256 transcribed verbatim from the consumer's reviewed + pin; Chronicle holds no bytes for this release and did not recompute the checksum. + - filename: chldcare.tab + access: licensed + licence: UK Data Service End User Licence + vintage: '2023_24' + sha256: 566e0ebca1d5e2f3e424e556c91f4cb583d17dadfdfa59feb3841eda7e5976a3 + size_bytes: 273837 + access_route: UK Data Service study SN 9367 under its End User Licence. Bytes + stay in the licensed environment the consumer already operates; no Chronicle + credential grants access to them. + doi: 10.5255/UKDA-SN-9367-2 + study: UK Data Service SN 9367 + hash_source: consumer_pin + attested_by: PolicyEngine/microcosm + pinned_from: + repository: PolicyEngine/microcosm + path: packages/microcosm-build/src/microcosm/build/uk/source_stages.json + commit: 2fb2e2f8a99c37725bd6e7a15ff4c2595c912b77 + notes: Microcosm's frs_spine stage cites 'UK Data Service SN 9367, DOI 10.5255/UKDA-SN-9367-2'; + its frs_employment, frs_council_tax, frs_education, and frs_legacy_proxies stages + cite 'SN 9252' for the same 2023_24 tabs. The tabs are the same bytes across + all five stages (identical SHA-256), so the registration carries the study reference + that also carries a DOI. SHA-256 transcribed verbatim from the consumer's reviewed + pin; Chronicle holds no bytes for this release and did not recompute the checksum. + - filename: extchild.tab + access: licensed + licence: UK Data Service End User Licence + vintage: '2023_24' + sha256: 8d358d7ee66ee4a7ceab87b4f24fbbf21ac86dc038dc7831e51fb271f96a57ec + size_bytes: 18677 + access_route: UK Data Service study SN 9367 under its End User Licence. Bytes + stay in the licensed environment the consumer already operates; no Chronicle + credential grants access to them. + doi: 10.5255/UKDA-SN-9367-2 + study: UK Data Service SN 9367 + hash_source: consumer_pin + attested_by: PolicyEngine/microcosm + pinned_from: + repository: PolicyEngine/microcosm + path: packages/microcosm-build/src/microcosm/build/uk/source_stages.json + commit: 2fb2e2f8a99c37725bd6e7a15ff4c2595c912b77 + notes: Microcosm's frs_spine stage cites 'UK Data Service SN 9367, DOI 10.5255/UKDA-SN-9367-2'; + its frs_employment, frs_council_tax, frs_education, and frs_legacy_proxies stages + cite 'SN 9252' for the same 2023_24 tabs. The tabs are the same bytes across + all five stages (identical SHA-256), so the registration carries the study reference + that also carries a DOI. SHA-256 transcribed verbatim from the consumer's reviewed + pin; Chronicle holds no bytes for this release and did not recompute the checksum. + - filename: househol.tab + access: licensed + licence: UK Data Service End User Licence + vintage: '2023_24' + sha256: 5fd26b8b675f33b3b30c9ac789a18da17de734790f77e00ded287d1c3a187b30 + size_bytes: 12387117 + access_route: UK Data Service study SN 9367 under its End User Licence. Bytes + stay in the licensed environment the consumer already operates; no Chronicle + credential grants access to them. + doi: 10.5255/UKDA-SN-9367-2 + study: UK Data Service SN 9367 + hash_source: consumer_pin + attested_by: PolicyEngine/microcosm + pinned_from: + repository: PolicyEngine/microcosm + path: packages/microcosm-build/src/microcosm/build/uk/source_stages.json + commit: 2fb2e2f8a99c37725bd6e7a15ff4c2595c912b77 + notes: Microcosm's frs_spine stage cites 'UK Data Service SN 9367, DOI 10.5255/UKDA-SN-9367-2'; + its frs_employment, frs_council_tax, frs_education, and frs_legacy_proxies stages + cite 'SN 9252' for the same 2023_24 tabs. The tabs are the same bytes across + all five stages (identical SHA-256), so the registration carries the study reference + that also carries a DOI. SHA-256 transcribed verbatim from the consumer's reviewed + pin; Chronicle holds no bytes for this release and did not recompute the checksum. + - filename: job.tab + access: licensed + licence: UK Data Service End User Licence + vintage: '2023_24' + sha256: 88b77ffe06865f029f713bb1d55ff12bdea8a1234de5bc293e72458fe64f3a74 + size_bytes: 10934873 + access_route: UK Data Service study SN 9367 under its End User Licence. Bytes + stay in the licensed environment the consumer already operates; no Chronicle + credential grants access to them. + doi: 10.5255/UKDA-SN-9367-2 + study: UK Data Service SN 9367 + hash_source: consumer_pin + attested_by: PolicyEngine/microcosm + pinned_from: + repository: PolicyEngine/microcosm + path: packages/microcosm-build/src/microcosm/build/uk/source_stages.json + commit: 2fb2e2f8a99c37725bd6e7a15ff4c2595c912b77 + notes: Microcosm's frs_spine stage cites 'UK Data Service SN 9367, DOI 10.5255/UKDA-SN-9367-2'; + its frs_employment, frs_council_tax, frs_education, and frs_legacy_proxies stages + cite 'SN 9252' for the same 2023_24 tabs. The tabs are the same bytes across + all five stages (identical SHA-256), so the registration carries the study reference + that also carries a DOI. SHA-256 transcribed verbatim from the consumer's reviewed + pin; Chronicle holds no bytes for this release and did not recompute the checksum. + - filename: maint.tab + access: licensed + licence: UK Data Service End User Licence + vintage: '2023_24' + sha256: f2dc924eb5a51b0c357791693d15b431327dc39c6421011efb313d88bf839695 + size_bytes: 15440 + access_route: UK Data Service study SN 9367 under its End User Licence. Bytes + stay in the licensed environment the consumer already operates; no Chronicle + credential grants access to them. + doi: 10.5255/UKDA-SN-9367-2 + study: UK Data Service SN 9367 + hash_source: consumer_pin + attested_by: PolicyEngine/microcosm + pinned_from: + repository: PolicyEngine/microcosm + path: packages/microcosm-build/src/microcosm/build/uk/source_stages.json + commit: 2fb2e2f8a99c37725bd6e7a15ff4c2595c912b77 + notes: Microcosm's frs_spine stage cites 'UK Data Service SN 9367, DOI 10.5255/UKDA-SN-9367-2'; + its frs_employment, frs_council_tax, frs_education, and frs_legacy_proxies stages + cite 'SN 9252' for the same 2023_24 tabs. The tabs are the same bytes across + all five stages (identical SHA-256), so the registration carries the study reference + that also carries a DOI. SHA-256 transcribed verbatim from the consumer's reviewed + pin; Chronicle holds no bytes for this release and did not recompute the checksum. + - filename: mortgage.tab + access: licensed + licence: UK Data Service End User Licence + vintage: '2023_24' + sha256: ce36b477d67837c469608a0d68f7ef269ac04758974235f1157d2f6b92cdbfdc + size_bytes: 631783 + access_route: UK Data Service study SN 9367 under its End User Licence. Bytes + stay in the licensed environment the consumer already operates; no Chronicle + credential grants access to them. + doi: 10.5255/UKDA-SN-9367-2 + study: UK Data Service SN 9367 + hash_source: consumer_pin + attested_by: PolicyEngine/microcosm + pinned_from: + repository: PolicyEngine/microcosm + path: packages/microcosm-build/src/microcosm/build/uk/source_stages.json + commit: 2fb2e2f8a99c37725bd6e7a15ff4c2595c912b77 + notes: Microcosm's frs_spine stage cites 'UK Data Service SN 9367, DOI 10.5255/UKDA-SN-9367-2'; + its frs_employment, frs_council_tax, frs_education, and frs_legacy_proxies stages + cite 'SN 9252' for the same 2023_24 tabs. The tabs are the same bytes across + all five stages (identical SHA-256), so the registration carries the study reference + that also carries a DOI. SHA-256 transcribed verbatim from the consumer's reviewed + pin; Chronicle holds no bytes for this release and did not recompute the checksum. + - filename: oddjob.tab + access: licensed + licence: UK Data Service End User Licence + vintage: '2023_24' + sha256: b4ba3dd3151f73a01422983c60514a3e38458ddfa4fb33ae4ed0326873406305 + size_bytes: 5165 + access_route: UK Data Service study SN 9367 under its End User Licence. Bytes + stay in the licensed environment the consumer already operates; no Chronicle + credential grants access to them. + doi: 10.5255/UKDA-SN-9367-2 + study: UK Data Service SN 9367 + hash_source: consumer_pin + attested_by: PolicyEngine/microcosm + pinned_from: + repository: PolicyEngine/microcosm + path: packages/microcosm-build/src/microcosm/build/uk/source_stages.json + commit: 2fb2e2f8a99c37725bd6e7a15ff4c2595c912b77 + notes: Microcosm's frs_spine stage cites 'UK Data Service SN 9367, DOI 10.5255/UKDA-SN-9367-2'; + its frs_employment, frs_council_tax, frs_education, and frs_legacy_proxies stages + cite 'SN 9252' for the same 2023_24 tabs. The tabs are the same bytes across + all five stages (identical SHA-256), so the registration carries the study reference + that also carries a DOI. SHA-256 transcribed verbatim from the consumer's reviewed + pin; Chronicle holds no bytes for this release and did not recompute the checksum. + - filename: penprov.tab + access: licensed + licence: UK Data Service End User Licence + vintage: '2023_24' + sha256: ee001461c40306ec24b38b2881e1774121114266a2ee449d606cd0a811c37731 + size_bytes: 522313 + access_route: UK Data Service study SN 9367 under its End User Licence. Bytes + stay in the licensed environment the consumer already operates; no Chronicle + credential grants access to them. + doi: 10.5255/UKDA-SN-9367-2 + study: UK Data Service SN 9367 + hash_source: consumer_pin + attested_by: PolicyEngine/microcosm + pinned_from: + repository: PolicyEngine/microcosm + path: packages/microcosm-build/src/microcosm/build/uk/source_stages.json + commit: 2fb2e2f8a99c37725bd6e7a15ff4c2595c912b77 + notes: Microcosm's frs_spine stage cites 'UK Data Service SN 9367, DOI 10.5255/UKDA-SN-9367-2'; + its frs_employment, frs_council_tax, frs_education, and frs_legacy_proxies stages + cite 'SN 9252' for the same 2023_24 tabs. The tabs are the same bytes across + all five stages (identical SHA-256), so the registration carries the study reference + that also carries a DOI. SHA-256 transcribed verbatim from the consumer's reviewed + pin; Chronicle holds no bytes for this release and did not recompute the checksum. + - filename: pension.tab + access: licensed + licence: UK Data Service End User Licence + vintage: '2023_24' + sha256: 150d6fad1fce81254fb7aea1526fbb00b63d4027d6e2ac4c26bb90aea3127eb7 + size_bytes: 1225838 + access_route: UK Data Service study SN 9367 under its End User Licence. Bytes + stay in the licensed environment the consumer already operates; no Chronicle + credential grants access to them. + doi: 10.5255/UKDA-SN-9367-2 + study: UK Data Service SN 9367 + hash_source: consumer_pin + attested_by: PolicyEngine/microcosm + pinned_from: + repository: PolicyEngine/microcosm + path: packages/microcosm-build/src/microcosm/build/uk/source_stages.json + commit: 2fb2e2f8a99c37725bd6e7a15ff4c2595c912b77 + notes: Microcosm's frs_spine stage cites 'UK Data Service SN 9367, DOI 10.5255/UKDA-SN-9367-2'; + its frs_employment, frs_council_tax, frs_education, and frs_legacy_proxies stages + cite 'SN 9252' for the same 2023_24 tabs. The tabs are the same bytes across + all five stages (identical SHA-256), so the registration carries the study reference + that also carries a DOI. SHA-256 transcribed verbatim from the consumer's reviewed + pin; Chronicle holds no bytes for this release and did not recompute the checksum. diff --git a/db/data/hmrc/spi_public_use_tape_2022_23/manifest.yaml b/db/data/hmrc/spi_public_use_tape_2022_23/manifest.yaml new file mode 100644 index 00000000..3080319e --- /dev/null +++ b/db/data/hmrc/spi_public_use_tape_2022_23/manifest.yaml @@ -0,0 +1,32 @@ +source_id: hmrc +package_id: hmrc-spi-public-use-tape-2022-23 +kind: microdata_release +dataset: hmrc_hmrc-spi-public-use-tape-2022-23 +publisher: HM Revenue and Customs +table: Survey of Personal Incomes Public Use Tape 2022-23 +files: + 2022: + - filename: put2223uk.tab + access: restricted + licence: UK Data Service End User Licence (study SN 9422) + vintage: 2022-23 + sha256: 5ef829461060c91a2a47be59ad541d9b519fc3976d66ca80d4920f711bb96f66 + size_bytes: 141323762 + access_route: UK Data Service study SN 9422. Microcosm reaches the bytes through + PolicyEngine's licensed copy in the private policyengine/policyengine-uk-data-private + Hugging Face repository (spi_2022_23.zip); no Chronicle credential grants access + to them. + doi: 10.5255/UKDA-SN-9422-1 + study: UK Data Service SN 9422 + hash_source: consumer_pin + attested_by: PolicyEngine/microcosm + pinned_from: + repository: PolicyEngine/microcosm + path: packages/microcosm-build/src/microcosm/build/uk/hmrc_income_source_stages.json + commit: de7451bd19ca46d2967e73cdf393908d29e72542 + notes: 'Microcosm classes this artifact kind: private_microdata with access: private_local_input. + Chronicle registers it restricted because the bytes are held only in a private + mirror; if the UKDS terms for SN 9422 are confirmed as End User Licence it can + be reclassified licensed, which changes nothing about storage — both classes + are hash-only. SHA-256 transcribed verbatim from the consumer''s reviewed pin; + Chronicle holds no bytes for this release and did not recompute the checksum.' diff --git a/db/pe_source_inventory.py b/db/pe_source_inventory.py index a52fff23..f64ed870 100644 --- a/db/pe_source_inventory.py +++ b/db/pe_source_inventory.py @@ -2,14 +2,15 @@ from __future__ import annotations -import os from pathlib import Path +from chronicle.env import env_value + from .schema import Jurisdiction from .source_files import SourceArtifactSpec, make_slug, make_url_slug -PE_US_DATA_ROOT_ENV = "LEDGER_PE_US_DATA_ROOT" -PE_UK_DATA_ROOT_ENV = "LEDGER_PE_UK_DATA_ROOT" +PE_US_DATA_ROOT_ENV = "CHRONICLE_PE_US_DATA_ROOT" +PE_UK_DATA_ROOT_ENV = "CHRONICLE_PE_UK_DATA_ROOT" SOURCE_SUFFIXES = { ".csv", @@ -272,21 +273,13 @@ ] -def _env_value(*names: str) -> str | None: - for name in names: - value = os.environ.get(name) - if value: - return value - return None - - def _resolve_required_root( root: Path | None, *, flag: str, env_var: str, ) -> Path: - value = root if root is not None else _env_value(env_var) + value = root if root is not None else env_value(env_var) if value is None: raise ValueError(f"{flag} or {env_var} is required.") path = Path(value).expanduser() @@ -612,8 +605,8 @@ def pe_source_specs( ) -> list[SourceArtifactSpec]: """Return source files used by the PE-US and PE-UK calibration pipelines.""" specs: list[SourceArtifactSpec] = [] - us_configured = pe_us_root is not None or _env_value(PE_US_DATA_ROOT_ENV) - uk_configured = pe_uk_root is not None or _env_value(PE_UK_DATA_ROOT_ENV) + us_configured = pe_us_root is not None or env_value(PE_US_DATA_ROOT_ENV) + uk_configured = pe_uk_root is not None or env_value(PE_UK_DATA_ROOT_ENV) if include_us and (us_configured or not include_uk or not uk_configured): specs.extend(pe_us_source_specs(pe_us_root)) if include_uk and (uk_configured or not include_us or not us_configured): diff --git a/db/supabase_client.py b/db/supabase_client.py index e14bb5bb..190d8b7e 100644 --- a/db/supabase_client.py +++ b/db/supabase_client.py @@ -8,7 +8,6 @@ from __future__ import annotations -import os from dataclasses import dataclass from functools import lru_cache from typing import Any, Dict, List, Optional @@ -16,18 +15,33 @@ from supabase import create_client, Client +from chronicle.env import default_chronicle_schema, env_value -def _env(*names: str) -> str | None: - """Read PolicyEngine-owned storage config.""" - for name in names: - value = os.environ.get(name) - if value: - return value - return None +TARGETS_SCHEMA_ENV = "POLICYENGINE_TARGETS_SCHEMA" +DEFAULT_TARGETS_SCHEMA = "targets" -LEDGER_SCHEMA = _env("POLICYENGINE_LEDGER_SCHEMA") or "ledger" -TARGETS_SCHEMA = _env("POLICYENGINE_TARGETS_SCHEMA") or "targets" +def chronicle_schema() -> str: + """Resolve the hosted Chronicle schema for a query. + + Read at call time, not bound at import: an import-time constant fixes the + schema at whatever the environment held when this module was first + imported, which the caller does not control (in the test suite that moment + is collection, before any fixture has isolated the environment). The + hosted schema is still named "ledger" -- only the variable that overrides + it has moved to the chronicle prefix, and renaming the schema value is a + later slice of PolicyEngine/chronicle#143. + """ + return default_chronicle_schema() + + +def targets_schema() -> str: + """Resolve the hosted targets schema. Read at call time, as above. + + ``POLICYENGINE_TARGETS_SCHEMA`` names a surface outside the ledger rename + window, so it is read literally. + """ + return env_value(TARGETS_SCHEMA_ENV, default=DEFAULT_TARGETS_SCHEMA) @dataclass @@ -49,14 +63,14 @@ def from_env(cls) -> "SupabaseConfig": Raises: ValueError: If required environment variables are missing """ - url = _env("POLICYENGINE_SUPABASE_URL") + url = env_value("POLICYENGINE_SUPABASE_URL") if not url: raise ValueError( "POLICYENGINE_SUPABASE_URL not set. " "Set this to your Supabase project URL." ) - secret_key = _env( + secret_key = env_value( "POLICYENGINE_SUPABASE_SERVICE_KEY", "POLICYENGINE_SUPABASE_SECRET_KEY", ) @@ -114,7 +128,7 @@ def query_sources( List of source records """ client = get_supabase_client() - query = _table(client, LEDGER_SCHEMA, "sources").select("*") + query = _table(client, chronicle_schema(), "sources").select("*") if jurisdiction: query = query.eq("jurisdiction", jurisdiction) @@ -143,7 +157,9 @@ def query_strata( List of strata records with nested constraints """ client = get_supabase_client() - query = _table(client, TARGETS_SCHEMA, "strata").select("*, stratum_constraints(*)") + query = _table(client, targets_schema(), "strata").select( + "*, stratum_constraints(*)" + ) if jurisdiction: query = query.eq("jurisdiction", jurisdiction) @@ -172,7 +188,7 @@ def query_targets( """ client = get_supabase_client() # Nested join: strata with their stratum_constraints - query = _table(client, TARGETS_SCHEMA, "targets").select( + query = _table(client, targets_schema(), "targets").select( "*, strata(*, stratum_constraints(*)), sources(*)" ) @@ -219,7 +235,7 @@ def insert_targets_batch( for i in range(0, len(targets), chunk_size): chunk = targets[i : i + chunk_size] - _table(client, TARGETS_SCHEMA, "targets").insert(chunk).execute() + _table(client, targets_schema(), "targets").insert(chunk).execute() total += len(chunk) return total diff --git a/docs/agent-source-package-harness.md b/docs/agent-source-package-harness.md index 5816ec47..909ed14c 100644 --- a/docs/agent-source-package-harness.md +++ b/docs/agent-source-package-harness.md @@ -14,21 +14,189 @@ lineage, provenance, constraints, and a passing `build-suite` report. The first gate for a new package is source-artifact acquisition. Agents should register raw source files with `uv run chronicle fetch-artifact` before authoring selectors. This writes the local artifact, captures checksum and retrieval -metadata in `manifest.yaml`, and can upload the exact bytes to the private -`ledger-raw` R2 bucket when Wrangler is authenticated. Agents can audit the local +metadata in `manifest.yaml`, and can upload the exact bytes to the private raw +R2 bucket (`ledger-raw` today; overridable with `CHRONICLE_R2_RAW_BUCKET`) when +Wrangler is authenticated. A publisher directory that feeds several source +packages keeps one manifest each, so pass `--manifest ` to address +the right one. Agents can audit the local artifact registry with `uv run chronicle inventory-artifacts --root db/data`. For already-downloaded manifest artifacts, agents should run `uv run chronicle publish-raw --root db/data` to upload checksum-verified bytes to R2 and write `storage.r2` metadata back into each manifest entry. +Both commands treat a manifest entry as a claim about specific bytes: by its +declared `sha256` from the moment it is registered, and by the content-addressed +key of its recorded `storage.r2` block once it is published. Re-fetching or +publishing bytes the entry does not identify is refused; when a publisher has re-published +under the same URL and vintage, register the revision with +`uv run chronicle fetch-artifact ... --record-revision`, which stores the new +bytes under their own key and keeps the superseded object in +`storage.previous_r2`. See +[Publisher Revisions](storage-architecture.md#publisher-revisions). + Builds do not require production raw bytes to be committed to Git. Source packages first read packaged fixture bytes, then -`LEDGER_SOURCE_ARTIFACT_CACHE_DIR` (defaulting to +`CHRONICLE_SOURCE_ARTIFACT_CACHE_DIR` (defaulting to `~/.cache/policyengine-chronicle/source-artifacts`). If a manifest artifact is -missing locally, set `LEDGER_SOURCE_ARTIFACT_FETCH=1` to fetch it from the +missing locally, set `CHRONICLE_SOURCE_ARTIFACT_FETCH=1` to fetch it from the manifest `source_url`, verify the declared SHA-256, and write it to that cache. -The old `CHRONICLE_`-prefixed environment variables remain accepted only as -migration fallbacks. +The ledger-era spellings `LEDGER_SOURCE_ARTIFACT_CACHE_DIR` and +`LEDGER_SOURCE_ARTIFACT_FETCH` are still honored during the rename window and +emit a one-time deprecation warning naming the `CHRONICLE_` variable to set +instead; see "Environment Variable Rename Window" in +[`docs/storage-architecture.md`](storage-architecture.md#environment-variable-rename-window). + +## Manifest Kinds + +Every manifest declares `kind`: `publisher_table` (one file per vintage, +parsed by a source package) or `microdata_release` (a registered release, +never parsed). The manifests that predate this rule are frozen, byte for +byte, in `chronicle/grandfathered_manifests.py` and read as publisher tables +only while they still match the freeze; `fetch-artifact` writes `kind` onto +any manifest it touches, so a frozen manifest leaves the freeze the first time +it is modified. A kindless manifest outside that list is an error at every +entry point — `fetch-artifact`, `publish-raw`, `inventory-artifacts`, +`validate-package` and the source-package byte reader — and never a publisher +table by default. A manifest that declares no file entry yet (a bare +`files:` line or an empty mapping) has nothing to classify, and the command +writing its first entry declares the kind. A manifest's kind is fixed once +declared: `fetch-artifact --kind` must match it, and a conflicting kind is +refused before anything is read. + +## Hash-Only Registrations + +Not every raw artifact a build starts from may be redistributed. Every +manifest file entry carries an `access` class from a closed set — `public`, +`licensed`, or `restricted` — and a `licence` naming the publisher's terms. +`public` is inferred when a publisher-table entry omits `access`, and +`fetch-artifact` writes the class explicitly onto every entry it touches. +Both fields are required on a `kind: microdata_release` manifest, as is the +attestation of who asserts the checksum: `hash_source` is one of +`chronicle_fetch` (`attested_by: chronicle`, `verified_at` = fetch date), +`consumer_attested` (`attested_by` = the consumer, `attestation_evidence`, +`verified_at`) or `consumer_pin` (`attested_by` = the consumer, `pinned_from` += repository, path and commit, and no `verified_at`). + +Only `public` bytes enter a Chronicle store. A `licensed` or `restricted` +artifact is registered *hash-only*: the manifest records the checksum, size, +vintage, licence, access route, and attestation, and no bytes are fetched, +written, or uploaded. Chronicle never fetched the bytes, so a hash-only +checksum is always the consumer's. Agents should register one with: + +```bash +uv run chronicle register-artifact \ + --source-id dwp \ + --package-id dwp-frs-2023-24 \ + --year 2023 \ + --out-dir db/data/dwp/frs_2023_24 \ + --filename adult.tab \ + --sha256 e09f9647d03585c81a528636028b2ed495f8f1fbcf64c5e7b4fe521b67367e06 \ + --size-bytes 35323384 \ + --vintage 2023_24 \ + --licence "UK Data Service End User Licence" \ + --access licensed \ + --doi 10.5255/UKDA-SN-9367-2 \ + --hash-source consumer_pin \ + --attested-by PolicyEngine/microcosm \ + --pinned-from-repository PolicyEngine/microcosm \ + --pinned-from-path packages/microcosm-build/src/microcosm/build/uk/source_stages.json \ + --pinned-from-commit 2fb2e2f8a99c37725bd6e7a15ff4c2595c912b77 +``` + +Agents should never invent a checksum to satisfy the command: `--sha256` must +be a lowercase 64-character digest taken from a reviewed pin, and a release +whose checksum nobody has published is a blocker to record, not a value to +guess. The command refuses `--access public`, refuses bytes sitting beside the +manifest, refuses to write into a `publisher_table` manifest, and refuses to +reclassify a filename the manifest already holds as `public` — an archived +release's object must be removed, with its `storage` record, before the entry +can become hash-only. + +Filenames are bare names compared case-insensitively: `./adult.tab`, +`ADULT.TAB` and `sub/../adult.tab` are the file `adult.tab`, and a manifest +never holds one path under two access classes. `2023` and `'2023'` are one +vintage key; a manifest that records both is refused rather than read through +one of them. Every command validates the whole manifest with the codes +`inventory-artifacts` reports before reading, writing, or uploading anything. + +The other commands enforce the same boundary from their side. `fetch-artifact` +refuses a `licensed` or `restricted` access class before reading anything, and +refuses to pull bytes over an entry already registered hash-only — including +when the filename is inferred from the URL, which happens before the read. +`publish-raw` refuses such an entry without reading or uploading its bytes; +pass `--skip-hash-only` to publish a tree that deliberately mixes both kinds. +`inventory-artifacts` treats a hash-only entry with no local file as valid — +the absent bytes are the correct state — and reports an error if the bytes +appear. The source-package byte reader refuses a hash-only entry before +consulting the package tree, the content-addressed cache, or the publisher, +whatever manifest kind it sits under; `validate-package` reports it as +`hash_only_artifact_not_parseable`, and `build-suite` refuses before creating +its output directory. + +A `public` microdata release is different: its bytes are redistributable, so +they are archived — but only with artifact-bound evidence. Being downloadable +is not a licence. The `licence` must be one of the allowlisted terms in +`chronicle/licences.py` (`US-Government-Work`, `OGL-UK-3.0`, `CC0-1.0`, +`CC-BY-4.0`), and the entry carries `licence_evidence` binding this file to +that term: the issuer, the licence identifier, a scope statement, a durable +evidence URL, and the reviewed SHA-256. The fetch therefore takes +`--expected-sha256` and refuses, before writing or uploading, bytes that hash +differently; `--record-revision` does not override that. Release bytes are +staged in a transient directory outside the repository +(`$CHRONICLE_MICRODATA_STAGING_DIR`, default +`~/.cache/policyengine-chronicle/microdata-staging`) and uploaded from there; +a file of that name beside the manifest is refused as tracked microdata bytes, +and the test suite guards that no release package tracks anything but its +manifest. Pass `--kind microdata_release` so the manifest declares what it is: + +```bash +uv run chronicle fetch-artifact \ + --url https://www2.census.gov/programs-surveys/acs/data/pums/2022/1-Year/csv_hus.zip \ + --source-id census_acs \ + --package-id census-acs-pums-2022-1yr \ + --year 2022 \ + --out-dir db/data/census/acs_pums_2022_1yr \ + --publisher "U.S. Census Bureau" \ + --vintage 2022 \ + --access public \ + --licence US-Government-Work \ + --kind microdata_release \ + --expected-sha256 \ + --licence-evidence-issuer "U.S. Census Bureau" \ + --licence-evidence-scope "Public-use file of a federal agency; 17 U.S.C. §105" \ + --licence-evidence-url \ + --upload-r2 +``` + +The entry records `hash_source: chronicle_fetch`, `attested_by: chronicle` +and the fetch date. A public release without licence evidence is classed +`licensed` and registered hash-only instead. + +Because several files can share one vintage, a `kind: microdata_release` +manifest gives `files[year]` as a list of entries rather than a single +mapping — the ACS household and person files above land side by side. A list +under any other manifest kind is an error. A fetch replaces only the entry for +its own filename, in place, so acquiring a second file never drops the first, +and re-fetching different bytes for one of them is a publisher revision under +the same guard as any other entry. + +No source package parses a microdata release. `validate-package` fails with +`microdata_release_not_parseable` if a package spec points at one, and no +microdata row, cell, or fact ever enters Chronicle. Registration is +manifest-level identity; see `docs/adr-chronicle-raw-microdata-identity.md`. + +`scripts/register_microdata_releases.py` drives both halves from a read-only +PolicyEngine/microcosm checkout: `emit` writes the hash-only manifests from +Microcosm's reviewed pins as `consumer_pin` registrations (recording the +consumer manifest's path and the last commit that changed it, or the commit +given with `--microcosm-commit [PATH=]COMMIT`), and `plan` prints the +`fetch-artifact` commands to run for public releases from a networked machine +with every reviewed identity as an argument. Anything Microcosm does not pin +prints as a `TODO` the command refuses to run with. The suite exercises the +script against `tests/fixtures/microcosm`, a synthetic snapshot of the +consumer manifests; re-deriving the catalogue means re-snapshotting that +fixture and regenerating `tests/fixtures/microcosm/golden_plan.json` in the +same change. For broad PE source migration, generate the agent queue from the manifest before assigning work: @@ -603,16 +771,19 @@ uv run chronicle build-suite packages/irs_soi/table_1_1 \ --require-axiom-validation ``` -The SQLite `ledger.db` is the source of hosted mirrors. To prepare tables for +The SQLite `chronicle.db` is the source of hosted mirrors. To prepare tables for Supabase/Postgres bulk loading, export the DB artifact rather than inserting cells through the Supabase client: ```bash -uv run chronicle export-db-tables --db /tmp/chronicle-suite/ledger.db --out /tmp/chronicle-mirror --replace +uv run chronicle export-db-tables --db /tmp/chronicle-suite/chronicle.db --out /tmp/chronicle-mirror --replace ``` -Accepted build-suite outputs can be published to the private `ledger-derived` R2 -bucket after validation: +Builds produced before the rename wrote `ledger.db`. That name is still read and +published unchanged, so point `--db` at whichever file the build emitted. + +Accepted build-suite outputs can be published to the private derived R2 bucket +after validation: ```bash uv run chronicle publish-derived \ diff --git a/docs/data-sources/be-silc-2023-registration-blocker.md b/docs/data-sources/be-silc-2023-registration-blocker.md new file mode 100644 index 00000000..53ad7ae4 --- /dev/null +++ b/docs/data-sources/be-silc-2023-registration-blocker.md @@ -0,0 +1,79 @@ +# BE-SILC 2023: Registration Blocker + +The Belgian SILC scientific-use files are a `restricted_microdata` root of +Microcosm's Belgian build, so they belong in Chronicle's microdata registry +under `docs/adr-chronicle-raw-microdata-identity.md`. They are **not +registered yet**, and this document records why. + +## Blocker + +A registration is identified by `{source_id, package_id, year, sha256, +filename}`. Microcosm's pin carries none of the last two. The whole artifact +entry in `packages/microcosm-build/src/microcosm/build/be/source_stages.json` +(stage `silc_load`) is: + +```json +{ + "format": "csv_or_spss", + "kind": "restricted_microdata", + "licence": "Statbel/Eurostat scientific-use; restricted — private artifacts only", + "locator": "Statbel BE-SILC scientific-use files: D (household register), R (personal register), H (household data), P (personal data)", + "vintage": "2023" +} +``` + +There is no `sha256`, no `size_bytes`, and no per-file `filename` — the +locator names four file *roles*, not four files. Every other pinned microdata +artifact in Microcosm carries a reviewed checksum; this one does not. + +`chronicle register-artifact` refuses the release rather than accepting a +placeholder, and `scripts/register_microdata_releases.py` reports it as a +blocker instead of emitting a manifest. No checksum is invented for a file +Chronicle has never seen and may never hold. + +## What The Registration Will Record Once Unblocked + +Everything except the identity is already known and is held in the script's +catalogue entry `statbel-be-silc-2023`: + +| Field | Value | +|-------|-------| +| `source_id` | `statbel` | +| `package_id` | `statbel-be-silc-2023` | +| `year` | 2023 | +| `access` | `restricted` | +| `licence` | Statbel/Eurostat scientific-use | +| `vintage` | 2023 | +| `source_page` | | +| `access_route` | Statbel BE-SILC scientific-use files: D, R, H, P | + +The access class is `restricted`, so the registration is hash-only whatever +the checksums turn out to be: no BE-SILC bytes enter any Chronicle store, and +no `ledger-raw` key exists for them. + +## To Unblock + +1. Microcosm publishes a reviewed SHA-256, size, and exact filename for each of + the four scientific-use files, in `be/source_stages.json` (tracked on the + consumer side in PolicyEngine/microcosm#848). +2. Re-run the emitter, which will pick the pins up with no catalogue change: + + ```bash + uv run python scripts/register_microdata_releases.py \ + --microcosm-root ~/PolicyEngine/microcosm \ + --root db/data \ + --release statbel-be-silc-2023 \ + emit + ``` + + The registration is a `consumer_pin`: it names PolicyEngine/microcosm as + the attester and records the consumer manifest's path and the commit the + pins were read from; it carries no `verified_at` of its own. + +3. Delete the `blocker` field from the `statbel-be-silc-2023` catalogue entry, + and delete this document. + +Until then `uv run chronicle inventory-artifacts --root db/data` reports 15 +hash-only registrations — the 14 DWP Family Resources Survey 2023-24 tabs and +the HMRC Survey of Personal Incomes Public Use Tape 2022-23 — and BE-SILC is +absent by design. diff --git a/docs/storage-architecture.md b/docs/storage-architecture.md index cf898933..d2c33848 100644 --- a/docs/storage-architecture.md +++ b/docs/storage-architecture.md @@ -9,17 +9,24 @@ class of Chronicle data belongs. Chronicle uses three storage layers with different jobs. -`ledger-raw` is the immutable source-byte archive. It stores exact publisher +The raw archive is the immutable source-byte store. It holds exact publisher artifacts as fetched: workbooks, CSVs, PDFs, ZIPs, HTML snapshots, and similar government-statistics release files. Raw objects are content-addressed by checksum and should never be overwritten in place. -`ledger-derived` is the reproducible artifact archive. It stores build outputs +The derived archive is the reproducible artifact store. It holds build outputs that Chronicle can regenerate from raw bytes, package specs, parser code, and build configuration. Examples include parsed-cell or parsed-row Parquet/JSONL files, -source record outputs, `ledger.db`, mirror JSONL exports, QA reports, Data +source record outputs, `chronicle.db`, mirror JSONL exports, QA reports, Data Package metadata, and RO-Crate metadata. +Both bucket names are configuration, not constants. The raw archive is +`$CHRONICLE_R2_RAW_BUCKET` and the derived archive is +`$CHRONICLE_R2_DERIVED_BUCKET`; the shipped defaults are still the ledger-era +`ledger-raw` and `ledger-derived`. [Bucket Cutover](#bucket-cutover) records how +those defaults move to `chronicle-raw` and `chronicle-derived` and why the +ledger-era buckets are preserved read-only rather than retired. + Supabase/Postgres is the queryable relational registry for accepted Chronicle builds. It stores rows that applications, agents, and downstream systems need to search and join: source artifacts, source rows/cells, source records, @@ -32,8 +39,8 @@ Hosted tables mirror accepted build outputs and provide a shared query surface. ## Ownership Matrix -| Data class | Git/local package | `ledger-raw` R2 | `ledger-derived` R2 | SQLite `ledger.db` | Supabase/Postgres | -|------------|-------------------|---------------|-------------------|------------------|-------------------| +| Data class | Git/local package | Raw R2 | Derived R2 | SQLite `chronicle.db` | Supabase/Postgres | +|------------|-------------------|--------|------------|-----------------------|-------------------| | Source package specs | Authoritative YAML and parser code | No | Optional packaged snapshot | No | Metadata only | | Raw publisher files | Tiny fixtures only | Authoritative bytes | No | Metadata only | Metadata plus R2 pointer | | Source manifests | Authoritative checked metadata | No | Optional snapshot | Metadata loaded into tables | Queryable artifact registry | @@ -76,14 +83,131 @@ Examples: ```text derived/uk/ons/ons-mye-2024-uk/2024/{build_id}/source_cells.jsonl -derived/nz/ird/ird-working-for-families-statistics-sept-2025/2024/{build_id}/ledger.db +derived/nz/ird/ird-working-for-families-statistics-sept-2025/2024/{build_id}/chronicle.db ``` Legacy US derived keys likewise remain `derived/{source_id}/...`. +A registered microdata release uses the same content-addressed key as any +other raw artifact, and it exists only when the release's `access` class is +`public`. A Census public-use file is a US publisher, so it keeps the legacy +shape: + +```text +raw/census_cps/census-cps-asec-2023/2023/{sha256}/asecpub23csv.zip +``` + +A `licensed` or `restricted` release has no key at all. Its registration is +`{source_id, package_id, year, sha256, filename}` recorded in `manifest.yaml`, +and no object is written to either bucket — so the DWP Family Resources Survey +tabs, which would otherwise route to `raw/uk/dwp/...`, occupy no key. See +`docs/adr-chronicle-raw-microdata-identity.md`. + +A public release's key exists only once the fetch has verified the bytes +against the reviewed `--expected-sha256` its `licence_evidence` covers. Its +bytes never enter the repository: they are staged under +`$CHRONICLE_MICRODATA_STAGING_DIR` (default +`~/.cache/policyengine-chronicle/microdata-staging`), keyed like the R2 object, +and uploaded from there. The source-artifact cache under +`~/.cache/policyengine-chronicle/source-artifacts` is a Chronicle store too: +the byte reader refuses a licensed or restricted entry before it would read, +fetch into, or serve from that cache. + Derived artifacts are reproducible and may be replaced by a new build, but a specific `{build_id}` path should be immutable once published. +## Publisher Revisions + +A raw key embeds the sha256 of the bytes it holds, so a manifest's recorded +`storage.r2` block is a claim about specific bytes, not a pointer to a file +name. Publishers do not always honor that: on 2026-09-02 the IRS re-published +`22in05ira.xlsx` and `22in06ira.xlsx` under their existing URLs +(PolicyEngine/chronicle#225). + +`fetch-artifact` therefore compares the entry's recorded identity with the +bytes it just fetched, before it writes anything. That identity is the recorded +key's `{sha256}/{filename}` tail once the entry has been published, and the +entry's own declared `sha256` before then — an entry that was registered +without an upload, or whose upload failed, still identifies its bytes, and gets +the same protection: + +- **Identical** — the recorded block is preserved exactly, whichever bucket is + configured now. Re-fetching after the bucket rename copies bytes; it does not + restate where they were first published. +- **Different** — the fetch is refused. Nothing is overwritten: not the cached + artifact, not the manifest entry, and no object is uploaded. The error names + the recorded and the fetched `sha256`/`size_bytes`. Per + `docs/adr-chronicle-fact-identity-v2.md`, the same vintage with new bytes is a + new release revision, so registering it is a decision an operator makes, not a + silent rewrite. + +`--record-revision` makes that decision explicit. The fetched bytes get their +own content-addressed key under the configured bucket — never the recorded key — +and the superseded block moves to `storage.previous_r2`: + +```yaml +files: + 2022: + filename: 22in05ira.xlsx + sha256: + size_bytes: + fetched_at: "2026-09-02T17:04:11+00:00" + storage: + r2: + provider: r2 + bucket: chronicle-raw + key: raw/irs_soi/soi-table-5/2022//22in05ira.xlsx + uri: r2://chronicle-raw/raw/irs_soi/soi-table-5/2022//22in05ira.xlsx + previous_r2: + - provider: r2 + bucket: ledger-raw + key: raw/irs_soi/soi-table-5/2022//22in05ira.xlsx + uri: r2://ledger-raw/raw/irs_soi/soi-table-5/2022//22in05ira.xlsx + sha256: + size_bytes: + fetched_at: "2026-06-11T14:22:05+00:00" + source_url: https://www.irs.gov/pub/irs-soi/22in05ira.xlsx + superseded_at: "2026-09-02T17:04:11+00:00" +``` + +`storage.r2` only ever names the object that holds the entry's current bytes, +and `previous_r2` lists superseded objects oldest first, so the bytes an +archived witness record pinned stay addressable at the URI it pinned. Every +reader — `inventory-artifacts`, `publish-raw`, source-package artifact loading, +and the suite's raw-R2-link acceptance check — reads `storage.r2` alone, so a +revised entry reads exactly like an unrevised one; `publish-raw` preserves the +rest of the `storage` block when it writes back. + +`publish-raw` applies the same identity check before treating a recorded block +as history. A local file the recorded object does not hold is reported as +`recorded_r2_identity_mismatch` and nothing is uploaded: registering a revision +is a fetch-time decision, not a publish-time rewrite. + +### Which manifest + +Most packages keep one `manifest.yaml`. A publisher directory that feeds +several source packages keeps one manifest each — +`db/data/irs_soi/ira_contributions/` holds +`manifest_traditional_source_package.yaml` beside +`manifest_roth_source_package.yaml` — and the entry being revised lives in +exactly one of them. `fetch-artifact --manifest ` selects it; +defaulting to `manifest.yaml` there would write a third manifest neither +package reads, and the recorded block would never be compared at all. The name +must be a filename inside `--out-dir`, not a path. + +### What a recorded block has to say + +A `storage.r2` block's `provider`, `bucket`, `key` and `uri` all describe one +object, so every field that is present is cross-checked against every other: +the key against the URI's path, the bucket against its authority, the provider +against its scheme, and the resulting key against the content-addressed +`{sha256}/{filename}` shape. A block whose fields disagree does not answer +"which bytes does this entry claim R2 holds", so it is an error rather than +something to preserve or publish under. Likewise a manifest that parses as +anything other than a mapping is refused rather than treated as absent — +reading it as absent would let the next fetch replace the file with a single +entry. + ## Relational Registry Contract The hosted `chronicle` schema should be the lookup surface for Chronicle, not the place @@ -114,20 +238,24 @@ The intended flow is: 1. Register raw source artifacts with `uv run chronicle fetch-artifact`, which writes local bytes, records checksums in `manifest.yaml`, and can upload the - exact bytes to `ledger-raw`. Existing manifest-declared artifacts can be + exact bytes to the raw archive. Re-fetching an entry whose bytes the + publisher has changed is refused unless the revision is registered with + `--record-revision`; see [Publisher Revisions](#publisher-revisions). + Existing manifest-declared artifacts can be checksum-validated, uploaded, and linked with `uv run chronicle publish-raw`. Production package specs may omit raw bytes from Git as long as the manifest keeps `source_url` and SHA-256 metadata; builds can fill - `LEDGER_SOURCE_ARTIFACT_CACHE_DIR` by setting - `LEDGER_SOURCE_ARTIFACT_FETCH=1`. The old `CHRONICLE_`-prefixed environment - variables remain accepted only as migration fallbacks. + `CHRONICLE_SOURCE_ARTIFACT_CACHE_DIR` by setting + `CHRONICLE_SOURCE_ARTIFACT_FETCH=1`. Ledger-era spellings of both still work; + see [Environment Variable Rename Window](#environment-variable-rename-window). 2. Validate and build a source package with `uv run chronicle validate-package` and `uv run chronicle build-suite`. 3. Produce local deterministic outputs: parsed rows/cells, source records, - aggregate facts, `ledger.db`, QA reports, Data Package metadata, and RO-Crate - metadata. + aggregate facts, `chronicle.db`, QA reports, Data Package metadata, and + RO-Crate metadata. Builds before this rename wrote `ledger.db`; every reader + still accepts that name. 4. Export relational mirror files with `uv run chronicle export-db-tables`. -5. Publish derived build outputs to `ledger-derived`: +5. Publish derived build outputs to the derived archive: ```bash uv run chronicle publish-derived \ @@ -150,6 +278,176 @@ The Supabase project must have the checked migration applied and the `chronicle` schema exposed in PostgREST/Data API settings before the REST loader can write to it. Use `--dry-run` to verify local JSONL files without writing. +## Environment Variable Rename Window + +Every Chronicle setting is read chronicle-first by one shared helper, +`chronicle/env.py`. For a setting `X`, the lookup order is: + +1. `CHRONICLE_X` +2. `POLICYENGINE_LEDGER_X` +3. `LEDGER_X` + +The first name that holds a non-empty value wins. When that name is a ledger-era +one, the process emits a single `ChronicleEnvDeprecationWarning` naming the +`CHRONICLE_`-prefixed variable to set instead. The warning fires once per legacy +name per process, and it subclasses `FutureWarning` rather than +`DeprecationWarning` so it actually reaches operators running the CLI. + +Two consequences are worth stating outright, because both are the reverse of +what a naive fallback would do: + +- The `CHRONICLE_` name wins even when its value reads false. An operator who + has migrated can set `CHRONICLE_SOURCE_ARTIFACT_FETCH=0` and have the flag + turn off, without first hunting down a stale `LEDGER_SOURCE_ARTIFACT_FETCH=1` + somewhere in their profile. +- An empty value counts as unset, so exporting an empty `CHRONICLE_` name does + not mask a set legacy name. + +| Chronicle name | Ledger-era names still accepted | Meaning | +|----------------|--------------------------------|---------| +| `CHRONICLE_SOURCE_ARTIFACT_CACHE_DIR` | `LEDGER_SOURCE_ARTIFACT_CACHE_DIR` | Where fetched raw bytes are cached; defaults to `~/.cache/policyengine-chronicle/source-artifacts` | +| `CHRONICLE_SOURCE_ARTIFACT_FETCH` | `LEDGER_SOURCE_ARTIFACT_FETCH` | Fetch a missing manifest artifact from its `source_url` during a build | +| `CHRONICLE_PE_US_DATA_ROOT` | `LEDGER_PE_US_DATA_ROOT` | Local checkout root for PE US source inventory | +| `CHRONICLE_PE_UK_DATA_ROOT` | `LEDGER_PE_UK_DATA_ROOT` | Local checkout root for PE UK source inventory | +| `CHRONICLE_SCHEMA` | `POLICYENGINE_LEDGER_SCHEMA`, `LEDGER_SCHEMA` | Postgres schema the Supabase client reads and `load-supabase-mirror` writes; defaults to `ledger` | +| `CHRONICLE_R2_RAW_BUCKET` | `LEDGER_R2_RAW_BUCKET` | Raw R2 archive bucket; defaults to `ledger-raw` | +| `CHRONICLE_R2_DERIVED_BUCKET` | `LEDGER_R2_DERIVED_BUCKET` | Derived R2 archive bucket; defaults to `ledger-derived` | + +The two R2 rows are new in this window rather than renamed: those buckets were +hardcoded before, so the ledger-era spellings are accepted for consistency, not +because anything ever set them. + +Variables carrying none of the three prefixes are read literally. This helper +renames the ledger-era surface, not every PolicyEngine variable, so +`POLICYENGINE_SUPABASE_URL`, `POLICYENGINE_SUPABASE_SERVICE_KEY` and +`POLICYENGINE_TARGETS_SCHEMA` keep their names and gain no aliases. + +The hosted schema *value* is a separate migration. `CHRONICLE_SCHEMA` renames +the variable that overrides the schema; the schema still defaults to `ledger`, +and the mirror table names are unchanged. Those move in a later slice +coordinated with the CI writers. + +Every reader of the setting resolves it the same way, at call time. That +includes the writer: `load-supabase-mirror` takes its `--schema` default from +`CHRONICLE_SCHEMA`, so setting the variable to rehearse a cutover moves the +mirror load with the client rather than leaving it pointed at `ledger`. An +explicit `--schema` still wins. + +## Bucket Cutover + +Chronicle's operational stores migrate by dual-run +(PolicyEngine/chronicle#143, mechanism 3): stand up the chronicle-named home, +backfill it, repoint writers, retire the old home. The R2 buckets take one +exception to the last step. Archived witness records pin raw R2 URLs by hash, so +`ledger-raw` and `ledger-derived` are preserved read-only forever rather than +deleted, and manifests keep the `storage.r2` URIs they already recorded as +historical truth. A backfill copies bytes into the new bucket; it never rewrites +where those bytes were first published. `publish-raw` and `fetch-artifact` +enforce that: a recorded block that addresses the bytes in hand is preserved +whichever bucket is configured, and `publish-raw` reports such an entry as +`skipped` (already published under the recorded bucket) rather than restating +it under a different one, so a sweep over a fully published tree stays green +after the flip. Bytes that the recorded object does not hold are not that +object's history at all; see [Publisher Revisions](#publisher-revisions). + +The cutover therefore has one irreversible-looking step that is in fact additive +(creating and filling the new buckets), one cheap reversible step (flipping the +defaults, which is a one-line change in `chronicle/artifacts.py`), and no +deletion step at all. + +### 1. Create the new buckets + +Bucket creation needs a Cloudflare login carrying R2 permissions, so it is an +operator step rather than something CI can do. `wrangler.toml` already pins the +PolicyEngine account (`account_id = "20d90f557651969925eece96e58e24dc"`), so no +`CLOUDFLARE_ACCOUNT_ID` is needed even for a user who belongs to several +accounts: + +```bash +bunx wrangler login +uv run chronicle bootstrap-r2 --raw-bucket chronicle-raw --derived-bucket chronicle-derived +``` + +`bootstrap-r2` verifies authentication with `wrangler whoami` before creating +anything, and creating a bucket that already exists is not an error. + +### 2. Enumerate what has to be copied + +Tracked manifests are the authoritative registry of raw objects. Every one of +them points at `ledger-raw` today: + +```bash +git ls-files '*manifest*.yaml' '*manifest*.yml' \ + | xargs grep -ho 'r2://ledger-raw/[^"'"'"' ]*' | sort -u > /tmp/chronicle-raw-objects.txt +wc -l < /tmp/chronicle-raw-objects.txt +``` + +That is 186 distinct objects at `ff3efd3`, spread over 154 manifest files. Recount +rather than trusting the number: source packages land continuously, and each new +package adds objects. + +The derived bucket needs no backfill. Derived artifacts are reproducible by +definition and are already keyed by `{build_id}`, so a rebuild republishes them +into whichever bucket is configured. + +### 3. Backfill-copy the raw objects + +Keys are content-addressed and identical across buckets, so the copy is a +straight get/put per object: + +```bash +mkdir -p /tmp/chronicle-r2-backfill +while read -r uri; do + key=${uri#r2://ledger-raw/} + dest=/tmp/chronicle-r2-backfill/$key + mkdir -p "$(dirname "$dest")" + bunx wrangler r2 object get "ledger-raw/$key" --file "$dest" --remote + bunx wrangler r2 object put "chronicle-raw/$key" --file "$dest" --remote +done < /tmp/chronicle-raw-objects.txt +``` + +### 4. Verify the copy against the keys themselves + +Every raw key ends `.../{sha256}/{filename}`, so the key is its own checksum +witness and verification needs no manifest lookup: + +```bash +while read -r uri; do + key=${uri#r2://ledger-raw/} + expected=$(printf '%s\n' "$key" | awk -F/ '{print $(NF-1)}') + actual=$(shasum -a 256 "/tmp/chronicle-r2-backfill/$key" | cut -d' ' -f1) + [ "$expected" = "$actual" ] || echo "MISMATCH $key" +done < /tmp/chronicle-raw-objects.txt +``` + +Silence means every downloaded object hashes to the checksum its key claims. +That covers the read from `ledger-raw`; to cover the write to `chronicle-raw`, +re-download each key from the new bucket into a second directory and rerun the +same loop against it. + +### 5. Flip the defaults, in a follow-up PR + +Once the new buckets are filled and verified, change `DEFAULT_R2_RAW_BUCKET` and +`DEFAULT_R2_DERIVED_BUCKET` in `chronicle/artifacts.py` to `chronicle-raw` and +`chronicle-derived`. Until then, operators can opt in per-shell: + +```bash +export CHRONICLE_R2_RAW_BUCKET=chronicle-raw +export CHRONICLE_R2_DERIVED_BUCKET=chronicle-derived +``` + +New raw publications land in the new bucket from that point. Manifests written +before the flip keep pointing at `ledger-raw`, which is why the old bucket stays +readable. A `publish-raw --root db/data` sweep after the flip reports every +already-published entry as `skipped` with its recorded `ledger-raw` location +(`skipped_count` in the report) and exits 0; only bytes that no recorded object +holds are uploaded, into `chronicle-raw`. + +### 6. Set the ledger-era buckets read-only + +`ledger-raw` and `ledger-derived` keep serving archived witness records after the +flip. They should accept no further writes and should never be deleted. + ## Non-Goals Supabase should not store large raw binary artifacts. It should point to R2. diff --git a/scripts/register_microdata_releases.py b/scripts/register_microdata_releases.py new file mode 100644 index 00000000..d4916b73 --- /dev/null +++ b/scripts/register_microdata_releases.py @@ -0,0 +1,1068 @@ +"""Register raw microdata releases from a Microcosm source-stages manifest. + +Chronicle registers every raw microdata release its consumers build from and +stores the bytes of only those a publisher permits it to redistribute +(``docs/adr-chronicle-raw-microdata-identity.md``). This script drives both +halves of that from the pins Microcosm already reviewed: + +``emit`` + Write hash-only ``kind: microdata_release`` manifests for ``licensed`` and + ``restricted`` releases. Every checksum, size, filename, and vintage is read + verbatim from the Microcosm source-stages JSON; nothing is recomputed and + nothing is invented. Each registration is a ``consumer_pin``: it names the + consumer as the attester and records the repository, path, and commit the + pin was read from, and carries no verification date of its own. A release + Microcosm pins without a checksum is reported as a blocker and never + registered. + +``plan`` + Print the exact ``chronicle fetch-artifact ... --upload-r2`` commands to run + from a networked machine for ``public`` releases, whose bytes Chronicle does + archive. Every command carries the reviewed identity as arguments: the + publisher, the vintage, and -- when Microcosm's pin is of the publisher + bytes -- ``--expected-sha256`` and ``--expected-size-bytes``, so the fetch + refuses a reissue before archiving it. Publisher URLs are copied verbatim + from the Microcosm manifest; a release whose manifest carries no URL, no + publisher-bytes checksum, or no licence-evidence URL prints a ``TODO`` + instead of a guess, and that command cannot run until the TODO is filled. + +The catalogue below is the only authored content: it maps a Microcosm artifact +onto Chronicle's ``{source_id, package_id, year, sha256, filename}`` identity +and records the publisher's terms. Run it read-only against a Microcosm +checkout; this script never writes to that repository. + +Usage:: + + python scripts/register_microdata_releases.py emit \\ + --microcosm-root ~/PolicyEngine/microcosm \\ + --root db/data + + python scripts/register_microdata_releases.py plan \\ + --microcosm-root ~/PolicyEngine/microcosm --root db/data +""" + +from __future__ import annotations + +import argparse +from collections.abc import Iterator, Mapping, Sequence +from dataclasses import dataclass, field +import json +from pathlib import Path +import re +import shlex +import subprocess +import sys +from typing import Any + +# Allow `python scripts/register_microdata_releases.py` from a checkout. +sys.path.insert(0, str(Path(__file__).resolve().parent.parent)) + +from chronicle.artifacts import default_r2_raw_bucket # noqa: E402 +from chronicle.registration import ( # noqa: E402 + ACCESS_PUBLIC, + HASH_SOURCE_CONSUMER_PIN, + MICRODATA_RELEASE_KIND, + HashOnlyRegistrationError, + register_hash_only_artifact, +) + +#: The consumer whose reviewed pins every registration here transcribes. +CONSUMER_REPOSITORY = "PolicyEngine/microcosm" + +#: Placeholders a planned command prints where Microcosm pins nothing. Each is +#: refused by fetch-artifact as written, so a command carrying one cannot run +#: until a reviewer replaces it. +TODO_PUBLISHER_URL = "TODO_PUBLISHER_URL" +TODO_REVIEWED_SHA256 = "TODO_REVIEWED_SHA256" +TODO_EVIDENCE_URL = "TODO_EVIDENCE_URL" + +#: Allowlisted licence identifier for a public-use file of a U.S. federal +#: statistical agency (chronicle/licences.py). +US_GOVERNMENT_WORK = "US-Government-Work" + +_COMMIT_RE = re.compile(r"^[0-9a-f]{40}$") + +#: Microcosm's per-artifact ``kind`` mapped onto Chronicle's access class. +#: +#: ``private_microdata`` maps to ``restricted`` rather than ``licensed``: the +#: bytes are held only in a private mirror, and over-classifying is the safe +#: direction because both classes are registered hash-only and neither ever +#: places bytes in a Chronicle store. +ACCESS_BY_MICROCOSM_KIND: dict[str, str] = { + "public_microdata": "public", + "licensed_microdata": "licensed", + "private_microdata": "restricted", + "restricted_microdata": "restricted", +} + +#: Provenance sentence written onto every registration this script emits. +HASH_PROVENANCE = ( + "SHA-256 transcribed verbatim from the consumer's reviewed pin; Chronicle " + "holds no bytes for this release and did not recompute the checksum." +) + + +@dataclass(frozen=True) +class ArtifactSelector: + """Locate one artifact inside a Microcosm source-stages JSON file. + + ``stage`` names the build stage; ``match`` is a set of artifact fields that + must equal the given values. A selector that matches nothing, or matches + inconsistent bytes across stages, is a hard error rather than a guess. + """ + + stage: str | None = None + match: Mapping[str, Any] = field(default_factory=dict) + kind: str | None = None + + +@dataclass(frozen=True) +class Release: + """One Chronicle registration drawn from a Microcosm pin.""" + + release_id: str + manifest: str + selector: ArtifactSelector + source_id: str + package_id: str + package_dir: str + year: int + table: str + publisher: str + licence: str + access: str + #: Artifact field holding the publisher filename, when not ``filename``. + filename_field: str = "filename" + #: Artifact field holding the publisher URL, for ``public`` releases. + url_field: str = "locator" + study: str | None = None + doi: str | None = None + source_page: str | None = None + access_route: str | None = None + vintage: str | None = None + notes: str | None = None + #: Whether Microcosm's pinned ``sha256`` is of the publisher artifact + #: itself. It is not when the pin covers a derived file or an extracted + #: archive member, and such a hash must never be presented as the checksum + #: a fetch should reproduce. + pinned_sha_is_publisher_bytes: bool = True + #: Licence evidence for a public release: who issued the file under the + #: allowlisted term, the scope statement, and the durable evidence URL. + #: A missing URL prints a TODO in the plan; it is never guessed. + licence_evidence_issuer: str | None = None + licence_evidence_scope: str | None = None + licence_evidence_url: str | None = None + #: Set when Microcosm pins the release without a checksum. + blocker: str | None = None + + +UK_STAGES = "packages/microcosm-build/src/microcosm/build/uk/source_stages.json" +UK_HMRC_STAGES = ( + "packages/microcosm-build/src/microcosm/build/uk/hmrc_income_source_stages.json" +) +BE_STAGES = "packages/microcosm-build/src/microcosm/build/be/source_stages.json" +US_STAGES = "packages/microcosm-build/src/microcosm/build/us/source_stages.json" +US_ACS_2024 = ( + "packages/microcosm-build/src/microcosm/build/us_runtime/acs_2024_1yr_sources.json" +) + +#: FRS 2023-24 tabs, in the order Microcosm's ``frs_spine`` stage lists them. +FRS_TABS: tuple[str, ...] = ( + "accounts", + "adult", + "benefits", + "benunit", + "child", + "chldcare", + "extchild", + "househol", + "job", + "maint", + "mortgage", + "oddjob", + "penprov", + "pension", +) + +FRS_LICENCE = "UK Data Service End User Licence" +FRS_STUDY = "UK Data Service SN 9367" +FRS_DOI = "10.5255/UKDA-SN-9367-2" +FRS_ACCESS_ROUTE = ( + "UK Data Service study SN 9367 under its End User Licence. Bytes stay in " + "the licensed environment the consumer already operates; no Chronicle " + "credential grants access to them." +) +FRS_NOTES = ( + "Microcosm's frs_spine stage cites 'UK Data Service SN 9367, DOI " + "10.5255/UKDA-SN-9367-2'; its frs_employment, frs_council_tax, " + "frs_education, and frs_legacy_proxies stages cite 'SN 9252' for the same " + "2023_24 tabs. The tabs are the same bytes across all five stages (identical " + "SHA-256), so the registration carries the study reference that also carries " + "a DOI. " + HASH_PROVENANCE +) + +CENSUS_SCOPE = ( + "Public-use microdata file published by the U.S. Census Bureau, a federal " + "agency; a work of the United States Government under 17 U.S.C. §105." +) +FED_SCOPE = ( + "Public data set published by the Board of Governors of the Federal " + "Reserve System, a federal agency; a work of the United States Government " + "under 17 U.S.C. §105." +) + +CATALOGUE: tuple[Release, ...] = ( + *( + Release( + release_id=f"dwp-frs-2023-24:{tab}", + manifest=UK_STAGES, + selector=ArtifactSelector( + stage="frs_spine", + kind="licensed_microdata", + match={"table": tab}, + ), + source_id="dwp", + package_id="dwp-frs-2023-24", + package_dir="dwp/frs_2023_24", + year=2023, + table="Family Resources Survey 2023-24", + publisher="Department for Work and Pensions", + licence=FRS_LICENCE, + access="licensed", + filename_field="locator", + study=FRS_STUDY, + doi=FRS_DOI, + access_route=FRS_ACCESS_ROUTE, + notes=FRS_NOTES, + ) + for tab in FRS_TABS + ), + Release( + release_id="hmrc-spi-public-use-tape-2022-23:put2223uk", + manifest=UK_HMRC_STAGES, + selector=ArtifactSelector( + stage="hmrc_spi_income", + kind="private_microdata", + match={"filename": "put2223uk.tab"}, + ), + source_id="hmrc", + package_id="hmrc-spi-public-use-tape-2022-23", + package_dir="hmrc/spi_public_use_tape_2022_23", + year=2022, + table="Survey of Personal Incomes Public Use Tape 2022-23", + publisher="HM Revenue and Customs", + licence="UK Data Service End User Licence (study SN 9422)", + access="restricted", + study="UK Data Service SN 9422", + doi="10.5255/UKDA-SN-9422-1", + access_route=( + "UK Data Service study SN 9422. Microcosm reaches the bytes through " + "PolicyEngine's licensed copy in the private " + "policyengine/policyengine-uk-data-private Hugging Face repository " + "(spi_2022_23.zip); no Chronicle credential grants access to them." + ), + notes=( + "Microcosm classes this artifact kind: private_microdata with " + "access: private_local_input. Chronicle registers it restricted " + "because the bytes are held only in a private mirror; if the UKDS " + "terms for SN 9422 are confirmed as End User Licence it can be " + "reclassified licensed, which changes nothing about storage — both " + "classes are hash-only. " + HASH_PROVENANCE + ), + ), + Release( + release_id="statbel-be-silc-2023", + manifest=BE_STAGES, + selector=ArtifactSelector( + stage="silc_load", + kind="restricted_microdata", + ), + source_id="statbel", + package_id="statbel-be-silc-2023", + package_dir="statbel/be_silc_2023", + year=2023, + table="BE-SILC 2023 scientific-use files (D, R, H, P)", + publisher="Statbel", + licence="Statbel/Eurostat scientific-use", + access="restricted", + source_page="https://statbel.fgov.be/en/themes/households/poverty-and-living-conditions", + access_route=( + "Statbel BE-SILC scientific-use files: D (household register), " + "R (personal register), H (household data), P (personal data)." + ), + blocker=( + "Microcosm's be/source_stages.json pins the BE-SILC scientific-use " + "files with no sha256, no size_bytes, and no per-file filename — it " + "names only the four file roles. A registration is identified by " + "{source_id, package_id, year, sha256, filename}, so this release " + "cannot be registered until the consumer publishes a reviewed " + "checksum per file. No hash is invented here." + ), + ), + # ---- public releases: bytes are archived, so these are fetch-and-upload ---- + Release( + release_id="census-cps-asec-2023", + manifest=US_STAGES, + selector=ArtifactSelector( + stage="weeks_unemployed_input", + kind="public_microdata", + match={"member": "pppub23.csv"}, + ), + source_id="census_cps", + package_id="census-cps-asec-2023", + package_dir="census/cps_asec_2023", + year=2023, + table="CPS Annual Social and Economic Supplement 2023 public-use files", + publisher="U.S. Census Bureau", + licence=US_GOVERNMENT_WORK, + licence_evidence_issuer="U.S. Census Bureau", + licence_evidence_scope=CENSUS_SCOPE, + access=ACCESS_PUBLIC, + source_page="https://www.census.gov/programs-surveys/cps/data/datasets.html", + ), + Release( + release_id="census-cps-basic-monthly-2024", + manifest=US_STAGES, + selector=ArtifactSelector( + stage="org_wages", + kind="public_microdata", + ), + source_id="census_cps", + package_id="census-cps-basic-monthly-2024", + package_dir="census/cps_basic_monthly_2024", + year=2024, + table="CPS basic monthly public-use files, January-December 2024", + publisher="U.S. Census Bureau", + licence=US_GOVERNMENT_WORK, + licence_evidence_issuer="U.S. Census Bureau", + licence_evidence_scope=CENSUS_SCOPE, + access=ACCESS_PUBLIC, + source_page="https://www2.census.gov/programs-surveys/cps/datasets/2024/basic/", + notes=( + "Microcosm pins the twelve monthly files as the locator string " + "'jan24pub through dec24pub' with no per-file URL, filename " + "extension, or checksum. The publisher directory is the stage " + "source; the twelve filenames must be read off that directory " + "before the fetch commands can be completed." + ), + ), + Release( + release_id="census-acs-pums-2022-household", + manifest=US_STAGES, + selector=ArtifactSelector( + stage="acs_rent", + kind="versioned_derived_microdata", + ), + source_id="census_acs", + package_id="census-acs-pums-2022-1yr", + package_dir="census/acs_pums_2022_1yr", + year=2022, + table="ACS 2022 1-Year PUMS household file", + publisher="U.S. Census Bureau", + licence=US_GOVERNMENT_WORK, + licence_evidence_issuer="U.S. Census Bureau", + licence_evidence_scope=CENSUS_SCOPE, + access=ACCESS_PUBLIC, + url_field="official_household_source", + source_page="https://www.census.gov/programs-surveys/acs", + pinned_sha_is_publisher_bytes=False, + notes=( + "The Microcosm artifact's own sha256 belongs to the derived " + "acs_2022.h5, not to the publisher zip; only the publisher URL is " + "reused here. The fetch computes the release checksum." + ), + ), + Release( + release_id="census-acs-pums-2022-person", + manifest=US_STAGES, + selector=ArtifactSelector( + stage="acs_rent", + kind="versioned_derived_microdata", + ), + source_id="census_acs", + package_id="census-acs-pums-2022-1yr", + package_dir="census/acs_pums_2022_1yr", + year=2022, + table="ACS 2022 1-Year PUMS person file", + publisher="U.S. Census Bureau", + licence=US_GOVERNMENT_WORK, + licence_evidence_issuer="U.S. Census Bureau", + licence_evidence_scope=CENSUS_SCOPE, + access=ACCESS_PUBLIC, + url_field="official_person_source", + source_page="https://www.census.gov/programs-surveys/acs", + pinned_sha_is_publisher_bytes=False, + notes=( + "The Microcosm artifact's own sha256 belongs to the derived " + "acs_2022.h5, not to the publisher zip; only the publisher URL is " + "reused here. The fetch computes the release checksum." + ), + ), + Release( + release_id="census-acs-pums-2024-household", + manifest=US_ACS_2024, + selector=ArtifactSelector(match={"role": "household"}), + source_id="census_acs", + package_id="census-acs-pums-2024-1yr", + package_dir="census/acs_pums_2024_1yr", + year=2024, + table="ACS 2024 1-Year PUMS household file", + publisher="U.S. Census Bureau", + licence=US_GOVERNMENT_WORK, + licence_evidence_issuer="U.S. Census Bureau", + licence_evidence_scope=CENSUS_SCOPE, + access=ACCESS_PUBLIC, + url_field="url", + vintage="2024", + source_page="https://www2.census.gov/programs-surveys/acs/data/pums/2024/1-Year/", + ), + Release( + release_id="census-acs-pums-2024-person", + manifest=US_ACS_2024, + selector=ArtifactSelector(match={"role": "person"}), + source_id="census_acs", + package_id="census-acs-pums-2024-1yr", + package_dir="census/acs_pums_2024_1yr", + year=2024, + table="ACS 2024 1-Year PUMS person file", + publisher="U.S. Census Bureau", + licence=US_GOVERNMENT_WORK, + licence_evidence_issuer="U.S. Census Bureau", + licence_evidence_scope=CENSUS_SCOPE, + access=ACCESS_PUBLIC, + url_field="url", + vintage="2024", + source_page="https://www2.census.gov/programs-surveys/acs/data/pums/2024/1-Year/", + ), + Release( + release_id="federal-reserve-scf-2022-summary", + manifest=US_STAGES, + selector=ArtifactSelector( + stage="scf_wealth", + kind="public_microdata", + match={"member": "rscfp2022.dta"}, + ), + source_id="federal_reserve", + package_id="federal-reserve-scf-2022", + package_dir="federal_reserve/scf_2022", + year=2022, + table="Survey of Consumer Finances 2022 summary extract", + publisher="Board of Governors of the Federal Reserve System", + licence=US_GOVERNMENT_WORK, + licence_evidence_issuer="Board of Governors of the Federal Reserve System", + licence_evidence_scope=FED_SCOPE, + access=ACCESS_PUBLIC, + source_page="https://www.federalreserve.gov/econres/scfindex.htm", + ), + Release( + release_id="federal-reserve-scf-2022-full", + manifest=US_STAGES, + selector=ArtifactSelector( + stage="scf_wealth", + kind="public_microdata", + match={"member": "p22i6.dta"}, + ), + source_id="federal_reserve", + package_id="federal-reserve-scf-2022", + package_dir="federal_reserve/scf_2022", + year=2022, + table="Survey of Consumer Finances 2022 full public data set", + publisher="Board of Governors of the Federal Reserve System", + licence=US_GOVERNMENT_WORK, + licence_evidence_issuer="Board of Governors of the Federal Reserve System", + licence_evidence_scope=FED_SCOPE, + access=ACCESS_PUBLIC, + source_page="https://www.federalreserve.gov/econres/scfindex.htm", + notes=( + "Microcosm records no checksum for this zip: 'Full-file SHA-256 " + "pending one network-enabled provisioning fetch'. The fetch below " + "computes and registers it." + ), + ), + Release( + release_id="census-sipp-2023", + manifest=US_STAGES, + selector=ArtifactSelector( + stage="scf_wealth", + kind="public_microdata", + match={"member": "pu2023.csv"}, + ), + source_id="census_sipp", + package_id="census-sipp-2023", + package_dir="census/sipp_2023", + year=2023, + table="Survey of Income and Program Participation 2023 public-use file", + publisher="U.S. Census Bureau", + licence=US_GOVERNMENT_WORK, + licence_evidence_issuer="U.S. Census Bureau", + licence_evidence_scope=CENSUS_SCOPE, + access=ACCESS_PUBLIC, + source_page="https://www.census.gov/programs-surveys/sipp.html", + pinned_sha_is_publisher_bytes=False, + notes=( + "Microcosm reaches this file through an immutable Hugging Face " + "mirror (revision 21280dca5995e978d706740a8a4b9b7860cfd7b6) and " + "records no Census URL, so the publisher URL must be read off the " + "SIPP dataset page before the fetch. Microcosm's pinned sha256 " + "5c30439e365fc26483318ef61d1d8f4bb2f0e9d6bb47c22c06756a7698733ee2 " + "and size 3726010471 are for the mirrored pu2023.csv member, not " + "for whatever archive the Census page serves, so they are not the " + "checksum this fetch should be expected to reproduce." + ), + ), +) + + +class CatalogueError(RuntimeError): + """Raised when the catalogue cannot be resolved against Microcosm.""" + + +def load_manifest(microcosm_root: Path, relative: str) -> dict[str, Any]: + """Load one Microcosm JSON manifest, read-only.""" + path = microcosm_root / relative + if not path.exists(): + raise CatalogueError(f"Microcosm manifest not found: {path}") + payload = json.loads(path.read_text(encoding="utf-8")) + if not isinstance(payload, dict): + raise CatalogueError(f"Microcosm manifest must be an object: {path}") + return payload + + +def iter_manifest_artifacts( + payload: Mapping[str, Any], +) -> Iterator[tuple[Mapping[str, Any], Mapping[str, Any]]]: + """Yield ``(stage, artifact)`` pairs from either Microcosm manifest shape. + + Source-stages manifests nest artifacts under ``stages``; the ACS runtime + manifest carries a flat top-level ``artifacts`` list with no stages. + """ + stages = payload.get("stages") + if isinstance(stages, list): + for stage in stages: + if not isinstance(stage, Mapping): + continue + for artifact in stage.get("artifacts") or (): + if isinstance(artifact, Mapping): + yield stage, artifact + return + for artifact in payload.get("artifacts") or (): + if isinstance(artifact, Mapping): + yield payload, artifact + + +def select_artifact( + payload: Mapping[str, Any], + selector: ArtifactSelector, + *, + release_id: str, +) -> tuple[Mapping[str, Any], Mapping[str, Any]]: + """Return the single ``(stage, artifact)`` a selector identifies. + + Several Microcosm stages reference the same bytes. Duplicates are accepted + only when they agree on every field the registration reads; disagreement is + an error, never a silent first-match. + """ + matches: list[tuple[Mapping[str, Any], Mapping[str, Any]]] = [] + for stage, artifact in iter_manifest_artifacts(payload): + if selector.stage is not None and stage.get("stage") != selector.stage: + continue + if selector.kind is not None and artifact.get("kind") != selector.kind: + continue + if any(artifact.get(key) != value for key, value in selector.match.items()): + continue + matches.append((stage, artifact)) + if not matches: + raise CatalogueError( + f"{release_id}: no Microcosm artifact matches {selector}. The " + "consumer manifest changed; re-derive the catalogue rather than " + "hand-editing a registration." + ) + first_stage, first = matches[0] + for _stage, artifact in matches[1:]: + if dict(artifact) != dict(first): + raise CatalogueError( + f"{release_id}: Microcosm pins conflicting values for this " + "artifact across stages; refusing to choose between them." + ) + return first_stage, first + + +@dataclass(frozen=True) +class ResolvedRelease: + """A catalogue entry resolved against Microcosm's pinned artifact.""" + + release: Release + stage: Mapping[str, Any] + artifact: Mapping[str, Any] + + @property + def filename(self) -> str | None: + """Publisher filename, from the release's declared artifact field.""" + value = self.artifact.get(self.release.filename_field) + if not value: + return None + # A locator may be a URL or a bare filename; take the last path segment. + return str(value).rstrip("/").rsplit("/", 1)[-1] + + @property + def url(self) -> str | None: + """Publisher URL, when the Microcosm artifact records one.""" + value = self.artifact.get(self.release.url_field) + text = str(value).strip() if value else "" + return text if text.startswith(("http://", "https://")) else None + + @property + def sha256(self) -> str | None: + """Checksum Microcosm pins for these bytes, if any.""" + value = self.artifact.get("sha256") + return str(value) if value else None + + @property + def size_bytes(self) -> int | None: + """Size Microcosm pins for these bytes, if any.""" + value = self.artifact.get("size_bytes") + return int(value) if isinstance(value, int) else None + + @property + def vintage(self) -> str: + """Publisher vintage label, from the release or the artifact.""" + return str(self.release.vintage or self.artifact.get("vintage") or "") + + +def resolve( + microcosm_root: Path, + releases: Sequence[Release], +) -> list[ResolvedRelease]: + """Resolve every catalogue entry against the Microcosm checkout.""" + payloads: dict[str, dict[str, Any]] = {} + resolved: list[ResolvedRelease] = [] + for release in releases: + if release.manifest not in payloads: + payloads[release.manifest] = load_manifest(microcosm_root, release.manifest) + stage, artifact = select_artifact( + payloads[release.manifest], + release.selector, + release_id=release.release_id, + ) + resolved.append( + ResolvedRelease(release=release, stage=stage, artifact=artifact) + ) + return resolved + + +def pin_commit(microcosm_root: Path, relative: str) -> str: + """Return the commit the consumer's pin is read from, read-only. + + The pin is the manifest blob, so the commit recorded is the last one that + changed that file: it addresses exactly the bytes the registration + transcribes, and it is stable across later, unrelated commits so repeated + ``emit`` runs stay byte-identical. + """ + try: + completed = subprocess.run( + [ + "git", + "-C", + str(microcosm_root), + "log", + "-1", + "--format=%H", + "--", + relative, + ], + capture_output=True, + text=True, + check=True, + ) + except (OSError, subprocess.CalledProcessError) as exc: + raise CatalogueError( + f"Cannot read the commit of {relative} in {microcosm_root}: {exc}. " + "Pass --microcosm-commit with the reviewed commit." + ) from exc + commit = completed.stdout.strip() + if not _COMMIT_RE.match(commit): + raise CatalogueError( + f"{microcosm_root} records no commit for {relative}; pass " + "--microcosm-commit with the reviewed commit." + ) + return commit + + +def parse_pin_commits(values: Sequence[str]) -> dict[str, str]: + """Parse ``--microcosm-commit`` values into ``{manifest path or '*': commit}``.""" + commits: dict[str, str] = {} + for value in values: + path, separator, commit = value.rpartition("=") + key = path if separator else "*" + if not _COMMIT_RE.match(commit): + raise CatalogueError( + f"--microcosm-commit must name a 40-hex commit, not {value!r}." + ) + if key in commits and commits[key] != commit: + raise CatalogueError( + f"--microcosm-commit names two commits for {key!r}; pass one." + ) + commits[key] = commit + return commits + + +def pinned_from(release: Release, commit: str) -> dict[str, str]: + """Return the ``pinned_from`` block a consumer_pin registration records.""" + return { + "repository": CONSUMER_REPOSITORY, + "path": release.manifest, + "commit": commit, + } + + +def emit( + resolved: Sequence[ResolvedRelease], + *, + root: Path, + pin_commits: Mapping[str, str], + allow_reissue: bool = False, +) -> tuple[list[dict[str, Any]], list[dict[str, str]]]: + """Write hash-only manifests for every registrable non-public release. + + ``pin_commits`` maps each consumer manifest path to the commit its pins + are read from. Returns ``(registrations, blockers)``. A release Microcosm + pins without a checksum is a blocker, not a registration: no hash is ever + invented. + """ + registrations: list[dict[str, Any]] = [] + blockers: list[dict[str, str]] = [] + for item in resolved: + release = item.release + if release.access == ACCESS_PUBLIC: + continue + if release.blocker: + blockers.append({"release": release.release_id, "reason": release.blocker}) + continue + filename = item.filename + checksum = item.sha256 + if not filename or not checksum: + blockers.append( + { + "release": release.release_id, + "reason": ( + "Microcosm pins this release without a " + f"{'filename' if not filename else 'sha256'}; a " + "registration needs both. No value is invented." + ), + } + ) + continue + report = register_hash_only_artifact( + source_id=release.source_id, + package_id=release.package_id, + year=release.year, + output_dir=root / release.package_dir, + filename=filename, + sha256=checksum, + licence=release.licence, + access=release.access, + vintage=item.vintage, + hash_source=HASH_SOURCE_CONSUMER_PIN, + attested_by=CONSUMER_REPOSITORY, + pinned_from=pinned_from(release, pin_commits[release.manifest]), + size_bytes=item.size_bytes, + source_page=release.source_page, + access_route=release.access_route, + doi=release.doi, + study=release.study, + table=release.table, + publisher=release.publisher, + notes=release.notes or HASH_PROVENANCE, + allow_reissue=allow_reissue, + ) + registrations.append(report.to_dict()) + return registrations, blockers + + +def fetch_command( + item: ResolvedRelease, + *, + root: Path, + r2_bucket: str, +) -> tuple[str, list[str]]: + """Return ``(command, todos)`` for one public release. + + The command is the exact ``chronicle fetch-artifact`` invocation to run from + a networked machine. Anything Microcosm does not pin becomes a TODO rather + than a fabricated value. + """ + release = item.release + todos: list[str] = [] + url = item.url + if url is None: + url = TODO_PUBLISHER_URL + todos.append( + f"{release.release_id}: Microcosm records no publisher URL " + f"(field {release.url_field!r}); read it off {release.source_page}." + ) + argv = [ + "uv", + "run", + "chronicle", + "fetch-artifact", + "--source-id", + release.source_id, + "--package-id", + release.package_id, + "--year", + str(release.year), + "--out-dir", + str(root / release.package_dir), + "--url", + url, + ] + if item.filename: + argv += ["--filename", item.filename] + if release.source_page: + argv += ["--source-page", release.source_page] + vintage = item.vintage + if not vintage: + todos.append( + f"{release.release_id}: Microcosm records no vintage; add " + "Release.vintage to the catalogue before running this command." + ) + argv += [ + "--table", + release.table, + "--publisher", + release.publisher, + "--vintage", + vintage or "TODO_VINTAGE", + "--access", + ACCESS_PUBLIC, + "--licence", + release.licence, + # A public microdata release is archived, but it is still a release: + # several files share one vintage and no source package parses it. + "--kind", + MICRODATA_RELEASE_KIND, + ] + # The reviewed identity travels as arguments, never as a comment: the + # fetch refuses bytes that hash differently before archiving anything. + if item.sha256 and release.pinned_sha_is_publisher_bytes: + argv += ["--expected-sha256", item.sha256] + if item.size_bytes: + argv += ["--expected-size-bytes", str(item.size_bytes)] + else: + argv += ["--expected-sha256", TODO_REVIEWED_SHA256] + if item.sha256: + todos.append( + f"{release.release_id}: Microcosm's pinned sha256 {item.sha256} " + "is NOT the publisher artifact's checksum. A public release is " + "archived only against a reviewed checksum its licence evidence " + "covers; review the publisher bytes and replace " + f"{TODO_REVIEWED_SHA256} before running this command." + ) + else: + todos.append( + f"{release.release_id}: Microcosm pins no checksum for this " + "release. A public release is archived only against a reviewed " + f"checksum; replace {TODO_REVIEWED_SHA256} before running this " + "command." + ) + argv += [ + "--licence-evidence-issuer", + release.licence_evidence_issuer or release.publisher, + "--licence-evidence-scope", + release.licence_evidence_scope or "TODO_EVIDENCE_SCOPE", + "--licence-evidence-url", + release.licence_evidence_url or TODO_EVIDENCE_URL, + ] + if not release.licence_evidence_url: + todos.append( + f"{release.release_id}: no durable licence-evidence URL is " + f"catalogued; replace {TODO_EVIDENCE_URL} with the publisher's " + "statement that this file is issued under " + f"{release.licence} before running this command." + ) + argv += ["--upload-r2", "--r2-bucket", r2_bucket] + if release.notes: + todos.append(f"{release.release_id}: {release.notes}") + return shlex.join(argv), todos + + +def plan( + resolved: Sequence[ResolvedRelease], + *, + root: Path, + r2_bucket: str, +) -> tuple[list[str], list[str]]: + """Return the fetch commands and TODOs for every public release.""" + commands: list[str] = [] + todos: list[str] = [] + for item in resolved: + if item.release.access != ACCESS_PUBLIC: + continue + command, item_todos = fetch_command(item, root=root, r2_bucket=r2_bucket) + commands.append(command) + todos.extend(item_todos) + return commands, todos + + +def build_parser() -> argparse.ArgumentParser: + """Build the command-line parser.""" + parser = argparse.ArgumentParser( + description=__doc__, + formatter_class=argparse.RawDescriptionHelpFormatter, + ) + parser.add_argument( + "--microcosm-root", + type=Path, + default=Path.home() / "PolicyEngine" / "microcosm", + help="Read-only path to a PolicyEngine/microcosm checkout.", + ) + parser.add_argument( + "--root", + type=Path, + default=Path("db/data"), + help="Chronicle data root that holds the package directories.", + ) + parser.add_argument( + "--release", + action="append", + default=None, + help="Limit to these release IDs. Repeatable.", + ) + parser.add_argument( + "--json", + action="store_true", + help="Emit machine-readable JSON instead of prose.", + ) + subparsers = parser.add_subparsers(dest="command", required=True) + + emit_parser = subparsers.add_parser( + "emit", + help="Write hash-only manifests for licensed and restricted releases", + ) + emit_parser.add_argument( + "--microcosm-commit", + action="append", + default=None, + metavar="[PATH=]COMMIT", + help=( + "Commit the consumer pins are read from, recorded as " + "pinned_from.commit on every registration. A bare COMMIT applies " + "to every consumer manifest; PATH=COMMIT (repeatable) names the " + "commit for one manifest path. Defaults to the last commit that " + "changed each consumer manifest, read from the checkout's git " + "history." + ), + ) + emit_parser.add_argument( + "--allow-reissue", + action="store_true", + help="Register different bytes alongside an existing pin for a filename.", + ) + + plan_parser = subparsers.add_parser( + "plan", + help="Print the fetch commands to run for public releases", + ) + plan_parser.add_argument( + "--r2-bucket", + default=None, + help=( + "Raw bucket the fetch should upload to. Defaults to " + "$CHRONICLE_R2_RAW_BUCKET, else the ledger-era default." + ), + ) + return parser + + +def main(argv: list[str] | None = None) -> int: + """Run the registration script.""" + args = build_parser().parse_args(argv) + releases = CATALOGUE + if args.release: + wanted = set(args.release) + releases = tuple(r for r in CATALOGUE if r.release_id in wanted) + missing = wanted - {r.release_id for r in releases} + if missing: + print(f"Unknown release IDs: {sorted(missing)}", file=sys.stderr) + return 2 + + microcosm_root = args.microcosm_root.expanduser() + try: + resolved = resolve(microcosm_root, releases) + except CatalogueError as exc: + print(str(exc), file=sys.stderr) + return 1 + + if args.command == "emit": + try: + declared = parse_pin_commits(args.microcosm_commit or ()) + except CatalogueError as exc: + print(str(exc), file=sys.stderr) + return 2 + # Only a registrable release needs its pin's commit: a public release + # is fetched, not transcribed, and a blocked one is never registered. + registrable = sorted( + { + item.release.manifest + for item in resolved + if item.release.access != ACCESS_PUBLIC and not item.release.blocker + } + ) + try: + pin_commits = { + manifest: declared.get(manifest) + or declared.get("*") + or pin_commit(microcosm_root, manifest) + for manifest in registrable + } + except CatalogueError as exc: + print(str(exc), file=sys.stderr) + return 1 + try: + registrations, blockers = emit( + resolved, + root=args.root, + pin_commits=pin_commits, + allow_reissue=args.allow_reissue, + ) + except HashOnlyRegistrationError as exc: + print(str(exc), file=sys.stderr) + return 1 + if args.json: + print( + json.dumps( + {"registrations": registrations, "blockers": blockers}, + indent=2, + sort_keys=True, + ) + ) + else: + for registration in registrations: + print(f"registered {registration['registration']}") + for blocker in blockers: + print(f"BLOCKED {blocker['release']}: {blocker['reason']}") + print( + f"\n{len(registrations)} registration(s), {len(blockers)} blocker(s)." + ) + return 0 + + commands, todos = plan( + resolved, + root=args.root, + r2_bucket=args.r2_bucket or default_r2_raw_bucket(), + ) + if args.json: + print(json.dumps({"commands": commands, "todos": todos}, indent=2)) + return 0 + print("# Run from a networked machine with R2 credentials.\n") + for command in commands: + print(command + "\n") + if todos: + print("# TODO") + for todo in todos: + print(f"# {todo}") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tests/conftest.py b/tests/conftest.py new file mode 100644 index 00000000..3d315441 --- /dev/null +++ b/tests/conftest.py @@ -0,0 +1,50 @@ +"""Shared fixtures for the Chronicle test suite. + +Chronicle is mid-rename (PolicyEngine/chronicle#143, mechanism 3), so its +settings answer to three prefixes at once: ``CHRONICLE_``, and the ledger-era +``POLICYENGINE_LEDGER_`` and ``LEDGER_``. Any of them can be set in an +operator's shell, and many tests assert the defaults those variables override. +Isolation therefore belongs to the whole suite, not to one module. +""" + +from __future__ import annotations + +import os + +import pytest + +from chronicle.env import ( + CHRONICLE_ENV_PREFIX, + LEGACY_ENV_PREFIXES, + reset_env_deprecation_state, +) + +RENAME_WINDOW_PREFIXES = (CHRONICLE_ENV_PREFIX, *LEGACY_ENV_PREFIXES) + + +def pytest_configure(config): + """Strip the rename window before collection imports a single module. + + The autouse fixture below runs per test, which is too late for anything a + module does while being imported. Collection happens after this hook, so + clearing here means no module can read an operator's ``LEDGER_SCHEMA`` + (warning as it goes, or freezing it into a constant) before a fixture has + had the chance to isolate it. Modules should resolve settings at call time + rather than at import; this hook makes that property testable instead of + depending on which shell ran pytest. + """ + for name in list(os.environ): + if name.startswith(RENAME_WINDOW_PREFIXES): + del os.environ[name] + reset_env_deprecation_state() + + +@pytest.fixture(autouse=True) +def isolated_rename_window_env(monkeypatch): + """Run every test with no rename-window variable inherited from the shell.""" + for name in list(os.environ): + if name.startswith(RENAME_WINDOW_PREFIXES): + monkeypatch.delenv(name, raising=False) + reset_env_deprecation_state() + yield + reset_env_deprecation_state() diff --git a/tests/fixtures/microcosm/golden_plan.json b/tests/fixtures/microcosm/golden_plan.json new file mode 100644 index 00000000..9941d5fb --- /dev/null +++ b/tests/fixtures/microcosm/golden_plan.json @@ -0,0 +1,36 @@ +{ + "commands": [ + "uv run chronicle fetch-artifact --source-id census_cps --package-id census-cps-asec-2023 --year 2023 --out-dir db/data/census/cps_asec_2023 --url https://www2.census.gov/programs-surveys/cps/datasets/2023/march/asecpub23csv.zip --source-page https://www.census.gov/programs-surveys/cps/data/datasets.html --table 'CPS Annual Social and Economic Supplement 2023 public-use files' --publisher 'U.S. Census Bureau' --vintage '2023 ASEC / 2022 income reference year' --access public --licence US-Government-Work --kind microdata_release --expected-sha256 d2e000250782adfbdd7f29c82b66d866591a30f0d330496698ec19f9c784ce11 --expected-size-bytes 150165063 --licence-evidence-issuer 'U.S. Census Bureau' --licence-evidence-scope 'Public-use microdata file published by the U.S. Census Bureau, a federal agency; a work of the United States Government under 17 U.S.C. \u00a7105.' --licence-evidence-url TODO_EVIDENCE_URL --upload-r2 --r2-bucket ledger-raw", + "uv run chronicle fetch-artifact --source-id census_cps --package-id census-cps-basic-monthly-2024 --year 2024 --out-dir db/data/census/cps_basic_monthly_2024 --url TODO_PUBLISHER_URL --source-page https://www2.census.gov/programs-surveys/cps/datasets/2024/basic/ --table 'CPS basic monthly public-use files, January-December 2024' --publisher 'U.S. Census Bureau' --vintage 2024 --access public --licence US-Government-Work --kind microdata_release --expected-sha256 TODO_REVIEWED_SHA256 --licence-evidence-issuer 'U.S. Census Bureau' --licence-evidence-scope 'Public-use microdata file published by the U.S. Census Bureau, a federal agency; a work of the United States Government under 17 U.S.C. \u00a7105.' --licence-evidence-url TODO_EVIDENCE_URL --upload-r2 --r2-bucket ledger-raw", + "uv run chronicle fetch-artifact --source-id census_acs --package-id census-acs-pums-2022-1yr --year 2022 --out-dir db/data/census/acs_pums_2022_1yr --url https://www2.census.gov/programs-surveys/acs/data/pums/2022/1-Year/csv_hus.zip --source-page https://www.census.gov/programs-surveys/acs --table 'ACS 2022 1-Year PUMS household file' --publisher 'U.S. Census Bureau' --vintage 2022 --access public --licence US-Government-Work --kind microdata_release --expected-sha256 TODO_REVIEWED_SHA256 --licence-evidence-issuer 'U.S. Census Bureau' --licence-evidence-scope 'Public-use microdata file published by the U.S. Census Bureau, a federal agency; a work of the United States Government under 17 U.S.C. \u00a7105.' --licence-evidence-url TODO_EVIDENCE_URL --upload-r2 --r2-bucket ledger-raw", + "uv run chronicle fetch-artifact --source-id census_acs --package-id census-acs-pums-2022-1yr --year 2022 --out-dir db/data/census/acs_pums_2022_1yr --url https://www2.census.gov/programs-surveys/acs/data/pums/2022/1-Year/csv_pus.zip --source-page https://www.census.gov/programs-surveys/acs --table 'ACS 2022 1-Year PUMS person file' --publisher 'U.S. Census Bureau' --vintage 2022 --access public --licence US-Government-Work --kind microdata_release --expected-sha256 TODO_REVIEWED_SHA256 --licence-evidence-issuer 'U.S. Census Bureau' --licence-evidence-scope 'Public-use microdata file published by the U.S. Census Bureau, a federal agency; a work of the United States Government under 17 U.S.C. \u00a7105.' --licence-evidence-url TODO_EVIDENCE_URL --upload-r2 --r2-bucket ledger-raw", + "uv run chronicle fetch-artifact --source-id census_acs --package-id census-acs-pums-2024-1yr --year 2024 --out-dir db/data/census/acs_pums_2024_1yr --url https://www2.census.gov/programs-surveys/acs/data/pums/2024/1-Year/csv_hus.zip --filename csv_hus.zip --source-page https://www2.census.gov/programs-surveys/acs/data/pums/2024/1-Year/ --table 'ACS 2024 1-Year PUMS household file' --publisher 'U.S. Census Bureau' --vintage 2024 --access public --licence US-Government-Work --kind microdata_release --expected-sha256 8281008e53de98f0ef81e7a2ee5a8725991dda1ecfd2713ead73246425e515d0 --expected-size-bytes 251500587 --licence-evidence-issuer 'U.S. Census Bureau' --licence-evidence-scope 'Public-use microdata file published by the U.S. Census Bureau, a federal agency; a work of the United States Government under 17 U.S.C. \u00a7105.' --licence-evidence-url TODO_EVIDENCE_URL --upload-r2 --r2-bucket ledger-raw", + "uv run chronicle fetch-artifact --source-id census_acs --package-id census-acs-pums-2024-1yr --year 2024 --out-dir db/data/census/acs_pums_2024_1yr --url https://www2.census.gov/programs-surveys/acs/data/pums/2024/1-Year/csv_pus.zip --filename csv_pus.zip --source-page https://www2.census.gov/programs-surveys/acs/data/pums/2024/1-Year/ --table 'ACS 2024 1-Year PUMS person file' --publisher 'U.S. Census Bureau' --vintage 2024 --access public --licence US-Government-Work --kind microdata_release --expected-sha256 afdc6d90c6e2f0bab365ed32d95ba4c4d8ac651162f46ac7861295b2dc469894 --expected-size-bytes 602847146 --licence-evidence-issuer 'U.S. Census Bureau' --licence-evidence-scope 'Public-use microdata file published by the U.S. Census Bureau, a federal agency; a work of the United States Government under 17 U.S.C. \u00a7105.' --licence-evidence-url TODO_EVIDENCE_URL --upload-r2 --r2-bucket ledger-raw", + "uv run chronicle fetch-artifact --source-id federal_reserve --package-id federal-reserve-scf-2022 --year 2022 --out-dir db/data/federal_reserve/scf_2022 --url https://www.federalreserve.gov/econres/files/scfp2022s.zip --source-page https://www.federalreserve.gov/econres/scfindex.htm --table 'Survey of Consumer Finances 2022 summary extract' --publisher 'Board of Governors of the Federal Reserve System' --vintage 2022 --access public --licence US-Government-Work --kind microdata_release --expected-sha256 3bb4d890ae2463ff6039ec7692e375f544dd98a55a37ca2cb2340354b9cc9d80 --licence-evidence-issuer 'Board of Governors of the Federal Reserve System' --licence-evidence-scope 'Public data set published by the Board of Governors of the Federal Reserve System, a federal agency; a work of the United States Government under 17 U.S.C. \u00a7105.' --licence-evidence-url TODO_EVIDENCE_URL --upload-r2 --r2-bucket ledger-raw", + "uv run chronicle fetch-artifact --source-id federal_reserve --package-id federal-reserve-scf-2022 --year 2022 --out-dir db/data/federal_reserve/scf_2022 --url https://www.federalreserve.gov/econres/files/scf2022s.zip --source-page https://www.federalreserve.gov/econres/scfindex.htm --table 'Survey of Consumer Finances 2022 full public data set' --publisher 'Board of Governors of the Federal Reserve System' --vintage 2022 --access public --licence US-Government-Work --kind microdata_release --expected-sha256 TODO_REVIEWED_SHA256 --licence-evidence-issuer 'Board of Governors of the Federal Reserve System' --licence-evidence-scope 'Public data set published by the Board of Governors of the Federal Reserve System, a federal agency; a work of the United States Government under 17 U.S.C. \u00a7105.' --licence-evidence-url TODO_EVIDENCE_URL --upload-r2 --r2-bucket ledger-raw", + "uv run chronicle fetch-artifact --source-id census_sipp --package-id census-sipp-2023 --year 2023 --out-dir db/data/census/sipp_2023 --url TODO_PUBLISHER_URL --source-page https://www.census.gov/programs-surveys/sipp.html --table 'Survey of Income and Program Participation 2023 public-use file' --publisher 'U.S. Census Bureau' --vintage 2023 --access public --licence US-Government-Work --kind microdata_release --expected-sha256 TODO_REVIEWED_SHA256 --licence-evidence-issuer 'U.S. Census Bureau' --licence-evidence-scope 'Public-use microdata file published by the U.S. Census Bureau, a federal agency; a work of the United States Government under 17 U.S.C. \u00a7105.' --licence-evidence-url TODO_EVIDENCE_URL --upload-r2 --r2-bucket ledger-raw" + ], + "todos": [ + "census-cps-asec-2023: no durable licence-evidence URL is catalogued; replace TODO_EVIDENCE_URL with the publisher's statement that this file is issued under US-Government-Work before running this command.", + "census-cps-basic-monthly-2024: Microcosm records no publisher URL (field 'locator'); read it off https://www2.census.gov/programs-surveys/cps/datasets/2024/basic/.", + "census-cps-basic-monthly-2024: Microcosm pins no checksum for this release. A public release is archived only against a reviewed checksum; replace TODO_REVIEWED_SHA256 before running this command.", + "census-cps-basic-monthly-2024: no durable licence-evidence URL is catalogued; replace TODO_EVIDENCE_URL with the publisher's statement that this file is issued under US-Government-Work before running this command.", + "census-cps-basic-monthly-2024: Microcosm pins the twelve monthly files as the locator string 'jan24pub through dec24pub' with no per-file URL, filename extension, or checksum. The publisher directory is the stage source; the twelve filenames must be read off that directory before the fetch commands can be completed.", + "census-acs-pums-2022-household: Microcosm's pinned sha256 0b319b496f19a6913066f9c5ea572edfda3d78a187be6f375846617d0b441bd4 is NOT the publisher artifact's checksum. A public release is archived only against a reviewed checksum its licence evidence covers; review the publisher bytes and replace TODO_REVIEWED_SHA256 before running this command.", + "census-acs-pums-2022-household: no durable licence-evidence URL is catalogued; replace TODO_EVIDENCE_URL with the publisher's statement that this file is issued under US-Government-Work before running this command.", + "census-acs-pums-2022-household: The Microcosm artifact's own sha256 belongs to the derived acs_2022.h5, not to the publisher zip; only the publisher URL is reused here. The fetch computes the release checksum.", + "census-acs-pums-2022-person: Microcosm's pinned sha256 0b319b496f19a6913066f9c5ea572edfda3d78a187be6f375846617d0b441bd4 is NOT the publisher artifact's checksum. A public release is archived only against a reviewed checksum its licence evidence covers; review the publisher bytes and replace TODO_REVIEWED_SHA256 before running this command.", + "census-acs-pums-2022-person: no durable licence-evidence URL is catalogued; replace TODO_EVIDENCE_URL with the publisher's statement that this file is issued under US-Government-Work before running this command.", + "census-acs-pums-2022-person: The Microcosm artifact's own sha256 belongs to the derived acs_2022.h5, not to the publisher zip; only the publisher URL is reused here. The fetch computes the release checksum.", + "census-acs-pums-2024-household: no durable licence-evidence URL is catalogued; replace TODO_EVIDENCE_URL with the publisher's statement that this file is issued under US-Government-Work before running this command.", + "census-acs-pums-2024-person: no durable licence-evidence URL is catalogued; replace TODO_EVIDENCE_URL with the publisher's statement that this file is issued under US-Government-Work before running this command.", + "federal-reserve-scf-2022-summary: no durable licence-evidence URL is catalogued; replace TODO_EVIDENCE_URL with the publisher's statement that this file is issued under US-Government-Work before running this command.", + "federal-reserve-scf-2022-full: Microcosm pins no checksum for this release. A public release is archived only against a reviewed checksum; replace TODO_REVIEWED_SHA256 before running this command.", + "federal-reserve-scf-2022-full: no durable licence-evidence URL is catalogued; replace TODO_EVIDENCE_URL with the publisher's statement that this file is issued under US-Government-Work before running this command.", + "federal-reserve-scf-2022-full: Microcosm records no checksum for this zip: 'Full-file SHA-256 pending one network-enabled provisioning fetch'. The fetch below computes and registers it.", + "census-sipp-2023: Microcosm records no publisher URL (field 'locator'); read it off https://www.census.gov/programs-surveys/sipp.html.", + "census-sipp-2023: Microcosm's pinned sha256 5c30439e365fc26483318ef61d1d8f4bb2f0e9d6bb47c22c06756a7698733ee2 is NOT the publisher artifact's checksum. A public release is archived only against a reviewed checksum its licence evidence covers; review the publisher bytes and replace TODO_REVIEWED_SHA256 before running this command.", + "census-sipp-2023: no durable licence-evidence URL is catalogued; replace TODO_EVIDENCE_URL with the publisher's statement that this file is issued under US-Government-Work before running this command.", + "census-sipp-2023: Microcosm reaches this file through an immutable Hugging Face mirror (revision 21280dca5995e978d706740a8a4b9b7860cfd7b6) and records no Census URL, so the publisher URL must be read off the SIPP dataset page before the fetch. Microcosm's pinned sha256 5c30439e365fc26483318ef61d1d8f4bb2f0e9d6bb47c22c06756a7698733ee2 and size 3726010471 are for the mirrored pu2023.csv member, not for whatever archive the Census page serves, so they are not the checksum this fetch should be expected to reproduce." + ] +} diff --git a/tests/fixtures/microcosm/packages/microcosm-build/src/microcosm/build/be/source_stages.json b/tests/fixtures/microcosm/packages/microcosm-build/src/microcosm/build/be/source_stages.json new file mode 100644 index 00000000..9ea696a9 --- /dev/null +++ b/tests/fixtures/microcosm/packages/microcosm-build/src/microcosm/build/be/source_stages.json @@ -0,0 +1,27 @@ +{ + "snapshot_of": { + "repository": "PolicyEngine/microcosm", + "commit": "fe2f92f8c79d36b897e6ae4d7e4b3c33fc7ebd28", + "note": "Synthetic consumer manifests in Microcosm's shapes, carrying the reviewed pins verbatim; decoy artifacts exercise selection." + }, + "version": "fixture", + "country": "be", + "policy": "synthetic Microcosm BE source-stages manifest", + "stages": [ + { + "stage": "silc_load", + "survey": "BE-SILC (Statbel national SILC)", + "source": "https://statbel.fgov.be/en/themes/households/poverty-and-living-conditions", + "grain": "person", + "artifacts": [ + { + "kind": "restricted_microdata", + "format": "csv_or_spss", + "vintage": "2023", + "locator": "Statbel BE-SILC scientific-use files: D (household register), R (personal register), H (household data), P (personal data)", + "licence": "Statbel/Eurostat scientific-use; restricted" + } + ] + } + ] +} diff --git a/tests/fixtures/microcosm/packages/microcosm-build/src/microcosm/build/uk/hmrc_income_source_stages.json b/tests/fixtures/microcosm/packages/microcosm-build/src/microcosm/build/uk/hmrc_income_source_stages.json new file mode 100644 index 00000000..85411e3f --- /dev/null +++ b/tests/fixtures/microcosm/packages/microcosm-build/src/microcosm/build/uk/hmrc_income_source_stages.json @@ -0,0 +1,45 @@ +{ + "snapshot_of": { + "repository": "PolicyEngine/microcosm", + "commit": "fe2f92f8c79d36b897e6ae4d7e4b3c33fc7ebd28", + "note": "Synthetic consumer manifests in Microcosm's shapes, carrying the reviewed pins verbatim; decoy artifacts exercise selection." + }, + "version": "fixture", + "country": "uk", + "policy": "synthetic Microcosm HMRC income source-stages manifest", + "stages": [ + { + "stage": "hmrc_spi_income", + "survey": "Survey of Personal Incomes Public Use Tape 2022-23", + "source": "https://example.invalid/collated_tables.ods", + "grain": "person", + "artifacts": [ + { + "role": "qrf_donor", + "kind": "private_microdata", + "format": "tab_delimited", + "survey": "Survey of Personal Incomes Public Use Tape 2022-23", + "vintage": "2022-23", + "tax_year_start": 2022, + "ukds_study_number": "SN 9422", + "doi": "10.5255/UKDA-SN-9422-1", + "filename": "put2223uk.tab", + "sha256": "5ef829461060c91a2a47be59ad541d9b519fc3976d66ca80d4920f711bb96f66", + "size_bytes": 141323762, + "access": "private_local_input", + "locator": "caller-supplied local input", + "runtime_sha256_required": true + }, + { + "role": "published_fact_surface", + "kind": "administrative_table", + "format": "ods", + "vintage": "2023-24", + "locator": "https://example.invalid/collated_tables.ods", + "sha256": "1111111111111111111111111111111111111111111111111111111111111111", + "size_bytes": 2 + } + ] + } + ] +} diff --git a/tests/fixtures/microcosm/packages/microcosm-build/src/microcosm/build/uk/source_stages.json b/tests/fixtures/microcosm/packages/microcosm-build/src/microcosm/build/uk/source_stages.json new file mode 100644 index 00000000..22d59100 --- /dev/null +++ b/tests/fixtures/microcosm/packages/microcosm-build/src/microcosm/build/uk/source_stages.json @@ -0,0 +1,212 @@ +{ + "snapshot_of": { + "repository": "PolicyEngine/microcosm", + "commit": "fe2f92f8c79d36b897e6ae4d7e4b3c33fc7ebd28", + "note": "Synthetic consumer manifests in Microcosm's shapes, carrying the reviewed pins verbatim; decoy artifacts exercise selection." + }, + "version": "fixture", + "country": "uk", + "policy": "synthetic Microcosm source-stages manifest for Chronicle tests", + "stages": [ + { + "stage": "frs_spine", + "survey": "Family Resources Survey 2023-24", + "source": "Department for Work and Pensions Family Resources Survey 2023-24, UK Data Service SN 9367, DOI 10.5255/UKDA-SN-9367-2", + "grain": "household", + "artifacts": [ + { + "role": "frs_table", + "table": "accounts", + "kind": "licensed_microdata", + "format": "tab", + "vintage": "2023_24", + "locator": "accounts.tab", + "sha256": "c5e31932bfd06087f835d2c83c0984c85a93409bf5ef85b699cb0958abcba1ea", + "size_bytes": 1807921, + "runtime_sha256_required": true + }, + { + "role": "frs_table", + "table": "adult", + "kind": "licensed_microdata", + "format": "tab", + "vintage": "2023_24", + "locator": "adult.tab", + "sha256": "e09f9647d03585c81a528636028b2ed495f8f1fbcf64c5e7b4fe521b67367e06", + "size_bytes": 35323384, + "runtime_sha256_required": true + }, + { + "role": "frs_table", + "table": "benefits", + "kind": "licensed_microdata", + "format": "tab", + "vintage": "2023_24", + "locator": "benefits.tab", + "sha256": "ff30d054cc659bcf23b44c492d98cfd701c0bfdb63e8e9aa9769b490ba9d636b", + "size_bytes": 4460292, + "runtime_sha256_required": true + }, + { + "role": "frs_table", + "table": "benunit", + "kind": "licensed_microdata", + "format": "tab", + "vintage": "2023_24", + "locator": "benunit.tab", + "sha256": "88946815eace8561516d5cbb442c27e319c1e90abc381fb2338f0126e3b9e05b", + "size_bytes": 21213867, + "runtime_sha256_required": true + }, + { + "role": "frs_table", + "table": "child", + "kind": "licensed_microdata", + "format": "tab", + "vintage": "2023_24", + "locator": "child.tab", + "sha256": "b5dc84fe8b002ee925e61fae23fed27b11537af9fb174f1d07d9cc1748b9702e", + "size_bytes": 2913156, + "runtime_sha256_required": true + }, + { + "role": "frs_table", + "table": "chldcare", + "kind": "licensed_microdata", + "format": "tab", + "vintage": "2023_24", + "locator": "chldcare.tab", + "sha256": "566e0ebca1d5e2f3e424e556c91f4cb583d17dadfdfa59feb3841eda7e5976a3", + "size_bytes": 273837, + "runtime_sha256_required": true + }, + { + "role": "frs_table", + "table": "extchild", + "kind": "licensed_microdata", + "format": "tab", + "vintage": "2023_24", + "locator": "extchild.tab", + "sha256": "8d358d7ee66ee4a7ceab87b4f24fbbf21ac86dc038dc7831e51fb271f96a57ec", + "size_bytes": 18677, + "runtime_sha256_required": true + }, + { + "role": "frs_table", + "table": "househol", + "kind": "licensed_microdata", + "format": "tab", + "vintage": "2023_24", + "locator": "househol.tab", + "sha256": "5fd26b8b675f33b3b30c9ac789a18da17de734790f77e00ded287d1c3a187b30", + "size_bytes": 12387117, + "runtime_sha256_required": true + }, + { + "role": "frs_table", + "table": "job", + "kind": "licensed_microdata", + "format": "tab", + "vintage": "2023_24", + "locator": "job.tab", + "sha256": "88b77ffe06865f029f713bb1d55ff12bdea8a1234de5bc293e72458fe64f3a74", + "size_bytes": 10934873, + "runtime_sha256_required": true + }, + { + "role": "frs_table", + "table": "maint", + "kind": "licensed_microdata", + "format": "tab", + "vintage": "2023_24", + "locator": "maint.tab", + "sha256": "f2dc924eb5a51b0c357791693d15b431327dc39c6421011efb313d88bf839695", + "size_bytes": 15440, + "runtime_sha256_required": true + }, + { + "role": "frs_table", + "table": "mortgage", + "kind": "licensed_microdata", + "format": "tab", + "vintage": "2023_24", + "locator": "mortgage.tab", + "sha256": "ce36b477d67837c469608a0d68f7ef269ac04758974235f1157d2f6b92cdbfdc", + "size_bytes": 631783, + "runtime_sha256_required": true + }, + { + "role": "frs_table", + "table": "oddjob", + "kind": "licensed_microdata", + "format": "tab", + "vintage": "2023_24", + "locator": "oddjob.tab", + "sha256": "b4ba3dd3151f73a01422983c60514a3e38458ddfa4fb33ae4ed0326873406305", + "size_bytes": 5165, + "runtime_sha256_required": true + }, + { + "role": "frs_table", + "table": "penprov", + "kind": "licensed_microdata", + "format": "tab", + "vintage": "2023_24", + "locator": "penprov.tab", + "sha256": "ee001461c40306ec24b38b2881e1774121114266a2ee449d606cd0a811c37731", + "size_bytes": 522313, + "runtime_sha256_required": true + }, + { + "role": "frs_table", + "table": "pension", + "kind": "licensed_microdata", + "format": "tab", + "vintage": "2023_24", + "locator": "pension.tab", + "sha256": "150d6fad1fce81254fb7aea1526fbb00b63d4027d6e2ac4c26bb90aea3127eb7", + "size_bytes": 1225838, + "runtime_sha256_required": true + }, + { + "kind": "administrative_table", + "format": "ods", + "vintage": "2023-24", + "locator": "https://example.invalid/decoy_table.ods", + "sha256": "0000000000000000000000000000000000000000000000000000000000000000", + "size_bytes": 1 + } + ] + }, + { + "stage": "frs_employment", + "survey": "Family Resources Survey 2023-24", + "source": "UK Data Service SN 9252; local licensed 2023_24 tabs.", + "grain": "person", + "artifacts": [ + { + "role": "frs_table", + "table": "adult", + "kind": "licensed_microdata", + "format": "tab", + "vintage": "2023_24", + "locator": "adult.tab", + "sha256": "e09f9647d03585c81a528636028b2ed495f8f1fbcf64c5e7b4fe521b67367e06", + "size_bytes": 35323384, + "runtime_sha256_required": true + }, + { + "role": "frs_table", + "table": "job", + "kind": "licensed_microdata", + "format": "tab", + "vintage": "2023_24", + "locator": "job.tab", + "sha256": "88b77ffe06865f029f713bb1d55ff12bdea8a1234de5bc293e72458fe64f3a74", + "size_bytes": 10934873, + "runtime_sha256_required": true + } + ] + } + ] +} diff --git a/tests/fixtures/microcosm/packages/microcosm-build/src/microcosm/build/us/source_stages.json b/tests/fixtures/microcosm/packages/microcosm-build/src/microcosm/build/us/source_stages.json new file mode 100644 index 00000000..1523b8b1 --- /dev/null +++ b/tests/fixtures/microcosm/packages/microcosm-build/src/microcosm/build/us/source_stages.json @@ -0,0 +1,112 @@ +{ + "snapshot_of": { + "repository": "PolicyEngine/microcosm", + "commit": "fe2f92f8c79d36b897e6ae4d7e4b3c33fc7ebd28", + "note": "Synthetic consumer manifests in Microcosm's shapes, carrying the reviewed pins verbatim; decoy artifacts exercise selection." + }, + "version": "fixture", + "country": "us", + "policy": "synthetic Microcosm US source-stages manifest", + "stages": [ + { + "stage": "weeks_unemployed_input", + "survey": "Census CPS ASEC", + "source": "https://www.census.gov/programs-surveys/cps.html", + "grain": "person", + "artifacts": [ + { + "kind": "public_microdata", + "format": "zip_csv", + "vintage": "2023 ASEC / 2022 income reference year", + "locator": "https://www2.census.gov/programs-surveys/cps/datasets/2023/march/asecpub23csv.zip", + "sha256": "d2e000250782adfbdd7f29c82b66d866591a30f0d330496698ec19f9c784ce11", + "size_bytes": 150165063, + "member": "pppub23.csv" + }, + { + "kind": "official_data_dictionary", + "format": "pdf", + "vintage": "2023", + "locator": "https://example.invalid/asec2023_ddl_pub_full.pdf" + } + ] + }, + { + "stage": "org_wages", + "survey": "CPS ORG", + "source": "https://www2.census.gov/programs-surveys/cps/datasets/2024/basic/", + "grain": "person", + "artifacts": [ + { + "kind": "public_microdata", + "format": "cps_basic_monthly_csv_or_zip", + "vintage": "2024", + "locator": "jan24pub through dec24pub; HRMIS 4 and 8" + }, + { + "kind": "administrative_table", + "format": "published_table", + "vintage": "2024", + "locator": "https://example.invalid/union2.t01.1" + } + ] + }, + { + "stage": "acs_rent", + "survey": "Census ACS 2022", + "source": "https://www.census.gov/programs-surveys/acs", + "grain": "household", + "artifacts": [ + { + "kind": "public_microdata", + "format": "census_asec_hdf5", + "vintage": "2022-2024 pooled", + "locator": "SHA-locked Census CPS ASEC person and household tables" + }, + { + "kind": "versioned_derived_microdata", + "format": "hdf5_arrays", + "vintage": "2022", + "locator": "Census ACS 2022 PUMS processed person/household arrays (acs_2022.h5)", + "sha256": "0b319b496f19a6913066f9c5ea572edfda3d78a187be6f375846617d0b441bd4", + "official_person_source": "https://www2.census.gov/programs-surveys/acs/data/pums/2022/1-Year/csv_pus.zip", + "official_household_source": "https://www2.census.gov/programs-surveys/acs/data/pums/2022/1-Year/csv_hus.zip" + } + ] + }, + { + "stage": "scf_wealth", + "survey": "Fed SCF 2022 + Census SIPP 2023", + "source": "https://www.federalreserve.gov/econres/scfindex.htm", + "grain": "household", + "artifacts": [ + { + "kind": "public_microdata", + "format": "stata_in_zip", + "vintage": "2022", + "locator": "https://www.federalreserve.gov/econres/files/scfp2022s.zip", + "member": "rscfp2022.dta", + "sha256": "3bb4d890ae2463ff6039ec7692e375f544dd98a55a37ca2cb2340354b9cc9d80" + }, + { + "kind": "public_microdata", + "format": "stata_in_zip", + "vintage": "2022", + "locator": "https://www.federalreserve.gov/econres/files/scf2022s.zip", + "member": "p22i6.dta", + "expected_rows": 22975 + }, + { + "kind": "public_microdata", + "format": "pipe_delimited_csv", + "vintage": "2023", + "locator": "Census SIPP 2023 public-use file; immutable Hugging Face mirror revision 21280dca5995e978d706740a8a4b9b7860cfd7b6", + "member": "pu2023.csv", + "revision": "21280dca5995e978d706740a8a4b9b7860cfd7b6", + "sha256": "5c30439e365fc26483318ef61d1d8f4bb2f0e9d6bb47c22c06756a7698733ee2", + "size_bytes": 3726010471 + } + ] + } + ] +} diff --git a/tests/fixtures/microcosm/packages/microcosm-build/src/microcosm/build/us_runtime/acs_2024_1yr_sources.json b/tests/fixtures/microcosm/packages/microcosm-build/src/microcosm/build/us_runtime/acs_2024_1yr_sources.json new file mode 100644 index 00000000..389950b1 --- /dev/null +++ b/tests/fixtures/microcosm/packages/microcosm-build/src/microcosm/build/us_runtime/acs_2024_1yr_sources.json @@ -0,0 +1,28 @@ +{ + "snapshot_of": { + "repository": "PolicyEngine/microcosm", + "commit": "fe2f92f8c79d36b897e6ae4d7e4b3c33fc7ebd28", + "note": "Synthetic consumer manifests in Microcosm's shapes, carrying the reviewed pins verbatim; decoy artifacts exercise selection." + }, + "version": "fixture", + "spine": "acs_2024_1yr", + "vintage": "2024", + "verified_on": "2026-08-01", + "source_directory": "https://www2.census.gov/programs-surveys/acs/data/pums/2024/1-Year/", + "artifacts": [ + { + "role": "household", + "filename": "csv_hus.zip", + "url": "https://www2.census.gov/programs-surveys/acs/data/pums/2024/1-Year/csv_hus.zip", + "sha256": "8281008e53de98f0ef81e7a2ee5a8725991dda1ecfd2713ead73246425e515d0", + "size_bytes": 251500587 + }, + { + "role": "person", + "filename": "csv_pus.zip", + "url": "https://www2.census.gov/programs-surveys/acs/data/pums/2024/1-Year/csv_pus.zip", + "sha256": "afdc6d90c6e2f0bab365ed32d95ba4c4d8ac651162f46ac7861295b2dc469894", + "size_bytes": 602847146 + } + ] +} diff --git a/tests/test_chronicle_artifacts.py b/tests/test_chronicle_artifacts.py index 7104ec03..5658c32a 100644 --- a/tests/test_chronicle_artifacts.py +++ b/tests/test_chronicle_artifacts.py @@ -4,12 +4,17 @@ import hashlib import json +import sqlite3 import pytest import yaml from chronicle.cli import main as cli_main from chronicle.artifacts import ( + AmbiguousManifestError, + MalformedManifestError, + RecordedR2LocatorError, + SourceArtifactRevisionError, build_artifact_key, build_artifact_rows, build_derived_r2_key, @@ -174,6 +179,7 @@ def test_fetch_source_artifact_writes_manifest_and_inventory(tmp_path): assert inventory.counts == { "artifact_count": 1, "checksum_mismatch_count": 0, + "hash_only_count": 0, "manifest_count": 1, "missing_count": 0, "r2_link_count": 0, @@ -240,8 +246,10 @@ def test_publish_source_artifacts_uploads_manifest_entries(tmp_path): assert report.counts == { "artifact_count": 1, "failed_count": 0, + "hash_only_refused_count": 0, "manifest_count": 1, "r2_link_count": 1, + "skipped_count": 0, "uploaded_count": 1, } assert storage["bucket"] == "ledger-raw" @@ -658,3 +666,1081 @@ def test_top_level_cli_dispatches_publish_derived(tmp_path, capsys, monkeypatch) assert exc.value.code == 0 assert payload["valid"] + + +def _sqlite_build(path, build_id): + """Write a minimal build database carrying one ledger_builds row.""" + with sqlite3.connect(path) as connection: + connection.execute("CREATE TABLE ledger_builds (build_id TEXT PRIMARY KEY)") + connection.execute("INSERT INTO ledger_builds VALUES (?)", (build_id,)) + + +@pytest.mark.parametrize("db_name", ["chronicle.db", "ledger.db"]) +def test_infer_build_id_reads_new_and_legacy_database_names(tmp_path, db_name): + suite = tmp_path / "suite" + suite.mkdir() + _sqlite_build(suite / db_name, "ledger.build.v1:from-db") + + assert infer_build_id(suite) == "ledger.build.v1:from-db" + + +def test_infer_build_id_prefers_the_chronicle_database(tmp_path): + suite = tmp_path / "suite" + suite.mkdir() + _sqlite_build(suite / "chronicle.db", "ledger.build.v1:chronicle") + _sqlite_build(suite / "ledger.db", "ledger.build.v1:legacy") + + assert infer_build_id(suite) == "ledger.build.v1:chronicle" + + +@pytest.mark.parametrize("db_name", ["chronicle.db", "ledger.db"]) +def test_publish_derived_classifies_both_database_names(tmp_path, db_name): + suite = tmp_path / "suite" + reports = suite / "reports" + reports.mkdir(parents=True) + build_id = "ledger.build.v1:kind" + (reports / "database.json").write_text(json.dumps({"build_id": build_id})) + (suite / db_name).write_bytes(b"db") + wrangler = tmp_path / "wrangler" + wrangler.write_text("#!/bin/sh\necho ok\n") + wrangler.chmod(0o755) + + report = publish_derived_artifacts( + suite, + source_id="irs_soi", + package_id="soi-table-1-1", + year=2023, + wrangler_command=str(wrangler), + ) + rows = {row["artifact_name"]: row for row in build_artifact_rows(report)} + + assert rows[db_name]["artifact_kind"] == "sqlite_database" + + +def test_publish_derived_uses_the_configured_bucket(tmp_path, monkeypatch): + monkeypatch.setenv("CHRONICLE_R2_DERIVED_BUCKET", "chronicle-derived") + suite = tmp_path / "suite" + reports = suite / "reports" + reports.mkdir(parents=True) + (reports / "database.json").write_text( + json.dumps({"build_id": "ledger.build.v1:bucket"}) + ) + (suite / "facts.jsonl").write_text("{}\n") + log = tmp_path / "wrangler.log" + wrangler = tmp_path / "wrangler" + wrangler.write_text(f"#!/bin/sh\nprintf '%s\\n' \"$*\" >> {log}\necho ok\n") + wrangler.chmod(0o755) + + report = publish_derived_artifacts( + suite, + source_id="irs_soi", + package_id="soi-table-1-1", + year=2023, + wrangler_command=str(wrangler), + ) + + assert report.valid + assert report.entries[0].r2_location.bucket == "chronicle-derived" + assert "chronicle-derived/derived/irs_soi/" in log.read_text() + + +def test_publish_raw_skips_an_object_already_held_by_a_preserved_bucket( + tmp_path, monkeypatch +): + """A recorded storage.r2 bucket is preserved history, not a publish target. + + Archived witness records pin raw R2 URLs by hash, so backfilling the same + bytes into a renamed bucket must not rewrite the manifest. The entry is + already published, so the sweep reports it skipped and stays green: after + the bucket-default flip every entry published before it takes this path. + """ + monkeypatch.setenv("CHRONICLE_R2_RAW_BUCKET", "chronicle-raw") + output_dir = tmp_path / "db" / "data" / "irs_soi" / "soi-table-1-1" + source = tmp_path / "soi.xlsx" + source.write_bytes(b"official SOI workbook") + fetch_source_artifact( + str(source), + source_id="irs_soi", + package_id="soi-table-1-1", + year=2023, + output_dir=output_dir, + ) + manifest_path = output_dir / "manifest.yaml" + manifest = yaml.safe_load(manifest_path.read_text()) + artifact = manifest["files"][2023] + recorded_key = ( + f"raw/irs_soi/soi-table-1-1/2023/{artifact['sha256']}/{artifact['filename']}" + ) + artifact["storage"] = { + "r2": { + "provider": "r2", + "bucket": "ledger-raw", + "key": recorded_key, + "uri": f"r2://ledger-raw/{recorded_key}", + } + } + manifest_path.write_text(yaml.safe_dump(manifest, sort_keys=False)) + log = tmp_path / "wrangler.log" + wrangler = tmp_path / "wrangler" + wrangler.write_text(f"#!/bin/sh\nprintf '%s\\n' \"$*\" >> {log}\necho ok\n") + wrangler.chmod(0o755) + + before = manifest_path.read_bytes() + report = publish_source_artifacts(output_dir, wrangler_command=str(wrangler)) + entry = report.entries[0] + + assert report.valid + assert entry.errors == () + assert entry.upload is None + assert entry.skipped == ( + "recorded_r2_bucket_is_preserved_history:" + "recorded=ledger-raw:requested=chronicle-raw" + ) + assert entry.r2_location is not None + assert entry.r2_location.bucket == "ledger-raw" + assert entry.r2_location.key == recorded_key + assert entry.to_dict()["skipped"] == entry.skipped + assert report.counts["skipped_count"] == 1 + assert report.counts["uploaded_count"] == 0 + assert report.counts["failed_count"] == 0 + assert not log.exists() + assert manifest_path.read_bytes() == before + + +def test_fetch_artifact_keeps_an_already_recorded_bucket(tmp_path, monkeypatch): + output_dir = tmp_path / "db" / "data" / "irs_soi" / "soi-table-1-1" + source = tmp_path / "soi.xlsx" + source.write_bytes(b"official SOI workbook") + log = tmp_path / "wrangler.log" + wrangler = tmp_path / "wrangler" + wrangler.write_text(f"#!/bin/sh\nprintf '%s\\n' \"$*\" >> {log}\necho ok\n") + wrangler.chmod(0o755) + fetch_source_artifact( + str(source), + source_id="irs_soi", + package_id="soi-table-1-1", + year=2023, + output_dir=output_dir, + upload_r2=True, + wrangler_command=str(wrangler), + ) + manifest_path = output_dir / "manifest.yaml" + first = yaml.safe_load(manifest_path.read_text()) + assert first["files"][2023]["storage"]["r2"]["bucket"] == "ledger-raw" + + monkeypatch.setenv("CHRONICLE_R2_RAW_BUCKET", "chronicle-raw") + report = fetch_source_artifact( + str(source), + source_id="irs_soi", + package_id="soi-table-1-1", + year=2023, + output_dir=output_dir, + upload_r2=True, + wrangler_command=str(wrangler), + ) + second = yaml.safe_load(manifest_path.read_text()) + + # The backfill copy really is uploaded to the new bucket, but the manifest + # keeps recording where the bytes were first published. + assert report.r2_location.bucket == "chronicle-raw" + assert "chronicle-raw" in log.read_text() + assert ( + second["files"][2023]["storage"]["r2"] == first["files"][2023]["storage"]["r2"] + ) + + +# --------------------------------------------------------------------------- +# Publisher revisions +# +# A raw R2 key is content-addressed, so a recorded storage.r2 block is a claim +# about specific bytes. On 2026-09-02 the IRS re-published 22in05ira.xlsx and +# 22in06ira.xlsx under their existing URLs (PolicyEngine/chronicle#225): a +# repeated fetch must never pair those new bytes with the old object's URI. +# --------------------------------------------------------------------------- + +REPUBLISHED_URL = "https://www.irs.gov/pub/irs-soi/22in05ira.xlsx" +REPUBLISHED_FILENAME = "22in05ira.xlsx" +FIRST_PUBLICATION = b"IRA table 5, first publication" +SECOND_PUBLICATION = b"IRA table 5, silently re-published with revised rows" + + +def _wrangler_stub(tmp_path, log): + wrangler = tmp_path / "wrangler" + wrangler.write_text(f"#!/bin/sh\nprintf '%s\\n' \"$*\" >> {log}\necho ok\n") + wrangler.chmod(0o755) + return wrangler + + +def _serve(monkeypatch, content): + """Serve ``content`` from the publisher URL, without touching the network.""" + + def _fake_read_artifact(source_url): + assert source_url == REPUBLISHED_URL + return content, REPUBLISHED_FILENAME + + monkeypatch.setattr("chronicle.artifacts._read_artifact", _fake_read_artifact) + + +def _fetch_republished(output_dir, wrangler, *, upload_r2=True, **kwargs): + return fetch_source_artifact( + REPUBLISHED_URL, + source_id="irs_soi", + package_id="soi-table-5", + year=2022, + output_dir=output_dir, + upload_r2=upload_r2, + wrangler_command=str(wrangler), + **kwargs, + ) + + +def test_repeated_fetch_of_identical_bytes_preserves_the_recorded_block( + tmp_path, monkeypatch +): + """Same bytes: the recorded block survives whatever bucket is configured.""" + output_dir = tmp_path / "db" / "data" / "irs_soi" / "soi-table-5" + log = tmp_path / "wrangler.log" + wrangler = _wrangler_stub(tmp_path, log) + manifest_path = output_dir / "manifest.yaml" + _serve(monkeypatch, FIRST_PUBLICATION) + _fetch_republished(output_dir, wrangler) + first = yaml.safe_load(manifest_path.read_text())["files"][2022] + + monkeypatch.setenv("CHRONICLE_R2_RAW_BUCKET", "chronicle-raw") + report = _fetch_republished(output_dir, wrangler) + second = yaml.safe_load(manifest_path.read_text())["files"][2022] + + assert report.valid + # The backfill copy really goes to the renamed bucket, but the manifest + # keeps recording where these bytes were first published. + assert report.r2_location.bucket == "chronicle-raw" + assert "chronicle-raw" in log.read_text() + assert second["storage"] == first["storage"] + # Field order too, so the block is byte-for-byte identical once dumped. + assert list(second["storage"]["r2"].items()) == list(first["storage"]["r2"].items()) + assert second["storage"]["r2"]["bucket"] == "ledger-raw" + assert "previous_r2" not in second["storage"] + assert second["sha256"] == first["sha256"] + + +@pytest.mark.parametrize( + ("upload_r2", "configured_bucket"), + [ + pytest.param(True, None, id="reuploaded"), + # The two routes that reached a manifest in the wild: a fetch that only + # registers the bytes, and a fetch once the bucket default has moved. + # Both preserved the recorded block while rewriting sha256/size_bytes. + pytest.param(False, None, id="registered-without-upload"), + pytest.param(True, "chronicle-raw", id="after-the-bucket-rename"), + ], +) +def test_repeated_fetch_of_different_bytes_is_refused( + tmp_path, monkeypatch, upload_r2, configured_bucket +): + """A publisher revision must not inherit the recorded object's provenance.""" + output_dir = tmp_path / "db" / "data" / "irs_soi" / "soi-table-5" + log = tmp_path / "wrangler.log" + wrangler = _wrangler_stub(tmp_path, log) + manifest_path = output_dir / "manifest.yaml" + artifact_path = output_dir / REPUBLISHED_FILENAME + _serve(monkeypatch, FIRST_PUBLICATION) + first_report = _fetch_republished(output_dir, wrangler) + manifest_before = manifest_path.read_bytes() + uploads_before = log.read_text() + + if configured_bucket: + monkeypatch.setenv("CHRONICLE_R2_RAW_BUCKET", configured_bucket) + _serve(monkeypatch, SECOND_PUBLICATION) + with pytest.raises(SourceArtifactRevisionError) as raised: + _fetch_republished(output_dir, wrangler, upload_r2=upload_r2) + + message = str(raised.value) + assert first_report.sha256 in message + assert hashlib.sha256(SECOND_PUBLICATION).hexdigest() in message + assert f"size_bytes={len(FIRST_PUBLICATION)}" in message + assert f"size_bytes={len(SECOND_PUBLICATION)}" in message + assert "release revision" in message + assert "--record-revision" in message + # Nothing was overwritten, copied or uploaded on the way to the refusal. + assert manifest_path.read_bytes() == manifest_before + assert artifact_path.read_bytes() == FIRST_PUBLICATION + assert log.read_text() == uploads_before + + +def test_record_revision_writes_a_new_key_and_keeps_the_previous_object( + tmp_path, monkeypatch +): + """The opt-in records the new bytes' own key under the configured bucket.""" + output_dir = tmp_path / "db" / "data" / "irs_soi" / "soi-table-5" + log = tmp_path / "wrangler.log" + wrangler = _wrangler_stub(tmp_path, log) + manifest_path = output_dir / "manifest.yaml" + _serve(monkeypatch, FIRST_PUBLICATION) + first_report = _fetch_republished(output_dir, wrangler) + superseded = yaml.safe_load(manifest_path.read_text())["files"][2022] + + monkeypatch.setenv("CHRONICLE_R2_RAW_BUCKET", "chronicle-raw") + _serve(monkeypatch, SECOND_PUBLICATION) + report = _fetch_republished(output_dir, wrangler, record_revision=True) + revised = yaml.safe_load(manifest_path.read_text())["files"][2022] + revised_sha256 = hashlib.sha256(SECOND_PUBLICATION).hexdigest() + + assert report.valid + # storage.r2 names the object that holds the entry's current bytes... + assert revised["sha256"] == revised_sha256 + assert revised["size_bytes"] == len(SECOND_PUBLICATION) + assert revised["storage"]["r2"]["bucket"] == "chronicle-raw" + assert revised["storage"]["r2"]["key"] == ( + f"raw/irs_soi/soi-table-5/2022/{revised_sha256}/{REPUBLISHED_FILENAME}" + ) + assert revised["storage"]["r2"]["uri"] == ( + f"r2://chronicle-raw/{revised['storage']['r2']['key']}" + ) + # ...and never the superseded key, which stays addressable as history. + previous = revised["storage"]["previous_r2"] + assert [entry["uri"] for entry in previous] == [superseded["storage"]["r2"]["uri"]] + assert previous[0]["bucket"] == "ledger-raw" + assert previous[0]["sha256"] == first_report.sha256 + assert previous[0]["size_bytes"] == len(FIRST_PUBLICATION) + assert previous[0]["fetched_at"] == superseded["fetched_at"] + assert previous[0]["superseded_at"] == revised["fetched_at"] + assert (output_dir / REPUBLISHED_FILENAME).read_bytes() == SECOND_PUBLICATION + assert f"chronicle-raw/{revised['storage']['r2']['key']}" in log.read_text() + + +def test_a_revised_manifest_still_reads_as_one_r2_linked_artifact( + tmp_path, monkeypatch +): + """storage.previous_r2 is a sibling key, so every storage.r2 reader is intact.""" + output_dir = tmp_path / "db" / "data" / "irs_soi" / "soi-table-5" + wrangler = _wrangler_stub(tmp_path, tmp_path / "wrangler.log") + _serve(monkeypatch, FIRST_PUBLICATION) + _fetch_republished(output_dir, wrangler) + _serve(monkeypatch, SECOND_PUBLICATION) + _fetch_republished(output_dir, wrangler, record_revision=True) + + inventory = inventory_source_artifacts(output_dir) + + assert inventory.valid + assert inventory.counts["r2_link_count"] == 1 + assert inventory.counts["checksum_mismatch_count"] == 0 + assert inventory.entries[0].r2["bucket"] == "ledger-raw" + assert inventory.entries[0].sha256_actual == ( + hashlib.sha256(SECOND_PUBLICATION).hexdigest() + ) + + +def test_publish_raw_refuses_a_file_the_recorded_object_does_not_hold( + tmp_path, monkeypatch +): + """Recorded sha256 != local sha256 is a revision, not a backfill.""" + output_dir = tmp_path / "db" / "data" / "irs_soi" / "soi-table-5" + log = tmp_path / "wrangler.log" + wrangler = _wrangler_stub(tmp_path, log) + manifest_path = output_dir / "manifest.yaml" + _serve(monkeypatch, FIRST_PUBLICATION) + first_report = _fetch_republished(output_dir, wrangler) + + # Reproduce the state a pre-fix fetch left behind: new bytes on disk, the + # entry's own hash rewritten, the recorded key still addressing the old + # bytes. + revised_sha256 = hashlib.sha256(SECOND_PUBLICATION).hexdigest() + (output_dir / REPUBLISHED_FILENAME).write_bytes(SECOND_PUBLICATION) + manifest = yaml.safe_load(manifest_path.read_text()) + manifest["files"][2022]["sha256"] = revised_sha256 + manifest["files"][2022]["size_bytes"] = len(SECOND_PUBLICATION) + manifest_path.write_text(yaml.safe_dump(manifest, sort_keys=False)) + manifest_before = manifest_path.read_bytes() + uploads_before = log.read_text() + + report = publish_source_artifacts(output_dir, wrangler_command=str(wrangler)) + + assert not report.valid + assert report.entries[0].upload is None + assert report.entries[0].r2_location is None + assert report.entries[0].errors == ( + "recorded_r2_identity_mismatch:" + f"recorded_sha256={first_report.sha256}:" + f"recorded_filename={REPUBLISHED_FILENAME}:" + f"local_sha256={revised_sha256}:" + f"local_filename={REPUBLISHED_FILENAME}", + ) + assert log.read_text() == uploads_before + assert manifest_path.read_bytes() == manifest_before + + +def test_publish_raw_uploads_a_registered_revision_and_keeps_its_history( + tmp_path, monkeypatch +): + """Once the revision is registered, publishing it is ordinary work.""" + output_dir = tmp_path / "db" / "data" / "irs_soi" / "soi-table-5" + log = tmp_path / "wrangler.log" + wrangler = _wrangler_stub(tmp_path, log) + manifest_path = output_dir / "manifest.yaml" + _serve(monkeypatch, FIRST_PUBLICATION) + first_report = _fetch_republished(output_dir, wrangler) + _serve(monkeypatch, SECOND_PUBLICATION) + _fetch_republished(output_dir, wrangler, record_revision=True) + revised_sha256 = hashlib.sha256(SECOND_PUBLICATION).hexdigest() + + report = publish_source_artifacts(output_dir, wrangler_command=str(wrangler)) + published = yaml.safe_load(manifest_path.read_text())["files"][2022] + + assert report.valid + assert published["storage"]["r2"]["key"].endswith( + f"/{revised_sha256}/{REPUBLISHED_FILENAME}" + ) + assert [entry["sha256"] for entry in published["storage"]["previous_r2"]] == [ + first_report.sha256 + ] + + +def test_fetch_artifact_cli_refuses_a_revision_then_records_it_on_request( + tmp_path, monkeypatch, capsys +): + output_dir = tmp_path / "db" / "data" / "irs_soi" / "soi-table-5" + wrangler = _wrangler_stub(tmp_path, tmp_path / "wrangler.log") + argv = [ + "fetch-artifact", + "--url", + REPUBLISHED_URL, + "--source-id", + "irs_soi", + "--package-id", + "soi-table-5", + "--year", + "2022", + "--out-dir", + str(output_dir), + "--upload-r2", + "--wrangler-command", + str(wrangler), + ] + _serve(monkeypatch, FIRST_PUBLICATION) + assert harness_main(argv) == 0 + capsys.readouterr() + + _serve(monkeypatch, SECOND_PUBLICATION) + refused = harness_main(argv) + refusal = capsys.readouterr() + + assert refused == 1 + assert "--record-revision" in refusal.err + assert refusal.out == "" + + assert harness_main([*argv, "--record-revision"]) == 0 + recorded = json.loads(capsys.readouterr().out) + + assert recorded["sha256"] == hashlib.sha256(SECOND_PUBLICATION).hexdigest() + assert recorded["r2_location"]["key"].endswith( + f"/{recorded['sha256']}/{REPUBLISHED_FILENAME}" + ) + + +def test_record_revision_without_an_upload_records_no_current_object( + tmp_path, monkeypatch +): + """An offline revision keeps history without claiming the new bytes exist. + + Registering a revision without ``--upload-r2`` leaves the entry with no + ``storage.r2`` at all rather than a pointer to bytes R2 does not hold. The + superseded object stays addressable, and a later publish-raw completes the + registration. + """ + output_dir = tmp_path / "db" / "data" / "irs_soi" / "soi-table-5" + log = tmp_path / "wrangler.log" + wrangler = _wrangler_stub(tmp_path, log) + manifest_path = output_dir / "manifest.yaml" + _serve(monkeypatch, FIRST_PUBLICATION) + first_report = _fetch_republished(output_dir, wrangler) + + _serve(monkeypatch, SECOND_PUBLICATION) + _fetch_republished(output_dir, wrangler, upload_r2=False, record_revision=True) + registered = yaml.safe_load(manifest_path.read_text())["files"][2022] + + assert "r2" not in registered["storage"] + assert [entry["sha256"] for entry in registered["storage"]["previous_r2"]] == [ + first_report.sha256 + ] + + report = publish_source_artifacts(output_dir, wrangler_command=str(wrangler)) + published = yaml.safe_load(manifest_path.read_text())["files"][2022] + revised_sha256 = hashlib.sha256(SECOND_PUBLICATION).hexdigest() + + assert report.valid + assert published["storage"]["r2"]["key"].endswith( + f"/{revised_sha256}/{REPUBLISHED_FILENAME}" + ) + assert [entry["sha256"] for entry in published["storage"]["previous_r2"]] == [ + first_report.sha256 + ] + + +def test_a_recorded_block_that_only_carries_a_uri_is_still_recognized( + tmp_path, monkeypatch +): + """Identity reads the URI when a hand-written block records no key.""" + output_dir = tmp_path / "db" / "data" / "irs_soi" / "soi-table-5" + wrangler = _wrangler_stub(tmp_path, tmp_path / "wrangler.log") + manifest_path = output_dir / "manifest.yaml" + _serve(monkeypatch, FIRST_PUBLICATION) + _fetch_republished(output_dir, wrangler) + manifest = yaml.safe_load(manifest_path.read_text()) + recorded = manifest["files"][2022]["storage"]["r2"] + manifest["files"][2022]["storage"]["r2"] = { + "provider": recorded["provider"], + "bucket": recorded["bucket"], + "uri": recorded["uri"], + } + manifest_path.write_text(yaml.safe_dump(manifest, sort_keys=False)) + uri_only = manifest["files"][2022]["storage"]["r2"] + + report = _fetch_republished(output_dir, wrangler) + preserved = yaml.safe_load(manifest_path.read_text())["files"][2022] + + assert report.valid + assert preserved["storage"]["r2"] == uri_only + + _serve(monkeypatch, SECOND_PUBLICATION) + with pytest.raises(SourceArtifactRevisionError): + _fetch_republished(output_dir, wrangler) + + +# --------------------------------------------------------------------------- +# Manifest addressing, entry identity, and recorded locators +# +# Everything below concerns the state a fetch reads before it writes: which +# manifest it reads, what that manifest's entry says its vintage holds, and +# whether the recorded R2 block names one object or two. +# --------------------------------------------------------------------------- + +TRADITIONAL_MANIFEST = "manifest_traditional_source_package.yaml" +ROTH_MANIFEST = "manifest_roth_source_package.yaml" + + +def _publish(tmp_path, name, content): + """Write bytes a fetch can read as a local publisher path.""" + path = tmp_path / "publisher" / name + path.parent.mkdir(parents=True, exist_ok=True) + path.write_bytes(content) + return path + + +def _fetch_local(output_dir, source_path, *, package_id="soi-table-5", **kwargs): + return fetch_source_artifact( + str(source_path), + source_id="irs_soi", + package_id=package_id, + year=2022, + output_dir=output_dir, + **kwargs, + ) + + +def _entry(manifest_path): + return yaml.safe_load(manifest_path.read_text())["files"][2022] + + +def test_fetch_artifact_writes_the_manifest_it_was_given(tmp_path): + """One publisher directory, two source packages, two manifests. + + db/data/irs_soi/ira_contributions keeps the traditional and Roth IRA + packages side by side. A fetch that always wrote manifest.yaml would write + a third manifest neither package reads. + """ + package = tmp_path / "db" / "data" / "irs_soi" / "ira_contributions" + traditional = _publish(tmp_path, "22in05ira.xlsx", b"traditional IRA table") + roth = _publish(tmp_path, "22in06ira.xlsx", b"roth IRA table") + + _fetch_local( + package, + traditional, + package_id="soi-ira-traditional-contributions-2022", + manifest_filename=TRADITIONAL_MANIFEST, + ) + _fetch_local( + package, + roth, + package_id="soi-ira-roth-contributions-2022", + manifest_filename=ROTH_MANIFEST, + ) + + assert sorted(path.name for path in package.glob("manifest*.yaml")) == [ + ROTH_MANIFEST, + TRADITIONAL_MANIFEST, + ] + assert not (package / "manifest.yaml").exists() + assert _entry(package / TRADITIONAL_MANIFEST)["filename"] == "22in05ira.xlsx" + assert _entry(package / ROTH_MANIFEST)["filename"] == "22in06ira.xlsx" + assert _entry(package / TRADITIONAL_MANIFEST)["sha256"] == ( + hashlib.sha256(b"traditional IRA table").hexdigest() + ) + + +def test_a_revision_is_refused_in_the_manifest_that_records_it(tmp_path): + """The IRA revision workflow the docs cite, on a two-manifest package.""" + package = tmp_path / "db" / "data" / "irs_soi" / "ira_contributions" + traditional = _publish(tmp_path, "22in05ira.xlsx", b"traditional IRA table") + _fetch_local( + package, + traditional, + package_id="soi-ira-traditional-contributions-2022", + manifest_filename=TRADITIONAL_MANIFEST, + ) + recorded = (package / TRADITIONAL_MANIFEST).read_bytes() + + # The IRS re-publishes under the same URL and vintage. + traditional.write_bytes(b"traditional IRA table, revised rows") + + with pytest.raises(SourceArtifactRevisionError) as raised: + _fetch_local( + package, + traditional, + package_id="soi-ira-traditional-contributions-2022", + manifest_filename=TRADITIONAL_MANIFEST, + ) + + assert TRADITIONAL_MANIFEST in str(raised.value) + assert (package / TRADITIONAL_MANIFEST).read_bytes() == recorded + assert (package / "22in05ira.xlsx").read_bytes() == b"traditional IRA table" + + # Without the flag the same fetch would address a manifest.yaml that no + # package reads and that protects nothing: the #225 path. It is refused, + # naming the manifests the directory keeps, and nothing is written. + with pytest.raises(AmbiguousManifestError) as stray: + _fetch_local( + package, + traditional, + package_id="soi-ira-traditional-contributions-2022", + ) + + assert TRADITIONAL_MANIFEST in str(stray.value) + assert "--manifest" in str(stray.value) + assert not (package / "manifest.yaml").exists() + assert (package / TRADITIONAL_MANIFEST).read_bytes() == recorded + assert (package / "22in05ira.xlsx").read_bytes() == b"traditional IRA table" + + +def test_fetch_artifact_cli_refuses_a_stray_default_manifest(tmp_path, capsys): + package = tmp_path / "db" / "data" / "irs_soi" / "ira_contributions" + traditional = _publish(tmp_path, "22in05ira.xlsx", b"traditional IRA table") + _fetch_local( + package, + traditional, + package_id="soi-ira-traditional-contributions-2022", + manifest_filename=TRADITIONAL_MANIFEST, + ) + argv = [ + "fetch-artifact", + "--url", + str(traditional), + "--source-id", + "irs_soi", + "--package-id", + "soi-ira-traditional-contributions-2022", + "--year", + "2022", + "--out-dir", + str(package), + ] + + assert harness_main(argv) == 1 + + err = capsys.readouterr().err + assert err.startswith("error: ") + assert TRADITIONAL_MANIFEST in err + assert not (package / "manifest.yaml").exists() + + +def test_a_same_bytes_rename_is_refused_by_name_not_as_a_revision(tmp_path): + """Identical bytes under another filename are neither a revision nor a + re-fetch: the entry's filename must keep agreeing with its recorded key.""" + package = tmp_path / "db" / "data" / "irs_soi" / "soi-table-5" + source = _publish(tmp_path, "22in05ira.xlsx", b"IRA table 5") + _fetch_local(package, source, filename="table-5.xlsx", upload_r2=False) + recorded = (package / "manifest.yaml").read_bytes() + + for record_revision in (False, True): + with pytest.raises(SourceArtifactRevisionError) as raised: + _fetch_local( + package, source, upload_r2=False, record_revision=record_revision + ) + message = str(raised.value) + assert "rename is not a release revision" in message + assert "filename=table-5.xlsx" in message + assert "names them 22in05ira.xlsx" in message + assert "--filename table-5.xlsx" in message + + assert (package / "manifest.yaml").read_bytes() == recorded + assert not (package / "22in05ira.xlsx").exists() + assert (package / "table-5.xlsx").read_bytes() == b"IRA table 5" + + +@pytest.mark.parametrize( + "manifest_filename", + ["../manifest.yaml", "nested/manifest.yaml", "", " ", ".", ".."], +) +def test_a_manifest_name_must_stay_inside_the_package(tmp_path, manifest_filename): + package = tmp_path / "db" / "data" / "irs_soi" / "soi-table-5" + source = _publish(tmp_path, "table.xlsx", b"table") + + with pytest.raises(ValueError, match="inside the package directory"): + _fetch_local(package, source, manifest_filename=manifest_filename) + + assert not package.exists() + + +def test_fetch_artifact_cli_reports_a_manifest_name_outside_the_package( + tmp_path, capsys +): + package = tmp_path / "db" / "data" / "irs_soi" / "soi-table-5" + source = _publish(tmp_path, "table.xlsx", b"table") + argv = [ + "fetch-artifact", + "--url", + str(source), + "--source-id", + "irs_soi", + "--package-id", + "soi-table-5", + "--year", + "2022", + "--out-dir", + str(package), + "--manifest", + "../manifest.yaml", + ] + + assert harness_main(argv) == 1 + + err = capsys.readouterr().err + assert err.startswith("error: ") + assert "inside the package directory" in err + assert not package.exists() + assert not (tmp_path / "db" / "data" / "irs_soi" / "manifest.yaml").exists() + + +def test_fetch_artifact_cli_targets_the_named_manifest(tmp_path, capsys): + package = tmp_path / "db" / "data" / "irs_soi" / "ira_contributions" + traditional = _publish(tmp_path, "22in05ira.xlsx", b"traditional IRA table") + argv = [ + "fetch-artifact", + "--url", + str(traditional), + "--source-id", + "irs_soi", + "--package-id", + "soi-ira-traditional-contributions-2022", + "--year", + "2022", + "--out-dir", + str(package), + "--manifest", + TRADITIONAL_MANIFEST, + ] + + assert harness_main(argv) == 0 + payload = json.loads(capsys.readouterr().out) + + assert payload["manifest_path"].endswith(TRADITIONAL_MANIFEST) + assert not (package / "manifest.yaml").exists() + + traditional.write_bytes(b"traditional IRA table, revised rows") + + assert harness_main(argv) == 1 + assert TRADITIONAL_MANIFEST in capsys.readouterr().err + + +# --------------------------------------------------------------------------- +# Identity without a recorded R2 object +# --------------------------------------------------------------------------- + + +def _failing_wrangler(tmp_path, log): + wrangler = tmp_path / "failing-wrangler" + wrangler.write_text(f"#!/bin/sh\nprintf '%s\\n' \"$*\" >> {log}\nexit 1\n") + wrangler.chmod(0o755) + return wrangler + + +def test_a_registered_entry_is_protected_before_it_is_ever_published(tmp_path): + """No storage.r2 yet is not no identity: the entry declares its bytes.""" + package = tmp_path / "db" / "data" / "irs_soi" / "soi-table-5" + source = _publish(tmp_path, "22in05ira.xlsx", b"IRA table 5, first publication") + first = _fetch_local(package, source, upload_r2=False) + recorded = (package / "manifest.yaml").read_bytes() + + assert "storage" not in _entry(package / "manifest.yaml") + + # Same bytes: an ordinary repeated fetch, not a revision. + assert _fetch_local(package, source, upload_r2=False).sha256 == first.sha256 + + source.write_bytes(b"IRA table 5, silently re-published") + with pytest.raises(SourceArtifactRevisionError) as raised: + _fetch_local(package, source, upload_r2=False) + + message = str(raised.value) + assert first.sha256 in message + assert hashlib.sha256(b"IRA table 5, silently re-published").hexdigest() in message + assert "size_bytes=30" in message + assert "--record-revision" in message + assert (package / "manifest.yaml").read_bytes() == recorded + assert (package / "22in05ira.xlsx").read_bytes() == ( + b"IRA table 5, first publication" + ) + + +def test_a_failed_upload_does_not_disable_revision_protection(tmp_path): + """The state #225 hit: bytes registered, upload failed, no storage.r2.""" + package = tmp_path / "db" / "data" / "irs_soi" / "soi-table-5" + log = tmp_path / "wrangler.log" + wrangler = _failing_wrangler(tmp_path, log) + source = _publish(tmp_path, "22in05ira.xlsx", b"IRA table 5, first publication") + + report = _fetch_local( + package, source, upload_r2=True, wrangler_command=str(wrangler) + ) + recorded = (package / "manifest.yaml").read_bytes() + + assert report.errors == ("r2_upload_failed",) + assert "storage" not in _entry(package / "manifest.yaml") + + source.write_bytes(b"IRA table 5, silently re-published") + with pytest.raises(SourceArtifactRevisionError): + _fetch_local(package, source, upload_r2=True, wrangler_command=str(wrangler)) + + assert (package / "manifest.yaml").read_bytes() == recorded + + +def test_record_revision_over_an_unpublished_entry_supersedes_nothing(tmp_path): + """There is no object to keep, so the entry gets no previous_r2 key.""" + package = tmp_path / "db" / "data" / "irs_soi" / "soi-table-5" + source = _publish(tmp_path, "22in05ira.xlsx", b"IRA table 5, first publication") + _fetch_local(package, source, upload_r2=False) + + source.write_bytes(b"IRA table 5, silently re-published") + report = _fetch_local(package, source, upload_r2=False, record_revision=True) + revised = _entry(package / "manifest.yaml") + + assert report.valid + assert revised["sha256"] == ( + hashlib.sha256(b"IRA table 5, silently re-published").hexdigest() + ) + assert "storage" not in revised + + +# --------------------------------------------------------------------------- +# Recorded locator cross-checks +# --------------------------------------------------------------------------- + + +def _recorded_package(tmp_path, content=b"IRA table 5, first publication"): + """A package whose entry records a published, content-addressed object.""" + package = tmp_path / "db" / "data" / "irs_soi" / "soi-table-5" + wrangler = _wrangler_stub(tmp_path, tmp_path / "wrangler.log") + source = _publish(tmp_path, "22in05ira.xlsx", content) + report = _fetch_local( + package, source, upload_r2=True, wrangler_command=str(wrangler) + ) + return package, source, report + + +def _rewrite_recorded_r2(package, mutate, manifest="manifest.yaml"): + manifest_path = package / manifest + payload = yaml.safe_load(manifest_path.read_text()) + mutate(payload["files"][2022]["storage"]) + manifest_path.write_text(yaml.safe_dump(payload, sort_keys=False)) + return manifest_path + + +def _other_sha256(): + return hashlib.sha256(b"some other object entirely").hexdigest() + + +def _contradict_key(storage): + key = storage["r2"]["key"] + storage["r2"]["key"] = key.replace(key.split("/")[-2], _other_sha256()) + + +def _contradict_bucket(storage): + storage["r2"]["bucket"] = "some-other-bucket" + + +def _contradict_provider(storage): + storage["r2"]["provider"] = "s3" + + +def _mangle_uri(storage): + storage["r2"]["uri"] = "r2:/ledger-raw-missing-a-slash" + + +def _drop_the_locator(storage): + storage["r2"] = {"provider": "r2", "bucket": "ledger-raw"} + + +def _flatten_the_key(storage): + storage["r2"]["key"] = "raw/irs_soi/22in05ira.xlsx" + storage["r2"]["uri"] = f"r2://ledger-raw/{storage['r2']['key']}" + + +@pytest.mark.parametrize( + ("mutate", "expected"), + [ + pytest.param(_contradict_key, "contradicts uri", id="key-vs-uri"), + pytest.param(_contradict_bucket, "contradicts uri", id="bucket-vs-uri"), + pytest.param(_contradict_provider, "contradicts uri", id="provider-vs-uri"), + pytest.param(_mangle_uri, "is not provider://bucket/key", id="uri-shape"), + pytest.param(_drop_the_locator, "records no key", id="no-locator"), + pytest.param( + _flatten_the_key, "is not content-addressed", id="not-content-addressed" + ), + ], +) +def test_a_recorded_block_that_names_two_objects_is_refused(tmp_path, mutate, expected): + """A contradictory locator is an error, never a silently preserved block. + + The key-vs-uri case is the one that used to pass: identity was read from + the key alone, so a block whose uri named different bytes was carried + forward verbatim, and the manifest kept publishing a URI for an object it + no longer described. + """ + package, source, _ = _recorded_package(tmp_path) + manifest_path = _rewrite_recorded_r2(package, mutate) + recorded = manifest_path.read_bytes() + + # Identical bytes: the fetch would otherwise preserve the recorded block. + with pytest.raises(RecordedR2LocatorError) as raised: + _fetch_local(package, source, upload_r2=False) + + assert expected in str(raised.value) + assert manifest_path.read_bytes() == recorded + + +def test_a_malformed_storage_block_is_not_treated_as_absent(tmp_path): + package, source, _ = _recorded_package(tmp_path) + manifest_path = _rewrite_recorded_r2( + package, lambda storage: storage.update({"r2": ["r2://ledger-raw/raw/key"]}) + ) + recorded = manifest_path.read_bytes() + + with pytest.raises(MalformedManifestError, match="must be a mapping"): + _fetch_local(package, source, upload_r2=False) + + assert manifest_path.read_bytes() == recorded + + +def test_publish_raw_refuses_a_contradictory_recorded_block(tmp_path): + """Nothing is uploaded under a block that does not name one object.""" + package, _, _ = _recorded_package(tmp_path) + log = tmp_path / "publish.log" + wrangler = _wrangler_stub(tmp_path, log) + manifest_path = _rewrite_recorded_r2(package, _contradict_key) + recorded = manifest_path.read_bytes() + + report = publish_source_artifacts(package, wrangler_command=str(wrangler)) + + assert not report.valid + assert report.entries[0].upload is None + assert report.entries[0].errors[0].startswith("recorded_r2_locator_invalid:") + assert "contradicts uri" in report.entries[0].errors[0] + assert not log.exists() + assert manifest_path.read_bytes() == recorded + + +# --------------------------------------------------------------------------- +# Malformed manifests +# --------------------------------------------------------------------------- + + +@pytest.mark.parametrize( + "document", + [ + pytest.param("- one entry\n- another\n", id="list"), + pytest.param("a bare scalar\n", id="scalar"), + pytest.param("files: [\n", id="unparseable"), + ], +) +def test_a_malformed_manifest_is_refused_before_anything_is_fetched(tmp_path, document): + """Not an absent manifest: refusing it protects what it still records. + + The publisher path does not exist, so reaching the fetch at all would raise + FileNotFoundError instead. Getting MalformedManifestError is what says the + manifest was read and refused first. + """ + package = tmp_path / "db" / "data" / "irs_soi" / "soi-table-5" + package.mkdir(parents=True) + manifest_path = package / "manifest.yaml" + manifest_path.write_text(document) + + with pytest.raises(MalformedManifestError): + _fetch_local(package, tmp_path / "publisher" / "never-read.xlsx") + + assert manifest_path.read_text() == document + assert list(package.iterdir()) == [manifest_path] + + +@pytest.mark.parametrize( + "document", + [ + "files:\n- not a mapping\n", + "files: 3\n", + "source_id: irs_soi\nfiles: text\n", + ], +) +def test_a_non_mapping_files_block_is_refused_before_anything_is_fetched( + tmp_path, document +): + """The same document inventory-artifacts and publish-raw report as + 'files must be a mapping'; a fetch must not overwrite the artifact first.""" + package = tmp_path / "db" / "data" / "irs_soi" / "soi-table-5" + package.mkdir(parents=True) + manifest_path = package / "manifest.yaml" + manifest_path.write_text(document) + + with pytest.raises(MalformedManifestError, match="files must be a mapping"): + _fetch_local(package, tmp_path / "publisher" / "never-read.xlsx") + + assert manifest_path.read_text() == document + assert list(package.iterdir()) == [manifest_path] + + +@pytest.mark.parametrize( + "document", + [ + "", + "\n", + "{}\n", + "# only a comment\n", + "files:\n", + "source_id: irs_soi\nfiles:\n", + ], +) +def test_an_empty_manifest_still_reads_as_absent(tmp_path, document): + """Including a bare ``files:`` line, which parses as an explicit null: the + fetch records into a fresh mapping rather than failing after the write.""" + package = tmp_path / "db" / "data" / "irs_soi" / "soi-table-5" + package.mkdir(parents=True) + (package / "manifest.yaml").write_text(document) + source = _publish(tmp_path, "22in05ira.xlsx", b"IRA table 5") + + report = _fetch_local(package, source, upload_r2=False) + + assert report.valid + assert _entry(package / "manifest.yaml")["filename"] == "22in05ira.xlsx" + + +def test_a_malformed_manifest_is_reported_by_inventory_and_publish(tmp_path): + """Neither sweep may crash on, or silently skip, a document it cannot read.""" + package = tmp_path / "db" / "data" / "irs_soi" / "soi-table-5" + package.mkdir(parents=True) + (package / "manifest.yaml").write_text("- not a mapping\n") + + inventory = inventory_source_artifacts(package) + published = publish_source_artifacts(package) + + assert not inventory.valid + assert inventory.entries == () + assert "must be a YAML mapping" in inventory.errors[0] + assert not published.valid + assert published.entries == () + assert "must be a YAML mapping" in published.errors[0] diff --git a/tests/test_chronicle_consumer_contract.py b/tests/test_chronicle_consumer_contract.py index cda97b66..9cc730c1 100644 --- a/tests/test_chronicle_consumer_contract.py +++ b/tests/test_chronicle_consumer_contract.py @@ -886,6 +886,41 @@ def test_export_consumer_facts_cli_rejects_contract_invalid_facts(tmp_path, caps }, "irs_soi.ty2024.table.us.taxable_interest_amount.ledger_derived", ), + # The guard used to match two hardcoded URI prefixes, so a URI naming + # any derived bucket other than `ledger-derived` did not match. Once the + # buckets are renamed (PolicyEngine/chronicle#143, mechanism 3) that is + # every derived URI, so the guard has to match on shape. + ( + { + "source_name": "irs_soi", + "source_file": "publisher.xlsx", + "raw_r2_bucket": None, + "raw_r2_key": None, + "raw_r2_uri": "r2://chronicle-derived/derived/source/fact.json", + }, + "publisher.raw.fact", + ), + ( + { + "source_name": "irs_soi", + "source_file": "chronicle-derived:taxable_interest.json", + "raw_r2_bucket": "ledger-raw", + "raw_r2_uri": "r2://ledger-raw/raw/source/publisher.xlsx", + }, + "publisher.raw.fact", + ), + # The derived-row marker renames with everything else, so the guard has + # to reject the chronicle spelling the same way it rejects the ledger + # one (PolicyEngine/chronicle#143, mechanism 3). + ( + { + "source_name": "irs_soi", + "source_file": "publisher.xlsx", + "raw_r2_bucket": "ledger-raw", + "raw_r2_uri": "r2://ledger-raw/raw/source/publisher.xlsx", + }, + "irs_soi.ty2024.table.us.taxable_interest_amount.chronicle_derived", + ), ], ) def test_consumer_contract_rejects_downstream_derived_target_facts( @@ -905,6 +940,46 @@ def test_consumer_contract_rejects_downstream_derived_target_facts( assert "derived_fact_provenance" in {error.code for error in report.errors} +def test_derived_record_marker_is_rejected_in_either_spelling(): + """Both rename-window spellings produce the identical boundary error.""" + fact = _soi_agi_fact() + base = "irs_soi.ty2024.table.us.taxable_interest_amount" + + reports = { + suffix: validate_consumer_fact_contract( + [replace(fact, source_record_id=f"{base}.{suffix}")] + ) + for suffix in ("ledger_derived", "chronicle_derived") + } + + ledger_errors = [ + (error.code, error.message) for error in reports["ledger_derived"].errors + ] + chronicle_errors = [ + (error.code, error.message) for error in reports["chronicle_derived"].errors + ] + assert ledger_errors == chronicle_errors + assert "derived_fact_provenance" in {code for code, _ in ledger_errors} + + +@pytest.mark.parametrize( + "source_record_id", + [ + # A publisher-backed row that merely contains the marker as a word, or + # carries it without the separating dot, is not a derived target row. + "irs_soi.ty2024.table.us.chronicle_derived_totals", + "irs_soi.ty2024.table.us.ledger_derived_totals", + "chronicle_derived", + ], +) +def test_derived_record_marker_matches_the_whole_final_segment(source_record_id): + fact = replace(_soi_agi_fact(), source_record_id=source_record_id) + + report = validate_consumer_fact_contract([fact]) + + assert report.valid + + def test_export_consumer_facts_cli_preserves_decimal_values(tmp_path, capsys): input_path = tmp_path / "facts.jsonl" output_path = tmp_path / "consumer_facts.jsonl" diff --git a/tests/test_chronicle_env.py b/tests/test_chronicle_env.py new file mode 100644 index 00000000..202d86f5 --- /dev/null +++ b/tests/test_chronicle_env.py @@ -0,0 +1,364 @@ +"""Tests for the chronicle-first environment read window. + +Chronicle's operational stores migrate by dual-run (PolicyEngine/chronicle#143, +mechanism 3): ``CHRONICLE_*`` names win, ledger-era names keep working behind a +deprecation warning. Every test here is hermetic — the suite-wide +``isolated_rename_window_env`` fixture in ``tests/conftest.py`` strips every +variable in the rename window from the ambient environment first. +""" + +from __future__ import annotations + +import importlib +import os +from pathlib import Path + +import pytest + +from chronicle.artifacts import ( + DEFAULT_R2_DERIVED_BUCKET, + DEFAULT_R2_RAW_BUCKET, + default_r2_derived_bucket, + default_r2_raw_bucket, +) +from chronicle.env import ( + CHRONICLE_ENV_PREFIX, + ChronicleEnvDeprecationWarning, + DEFAULT_CHRONICLE_SCHEMA, + LEGACY_ENV_PREFIXES, + default_chronicle_schema, + env_flag, + env_names, + env_value, +) +from chronicle.harness import main as harness_main +from chronicle.source_package import ( + SOURCE_ARTIFACT_CACHE_ENV, + SOURCE_ARTIFACT_FETCH_ENV, +) + + +def _fake_wrangler(tmp_path, log): + wrangler = tmp_path / "wrangler" + wrangler.write_text(f"#!/bin/sh\nprintf '%s\\n' \"$*\" >> {log}\necho ok\n") + wrangler.chmod(0o755) + return wrangler + + +# --------------------------------------------------------------------------- +# Lookup order +# --------------------------------------------------------------------------- + + +def test_every_test_runs_with_the_rename_window_cleared(): + """Isolation is suite-wide (tests/conftest.py), not module-scoped. + + Modules well outside this one assert the defaults these variables override + — the raw and derived bucket names, the Supabase schema — so an operator's + shell must not reach any test. + """ + leaked = sorted( + name + for name in os.environ + if name.startswith((CHRONICLE_ENV_PREFIX, *LEGACY_ENV_PREFIXES)) + ) + + assert leaked == [] + + +def test_env_names_puts_chronicle_first_then_ledger_era_names(): + assert env_names("CHRONICLE_SOURCE_ARTIFACT_FETCH") == ( + "CHRONICLE_SOURCE_ARTIFACT_FETCH", + "POLICYENGINE_LEDGER_SOURCE_ARTIFACT_FETCH", + "LEDGER_SOURCE_ARTIFACT_FETCH", + ) + + +def test_env_names_expands_a_ledger_era_name_to_the_same_ladder(): + assert env_names("LEDGER_PE_US_DATA_ROOT") == env_names("CHRONICLE_PE_US_DATA_ROOT") + + +def test_env_names_leaves_variables_outside_the_rename_window_alone(): + assert env_names("POLICYENGINE_SUPABASE_URL") == ("POLICYENGINE_SUPABASE_URL",) + assert env_names("POLICYENGINE_TARGETS_SCHEMA") == ("POLICYENGINE_TARGETS_SCHEMA",) + + +def test_bare_prefix_is_not_treated_as_a_renamed_variable(): + assert env_names("LEDGER_") == ("LEDGER_",) + + +# --------------------------------------------------------------------------- +# Precedence and the deprecation warning +# --------------------------------------------------------------------------- + + +def test_chronicle_name_wins_over_both_ledger_era_names(monkeypatch, recwarn): + monkeypatch.setenv("CHRONICLE_PE_US_DATA_ROOT", "/chronicle") + monkeypatch.setenv("LEDGER_PE_US_DATA_ROOT", "/ledger") + monkeypatch.setenv("POLICYENGINE_LEDGER_PE_US_DATA_ROOT", "/policyengine-ledger") + + assert env_value("CHRONICLE_PE_US_DATA_ROOT") == "/chronicle" + assert not [ + warning + for warning in recwarn.list + if issubclass(warning.category, ChronicleEnvDeprecationWarning) + ] + + +def test_ledger_name_alone_still_works_and_warns(monkeypatch): + monkeypatch.setenv("LEDGER_PE_US_DATA_ROOT", "/ledger") + + with pytest.warns(ChronicleEnvDeprecationWarning) as warnings_raised: + assert env_value("CHRONICLE_PE_US_DATA_ROOT") == "/ledger" + + message = str(warnings_raised[0].message) + assert "LEDGER_PE_US_DATA_ROOT" in message + assert "CHRONICLE_PE_US_DATA_ROOT" in message + + +def test_policyengine_ledger_name_alone_still_works_and_warns(monkeypatch): + monkeypatch.setenv("POLICYENGINE_LEDGER_SCHEMA", "ledger") + + with pytest.warns(ChronicleEnvDeprecationWarning) as warnings_raised: + assert env_value("CHRONICLE_SCHEMA") == "ledger" + + message = str(warnings_raised[0].message) + assert "POLICYENGINE_LEDGER_SCHEMA" in message + assert "CHRONICLE_SCHEMA" in message + + +def test_deprecation_warning_is_raised_once_per_process(monkeypatch, recwarn): + monkeypatch.setenv("LEDGER_PE_UK_DATA_ROOT", "/ledger") + + for _ in range(3): + assert env_value("CHRONICLE_PE_UK_DATA_ROOT") == "/ledger" + + deprecations = [ + warning + for warning in recwarn.list + if issubclass(warning.category, ChronicleEnvDeprecationWarning) + ] + assert len(deprecations) == 1 + + +def test_deprecation_warning_is_attributed_to_the_calling_module(monkeypatch): + monkeypatch.setenv("LEDGER_SOURCE_ARTIFACT_FETCH", "1") + + with pytest.warns(ChronicleEnvDeprecationWarning) as warnings_raised: + assert env_flag(SOURCE_ARTIFACT_FETCH_ENV) + + # env_flag and env_value must report at the same depth, or operators get a + # notice pointing at Chronicle's own source instead of their call site. + assert Path(warnings_raised[0].filename).name == "test_chronicle_env.py" + + +def test_unset_variables_fall_back_to_the_default(): + assert env_value("CHRONICLE_PE_US_DATA_ROOT") is None + assert env_value("CHRONICLE_PE_US_DATA_ROOT", default="/fallback") == "/fallback" + + +def test_empty_values_count_as_unset(monkeypatch): + monkeypatch.setenv("CHRONICLE_PE_US_DATA_ROOT", "") + monkeypatch.setenv("LEDGER_PE_US_DATA_ROOT", "/ledger") + + with pytest.warns(ChronicleEnvDeprecationWarning): + assert env_value("CHRONICLE_PE_US_DATA_ROOT") == "/ledger" + + +# --------------------------------------------------------------------------- +# Flags +# --------------------------------------------------------------------------- + + +@pytest.mark.parametrize("value", ["1", "true", "TRUE", "yes", "on", " on "]) +def test_env_flag_accepts_truthy_spellings(monkeypatch, value): + monkeypatch.setenv("CHRONICLE_SOURCE_ARTIFACT_FETCH", value) + assert env_flag(SOURCE_ARTIFACT_FETCH_ENV) + + +@pytest.mark.parametrize("value", ["0", "false", "no", "off", "maybe"]) +def test_env_flag_rejects_other_values(monkeypatch, value): + monkeypatch.setenv("CHRONICLE_SOURCE_ARTIFACT_FETCH", value) + assert not env_flag(SOURCE_ARTIFACT_FETCH_ENV) + + +def test_env_flag_lets_the_chronicle_name_turn_a_legacy_flag_off(monkeypatch): + monkeypatch.setenv("CHRONICLE_SOURCE_ARTIFACT_FETCH", "0") + monkeypatch.setenv("LEDGER_SOURCE_ARTIFACT_FETCH", "1") + + # An operator who has migrated must be able to turn the flag off without + # first hunting down the stale ledger-era variable. + assert not env_flag(SOURCE_ARTIFACT_FETCH_ENV) + + +# --------------------------------------------------------------------------- +# Real call sites +# --------------------------------------------------------------------------- + + +def test_source_artifact_env_constants_are_chronicle_named(): + assert SOURCE_ARTIFACT_CACHE_ENV == "CHRONICLE_SOURCE_ARTIFACT_CACHE_DIR" + assert SOURCE_ARTIFACT_FETCH_ENV == "CHRONICLE_SOURCE_ARTIFACT_FETCH" + + +@pytest.mark.parametrize( + "name", + ["CHRONICLE_SOURCE_ARTIFACT_CACHE_DIR", "LEDGER_SOURCE_ARTIFACT_CACHE_DIR"], +) +def test_source_artifact_cache_dir_honors_both_names(monkeypatch, tmp_path, name): + from chronicle.source_package import _source_artifact_cache_path + + monkeypatch.setenv(name, str(tmp_path)) + + cache_path = _source_artifact_cache_path( + {"filename": "table.xlsx", "sha256": "abc123"} + ) + + assert cache_path == tmp_path / "abc123" / "table.xlsx" + + +@pytest.mark.parametrize( + "name", ["CHRONICLE_PE_US_DATA_ROOT", "LEDGER_PE_US_DATA_ROOT"] +) +def test_pe_source_root_cli_default_honors_both_names(monkeypatch, name): + from db.cli import _pe_source_root_env_default + + monkeypatch.setenv(name, "/pe-us") + + assert _pe_source_root_env_default("us") == "/pe-us" + + +def test_pe_source_inventory_env_constants_are_chronicle_named(): + from db.pe_source_inventory import PE_UK_DATA_ROOT_ENV, PE_US_DATA_ROOT_ENV + + assert PE_US_DATA_ROOT_ENV == "CHRONICLE_PE_US_DATA_ROOT" + assert PE_UK_DATA_ROOT_ENV == "CHRONICLE_PE_UK_DATA_ROOT" + + +def test_db_cli_parser_builds_with_the_env_backed_defaults(monkeypatch, capsys): + """The db CLI builds its parser before dispatching any subcommand. + + Its --pe-us-root/--pe-uk-root defaults call into the env helper, so an + import error there breaks `chronicle init`, `load` and `stats` alike while + the rest of the test suite stays green. + """ + import db.cli + + monkeypatch.setenv("CHRONICLE_PE_US_DATA_ROOT", "/pe-us") + monkeypatch.setattr("sys.argv", ["chronicle", "--help"]) + + with pytest.raises(SystemExit) as exit_info: + db.cli.main() + + assert exit_info.value.code == 0 + assert "Manage Chronicle target input data" in capsys.readouterr().out + + +@pytest.mark.parametrize( + "name", + ["CHRONICLE_SCHEMA", "POLICYENGINE_LEDGER_SCHEMA", "LEDGER_SCHEMA"], +) +def test_supabase_schema_honors_every_name_in_the_window(monkeypatch, name): + """Set after import and still honored: the schema is read at call time.""" + import db.supabase_client + + monkeypatch.setenv(name, "chronicle_probe") + + assert db.supabase_client.chronicle_schema() == "chronicle_probe" + assert default_chronicle_schema() == "chronicle_probe" + + +def test_supabase_schema_default_is_unchanged(): + import db.supabase_client + + # The hosted schema name itself is out of this slice; only the variable + # that overrides it moved. + assert DEFAULT_CHRONICLE_SCHEMA == "ledger" + assert default_chronicle_schema() == "ledger" + assert db.supabase_client.chronicle_schema() == "ledger" + assert db.supabase_client.targets_schema() == "targets" + + +def test_supabase_schema_is_not_bound_at_import(monkeypatch): + """No module-level constant may freeze the schema at import time. + + A reload under a set variable is the pre-fix behavior this guards against: + it proves nothing about a module that resolved the value once, at + collection, and answers with the stale constant forever after. + """ + import db.supabase_client + + assert not [ + name + for name, value in vars(db.supabase_client).items() + if name.isupper() and value == "ledger" + ] + + monkeypatch.setenv("CHRONICLE_SCHEMA", "chronicle_probe") + unreloaded = importlib.import_module("db.supabase_client") + + assert unreloaded.chronicle_schema() == "chronicle_probe" + + +# --------------------------------------------------------------------------- +# R2 bucket configuration +# --------------------------------------------------------------------------- + + +def test_r2_bucket_defaults_are_still_the_ledger_era_names(): + assert DEFAULT_R2_RAW_BUCKET == "ledger-raw" + assert DEFAULT_R2_DERIVED_BUCKET == "ledger-derived" + assert default_r2_raw_bucket() == "ledger-raw" + assert default_r2_derived_bucket() == "ledger-derived" + + +def test_r2_buckets_follow_the_chronicle_env_vars(monkeypatch): + monkeypatch.setenv("CHRONICLE_R2_RAW_BUCKET", "chronicle-raw") + monkeypatch.setenv("CHRONICLE_R2_DERIVED_BUCKET", "chronicle-derived") + + assert default_r2_raw_bucket() == "chronicle-raw" + assert default_r2_derived_bucket() == "chronicle-derived" + + +def test_r2_buckets_honor_ledger_era_names_with_a_warning(monkeypatch): + monkeypatch.setenv("LEDGER_R2_RAW_BUCKET", "legacy-raw") + + with pytest.warns(ChronicleEnvDeprecationWarning): + assert default_r2_raw_bucket() == "legacy-raw" + + +def test_bootstrap_r2_cli_creates_the_configured_buckets(monkeypatch, tmp_path): + log = tmp_path / "wrangler.log" + wrangler = _fake_wrangler(tmp_path, log) + monkeypatch.setenv("CHRONICLE_R2_RAW_BUCKET", "chronicle-raw") + monkeypatch.setenv("CHRONICLE_R2_DERIVED_BUCKET", "chronicle-derived") + + exit_code = harness_main(["bootstrap-r2", "--wrangler-command", str(wrangler)]) + + commands = log.read_text() + assert exit_code == 0 + assert "r2 bucket create chronicle-raw" in commands + assert "r2 bucket create chronicle-derived" in commands + + +def test_bootstrap_r2_cli_flags_still_override_the_environment(monkeypatch, tmp_path): + log = tmp_path / "wrangler.log" + wrangler = _fake_wrangler(tmp_path, log) + monkeypatch.setenv("CHRONICLE_R2_RAW_BUCKET", "chronicle-raw") + + harness_main( + [ + "bootstrap-r2", + "--raw-bucket", + "explicit-raw", + "--derived-bucket", + "explicit-derived", + "--wrangler-command", + str(wrangler), + ] + ) + + commands = log.read_text() + assert "r2 bucket create explicit-raw" in commands + assert "r2 bucket create explicit-derived" in commands + assert "chronicle-raw" not in commands diff --git a/tests/test_chronicle_manifest_kind.py b/tests/test_chronicle_manifest_kind.py new file mode 100644 index 00000000..e5409b0b --- /dev/null +++ b/tests/test_chronicle_manifest_kind.py @@ -0,0 +1,314 @@ +"""The explicit-kind rule and the repository guard for microdata bytes. + +Every manifest created or modified after the microdata-identity ADR declares +``kind``. Manifests that predate the rule are frozen, byte for byte, in +``chronicle/grandfathered_manifests.py``; any other kindless manifest is an +error at every entry point -- ``fetch-artifact``, ``publish-raw``, +``inventory-artifacts``, ``validate-package`` and the source-package byte +reader -- and never a publisher table by default. A ``kind: microdata_release`` +package directory holds only manifests: public release bytes are archived in +R2 from a staging directory outside the tree, never committed. +""" + +from __future__ import annotations + +import subprocess +import sys +import uuid +from pathlib import Path + +import pytest +import yaml + +from chronicle.artifacts import ( + fetch_source_artifact, + inventory_source_artifacts, + publish_source_artifacts, +) +from chronicle.grandfathered_manifests import ( + GRANDFATHERED_KINDLESS_MANIFESTS, + grandfathered_manifest_key, + is_grandfathered_manifest, + manifest_digest, +) +from chronicle.registration import ( + ManifestAccessError, + ManifestKindError, + manifest_kind, + safe_manifest_kind, +) +from chronicle.source_package import SourceArtifactSpec, validate_source_package + + +REPO_ROOT = Path(__file__).resolve().parents[1] +FREEZE_SIZE = 161 + + +def _tracked_manifests() -> list[Path]: + listed = subprocess.run( + ["git", "-C", str(REPO_ROOT), "ls-files", "db/data"], + capture_output=True, + text=True, + check=True, + ).stdout.split() + return [ + REPO_ROOT / path + for path in listed + if Path(path).name.startswith("manifest") and path.endswith((".yaml", ".yml")) + ] + + +def _kindless(path: Path) -> bool: + payload = yaml.safe_load(path.read_text()) or {} + return isinstance(payload, dict) and "kind" not in payload + + +def test_every_kindless_manifest_in_the_tree_is_frozen_unmodified(): + tracked = _tracked_manifests() + assert tracked + kindless = { + path.relative_to(REPO_ROOT).as_posix(): path + for path in tracked + if _kindless(path) + } + + missing = sorted(set(kindless) - set(GRANDFATHERED_KINDLESS_MANIFESTS)) + assert missing == [], ( + "kindless manifests outside the frozen list; declare `kind:` in them: " + f"{missing}" + ) + modified = sorted( + key + for key, path in kindless.items() + if manifest_digest(path) != GRANDFATHERED_KINDLESS_MANIFESTS[key] + ) + assert modified == [], ( + "frozen manifests modified without declaring `kind:`; declare it and " + f"drop them from the frozen list: {modified}" + ) + + +def test_the_frozen_list_only_shrinks(): + # Entries leave the list once a manifest declares its kind; none is added. + assert len(GRANDFATHERED_KINDLESS_MANIFESTS) <= FREEZE_SIZE + stale = sorted( + key + for key in GRANDFATHERED_KINDLESS_MANIFESTS + if not (REPO_ROOT / key).exists() or not _kindless(REPO_ROOT / key) + ) + assert stale == [], f"drop from the frozen list, the kind is declared: {stale}" + + +def _frozen_copy(tmp_path: Path) -> tuple[Path, str]: + key = "db/data/irs_soi/table_1_2/manifest.yaml" + assert key in GRANDFATHERED_KINDLESS_MANIFESTS + copy = tmp_path / key + copy.parent.mkdir(parents=True) + copy.write_bytes((REPO_ROOT / key).read_bytes()) + return copy, key + + +def test_a_frozen_manifest_reads_as_a_publisher_table_until_it_is_modified(tmp_path): + copy, key = _frozen_copy(tmp_path) + payload = yaml.safe_load(copy.read_text()) + + assert grandfathered_manifest_key(copy) == key + assert is_grandfathered_manifest(copy) + assert manifest_kind(payload, manifest_path=copy) == "publisher_table" + assert safe_manifest_kind(payload, manifest_path=copy) == ("publisher_table", None) + + copy.write_bytes(copy.read_bytes() + b"# touched\n") + assert not is_grandfathered_manifest(copy) + with pytest.raises(ManifestKindError, match="declares no kind"): + manifest_kind(payload, manifest_path=copy) + assert safe_manifest_kind(payload, manifest_path=copy) == ( + "publisher_table", + "manifest_kind_missing", + ) + + +def test_a_fetch_into_a_frozen_manifest_declares_its_kind(tmp_path): + copy, _key = _frozen_copy(tmp_path) + source = tmp_path / "24in12ms.xls" + source.write_bytes(b"the next vintage") + + report = fetch_source_artifact( + str(source), + source_id="irs_soi", + package_id="soi-table-1-2", + year=2099, + output_dir=copy.parent, + ) + payload = yaml.safe_load(copy.read_text()) + + assert report.valid + assert payload["kind"] == "publisher_table" + assert list(payload)[:3] == ["source_id", "package_id", "kind"] + # The manifest left the freeze by declaring itself. + assert not is_grandfathered_manifest(copy) + assert manifest_kind(payload, manifest_path=copy) == "publisher_table" + + +def _kindless_package(tmp_path: Path) -> Path: + package = tmp_path / "db" / "data" / "dwp" / "new_tables" + package.mkdir(parents=True) + (package / "table.ods").write_bytes(b"a table") + (package / "manifest.yaml").write_text( + yaml.safe_dump( + { + "source_id": "dwp", + "package_id": "dwp-new-tables", + "files": { + 2023: { + "filename": "table.ods", + "source_url": "https://publisher.example/table.ods", + } + }, + }, + sort_keys=False, + ) + ) + return package + + +def test_a_new_kindless_manifest_is_refused_by_fetch_before_reading( + tmp_path, monkeypatch +): + package = _kindless_package(tmp_path) + original = (package / "manifest.yaml").read_bytes() + reads: list[str] = [] + monkeypatch.setattr( + "chronicle.artifacts._read_artifact", + lambda url: reads.append(url) or (b"x", "other.ods"), + ) + + for kind in (None, "publisher_table", "microdata_release"): + with pytest.raises(ManifestAccessError, match="declares no kind"): + fetch_source_artifact( + "https://publisher.example/other.ods", + source_id="dwp", + package_id="dwp-new-tables", + year=2024, + output_dir=package, + kind=kind, + licence="OGL-UK-3.0", + ) + + assert reads == [] + assert (package / "manifest.yaml").read_bytes() == original + + +def test_a_new_kindless_manifest_is_reported_and_skipped_by_the_sweeps( + tmp_path, monkeypatch +): + package = _kindless_package(tmp_path) + original = (package / "manifest.yaml").read_bytes() + monkeypatch.setattr( + "chronicle.artifacts._upload_r2_object", + lambda *args, **kwargs: (_ for _ in ()).throw(AssertionError("no upload")), + ) + + inventory = inventory_source_artifacts(tmp_path / "db" / "data") + published = publish_source_artifacts(tmp_path / "db" / "data") + + assert not inventory.valid + assert any(error.startswith("manifest_kind_missing") for error in inventory.errors) + assert not published.valid + assert published.entries == () + assert any(error.startswith("manifest_kind_missing") for error in published.errors) + assert (package / "manifest.yaml").read_bytes() == original + + +def test_a_new_kindless_manifest_is_refused_by_the_byte_reader(tmp_path, monkeypatch): + package_name = f"chronicle_kind_{uuid.uuid4().hex}" + resource_dir = tmp_path / "pkgroot" / package_name / "data" / "dwp" / "tables" + resource_dir.mkdir(parents=True) + (resource_dir / "table.csv").write_bytes(b"a,b\n1,2\n") + (resource_dir / "manifest.yaml").write_text( + yaml.safe_dump( + { + "source_id": "dwp", + "package_id": "dwp-tables", + "files": {2023: {"filename": "table.csv", "source_url": "x"}}, + } + ) + ) + monkeypatch.syspath_prepend(str(tmp_path / "pkgroot")) + monkeypatch.delitem(sys.modules, package_name, raising=False) + spec = SourceArtifactSpec( + source_name="dwp", + source_table="Tables", + resource_package=package_name, + resource_directory="data/dwp/tables", + manifest="manifest.yaml", + vintage="2023", + extracted_at="2026-09-02", + extraction_method="none", + parser="delimited_text_full_rows", + artifact_year=2023, + ) + + with pytest.raises(ManifestKindError): + spec.assert_parseable_manifest() + with pytest.raises(ManifestKindError): + spec.build_source_rows(2023) + + package_dir = tmp_path / "package" + package_dir.mkdir() + (package_dir / "source_package.yaml").write_text( + yaml.safe_dump( + { + "schema_version": "ledger.source_package.v1", + "package_id": "dwp-tables-parse-attempt", + "label": "A kindless manifest", + "artifact": { + "source_name": "dwp", + "source_table": "Tables", + "resource_package": package_name, + "resource_directory": "data/dwp/tables", + "manifest": "manifest.yaml", + "vintage": "2023", + "extracted_at": "2026-09-02", + "extraction_method": "none", + "parser": "delimited_text_full_rows", + "artifact_year": 2023, + }, + "record_sets": [], + }, + sort_keys=False, + ) + ) + report = validate_source_package(package_dir, year=2023) + assert not report.valid + assert [issue.code for issue in report.errors] == ["manifest_kind_missing"] + + +def test_no_tracked_microdata_bytes(): + """The repository guard: a release package holds manifests and nothing else.""" + release_dirs = { + path.parent + for path in _tracked_manifests() + if (yaml.safe_load(path.read_text()) or {}).get("kind") == "microdata_release" + } + assert release_dirs + + tracked = subprocess.run( + ["git", "-C", str(REPO_ROOT), "ls-files", "db/data"], + capture_output=True, + text=True, + check=True, + ).stdout.split() + for directory in release_dirs: + relative = directory.relative_to(REPO_ROOT).as_posix() + files = sorted( + Path(path).name + for path in tracked + if Path(path).parent.as_posix() == relative + ) + assert files == ["manifest.yaml"], f"{relative} tracks microdata bytes: {files}" + assert sorted(item.name for item in directory.iterdir()) == ["manifest.yaml"] + + +def test_the_committed_tree_passes_the_kind_rule(): + report = inventory_source_artifacts(REPO_ROOT / "db" / "data") + assert not [error for error in report.errors if "manifest_kind" in error] diff --git a/tests/test_chronicle_microdata_catalogue.py b/tests/test_chronicle_microdata_catalogue.py new file mode 100644 index 00000000..dc8d17c6 --- /dev/null +++ b/tests/test_chronicle_microdata_catalogue.py @@ -0,0 +1,479 @@ +"""Hermetic tests for ``scripts/register_microdata_releases.py``. + +The catalogue is resolved against ``tests/fixtures/microcosm``: synthetic +consumer manifests in Microcosm's two shapes (staged ``source_stages.json`` +files and the flat ACS runtime manifest) that carry the reviewed pins +verbatim beside decoy artifacts the selectors must ignore. Nothing here reads +a checkout outside the repository, and nothing skips: a resolution failure is +a failure. + +``emit`` must reproduce the two committed hash-only manifests byte for byte +from the fixture, and ``plan`` must print exactly the golden commands in +``tests/fixtures/microcosm/golden_plan.json`` -- every reviewed identity as an +argument, every unknown as a ``TODO`` that the real CLI refuses to run. +""" + +from __future__ import annotations + +import argparse +import contextlib +import json +import shlex +import subprocess +import sys +from pathlib import Path + +import pytest +import yaml + +from chronicle.harness import main as harness_main + + +REPO_ROOT = Path(__file__).resolve().parents[1] +FIXTURE_ROOT = REPO_ROOT / "tests" / "fixtures" / "microcosm" +GOLDEN_PLAN = FIXTURE_ROOT / "golden_plan.json" +FRS_MANIFEST = REPO_ROOT / "db" / "data" / "dwp" / "frs_2023_24" / "manifest.yaml" +SPI_MANIFEST = ( + REPO_ROOT / "db" / "data" / "hmrc" / "spi_public_use_tape_2022_23" / "manifest.yaml" +) +UK_STAGES = "packages/microcosm-build/src/microcosm/build/uk/source_stages.json" +UK_HMRC_STAGES = ( + "packages/microcosm-build/src/microcosm/build/uk/hmrc_income_source_stages.json" +) +#: The commits the committed registrations were transcribed from: the last +#: commit that changed each consumer manifest. +PIN_COMMITS = { + UK_STAGES: "2fb2e2f8a99c37725bd6e7a15ff4c2595c912b77", + UK_HMRC_STAGES: "de7451bd19ca46d2967e73cdf393908d29e72542", +} +PIN_COMMIT_ARGS = [ + arg + for path, commit in PIN_COMMITS.items() + for arg in ("--microcosm-commit", f"{path}={commit}") +] + +sys.path.insert(0, str(REPO_ROOT / "scripts")) +import register_microdata_releases as script # noqa: E402 + + +def _run(argv: list[str], capsys) -> tuple[int, str, str]: + exit_code = script.main(argv) + captured = capsys.readouterr() + return exit_code, captured.out, captured.err + + +def _harness_parser(monkeypatch) -> argparse.ArgumentParser: + """Capture the real ``chronicle`` harness parser.""" + captured: dict[str, argparse.ArgumentParser] = {} + + def capture(self, args=None, namespace=None): + captured["parser"] = self + raise SystemExit(0) + + monkeypatch.setattr(argparse.ArgumentParser, "parse_args", capture) + with contextlib.suppress(SystemExit): + harness_main(["--help"]) + monkeypatch.undo() + return captured["parser"] + + +# -------------------------------------------------------------------------- +# The fixture checkout +# -------------------------------------------------------------------------- + + +def test_fixture_checkout_holds_every_catalogued_manifest(): + catalogued = sorted({release.manifest for release in script.CATALOGUE}) + assert catalogued + for relative in catalogued: + path = FIXTURE_ROOT / relative + assert path.exists(), relative + payload = json.loads(path.read_text()) + assert payload["snapshot_of"]["repository"] == "PolicyEngine/microcosm" + + +def test_selection_ignores_decoys_and_resolves_every_catalogue_entry(): + resolved = script.resolve(FIXTURE_ROOT, script.CATALOGUE) + by_id = {item.release.release_id: item for item in resolved} + frs = yaml.safe_load(FRS_MANIFEST.read_text())["files"][2023] + committed = {entry["filename"]: entry for entry in frs} + + assert len(by_id) == len(script.CATALOGUE) + for tab in script.FRS_TABS: + item = by_id[f"dwp-frs-2023-24:{tab}"] + expected = committed[f"{tab}.tab"] + assert item.stage["stage"] == "frs_spine" + assert item.artifact["kind"] == "licensed_microdata" + assert (item.filename, item.sha256, item.size_bytes, item.vintage) == ( + f"{tab}.tab", + expected["sha256"], + expected["size_bytes"], + expected["vintage"], + ) + spi = by_id["hmrc-spi-public-use-tape-2022-23:put2223uk"] + spi_committed = yaml.safe_load(SPI_MANIFEST.read_text())["files"][2022][0] + assert (spi.filename, spi.sha256, spi.size_bytes, spi.vintage) == ( + "put2223uk.tab", + spi_committed["sha256"], + spi_committed["size_bytes"], + spi_committed["vintage"], + ) + assert spi.url is None + silc = by_id["statbel-be-silc-2023"] + assert (silc.sha256, silc.size_bytes, silc.filename) == (None, None, None) + assert by_id["census-cps-basic-monthly-2024"].url is None + assert by_id["census-sipp-2023"].url is None + assert by_id["federal-reserve-scf-2022-full"].sha256 is None + assert by_id["census-acs-pums-2024-household"].filename == "csv_hus.zip" + + +def test_selection_accepts_agreeing_duplicates_and_refuses_conflicts(): + payload = json.loads((FIXTURE_ROOT / UK_STAGES).read_text()) + selector = script.ArtifactSelector( + kind="licensed_microdata", match={"table": "adult"} + ) + stage, artifact = script.select_artifact(payload, selector, release_id="x") + # adult.tab is listed by two stages with identical bytes; the first wins. + assert stage["stage"] == "frs_spine" + + conflicting = json.loads(json.dumps(payload)) + conflicting["stages"][1]["artifacts"][0]["sha256"] = "f" * 64 + with pytest.raises(script.CatalogueError, match="conflicting values"): + script.select_artifact(conflicting, selector, release_id="x") + + with pytest.raises(script.CatalogueError, match="no Microcosm artifact matches"): + script.select_artifact( + payload, + script.ArtifactSelector(kind="licensed_microdata", match={"table": "nope"}), + release_id="x", + ) + + +def test_resolve_refuses_a_missing_consumer_manifest(tmp_path): + with pytest.raises(script.CatalogueError, match="manifest not found"): + script.resolve(tmp_path / "no-such-checkout", script.CATALOGUE[:1]) + + +# -------------------------------------------------------------------------- +# emit +# -------------------------------------------------------------------------- + + +def test_emit_from_the_fixture_reproduces_the_committed_manifests_byte_for_byte( + tmp_path, capsys +): + root = tmp_path / "data" + + exit_code, out, err = _run( + [ + "--microcosm-root", + str(FIXTURE_ROOT), + "--root", + str(root), + "--json", + "emit", + *PIN_COMMIT_ARGS, + ], + capsys, + ) + payload = json.loads(out) + + assert exit_code == 0, err + assert len(payload["registrations"]) == 15 + assert [blocker["release"] for blocker in payload["blockers"]] == [ + "statbel-be-silc-2023" + ] + assert "No hash is invented" in payload["blockers"][0]["reason"] + assert all(r["hash_source"] == "consumer_pin" for r in payload["registrations"]) + assert all(r["r2_location"] is None for r in payload["registrations"]) + written = sorted( + path.relative_to(root).as_posix() for path in root.rglob("*") if path.is_file() + ) + assert written == [ + "dwp/frs_2023_24/manifest.yaml", + "hmrc/spi_public_use_tape_2022_23/manifest.yaml", + ] + assert (root / "dwp/frs_2023_24/manifest.yaml").read_bytes() == ( + FRS_MANIFEST.read_bytes() + ) + assert (root / "hmrc/spi_public_use_tape_2022_23/manifest.yaml").read_bytes() == ( + SPI_MANIFEST.read_bytes() + ) + + +def test_emit_is_idempotent_over_the_committed_manifests(tmp_path, capsys): + root = tmp_path / "data" + for manifest in (FRS_MANIFEST, SPI_MANIFEST): + target = root / manifest.relative_to(REPO_ROOT / "db" / "data") + target.parent.mkdir(parents=True) + target.write_bytes(manifest.read_bytes()) + + exit_code, out, _err = _run( + [ + "--microcosm-root", + str(FIXTURE_ROOT), + "--root", + str(root), + "--json", + "emit", + *PIN_COMMIT_ARGS, + ], + capsys, + ) + + assert exit_code == 0 + assert all(r["replaced"] for r in json.loads(out)["registrations"]) + assert ( + root / "dwp/frs_2023_24/manifest.yaml" + ).read_bytes() == FRS_MANIFEST.read_bytes() + + +def test_emit_refuses_a_pin_that_drifted_from_the_committed_one(tmp_path, capsys): + root = tmp_path / "data" + target = root / "dwp" / "frs_2023_24" / "manifest.yaml" + target.parent.mkdir(parents=True) + target.write_bytes(FRS_MANIFEST.read_bytes()) + checkout = tmp_path / "consumer" + for path in FIXTURE_ROOT.rglob("*.json"): + copy = checkout / path.relative_to(FIXTURE_ROOT) + copy.parent.mkdir(parents=True, exist_ok=True) + copy.write_bytes(path.read_bytes()) + stages = json.loads((checkout / UK_STAGES).read_text()) + for stage in stages["stages"]: + for artifact in stage["artifacts"]: + if artifact.get("table") == "adult": + artifact["sha256"] = "f" * 64 + (checkout / UK_STAGES).write_text(json.dumps(stages)) + + exit_code, _out, err = _run( + [ + "--microcosm-root", + str(checkout), + "--root", + str(root), + "emit", + *PIN_COMMIT_ARGS, + ], + capsys, + ) + + assert exit_code == 1 + assert "pass --allow-reissue" in err + assert target.read_bytes() == FRS_MANIFEST.read_bytes() + + +def _fixture_copy(destination: Path) -> Path: + for path in FIXTURE_ROOT.rglob("*.json"): + copy = destination / path.relative_to(FIXTURE_ROOT) + copy.parent.mkdir(parents=True, exist_ok=True) + copy.write_bytes(path.read_bytes()) + return destination + + +def _git(repo: Path, *args: str) -> str: + return subprocess.run( + ["git", "-C", str(repo), *args], capture_output=True, text=True, check=True + ).stdout.strip() + + +def test_emit_needs_a_commit_it_can_read_or_be_told(tmp_path, capsys): + # The fixture inside this repository is committed, so a run against it + # would read a Chronicle commit as if it were the consumer's. Outside any + # repository there is no commit to read; inside one whose manifests are + # untracked there is none either. Both refuse before writing. + outside = _fixture_copy(tmp_path / "outside-git") + assert not (outside / ".git").exists() + untracked = _fixture_copy(tmp_path / "untracked") + _git(untracked, "init", "-q") + _git(untracked, "config", "user.email", "t@example.com") + _git(untracked, "config", "user.name", "t") + (untracked / "README").write_text("nothing pinned here") + _git(untracked, "add", "README") + _git(untracked, "commit", "-q", "-m", "unrelated") + + for checkout in (outside, untracked): + exit_code, _out, err = _run( + [ + "--microcosm-root", + str(checkout), + "--root", + str(tmp_path / "data"), + "emit", + ], + capsys, + ) + assert exit_code == 1 + assert "--microcosm-commit" in err + assert not (tmp_path / "data").exists() + + exit_code, _out, err = _run( + [ + "--microcosm-root", + str(outside), + "--root", + str(tmp_path / "data"), + "emit", + "--microcosm-commit", + "not-a-commit", + ], + capsys, + ) + assert exit_code == 2 + assert "40-hex commit" in err + assert not (tmp_path / "data").exists() + + +def test_emit_from_a_checkout_outside_git_registers_with_the_given_commit( + tmp_path, capsys +): + outside = _fixture_copy(tmp_path / "outside-git") + + exit_code, out, err = _run( + [ + "--microcosm-root", + str(outside), + "--root", + str(tmp_path / "data"), + "--json", + "emit", + *PIN_COMMIT_ARGS, + ], + capsys, + ) + + assert exit_code == 0, err + assert len(json.loads(out)["registrations"]) == 15 + assert (tmp_path / "data" / "dwp/frs_2023_24/manifest.yaml").read_bytes() == ( + FRS_MANIFEST.read_bytes() + ) + + +def test_pin_commit_reads_the_last_commit_that_changed_the_file(tmp_path): + repo = tmp_path / "consumer" + (repo / "build").mkdir(parents=True) + git = ["git", "-C", str(repo)] + subprocess.run([*git, "init", "-q"], check=True) + subprocess.run([*git, "config", "user.email", "t@example.com"], check=True) + subprocess.run([*git, "config", "user.name", "t"], check=True) + (repo / "build" / "stages.json").write_text("{}") + subprocess.run([*git, "add", "."], check=True) + subprocess.run([*git, "commit", "-q", "-m", "pin"], check=True) + pinned = subprocess.run( + [*git, "rev-parse", "HEAD"], capture_output=True, text=True, check=True + ).stdout.strip() + (repo / "other.txt").write_text("unrelated") + subprocess.run([*git, "add", "."], check=True) + subprocess.run([*git, "commit", "-q", "-m", "unrelated"], check=True) + + assert script.pin_commit(repo, "build/stages.json") == pinned + with pytest.raises(script.CatalogueError, match="records no commit"): + script.pin_commit(repo, "build/missing.json") + + +# -------------------------------------------------------------------------- +# plan +# -------------------------------------------------------------------------- + + +def test_plan_matches_the_golden_commands_exactly(capsys): + exit_code, out, _err = _run( + ["--microcosm-root", str(FIXTURE_ROOT), "--root", "db/data", "--json", "plan"], + capsys, + ) + golden = json.loads(GOLDEN_PLAN.read_text()) + + assert exit_code == 0 + assert json.loads(out) == golden + assert len(golden["commands"]) == 9 + + +def test_every_planned_command_parses_with_the_reviewed_identity_as_arguments( + monkeypatch, +): + parser = _harness_parser(monkeypatch) + golden = json.loads(GOLDEN_PLAN.read_text()) + + for command in golden["commands"]: + argv = shlex.split(command) + assert argv[:3] == ["uv", "run", "chronicle"] + namespace = parser.parse_args(argv[3:]) + assert namespace.command == "fetch-artifact" + assert namespace.access == "public" + assert namespace.kind == "microdata_release" + assert namespace.licence == "US-Government-Work" + assert namespace.publisher + assert namespace.vintage + assert namespace.expected_sha256 + assert namespace.licence_evidence_issuer + assert namespace.licence_evidence_scope + assert namespace.licence_evidence_url + assert namespace.upload_r2 is True + + +def test_plan_prints_a_todo_and_never_a_guess(capsys): + golden = json.loads(GOLDEN_PLAN.read_text()) + by_release = { + shlex.split(command)[shlex.split(command).index("--package-id") + 1]: command + for command in golden["commands"] + } + + # Every reviewed publisher checksum travels as an argument. + assert ( + "--expected-sha256 d2e000250782adfbdd7f29c82b66d866591a30f0d330496698ec19f9c784ce11" + in by_release["census-cps-asec-2023"] + ) + assert "--expected-size-bytes 150165063" in by_release["census-cps-asec-2023"] + # Unknowns are TODOs the CLI refuses: no URL, no publisher-bytes checksum, + # no catalogued evidence URL. + assert "--url TODO_PUBLISHER_URL" in by_release["census-cps-basic-monthly-2024"] + assert "--url TODO_PUBLISHER_URL" in by_release["census-sipp-2023"] + for package_id in ( + "census-cps-basic-monthly-2024", + "census-acs-pums-2022-1yr", + "federal-reserve-scf-2022", + "census-sipp-2023", + ): + assert any( + "--expected-sha256 TODO_REVIEWED_SHA256" in command + for package, command in by_release.items() + if package == package_id + ) + assert all("TODO_EVIDENCE_URL" in command for command in golden["commands"]) + assert not any("TODO_VINTAGE" in command for command in golden["commands"]) + # The derived-h5 hash Microcosm pins for ACS 2022 never masquerades as + # the publisher checksum. + assert ( + "0b319b496f19a6913066f9c5ea572edfda3d78a187be6f375846617d0b441bd4" + not in "\n".join(golden["commands"]) + ) + + +def test_a_planned_command_with_a_todo_is_refused_before_anything_is_read( + tmp_path, monkeypatch, capsys +): + golden = json.loads(GOLDEN_PLAN.read_text()) + reads: list[str] = [] + + def unexpected_read(source_url): + reads.append(source_url) + raise AssertionError("a planned command with a TODO must not read") + + monkeypatch.setattr("chronicle.artifacts._read_artifact", unexpected_read) + monkeypatch.setattr( + "chronicle.artifacts._upload_r2_object", + lambda *args, **kwargs: (_ for _ in ()).throw(AssertionError("no upload")), + ) + + for command in golden["commands"]: + argv = shlex.split(command)[3:] + out_index = argv.index("--out-dir") + 1 + argv[out_index] = str(tmp_path / argv[out_index]) + exit_code = harness_main([*argv, "--staging-dir", str(tmp_path / "staging")]) + captured = capsys.readouterr() + assert exit_code == 1 + assert captured.out == "" + assert "error:" in captured.err + + assert reads == [] + assert not (tmp_path / "db").exists() + assert not (tmp_path / "staging").exists() diff --git a/tests/test_chronicle_microdata_registration.py b/tests/test_chronicle_microdata_registration.py new file mode 100644 index 00000000..ecef4e86 --- /dev/null +++ b/tests/test_chronicle_microdata_registration.py @@ -0,0 +1,2884 @@ +"""Tests for microdata-release registration: identity without content. + +Chronicle registers every raw microdata release its consumers build from and +stores the bytes of only the ones a publisher permits it to redistribute +(``docs/adr-chronicle-raw-microdata-identity.md``). These tests pin the whole +refusal surface: which access classes may carry bytes, which commands refuse +them, that every refusal happens before any filesystem or network side effect, +and that a hash-only registration is a complete, valid artifact record with no +local file and no R2 key. + +The catalogue and the checked-in synthetic consumer manifests are covered in +``tests/test_chronicle_microdata_catalogue.py``; the explicit-kind rule and the +repository guard in ``tests/test_chronicle_manifest_kind.py``. +""" + +from __future__ import annotations + +import hashlib +import json +import sys +import uuid +from pathlib import Path + +import pytest +import yaml + +from chronicle.artifacts import ( + ArtifactCommandResult, + ExpectedArtifactIdentityError, + MalformedManifestError, + SourceArtifactRevisionError, + _expected_identity, + _upsert_manifest, + fetch_source_artifact, + inventory_source_artifacts, + microdata_staging_path, + publish_source_artifacts, +) +from chronicle.harness import main as harness_main +from chronicle.registration import ( + ACCESS_CLASSES, + HASH_SOURCES, + AmbiguousVintageKeyError, + ArtifactFilenameError, + HashOnlyRegistrationError, + ListSpecRejected, + ManifestAccessError, + MicrodataReleaseNotParseableError, + bare_filename, + entry_access, + filename_key, + has_file_entries, + is_bare_filename, + is_hash_only, + is_microdata_release, + iter_file_specs, + manifest_kind, + normalize_access, + register_hash_only_artifact, + registration_id, + resolve_vintage_key, + safe_entry_access, + stores_bytes, + validate_file_entry, + validate_manifest_files, + vintage_key_forms, +) +from chronicle.source_package import ( + SOURCE_ARTIFACT_CACHE_ENV, + SOURCE_ARTIFACT_FETCH_ENV, + SourceArtifactSpec, + _read_source_artifact_content, + validate_source_package, +) + + +REPO_ROOT = Path(__file__).resolve().parents[1] +FRS_PACKAGE = REPO_ROOT / "db" / "data" / "dwp" / "frs_2023_24" +SPI_PACKAGE = REPO_ROOT / "db" / "data" / "hmrc" / "spi_public_use_tape_2022_23" + +# A syntactically valid checksum that identifies no real publisher bytes. +FIXTURE_SHA = "a" * 64 +OTHER_SHA = "b" * 64 +FIXTURE_COMMIT = "2fb2e2f8a99c37725bd6e7a15ff4c2595c912b77" + +ATTESTED = { + "hash_source": "consumer_attested", + "attested_by": "PolicyEngine/microcosm", + "attestation_evidence": ( + "microcosm uk/source_stages.json frs_spine pin, verified against the " + "licensed copy" + ), + "verified_at": "2026-09-02", +} +PINNED = { + "hash_source": "consumer_pin", + "attested_by": "PolicyEngine/microcosm", + "pinned_from": { + "repository": "PolicyEngine/microcosm", + "path": "packages/microcosm-build/src/microcosm/build/uk/source_stages.json", + "commit": FIXTURE_COMMIT, + }, +} + +# Evidence binding a public release to an allowlisted term. The URL is a +# test value; the check is that it is a durable http(s) location. +EVIDENCE = { + "issuer": "U.S. Census Bureau", + "scope": ( + "Public-use microdata file published by the U.S. Census Bureau, a " + "federal agency; a work of the United States Government under 17 " + "U.S.C. §105." + ), + "url": "https://evidence.example/census/public-use-files", +} +PUBLIC_BYTES = b"public household pums" +PUBLIC_SHA = hashlib.sha256(PUBLIC_BYTES).hexdigest() + + +def _register(output_dir: Path, **overrides: object) -> object: + """Register a fixture licensed artifact, with per-test overrides.""" + kwargs: dict[str, object] = { + "source_id": "dwp", + "package_id": "dwp-frs-2023-24", + "year": 2023, + "output_dir": output_dir, + "filename": "adult.tab", + "sha256": FIXTURE_SHA, + "licence": "UK Data Service End User Licence", + "access": "licensed", + "vintage": "2023_24", + "size_bytes": 35323384, + "doi": "10.5255/UKDA-SN-9367-2", + **ATTESTED, + } + kwargs.update(overrides) + return register_hash_only_artifact(**kwargs) # type: ignore[arg-type] + + +def _manifest(output_dir: Path) -> dict: + return yaml.safe_load((output_dir / "manifest.yaml").read_text()) + + +def _refuse_read(monkeypatch, message: str = "the publisher was read") -> list: + """Make any publisher read an ordering violation; return the read log.""" + reads: list[str] = [] + + def unexpected_read(source_url): + reads.append(source_url) + raise AssertionError(f"ORDERING VIOLATION: {message} before the refusal") + + monkeypatch.setattr("chronicle.artifacts._read_artifact", unexpected_read) + return reads + + +def _serve(monkeypatch, content: bytes, filename: str | None = None) -> list: + """Serve ``content`` for any URL, recording each read.""" + reads: list[str] = [] + + def fake_read(source_url): + reads.append(source_url) + return content, filename or Path(source_url).name + + monkeypatch.setattr("chronicle.artifacts._read_artifact", fake_read) + return reads + + +def _record_uploads(monkeypatch) -> list[tuple[str, str]]: + uploads: list[tuple[str, str]] = [] + + def fake_upload(location, local_path, *, wrangler_command): + uploads.append((location.uri, str(local_path))) + return ArtifactCommandResult( + command=("stub",), returncode=0, stdout="ok", stderr="" + ) + + monkeypatch.setattr("chronicle.artifacts._upload_r2_object", fake_upload) + return uploads + + +def _forbid_uploads(monkeypatch) -> None: + monkeypatch.setattr( + "chronicle.artifacts._upload_r2_object", + lambda *args, **kwargs: (_ for _ in ()).throw( + AssertionError("nothing may be uploaded on the way to a refusal") + ), + ) + + +def _fetch_table(source: Path, output_dir: Path, **overrides: object): + kwargs: dict[str, object] = { + "source_id": "irs_soi", + "package_id": "soi-table-1-2", + "year": 2023, + "output_dir": output_dir, + } + kwargs.update(overrides) + return fetch_source_artifact(str(source), **kwargs) # type: ignore[arg-type] + + +def _fetch_release( + output_dir: Path, + *, + staging_dir: Path, + filename: str = "csv_hus.zip", + content: bytes = PUBLIC_BYTES, + **overrides: object, +): + """Fetch a public microdata release with complete evidence.""" + kwargs: dict[str, object] = { + "source_id": "census_acs", + "package_id": "census-acs-pums-2022-1yr", + "year": 2022, + "output_dir": output_dir, + "filename": filename, + "licence": "US-Government-Work", + "access": "public", + "kind": "microdata_release", + "publisher": "U.S. Census Bureau", + "vintage": "2022", + "expected_sha256": hashlib.sha256(content).hexdigest(), + "licence_evidence": EVIDENCE, + "staging_dir": staging_dir, + } + kwargs.update(overrides) + return fetch_source_artifact( + f"https://publisher.example/pums/{filename}", + **kwargs, # type: ignore[arg-type] + ) + + +# -------------------------------------------------------------------------- +# Access, kind, filename and vintage vocabularies +# -------------------------------------------------------------------------- + + +def test_access_classes_are_the_closed_contract_set(): + assert ACCESS_CLASSES == ("public", "licensed", "restricted") + + +@pytest.mark.parametrize("access", ACCESS_CLASSES) +def test_only_public_access_may_carry_bytes(access): + assert stores_bytes(access) is (access == "public") + assert is_hash_only(access) is (access != "public") + + +def test_absent_access_is_inferred_public_for_a_table_entry(): + assert normalize_access(None) == "public" + assert entry_access({"filename": "table.xlsx"}) == "public" + + +def test_unknown_access_class_is_refused(): + with pytest.raises(ManifestAccessError, match="Unknown access class"): + normalize_access("internal") + + +def test_unknown_access_class_falls_back_to_restricted_not_public(): + # Never upload bytes because a class failed to parse. + assert safe_entry_access({"access": "internal"}) == "restricted" + + +def test_manifest_kind_is_explicit_except_for_an_absent_manifest(): + assert manifest_kind(None) == "publisher_table" + assert manifest_kind({}) == "publisher_table" + assert is_microdata_release({"kind": "microdata_release"}) is True + with pytest.raises(ManifestAccessError, match="Unknown manifest kind"): + manifest_kind({"kind": "microdata_rows"}) + # A manifest with content and no kind is an error, never a table. + with pytest.raises(ManifestAccessError, match="declares no kind"): + manifest_kind({"files": {2023: {"filename": "table.xlsx"}}}) + + +def test_a_manifest_with_no_entries_has_nothing_to_classify(): + # A bare ``files:`` line or an empty mapping declares no entry that could + # be misread as a publisher table, so the manifest reads like an absent + # one and the command writing its first entry declares the kind. + assert has_file_entries({"source_id": "irs_soi", "files": None}) is False + assert has_file_entries({"source_id": "irs_soi", "files": {}}) is False + assert has_file_entries({"files": {2023: {"filename": "t.xlsx"}}}) is True + assert has_file_entries({"files": {2023: []}}) is False + assert has_file_entries({"files": ["not", "a", "mapping"]}) is True + assert manifest_kind({"source_id": "irs_soi", "files": None}) == "publisher_table" + assert manifest_kind({"source_id": "irs_soi", "files": {}}) == "publisher_table" + with pytest.raises(ManifestAccessError, match="declares no kind"): + manifest_kind({"source_id": "irs_soi", "files": {2023: {"filename": "t"}}}) + + +def test_hash_sources_are_the_closed_contract_set(): + assert HASH_SOURCES == ("chronicle_fetch", "consumer_attested", "consumer_pin") + + +def test_registration_identity_is_the_contract_tuple(): + assert ( + registration_id( + source_id="dwp", + package_id="dwp-frs-2023-24", + year=2023, + sha256=FIXTURE_SHA, + filename="adult.tab", + ) + == f"dwp/dwp-frs-2023-24/2023/{FIXTURE_SHA}/adult.tab" + ) + + +@pytest.mark.parametrize( + "alias", + [ + "./adult.tab", + "sub/../adult.tab", + "adult.tab/", + "../pkg/adult.tab", + "/x/adult.tab", + ".", + "..", + " adult.tab", + "", + ], +) +def test_bare_filename_refuses_every_alias(alias): + assert not is_bare_filename(alias) + with pytest.raises(ArtifactFilenameError, match="bare filename"): + bare_filename(alias) + + +def test_bare_filename_accepts_a_plain_name_and_keys_case_folded(): + assert bare_filename("adult.tab") == "adult.tab" + assert filename_key("ADULT.TAB") == filename_key("adult.tab") + assert filename_key("./Adult.tab") == "adult.tab" + + +def test_vintage_key_forms_pair_the_two_spellings_of_a_year_only(): + assert vintage_key_forms(2023) == (2023, "2023") + assert vintage_key_forms("2023") == ("2023", 2023) + assert vintage_key_forms("A_1") == ("A_1",) + assert vintage_key_forms("0123") == ("0123",) + assert resolve_vintage_key({"2023": {}}, 2023) == "2023" + assert resolve_vintage_key({2023: {}}, "2023") == 2023 + assert resolve_vintage_key({}, 2023) is None + with pytest.raises(AmbiguousVintageKeyError, match="both keys"): + resolve_vintage_key({2023: {}, "2023": {}}, 2023) + + +# -------------------------------------------------------------------------- +# Entry and manifest validation vocabulary +# -------------------------------------------------------------------------- + + +def test_microdata_release_entry_requires_access_licence_and_attestation(): + errors = validate_file_entry( + {"filename": "adult.tab", "sha256": FIXTURE_SHA}, + kind="microdata_release", + manifest={}, + local_file_exists=False, + ) + + assert "missing_access" in errors + assert "missing_licence" in errors + assert "missing_hash_source" in errors + + +def _attested_entry(**mutation: object) -> dict: + entry = { + "filename": "adult.tab", + "access": "licensed", + "licence": "UK Data Service End User Licence", + "vintage": "2023_24", + "sha256": FIXTURE_SHA, + "doi": "10.5255/UKDA-SN-9367-2", + **ATTESTED, + } + entry.update(mutation) + return {key: value for key, value in entry.items() if value is not None} + + +@pytest.mark.parametrize( + ("mutation", "expected_code"), + [ + ({"licence": None}, "missing_licence"), + ({"sha256": None}, "missing_sha256"), + ({"sha256": "not-a-checksum"}, "malformed_sha256"), + ({"sha256": FIXTURE_SHA.upper()}, "malformed_sha256"), + ({"vintage": None}, "missing_vintage"), + ({"doi": None}, "missing_access_route"), + ({"hash_source": None}, "missing_hash_source"), + ({"hash_source": "transcribed"}, "unknown_hash_source:transcribed"), + ({"attested_by": None}, "missing_attested_by"), + ({"attestation_evidence": None}, "missing_attestation_evidence"), + ({"verified_at": None}, "missing_verified_at"), + ({"filename": "./adult.tab"}, "non_canonical_filename:./adult.tab"), + ], +) +def test_hash_only_entry_reports_each_missing_field(mutation, expected_code): + errors = validate_file_entry( + _attested_entry(**mutation), + kind="microdata_release", + manifest={}, + local_file_exists=False, + ) + + assert expected_code in errors + + +@pytest.mark.parametrize( + ("mutation", "expected_code"), + [ + ({"pinned_from": None}, "missing_pinned_from"), + ({"pinned_from": "microcosm@abc"}, "malformed_pinned_from"), + ( + {"pinned_from": {"repository": "PolicyEngine/microcosm"}}, + "pinned_from_missing_field:commit", + ), + ( + {"pinned_from": {**PINNED["pinned_from"], "commit": "abc123"}}, + "malformed_pinned_from_commit", + ), + ({"verified_at": "2026-09-02"}, "verified_at_forbidden_for_consumer_pin"), + ], +) +def test_consumer_pin_entry_reports_its_own_fields(mutation, expected_code): + entry = _attested_entry(attestation_evidence=None, verified_at=None, **PINNED) + entry.update(mutation) + entry = {key: value for key, value in entry.items() if value is not None} + + errors = validate_file_entry( + entry, kind="microdata_release", manifest={}, local_file_exists=False + ) + + assert expected_code in errors + + +def test_chronicle_fetch_entry_is_attested_by_chronicle_with_a_date(): + entry = _attested_entry( + hash_source="chronicle_fetch", attested_by="microcosm", verified_at=None + ) + errors = validate_file_entry( + entry, kind="microdata_release", manifest={}, local_file_exists=False + ) + assert "attested_by_not_chronicle" in errors + assert "missing_verified_at" in errors + + +def test_hash_only_entry_flags_bytes_and_r2_locations(): + entry = _attested_entry( + access="restricted", + storage={ + "r2": {"bucket": "ledger-raw", "key": "raw/x/y/z"}, + "previous_r2": [{"uri": "r2://ledger-raw/raw/x/y/old"}], + }, + ) + + errors = validate_file_entry( + entry, + kind="microdata_release", + manifest={}, + local_file_exists=True, + ) + + assert "bytes_present_for_hash_only_entry" in errors + assert "r2_location_for_hash_only_entry" in errors + assert "r2_history_for_hash_only_entry" in errors + + +def test_public_table_entry_needs_no_licence_access_route_or_attestation(): + assert ( + validate_file_entry( + {"filename": "table.xlsx", "sha256": FIXTURE_SHA}, + kind="publisher_table", + manifest={}, + local_file_exists=True, + ) + == () + ) + + +def _public_release_entry(**mutation: object) -> dict: + entry = { + "filename": "csv_hus.zip", + "access": "public", + "licence": "US-Government-Work", + "licence_evidence": { + **EVIDENCE, + "licence": "US-Government-Work", + "sha256": PUBLIC_SHA, + }, + "vintage": "2022", + "sha256": PUBLIC_SHA, + "hash_source": "chronicle_fetch", + "attested_by": "chronicle", + "verified_at": "2026-09-03", + } + entry.update(mutation) + return {key: value for key, value in entry.items() if value is not None} + + +def test_public_release_entry_is_complete_with_allowlisted_evidence(): + assert ( + validate_file_entry( + _public_release_entry(), + kind="microdata_release", + manifest={}, + local_file_exists=False, + ) + == () + ) + + +@pytest.mark.parametrize( + ("mutation", "expected_code"), + [ + ( + {"licence": "U.S. Census Bureau public-use file"}, + "licence_not_redistributable:U.S. Census Bureau public-use file", + ), + ({"licence_evidence": None}, "missing_licence_evidence"), + ({"licence_evidence": "see website"}, "malformed_licence_evidence"), + ( + { + "licence_evidence": { + **EVIDENCE, + "licence": "US-Government-Work", + "sha256": PUBLIC_SHA, + "url": "", + } + }, + "licence_evidence_missing_field:url", + ), + ( + { + "licence_evidence": { + **EVIDENCE, + "licence": "CC0-1.0", + "sha256": PUBLIC_SHA, + } + }, + "licence_evidence_licence_mismatch", + ), + ( + { + "licence_evidence": { + **EVIDENCE, + "licence": "US-Government-Work", + "sha256": OTHER_SHA, + } + }, + "licence_evidence_sha256_mismatch", + ), + ( + { + "licence_evidence": { + **EVIDENCE, + "licence": "US-Government-Work", + "sha256": PUBLIC_SHA, + "url": "see the website", + } + }, + "licence_evidence_url_not_durable", + ), + ({"vintage": None}, "missing_vintage"), + ({"sha256": None}, "missing_sha256"), + ({"hash_source": None}, "missing_hash_source"), + ], +) +def test_public_release_entry_reports_missing_evidence(mutation, expected_code): + errors = validate_file_entry( + _public_release_entry(**mutation), + kind="microdata_release", + manifest={}, + local_file_exists=False, + ) + assert expected_code in errors + + +def test_public_release_bytes_beside_the_manifest_are_a_repository_violation(): + errors = validate_file_entry( + _public_release_entry(), + kind="microdata_release", + manifest={}, + local_file_exists=True, + ) + assert "bytes_present_for_microdata_release_entry" in errors + + +def test_list_file_spec_expands_only_for_a_microdata_release(): + specs = [{"filename": "adult.tab"}, {"filename": "child.tab"}] + + assert iter_file_specs(specs, kind="microdata_release") == tuple(specs) + + rejected = iter_file_specs(specs, kind="publisher_table") + assert len(rejected) == 1 + assert isinstance(rejected[0], ListSpecRejected) + assert validate_file_entry( + rejected[0], + kind="publisher_table", + manifest={}, + local_file_exists=False, + ) == ("list_file_spec_requires_microdata_release_kind",) + + +def test_manifest_level_validation_reports_keys_names_and_collisions(): + manifest = { + "files": { + 2023: [ + {"filename": "adult.tab", "access": "licensed", "sha256": FIXTURE_SHA}, + {"filename": "./adult.tab", "access": "public", "sha256": OTHER_SHA}, + {"filename": "child.tab", "sha256": FIXTURE_SHA}, + {"filename": "child.tab", "sha256": FIXTURE_SHA}, + ], + "2023": {"filename": "other.csv"}, + 2022: {"filename": "table.xlsx", "sha256": FIXTURE_SHA}, + 2021: {"filename": "TABLE.xlsx", "sha256": OTHER_SHA}, + } + } + + errors = validate_manifest_files(manifest) + + assert "duplicate_vintage_key:2023" in errors + assert "non_canonical_filename:./adult.tab" in errors + assert "filename_collision:adult.tab" in errors + assert "duplicate_filename_in_vintage:child.tab" in errors + # Two public entries, one path, different bytes: the tree can hold one. + assert "filename_collision:table.xlsx" in errors + + +def test_manifest_level_validation_accepts_the_same_file_under_two_keys(): + # The SSA manifests register one file under a year and a label key. + manifest = { + "files": { + 2024: {"filename": "ssa.csv", "sha256": FIXTURE_SHA}, + "extracted_targets": {"filename": "ssa.csv", "sha256": FIXTURE_SHA}, + } + } + assert validate_manifest_files(manifest) == () + + +def test_manifest_level_validation_accepts_a_hash_only_reissue(): + manifest = { + "files": { + 2023: [ + {"filename": "adult.tab", "access": "licensed", "sha256": FIXTURE_SHA}, + {"filename": "adult.tab", "access": "licensed", "sha256": OTHER_SHA}, + ] + } + } + assert validate_manifest_files(manifest) == () + assert validate_manifest_files({"files": ["not", "a", "mapping"]}) == ( + "files_not_a_mapping", + ) + + +# -------------------------------------------------------------------------- +# register-artifact +# -------------------------------------------------------------------------- + + +def test_register_writes_identity_without_bytes_or_an_r2_key(tmp_path): + output_dir = tmp_path / "pkg" + + report = _register(output_dir) + manifest = _manifest(output_dir) + entry = manifest["files"][2023][0] + + assert report.valid + assert report.replaced is False + assert report.hash_source == "consumer_attested" + assert report.attested_by == "PolicyEngine/microcosm" + assert report.registration == f"dwp/dwp-frs-2023-24/2023/{FIXTURE_SHA}/adult.tab" + assert manifest["kind"] == "microdata_release" + assert entry["access"] == "licensed" + assert entry["sha256"] == FIXTURE_SHA + assert entry["hash_source"] == "consumer_attested" + assert entry["attested_by"] == "PolicyEngine/microcosm" + assert entry["attestation_evidence"] == ATTESTED["attestation_evidence"] + assert entry["verified_at"] == "2026-09-02" + assert "storage" not in entry + assert sorted(path.name for path in output_dir.iterdir()) == ["manifest.yaml"] + + +def test_register_a_consumer_pin_records_where_it_was_read(tmp_path): + output_dir = tmp_path / "pkg" + + _register(output_dir, attestation_evidence=None, verified_at=None, **PINNED) + entry = _manifest(output_dir)["files"][2023][0] + + assert entry["hash_source"] == "consumer_pin" + assert entry["pinned_from"] == PINNED["pinned_from"] + assert "verified_at" not in entry + assert list(entry).index("hash_source") < list(entry).index("pinned_from") + + +def test_register_refuses_public_access(tmp_path): + with pytest.raises(HashOnlyRegistrationError, match="refuses access='public'"): + _register(tmp_path / "pkg", access="public") + + +@pytest.mark.parametrize("sha256", ["", "abc123", FIXTURE_SHA.upper(), "z" * 64]) +def test_register_never_invents_a_hash(tmp_path, sha256): + with pytest.raises(HashOnlyRegistrationError, match="Never invent a hash"): + _register(tmp_path / "pkg", sha256=sha256) + assert not (tmp_path / "pkg").exists() + + +@pytest.mark.parametrize( + ("overrides", "expected"), + [ + ({"licence": ""}, "must record the publisher licence"), + ({"vintage": ""}, "must record the artifact vintage"), + ({"filename": "../adult.tab"}, "must be a bare filename"), + ({"filename": "./adult.tab"}, "must be a bare filename"), + ({"hash_source": "chronicle_fetch"}, "does not apply"), + ({"hash_source": "guessed"}, "Unknown hash_source"), + ({"attested_by": ""}, "pass --attested-by"), + ({"attestation_evidence": None}, "pass --attestation-evidence"), + ({"verified_at": None}, "pass --verified-at"), + ], +) +def test_register_refuses_an_incomplete_registration(tmp_path, overrides, expected): + with pytest.raises(HashOnlyRegistrationError, match=expected): + _register(tmp_path / "pkg", **overrides) + assert not (tmp_path / "pkg").exists() + + +@pytest.mark.parametrize( + ("overrides", "expected"), + [ + ({"pinned_from": None}, "pass --pinned-from-repository"), + ({"pinned_from": {"repository": "PolicyEngine/microcosm"}}, "40-hex commit"), + ({"verified_at": "2026-09-02"}, "carries no verified_at"), + ], +) +def test_register_refuses_an_incomplete_consumer_pin(tmp_path, overrides, expected): + kwargs = {"hash_source": None, "attestation_evidence": None, "verified_at": None} + kwargs.update(PINNED) + kwargs.update(overrides) + with pytest.raises(HashOnlyRegistrationError, match=expected): + _register(tmp_path / "pkg", **kwargs) + + +def test_register_refuses_without_an_access_route(tmp_path): + with pytest.raises(HashOnlyRegistrationError, match="how the bytes are reached"): + _register(tmp_path / "pkg", doi=None) + + +def test_register_refuses_while_the_bytes_are_present(tmp_path): + output_dir = tmp_path / "pkg" + output_dir.mkdir() + (output_dir / "adult.tab").write_bytes(b"licensed microdata must not live here") + + with pytest.raises(HashOnlyRegistrationError, match="while its bytes"): + _register(output_dir) + + +def test_register_refuses_a_publisher_table_manifest(tmp_path): + output_dir = tmp_path / "pkg" + output_dir.mkdir() + (output_dir / "manifest.yaml").write_text( + yaml.safe_dump( + { + "source_id": "dwp", + "package_id": "dwp-frs-2023-24", + "kind": "publisher_table", + "files": {2023: {"filename": "table.ods", "sha256": FIXTURE_SHA}}, + } + ) + ) + + with pytest.raises(HashOnlyRegistrationError, match="publisher_table manifest"): + _register(output_dir) + + +def test_register_refuses_a_kindless_manifest_with_content(tmp_path): + output_dir = tmp_path / "pkg" + output_dir.mkdir() + (output_dir / "manifest.yaml").write_text( + yaml.safe_dump( + { + "source_id": "dwp", + "package_id": "dwp-frs-2023-24", + "files": {2023: {"filename": "table.ods", "sha256": FIXTURE_SHA}}, + } + ) + ) + original = (output_dir / "manifest.yaml").read_bytes() + + with pytest.raises(HashOnlyRegistrationError, match="declares no kind"): + _register(output_dir) + + assert (output_dir / "manifest.yaml").read_bytes() == original + + +def _entryless_manifest(tmp_path: Path, **fields: object) -> Path: + output_dir = tmp_path / "pkg" + output_dir.mkdir() + payload = {"source_id": "dwp", "package_id": "dwp-frs-2023-24", **fields} + (output_dir / "manifest.yaml").write_text( + yaml.safe_dump(payload, sort_keys=False).replace("files: null", "files:") + ) + return output_dir + + +@pytest.mark.parametrize("files", [None, {}], ids=["explicit-null", "empty-mapping"]) +def test_register_declares_the_kind_of_an_entryless_kindless_manifest(tmp_path, files): + output_dir = _entryless_manifest(tmp_path, files=files) + + report = _register(output_dir) + manifest = _manifest(output_dir) + + assert report.valid + assert manifest["kind"] == "microdata_release" + assert manifest["files"][2023][0]["filename"] == "adult.tab" + + +def test_register_never_reclassifies_a_declared_publisher_table(tmp_path): + # An explicit kind is fixed even when the manifest holds no entry yet. + output_dir = _entryless_manifest(tmp_path, kind="publisher_table", files={}) + original = (output_dir / "manifest.yaml").read_bytes() + + with pytest.raises(HashOnlyRegistrationError, match="publisher_table manifest"): + _register(output_dir) + + assert (output_dir / "manifest.yaml").read_bytes() == original + + +def test_register_refuses_a_manifest_for_a_different_source(tmp_path): + output_dir = tmp_path / "pkg" + _register(output_dir) + + with pytest.raises(HashOnlyRegistrationError, match="declares source_id='dwp'"): + _register(output_dir, source_id="ons", filename="other.tab") + + +def test_register_is_idempotent_and_byte_stable(tmp_path): + output_dir = tmp_path / "pkg" + _register(output_dir) + first = (output_dir / "manifest.yaml").read_bytes() + + report = _register(output_dir) + + assert report.replaced is True + assert (output_dir / "manifest.yaml").read_bytes() == first + + +def test_register_refuses_a_reissue_unless_it_is_asked_for(tmp_path): + output_dir = tmp_path / "pkg" + _register(output_dir) + original = (output_dir / "manifest.yaml").read_bytes() + + with pytest.raises(HashOnlyRegistrationError, match="pass --allow-reissue"): + _register(output_dir, sha256=OTHER_SHA) + assert (output_dir / "manifest.yaml").read_bytes() == original + + report = _register(output_dir, sha256=OTHER_SHA, allow_reissue=True) + entries = _manifest(output_dir)["files"][2023] + + assert report.replaced is False + assert [entry["sha256"] for entry in entries] == [FIXTURE_SHA, OTHER_SHA] + + +def test_register_keeps_distinct_files_under_one_vintage(tmp_path): + output_dir = tmp_path / "pkg" + _register(output_dir, filename="adult.tab") + _register(output_dir, filename="child.tab", sha256=OTHER_SHA) + + entries = _manifest(output_dir)["files"][2023] + + assert [entry["filename"] for entry in entries] == ["adult.tab", "child.tab"] + + +def test_reregistering_the_current_pin_stays_idempotent_after_a_reissue(tmp_path): + output_dir = tmp_path / "pkg" + _register(output_dir, sha256=FIXTURE_SHA) + _register(output_dir, sha256=OTHER_SHA, allow_reissue=True) + + report = _register(output_dir, sha256=OTHER_SHA) + + assert report.replaced is True + entries = _manifest(output_dir)["files"][2023] + assert [entry["sha256"] for entry in entries] == [FIXTURE_SHA, OTHER_SHA] + + +# Finding 7: a quoted string year key is the same vintage as the integer. + + +def _quoted_year_release(tmp_path: Path, **entry_overrides: object) -> Path: + output_dir = tmp_path / "pkg" + output_dir.mkdir() + entry = _attested_entry(**entry_overrides) + (output_dir / "manifest.yaml").write_text( + yaml.safe_dump( + { + "source_id": "dwp", + "package_id": "dwp-frs-2023-24", + "kind": "microdata_release", + "files": {"2023": [entry]}, + }, + sort_keys=False, + ) + ) + return output_dir + + +def test_register_sees_a_registration_under_a_quoted_year_key(tmp_path): + output_dir = _quoted_year_release(tmp_path) + original = (output_dir / "manifest.yaml").read_bytes() + + with pytest.raises(HashOnlyRegistrationError, match="already registers"): + _register(output_dir, sha256=OTHER_SHA) + assert (output_dir / "manifest.yaml").read_bytes() == original + + report = _register(output_dir) + manifest = _manifest(output_dir) + + assert report.replaced is True + # The manifest's own key spelling is retained; no parallel key appears. + assert list(manifest["files"]) == ["2023"] + + +def test_register_refuses_a_vintage_recorded_under_both_key_spellings(tmp_path): + output_dir = _quoted_year_release(tmp_path) + manifest = _manifest(output_dir) + manifest["files"][2023] = [_attested_entry(filename="child.tab")] + (output_dir / "manifest.yaml").write_text(yaml.safe_dump(manifest, sort_keys=False)) + original = (output_dir / "manifest.yaml").read_bytes() + + with pytest.raises(HashOnlyRegistrationError, match="duplicate_vintage_key"): + _register(output_dir, filename="job.tab", sha256=OTHER_SHA) + + assert (output_dir / "manifest.yaml").read_bytes() == original + + +# Finding 3: a public identity with an object in R2 is not reclassified. + + +def _archived_release(tmp_path: Path, monkeypatch, *, filename="asecpub23csv.zip"): + """A public release whose bytes were archived, then cleaned up locally.""" + output_dir = tmp_path / "pkg" + staging = tmp_path / "staging" + uploads = _record_uploads(monkeypatch) + _serve(monkeypatch, PUBLIC_BYTES) + _fetch_release( + output_dir, + staging_dir=staging, + filename=filename, + source_id="census_cps", + package_id="census-cps-asec-2023", + year=2023, + upload_r2=True, + ) + assert uploads + staged = microdata_staging_path( + staging_dir=staging, + source_id="census_cps", + package_id="census-cps-asec-2023", + year=2023, + sha256=PUBLIC_SHA, + filename=filename, + ) + staged.unlink() + return output_dir, uploads + + +@pytest.mark.parametrize("access", ["licensed", "restricted"]) +def test_register_refuses_to_reclassify_an_identity_r2_still_holds( + tmp_path, monkeypatch, access +): + output_dir, uploads = _archived_release(tmp_path, monkeypatch) + original = (output_dir / "manifest.yaml").read_bytes() + + with pytest.raises(HashOnlyRegistrationError, match="records the R2 object"): + _register( + output_dir, + source_id="census_cps", + package_id="census-cps-asec-2023", + filename="asecpub23csv.zip", + sha256=PUBLIC_SHA, + access=access, + licence="Some licence", + vintage="2023", + ) + + assert (output_dir / "manifest.yaml").read_bytes() == original + entry = _manifest(output_dir)["files"][2023][0] + assert entry["storage"]["r2"]["uri"] == uploads[0][0] + inventory = inventory_source_artifacts(output_dir, staging_dir=tmp_path / "staging") + assert inventory.counts["r2_link_count"] == 1 + + +def test_register_refuses_to_reclassify_an_unarchived_public_entry(tmp_path): + output_dir = tmp_path / "pkg" + output_dir.mkdir() + (output_dir / "manifest.yaml").write_text( + yaml.safe_dump( + { + "source_id": "census_cps", + "package_id": "census-cps-asec-2023", + "kind": "microdata_release", + "files": {2023: [_public_release_entry(filename="asecpub23csv.zip")]}, + }, + sort_keys=False, + ) + ) + original = (output_dir / "manifest.yaml").read_bytes() + + with pytest.raises(HashOnlyRegistrationError, match="explicit decision"): + _register( + output_dir, + source_id="census_cps", + package_id="census-cps-asec-2023", + filename="ASECPUB23CSV.ZIP", + sha256=PUBLIC_SHA, + licence="Some licence", + vintage="2023", + ) + + assert (output_dir / "manifest.yaml").read_bytes() == original + + +def test_register_refuses_a_collision_with_a_public_alias(tmp_path): + # Finding 1 from the registration side: a manifest never holds one path + # under two access classes, whatever spelling the other entry uses. + output_dir = tmp_path / "pkg" + output_dir.mkdir() + (output_dir / "manifest.yaml").write_text( + yaml.safe_dump( + { + "source_id": "dwp", + "package_id": "dwp-frs-2023-24", + "kind": "microdata_release", + "files": {2022: [_public_release_entry(filename="Adult.tab")]}, + }, + sort_keys=False, + ) + ) + + with pytest.raises(HashOnlyRegistrationError, match="explicit decision"): + _register(output_dir) + + +# -------------------------------------------------------------------------- +# fetch-artifact +# -------------------------------------------------------------------------- + + +def test_fetch_refuses_a_hash_only_access_class(tmp_path, monkeypatch): + source = tmp_path / "adult.tab" + source.write_bytes(b"licensed microdata") + _refuse_read(monkeypatch) + + with pytest.raises(ManifestAccessError, match="fetch-artifact stores bytes"): + fetch_source_artifact( + str(source), + source_id="dwp", + package_id="dwp-frs-2023-24", + year=2023, + output_dir=tmp_path / "pkg", + access="licensed", + ) + + +def test_fetch_refuses_to_pull_bytes_over_a_hash_only_registration( + tmp_path, monkeypatch +): + output_dir = tmp_path / "pkg" + _register(output_dir) + original_manifest = (output_dir / "manifest.yaml").read_bytes() + source = tmp_path / "adult.tab" + source.write_bytes(b"licensed microdata") + _refuse_read(monkeypatch) + + with pytest.raises(ManifestAccessError, match="Its bytes must not enter"): + fetch_source_artifact( + str(source), + source_id="dwp", + package_id="dwp-frs-2023-24", + year=2023, + output_dir=output_dir, + filename="adult.tab", + access="public", + ) + + assert not (output_dir / "adult.tab").exists() + assert (output_dir / "manifest.yaml").read_bytes() == original_manifest + + +# Finding 5: the inferred filename is refused before the read, not after. + + +@pytest.mark.parametrize("year", [2023, 2022], ids=["same-vintage", "other-vintage"]) +def test_fetch_without_filename_refuses_a_hash_only_registration_before_reading( + tmp_path, monkeypatch, year +): + output_dir = tmp_path / "pkg" + _register(output_dir) + original_manifest = (output_dir / "manifest.yaml").read_bytes() + reads = _refuse_read(monkeypatch, "the artifact was downloaded") + + with pytest.raises(ManifestAccessError, match="Its bytes must not enter"): + fetch_source_artifact( + "https://beta.ukdataservice.ac.uk/Umbraco/Surface/Download/9367/adult.tab", + source_id="dwp", + package_id="dwp-frs-2023-24", + year=year, + output_dir=output_dir, + licence="UK Data Service End User Licence", + access="public", + ) + + assert reads == [] + assert (output_dir / "manifest.yaml").read_bytes() == original_manifest + assert sorted(path.name for path in output_dir.iterdir()) == ["manifest.yaml"] + + +def test_fetch_refuses_an_uninferrable_or_unsupported_source_before_reading( + tmp_path, monkeypatch +): + reads = _refuse_read(monkeypatch) + common = { + "source_id": "irs_soi", + "package_id": "soi-table-1-2", + "year": 2023, + "output_dir": tmp_path / "pkg", + } + with pytest.raises(ArtifactFilenameError, match="inferred from the URL"): + fetch_source_artifact("https://publisher.example/", **common) + with pytest.raises(ValueError, match="Unsupported source URL scheme"): + fetch_source_artifact("ftp://publisher.example/table.xlsx", **common) + assert reads == [] + assert not (tmp_path / "pkg").exists() + + +# Finding 1: no alias of a registered name slips past the byte boundary. + +ALIASES = { + "dot-slash": lambda out: "./adult.tab", + "dot-dot-segment": lambda out: "sub/../adult.tab", + "trailing-slash": lambda out: "adult.tab/", + "parent-then-back": lambda out: f"../{out.name}/adult.tab", + "absolute": lambda out: str(out / "adult.tab"), + "case": lambda out: "ADULT.TAB", +} + + +@pytest.mark.parametrize("alias", ALIASES.values(), ids=ALIASES.keys()) +def test_fetch_refuses_every_alias_of_a_hash_only_registration_before_reading( + tmp_path, monkeypatch, alias +): + output_dir = tmp_path / "pkg" + _register(output_dir) + original_manifest = (output_dir / "manifest.yaml").read_bytes() + reads = _refuse_read(monkeypatch) + _forbid_uploads(monkeypatch) + + with pytest.raises(ManifestAccessError): + fetch_source_artifact( + "https://publisher.example/frs/adult.tab", + source_id="dwp", + package_id="dwp-frs-2023-24", + year=2023, + output_dir=output_dir, + filename=alias(output_dir), + licence="Open Government Licence v3", + access="public", + upload_r2=True, + ) + + assert reads == [] + assert sorted(path.name for path in output_dir.iterdir()) == ["manifest.yaml"] + assert (output_dir / "manifest.yaml").read_bytes() == original_manifest + + +def test_fetch_refuses_a_path_that_escapes_the_package_directory(tmp_path, monkeypatch): + reads = _refuse_read(monkeypatch) + with pytest.raises(ArtifactFilenameError, match="bare filename"): + _fetch_table( + tmp_path / "table.xlsx", + tmp_path / "pkg", + filename="../../escaped.xlsx", + ) + assert reads == [] + assert not (tmp_path / "escaped.xlsx").exists() + + +def test_fetch_writes_the_access_class_and_kind_explicitly(tmp_path): + source = tmp_path / "table.xlsx" + source.write_bytes(b"publisher table") + output_dir = tmp_path / "pkg" + + _fetch_table(source, output_dir) + manifest = _manifest(output_dir) + + assert list(manifest)[:3] == ["source_id", "package_id", "kind"] + assert manifest["kind"] == "publisher_table" + assert manifest["files"][2023]["access"] == "public" + + +def test_fetch_into_a_microdata_release_manifest_requires_the_evidence( + tmp_path, monkeypatch +): + output_dir = tmp_path / "pkg" + _register(output_dir) + original = (output_dir / "manifest.yaml").read_bytes() + reads = _refuse_read(monkeypatch) + base = { + "source_id": "dwp", + "package_id": "dwp-frs-2023-24", + "year": 2023, + "output_dir": output_dir, + "filename": "codebook.pdf", + "access": "public", + "licence": "OGL-UK-3.0", + "publisher": "Department for Work and Pensions", + "vintage": "2023_24", + "expected_sha256": FIXTURE_SHA, + "licence_evidence": {**EVIDENCE, "issuer": "DWP"}, + "staging_dir": tmp_path / "staging", + } + cases = [ + ({"licence": None}, "pass --licence"), + ({"vintage": None}, "pass --vintage"), + ({"publisher": None}, "pass --publisher"), + ({"expected_sha256": None}, "pass --expected-sha256"), + ( + {"licence": "UK Data Service End User Licence"}, + "not on Chronicle's allowlist", + ), + ({"licence_evidence": None}, "licence_evidence_missing_field:issuer"), + ( + {"licence_evidence": {**EVIDENCE, "url": "ask the archive"}}, + "url_not_durable", + ), + ] + for overrides, expected in cases: + kwargs = {**base, **overrides} + with pytest.raises(ManifestAccessError, match=expected): + fetch_source_artifact("https://publisher.example/codebook.pdf", **kwargs) + + assert reads == [] + assert (output_dir / "manifest.yaml").read_bytes() == original + + +def test_fetch_refuses_a_registration_recorded_under_another_vintage( + tmp_path, monkeypatch +): + # The write target is a path in the package directory, not a year, so a + # registration under 2022 must still block a fetch requested for 2023. + package = tmp_path / "pkg" + _register(package, year=2022, vintage="2022_23") + reads = _refuse_read(monkeypatch) + + with pytest.raises(ManifestAccessError, match="for 2022 as access='licensed'"): + fetch_source_artifact( + "https://publisher.example/adult.tab", + source_id="dwp", + package_id="dwp-frs-2023-24", + year=2023, + output_dir=package, + filename="adult.tab", + licence="OGL-UK-3.0", + access="public", + ) + + assert reads == [] + assert not (package / "adult.tab").exists() + + +def test_fetch_refuses_a_list_entry_in_a_manifest_without_a_kind(tmp_path, monkeypatch): + # A missing kind must not make the guard blind to the entries the + # manifest actually holds: the byte boundary is refused first. + package = tmp_path / "pkg" + package.mkdir() + (package / "manifest.yaml").write_text( + yaml.safe_dump( + { + "source_id": "dwp", + "package_id": "dwp-frs-2023-24", + "files": {2023: [_attested_entry()]}, + } + ) + ) + reads = _refuse_read(monkeypatch) + + with pytest.raises(ManifestAccessError, match="Its bytes must not enter"): + fetch_source_artifact( + "https://publisher.example/adult.tab", + source_id="dwp", + package_id="dwp-frs-2023-24", + year=2023, + output_dir=package, + filename="adult.tab", + access="public", + ) + + assert reads == [] + assert not (package / "adult.tab").exists() + + +# Finding 4: the existing manifest is validated strictly before any I/O. + + +def test_fetch_refuses_a_kind_that_conflicts_with_the_manifest(tmp_path, monkeypatch): + release = tmp_path / "release" + _register(release) + table = tmp_path / "table" + (tmp_path / "t.xlsx").write_bytes(b"table") + _fetch_table(tmp_path / "t.xlsx", table) + reads = _refuse_read(monkeypatch) + _forbid_uploads(monkeypatch) + before = {path: (path / "manifest.yaml").read_bytes() for path in (release, table)} + + with pytest.raises(ManifestAccessError, match="is a microdata_release manifest"): + fetch_source_artifact( + "https://publisher.example/codebook.pdf", + source_id="dwp", + package_id="dwp-frs-2023-24", + year=2023, + output_dir=release, + kind="publisher_table", + upload_r2=True, + ) + with pytest.raises(ManifestAccessError, match="is a publisher_table manifest"): + fetch_source_artifact( + "https://publisher.example/22in05ira.xlsx", + source_id="irs_soi", + package_id="soi-table-1-2", + year=2022, + output_dir=table, + kind="microdata_release", + licence="US-Government-Work", + upload_r2=True, + ) + + assert reads == [] + for path, original in before.items(): + assert (path / "manifest.yaml").read_bytes() == original + assert not (path / "codebook.pdf").exists() + + +@pytest.mark.parametrize("files", [None, {}], ids=["explicit-null", "empty-mapping"]) +def test_fetch_declares_the_requested_kind_on_an_entryless_kindless_manifest( + tmp_path, monkeypatch, files +): + output_dir = _entryless_manifest(tmp_path, files=files) + _serve(monkeypatch, PUBLIC_BYTES) + + report = _fetch_release( + output_dir, + staging_dir=tmp_path / "staging", + source_id="dwp", + package_id="dwp-frs-2023-24", + year=2023, + licence="OGL-UK-3.0", + publisher="Department for Work and Pensions", + vintage="2023_24", + licence_evidence={**EVIDENCE, "issuer": "DWP"}, + ) + manifest = _manifest(output_dir) + + assert report.valid + assert manifest["kind"] == "microdata_release" + assert [entry["filename"] for entry in manifest["files"][2023]] == ["csv_hus.zip"] + + +def test_fetch_never_reclassifies_a_declared_kind_without_entries( + tmp_path, monkeypatch +): + output_dir = _entryless_manifest(tmp_path, kind="publisher_table", files={}) + original = (output_dir / "manifest.yaml").read_bytes() + reads = _refuse_read(monkeypatch) + + with pytest.raises(ManifestAccessError, match="is a publisher_table manifest"): + _fetch_release( + output_dir, + staging_dir=tmp_path / "staging", + source_id="dwp", + package_id="dwp-frs-2023-24", + year=2023, + licence="OGL-UK-3.0", + publisher="Department for Work and Pensions", + vintage="2023_24", + licence_evidence={**EVIDENCE, "issuer": "DWP"}, + ) + + assert reads == [] + assert (output_dir / "manifest.yaml").read_bytes() == original + + +def test_fetch_refuses_a_stored_unknown_kind_even_with_an_explicit_kind( + tmp_path, monkeypatch +): + package = tmp_path / "pkg" + package.mkdir() + (package / "manifest.yaml").write_text( + yaml.safe_dump( + { + "source_id": "dwp", + "package_id": "dwp-frs-2023-24", + "kind": "microdata_rows", + "files": {2023: {"filename": "table.ods", "sha256": FIXTURE_SHA}}, + } + ) + ) + original = (package / "manifest.yaml").read_bytes() + reads = _refuse_read(monkeypatch) + + for kind in (None, "publisher_table"): + with pytest.raises(ManifestAccessError, match="unknown manifest kind"): + fetch_source_artifact( + "https://publisher.example/codebook.pdf", + source_id="dwp", + package_id="dwp-frs-2023-24", + year=2023, + output_dir=package, + kind=kind, + ) + + assert reads == [] + assert (package / "manifest.yaml").read_bytes() == original + + +def test_fetch_refuses_a_release_entry_without_an_access_class(tmp_path, monkeypatch): + package = tmp_path / "pkg" + package.mkdir() + entry = _attested_entry() + del entry["access"] + (package / "manifest.yaml").write_text( + yaml.safe_dump( + { + "source_id": "dwp", + "package_id": "dwp-frs-2023-24", + "kind": "microdata_release", + "publisher": "DWP", + "files": {2023: [entry]}, + } + ) + ) + original = (package / "manifest.yaml").read_bytes() + reads = _refuse_read(monkeypatch) + _forbid_uploads(monkeypatch) + + with pytest.raises(ManifestAccessError, match="missing_access"): + fetch_source_artifact( + "https://publisher.example/adult.tab", + source_id="dwp", + package_id="dwp-frs-2023-24", + year=2023, + output_dir=package, + filename="adult.tab", + licence="OGL-UK-3.0", + vintage="2023_24", + expected_sha256=FIXTURE_SHA, + licence_evidence={**EVIDENCE, "issuer": "DWP"}, + upload_r2=True, + ) + + assert reads == [] + assert (package / "manifest.yaml").read_bytes() == original + assert not (package / "adult.tab").exists() + + +def test_fetch_refuses_an_invalid_manifest_before_reading(tmp_path, monkeypatch): + package = tmp_path / "pkg" + package.mkdir() + (package / "manifest.yaml").write_text( + yaml.safe_dump( + { + "source_id": "dwp", + "package_id": "dwp-frs-2023-24", + "kind": "microdata_release", + "files": { + 2023: [ + _attested_entry(filename="adult.tab"), + _public_release_entry(filename="./adult.tab"), + ] + }, + } + ) + ) + original = (package / "manifest.yaml").read_bytes() + reads = _refuse_read(monkeypatch) + + with pytest.raises(ManifestAccessError, match="not a valid microdata_release"): + _fetch_release( + package, + staging_dir=tmp_path / "staging", + filename="job.tab", + source_id="dwp", + package_id="dwp-frs-2023-24", + year=2023, + ) + + assert reads == [] + assert (package / "manifest.yaml").read_bytes() == original + + +def test_fetch_never_drops_an_existing_registration(tmp_path, monkeypatch): + output_dir = tmp_path / "pkg" + _register(output_dir) + _serve(monkeypatch, b"a public codebook in the same package") + + _fetch_release( + output_dir, + staging_dir=tmp_path / "staging", + filename="codebook.pdf", + content=b"a public codebook in the same package", + source_id="dwp", + package_id="dwp-frs-2023-24", + year=2023, + licence="OGL-UK-3.0", + publisher="Department for Work and Pensions", + vintage="2023_24", + licence_evidence={**EVIDENCE, "issuer": "DWP"}, + ) + + entries = _manifest(output_dir)["files"][2023] + + assert [entry["filename"] for entry in entries] == ["adult.tab", "codebook.pdf"] + assert entries[0]["access"] == "licensed" + assert entries[0]["sha256"] == FIXTURE_SHA + assert entries[0]["attestation_evidence"] == ATTESTED["attestation_evidence"] + + +# -------------------------------------------------------------------------- +# Public microdata releases: bytes only with evidence, staged outside the tree +# -------------------------------------------------------------------------- + + +def test_fetch_archives_a_public_release_from_a_staging_directory( + tmp_path, monkeypatch +): + output_dir = tmp_path / "db" / "data" / "census" / "acs_pums_2022_1yr" + staging = tmp_path / "staging" + uploads = _record_uploads(monkeypatch) + _serve(monkeypatch, PUBLIC_BYTES) + + report = _fetch_release(output_dir, staging_dir=staging, upload_r2=True) + person = b"public person pums" + _serve(monkeypatch, person) + _fetch_release( + output_dir, + staging_dir=staging, + filename="csv_pus.zip", + content=person, + upload_r2=True, + ) + manifest = _manifest(output_dir) + entries = manifest["files"][2022] + + staged = microdata_staging_path( + staging_dir=staging, + source_id="census_acs", + package_id="census-acs-pums-2022-1yr", + year=2022, + sha256=PUBLIC_SHA, + filename="csv_hus.zip", + ) + assert report.valid + assert Path(report.local_path) == staged + assert staged.read_bytes() == PUBLIC_BYTES + # No release bytes ever land beside the manifest. + assert sorted(path.name for path in output_dir.iterdir()) == ["manifest.yaml"] + assert uploads[0][1] == str(staged) + assert uploads[0][0].endswith(f"/2022/{PUBLIC_SHA}/csv_hus.zip") + assert manifest["kind"] == "microdata_release" + assert manifest["publisher"] == "U.S. Census Bureau" + assert [entry["filename"] for entry in entries] == ["csv_hus.zip", "csv_pus.zip"] + first = entries[0] + assert first["access"] == "public" + assert first["licence"] == "US-Government-Work" + assert first["licence_evidence"] == { + **EVIDENCE, + "licence": "US-Government-Work", + "sha256": PUBLIC_SHA, + } + assert first["vintage"] == "2022" + assert first["hash_source"] == "chronicle_fetch" + assert first["attested_by"] == "chronicle" + assert first["verified_at"] == first["fetched_at"][:10] + assert first["storage"]["r2"]["uri"] == uploads[0][0] + + inventory = inventory_source_artifacts(output_dir, staging_dir=staging) + assert inventory.valid + assert inventory.counts["hash_only_count"] == 0 + assert inventory.counts["r2_link_count"] == 2 + assert all(entry.exists for entry in inventory.entries) + + +def test_a_public_release_without_a_recorded_object_is_incomplete( + tmp_path, monkeypatch +): + output_dir = tmp_path / "pkg" + staging = tmp_path / "staging" + _serve(monkeypatch, PUBLIC_BYTES) + _fetch_release(output_dir, staging_dir=staging) + + inventory = inventory_source_artifacts(output_dir, staging_dir=staging) + + assert not inventory.valid + assert inventory.entries[0].errors == ("r2_object_not_recorded",) + + +# Finding 8: the fetch refuses bytes the reviewed pin does not cover. + + +def test_fetch_refuses_a_reissue_before_any_side_effect(tmp_path, monkeypatch): + output_dir = tmp_path / "pkg" + staging = tmp_path / "staging" + _serve(monkeypatch, b"silently re-published bytes") + _forbid_uploads(monkeypatch) + + with pytest.raises(ExpectedArtifactIdentityError) as raised: + _fetch_release( + output_dir, + staging_dir=staging, + expected_sha256=PUBLIC_SHA, + upload_r2=True, + ) + + message = str(raised.value) + assert PUBLIC_SHA in message + assert hashlib.sha256(b"silently re-published bytes").hexdigest() in message + assert "unreviewed reissue" in message + assert not output_dir.exists() + assert not staging.exists() + + +def test_fetch_refuses_a_size_that_disagrees_with_the_pin(tmp_path, monkeypatch): + _serve(monkeypatch, PUBLIC_BYTES) + with pytest.raises(ExpectedArtifactIdentityError, match="size_bytes=1"): + _fetch_release( + tmp_path / "pkg", + staging_dir=tmp_path / "staging", + expected_size_bytes=1, + ) + assert not (tmp_path / "pkg").exists() + + +def test_record_revision_does_not_override_the_expected_identity(tmp_path, monkeypatch): + output_dir = tmp_path / "pkg" + staging = tmp_path / "staging" + _serve(monkeypatch, PUBLIC_BYTES) + _fetch_release(output_dir, staging_dir=staging) + original = (output_dir / "manifest.yaml").read_bytes() + _serve(monkeypatch, b"revised bytes") + + with pytest.raises(ExpectedArtifactIdentityError): + _fetch_release( + output_dir, + staging_dir=staging, + expected_sha256=PUBLIC_SHA, + record_revision=True, + ) + + assert (output_dir / "manifest.yaml").read_bytes() == original + + +def test_an_expectation_that_contradicts_the_record_is_refused_before_reading( + tmp_path, monkeypatch +): + output_dir = tmp_path / "pkg" + staging = tmp_path / "staging" + _serve(monkeypatch, PUBLIC_BYTES) + _fetch_release(output_dir, staging_dir=staging) + reads = _refuse_read(monkeypatch) + + with pytest.raises(ExpectedArtifactIdentityError, match="already records"): + _fetch_release( + output_dir, + staging_dir=staging, + expected_sha256=OTHER_SHA, + licence_evidence=EVIDENCE, + ) + + assert reads == [] + + +@pytest.mark.parametrize("bad", ["", "abc", FIXTURE_SHA.upper()]) +def test_fetch_never_accepts_an_invented_expected_hash(tmp_path, monkeypatch, bad): + reads = _refuse_read(monkeypatch) + with pytest.raises(ExpectedArtifactIdentityError, match="Never invent a hash"): + _fetch_table(tmp_path / "t.xlsx", tmp_path / "pkg", expected_sha256=bad) + assert reads == [] + + +def test_a_table_fetch_honours_an_expected_hash_before_writing(tmp_path, monkeypatch): + source = tmp_path / "table.xlsx" + source.write_bytes(b"publisher table") + output_dir = tmp_path / "pkg" + + with pytest.raises(ExpectedArtifactIdentityError): + _fetch_table(source, output_dir, expected_sha256=OTHER_SHA) + assert not output_dir.exists() + + report = _fetch_table( + source, + output_dir, + expected_sha256=hashlib.sha256(b"publisher table").hexdigest(), + expected_size_bytes=len(b"publisher table"), + ) + assert report.valid + + +def test_fetch_refuses_release_bytes_already_tracked_beside_the_manifest( + tmp_path, monkeypatch +): + output_dir = tmp_path / "pkg" + output_dir.mkdir() + (output_dir / "csv_hus.zip").write_bytes(PUBLIC_BYTES) + reads = _refuse_read(monkeypatch) + + with pytest.raises(ManifestAccessError, match="staged outside the package tree"): + _fetch_release(output_dir, staging_dir=tmp_path / "staging") + + assert reads == [] + + +# Finding 6: a release vintage is a list, and every entry keeps its identity. + + +def test_refetching_a_release_file_with_different_bytes_is_refused( + tmp_path, monkeypatch +): + output_dir = tmp_path / "pkg" + staging = tmp_path / "staging" + uploads = _record_uploads(monkeypatch) + _serve(monkeypatch, PUBLIC_BYTES) + _fetch_release(output_dir, staging_dir=staging, upload_r2=True) + person = b"public person pums" + _serve(monkeypatch, person) + _fetch_release( + output_dir, + staging_dir=staging, + filename="csv_pus.zip", + content=person, + upload_r2=True, + ) + original = (output_dir / "manifest.yaml").read_bytes() + revised = b"household pums, silently re-published" + reads = _refuse_read(monkeypatch) + uploads_before = list(uploads) + + # The list entry's recorded identity is live: a reviewed pin for other + # bytes contradicts it, and the contradiction is refused before the read. + with pytest.raises( + ExpectedArtifactIdentityError, match="already records" + ) as raised: + _fetch_release(output_dir, staging_dir=staging, content=revised, upload_r2=True) + + assert PUBLIC_SHA in str(raised.value) + assert "csv_hus.zip" in str(raised.value) + assert reads == [] + assert (output_dir / "manifest.yaml").read_bytes() == original + assert uploads == uploads_before + + # Served bytes that differ from the pin the record and the evidence agree + # on are refused after the read, before any write or upload. + _serve(monkeypatch, revised) + with pytest.raises(ExpectedArtifactIdentityError, match="unreviewed reissue"): + _fetch_release( + output_dir, + staging_dir=staging, + content=revised, + upload_r2=True, + expected_sha256=PUBLIC_SHA, + ) + assert (output_dir / "manifest.yaml").read_bytes() == original + assert uploads == uploads_before + assert not microdata_staging_path( + staging_dir=staging, + source_id="census_acs", + package_id="census-acs-pums-2022-1yr", + year=2022, + sha256=hashlib.sha256(revised).hexdigest(), + filename="csv_hus.zip", + ).exists() + + +def test_refetching_identical_release_bytes_preserves_the_entry_in_place( + tmp_path, monkeypatch +): + output_dir = tmp_path / "pkg" + staging = tmp_path / "staging" + _record_uploads(monkeypatch) + _serve(monkeypatch, PUBLIC_BYTES) + _fetch_release(output_dir, staging_dir=staging, upload_r2=True) + person = b"public person pums" + _serve(monkeypatch, person) + _fetch_release( + output_dir, + staging_dir=staging, + filename="csv_pus.zip", + content=person, + upload_r2=True, + ) + before = _manifest(output_dir)["files"][2022] + + monkeypatch.setenv("CHRONICLE_R2_RAW_BUCKET", "chronicle-raw") + _serve(monkeypatch, PUBLIC_BYTES) + report = _fetch_release(output_dir, staging_dir=staging, upload_r2=True) + after = _manifest(output_dir)["files"][2022] + + assert report.valid + assert [entry["filename"] for entry in after] == ["csv_hus.zip", "csv_pus.zip"] + # The recorded object is history: the backfill copy does not restate it. + assert after[0]["storage"] == before[0]["storage"] + assert after[0]["storage"]["r2"]["bucket"] == "ledger-raw" + assert after[1] == before[1] + + +def test_record_revision_of_a_release_file_supersedes_only_that_entry( + tmp_path, monkeypatch +): + output_dir = tmp_path / "pkg" + staging = tmp_path / "staging" + _record_uploads(monkeypatch) + _serve(monkeypatch, PUBLIC_BYTES) + _fetch_release(output_dir, staging_dir=staging, upload_r2=True) + person = b"public person pums" + _serve(monkeypatch, person) + _fetch_release( + output_dir, + staging_dir=staging, + filename="csv_pus.zip", + content=person, + upload_r2=True, + ) + before = _manifest(output_dir)["files"][2022] + revised = b"household pums, revised" + _serve(monkeypatch, revised) + + _fetch_release( + output_dir, + staging_dir=staging, + content=revised, + upload_r2=True, + record_revision=True, + ) + after = _manifest(output_dir)["files"][2022] + + assert [entry["filename"] for entry in after] == ["csv_hus.zip", "csv_pus.zip"] + assert after[0]["sha256"] == hashlib.sha256(revised).hexdigest() + assert after[0]["licence_evidence"]["sha256"] == after[0]["sha256"] + assert [entry["uri"] for entry in after[0]["storage"]["previous_r2"]] == [ + before[0]["storage"]["r2"]["uri"] + ] + assert after[1] == before[1] + + +def test_a_second_file_never_turns_a_publisher_table_vintage_into_a_list( + tmp_path, monkeypatch +): + output_dir = tmp_path / "pkg" + (tmp_path / "22in05ira.xlsx").write_bytes(b"IRA table 5") + _fetch_table(tmp_path / "22in05ira.xlsx", output_dir, year=2022) + original = (output_dir / "manifest.yaml").read_bytes() + (tmp_path / "22in05ira_rev.xlsx").write_bytes(b"IRA table 5, renamed") + + with pytest.raises(SourceArtifactRevisionError): + _fetch_table(tmp_path / "22in05ira_rev.xlsx", output_dir, year=2022) + assert (output_dir / "manifest.yaml").read_bytes() == original + + _fetch_table( + tmp_path / "22in05ira_rev.xlsx", output_dir, year=2022, record_revision=True + ) + revised = _manifest(output_dir)["files"][2022] + + assert isinstance(revised, dict) + assert revised["filename"] == "22in05ira_rev.xlsx" + + +def test_the_manifest_write_refuses_a_same_bytes_rename_by_itself(tmp_path): + # PR #226's rule: identical bytes under another filename are a rename, + # not a revision, so --record-revision does not apply. The write path + # repeats the guard so no caller reaches a false-provenance write. + output_dir = tmp_path / "pkg" + (tmp_path / "22in05ira.xlsx").write_bytes(b"IRA table 5") + _fetch_table(tmp_path / "22in05ira.xlsx", output_dir, year=2022) + original = (output_dir / "manifest.yaml").read_bytes() + + for record_revision in (False, True): + with pytest.raises(SourceArtifactRevisionError, match="rename is not"): + _upsert_manifest( + output_dir / "manifest.yaml", + source_id="irs_soi", + package_id="soi-table-1-2", + dataset="irs_soi_soi-table-1-2", + source_page=None, + table=None, + publisher=None, + year=2022, + filename="table-5.xlsx", + source_url="https://publisher.example/table-5.xlsx", + sha256=hashlib.sha256(b"IRA table 5").hexdigest(), + size_bytes=len(b"IRA table 5"), + fetched_at="2026-09-04T00:00:00+00:00", + access="public", + licence=None, + kind="publisher_table", + vintage=None, + licence_evidence=None, + expected=_expected_identity(None, None), + r2_location=None, + record_revision=record_revision, + ) + + assert (output_dir / "manifest.yaml").read_bytes() == original + + +def test_a_second_file_over_an_unidentified_table_entry_is_refused( + tmp_path, monkeypatch +): + output_dir = tmp_path / "pkg" + output_dir.mkdir() + (output_dir / "manifest.yaml").write_text( + yaml.safe_dump( + { + "source_id": "irs_soi", + "package_id": "soi-table-1-2", + "kind": "publisher_table", + "files": {2022: {"filename": "22in05ira.xlsx", "source_url": "x"}}, + } + ) + ) + original = (output_dir / "manifest.yaml").read_bytes() + reads = _refuse_read(monkeypatch) + + with pytest.raises(MalformedManifestError, match="one file per vintage"): + _fetch_table(tmp_path / "other.xlsx", output_dir, year=2022) + + assert reads == [] + assert (output_dir / "manifest.yaml").read_bytes() == original + + +# Finding 7 on the fetch side. + + +def test_fetch_refuses_a_revision_recorded_under_a_quoted_year_key( + tmp_path, monkeypatch +): + output_dir = tmp_path / "pkg" + (tmp_path / "t.xlsx").write_bytes(b"first") + _fetch_table(tmp_path / "t.xlsx", output_dir, year=2022) + manifest = _manifest(output_dir) + manifest["files"] = {"2022": manifest["files"][2022]} + (output_dir / "manifest.yaml").write_text(yaml.safe_dump(manifest, sort_keys=False)) + original = (output_dir / "manifest.yaml").read_bytes() + (tmp_path / "t.xlsx").write_bytes(b"second") + + with pytest.raises(SourceArtifactRevisionError, match="entry '2022'"): + _fetch_table(tmp_path / "t.xlsx", output_dir, year=2022) + assert (output_dir / "manifest.yaml").read_bytes() == original + + (tmp_path / "t.xlsx").write_bytes(b"first") + _fetch_table(tmp_path / "t.xlsx", output_dir, year=2022) + assert list(_manifest(output_dir)["files"]) == ["2022"] + + +def test_fetch_refuses_a_vintage_recorded_under_both_key_spellings( + tmp_path, monkeypatch +): + output_dir = tmp_path / "pkg" + (tmp_path / "t.xlsx").write_bytes(b"first") + _fetch_table(tmp_path / "t.xlsx", output_dir, year=2022) + manifest = _manifest(output_dir) + manifest["files"]["2022"] = dict(manifest["files"][2022]) + (output_dir / "manifest.yaml").write_text(yaml.safe_dump(manifest, sort_keys=False)) + original = (output_dir / "manifest.yaml").read_bytes() + reads = _refuse_read(monkeypatch) + + with pytest.raises(ManifestAccessError, match="duplicate_vintage_key"): + _fetch_table(tmp_path / "t.xlsx", output_dir, year=2022) + + assert reads == [] + assert (output_dir / "manifest.yaml").read_bytes() == original + + +# -------------------------------------------------------------------------- +# publish-raw +# -------------------------------------------------------------------------- + + +def _hash_only_tree(tmp_path: Path) -> Path: + """Build a data root holding one hash-only registration.""" + root = tmp_path / "data" + _register(root / "dwp" / "frs_2023_24") + return root + + +def test_publish_raw_refuses_hash_only_entries_without_reading_bytes( + tmp_path, monkeypatch +): + root = _hash_only_tree(tmp_path) + _forbid_uploads(monkeypatch) + + report = publish_source_artifacts(root) + + assert not report.valid + assert report.counts["hash_only_refused_count"] == 1 + assert report.counts["uploaded_count"] == 0 + entry = report.entries[0] + assert entry.skipped == "hash_only_access:licensed" + assert entry.errors == ("hash_only_access_refuses_bytes:licensed",) + assert entry.r2_location is None + assert entry.upload is None + + +def test_publish_raw_skip_hash_only_reports_the_skip_without_failing( + tmp_path, monkeypatch +): + root = _hash_only_tree(tmp_path) + _forbid_uploads(monkeypatch) + + report = publish_source_artifacts(root, skip_hash_only=True) + + assert report.valid + assert report.counts["hash_only_refused_count"] == 1 + assert report.counts["uploaded_count"] == 0 + assert report.entries[0].errors == () + assert report.entries[0].r2_location is None + + +def test_publish_raw_leaves_the_manifest_untouched(tmp_path, monkeypatch): + root = _hash_only_tree(tmp_path) + manifest_path = root / "dwp" / "frs_2023_24" / "manifest.yaml" + original = manifest_path.read_bytes() + _forbid_uploads(monkeypatch) + + publish_source_artifacts(root, skip_hash_only=True) + + assert manifest_path.read_bytes() == original + + +def test_publish_raw_reports_a_violation_even_when_skipping(tmp_path, monkeypatch): + root = _hash_only_tree(tmp_path) + (root / "dwp" / "frs_2023_24" / "adult.tab").write_bytes(b"leaked bytes") + _forbid_uploads(monkeypatch) + + report = publish_source_artifacts(root, skip_hash_only=True) + + # --skip-hash-only turns off the refusal, not the contract check. + assert not report.valid + assert "bytes_present_for_hash_only_entry" in report.entries[0].errors + + +@pytest.mark.parametrize("skip_hash_only", [False, True]) +def test_publish_raw_never_uploads_a_hash_only_file_through_a_public_alias( + tmp_path, monkeypatch, skip_hash_only +): + # Finding 1: a hand-edited public alias of the licensed entry must not + # carry its bytes to the bucket; the manifest is refused whole. + root = _hash_only_tree(tmp_path) + package = root / "dwp" / "frs_2023_24" + (package / "adult.tab").write_bytes(b"leaked licensed bytes") + manifest = _manifest(package) + manifest["files"][2023].append( + { + **_public_release_entry(filename="./adult.tab"), + "sha256": hashlib.sha256(b"leaked licensed bytes").hexdigest(), + } + ) + (package / "manifest.yaml").write_text(yaml.safe_dump(manifest, sort_keys=False)) + original = (package / "manifest.yaml").read_bytes() + _forbid_uploads(monkeypatch) + + report = publish_source_artifacts(root, skip_hash_only=skip_hash_only) + + assert not report.valid + assert report.entries == () + assert any("filename_collision:adult.tab" in error for error in report.errors) + assert any("non_canonical_filename:./adult.tab" in error for error in report.errors) + assert (package / "manifest.yaml").read_bytes() == original + + +def test_publish_raw_uploads_a_staged_release_and_never_the_tree(tmp_path, monkeypatch): + output_dir = tmp_path / "data" / "census" / "acs_pums_2022_1yr" + staging = tmp_path / "staging" + _serve(monkeypatch, PUBLIC_BYTES) + _fetch_release(output_dir, staging_dir=staging) + uploads = _record_uploads(monkeypatch) + + report = publish_source_artifacts(tmp_path / "data", staging_dir=staging) + + assert report.valid + assert report.counts["uploaded_count"] == 1 + assert uploads[0][1].startswith(str(staging)) + assert ( + _manifest(output_dir)["files"][2022][0]["storage"]["r2"]["uri"] + == (uploads[0][0]) + ) + + (output_dir / "csv_hus.zip").write_bytes(PUBLIC_BYTES) + report = publish_source_artifacts(tmp_path / "data", staging_dir=staging) + assert not report.valid + assert "bytes_present_for_microdata_release_entry" in report.entries[0].errors + + +# -------------------------------------------------------------------------- +# inventory-artifacts +# -------------------------------------------------------------------------- + + +def test_inventory_accepts_a_hash_only_entry_with_no_local_file(tmp_path): + root = _hash_only_tree(tmp_path) + + report = inventory_source_artifacts(root) + + assert report.valid + assert report.counts["hash_only_count"] == 1 + assert report.counts["missing_count"] == 0 + entry = report.entries[0] + assert entry.valid + assert entry.exists is False + assert entry.hash_only is True + assert entry.access == "licensed" + assert entry.licence == "UK Data Service End User Licence" + assert entry.sha256_expected == FIXTURE_SHA + assert entry.sha256_actual is None + # The size is the publisher's, recorded rather than measured. + assert entry.size_bytes == 35323384 + + +def test_inventory_flags_a_hash_only_entry_whose_bytes_are_present(tmp_path): + root = _hash_only_tree(tmp_path) + (root / "dwp" / "frs_2023_24" / "adult.tab").write_bytes(b"leaked bytes") + + report = inventory_source_artifacts(root) + + assert not report.valid + assert "bytes_present_for_hash_only_entry" in report.entries[0].errors + + +def test_inventory_rejects_a_list_entry_outside_a_microdata_release(tmp_path): + package = tmp_path / "data" / "dwp" / "tables" + package.mkdir(parents=True) + (package / "manifest.yaml").write_text( + yaml.safe_dump( + { + "source_id": "dwp", + "package_id": "dwp-tables", + "kind": "publisher_table", + "files": {2023: [{"filename": "a.ods", "sha256": FIXTURE_SHA}]}, + } + ) + ) + + report = inventory_source_artifacts(tmp_path / "data") + + assert not report.valid + assert report.entries[0].errors == ( + "list_file_spec_requires_microdata_release_kind", + ) + + +def test_inventory_reports_manifest_level_defects(tmp_path): + package = tmp_path / "data" / "dwp" / "tables" + package.mkdir(parents=True) + (package / "manifest.yaml").write_text( + yaml.safe_dump( + { + "source_id": "dwp", + "package_id": "dwp-tables", + "kind": "publisher_table", + "files": { + 2023: {"filename": "../a.ods", "sha256": FIXTURE_SHA}, + "2023": {"filename": "b.ods", "sha256": FIXTURE_SHA}, + }, + } + ) + ) + + report = inventory_source_artifacts(tmp_path / "data") + + assert not report.valid + assert any( + error.startswith("duplicate_vintage_key:2023") for error in report.errors + ) + assert any( + error.startswith("non_canonical_filename:../a.ods") for error in report.errors + ) + # A non-bare name is never resolved to a path outside the package. + first = next(entry for entry in report.entries if entry.filename == "../a.ods") + assert first.local_path == str(package) + assert "missing_file" not in first.errors + + +# -------------------------------------------------------------------------- +# Source packages never parse a microdata release or a hash-only entry +# -------------------------------------------------------------------------- + + +def test_source_artifact_spec_refuses_to_parse_a_registered_release(): + spec = SourceArtifactSpec( + source_name="dwp", + source_table="Family Resources Survey 2023-24", + resource_package="db", + resource_directory="data/dwp/frs_2023_24", + manifest="manifest.yaml", + vintage="2023_24", + extracted_at="2026-09-02", + extraction_method="none", + parser="delimited_text_full_rows", + artifact_year=2023, + ) + + with pytest.raises(MicrodataReleaseNotParseableError, match="registers a"): + spec.assert_parseable_manifest() + with pytest.raises(MicrodataReleaseNotParseableError): + spec._artifact_content(2023) + + +def test_year_mapping_refuses_a_multi_file_vintage_and_both_key_spellings(): + from chronicle.source_package import _year_mapping + + with pytest.raises(ValueError, match="list of 2 entries"): + _year_mapping( + {2023: [{"filename": "adult.tab"}, {"filename": "child.tab"}]}, 2023 + ) + with pytest.raises(ValueError, match="both keys"): + _year_mapping({2023: {"filename": "a"}, "2023": {"filename": "b"}}, 2023) + assert _year_mapping({"2023": {"filename": "a"}}, 2023) == {"filename": "a"} + + +def test_validate_package_reports_a_microdata_release_carve_out(tmp_path): + package_dir = tmp_path / "frs_2023_24" + package_dir.mkdir() + (package_dir / "source_package.yaml").write_text( + yaml.safe_dump( + { + "schema_version": "ledger.source_package.v1", + "package_id": "dwp-frs-2023-24-parse-attempt", + "label": "Attempt to parse a registered microdata release", + "artifact": { + "source_name": "dwp", + "source_table": "Family Resources Survey 2023-24", + "resource_package": "db", + "resource_directory": "data/dwp/frs_2023_24", + "manifest": "manifest.yaml", + "vintage": "2023_24", + "extracted_at": "2026-09-02", + "extraction_method": "none", + "parser": "delimited_text_full_rows", + "artifact_year": 2023, + }, + "record_sets": [], + }, + sort_keys=False, + ) + ) + + report = validate_source_package(package_dir, year=2023) + + assert not report.valid + assert "microdata_release_not_parseable" in {issue.code for issue in report.errors} + + +# Finding 2: the byte reader refuses a hash-only entry under any manifest kind. + +LICENSED_BYTES = b"sernum\tage\n1\t45\n2\t31\n" + + +class _UnreadableArtifactPath: + """Bytes sit in the package tree; a refusal must not read them.""" + + def read_bytes(self) -> bytes: + raise AssertionError("a hash-only entry's bytes must not be read") + + +def _isolated_reader(tmp_path: Path, monkeypatch) -> Path: + cache_root = tmp_path / "cache" + monkeypatch.setenv(SOURCE_ARTIFACT_CACHE_ENV, str(cache_root)) + monkeypatch.setattr( + "chronicle.source_package._fetch_source_artifact_content", + lambda _url: (_ for _ in ()).throw( + AssertionError("a hash-only entry must never be fetched") + ), + ) + return cache_root + + +def _licensed_table_spec( + tmp_path: Path, monkeypatch, *, kind: str | None, access: str +) -> tuple[SourceArtifactSpec, Path]: + """A publisher-table package whose single entry is hash-only. + + The resource package is a namespace package on ``sys.path`` with a unique + name, so ``importlib.resources.files`` resolves it exactly as it resolves + the repository's ``db`` package. + """ + package_name = f"chronicle_test_{uuid.uuid4().hex}" + resource_dir = tmp_path / "pkgroot" / package_name / "data" / "dwp" / "frs" + resource_dir.mkdir(parents=True) + manifest: dict = {"source_id": "dwp", "package_id": "dwp-frs"} + if kind is not None: + manifest["kind"] = kind + entry = _attested_entry( + access=access, + sha256=hashlib.sha256(LICENSED_BYTES).hexdigest(), + source_url="https://ukdataservice.example/adult.tab", + ) + manifest["files"] = {2023: entry} + (resource_dir / "manifest.yaml").write_text( + yaml.safe_dump(manifest, sort_keys=False) + ) + (resource_dir / "adult.tab").write_bytes(LICENSED_BYTES) + monkeypatch.syspath_prepend(str(tmp_path / "pkgroot")) + monkeypatch.delitem(sys.modules, package_name, raising=False) + spec = SourceArtifactSpec( + source_name="dwp", + source_table="Family Resources Survey", + resource_package=package_name, + resource_directory="data/dwp/frs", + manifest="manifest.yaml", + vintage="2023_24", + extracted_at="2026-09-02", + extraction_method="none", + parser="delimited_text_full_rows", + delimiter="\t", + artifact_year=2023, + ) + return spec, resource_dir + + +@pytest.mark.parametrize("access", ["licensed", "restricted"]) +def test_byte_reader_refuses_a_hash_only_entry_before_any_store( + tmp_path, monkeypatch, access +): + cache_root = _isolated_reader(tmp_path, monkeypatch) + entry = _attested_entry(access=access, source_url="https://x.example/adult.tab") + cached = cache_root / FIXTURE_SHA / "adult.tab" + cached.parent.mkdir(parents=True) + cached.write_bytes(LICENSED_BYTES) + monkeypatch.setenv(SOURCE_ARTIFACT_FETCH_ENV, "1") + + with pytest.raises(ManifestAccessError, match=f"access={access!r}"): + _read_source_artifact_content(_UnreadableArtifactPath(), entry) + + +def test_byte_reader_treats_an_unknown_access_class_as_unreadable( + tmp_path, monkeypatch +): + _isolated_reader(tmp_path, monkeypatch) + entry = _attested_entry(access="internal") + with pytest.raises(ManifestAccessError, match="Unknown access class 'internal'"): + _read_source_artifact_content(_UnreadableArtifactPath(), entry) + + +@pytest.mark.parametrize("kind", ["publisher_table"]) +@pytest.mark.parametrize("access", ["licensed", "restricted"]) +def test_artifact_content_refuses_a_hash_only_mapping_entry( + tmp_path, monkeypatch, kind, access +): + _isolated_reader(tmp_path, monkeypatch) + spec, _resource_dir = _licensed_table_spec( + tmp_path, monkeypatch, kind=kind, access=access + ) + + with pytest.raises(ManifestAccessError, match="identity only"): + spec.assert_parseable(2023) + with pytest.raises(ManifestAccessError): + spec._artifact_content(2023) + with pytest.raises(ManifestAccessError): + spec.build_source_rows(2023) + + +def test_validate_package_reports_a_hash_only_entry_without_reading( + tmp_path, monkeypatch +): + _isolated_reader(tmp_path, monkeypatch) + spec, _resource_dir = _licensed_table_spec( + tmp_path, monkeypatch, kind="publisher_table", access="licensed" + ) + package_dir = tmp_path / "package" + package_dir.mkdir() + (package_dir / "source_package.yaml").write_text( + yaml.safe_dump( + { + "schema_version": "ledger.source_package.v1", + "package_id": "dwp-frs-parse-attempt", + "label": "Attempt to parse a hash-only table entry", + "artifact": { + "source_name": spec.source_name, + "source_table": spec.source_table, + "resource_package": spec.resource_package, + "resource_directory": spec.resource_directory, + "manifest": "manifest.yaml", + "vintage": "2023_24", + "extracted_at": "2026-09-02", + "extraction_method": "none", + "parser": "delimited_text_full_rows", + "delimiter": "\t", + "artifact_year": 2023, + }, + "record_sets": [], + }, + sort_keys=False, + ) + ) + + report = validate_source_package(package_dir, year=2023) + + assert not report.valid + assert [issue.code for issue in report.errors] == [ + "hash_only_artifact_not_parseable" + ] + + +def test_build_suite_refuses_a_hash_only_entry_before_touching_the_output( + tmp_path, monkeypatch +): + from chronicle.suite import build_source_suite + + _isolated_reader(tmp_path, monkeypatch) + spec, _resource_dir = _licensed_table_spec( + tmp_path, monkeypatch, kind="publisher_table", access="licensed" + ) + package_dir = tmp_path / "package" + package_dir.mkdir() + (package_dir / "source_package.yaml").write_text( + yaml.safe_dump( + { + "schema_version": "ledger.source_package.v1", + "package_id": "dwp-frs-parse-attempt", + "label": "Attempt to build a hash-only table entry", + "artifact": { + "source_name": spec.source_name, + "source_table": spec.source_table, + "resource_package": spec.resource_package, + "resource_directory": spec.resource_directory, + "manifest": "manifest.yaml", + "vintage": "2023_24", + "extracted_at": "2026-09-02", + "extraction_method": "none", + "parser": "delimited_text_full_rows", + "delimiter": "\t", + "artifact_year": 2023, + }, + "record_sets": [], + }, + sort_keys=False, + ) + ) + output_dir = tmp_path / "suite" + + with pytest.raises(ManifestAccessError): + build_source_suite(package_dir, output_dir, year=2023) + + assert not output_dir.exists() + + +# -------------------------------------------------------------------------- +# CLI +# -------------------------------------------------------------------------- + + +def test_cli_register_artifact_round_trips(tmp_path, capsys): + output_dir = tmp_path / "pkg" + + exit_code = harness_main( + [ + "register-artifact", + "--source-id", + "hmrc", + "--package-id", + "hmrc-spi-public-use-tape-2022-23", + "--year", + "2022", + "--out-dir", + str(output_dir), + "--filename", + "put2223uk.tab", + "--sha256", + FIXTURE_SHA, + "--size-bytes", + "141323762", + "--vintage", + "2022-23", + "--licence", + "UK Data Service End User Licence (study SN 9422)", + "--access", + "restricted", + "--doi", + "10.5255/UKDA-SN-9422-1", + "--hash-source", + "consumer_pin", + "--attested-by", + "PolicyEngine/microcosm", + "--pinned-from-repository", + "PolicyEngine/microcosm", + "--pinned-from-path", + "packages/microcosm-build/src/microcosm/build/uk/hmrc_income_source_stages.json", + "--pinned-from-commit", + FIXTURE_COMMIT, + ] + ) + payload = json.loads(capsys.readouterr().out) + + assert exit_code == 0 + assert payload["valid"] is True + assert payload["access"] == "restricted" + assert payload["hash_source"] == "consumer_pin" + assert payload["r2_location"] is None + assert payload["registration"] == ( + f"hmrc/hmrc-spi-public-use-tape-2022-23/2022/{FIXTURE_SHA}/put2223uk.tab" + ) + assert not (output_dir / "put2223uk.tab").exists() + entry = _manifest(output_dir)["files"][2022][0] + assert entry["pinned_from"]["commit"] == FIXTURE_COMMIT + + +def test_cli_register_artifact_rejects_public_access(tmp_path): + with pytest.raises(SystemExit): + harness_main( + [ + "register-artifact", + "--source-id", + "census_cps", + "--package-id", + "census-cps-asec-2023", + "--year", + "2023", + "--out-dir", + str(tmp_path / "pkg"), + "--filename", + "asecpub23csv.zip", + "--sha256", + FIXTURE_SHA, + "--vintage", + "2023", + "--licence", + "Public domain", + "--access", + "public", + "--hash-source", + "consumer_pin", + "--attested-by", + "x", + ] + ) + + +def test_cli_refusals_print_an_error_and_exit_1(tmp_path, monkeypatch, capsys): + output_dir = tmp_path / "pkg" + _register(output_dir) + reads = _refuse_read(monkeypatch) + + exit_code = harness_main( + [ + "fetch-artifact", + "--url", + "https://publisher.example/adult.tab", + "--source-id", + "dwp", + "--package-id", + "dwp-frs-2023-24", + "--year", + "2023", + "--out-dir", + str(output_dir), + ] + ) + captured = capsys.readouterr() + + assert exit_code == 1 + assert captured.out == "" + assert "Its bytes must not enter" in captured.err + assert reads == [] + + exit_code = harness_main( + [ + "register-artifact", + "--source-id", + "dwp", + "--package-id", + "dwp-frs-2023-24", + "--year", + "2023", + "--out-dir", + str(output_dir), + "--filename", + "adult.tab", + "--sha256", + OTHER_SHA, + "--vintage", + "2023_24", + "--licence", + "UK Data Service End User Licence", + "--access", + "licensed", + "--doi", + "10.5255/UKDA-SN-9367-2", + "--hash-source", + "consumer_pin", + "--attested-by", + "PolicyEngine/microcosm", + "--pinned-from-repository", + "PolicyEngine/microcosm", + "--pinned-from-path", + "p", + "--pinned-from-commit", + FIXTURE_COMMIT, + ] + ) + captured = capsys.readouterr() + assert exit_code == 1 + assert "pass --allow-reissue" in captured.err + + +def test_cli_fetch_archives_a_release_with_the_reviewed_identity( + tmp_path, monkeypatch, capsys +): + output_dir = tmp_path / "pkg" + staging = tmp_path / "staging" + uploads = _record_uploads(monkeypatch) + _serve(monkeypatch, PUBLIC_BYTES) + argv = [ + "fetch-artifact", + "--url", + "https://www2.census.gov/programs-surveys/cps/datasets/2023/march/asecpub23csv.zip", + "--source-id", + "census_cps", + "--package-id", + "census-cps-asec-2023", + "--year", + "2023", + "--out-dir", + str(output_dir), + "--publisher", + "U.S. Census Bureau", + "--vintage", + "2023 ASEC / 2022 income reference year", + "--access", + "public", + "--licence", + "US-Government-Work", + "--kind", + "microdata_release", + "--expected-sha256", + PUBLIC_SHA, + "--expected-size-bytes", + str(len(PUBLIC_BYTES)), + "--licence-evidence-issuer", + EVIDENCE["issuer"], + "--licence-evidence-scope", + EVIDENCE["scope"], + "--licence-evidence-url", + EVIDENCE["url"], + "--staging-dir", + str(staging), + "--upload-r2", + ] + + assert harness_main(argv) == 0 + payload = json.loads(capsys.readouterr().out) + entry = _manifest(output_dir)["files"][2023][0] + + assert payload["valid"] is True + assert payload["sha256"] == PUBLIC_SHA + assert entry["filename"] == "asecpub23csv.zip" + assert entry["vintage"] == "2023 ASEC / 2022 income reference year" + assert entry["licence_evidence"]["sha256"] == PUBLIC_SHA + assert uploads[0][0].endswith("/asecpub23csv.zip") + + _serve(monkeypatch, b"a reissue") + assert harness_main(argv) == 1 + captured = capsys.readouterr() + assert PUBLIC_SHA in captured.err + assert captured.out == "" + + +# -------------------------------------------------------------------------- +# The committed registrations +# -------------------------------------------------------------------------- + +#: Every committed hash-only pin, keyed by registration identity. Values are +#: (sha256, size_bytes, vintage, access, licence); the checksums are the +#: reviewed consumer pins and any change here is a change of identity. +GOLDEN_PINS: dict[tuple[str, str, int, str], tuple[str, int, str, str, str]] = { + ("dwp", "dwp-frs-2023-24", 2023, "accounts.tab"): ( + "c5e31932bfd06087f835d2c83c0984c85a93409bf5ef85b699cb0958abcba1ea", + 1807921, + "2023_24", + "licensed", + "UK Data Service End User Licence", + ), + ("dwp", "dwp-frs-2023-24", 2023, "adult.tab"): ( + "e09f9647d03585c81a528636028b2ed495f8f1fbcf64c5e7b4fe521b67367e06", + 35323384, + "2023_24", + "licensed", + "UK Data Service End User Licence", + ), + ("dwp", "dwp-frs-2023-24", 2023, "benefits.tab"): ( + "ff30d054cc659bcf23b44c492d98cfd701c0bfdb63e8e9aa9769b490ba9d636b", + 4460292, + "2023_24", + "licensed", + "UK Data Service End User Licence", + ), + ("dwp", "dwp-frs-2023-24", 2023, "benunit.tab"): ( + "88946815eace8561516d5cbb442c27e319c1e90abc381fb2338f0126e3b9e05b", + 21213867, + "2023_24", + "licensed", + "UK Data Service End User Licence", + ), + ("dwp", "dwp-frs-2023-24", 2023, "child.tab"): ( + "b5dc84fe8b002ee925e61fae23fed27b11537af9fb174f1d07d9cc1748b9702e", + 2913156, + "2023_24", + "licensed", + "UK Data Service End User Licence", + ), + ("dwp", "dwp-frs-2023-24", 2023, "chldcare.tab"): ( + "566e0ebca1d5e2f3e424e556c91f4cb583d17dadfdfa59feb3841eda7e5976a3", + 273837, + "2023_24", + "licensed", + "UK Data Service End User Licence", + ), + ("dwp", "dwp-frs-2023-24", 2023, "extchild.tab"): ( + "8d358d7ee66ee4a7ceab87b4f24fbbf21ac86dc038dc7831e51fb271f96a57ec", + 18677, + "2023_24", + "licensed", + "UK Data Service End User Licence", + ), + ("dwp", "dwp-frs-2023-24", 2023, "househol.tab"): ( + "5fd26b8b675f33b3b30c9ac789a18da17de734790f77e00ded287d1c3a187b30", + 12387117, + "2023_24", + "licensed", + "UK Data Service End User Licence", + ), + ("dwp", "dwp-frs-2023-24", 2023, "job.tab"): ( + "88b77ffe06865f029f713bb1d55ff12bdea8a1234de5bc293e72458fe64f3a74", + 10934873, + "2023_24", + "licensed", + "UK Data Service End User Licence", + ), + ("dwp", "dwp-frs-2023-24", 2023, "maint.tab"): ( + "f2dc924eb5a51b0c357791693d15b431327dc39c6421011efb313d88bf839695", + 15440, + "2023_24", + "licensed", + "UK Data Service End User Licence", + ), + ("dwp", "dwp-frs-2023-24", 2023, "mortgage.tab"): ( + "ce36b477d67837c469608a0d68f7ef269ac04758974235f1157d2f6b92cdbfdc", + 631783, + "2023_24", + "licensed", + "UK Data Service End User Licence", + ), + ("dwp", "dwp-frs-2023-24", 2023, "oddjob.tab"): ( + "b4ba3dd3151f73a01422983c60514a3e38458ddfa4fb33ae4ed0326873406305", + 5165, + "2023_24", + "licensed", + "UK Data Service End User Licence", + ), + ("dwp", "dwp-frs-2023-24", 2023, "penprov.tab"): ( + "ee001461c40306ec24b38b2881e1774121114266a2ee449d606cd0a811c37731", + 522313, + "2023_24", + "licensed", + "UK Data Service End User Licence", + ), + ("dwp", "dwp-frs-2023-24", 2023, "pension.tab"): ( + "150d6fad1fce81254fb7aea1526fbb00b63d4027d6e2ac4c26bb90aea3127eb7", + 1225838, + "2023_24", + "licensed", + "UK Data Service End User Licence", + ), + ("hmrc", "hmrc-spi-public-use-tape-2022-23", 2022, "put2223uk.tab"): ( + "5ef829461060c91a2a47be59ad541d9b519fc3976d66ca80d4920f711bb96f66", + 141323762, + "2022-23", + "restricted", + "UK Data Service End User Licence (study SN 9422)", + ), +} + +CONSUMER_PIN_COMMITS = { + "packages/microcosm-build/src/microcosm/build/uk/source_stages.json": ( + "2fb2e2f8a99c37725bd6e7a15ff4c2595c912b77" + ), + "packages/microcosm-build/src/microcosm/build/uk/hmrc_income_source_stages.json": ( + "de7451bd19ca46d2967e73cdf393908d29e72542" + ), +} + + +def _committed_entries(package: Path) -> list[tuple[dict, int, dict]]: + manifest = yaml.safe_load((package / "manifest.yaml").read_text()) + assert manifest["kind"] == "microdata_release" + return [ + (manifest, year, entry) + for year, entries in manifest["files"].items() + for entry in entries + ] + + +def test_committed_pins_match_the_golden_mapping_exactly(): + committed = { + (manifest["source_id"], manifest["package_id"], year, entry["filename"]): ( + entry["sha256"], + entry["size_bytes"], + entry["vintage"], + entry["access"], + entry["licence"], + ) + for package in (FRS_PACKAGE, SPI_PACKAGE) + for manifest, year, entry in _committed_entries(package) + } + + assert committed == GOLDEN_PINS + assert len(GOLDEN_PINS) == 15 + + +def test_committed_frs_registration_covers_every_pinned_tab(): + filenames = [entry["filename"] for _m, _y, entry in _committed_entries(FRS_PACKAGE)] + assert filenames == sorted( + key[3] for key in GOLDEN_PINS if key[1] == "dwp-frs-2023-24" + ) + + +@pytest.mark.parametrize("package", [FRS_PACKAGE, SPI_PACKAGE]) +def test_committed_registrations_are_consumer_pins_without_bytes(package): + entries = _committed_entries(package) + + assert entries + for _manifest_payload, _year, entry in entries: + assert entry["access"] in {"licensed", "restricted"} + assert entry["hash_source"] == "consumer_pin" + assert entry["attested_by"] == "PolicyEngine/microcosm" + assert entry["pinned_from"]["repository"] == "PolicyEngine/microcosm" + assert ( + entry["pinned_from"]["commit"] + == (CONSUMER_PIN_COMMITS[entry["pinned_from"]["path"]]) + ) + assert "verified_at" not in entry + assert "storage" not in entry + # No bytes accompany a hash-only registration. + assert not (package / entry["filename"]).exists() + assert sorted(path.name for path in package.iterdir()) == ["manifest.yaml"] + + +def test_committed_registrations_pass_inventory_with_the_golden_identities(): + report = inventory_source_artifacts(REPO_ROOT / "db" / "data") + hash_only = [entry for entry in report.entries if entry.hash_only] + + assert report.valid + assert all(entry.valid and not entry.exists for entry in hash_only) + assert all(entry.r2 is None for entry in hash_only) + identities = { + ( + yaml.safe_load(Path(entry.manifest_path).read_text())["source_id"], + yaml.safe_load(Path(entry.manifest_path).read_text())["package_id"], + int(entry.year), + entry.filename, + ): (entry.sha256_expected, entry.size_bytes) + for entry in hash_only + } + assert identities == { + key: (value[0], value[1]) for key, value in GOLDEN_PINS.items() + } + + +def test_no_committed_registration_computes_an_r2_key(): + for package in (FRS_PACKAGE, SPI_PACKAGE): + text = (package / "manifest.yaml").read_text() + assert "storage:" not in text + assert "ledger-raw" not in text + assert "r2://" not in text + + +def test_registration_hashes_are_not_hashes_of_anything_chronicle_holds(): + # A registered checksum identifies publisher bytes Chronicle never sees; + # it must never coincide with the hash of the manifest that records it. + for package in (FRS_PACKAGE, SPI_PACKAGE): + manifest_hash = hashlib.sha256( + (package / "manifest.yaml").read_bytes() + ).hexdigest() + assert manifest_hash not in (package / "manifest.yaml").read_text() diff --git a/tests/test_chronicle_mirror.py b/tests/test_chronicle_mirror.py index 787429c1..6b2e694d 100644 --- a/tests/test_chronicle_mirror.py +++ b/tests/test_chronicle_mirror.py @@ -4,6 +4,9 @@ import json +import pytest + +from chronicle.env import ChronicleEnvDeprecationWarning from chronicle.harness import main as harness_main from chronicle.mirror import ( LEDGER_MIRROR_TABLES, @@ -18,7 +21,7 @@ def test_export_chronicle_db_tables_writes_jsonl_and_manifest(tmp_path): - db_path = tmp_path / "ledger.db" + db_path = tmp_path / "chronicle.db" output_dir = tmp_path / "mirror" build_chronicle_db( build_soi_table_1_1_facts(2023), @@ -53,7 +56,7 @@ def test_export_chronicle_db_tables_writes_jsonl_and_manifest(tmp_path): def test_export_chronicle_db_tables_orders_rows_deterministically(tmp_path): - db_path = tmp_path / "ledger.db" + db_path = tmp_path / "chronicle.db" first_output_dir = tmp_path / "mirror-first" second_output_dir = tmp_path / "mirror-second" build_chronicle_db( @@ -72,7 +75,7 @@ def test_export_chronicle_db_tables_orders_rows_deterministically(tmp_path): def test_export_db_tables_cli_emits_manifest_summary(tmp_path, capsys): - db_path = tmp_path / "ledger.db" + db_path = tmp_path / "chronicle.db" output_dir = tmp_path / "mirror" build_chronicle_db( build_soi_table_1_1_facts(2023), @@ -99,7 +102,7 @@ def test_export_db_tables_cli_emits_manifest_summary(tmp_path, capsys): def test_load_supabase_mirror_dry_run_counts_exported_rows(tmp_path): - db_path = tmp_path / "ledger.db" + db_path = tmp_path / "chronicle.db" output_dir = tmp_path / "mirror" build_chronicle_db( build_soi_table_1_1_facts(2023), @@ -189,6 +192,127 @@ def test_load_supabase_mirror_cli_dry_run(tmp_path, capsys): assert payload["table_count"] == len(LEDGER_MIRROR_TABLES) +# --------------------------------------------------------------------------- +# Schema configuration +# +# The mirror loader is the primary writer into the hosted schema, so it is the +# call site CHRONICLE_SCHEMA has to reach (PolicyEngine/chronicle#143, +# mechanism 3). It defaulted to the literal "ledger" while only the read-side +# client honored the renamed variable, which would have sent a rehearsal load +# into production the moment an operator set it. +# --------------------------------------------------------------------------- + + +def _empty_mirror(tmp_path): + mirror_dir = tmp_path / "mirror" + mirror_dir.mkdir() + for table in LEDGER_MIRROR_TABLES: + (mirror_dir / f"{table}.jsonl").write_text("") + return mirror_dir + + +def _one_build_artifact(tmp_path): + path = tmp_path / "build_artifacts.jsonl" + path.write_text( + json.dumps( + { + "build_artifact_key": "ledger.build_artifact.v1:test", + "build_id": "ledger.build.v1:test", + "artifact_kind": "json", + "artifact_name": "reports/build_summary.json", + "sha256": "abc", + "size_bytes": 3, + "r2_bucket": "ledger-derived", + "r2_key": "derived/test", + "r2_uri": "r2://ledger-derived/derived/test", + }, + sort_keys=True, + ) + + "\n" + ) + return path + + +def _load_into_fake_client(tmp_path, **kwargs): + client = _FakeSupabaseClient() + report = load_supabase_mirror( + _empty_mirror(tmp_path), + table_paths={"build_artifacts": _one_build_artifact(tmp_path)}, + client=client, + **kwargs, + ) + return report, client + + +def test_load_supabase_mirror_defaults_to_the_ledger_schema(tmp_path): + report, client = _load_into_fake_client(tmp_path) + + assert report.schema == "ledger" + assert [upsert[0] for upsert in client.upserts] == ["ledger"] + + +def test_load_supabase_mirror_writes_to_the_chronicle_schema(tmp_path, monkeypatch): + """The renamed variable configures the writer, not just the reader.""" + monkeypatch.setenv("CHRONICLE_SCHEMA", "chronicle_probe") + + report, client = _load_into_fake_client(tmp_path) + + assert report.schema == "chronicle_probe" + assert [upsert[0] for upsert in client.upserts] == ["chronicle_probe"] + + +@pytest.mark.parametrize("name", ["POLICYENGINE_LEDGER_SCHEMA", "LEDGER_SCHEMA"]) +def test_load_supabase_mirror_honors_a_ledger_era_schema_name( + tmp_path, monkeypatch, name +): + monkeypatch.setenv(name, "legacy_probe") + + with pytest.warns(ChronicleEnvDeprecationWarning): + report, client = _load_into_fake_client(tmp_path) + + assert report.schema == "legacy_probe" + assert [upsert[0] for upsert in client.upserts] == ["legacy_probe"] + + +def test_an_explicit_schema_still_wins_over_the_environment(tmp_path, monkeypatch): + monkeypatch.setenv("CHRONICLE_SCHEMA", "chronicle_probe") + + report, client = _load_into_fake_client(tmp_path, schema="explicit_probe") + + assert report.schema == "explicit_probe" + assert [upsert[0] for upsert in client.upserts] == ["explicit_probe"] + + +def test_load_supabase_mirror_cli_writes_to_the_configured_schema( + tmp_path, monkeypatch, capsys +): + """The CLI resolves the same way when no --schema is supplied.""" + client = _FakeSupabaseClient() + monkeypatch.setattr("chronicle.mirror._get_supabase_client", lambda: client) + monkeypatch.setenv("CHRONICLE_SCHEMA", "chronicle_probe") + argv = [ + "load-supabase-mirror", + "--dir", + str(_empty_mirror(tmp_path)), + "--build-artifacts", + str(_one_build_artifact(tmp_path)), + ] + + exit_code = harness_main(argv) + payload = json.loads(capsys.readouterr().out) + + assert exit_code == 0 + assert payload["schema"] == "chronicle_probe" + assert [upsert[0] for upsert in client.upserts] == ["chronicle_probe"] + + assert harness_main([*argv, "--schema", "explicit_probe"]) == 0 + assert json.loads(capsys.readouterr().out)["schema"] == "explicit_probe" + assert [upsert[0] for upsert in client.upserts] == [ + "chronicle_probe", + "explicit_probe", + ] + + class _FakeSupabaseClient: def __init__(self): self.upserts = [] diff --git a/tests/test_chronicle_namespace.py b/tests/test_chronicle_namespace.py index 9b0ffd41..5dad695e 100644 --- a/tests/test_chronicle_namespace.py +++ b/tests/test_chronicle_namespace.py @@ -8,11 +8,7 @@ query_targets, ) from db.schema import Target as DbTarget -from db.supabase_client import ( - LEDGER_SCHEMA, - TARGETS_SCHEMA, - query_targets as db_query_targets, -) +from db.supabase_client import query_targets as db_query_targets def test_chronicle_targets_reexport_schema_objects(): @@ -29,8 +25,30 @@ def test_chronicle_client_reexports_supabase_client(): def test_chronicle_supabase_schema_boundaries_are_defaulted(): - assert LEDGER_SCHEMA == "ledger" - assert TARGETS_SCHEMA == "targets" + """The schema names resolve per call, so this reads the cleared window. + + ``db.supabase_client`` is imported at collection, before any fixture runs. + Resolving the schema there — as an import-time constant — would bind an + operator's ``CHRONICLE_SCHEMA`` (or a ledger-era name, warning as it went) + into the module for the whole session, and no fixture could take it back. + Reading at call time is what makes this a test of the defaults rather than + of the shell. + """ + from db import supabase_client + + assert supabase_client.chronicle_schema() == "ledger" + assert supabase_client.targets_schema() == "targets" + + +def test_chronicle_supabase_schema_follows_the_environment(monkeypatch): + """The renamed variable reaches the client after it has been imported.""" + from db import supabase_client + + monkeypatch.setenv("CHRONICLE_SCHEMA", "chronicle_probe") + monkeypatch.setenv("POLICYENGINE_TARGETS_SCHEMA", "targets_probe") + + assert supabase_client.chronicle_schema() == "chronicle_probe" + assert supabase_client.targets_schema() == "targets_probe" def test_chronicle_normalization_exports_helpers(): diff --git a/tests/test_chronicle_suite.py b/tests/test_chronicle_suite.py index 6911d6e2..c08f8abe 100644 --- a/tests/test_chronicle_suite.py +++ b/tests/test_chronicle_suite.py @@ -96,7 +96,7 @@ def test_build_source_suite_writes_artifacts_and_reports(tmp_path): assert (output_dir / "source_regions.jsonl").exists() assert (output_dir / "facts.jsonl").exists() assert (output_dir / "consumer_facts.jsonl").exists() - assert (output_dir / "ledger.db").exists() + assert (output_dir / "chronicle.db").exists() assert (output_dir / "datapackage.json").exists() assert (output_dir / "ro-crate-metadata.json").exists() assert (output_dir / "reports" / "source_regions.json").exists() @@ -124,7 +124,7 @@ def test_build_source_suite_writes_artifacts_and_reports(tmp_path): "source_regions.jsonl", "facts.jsonl", "consumer_facts.jsonl", - "ledger.db", + "chronicle.db", "reports/build_summary.json", "reports/source_regions.json", "reports/selectors.json", @@ -150,7 +150,7 @@ def test_build_source_suite_writes_artifacts_and_reports(tmp_path): "concept_alignment_validation_skipped" ] - with sqlite3.connect(output_dir / "ledger.db") as connection: + with sqlite3.connect(output_dir / "chronicle.db") as connection: facts_count = connection.execute( "SELECT COUNT(*) FROM aggregate_facts" ).fetchone()[0] @@ -184,7 +184,7 @@ def test_build_source_suite_supports_soi_table_1_4(tmp_path): "concept_alignment_validation_skipped" ) assert (output_dir / "source_regions.jsonl").exists() - assert (output_dir / "ledger.db").exists() + assert (output_dir / "chronicle.db").exists() def test_agent_acceptance_accepts_aggregate_income_range_source_rows(): @@ -739,7 +739,7 @@ def test_build_suite_cli_emits_json_summary(tmp_path, capsys): assert payload["outputs"]["source_regions"] == str( output_dir / "source_regions.jsonl" ) - assert payload["outputs"]["database"] == str(output_dir / "ledger.db") + assert payload["outputs"]["database"] == str(output_dir / "chronicle.db") assert payload["outputs"]["consumer_facts"] == str( output_dir / "consumer_facts.jsonl" )