Operational rename, slice 1: CHRONICLE_* env names, chronicle.db output, configurable buckets, cutover checklist (#143, mechanism 3)

[MaxGhenis]

Summary

First slice of #143 mechanism 3 (operational stores migrate by dual-run). Code and docs only; no infrastructure changes and no defaults flipped.

• chronicle/env.py — one shared reader for the rename window: CHRONICLE_<X> is read first, then POLICYENGINE_LEDGER_<X>, then LEDGER_<X>, with a once-per-process FutureWarning (so CLI operators actually see it) naming the preferred variable. Replaces the three ad-hoc _env helpers in db/supabase_client.py, chronicle/source_package.py, and db/pe_source_inventory.py. Names outside the window are read literally.
• R2 bucket names are configurable (CHRONICLE_R2_RAW_BUCKET, CHRONICLE_R2_DERIVED_BUCKET, plumbed through fetch-artifact, publish-raw, publish-derived, bootstrap-r2 and their --r2-bucket defaults). Defaults stay ledger-raw / ledger-derived; the flip is a follow-up PR once consumers repoint.
• chronicle.db for new suite outputs; ledger.db still read and inferred.
• Docs: docs/storage-architecture.md gains an "Environment Variable Rename Window" section (the previous sentence had the fallback direction backwards) and a "Bucket Cutover" checklist; README and the harness doc swept.

Two defects found and fixed on the way, each with a regression test that passes on main's behavior and fails on the bug:

• db/cli.py raised ImportError on every db CLI subcommand (a private helper import that pytest never exercised).
• The consumer-fact boundary guard matched the hard-coded ledger-derived: / r2://ledger-derived/ strings, so the bucket rename would have made it stop firing silently. It now matches on shape (*-derived bucket or derived/ key prefix).

Out of scope, deliberately: the ledger CLI alias, Supabase schema and mirror table names, governance role ids and concept authorities (hashed → mechanism 1), hash domains and schema ids (the epoch PR).

Note for the cutover checklist: this branch's count of manifest-declared raw objects differs from the one on #143 (the checklist tells the operator to recount rather than trust a number). The backfill performed today used the tracked manifests on main: 157 distinct keys, all now present in chronicle-raw (#225 has the integrity findings).

Review fixes

Gate round 1

Both findings are applied on this branch.

[high] fetch-artifact could attach a recorded R2 URI to new bytes. The preserve rule keyed on the bucket, so a repeated fetch that did not re-upload into the same bucket kept the recorded storage.r2 block while rewriting the entry's sha256/size_bytes. Reproduced against this branch's parent by serving two different bodies from one URL: the entry ends up declaring the fetched bytes' sha256 under a key addressed by the superseded bytes' one, both when the fetch only registers the bytes and when the bucket default has moved — the shape #225 hit when the IRS re-published the 2022 IRA tables.

The rule now keys on identity: the recorded key's {sha256}/{filename} tail against the fetched bytes.

• Identical — the recorded block is preserved exactly, whichever bucket is configured now.
• Different — SourceArtifactRevisionError, raised before the cached artifact or its manifest entry is touched and before anything is uploaded, naming recorded and fetched sha256/size_bytes and the ADR rule that same vintage plus new bytes is a new release_revision.
• --record-revision opts in: the fetched bytes get their own content-addressed key under the configured bucket — never the old key — and the superseded block moves to storage.previous_r2 with its sha256, size_bytes and fetched_at, so the earlier bytes stay addressable at the URI archived witness records pin.
• publish-raw applies the same check before treating a recorded block as history: a local file the recorded object does not hold is refused as recorded_r2_identity_mismatch with nothing uploaded.

storage.previous_r2 is a sibling key rather than a new shape for storage.r2, because every reader — inventory-artifacts, publish-raw, source_package._artifact_content, the suite's raw-R2-link acceptance check — reads storage.r2 alone, and publish-raw already spreads the rest of the storage block when it writes back, so a revision survives publication untouched. All 180 tracked manifest entries carrying a storage.r2 block are content-addressed and agree with their declared sha256 and filename, so the check never fires on tracked data.

[low] Env isolation was scoped to one module. The autouse fixture moves to tests/conftest.py and clears all three prefixes for every test. db.supabase_client resolves LEDGER_SCHEMA at import — during collection, before any fixture runs — so tests/test_chronicle_namespace.py re-imports it under the cleared environment instead of asserting the constant it bound at collection time.

New tests in tests/test_chronicle_artifacts.py (fake downloader, no network): a repeated fetch of identical bytes preserves the block; a repeated fetch of different bytes is refused on all three routes (re-uploaded, registered without an upload, after the bucket rename) with the cache, manifest and upload log untouched; --record-revision records the new key under the configured bucket and retains the previous object; a revised manifest still inventories as one R2-linked artifact; publish-raw refuses a mismatched local file and publishes a registered revision; and the CLI exits 1 with the message, then 0 with the flag. tests/test_chronicle_env.py asserts no rename-window variable reaches a test.

Verification on the fix head: uv run pytest -q green; CHRONICLE_R2_RAW_BUCKET=zzz CHRONICLE_SCHEMA=zzz uv run pytest -q green (five tests failed this way before the shared fixture); uv run ruff check . clean.

Gate round 2

Seven findings, all applied on this branch. Each has a regression test, and each was first reproduced against the round-1 head (34d1d0f) — the pre-fix behavior quoted under each item is that run's output, not an inference.

[high] CHRONICLE_SCHEMA did not configure the primary Supabase mirror writer. chronicle/mirror.py, its harness wrapper and the --schema CLI default all defaulted to the literal ledger, so setting the variable to rehearse a cutover moved the read-side client and left the writer pointed at production. chronicle/env.py now owns default_chronicle_schema() — one home for the CHRONICLE_SCHEMA → POLICYENGINE_LEDGER_SCHEMA → LEDGER_SCHEMA → "ledger" ladder — and the loader resolves through it whenever no schema is supplied. An explicit --schema still wins, and the default is unchanged. Pre-fix: with CHRONICLE_SCHEMA=chronicle_probe, load_supabase_mirror still reports schema='ledger'.

[high] The derived-fact boundary was not rename-safe. chronicle/consumer_contract.py matched the .ledger_derived suffix literally. It now matches the whole final dot-segment against both ledger_derived and chronicle_derived, so a producer that renames its derived rows cannot walk a downstream target fact through the guard. Pre-fix: '.chronicle_derived'.endswith('.ledger_derived') is False — the guard never fired.

[high] fetch-artifact could not address a package's non-default manifest. Seven tracked packages keep a manifest_*_source_package.yaml, and three publisher directories keep two of them: db/data/irs_soi/ira_contributions/ holds the traditional manifest beside the Roth one. A fetch that always wrote manifest.yaml would write a third manifest neither package reads, so the IRA revision workflow the docs cite would never see the recorded block. --manifest <filename> selects it (default manifest.yaml); the name must be a filename inside --out-dir, not a path. Pre-fix: fetch_source_artifact() rejects manifest_filename as an unexpected keyword, and a fetch into that directory writes manifest.yaml.

[high] Revision protection vanished when the entry had no storage.r2. A fetch that only registered bytes, or one whose upload failed — the state #225 landed in — left an entry with a declared sha256 and no recorded block, and the guard skipped it entirely. A manifest entry identifies its bytes from the moment it is registered, so the comparison is now against the entry's recorded identity: the recorded key's {sha256}/{filename} once published, the declared sha256 before that. An ordinary fetch of different bytes over either is refused unless --record-revision opts in; a revision over a never-published entry supersedes nothing and gets no empty previous_r2. Pre-fix: re-fetching different bytes over a registered entry rewrote its sha256 with no refusal.

[medium] Recorded-R2 validation read the key or the uri, whichever came first. _validated_recorded_r2 now cross-checks every supplied locator field against every other — key against the URI's path, bucket against its authority, provider against its scheme — and the resulting key against the content-addressed {sha256}/{filename} shape. A contradiction raises RecordedR2LocatorError at fetch time and refuses as recorded_r2_locator_invalid at publish time, with nothing uploaded. Pre-fix: a block whose key and uri named different objects was preserved verbatim — key sha c63744a4… beside uri sha 1e9b3fdb… — so the entry kept publishing a URI for bytes it no longer described.

[medium] A malformed manifest read as an absent one. _read_manifest now refuses a document that parses as anything but a mapping, and an unparseable one, with MalformedManifestError — raised before the publisher is read at all, so a refusal costs no fetch. inventory-artifacts and publish-raw report it as a manifest error instead of crashing on .get. An absent or empty document still reads as an empty mapping. Pre-fix: a list-valued manifest.yaml was overwritten by the fetch.

[low] Schema resolution happened at import, during collection. db.supabase_client resolved both schemas into module constants when it was first imported — at collection, before any fixture — so a legacy variable could warn and seed state that no fixture could take back. Both are now functions (chronicle_schema(), targets_schema()), and tests/conftest.py additionally strips the rename window in pytest_configure, before collection imports anything. Pre-fix: importing under LEDGER_SCHEMA=zzz bound LEDGER_SCHEMA='zzz' and emitted a FutureWarning.

All four refusals share a SourceArtifactManifestError base, so fetch-artifact reports every one as exit 1 with nothing written.

New round-2 tests: tests/test_chronicle_mirror.py (the loader and its CLI write to the configured schema, under each name in the window, with an explicit --schema still winning); tests/test_chronicle_artifacts.py (a two-manifest package addressed by name and its revision refused in the right manifest, on the direct and CLI paths; a manifest name that tries to leave the package; revision protection over a registered entry and over a failed upload; every locator contradiction, at fetch and at publish time; malformed manifests refused before the fetch and reported by both sweeps); tests/test_chronicle_consumer_contract.py (the chronicle spelling rejected identically, and the marker matched as a whole segment); tests/test_chronicle_namespace.py and tests/test_chronicle_env.py (schema read per call, no constant frozen at import).

Verification on the round-2 head (ea67f0a): uv run pytest -q — 839 passed, 1 skipped; LEDGER_SCHEMA=zzz uv run pytest -q -W error::FutureWarning tests/test_chronicle_namespace.py — 6 passed; CHRONICLE_R2_RAW_BUCKET=zzz CHRONICLE_SCHEMA=zzz uv run pytest -q — 839 passed, 1 skipped; uv run ruff check . clean; uv run ruff format --check . clean for every file this branch touches (the 13 unformatted files are pre-existing on main and none are touched here).

Scope is unchanged: no boundary change, nothing under releases/ or db/data manifests, defaults still ledger-raw / ledger-derived / ledger, and the shared functions #227 will rebase onto keep their signatures with the new arguments defaulted.

Chronicle Governance

This PR does not touch the source-data boundary: no source packages, facts, profiles, or schemas change. Storage configuration, CLI defaults, and docs only.

Tests

tests/test_chronicle_env.py (new, hermetic: chronicle-first precedence, legacy fallback with one warning per process, out-of-window names read literally), tests/test_chronicle_artifacts.py (bucket configuration, chronicle.db output with ledger.db read-back, the db/cli.py regression), tests/test_chronicle_consumer_contract.py (boundary guard by shape). CI runs the full suite on this PR; the lane that authored the branch ran out of budget before pasting its local run, so treat CI here as the verification of record.

🤖 Generated with Claude Code

[MaxGhenis]

Fable+Sol gate: changes requested, and the high finding is exactly the failure mode today's backfill hit in the wild (IRS re-published the 2022 IRA tables with different bytes under the same URL, #225):

1. [high] fetch-artifact can attach the recorded R2 URI to new bytes. When the recorded storage.r2 block is preserved on a repeated fetch, the manifest is rewritten with the newly fetched sha256 and size_bytes but keeps the old content-addressed key and URI. A publisher revision then produces a manifest whose provenance is false. Fix: compare the recorded identity (filename + sha256) with the fetched bytes before preserving anything; identical → preserve the recorded block; different → refuse, or record a new content-addressed key under the configured bucket as a new revision, never the old key. The regression test must repeat with different bytes, not identical ones.
2. [low] Env isolation is scoped to one test module. Other tests assert default buckets/schemas without clearing CHRONICLE_R2_RAW_BUCKET / CHRONICLE_SCHEMA; move isolation to a shared fixture or clear explicitly where defaults are asserted.

A fix lane is applying both on this branch; re-gate after.

[MaxGhenis]

Gate round 2: changes requested again, all agreed and being fixed on this branch (round-2 lane): CHRONICLE_SCHEMA did not reach the primary Supabase mirror writer (harness/mirror still default to the literal ledger schema); the .ledger_derived suffix check is not rename-safe; fetch-artifact cannot address a package's non-default manifest file (--manifest added), so the IRA revision workflow the docs cite would miss the recorded block; revision protection vanished when an entry lacked storage.r2; recorded-R2 locator fields must be cross-checked, not key-or-URI; malformed manifests must be rejected rather than treated as absent; collection-time schema resolution must be lazy. Re-gate follows.