Operational rename, slice 1: CHRONICLE_* env names, chronicle.db output, configurable buckets, cutover checklist (#143, mechanism 3)

PROGRESS.md

@@ -1,51 +1,190 @@
-# Lane C5 progress
+# Operational rename, slice 1 (chronicle#143, mechanism 3)
+
+Lane C5's handoff notes previously lived here; its durable record is
+`LANE_C5_REPORT.md`. This file now tracks the active lane on this branch.
 
 ## State
 
-- Branch: `be-2025-vintages` from `origin/main` at `5c15bfd`.
-- Worktree inputs are staged under `.lane-raw/` and must remain uncommitted.
-- Lane C5 is complete, validated, independently reviewed, and ready for handoff.
-- The requested staged C2 report is absent, but root `LANE_C2_REPORT.md` is byte-identical
-  to the sibling lane's staged copy (SHA-256 `4590e0dc...50f06e7`) and is the pattern used.
+- Branch: `ops-rename-slice1`, cut from `origin/main` at `ff3efd3`.
+- Scope: env names, R2 bucket configurability, `ledger.db` -> `chronicle.db`,
+  and the docs for all three. Code and docs only; no infrastructure changes.
+- Out of scope and deliberately untouched: the `ledger` console-script alias,
+  the Supabase `"ledger"` schema and mirror table names, governance role ids and
+  concept authorities, hash domains and schema ids, anything under `releases/`.
+- This PR does not touch the source-data boundary. No package spec, parser,
+  selector, manifest, or fact value changes.
 
 ## Done
 
-- Read the repository Chronicle boundary rules in `AGENTS.md`.
-- Read `.lane-raw/SOURCES.md` and confirmed all five named publisher artifacts are present.
-- Confirmed the worktree is otherwise clean apart from `.lane-raw/` and the shared `.venv` link.
-- Verified all five staged artifact SHA-256 pins exactly.
-- Mapped FPB workbook cells: 990 facts across T01/T06/T07/T11/T17/T24, with
-  2022–2025 observations and 2026–2031 `source_projection` facts.
-- Confirmed PDF boundary evidence: printed page 19 calls 2026 the first projection year;
-  annex table units appear on printed pages 45, 48, 49, 53, 58, and 65.
-- Chosen Eurostat layout: two vintage-specific source-package aliases share new manifest
-  entries, preserving the prior package YAMLs, raw bytes, and fact outputs unchanged.
-- Reproduced the Statbel curator logic: 18 NUTS1 × sex × age-band cells totaling 11,825,551.
-- Added the hash-pinned FPB workbook and publication PDF plus the
-  `fpb-economic-outlook-2026-2031-june-2026` package alias.
-- Built 990 line-specific publisher facts (99 per year): 396 observations for
-  2022–2025 and 594 `source_projection` facts for 2026–2031.
-- Passed FPB `validate-package` and `build-suite`: 990 facts, full cell lineage,
-  zero acceptance errors, and pinned 2025 cells 320578 / 77771 / 5602 million euro.
-- Re-ran the Statbel 2026 curator logic on the 2025 ZIP and added the hash-pinned
-  raw capture plus its deterministic 18-row curated CSV.
-- Passed Statbel 2025 `validate-package` and `build-suite`: 18 facts totaling
-  11,825,551, 66 constraints, full lineage, and zero acceptance errors.
-- Added the Eurostat `gov_10a_taxag` 2025 and `spr_exp_func` 2024 manifest
-  entries plus vintage-specific package aliases, without modifying either
-  prior artifact or prior package specification.
-- Passed both new Eurostat package validations and suite builds: 12 tax facts
-  and 9 ESSPROS facts, full lineage, and zero acceptance errors.
-- Extended Belgium and Eurostat regressions for FPB table counts/cells and
-  assertion boundary, vintage non-overlap, prior-output digests, Statbel pins,
-  and the declared 0.25% Statbel/FPB population comparison tolerance.
-- Passed 43 focused tests and the full merged-bundle regression: 157,177 facts,
-  148 packages, zero aggregate-key duplicates, and expected goldens throughout.
-- Recorded pins, counts, boundary evidence, curator commands, validation tails,
-  and consumer fact families in `LANE_C5_REPORT.md`.
-- Passed independent `ledger-source-fidelity` and `ledger-boundary` reviews with
-  no required corrections.
+- Read `AGENTS.md`, `docs/storage-architecture.md`,
+  `docs/agent-source-package-harness.md`, and the mechanism-3 migration spec in
+  the first comment of PolicyEngine/chronicle#143.
+- Enumerated every ledger-named env read in tracked Python: the four real
+  variables (`LEDGER_SOURCE_ARTIFACT_CACHE_DIR`, `LEDGER_SOURCE_ARTIFACT_FETCH`,
+  `LEDGER_PE_US_DATA_ROOT`, `LEDGER_PE_UK_DATA_ROOT`) plus
+  `POLICYENGINE_LEDGER_SCHEMA`. `LEDGER_MIRROR_TABLES`,
+  `LEDGER_MIRROR_PRIMARY_KEYS`, and `LEDGER_DB_SCHEMA_VERSION` are module
+  constants, not env reads, and name out-of-scope surfaces.
+- Added `chronicle/env.py`: one shared `env_value`/`env_flag`/`env_names`
+  helper reading `CHRONICLE_<X>` first, then `POLICYENGINE_LEDGER_<X>` and
+  `LEDGER_<X>` with a once-per-process `ChronicleEnvDeprecationWarning` naming
+  the preferred variable.
+- Replaced all three ad-hoc helpers (`db/supabase_client._env`,
+  `chronicle/source_package._env_value`/`_truthy_env`,
+  `db/pe_source_inventory._env_value`) with the shared helper.
+- Made the R2 bucket names configurable via `CHRONICLE_R2_RAW_BUCKET` and
+  `CHRONICLE_R2_DERIVED_BUCKET`, plumbed through fetch-artifact, publish-raw,
+  publish-derived and bootstrap-r2. Defaults unchanged at `ledger-raw` and
+  `ledger-derived`. Both manifest write paths now preserve a recorded
+  `storage.r2` block instead of restating it under a renamed bucket.
+- Emitted `chronicle.db` for new suite outputs, with `ledger.db` still accepted
+  on read and on derived-artifact kind inference.
+- Added `tests/test_chronicle_env.py` plus artifact tests: 75 hermetic tests
+  covering the lookup ladder, precedence, the once-per-process warning, and every
+  real call site.
+- Swept the docs. `docs/storage-architecture.md` gained an "Environment Variable
+  Rename Window" section (the old text stated the fallback direction backwards)
+  and a "Bucket Cutover" section; `docs/agent-source-package-harness.md` and
+  `README.md` follow. Verified 186 distinct `ledger-raw` objects across 154
+  tracked manifest files, every key content-addressed by sha256.
+
+## Review fixes (gate round 1)
+
+The Fable+Sol gate requested changes; both findings are applied on this branch.
+
+- **[high] `fetch-artifact` could attach a recorded R2 URI to new bytes.** The
+  preserve rule keyed on the bucket, so a repeated fetch that did not re-upload
+  into the same bucket kept the recorded `storage.r2` block while rewriting the
+  entry's `sha256`/`size_bytes`. Reproduced against this branch's parent by
+  serving two different bodies from one URL: the entry ends up declaring the
+  fetched bytes' `sha256` under a key addressed by the superseded bytes' one,
+  both when the fetch only registers the bytes and when the bucket default has
+  moved.
+  The rule now keys on identity — the recorded key's `{sha256}/{filename}` tail
+  against the fetched bytes. Identical preserves the block exactly; different
+  raises `SourceArtifactRevisionError` before the cached artifact or its
+  manifest entry is touched, naming recorded and fetched `sha256`/`size_bytes`
+  and the ADR rule that same vintage plus new bytes is a new release revision.
+  `--record-revision` opts in: the new bytes get their own content-addressed key
+  under the configured bucket, never the old key, and the superseded block moves
+  to `storage.previous_r2`. `publish-raw` applies the same check before treating
+  a recorded block as history (`recorded_r2_identity_mismatch`, nothing
+  uploaded).
+- **[low] Env isolation was scoped to one module.** The autouse fixture moved to
+  `tests/conftest.py` and now clears all three prefixes for every test.
+  `db.supabase_client` resolves `LEDGER_SCHEMA` at import — during collection,
+  before any fixture — so `tests/test_chronicle_namespace.py` re-imports it
+  under the cleared environment instead of asserting the constant it bound at
+  collection time.
+
+`storage.previous_r2` is a sibling key, chosen because every reader
+(`inventory-artifacts`, `publish-raw`, `source_package._artifact_content`, the
+suite's raw-R2-link acceptance check) reads `storage.r2` alone, and
+`publish-raw` already spreads the rest of the `storage` block when it writes
+back, so a revision survives publication untouched. All 180 tracked manifest
+entries that carry a `storage.r2` block are content-addressed and agree with
+their declared `sha256` and `filename`, so the identity check never fires on
+tracked data.
+
+## Review fixes (gate round 2)
+
+The second Fable+Sol gate requested changes again. Seven findings, each fixed
+with a regression test on this branch. Plan, in dependency order:
+
+1. **[high] `CHRONICLE_SCHEMA` does not reach the Supabase mirror writer.**
+   `chronicle/harness.py` and `chronicle/mirror.py` default the schema to the
+   literal `"ledger"`; only `db.supabase_client` reads the renamed variable.
+   Resolve through the shared helper whenever no explicit `--schema` is given.
+2. **[high] The derived-fact boundary check is not rename-safe.**
+   `chronicle/consumer_contract.py` matches the `.ledger_derived` suffix only.
+3. **[high] `fetch-artifact` cannot address a package's non-default manifest.**
+   Seven tracked packages keep a `manifest_*_source_package.yaml`; three
+   directories keep two. A fetch into one of them writes a third manifest and
+   never sees the recorded block.
+4. **[high] Revision protection vanishes when the entry has no `storage.r2`.**
+5. **[medium] Recorded-R2 locator fields must be cross-checked**, not read as
+   key-or-URI, before a block is preserved or published.
+6. **[medium] `_read_manifest` must reject a malformed document**, not treat a
+   non-mapping YAML payload as an absent manifest.
+7. **[low] Schema resolution must be lazy** so no legacy variable is read at
+   collection, before the autouse isolation fixture runs.
+
+## State (round 2)
+
+- Read both gate rounds on PolicyEngine/chronicle#226 and the code each finding
+  names.
+- Scanned all 154 tracked manifest files (187 `files` entries, every one
+  carrying `storage.r2`): every recorded block supplies provider, bucket, key
+  and uri; every key is content-addressed; every declared `sha256`/`filename`
+  agrees with its key tail; no `uri` contradicts its `key`. Strict locator
+  validation therefore refuses nothing that is tracked today.
+- All seven findings are applied, each with a regression test, and each
+  reproduced against this branch's previous head (`34d1d0f`) first.
+
+### What each fix does
+
+1. `chronicle/env.py` gains `default_chronicle_schema()`: one home for the
+   `CHRONICLE_SCHEMA` -> `POLICYENGINE_LEDGER_SCHEMA` -> `LEDGER_SCHEMA` ->
+   `"ledger"` ladder. `load_supabase_mirror`, its harness wrapper and the
+   `--schema` CLI default all resolve through it when no schema is supplied;
+   an explicit `--schema` still wins. Defaults unchanged.
+2. `chronicle/consumer_contract.py` matches the whole final dot-segment of a
+   `source_record_id` against both `ledger_derived` and `chronicle_derived`.
+3. `fetch-artifact --manifest <filename>` selects which of a package's
+   manifests the entry belongs to (default `manifest.yaml`); the name must be
+   a filename inside `--out-dir`.
+4. Revision protection now compares against the entry's recorded identity --
+   the recorded key's `{sha256}/{filename}` once published, the declared
+   `sha256` before that -- so a registered-but-unpublished entry, or one whose
+   upload failed, is protected exactly like a published one.
+5. `_validated_recorded_r2` cross-checks every supplied locator field against
+   every other and against the content-addressed key shape. A contradiction is
+   `RecordedR2LocatorError` at fetch time and `recorded_r2_locator_invalid` at
+   publish time, never a preserved block.
+6. `_read_manifest` refuses a non-mapping or unparseable document
+   (`MalformedManifestError`) before the publisher is read at all;
+   `inventory-artifacts` and `publish-raw` report it instead of crashing.
+7. `db.supabase_client` resolves both schemas per call rather than at import,
+   and `tests/conftest.py` strips the rename window in `pytest_configure`, so
+   no module can read or warn from an operator's shell during collection.
+
+All four refusals share a `SourceArtifactManifestError` base, so the
+`fetch-artifact` CLI reports every one as exit 1 with nothing written.
+
+### Reproduced against `34d1d0f` (the round-1 head)
+
+Running the same operations against a checkout of the previous head:
+
+1. `load_supabase_mirror` default `schema='ledger'`; with
+   `CHRONICLE_SCHEMA=chronicle_probe` the load still reports `schema='ledger'`.
+2. `'.chronicle_derived'.endswith('.ledger_derived')` is False: the boundary
+   never fired for the chronicle spelling.
+3. `fetch_source_artifact()` rejects `manifest_filename` as an unexpected
+   keyword; a fetch into `ira_contributions/` writes `manifest.yaml`.
+4. A fetch of different bytes over a registered (unpublished) entry was
+   accepted silently: the entry's `sha256` was rewritten with no refusal.
+5. A block whose `key` and `uri` named different objects was preserved
+   verbatim, key sha `c63744a4...` beside uri sha `1e9b3fdb...`.
+6. A list-valued `manifest.yaml` was overwritten by the fetch.
+7. Importing `db.supabase_client` under `LEDGER_SCHEMA=zzz` bound
+   `LEDGER_SCHEMA='zzz'` and emitted a `FutureWarning` at collection.
+
+## Verification
+
+- `uv run pytest -q`: green.
+- `uv run ruff check .`: clean.
+- `uv run ruff format --check .`: clean for every file this branch touches. 13
+  files are unformatted on `main` already and are byte-identical here; CI runs
+  `ruff check` only, so they are pre-existing and out of scope.
+- CI's db CLI gate (`chronicle init` / `load all` / `stats`): passes.
+- `CHRONICLE_R2_RAW_BUCKET=zzz CHRONICLE_SCHEMA=zzz uv run pytest -q`: green.
+  Before the shared fixture it failed five tests — four bucket-default
+  assertions in `tests/test_chronicle_artifacts.py` and the collection-time
+  schema constant in `tests/test_chronicle_namespace.py`.
 
 ## Next
 
-- None; ready for handoff. No push was performed.
+- Push and open the PR against `main`. Do not merge.
+- Follow-up PR, after Max creates and backfills the new buckets: flip
+  `DEFAULT_R2_RAW_BUCKET` / `DEFAULT_R2_DERIVED_BUCKET` to `chronicle-raw` /
+  `chronicle-derived`.

README.md

@@ -65,10 +65,12 @@ contract that aligns it to another period (see
 | Microcosm Target Contracts | Selection, measurement bindings, and active subset | Period alignment, support-aware activation, solver inputs, diagnostics |
 
 The storage split is documented in
-[`docs/storage-architecture.md`](docs/storage-architecture.md): `ledger-raw`
-stores immutable source bytes, `ledger-derived` stores reproducible build
+[`docs/storage-architecture.md`](docs/storage-architecture.md): a raw R2 archive
+stores immutable source bytes, a derived R2 archive stores reproducible build
 artifacts, and Supabase/Postgres hosts the queryable relational Chronicle registry
-mirrored from accepted builds.
+mirrored from accepted builds. The bucket names are configuration
+(`$CHRONICLE_R2_RAW_BUCKET` and `$CHRONICLE_R2_DERIVED_BUCKET`), still defaulting
+to the ledger-era `ledger-raw` and `ledger-derived`.
 
 ## Repository Model
 
@@ -247,7 +249,7 @@ This writes:
   source_regions.jsonl
   facts.jsonl
   consumer_facts.jsonl
-  ledger.db
+  chronicle.db
   reports/
     source_rows.json
     source_cells.json
@@ -349,11 +351,14 @@ needed, even when your Cloudflare user belongs to several accounts:
 # One-time per machine (opens a browser consent page):
 bunx wrangler login
 
-# One-time per account (already done for the PolicyEngine account):
-uv run chronicle bootstrap-r2 --raw-bucket ledger-raw --derived-bucket ledger-derived
+# One-time per account (already done for the PolicyEngine account). The bucket
+# flags default to $CHRONICLE_R2_RAW_BUCKET / $CHRONICLE_R2_DERIVED_BUCKET:
+uv run chronicle bootstrap-r2
 
 # Fetch/register a source artifact, write db/data/.../manifest.yaml, and upload
-# the exact bytes to R2 when Wrangler is authenticated:
+# the exact bytes to R2 when Wrangler is authenticated. Pass --manifest when the
+# package directory keeps more than one manifest (ira_contributions keeps a
+# traditional and a Roth one):
 uv run chronicle fetch-artifact \
   --url https://www.irs.gov/pub/irs-soi/23in12ms.xls \
   --source-id irs_soi \
@@ -364,11 +369,20 @@ uv run chronicle fetch-artifact \
   --table "Publication 1304 Table 1.2" \
   --upload-r2
 
+# Re-fetching is safe: identical bytes keep the recorded storage.r2 block, and
+# bytes that disagree with what the entry identifies -- its declared sha256, or
+# its recorded content-addressed key once published -- are refused. When a
+# publisher has re-published
+# under the same URL and vintage, register the revision explicitly — the new
+# bytes get their own content-addressed key and the superseded object is kept
+# in storage.previous_r2:
+uv run chronicle fetch-artifact ... --record-revision
+
 # Audit local manifests and checksums:
 uv run chronicle inventory-artifacts --root db/data
 
-# Upload all existing manifest-declared local artifacts to ledger-raw and write
-# storage.r2 metadata back into the manifests:
+# Upload all existing manifest-declared local artifacts to the raw archive and
+# write storage.r2 metadata back into the manifests:
 uv run chronicle publish-raw --root db/data
 ```
 
@@ -406,10 +420,10 @@ To prepare the deterministic SQLite artifact for a hosted Supabase/Postgres
 mirror, export each relational table to JSONL plus a manifest:
 
 ```bash
-uv run chronicle export-db-tables --db /tmp/chronicle-suite/ledger.db --out /tmp/chronicle-mirror --replace
+uv run chronicle export-db-tables --db /tmp/chronicle-suite/chronicle.db --out /tmp/chronicle-mirror --replace
 ```
 
-To publish the deterministic build outputs to the `ledger-derived` R2 bucket:
+To publish the deterministic build outputs to the derived R2 archive:
 
 ```bash
 uv run chronicle publish-derived \
@@ -437,6 +451,14 @@ uv run chronicle load-supabase-mirror \
 Use `--dry-run` first to validate JSONL row counts and file coverage without
 writing to Supabase.
 
+Chronicle settings are read chronicle-first: `CHRONICLE_X` wins, and the
+ledger-era `POLICYENGINE_LEDGER_X` and `LEDGER_X` spellings still work behind a
+one-time deprecation warning naming the variable to move to.
+[`docs/storage-architecture.md`](docs/storage-architecture.md#environment-variable-rename-window)
+lists every variable in that window, and
+[Bucket Cutover](docs/storage-architecture.md#bucket-cutover) covers the R2
+bucket rename.
+
 Chronicle facts keep source concepts and canonical concepts separately. For example,
 the SOI Table 1.1 adjusted gross income column is preserved as
 `irs_soi.adjusted_gross_income`, while the canonical concept is

chronicle/__init__.py

@@ -13,6 +13,7 @@
     "consumer_contract",
     "core",
     "database",
+    "env",
     "facts",
     "harness",
     "jurisdictions",