diff --git a/PROGRESS-chronicle-dual-accept-round2.md b/PROGRESS-chronicle-dual-accept-round2.md new file mode 100644 index 000000000..8406d02f9 --- /dev/null +++ b/PROGRESS-chronicle-dual-accept-round2.md @@ -0,0 +1,89 @@ +# Round-2 review fixes — PR #849 (`chronicle-dual-accept`) + +Journal for the gate-findings pass on top of `1f78847c`. History, not state: +check git/GitHub for current truth. + +## Verified ground truth (read this session, 2026-09-02) + +Read from `PolicyEngine/chronicle` at `origin/main`: + +| Identity | Where | Epoch | +|---|---|---| +| `policyengine_ledger.consumer_artifact.v2` | `policyengine_chronicle/consumer.py:30`, README line 299 | ledger | +| `ledger.consumer_fact.v1` | `chronicle/consumer_contract.py:28` | ledger | +| `ledger.fact.v1` | `chronicle/core.py:103` | ledger | +| `ledger.source_cell.v1` | `chronicle/sources/cells.py:24` | ledger | +| `ledger.source_row.v1`, `ledger.source_column.v1`, `ledger.source_row_value.v1` | `chronicle/sources/rows.py:27-29` | ledger | +| `ledger.{aggregate_fact,semantic_fact,concept_alignment,dimension_set,observed_measure,source_release,source_series,universe_constraint_set}.v2` | `chronicle/consumer_contract.py`, `policyengine_chronicle/consumer.py:195` | ledger | + +`policyengine_ledger.consumer_artifact.v1` is what **Microcosm** mints +(`import_entry_facts.py:164`), never what Chronicle emits — it is ledger-era +history that must keep loading. + +There is **no** `origin/epoch-dual-domain` branch on chronicle +(`git ls-remote --heads origin | grep -i epoch` → empty), so the chronicle-era +successors are declared here by the migration rule chronicle#143 states — +new namespace, same family, version bumped by one — and the brief pins the two +that matter: `policyengine_chronicle.consumer_artifact.v3` and +`chronicle.consumer_fact.v2`. + +## State + +Round 2 complete and pushed to `chronicle-dual-accept`; the PR body carries a +"Review fixes (round 2)" section with the same content as this journal's +verification block. **DO NOT MERGE** — the lane was authorized to fix and +push only. + +## Verification (2026-09-02) + +Both full `packages/microcosm-build` suites — clean shell and the +polluted-Logbook shell finding 2 specifies — returned **outcome-identical** +results: 7116 passed, 47 skipped, 2 failed, out of 7165. + +The 2 failures are `test_release_target_parity.py::TestRegeneration` and are +**not this branch's**. They reproduce identically at `1f78847c` (this +branch's previous head) and at `origin/main` `47c74225`, both verified in a +throwaway worktree this session: + + ValueError: Ledger target reference + 'cbo.revenue_projection.ty2024.income_by_source.adjusted_gross_income.projected_amount' + assertion_policy='observed_only' does not allow resolved fact assertion + 'source_projection'. + +They run only where the pinned feed exists — +`~/PolicyEngine/_buildh-runtime/inputs/consumer_facts_buildn_v9_4.jsonl`, +outside the repo — and skip everywhere else, PR CI included. Worth a separate +issue: on a machine holding that feed, the committed +`cbo.revenue_projection.*` reference and the feed row it resolves to disagree +about assertion class. + +`uv run ruff format --check .` is red repo-wide (116 files) at this branch +*and* at its merge base: ruff 0.15.16's formatter against a repo pinned to +`ruff>=0.8`, whose CI lint lane runs `ruff check` only. Every file this +branch touches is format-clean before and after. + +## What changed structurally, for the next reader + +The first pass treated epoch as something you could *read off* a key's +namespace. That is why it accepted a two-id set that excluded the id +Chronicle actually emits, and why `chronicle.anything.vN` was witnessed as +Chronicle-issued identity. Round 2 replaces it with a declared registry: +`DECLARED_IDENTITIES` in `chronicle_epoch.py`, keyed by +`(namespace, family, version)`. Anything not in that table is `undeclared` +and is reported as such. If Chronicle publishes its own successor +enumeration, that table is the one place that changes. + +## Done + +- [x] (1) Declared identity registry keyed by `(namespace, family, version)` +- [x] (2) Autouse Logbook env isolation + fail-closed network in tests +- [x] (3) UK `_ledger_provenance` delegates to `provenance()` +- [x] (4) Undeclared chronicle-namespace identities labelled `undeclared` +- [x] (5) `microcosm.build.logbook_env` module import no longer shadowed +- [x] (6) `CHRONICLE_US_SOURCE_COVERAGE_CONTRACT_COMMIT` in the us_runtime barrel +- [x] (7) Non-string `schema_version` raises the documented ValueError +- [x] (8) `tools/logbook.py --remote` help prefers `LOGBOOK_*` + +## Next + +Nothing in this lane. Reviewer decides on merge. diff --git a/PROGRESS-chronicle-dual-accept.md b/PROGRESS-chronicle-dual-accept.md new file mode 100644 index 000000000..604b9ec9a --- /dev/null +++ b/PROGRESS-chronicle-dual-accept.md @@ -0,0 +1,163 @@ +# Chronicle dual-acceptance (consumer side) — chronicle#143 + +Branch `chronicle-dual-accept`. Journal for this lane. Root journals are +history, not state (see CLAUDE.md); this one is current only while the branch +is open. + +> **Corrected 2026-09-02 by the round-2 review pass.** Several identity and +> design claims below were accurate to the first pass and are wrong now. The +> accepted artifact set is three ids, not two; the chronicle-era successor of +> the artifact id is `.v3`; epoch resolution is a declared registry, not +> namespace parsing; and the env half is `LOGBOOK_*`, not `CHRONICLE_*`. Each +> is annotated in place below. See `PROGRESS-chronicle-dual-accept-round2.md` +> and the PR's "Review fixes (round 2)" section for what is actually in the +> branch. + +## Goal + +Microcosm must accept BOTH ledger-era and chronicle-era Chronicle identities +*before* Chronicle flips emit: + +- schema ids: `policyengine_ledger.consumer_artifact.v1` **and** + `policyengine_chronicle.consumer_artifact.v2`; `ledger.consumer_fact.v1` + **and** `chronicle.consumer_fact.v2` + + *(Corrected 2026-09-02: wrong at both ends. Chronicle's `main` emits + `policyengine_ledger.consumer_artifact.v2` today, so that pair rejected + every artifact Chronicle publishes; and the chronicle-era successor of the + v2 id is `policyengine_chronicle.consumer_artifact.v3`. The branch now + accepts all three.)* +- hash domains: `ledger..v2` **and** `chronicle..v3` (same canonical + payload, new domain string) + + *(Corrected 2026-09-02: the version numbers differ by family. The + source-side domains are ledger `v1` → chronicle `v2`; only the derived + families are `v2` → `v3`.)* +- env names: `CHRONICLE_*` preferred, legacy honored with a once-per-process + deprecation warning + + *(Corrected on the PR before merge, and again here: these are the + **Logbook** store's credentials, so the preferred spellings are + `LOGBOOK_*`. See the PR review comment of 2026-09-01.)* + +Frozen (microcosm#639): nothing on disk or in artifacts renames. Diagnostic +field names (`ledger_aggregate_fact_key`, `ledger_commit`), H5 attrs, +`populace_*` ids, fact keys, goldens and fixtures stay at v1. + +**Acceptance widens; nothing narrows.** This is the rule the lane's own first +pass broke and the second pass restored — see "The regression that mattered". + +## State + +Implementation and tests landed; verification run recorded in the PR body and +in the lane's report. PR #849, open, do not merge. + +## Done + +- `microcosm/build/chronicle_epoch.py` — the single epoch authority. Epoch + detection is **structural** (the namespace segment of a key domain), not a + lookup in a frozen domain list, because chronicle#143 declares the `v3` + spelling only for the aggregate and semantic fact families. Only identity + strings the spec names explicitly are pinned as literals. + + *(Corrected 2026-09-02: structural detection reported + `chronicle..vN` as Chronicle-**issued** identity, which is a + guess dressed as a witness. Resolution is now a declared registry keyed by + `(namespace, family, version)`; an undeclared spelling in a Chronicle + namespace is reported as `undeclared`. The claim that chronicle#143 + "declares the `v3` spelling" for two families also overstated the issue, + which lists `dual-hash window or v3 domains` as open options.)* +- `microcosm/build/chronicle_env.py` — the env dual-read window, one helper, + one `DeprecationWarning` per process per legacy name. + + *(Renamed before merge to `microcosm/build/logbook_env.py`; there is no + `chronicle_env.py` on the branch.)* +- `ledger_artifact.py` — the manifest `schema_version` is a membership test + over both eras; per-row schema ids and fact keys are carried as published; + `provenance()` records the observed manifest id, `schema_epoch`, + `fact_key_epochs`, and `fact_schema_versions`. +- `us_trade/import_entry_facts.py` — emission stays ledger-era (bytes are + pinned; these rows are minted from Census/CBP bytes, so there is no source + epoch to inherit), but the declared id is now an argument checked against + both eras. +- Chronicle-era aliases for the four `LEDGER_*` module constants that only + look like env vars to a grep. +- Tests: `test_chronicle_epoch.py`, `test_chronicle_env.py`, plus mixed-epoch + cases in `test_ledger_targets.py`, epoch-independence of the minted + `microcosm.derived_fact.*` keys in + `test_us_congressional_district_vintage.py`, and dual-era emission and + acceptance in `test_us_trade_facts.py`. + +## The regression that mattered + +The lane's first pass added a per-row `schema_version` check to +`_load_fact_rows` that rejected any id outside the two chronicle#143 names — +validation `main` never performed, on a field `main` never read. Real feeds do +not honor that set. The pinned US fiscal-refresh feed +`consumer_facts_buildn_v9_4.jsonl`, which the release path loads through this +loader, declares `arch.consumer_fact.v1` on 37,006 of its first 200,000 rows +and `ledger.consumer_fact.v1` on 399. The check failed the build closed on its +own pinned input, and **PR CI could not see it**: the feed is gated data +outside PR CI. Reproduced directly, fixed, and pinned by a test. + +The same pass witnessed fact-key epochs from only the four fields targets +resolve a fact by. Published rows carry eleven key-bearing paths; the pinned +feed exercises all of them. A row straddling the cutover — ledger-era +aggregate key, chronicle-era source-release key — was reported as pure +ledger-era. The inventory is now complete and pinned to the captured feed +fixture. + +## Audit result + +Every site that compares, parses, or mints a Chronicle fact key or schema id +was inventoried. After this branch, no non-test source file hard-codes a +single epoch in a validator. `ledger_targets.py` carries keys opaquely and +contains no epoch literal at all; `congressional_district_vintage.py` and +`us_trade/import_entry_facts.py` mint into Microcosm-owned namespaces from +digest payloads that contain no Chronicle key, so a source row's epoch cannot +move a minted key; the UK runtime is clean; no golden embeds a Chronicle +domain and no H5 attribute name embeds `ledger`. `require_pins` does not exist +in this repo. + +## Open question for the Chronicle side + +The pinned feeds carry an `arch.*` key namespace alongside `ledger.*` — 37,006 +rows against 399 in `consumer_facts_buildn_v9_4.jsonl`, across all eleven +families. Microcosm treats those keys opaquely, so nothing here depends on +what `arch` is, and the epoch module reports it as outside both declared eras. +Whether chronicle#143's cutover is meant to re-epoch `arch.*` rows too is a +question for the Chronicle lane; this branch does not guess. + +## Next + +Nothing outstanding on this branch. Chronicle's own acceptance half lands in +the parallel lane; the emit flip is a separate, later cutover. + +## Correction (2026-09-02): the env half was misnamed + +PR #849 review (Fable, main) caught that `POPULACE_LEDGER_URL` / +`_KEY` / `_API_KEY` / `_EXPORT_KEY` are **Logbook** store credentials +(Supabase `logbook` schema, `logbook_writer` / `logbook_exporter` roles — +see `logbook.py`'s docstring and `logbook/README.md`), not Chronicle fact- +store ones. "Ledger" there was the generic build-ledger sense renamed to +Logbook on 2026-08-08 (microcosm#632) specifically to stop colliding with +Chronicle. Naming the preferred spellings `CHRONICLE_*` would have recreated +that exact collision. + +The epoch half above (`chronicle_epoch.py`, `ledger_artifact.py`, +`import_entry_facts.py`, `ledger_targets.py`) is unaffected and was approved +as-is. The env half only: `chronicle_env.py` renamed to `logbook_env.py`, +`CHRONICLE_*_ENV`/`chronicle_env`/`chronicle_env_names`/ +`describe_chronicle_env`/`reset_chronicle_env_deprecation_warnings` renamed +to `LOGBOOK_*_ENV`/`logbook_env`/`logbook_env_names`/`describe_logbook_env`/ +`reset_logbook_env_deprecation_warnings`, and every caller (`logbook.py`, +`tools/logbook.py`, `build/__init__.py` exports, `logbook/README.md`, the +changelog fragment, this file, and the module-path comments in +`firm_generation.py`/`source_coverage.py`) updated to match. The +`POPULACE_LEDGER_*` legacy names and the once-per-process +`DeprecationWarning` behavior are unchanged; the warning text now cites +microcosm#632 instead of chronicle#143, since this window is a Logbook +naming cleanup riding along on this branch, not part of the chronicle#143 +epoch migration. `CHRONICLE_*_BANDS` and +`CHRONICLE_US_SOURCE_COVERAGE_CONTRACT_COMMIT` are untouched — those really +do translate Chronicle ids and pin a Chronicle commit. diff --git a/changelog.d/chronicle-dual-accept.added.md b/changelog.d/chronicle-dual-accept.added.md new file mode 100644 index 000000000..0af1451c3 --- /dev/null +++ b/changelog.d/chronicle-dual-accept.added.md @@ -0,0 +1 @@ +Accept chronicle-era Chronicle identities everywhere Microcosm consumes them, from a declared identity registry keyed by `(namespace, family, version)`. Consumer-artifact manifests load under every id a real producer stamps — Microcosm's own minted `policyengine_ledger.consumer_artifact.v1`, the `policyengine_ledger.consumer_artifact.v2` Chronicle emits today, and the chronicle-era `policyengine_chronicle.consumer_artifact.v3` successor — and per-row `ledger.consumer_fact.v1` / `chronicle.consumer_fact.v2` alike. Mixed-epoch feeds compile, and provenance records the observed manifest id, its epoch, the fact-key epoch labels present in the feed, any undeclared Chronicle-namespace key domain among them, and the per-row schema ids the rows declare. An identity in a Chronicle namespace that the registry does not carry is reported as `undeclared`, never counted as Chronicle-issued. Only the manifest id is gated: per-row schema ids and fact keys are carried as published, since real feeds mint rows in namespaces belonging to neither era. Nothing renames on disk: emitted ids, diagnostic field names, `populace_*` key namespaces, goldens and fixtures stay frozen at v1. diff --git a/changelog.d/chronicle-dual-accept.changed.md b/changelog.d/chronicle-dual-accept.changed.md new file mode 100644 index 000000000..ec894c9c6 --- /dev/null +++ b/changelog.d/chronicle-dual-accept.changed.md @@ -0,0 +1 @@ +Give the Logbook store's environment variables a `LOGBOOK_*` dual-read window, with the `POPULACE_LEDGER_*` names honored under a once-per-process deprecation warning and preferred in the `tools/logbook.py --remote` help text (microcosm#632). These are the Logbook store's own credentials, unrelated to the Chronicle fact store. Every test in the build shard is now detached from a live Logbook by an autouse fixture that clears both generations of the variables and refuses unstubbed Logbook HTTP. diff --git a/logbook/README.md b/logbook/README.md index a148dd142..81dc42fdd 100644 --- a/logbook/README.md +++ b/logbook/README.md @@ -90,8 +90,10 @@ for `published` and `certified` rows. ## The live store -The best-effort Supabase insert (`POPULACE_LEDGER_URL` + -`POPULACE_LEDGER_KEY`, the migration's insert-only `logbook_writer` role) +The best-effort Supabase insert (`LOGBOOK_URL` + `LOGBOOK_KEY`, the +migration's insert-only `logbook_writer` role; the ledger-era +`POPULACE_LEDGER_URL` / `POPULACE_LEDGER_KEY` stay honored for the +Logbook dual-read window (microcosm#632) and warn once per process) uses the same scope rule as the archives after this PR's migration is applied by the project owner. The writer key is unaffected: rows are still inserted through the same role and the scope is derived from the hashed `pipeline` diff --git a/out.md b/out.md deleted file mode 100644 index a727ccd79..000000000 --- a/out.md +++ /dev/null @@ -1,119 +0,0 @@ -# Armenia country package final report (#814) - -Date: 2026-08-28 - -Branch: `armenia-country-package-814` - -## Outcome - -The Armenia package is complete as a schema-valid, spec-only, engine-free -greenfield contract. It is not represented as an executable build: the exact -`populace-us` artifact, Armenian facts, generated cell bindings, community -distribution, gate policy, and shared runtime kernels remain explicit -prerequisites. - -The closed `am/` inventory mirrors Belgium's six typed resources and five -generation-zero JSON projections. It declares: - -- a two-stage load of a to-be-authenticated public `populace-us` artifact and - target-derived marz assignment; every record remains a US donor support - record and is never described as Armenian microdata; -- a 2022-census-vintage community spine constrained to the assigned marz, with - 10 marzes plus Yerevan and 71 consolidated communities documented but no - invented code roster; -- eight Ledger-only, count/indicator calibration authoring contracts spanning - demography, household structure, consumption, labour, earnings, pensions, - and family benefits; -- real-resolver refusal of unexpanded multi-cell tables: Chronicle must generate - cell-pinned Ledger references and direct/pre-built candidate bindings before - runtime activation; -- wage/payment, raw-income diagnostic, and national-accounts facts outside the - solver manifest until their validation role or AMD-compatible pre-built bridge - is enforceable; -- greenfield aggregate-admin, per-family-fit, target-coverage, macro-realism, - support, weight-audit, ESS, and ratio gate declarations, plus active - release-blocking reference-coverage/support/output checks; and -- a public 2024 release contract for `populace_am_{year}.h5`, with ArmStat open - dissemination stated and exact ArmStat/donor licence text left as a mandatory - verification item. - -`HARVEST.md` has exactly one solver worklist row per live target-reference key, -plus separate deferred validation/amount, geography, source-authentication, and -external-oracle worklists. `NOTES.md` records the Belgium solver lessons, -permanent survey tax-benefit holdouts, the amount/currency boundary, -`populace#263/#265`, and the future Axiom-backed `rulespec-am` boundary. - -Package identity after review: - -- CountrySpec fingerprint: - `64f50fa39e68e9ba6c451e3a47a2f2adeaba5a5ccb147cb80297f942de433ca8` -- Typed spec SHA-256: - `659b6baf5ebbd71fb7786ec4c4d49df565b2bddabeb868a9385ed226c56880f9` - -## Verification - -All commands ran offline. The package-wide command used the exact lock-required -`policyengine-us==1.819.0` already present in the local uv cache, so the two -engine-only test files ran inside the same aggregate rather than failing for a -missing optional dependency. - -Exact requested package-wide command: - -```sh -UV_NO_SYNC=1 UV_PROJECT_ENVIRONMENT=/tmp/armenia-uv-env.ME3TCl UV_CACHE_DIR=/tmp/uv-cache-armenia-814 PYTHONPATH=packages/microcosm-build/src:packages/microcosm-calibrate/src:packages/microcosm-frame/src:packages/microcosm-fit/src:packages/microcosm-data/src:/Users/maxghenis/.cache/uv/archive-v0/ewqqbcYNhWejPQ-OfsFxl:/tmp/armenia-no-engine-site.vZ5Kv8:/Users/maxghenis/PolicyEngine/chronicle/.venv/lib/python3.14/site-packages uv run pytest packages/microcosm-build -``` - -Result: **6,556 passed, 45 skipped, 2,351 warnings, 0 failed** in 3,671.74 -seconds (1:01:11). The warnings are existing numerical, pandas copy/fragmentation, -and PolicyEngine-US runtime warnings; none is Armenia-specific. - -Additional checks: - -- Focused package/golden/compiler suite: - `uv run pytest packages/microcosm-build/tests/test_spec_only_country_packages.py packages/microcosm-build/tests/test_country_spec.py packages/microcosm-build/tests/test_spec_engine_country_bundles.py -p no:cacheprovider` - — **101 passed** in 29.82 seconds. -- Authoritative `shared-spec` CI group: - `python3 tools/ci_test_groups.py --list shared-spec | xargs uv run pytest -p no:cacheprovider` - — **1,334 passed, 42 skipped, 1 warning** in 418.20 seconds. -- `ruff check .` — **passed**. -- `python3 tools/ci_test_groups.py --verify` — **verification=ok**, 313 test - files tracked. -- `git diff --check` — **passed**. - -## Deviations from Belgium and why - -1. Armenia consumes a pre-built public US donor pool instead of native, - restricted SILC. No `support_spine.json` is present because the current - vocabulary describes raw ASEC pool construction, not an existing artifact. -2. Armenia is engine-free. No target, gate, or release file requires - `rulespec-am`; any later rules leg must use Frame's `RulesEngine` protocol - through the Axiom adapter. -3. Marz is target-assigned before community cloning. The clone factor is the - compile-safe minimum of one, not Belgium's 20: collision-avoiding fanout must - wait for the 71-to-11 roster and within-marz support evidence. -4. There is no incumbent, so parity/export/target-surface gates are absent. - National accounts back a deferred macro-realism band; CEQ and World Bank - estimates remain documentation-only band candidates until harvested. -5. The live solver manifest contains eight count/indicator series contracts, - not guessed scalar cells or AMD amount targets. Those series will expand to - the reviewed cell-level profile after Chronicle harvest; the eventual - 10–16-margin selection is not fabricated in this package. - -## Top five maintainer questions - -1. Which immutable `populace-us` revision/file, hash, licence, and column - inventory certify the donor input? -2. Which exact Statbank, ILCS, LFS, SRC, pension, and benefit tables/cells define - the 2024 profile, and what generated cell-reference/binding artifact owns - their fanout? -3. Which scale-free indicator or reviewed AMD-compatible pre-built bridge makes - donor consumption bands and any future amount rows conceptually comparable? -4. What ex-ante Armenia aggregate-fit, family-fit, macro-realism, ESS, and - weight-ratio thresholds—and which shared evaluators—activate the deferred - gate declarations? -5. What is the authoritative 2022 marz/community roster and assignment table, - and when will the shared geography/source-coverage runtimes land under - `populace#263/#265`? - -No network fetch, push, PR, artifact build, release, or publication was -performed. The repository-root `PROGRESS.md` was not touched. diff --git a/packages/microcosm-build/src/microcosm/build/__init__.py b/packages/microcosm-build/src/microcosm/build/__init__.py index 352f53b74..b50754703 100644 --- a/packages/microcosm-build/src/microcosm/build/__init__.py +++ b/packages/microcosm-build/src/microcosm/build/__init__.py @@ -49,6 +49,22 @@ def _assert_frame_compatible(version: str, required: tuple[int, int]) -> None: _assert_frame_compatible(_frame_version, _REQUIRED_FRAME_SERIES) +from microcosm.build.chronicle_epoch import ( # noqa: E402 - after the compat gate + ACCEPTED_CONSUMER_ARTIFACT_SCHEMA_VERSIONS, + ACCEPTED_CONSUMER_FACT_SCHEMA_VERSIONS, + CHRONICLE_EPOCH, + DECLARED_IDENTITIES, + DECLARED_IDENTITY_EPOCHS, + EPOCHS, + LEDGER_EPOCH, + UNDECLARED, + fact_key_epoch, + fact_key_epoch_label, + feed_fact_key_epochs, + feed_undeclared_fact_key_domains, + is_chronicle_fact_key, + parse_fact_key, +) from microcosm.build.country_spec import ( # noqa: E402 - after the compat gate CountryResourceRow, CountrySpec, @@ -126,6 +142,15 @@ def _assert_frame_compatible(version: str, required: tuple[int, int]) -> None: select_ledger_targets_from_jsonl, target_spec_from_ledger_fact, ) + +# Only ``logbook_env_names`` is re-exported here. The reader function is +# spelled ``logbook_env`` — the same name as its module — so binding it on +# the package would make ``import microcosm.build.logbook_env as env`` hand +# back the function instead of the module. Import the function from its +# module: ``from microcosm.build.logbook_env import logbook_env``. +from microcosm.build.logbook_env import ( # noqa: E402 - after the compat gate + logbook_env_names, +) from microcosm.build.monetary_profile import ( # noqa: E402 - after compat gate MonetaryTargetContract, MonetaryTargetProfile, @@ -196,6 +221,14 @@ def _assert_frame_compatible(version: str, required: tuple[int, int]) -> None: "StagingTelemetry", "TargetCoverageRequirement", "TargetFitRequirement", + "ACCEPTED_CONSUMER_ARTIFACT_SCHEMA_VERSIONS", + "ACCEPTED_CONSUMER_FACT_SCHEMA_VERSIONS", + "CHRONICLE_EPOCH", + "DECLARED_IDENTITIES", + "DECLARED_IDENTITY_EPOCHS", + "EPOCHS", + "LEDGER_EPOCH", + "UNDECLARED", "LedgerConsumerArtifact", "LedgerTargetMapping", "LedgerTargetSelection", @@ -204,6 +237,13 @@ def _assert_frame_compatible(version: str, required: tuple[int, int]) -> None: "MonetaryTargetProfile", "PreparedMonetaryMeasure", "add_ledger_artifact_args", + "logbook_env_names", + "fact_key_epoch", + "fact_key_epoch_label", + "feed_fact_key_epochs", + "feed_undeclared_fact_key_domains", + "is_chronicle_fact_key", + "parse_fact_key", "aggregate_admin_gate", "area_support_gate", "column_implication_gate", diff --git a/packages/microcosm-build/src/microcosm/build/chronicle_epoch.py b/packages/microcosm-build/src/microcosm/build/chronicle_epoch.py new file mode 100644 index 000000000..86fdf09a3 --- /dev/null +++ b/packages/microcosm-build/src/microcosm/build/chronicle_epoch.py @@ -0,0 +1,585 @@ +"""Epoch tolerance for Chronicle (formerly Ledger) identities. + +PolicyEngine Ledger is being renamed Chronicle. Per the migration spec on +PolicyEngine/chronicle#143 the rename of *identities* happens **by epoch, +never in place**: Chronicle introduces chronicle-era hash domains and +chronicle-named schema ids for **newly emitted** rows at a declared cutover +release, and ledger-era ids stay valid *history* forever. No golden +regenerates and no witnessed row is rewritten. + +Microcosm is a consumer, so it must accept both eras **before** Chronicle +flips emit. This module is the single place that knows which identities are +which era. The rules it encodes: + +1. **Fact keys are opaque.** Microcosm never reconstructs a Chronicle key + from a payload, so it never needs to know how the domain string feeds the + hash. Comparison stays exact string equality, which is correct in both + eras; what changes is that a validator asking "is this a Chronicle fact + key?" must answer yes for either epoch. +2. **Epoch is a declaration, not a guess.** Every identity this module calls + Chronicle-issued is listed in :data:`DECLARED_IDENTITIES` below, keyed by + ``(namespace, family, version)``. Structural parsing of + ``..v:`` still happens, but only so an + *undeclared* key in a Chronicle namespace can be reported as + :data:`UNDECLARED` in provenance — never silently promoted to issued + identity. A cutover that ships a spelling this table does not carry is a + one-row diff here, made deliberately, with the reader able to see which + spellings the build was actually prepared for. +3. **Microcosm-minted keys are epoch-independent.** Keys Microcosm mints live + in Microcosm-owned namespaces (``microcosm.derived_fact.*``, + ``populace_us_trade.*``) and are frozen at v1 by microcosm#639. They carry + no Chronicle epoch, and a chronicle-era source row must mint the + byte-identical Microcosm key a ledger-era source row does — otherwise the + cutover would silently re-identify derived facts. +4. **Nothing renames on disk.** Diagnostic field names + (``ledger_aggregate_fact_key``, ``ledger_commit``), H5 attributes, + ``populace_*`` ids, goldens, and fixtures are frozen at v1. + +Provenance for the ledger-era half of the table was read out of +``PolicyEngine/chronicle`` at ``origin/main`` rather than restated from a +plan: ``policyengine_chronicle/consumer.py`` for the artifact id, +``chronicle/consumer_contract.py`` for the per-row id and the ``.v2`` fact +domains, ``chronicle/core.py`` and ``chronicle/sources/`` for the ``.v1`` +source domains. The chronicle-era half is the successor each of those takes +under the rule chronicle#143 states for the rename — same family, new +namespace, version bumped by one — with the two ids the consumer migration +pins spelled out explicitly. Chronicle has published no per-family +enumeration of its own; when it does, this table is what changes. +""" + +from __future__ import annotations + +import re +from collections.abc import Iterable, Mapping +from dataclasses import dataclass +from typing import Any + +__all__ = [ + "ACCEPTED_CONSUMER_ARTIFACT_SCHEMA_VERSIONS", + "ACCEPTED_CONSUMER_FACT_SCHEMA_VERSIONS", + "CHRONICLE_CONSUMER_ARTIFACT_SCHEMA_VERSION", + "CHRONICLE_CONSUMER_FACT_SCHEMA_VERSION", + "CHRONICLE_EPOCH", + "CHRONICLE_FACT_KEY_DOMAINS", + "CHRONICLE_NAMESPACES", + "DECLARED_IDENTITIES", + "DECLARED_IDENTITY_EPOCHS", + "DECLARED_IDENTITY_REGISTRY", + "EPOCHS", + "FACT_KEY_EPOCH_LABELS", + "FACT_KEY_FIELDS", + "FACT_KEY_IDENTITY_KIND", + "CONSUMER_ARTIFACT_SCHEMA_IDENTITY_KIND", + "CONSUMER_FACT_SCHEMA_IDENTITY_KIND", + "DeclaredIdentity", + "FactKeyIdentity", + "LEDGER_CONSUMER_FACT_SCHEMA_VERSION", + "LEDGER_EPOCH", + "LEDGER_FACT_KEY_DOMAINS", + "MICROCOSM_CONSUMER_ARTIFACT_SCHEMA_VERSION", + "PUBLISHED_CONSUMER_ARTIFACT_SCHEMA_VERSION", + "UNDECLARED", + "consumer_artifact_schema_epoch", + "consumer_fact_schema_epoch", + "declared_identity", + "describe_accepted_consumer_artifact_schema_versions", + "describe_accepted_consumer_fact_schema_versions", + "fact_key_epoch", + "fact_key_epoch_label", + "feed_fact_key_epochs", + "feed_undeclared_fact_key_domains", + "is_accepted_consumer_artifact_schema_version", + "is_accepted_consumer_fact_schema_version", + "is_chronicle_fact_key", + "parse_fact_key", + "row_fact_key_epochs", +] + +#: The ledger era: everything Chronicle issued under its former name. +LEDGER_EPOCH = "ledger" + +#: The chronicle era: everything issued from the declared cutover forward. +CHRONICLE_EPOCH = "chronicle" + +#: Both eras, oldest first. A validator iterates this; it never names one. +EPOCHS: tuple[str, ...] = (LEDGER_EPOCH, CHRONICLE_EPOCH) + +#: The label for an identity in a Chronicle namespace that this module's +#: declaration table does not carry — an unrecognized family, or a family at +#: a version nobody declared. Such an identity is *reported*, so a release +#: manifest shows that the feed contained something the build was not +#: prepared for, but it is never counted as Chronicle-issued identity. +UNDECLARED = "undeclared" + +#: The labels :func:`feed_fact_key_epochs` can emit, in report order. +FACT_KEY_EPOCH_LABELS: tuple[str, ...] = (*EPOCHS, UNDECLARED) + +#: Namespace -> epoch, for every namespace Chronicle issues identities in. +#: Fact keys and per-row schema ids use the bare ``ledger`` / ``chronicle`` +#: namespace; the consumer-artifact manifest id uses the package-qualified +#: ``policyengine_ledger`` / ``policyengine_chronicle`` one. +CHRONICLE_NAMESPACES: Mapping[str, str] = { + "ledger": LEDGER_EPOCH, + "policyengine_ledger": LEDGER_EPOCH, + "chronicle": CHRONICLE_EPOCH, + "policyengine_chronicle": CHRONICLE_EPOCH, +} + +#: Identity kinds. An identity's kind says where it is allowed to appear: +#: a fact-key domain never gates a manifest and a manifest id is never a key. +CONSUMER_ARTIFACT_SCHEMA_IDENTITY_KIND = "consumer_artifact_schema" +CONSUMER_FACT_SCHEMA_IDENTITY_KIND = "consumer_fact_schema" +FACT_KEY_IDENTITY_KIND = "fact_key" + +_VERSION_PATTERN = re.compile(r"^v\d+$") + + +@dataclass(frozen=True) +class DeclaredIdentity: + """One identity spelling the migration declares, and the epoch that owns it.""" + + namespace: str + family: str + version: str + epoch: str + kind: str + + @property + def identity(self) -> str: + """The identity string as it appears in an artifact, ``a.b.vN``.""" + return f"{self.namespace}.{self.family}.{self.version}" + + @property + def registry_key(self) -> tuple[str, str, str]: + """The ``(namespace, family, version)`` triple this is registered under.""" + return (self.namespace, self.family, self.version) + + +def _declare(identity: str, *, epoch: str, kind: str) -> DeclaredIdentity: + """Build one registry row from its literal identity string. + + Every argument is a literal at the call site below, so the table reads as + the list of spellings it is, and a typo fails at import rather than at + cutover. + """ + namespace, _, remainder = identity.partition(".") + family, _, version = remainder.rpartition(".") + if not namespace or not family or not _VERSION_PATTERN.match(version): + raise ValueError(f"Not a Chronicle identity spelling: {identity!r}.") + if CHRONICLE_NAMESPACES.get(namespace) != epoch: + raise ValueError( + f"{identity!r} is declared for the {epoch!r} epoch but its " + f"namespace {namespace!r} belongs to " + f"{CHRONICLE_NAMESPACES.get(namespace)!r}." + ) + return DeclaredIdentity( + namespace=namespace, family=family, version=version, epoch=epoch, kind=kind + ) + + +#: The consumer-artifact manifest ``schema_version`` Microcosm's **own** +#: minted artifacts declare (``import_entry_facts.write_consumer_artifact``). +#: Frozen at v1 by microcosm#639: these bytes must not move, and they must +#: keep loading forever. +MICROCOSM_CONSUMER_ARTIFACT_SCHEMA_VERSION = "policyengine_ledger.consumer_artifact.v1" + +#: The consumer-artifact manifest ``schema_version`` **Chronicle** stamps +#: today (``policyengine_chronicle/consumer.py`` on chronicle ``main``). Still +#: ledger-era: chronicle#143 has not flipped the namespace yet. +PUBLISHED_CONSUMER_ARTIFACT_SCHEMA_VERSION = "policyengine_ledger.consumer_artifact.v2" + +#: The chronicle-era successor of the id above: same payload, chronicle +#: namespace, version bumped. +CHRONICLE_CONSUMER_ARTIFACT_SCHEMA_VERSION = ( + "policyengine_chronicle.consumer_artifact.v3" +) + +#: The per-row consumer-fact ``schema_version`` Chronicle stamps today +#: (``chronicle/consumer_contract.py`` on chronicle ``main``), and its +#: chronicle-era successor. +LEDGER_CONSUMER_FACT_SCHEMA_VERSION = "ledger.consumer_fact.v1" +CHRONICLE_CONSUMER_FACT_SCHEMA_VERSION = "chronicle.consumer_fact.v2" + +#: Fact-key domains, ledger-era spelling beside its chronicle-era successor. +#: The ledger column is what Chronicle's ``main`` hashes with today; the +#: version numbers differ by family (the source-side domains are still v1 +#: where the derived ones are v2), which is why a version can never be +#: assumed and why this table is explicit rather than computed at the call +#: site. +_FACT_KEY_DOMAIN_DECLARATIONS: tuple[tuple[str, str], ...] = ( + # chronicle/core.py + ("ledger.fact.v1", "chronicle.fact.v2"), + # chronicle/sources/cells.py, chronicle/sources/rows.py + ("ledger.source_cell.v1", "chronicle.source_cell.v2"), + ("ledger.source_row.v1", "chronicle.source_row.v2"), + ("ledger.source_column.v1", "chronicle.source_column.v2"), + ("ledger.source_row_value.v1", "chronicle.source_row_value.v2"), + # chronicle/consumer_contract.py, policyengine_chronicle/consumer.py + ("ledger.aggregate_fact.v2", "chronicle.aggregate_fact.v3"), + ("ledger.semantic_fact.v2", "chronicle.semantic_fact.v3"), + ("ledger.concept_alignment.v2", "chronicle.concept_alignment.v3"), + ("ledger.dimension_set.v2", "chronicle.dimension_set.v3"), + ("ledger.observed_measure.v2", "chronicle.observed_measure.v3"), + ("ledger.source_release.v2", "chronicle.source_release.v3"), + ("ledger.source_series.v2", "chronicle.source_series.v3"), + ("ledger.universe_constraint_set.v2", "chronicle.universe_constraint_set.v3"), +) + +#: Every identity spelling this consumer is declared to understand. The +#: registry below is keyed off it; nothing outside this tuple is Chronicle +#: identity as far as Microcosm is concerned. +DECLARED_IDENTITIES: tuple[DeclaredIdentity, ...] = ( + _declare( + MICROCOSM_CONSUMER_ARTIFACT_SCHEMA_VERSION, + epoch=LEDGER_EPOCH, + kind=CONSUMER_ARTIFACT_SCHEMA_IDENTITY_KIND, + ), + _declare( + PUBLISHED_CONSUMER_ARTIFACT_SCHEMA_VERSION, + epoch=LEDGER_EPOCH, + kind=CONSUMER_ARTIFACT_SCHEMA_IDENTITY_KIND, + ), + _declare( + CHRONICLE_CONSUMER_ARTIFACT_SCHEMA_VERSION, + epoch=CHRONICLE_EPOCH, + kind=CONSUMER_ARTIFACT_SCHEMA_IDENTITY_KIND, + ), + _declare( + LEDGER_CONSUMER_FACT_SCHEMA_VERSION, + epoch=LEDGER_EPOCH, + kind=CONSUMER_FACT_SCHEMA_IDENTITY_KIND, + ), + _declare( + CHRONICLE_CONSUMER_FACT_SCHEMA_VERSION, + epoch=CHRONICLE_EPOCH, + kind=CONSUMER_FACT_SCHEMA_IDENTITY_KIND, + ), + *( + _declare(identity, epoch=epoch, kind=FACT_KEY_IDENTITY_KIND) + for ledger_domain, chronicle_domain in _FACT_KEY_DOMAIN_DECLARATIONS + for identity, epoch in ( + (ledger_domain, LEDGER_EPOCH), + (chronicle_domain, CHRONICLE_EPOCH), + ) + ), +) + +#: The registry: ``(namespace, family, version)`` -> the declared identity. +#: One triple can only belong to one epoch, and the constructor above +#: refuses a row whose namespace disagrees with the epoch it claims. +DECLARED_IDENTITY_REGISTRY: Mapping[tuple[str, str, str], DeclaredIdentity] = { + declaration.registry_key: declaration for declaration in DECLARED_IDENTITIES +} +if len(DECLARED_IDENTITY_REGISTRY) != len(DECLARED_IDENTITIES): # pragma: no cover + raise ValueError("Duplicate (namespace, family, version) in DECLARED_IDENTITIES.") + +#: The same registry reduced to what epoch resolution actually asks it: +#: ``(namespace, family, version)`` -> epoch. +DECLARED_IDENTITY_EPOCHS: Mapping[tuple[str, str, str], str] = { + registry_key: declaration.epoch + for registry_key, declaration in DECLARED_IDENTITY_REGISTRY.items() +} + + +def _identities_of(kind: str) -> tuple[str, ...]: + """Declared identity strings of one kind, oldest era first.""" + return tuple( + declaration.identity + for era in EPOCHS + for declaration in DECLARED_IDENTITIES + if declaration.kind == kind and declaration.epoch == era + ) + + +#: Membership sets. Loaders test membership, never equality with one era. +#: The artifact set carries three spellings, not two: Microcosm's own minted +#: v1, Chronicle's published v2, and the chronicle-era v3 successor. +ACCEPTED_CONSUMER_ARTIFACT_SCHEMA_VERSIONS = frozenset( + _identities_of(CONSUMER_ARTIFACT_SCHEMA_IDENTITY_KIND) +) +ACCEPTED_CONSUMER_FACT_SCHEMA_VERSIONS = frozenset( + _identities_of(CONSUMER_FACT_SCHEMA_IDENTITY_KIND) +) + +#: Fact-key domains by family, per epoch. Nothing consults these at runtime — +#: resolution goes through the registry — but they document the table and let +#: tests pin it against the feeds this repo carries. +LEDGER_FACT_KEY_DOMAINS: Mapping[str, str] = { + declaration.family: declaration.identity + for declaration in DECLARED_IDENTITIES + if declaration.kind == FACT_KEY_IDENTITY_KIND and declaration.epoch == LEDGER_EPOCH +} +CHRONICLE_FACT_KEY_DOMAINS: Mapping[str, str] = { + declaration.family: declaration.identity + for declaration in DECLARED_IDENTITIES + if declaration.kind == FACT_KEY_IDENTITY_KIND + and declaration.epoch == CHRONICLE_EPOCH +} + +#: Consumer-fact row fields that carry a single Chronicle key. The first four +#: are the identifiers :mod:`microcosm.build.ledger_targets` resolves a fact +#: by; the rest are the remaining single-key fields published rows carry, all +#: of them observed on the captured feed in +#: ``tests/fixtures/uk_target_reference_feed_rows.jsonl``. The list is for +#: *witnessing* epochs, so it is deliberately wider than the resolution set: a +#: row whose aggregate key is still ledger-era but whose source-release key has +#: crossed the cutover must report both epochs, not one. +FACT_KEY_FIELDS: tuple[str, ...] = ( + "aggregate_fact_key", + "semantic_fact_key", + "fact_key", + "legacy_fact_key", + "dimension_set_key", + "observed_measure_key", + "source_release_key", + "source_series_key", + "universe_constraint_set_key", +) + +#: Row fields carrying a *list* of Chronicle keys. +_FACT_KEY_LIST_PATHS: tuple[tuple[str, ...], ...] = ( + ("lineage", "source_cell_keys"), + ("lineage", "source_row_keys"), +) + +#: Nested single-key paths. +_FACT_KEY_NESTED_PATHS: tuple[tuple[str, ...], ...] = ( + ("concept_alignment", "concept_alignment_key"), +) + + +@dataclass(frozen=True) +class FactKeyIdentity: + """The structure of a Chronicle-shaped key, ``:``. + + ``epoch`` is the *declared* epoch — ``"ledger"`` or ``"chronicle"`` — and + is ``None`` for anything :data:`DECLARED_IDENTITY_REGISTRY` does not carry. + ``namespace_epoch`` is the weaker, structural answer: the epoch the key's + namespace belongs to, whatever family or version follows it. The two + differ exactly for a key in a Chronicle namespace whose spelling was never + declared, which :attr:`epoch_label` reports as :data:`UNDECLARED` rather + than promoting it to issued identity. + + Both are ``None`` for keys minted in some other namespace — Microcosm's + own ``microcosm.derived_fact.*`` and ``populace_us_trade.*`` derived keys, + for instance, which are deliberately outside both eras. + """ + + domain: str + namespace: str + family: str + version: str + digest: str + epoch: str | None + namespace_epoch: str | None + + @property + def declared(self) -> bool: + """True when this exact spelling is in the declaration table.""" + return self.epoch is not None + + @property + def epoch_label(self) -> str | None: + """The declared epoch, :data:`UNDECLARED`, or ``None`` if foreign.""" + if self.epoch is not None: + return self.epoch + return UNDECLARED if self.namespace_epoch is not None else None + + +def parse_fact_key(key: object) -> FactKeyIdentity | None: + """Split ``..v:``; ``None`` if not that shape. + + Parsing is deliberately shallow. Microcosm does not recompute Chronicle + digests, so it needs only enough structure to look the spelling up in + :data:`DECLARED_IDENTITY_REGISTRY` — and, when the lookup misses, to say + whether the miss was in a Chronicle namespace (undeclared, worth + reporting) or in some other namespace entirely (not Chronicle's at all). + """ + if not isinstance(key, str) or not key: + return None + domain, separator, digest = key.partition(":") + if not separator or not domain or not digest: + return None + segments = domain.split(".") + if len(segments) < 3: + return None + version = segments[-1] + if not _VERSION_PATTERN.match(version): + return None + namespace = segments[0] + family = ".".join(segments[1:-1]) + if not namespace or not family: + return None + declaration = DECLARED_IDENTITY_REGISTRY.get((namespace, family, version)) + if declaration is not None and declaration.kind != FACT_KEY_IDENTITY_KIND: + # A manifest or per-row schema id is a declared identity, but it is + # not a fact-key domain: keys never carry it. + declaration = None + return FactKeyIdentity( + domain=domain, + namespace=namespace, + family=family, + version=version, + digest=digest, + epoch=None if declaration is None else declaration.epoch, + namespace_epoch=CHRONICLE_NAMESPACES.get(namespace), + ) + + +def declared_identity(identity: object) -> DeclaredIdentity | None: + """The declaration for an identity string, or ``None`` if undeclared.""" + if not isinstance(identity, str) or not identity: + return None + namespace, _, remainder = identity.partition(".") + family, _, version = remainder.rpartition(".") + if not namespace or not family: + return None + return DECLARED_IDENTITY_REGISTRY.get((namespace, family, version)) + + +def fact_key_epoch(key: object) -> str | None: + """The epoch that *declared* ``key``'s domain, or ``None``. + + ``None`` covers both a foreign namespace and a Chronicle namespace whose + exact spelling was never declared; :func:`fact_key_epoch_label` + distinguishes them. + """ + identity = parse_fact_key(key) + return None if identity is None else identity.epoch + + +def fact_key_epoch_label(key: object) -> str | None: + """``"ledger"``, ``"chronicle"``, :data:`UNDECLARED`, or ``None``.""" + identity = parse_fact_key(key) + return None if identity is None else identity.epoch_label + + +def is_chronicle_fact_key(key: object) -> bool: + """True when ``key`` is a *declared* Chronicle key of either epoch. + + An undeclared ``chronicle.*`` spelling is deliberately false here: it may + turn out to be Chronicle's, but this build has not been told so, and a + validator that answered yes would be treating a guess as identity. + """ + return fact_key_epoch(key) is not None + + +def row_fact_key_epochs(row: object) -> frozenset[str]: + """Every epoch label appearing in one consumer-fact row's Chronicle keys. + + Contains :data:`UNDECLARED` when the row carries a key in a Chronicle + namespace whose spelling this consumer does not declare. Empty for a row + whose keys are all Microcosm-minted: those namespaces are outside both + eras by design. + """ + return frozenset(label for label, _ in _row_labelled_keys(row)) + + +def feed_undeclared_fact_key_domains(rows: Iterable[Any]) -> tuple[str, ...]: + """Sorted Chronicle-namespace key domains the declaration table lacks. + + Reported alongside the epochs so a release manifest names *what* it did + not recognise, rather than only that something went unrecognised. + """ + undeclared: set[str] = set() + for row in rows: + for label, domain in _row_labelled_keys(row): + if label == UNDECLARED: + undeclared.add(domain) + return tuple(sorted(undeclared)) + + +def feed_fact_key_epochs(rows: Iterable[Any]) -> tuple[str, ...]: + """Epoch labels observed across a whole feed, in report order. + + A mixed-epoch feed — ledger-era history beside chronicle-era rows — is + expected during the cutover window and is reported, not rejected. So is + an undeclared spelling: it appears as :data:`UNDECLARED`, never folded + into either era. + """ + observed: set[str] = set() + for row in rows: + observed |= row_fact_key_epochs(row) + return tuple(label for label in FACT_KEY_EPOCH_LABELS if label in observed) + + +def is_accepted_consumer_artifact_schema_version(value: object) -> bool: + """True for a declared consumer-artifact manifest schema id, either epoch. + + Non-string JSON — a list or object where a schema id belongs — is simply + not accepted. It must not raise here: the loader turns a rejection into + the documented unsupported-schema ``ValueError``, and a ``TypeError`` + escaping a membership test would bypass that message entirely. + """ + return ( + isinstance(value, str) and value in ACCEPTED_CONSUMER_ARTIFACT_SCHEMA_VERSIONS + ) + + +def is_accepted_consumer_fact_schema_version(value: object) -> bool: + """True for a declared per-row consumer-fact schema id, either epoch.""" + return isinstance(value, str) and value in ACCEPTED_CONSUMER_FACT_SCHEMA_VERSIONS + + +def consumer_artifact_schema_epoch(value: object) -> str | None: + """The epoch of a consumer-artifact schema id, or ``None`` if undeclared.""" + return _schema_epoch(value, CONSUMER_ARTIFACT_SCHEMA_IDENTITY_KIND) + + +def consumer_fact_schema_epoch(value: object) -> str | None: + """The epoch of a per-row consumer-fact schema id, or ``None``.""" + return _schema_epoch(value, CONSUMER_FACT_SCHEMA_IDENTITY_KIND) + + +def describe_accepted_consumer_artifact_schema_versions() -> str: + """Every accepted artifact schema id, oldest era first, for messages.""" + return _describe(CONSUMER_ARTIFACT_SCHEMA_IDENTITY_KIND) + + +def describe_accepted_consumer_fact_schema_versions() -> str: + """Every accepted per-row fact schema id, oldest era first, for messages.""" + return _describe(CONSUMER_FACT_SCHEMA_IDENTITY_KIND) + + +def _describe(kind: str) -> str: + return ", ".join(repr(identity) for identity in _identities_of(kind)) + + +def _schema_epoch(value: object, kind: str) -> str | None: + declaration = declared_identity(value) + if declaration is None or declaration.kind != kind: + return None + return declaration.epoch + + +def _row_labelled_keys(row: object) -> tuple[tuple[str, str], ...]: + """``(epoch label, key domain)`` for every Chronicle key in one row.""" + if not isinstance(row, Mapping): + return () + labelled: list[tuple[str, str]] = [] + + def record(value: object) -> None: + identity = parse_fact_key(value) + if identity is None or identity.epoch_label is None: + return + labelled.append((identity.epoch_label, identity.domain)) + + for field in FACT_KEY_FIELDS: + record(row.get(field)) + for path in _FACT_KEY_NESTED_PATHS: + record(_at(row, path)) + for path in _FACT_KEY_LIST_PATHS: + values = _at(row, path) + if isinstance(values, Iterable) and not isinstance(values, str | bytes): + for value in values: + record(value) + return tuple(labelled) + + +def _at(row: Mapping[str, Any], path: tuple[str, ...]) -> Any: + current: Any = row + for segment in path: + if not isinstance(current, Mapping): + return None + current = current.get(segment) + return current diff --git a/packages/microcosm-build/src/microcosm/build/ledger_artifact.py b/packages/microcosm-build/src/microcosm/build/ledger_artifact.py index ff9991a5e..815da04a7 100644 --- a/packages/microcosm-build/src/microcosm/build/ledger_artifact.py +++ b/packages/microcosm-build/src/microcosm/build/ledger_artifact.py @@ -1,16 +1,39 @@ -"""Pinned loading for PolicyEngine Ledger consumer artifacts. +"""Pinned loading for PolicyEngine Chronicle consumer artifacts. -Ledger publishes consumer artifacts: a directory with ``manifest.json`` -(schema version, content hashes, embedded profile hashes), +Chronicle (formerly Ledger) publishes consumer artifacts: a directory with +``manifest.json`` (schema version, content hashes, embedded profile hashes), ``consumer_facts.jsonl``, per-profile JSON, and coverage diagnostics. A Microcosm build should consume facts through this loader so the release -manifest can record exactly which Ledger data resolved its target values +manifest can record exactly which Chronicle data resolved its target values (PolicyEngine/microcosm#160, #271) and so tampered or mismatched feeds fail before they calibrate anything. -Like the rest of Microcosm's Ledger consumption, this module is duck-typed -against the published artifact contract (stdlib only); it does not import -the Ledger implementation package. +Like the rest of Microcosm's Chronicle consumption, this module is duck-typed +against the published artifact contract (stdlib only); it does not import the +Chronicle implementation package. + +**Every declared era loads.** The manifest's ``schema_version`` is checked +for membership in :data:`ACCEPTED_CONSUMER_ARTIFACT_SCHEMA_VERSIONS`, never +for equality with one era, so an artifact published after Chronicle's rename +cutover loads here without a code change — see +:mod:`microcosm.build.chronicle_epoch` and PolicyEngine/chronicle#143. Three +spellings are accepted, not two: Microcosm's own minted +``policyengine_ledger.consumer_artifact.v1``, the +``policyengine_ledger.consumer_artifact.v2`` Chronicle's ``main`` emits +today, and the chronicle-era +``policyengine_chronicle.consumer_artifact.v3`` successor. + +The observed id, its epoch, the fact-key epochs present in the feed, any +Chronicle-namespace key domain the declaration table does not carry, and the +per-row schema ids the rows themselves declare are all recorded in +:meth:`LedgerConsumerArtifact.provenance`, so a release manifest witnesses +which era it actually consumed rather than which era it assumed. + +Acceptance widened here; nothing narrowed. Only the *manifest* schema id is +gated, and only against the declared ids. Row-level ids and fact keys are +carried as published — real feeds mint rows in namespaces that belong to +neither era, and a consumer that rejected them would fail closed on data that +has always loaded. """ from __future__ import annotations @@ -22,7 +45,18 @@ from pathlib import Path from typing import Any +from microcosm.build.chronicle_epoch import ( + ACCEPTED_CONSUMER_ARTIFACT_SCHEMA_VERSIONS, + MICROCOSM_CONSUMER_ARTIFACT_SCHEMA_VERSION, + consumer_artifact_schema_epoch, + describe_accepted_consumer_artifact_schema_versions, + feed_fact_key_epochs, + feed_undeclared_fact_key_domains, + is_accepted_consumer_artifact_schema_version, +) + __all__ = [ + "ACCEPTED_CONSUMER_ARTIFACT_SCHEMA_VERSIONS", "ALLOWED_LEDGER_ASSERTIONS", "CONSUMER_ARTIFACT_SCHEMA_VERSION", "DEFAULT_LEDGER_ASSERTION", @@ -32,21 +66,30 @@ "resolve_ledger_artifact", ] -CONSUMER_ARTIFACT_SCHEMA_VERSION = "policyengine_ledger.consumer_artifact.v1" +#: The id Microcosm's own minted artifacts still declare. Frozen at v1 by +#: microcosm#639; loading is governed by the accepted *set*, not by this. It +#: is deliberately *not* the id Chronicle emits — Chronicle's ``main`` stamps +#: ``policyengine_ledger.consumer_artifact.v2``, which the set also accepts. +CONSUMER_ARTIFACT_SCHEMA_VERSION = MICROCOSM_CONSUMER_ARTIFACT_SCHEMA_VERSION ALLOWED_LEDGER_ASSERTIONS = frozenset(("observation", "source_projection")) DEFAULT_LEDGER_ASSERTION = "observation" @dataclass(frozen=True) class LedgerConsumerArtifact: - """A loaded, hash-verified Ledger consumer fact feed. + """A loaded, hash-verified Chronicle consumer fact feed. ``manifest`` and ``manifest_sha256`` are ``None`` when the feed was a bare ``consumer_facts.jsonl`` file rather than an artifact directory; bare feeds are still content-addressed by ``facts_sha256`` so builds can - pin them, but they carry no Ledger-side provenance. Rows are exactly as + pin them, but they carry no Chronicle-side provenance. Rows are exactly as published: an ``assertion`` field is validated when present and never fabricated when absent (missing means observation-by-default to readers). + + The class name is ledger-era and stays: renaming an exported symbol + Microcosm's tools and experiments import buys nothing the alias in + :mod:`microcosm.build` does not, and chronicle#143 migrates identities, + not vocabulary. """ path: Path @@ -60,8 +103,65 @@ def fact_row_count(self) -> int: """Number of consumer fact rows in the feed.""" return len(self.facts) + @property + def schema_version(self) -> str | None: + """The schema id this feed actually declared, verbatim.""" + if self.manifest is None: + return None + schema_version = self.manifest.get("schema_version") + return None if schema_version is None else str(schema_version) + + @property + def schema_epoch(self) -> str | None: + """``"ledger"`` or ``"chronicle"`` for the declared schema id.""" + return consumer_artifact_schema_epoch(self.schema_version) + + @property + def fact_key_epochs(self) -> tuple[str, ...]: + """Epoch labels observed across the feed's Chronicle fact keys. + + Empty when the feed carries only Microcosm-minted keys, and both + epochs when a cutover-window feed mixes ledger-era history with + chronicle-era rows. A Chronicle-namespace key whose exact spelling is + undeclared appears as ``"undeclared"`` — reported, never folded into + an era it was not declared for. + """ + return feed_fact_key_epochs(self.facts) + + @property + def undeclared_fact_key_domains(self) -> tuple[str, ...]: + """Chronicle-namespace key domains the declaration table lacks. + + Normally empty. A non-empty tuple means the feed carried a spelling + this build was not told about, and names it, so the manifest says + *what* went unrecognised instead of only that something did. + """ + return feed_undeclared_fact_key_domains(self.facts) + + @property + def fact_schema_versions(self) -> tuple[str, ...]: + """Distinct per-row ``schema_version`` values in the feed, sorted. + + Reported verbatim and never gated on — see :func:`_load_fact_rows`. + Empty when no row declares one; more than one entry when a feed mixes + producers, which the pinned US fiscal-refresh feed already does. + """ + observed = { + str(row["schema_version"]) + for row in self.facts + if isinstance(row, dict) and row.get("schema_version") is not None + } + return tuple(sorted(observed)) + def provenance(self) -> dict[str, Any]: - """Ledger-artifact identity block for build and release manifests.""" + """Chronicle-artifact identity block for build and release manifests. + + Records the schema id as *observed*, plus the epoch it belongs to, + the epoch labels of the fact keys in the feed, and any undeclared + Chronicle-namespace domain among them, so a manifest witnesses which + era of Chronicle actually resolved its targets — and says so when the + feed carried a spelling this build did not declare. + """ payload: dict[str, Any] = { "path_name": self.path.name, "fact_row_count": self.fact_row_count, @@ -82,6 +182,10 @@ def provenance(self) -> dict[str, Any]: else: payload["schema_version"] = None payload["manifest_sha256"] = None + payload["schema_epoch"] = self.schema_epoch + payload["fact_key_epochs"] = list(self.fact_key_epochs) + payload["undeclared_fact_key_domains"] = list(self.undeclared_fact_key_domains) + payload["fact_schema_versions"] = list(self.fact_schema_versions) return payload @@ -123,11 +227,17 @@ def load_ledger_consumer_artifact( f"Ledger consumer artifact manifest must be an object: {manifest_path}" ) schema_version = manifest.get("schema_version") - if schema_version != CONSUMER_ARTIFACT_SCHEMA_VERSION: + # Membership, not equality: ledger-era and chronicle-era artifacts are + # the same contract under different names, and all of them must load + # through the rename cutover (chronicle#143). The predicate is total + # over JSON — a manifest whose schema_version is a list or an object + # is unsupported, and must say so with this message rather than + # escaping as a TypeError from a membership test. + if not is_accepted_consumer_artifact_schema_version(schema_version): raise ValueError( - "Unsupported Ledger consumer artifact schema_version " - f"{schema_version!r}; expected " - f"{CONSUMER_ARTIFACT_SCHEMA_VERSION!r}." + "Unsupported Chronicle consumer artifact schema_version " + f"{schema_version!r}; expected one of " + f"{describe_accepted_consumer_artifact_schema_versions()}." ) else: facts_path = artifact_path @@ -222,9 +332,17 @@ def _load_fact_rows(path: Path) -> tuple[dict[str, Any], ...]: ) from exc if not isinstance(row, dict): raise ValueError( - f"Invalid Ledger facts JSONL row {line_number}: expected " + f"Invalid Chronicle facts JSONL row {line_number}: expected " f"object, got {type(row).__name__}." ) + # A per-row ``schema_version`` is carried, never gated on. Real + # feeds stamp ids from namespaces that are neither era: the pinned + # US fiscal-refresh feed (consumer_facts_buildn_v9_4.jsonl) mixes + # 'arch.consumer_fact.v1' with 'ledger.consumer_fact.v1'. Rejecting + # an unrecognized id would fail the build closed on its own pinned + # input, and rejecting is not what dual acceptance asks for. The + # observed ids are reported through + # :attr:`LedgerConsumerArtifact.fact_schema_versions` instead. assertion = row.get("assertion", DEFAULT_LEDGER_ASSERTION) if assertion not in ALLOWED_LEDGER_ASSERTIONS: raise ValueError( diff --git a/packages/microcosm-build/src/microcosm/build/logbook.py b/packages/microcosm-build/src/microcosm/build/logbook.py index 5047bb77a..6cfb75d12 100644 --- a/packages/microcosm-build/src/microcosm/build/logbook.py +++ b/packages/microcosm-build/src/microcosm/build/logbook.py @@ -7,12 +7,17 @@ a best-effort Supabase REST insert. Remote availability is never part of build correctness. If either -``POPULACE_LEDGER_URL`` or ``POPULACE_LEDGER_KEY`` is absent, the validated row -stays in spool-only mode without error. Network and HTTP failures are returned -as receipt data and never raised. Local validation and durable-spool failures +``LOGBOOK_URL`` or ``LOGBOOK_KEY`` is absent, the validated row stays in +spool-only mode without error. Network and HTTP failures are returned as +receipt data and never raised. Local validation and durable-spool failures remain fatal: callers must not claim an attempt was recorded if its local row was invalid or could not be persisted. +The ledger-era spellings (``POPULACE_LEDGER_URL``, ``POPULACE_LEDGER_KEY``, +``POPULACE_LEDGER_API_KEY``) are still honored for the Logbook dual-read +window (microcosm#632) and warn once per process; see +:mod:`microcosm.build.logbook_env`. + The caller owns chain coordination and must provide ``prev_row_digest`` for the ledger head it extends. The row digest is @@ -28,7 +33,7 @@ The Supabase key must identify the migration's ``logbook_writer`` role, not the service role. Hosted Supabase projects should additionally provide the -project gateway key as ``POPULACE_LEDGER_API_KEY``; single-key deployments may +project gateway key as ``LOGBOOK_API_KEY``; single-key deployments may omit it. The ``logbook`` schema must also be enabled in the hosted project's PostgREST exposed-schema setting. The US stacked driver and the three UK drivers record through this seam; this module remains driver-agnostic. @@ -56,6 +61,14 @@ from urllib.parse import urlencode, urlsplit from urllib.request import HTTPRedirectHandler, Request, build_opener +from microcosm.build.logbook_env import ( + LEGACY_API_KEY_ENV, + LOGBOOK_API_KEY_ENV, + LOGBOOK_KEY_ENV, + LOGBOOK_URL_ENV, + logbook_env, +) + __all__ = [ "BUILD_DISPOSITIONS", "LOGBOOK_ROW_FIELDS", @@ -91,7 +104,11 @@ _DIGEST_PATTERN = re.compile(r"^[0-9a-f]{64}$") _BUILD_ID_PATTERN = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._:-]{0,254}$") LOGBOOK_RUNGS = frozenset({"f001", "f004", "f010", "f025", "f100"}) -LEDGER_API_KEY_ENV = "POPULACE_LEDGER_API_KEY" +#: Ledger-era name for the project gateway key, kept because callers and +#: operator runbooks still name it. Reads go through +#: :func:`microcosm.build.logbook_env.logbook_env`, which prefers +#: ``LOGBOOK_API_KEY`` and warns once when it falls back to this one. +LEDGER_API_KEY_ENV = LEGACY_API_KEY_ENV LOGBOOK_ROW_FIELDS = frozenset( { "build_id", @@ -820,11 +837,11 @@ def urlopen(request: Request, *, timeout: float) -> Any: def _remote_config() -> tuple[str, str, str] | None: - url = os.environ.get("POPULACE_LEDGER_URL") - key = os.environ.get("POPULACE_LEDGER_KEY") + url = logbook_env(LOGBOOK_URL_ENV) + key = logbook_env(LOGBOOK_KEY_ENV) if not url or not key: return None - api_key = os.environ.get(LEDGER_API_KEY_ENV) or key + api_key = logbook_env(LOGBOOK_API_KEY_ENV) or key return url, key, api_key diff --git a/packages/microcosm-build/src/microcosm/build/logbook_env.py b/packages/microcosm-build/src/microcosm/build/logbook_env.py new file mode 100644 index 000000000..6a7ff6f9b --- /dev/null +++ b/packages/microcosm-build/src/microcosm/build/logbook_env.py @@ -0,0 +1,150 @@ +"""Dual-read for the Logbook (formerly Ledger) environment variables. + +Logbook is the build ledger's own name (microcosm#632), chosen precisely to +stop colliding with Chronicle, the fact store this package also consumes. +This module gives the Logbook store's credentials — buckets, database +schema, role ids, and env names — a **dual-read window**: ``LOGBOOK_*`` is +preferred, the legacy name is still honored, and honoring it emits a +deprecation warning so publish flows and build scripts migrate on their own +schedule. This module is that window, in one place. + +Every variable here is read through :func:`logbook_env`, which tries the +preferred ``LOGBOOK_*`` name first and falls back to the legacy name, +warning once per process per legacy name. Nothing is renamed on disk: the +legacy names keep working for as long as the window is open, and the legacy +name constants stay exported so error messages and tests can still name them. + +Note on scope: the identifiers ``LEDGER_HMRC_BANDS``, +``LEDGER_ONS_TURNOVER_BANDS``, ``LEDGER_ONS_EMPLOYMENT_BANDS``, and +``LEDGER_US_SOURCE_COVERAGE_CONTRACT_COMMIT`` look like environment variables +to a grep but are plain Python module constants (band maps and a git commit +pin). They translate Chronicle identities and get Chronicle-named *aliases* +beside their modules rather than a dual-read, because there is no +environment to read them from and they have nothing to do with the Logbook +store this module covers. +""" + +from __future__ import annotations + +import os +import threading +import warnings +from collections.abc import Mapping + +__all__ = [ + "LEGACY_API_KEY_ENV", + "LEGACY_EXPORT_KEY_ENV", + "LEGACY_KEY_ENV", + "LEGACY_URL_ENV", + "LOGBOOK_API_KEY_ENV", + "LOGBOOK_ENV_LEGACY_NAMES", + "LOGBOOK_EXPORT_KEY_ENV", + "LOGBOOK_KEY_ENV", + "LOGBOOK_URL_ENV", + "describe_logbook_env", + "logbook_env", + "logbook_env_names", + "reset_logbook_env_deprecation_warnings", +] + +#: Preferred, Logbook-named names. +LOGBOOK_URL_ENV = "LOGBOOK_URL" +LOGBOOK_KEY_ENV = "LOGBOOK_KEY" +LOGBOOK_API_KEY_ENV = "LOGBOOK_API_KEY" +LOGBOOK_EXPORT_KEY_ENV = "LOGBOOK_EXPORT_KEY" + +#: Legacy, ledger-era names. Still honored; still named in error messages so +#: an operator running the old environment recognises what is being asked for. +LEGACY_URL_ENV = "POPULACE_LEDGER_URL" +LEGACY_KEY_ENV = "POPULACE_LEDGER_KEY" +LEGACY_API_KEY_ENV = "POPULACE_LEDGER_API_KEY" +LEGACY_EXPORT_KEY_ENV = "POPULACE_LEDGER_EXPORT_KEY" + +#: Preferred name -> legacy names, most recent legacy spelling first. Adding a +#: variable to the dual-read window means adding a row here and nothing else. +LOGBOOK_ENV_LEGACY_NAMES: Mapping[str, tuple[str, ...]] = { + LOGBOOK_URL_ENV: (LEGACY_URL_ENV,), + LOGBOOK_KEY_ENV: (LEGACY_KEY_ENV,), + LOGBOOK_API_KEY_ENV: (LEGACY_API_KEY_ENV,), + LOGBOOK_EXPORT_KEY_ENV: (LEGACY_EXPORT_KEY_ENV,), +} + +_WARNED_LEGACY_NAMES: set[str] = set() +_WARNED_LOCK = threading.Lock() + + +def logbook_env( + name: str, + default: str | None = None, + *, + environ: Mapping[str, str] | None = None, +) -> str | None: + """Read one Logbook variable, preferring ``name`` over its legacy spelling. + + ``name`` is the preferred ``LOGBOOK_*`` name. When it is unset but a + legacy name carries a value, that value is returned and a + :class:`DeprecationWarning` is emitted **once per process per legacy + name** — repeated reads in a build loop must not turn into a warning + storm. An empty value is treated as unset, matching how the callers here + already test these variables. + """ + source = os.environ if environ is None else environ + if name not in LOGBOOK_ENV_LEGACY_NAMES: + raise KeyError( + f"{name!r} is not a Logbook environment variable; expected one of " + f"{sorted(LOGBOOK_ENV_LEGACY_NAMES)}." + ) + value = source.get(name) + if value: + return value + for legacy_name in LOGBOOK_ENV_LEGACY_NAMES[name]: + legacy_value = source.get(legacy_name) + if legacy_value: + _warn_once(legacy_name, preferred=name) + return legacy_value + return default + + +def logbook_env_names(name: str) -> tuple[str, ...]: + """The preferred name followed by every legacy name still honored.""" + if name not in LOGBOOK_ENV_LEGACY_NAMES: + raise KeyError( + f"{name!r} is not a Logbook environment variable; expected one of " + f"{sorted(LOGBOOK_ENV_LEGACY_NAMES)}." + ) + return (name, *LOGBOOK_ENV_LEGACY_NAMES[name]) + + +def describe_logbook_env(*names: str) -> str: + """Render required variables for an error message, legacy names included. + + Error text names both spellings on purpose: an operator whose environment + predates the rename must still be able to match the message against what + they have set. + """ + preferred = ", ".join(names) + legacy = ", ".join( + legacy_name for name in names for legacy_name in LOGBOOK_ENV_LEGACY_NAMES[name] + ) + return f"{preferred} (legacy {legacy} still honored)" + + +def reset_logbook_env_deprecation_warnings() -> None: + """Forget which legacy names have warned. For tests only.""" + with _WARNED_LOCK: + _WARNED_LEGACY_NAMES.clear() + + +def _warn_once(legacy_name: str, *, preferred: str) -> None: + with _WARNED_LOCK: + if legacy_name in _WARNED_LEGACY_NAMES: + return + _WARNED_LEGACY_NAMES.add(legacy_name) + warnings.warn( + f"{legacy_name} is the pre-rename name for {preferred}; the build " + "ledger is now Logbook (microcosm#632). Set " + f"{preferred} instead — {legacy_name} stays honored only for the " + "dual-read window.", + DeprecationWarning, + stacklevel=3, + ) diff --git a/packages/microcosm-build/src/microcosm/build/uk_runtime/calibration_run.py b/packages/microcosm-build/src/microcosm/build/uk_runtime/calibration_run.py index 6864a9eff..527961aa0 100644 --- a/packages/microcosm-build/src/microcosm/build/uk_runtime/calibration_run.py +++ b/packages/microcosm-build/src/microcosm/build/uk_runtime/calibration_run.py @@ -919,23 +919,62 @@ def uk_aggregate_admin_totals( return totals, receipt +#: The manifest fields the UK run seals into ``run_config``. Narrower than +#: the artifact's whole manifest on purpose: only the fields that identify +#: *which* published artifact was compiled belong in the identity digest. +_LEDGER_MANIFEST_IDENTITY_FIELDS = ( + "artifact_id", + "profile", + "schema_version", + "generated_at", +) + +#: Epoch witnesses :meth:`LedgerConsumerArtifact.provenance` supplies, split +#: by shape. Named here so the delegation below cannot quietly drop one: a +#: run that compiled a chronicle-era or mixed-epoch feed has to say so in its +#: own evidence, not only in the loader's return value +#: (PolicyEngine/chronicle#143). +_LEDGER_EPOCH_WITNESS_SCALARS = ("schema_epoch",) +_LEDGER_EPOCH_WITNESS_LISTS = ( + "fact_key_epochs", + "undeclared_fact_key_domains", + "fact_schema_versions", +) + + def _ledger_provenance(artifact: Any) -> dict[str, object]: - """The verified identity of the Ledger consumer feed this run compiled. + """The verified identity of the Chronicle consumer feed this run compiled. + + Delegates to :meth:`microcosm.build.ledger_artifact.LedgerConsumerArtifact.provenance` + rather than rebuilding the block field by field. Rebuilding it is how the + epoch witnesses went missing here in the first place: the loader learned + which era resolved the targets and the UK run kept reporting only the + hashes. Anything the shared block gains, this block gains. + + Only the manifest sub-block is UK-shaped, and it stays narrow because it + feeds the run's identity digest. A bare ``consumer_facts.jsonl`` feed carries no manifest, so its - Ledger-side provenance is recorded as absent rather than invented. + Chronicle-side provenance is recorded as absent rather than invented. So + is a stand-in that predates the shared block: the fields it cannot supply + are recorded ``None``/empty rather than fabricated. """ + shared = getattr(artifact, "provenance", None) + block: Mapping[str, Any] = shared() if callable(shared) else {} provenance: dict[str, object] = { - "facts_sha256": getattr(artifact, "facts_sha256", None), - "fact_row_count": getattr(artifact, "fact_row_count", None), - "manifest_sha256": getattr(artifact, "manifest_sha256", None), + field: block.get(field, getattr(artifact, field, None)) + for field in ("facts_sha256", "fact_row_count", "manifest_sha256") } + for field in _LEDGER_EPOCH_WITNESS_SCALARS: + provenance[field] = block.get(field) + for field in _LEDGER_EPOCH_WITNESS_LISTS: + provenance[field] = list(block.get(field) or ()) manifest = getattr(artifact, "manifest", None) if isinstance(manifest, Mapping): provenance["manifest"] = { key: manifest.get(key) - for key in ("artifact_id", "profile", "schema_version", "generated_at") + for key in _LEDGER_MANIFEST_IDENTITY_FIELDS if manifest.get(key) is not None } return provenance diff --git a/packages/microcosm-build/src/microcosm/build/uk_runtime/firm_generation.py b/packages/microcosm-build/src/microcosm/build/uk_runtime/firm_generation.py index 8e4da91cb..7df3399c8 100644 --- a/packages/microcosm-build/src/microcosm/build/uk_runtime/firm_generation.py +++ b/packages/microcosm-build/src/microcosm/build/uk_runtime/firm_generation.py @@ -159,6 +159,16 @@ class UKFirmLedgerTargetProfile: "greater_than_10m": "Greater_than_£10m", } +# Chronicle-era spellings of the three band maps above. These identifiers only +# look like environment variables to a grep — they are plain support-layout +# data, so they take aliases rather than the dual-read window in +# :mod:`microcosm.build.logbook_env`. The ledger-era names stay: they are what +# the generator below imports, and the value ids they translate are published +# Chronicle ids, unaffected by the rename. +CHRONICLE_ONS_TURNOVER_BANDS = LEDGER_ONS_TURNOVER_BANDS +CHRONICLE_ONS_EMPLOYMENT_BANDS = LEDGER_ONS_EMPLOYMENT_BANDS +CHRONICLE_HMRC_BANDS = LEDGER_HMRC_BANDS + HMRC_BAND_COLUMNS: tuple[str, ...] = ( "Negative_or_Zero", "£1_to_Threshold", diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/__init__.py b/packages/microcosm-build/src/microcosm/build/us_runtime/__init__.py index 518baacb6..9437b99e8 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/__init__.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/__init__.py @@ -885,6 +885,7 @@ with_us_snap_take_up_inputs, ) from microcosm.build.us_runtime.source_coverage import ( + CHRONICLE_US_SOURCE_COVERAGE_CONTRACT_COMMIT, LEDGER_US_SOURCE_COVERAGE_CONTRACT_COMMIT, US_SOURCE_COVERAGE, hard_target_package_aliases, @@ -1107,6 +1108,7 @@ "SimpleTaxExpenditureReform", "ReformValidationSpec", "REFORM_VALIDATION_SCHEMA_VERSION", + "CHRONICLE_US_SOURCE_COVERAGE_CONTRACT_COMMIT", "LEDGER_US_SOURCE_COVERAGE_CONTRACT_COMMIT", "AgeBand", "AGE_BANDS", diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/source_coverage.py b/packages/microcosm-build/src/microcosm/build/us_runtime/source_coverage.py index 76214ad50..d37e3878e 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/source_coverage.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/source_coverage.py @@ -18,6 +18,7 @@ from microcosm.build.gates import GateResult, source_coverage_gate __all__ = [ + "CHRONICLE_US_SOURCE_COVERAGE_CONTRACT_COMMIT", "LEDGER_US_SOURCE_COVERAGE_CONTRACT_COMMIT", "CoverageRole", "SourceCoverageEntry", @@ -31,6 +32,13 @@ ] LEDGER_US_SOURCE_COVERAGE_CONTRACT_COMMIT = "e2fc882c35f9203c788a7159e7b08b0b5e5ceb58" +#: Chronicle-era spelling of the same pin. This identifier only *looks* like an +#: environment variable to a grep — it is a git commit of the source-coverage +#: contract, so it takes an alias rather than the dual-read window in +#: :mod:`microcosm.build.logbook_env`. The ledger-era name stays exported: it +#: is what callers import, and the ``ledger_commit`` diagnostic field it feeds +#: is frozen at v1 (microcosm#639). +CHRONICLE_US_SOURCE_COVERAGE_CONTRACT_COMMIT = LEDGER_US_SOURCE_COVERAGE_CONTRACT_COMMIT CoverageRole = Literal["hard_target", "validation_only", "source_gap"] diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/us_trade/import_entry_facts.py b/packages/microcosm-build/src/microcosm/build/us_runtime/us_trade/import_entry_facts.py index fd953f178..3525b9234 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/us_trade/import_entry_facts.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/us_trade/import_entry_facts.py @@ -1,6 +1,6 @@ -"""Ledger consumer-artifact emission for US import-entry margins. +"""Chronicle consumer-artifact emission for US import-entry margins. -Microcosm's ledger leg consumes sha-pinned consumer artifacts +Microcosm's Chronicle leg consumes sha-pinned consumer artifacts (``manifest.json`` + ``consumer_facts.jsonl``, :mod:`microcosm.build.ledger_artifact`) whose rows follow the ``ledger.consumer_fact.v1`` shape that :mod:`microcosm.build.ledger_targets` @@ -8,6 +8,19 @@ margin series to that exact contract from the Census ingest and archived CBP statistics. +The contract has a chronicle-era spelling too (``chronicle.consumer_fact.v2`` +under ``policyengine_chronicle.consumer_artifact.v3``, chronicle#143), and the +loader accepts every declared spelling. What this module *emits* stays +ledger-era, and specifically stays at +``policyengine_ledger.consumer_artifact.v1`` — the id Microcosm's own minted +artifacts have always declared, which is a generation behind the +``...consumer_artifact.v2`` Chronicle itself publishes. The rows are +byte-pinned goldens, and they are minted from official source bytes rather +than derived from a Chronicle row whose epoch they would have to inherit. +:func:`write_consumer_artifact` takes the declared id as an argument, checked +for membership in the declared set, so flipping the declaration later is a +caller change rather than a code change. + Producer identity is explicit: these rows are **microcosm-minted** from official source bytes — not an export of a PolicyEngine/ledger build — and every row's ``source.extraction_method`` and the artifact manifest's @@ -39,6 +52,13 @@ import pandas as pd +from microcosm.build.chronicle_epoch import ( + ACCEPTED_CONSUMER_ARTIFACT_SCHEMA_VERSIONS, + LEDGER_CONSUMER_FACT_SCHEMA_VERSION, + MICROCOSM_CONSUMER_ARTIFACT_SCHEMA_VERSION, + describe_accepted_consumer_artifact_schema_versions, + is_accepted_consumer_artifact_schema_version, +) from microcosm.build.us_runtime.us_trade.cbp_entry_stats import ( CBP_TRADE_STATS_URL, CbpEntryStats, @@ -50,6 +70,7 @@ from microcosm.build.us_runtime.us_trade.imdb_bulk import IMDB_URL_TEMPLATE __all__ = [ + "ACCEPTED_CONSUMER_ARTIFACT_SCHEMA_VERSIONS", "ALL_IMPORT_ENTRY_FACT_GRAINS", "CONSUMER_ARTIFACT_SCHEMA_VERSION", "DISTRICT_ENTRY_FACT_GRAIN", @@ -138,8 +159,17 @@ class FactSourceLeg: ), ) -CONSUMER_ARTIFACT_SCHEMA_VERSION = "policyengine_ledger.consumer_artifact.v1" -_FACT_SCHEMA_VERSION = "ledger.consumer_fact.v1" +#: The schema ids these microcosm-minted rows *declare*. Both stay ledger-era: +#: the emitted bytes are golden-pinned (microcosm#639) and these facts are +#: minted from Census/CBP source bytes, so there is no chronicle-era source row +#: whose epoch they could inherit. Chronicle's own cutover moves what Chronicle +#: emits, not what Microcosm mints. Acceptance is the separate, dual-era +#: question, and it lives in :mod:`microcosm.build.chronicle_epoch`. +CONSUMER_ARTIFACT_SCHEMA_VERSION = MICROCOSM_CONSUMER_ARTIFACT_SCHEMA_VERSION +_FACT_SCHEMA_VERSION = LEDGER_CONSUMER_FACT_SCHEMA_VERSION +#: Microcosm's own key namespace: outside both Chronicle eras by construction, +#: so these keys can never collide with or impersonate Chronicle-built keys — +#: and a Chronicle-era cutover cannot silently re-identify them. _KEY_NAMESPACE = "populace_us_trade" #: Margins-feed grains (the aggregations of the HTS10 × country margins @@ -583,15 +613,26 @@ def write_consumer_artifact( *, retrieval_manifest: Iterable[Mapping[str, Any]], generator: Mapping[str, Any], + schema_version: str = CONSUMER_ARTIFACT_SCHEMA_VERSION, ) -> dict[str, Any]: """Write ``consumer_facts.jsonl`` + ``manifest.json`` and return the manifest. The directory loads through :func:`microcosm.build.ledger_artifact.load_ledger_consumer_artifact`, which re-hashes the fact file against ``manifest.facts_sha256``. + + ``schema_version`` defaults to the ledger-era id these artifacts have + always declared and is validated for membership in the declared set, so + the chronicle-era cutover is a caller decision rather than an edit here. """ if not fact_rows: raise ValueError("Refusing to write an empty consumer artifact.") + if not is_accepted_consumer_artifact_schema_version(schema_version): + raise ValueError( + f"Refusing to write a consumer artifact declaring schema_version " + f"{schema_version!r}; expected one of " + f"{describe_accepted_consumer_artifact_schema_versions()}." + ) retrieval_entries = [dict(entry) for entry in retrieval_manifest] if not retrieval_entries: raise ValueError( @@ -621,7 +662,7 @@ def write_consumer_artifact( facts_path.write_bytes(payload) retrievals = retrieval_entries manifest = { - "schema_version": CONSUMER_ARTIFACT_SCHEMA_VERSION, + "schema_version": schema_version, "facts_sha256": hashlib.sha256(payload).hexdigest(), "fact_row_count": len(fact_rows), "profiles": {}, diff --git a/packages/microcosm-build/tests/conftest.py b/packages/microcosm-build/tests/conftest.py index ac3f6b9dc..cffd6547f 100644 --- a/packages/microcosm-build/tests/conftest.py +++ b/packages/microcosm-build/tests/conftest.py @@ -1,5 +1,7 @@ -"""Shared fixtures: a small person+household frame, and an ODS writer.""" +"""Shared fixtures: a small person+household frame, an ODS writer, and the +autouse guard that keeps every test in this shard away from a live Logbook.""" +import sys from pathlib import Path from zipfile import ZIP_DEFLATED, ZIP_STORED, ZipFile @@ -104,3 +106,111 @@ class ODSBuilder: def ods() -> ODSBuilder: """Helpers for writing a small ODS file to a temporary path.""" return ODSBuilder() + + +# --- Logbook isolation -------------------------------------------------- +# +# The Logbook live store is append-only: a row that reaches it cannot be +# taken back. These tests must never be able to reach it, and "must never" +# has to hold in a shell that already has an operator's credentials +# exported — a developer who has just run an export, or a runner where the +# variables are set for a later job. Per-module ``delenv`` lists did not +# hold: they were written before the ``LOGBOOK_*`` spelling existed, so a +# shell carrying the new names walked straight past them into +# ``_remote_config()`` and out to Supabase. +# +# Two independent guards, because either one alone is a single point of +# failure: the environment is cleared for every test, and the Logbook +# network call is replaced with one that raises. A test that means to +# exercise the remote path stubs ``urlopen`` itself, as several already do; +# a test that means to reach the real network asks for +# ``allow_logbook_network`` and says so in its own body. + + +def _logbook_environment_names() -> tuple[str, ...]: + """Every environment variable that can point tests at a live Logbook. + + Both generations of the dual-read window, taken from the window itself + rather than restated, so adding a variable there cannot leave a hole + here. ``POPULACE_LOGBOOK_PREV_ROW_DIGEST`` is not in the window (it has + only ever had one spelling) but it is read from the environment and it + changes which chain a row claims to extend, so it is cleared too. + """ + from microcosm.build.logbook_env import LOGBOOK_ENV_LEGACY_NAMES + + names = {"POPULACE_LOGBOOK_PREV_ROW_DIGEST"} + for preferred, legacy_names in LOGBOOK_ENV_LEGACY_NAMES.items(): + names.add(preferred) + names.update(legacy_names) + return tuple(sorted(names)) + + +class UnstubbedLogbookNetworkError(AssertionError): + """Raised when a test reaches the Logbook live store for real.""" + + +def _refuse_logbook_network(*_args: object, **_kwargs: object) -> None: + raise UnstubbedLogbookNetworkError( + "A test tried to open a real Logbook HTTP request. The live store is " + "append-only, so this is never a harmless mistake. Stub the module's " + "'urlopen' (see test_logbook.py) to exercise the remote path, or " + "request the 'allow_logbook_network' fixture if the network is " + "genuinely the thing under test." + ) + + +def _logbook_urlopen_holders(real_urlopen: object) -> list[object]: + """Every imported module whose ``urlopen`` is the Logbook one. + + ``tools/logbook.py`` binds the function at import + (``from microcosm.build.logbook import urlopen``), so patching the + defining module alone leaves the CLI's copy live. Rather than name the + importers — the next one would be missed — this finds every module + currently holding the same object. + """ + return [ + module + for module in list(sys.modules.values()) + if getattr(module, "urlopen", None) is real_urlopen + ] + + +@pytest.fixture(autouse=True) +def _isolate_logbook(monkeypatch: pytest.MonkeyPatch) -> dict[str, object]: + """Detach every test from any live Logbook the shell may point at. + + Returns the callables it displaced, so the opt-in fixture below can put + the real ones back without having to guess what they were. + """ + from microcosm.build import logbook + + for name in _logbook_environment_names(): + monkeypatch.delenv(name, raising=False) + + displaced = { + "urlopen": logbook.urlopen, + "open": logbook._NO_REDIRECT_OPENER.open, + } + for module in _logbook_urlopen_holders(displaced["urlopen"]): + monkeypatch.setattr(module, "urlopen", _refuse_logbook_network) + # The opener is what a surviving reference to the real ``urlopen`` would + # ultimately call, so guarding it closes the path even for a caller this + # fixture did not find. + monkeypatch.setattr(logbook._NO_REDIRECT_OPENER, "open", _refuse_logbook_network) + return displaced + + +@pytest.fixture +def allow_logbook_network( + monkeypatch: pytest.MonkeyPatch, _isolate_logbook: dict[str, object] +) -> None: + """Opt back in to real Logbook HTTP. Nothing in this suite should need it. + + It exists so the guard above is a policy with a documented exception + rather than a wall, and so the guard itself can be tested. + """ + from microcosm.build import logbook + + for module in _logbook_urlopen_holders(_refuse_logbook_network): + monkeypatch.setattr(module, "urlopen", _isolate_logbook["urlopen"]) + monkeypatch.setattr(logbook._NO_REDIRECT_OPENER, "open", _isolate_logbook["open"]) diff --git a/packages/microcosm-build/tests/test_chronicle_epoch.py b/packages/microcosm-build/tests/test_chronicle_epoch.py new file mode 100644 index 000000000..53d5516bb --- /dev/null +++ b/packages/microcosm-build/tests/test_chronicle_epoch.py @@ -0,0 +1,701 @@ +"""Dual-era acceptance of Chronicle (formerly Ledger) identities. + +Chronicle introduces chronicle-era hash domains and chronicle-named schema +ids for newly emitted rows at a declared cutover; ledger-era ids stay valid +history forever (PolicyEngine/chronicle#143). Microcosm consumes both, and +these tests are the contract that says so: no validator may reject a row for +being on the other side of the cutover, and no minted Microcosm identity may +move because a source row crossed it. + +The other half of the contract is that acceptance follows a *declaration*. +An identity Microcosm has not been told about is reported as undeclared, not +waved through as Chronicle's — a consumer that inferred issued identity from +the shape of a string would accept anything that happened to be spelled like +a Chronicle key. +""" + +from __future__ import annotations + +import hashlib +import json +import re +from pathlib import Path + +import pytest + +from microcosm.build.chronicle_epoch import ( + ACCEPTED_CONSUMER_ARTIFACT_SCHEMA_VERSIONS, + ACCEPTED_CONSUMER_FACT_SCHEMA_VERSIONS, + CHRONICLE_CONSUMER_ARTIFACT_SCHEMA_VERSION, + CHRONICLE_CONSUMER_FACT_SCHEMA_VERSION, + CHRONICLE_EPOCH, + CHRONICLE_FACT_KEY_DOMAINS, + CHRONICLE_NAMESPACES, + DECLARED_IDENTITIES, + DECLARED_IDENTITY_EPOCHS, + EPOCHS, + FACT_KEY_FIELDS, + FACT_KEY_IDENTITY_KIND, + LEDGER_CONSUMER_FACT_SCHEMA_VERSION, + LEDGER_EPOCH, + LEDGER_FACT_KEY_DOMAINS, + MICROCOSM_CONSUMER_ARTIFACT_SCHEMA_VERSION, + PUBLISHED_CONSUMER_ARTIFACT_SCHEMA_VERSION, + UNDECLARED, + consumer_artifact_schema_epoch, + consumer_fact_schema_epoch, + declared_identity, + fact_key_epoch, + fact_key_epoch_label, + feed_fact_key_epochs, + feed_undeclared_fact_key_domains, + is_accepted_consumer_artifact_schema_version, + is_accepted_consumer_fact_schema_version, + is_chronicle_fact_key, + parse_fact_key, + row_fact_key_epochs, +) +from microcosm.build.ledger_artifact import load_ledger_consumer_artifact + +#: A chronicle-era row as the migration spec describes it: identical canonical +#: payload, ``chronicle.*.v3`` domain in place of ``ledger.*.v2``. +_CHRONICLE_AGGREGATE_KEY = "chronicle.aggregate_fact.v3:abc123" +_LEDGER_AGGREGATE_KEY = "ledger.aggregate_fact.v2:abc123" + +#: Every consumer-artifact manifest id a real producer stamps. Microcosm's +#: own minted v1 and Chronicle's published v2 are both ledger-era; v3 is the +#: chronicle-era successor. +_ARTIFACT_SCHEMA_IDS_BY_EPOCH = ( + (MICROCOSM_CONSUMER_ARTIFACT_SCHEMA_VERSION, LEDGER_EPOCH), + (PUBLISHED_CONSUMER_ARTIFACT_SCHEMA_VERSION, LEDGER_EPOCH), + (CHRONICLE_CONSUMER_ARTIFACT_SCHEMA_VERSION, CHRONICLE_EPOCH), +) + + +def _fact_row(**overrides): + row = { + "aggregate_fact_key": _LEDGER_AGGREGATE_KEY, + "semantic_fact_key": "ledger.semantic_fact.v2:abc123", + "value": 100, + "period": {"type": "tax_year", "value": 2023}, + "geography": {"level": "country", "id": "0100000US"}, + "entity": {"name": "tax_unit"}, + "aggregation": {"method": "sum"}, + "observed_measure": {"source_name": "irs_soi", "unit": "usd"}, + "source": {"source_name": "irs_soi"}, + "lineage": { + "source_record_id": "irs_soi.ty2023.t.all.agi", + "source_cell_keys": ["ledger.source_cell.v1:cell"], + }, + } + row.update(overrides) + return row + + +def _chronicle_fact_row(**overrides): + """The same row after Chronicle's cutover: every key domain re-epoched. + + Each successor spelling comes from the declaration table rather than + being written out here, so a row this fixture builds is one the loader + recognises as chronicle-era rather than one it reports as undeclared. + """ + row = _fact_row( + aggregate_fact_key=_CHRONICLE_AGGREGATE_KEY, + semantic_fact_key="chronicle.semantic_fact.v3:abc123", + ) + row["lineage"] = dict(row["lineage"]) + row["lineage"]["source_cell_keys"] = [ + CHRONICLE_FACT_KEY_DOMAINS["source_cell"] + ":cell" + ] + row.update(overrides) + return row + + +#: The captured Chronicle feed this repo carries. Its rows are ledger-era and +#: frozen: they are the only published Chronicle rows available in-tree. +_CAPTURED_FEED = ( + Path(__file__).parent / "fixtures" / "uk_target_reference_feed_rows.jsonl" +) + + +def _captured_feed_rows() -> list[dict]: + return [ + json.loads(line) + for line in _CAPTURED_FEED.read_text().splitlines() + if line.strip() + ] + + +def _keys_in(value, _pattern=re.compile(r"^[a-z][a-z0-9_]*(?:\.[a-z0-9_]+)+\.v\d+:")): + """Every Chronicle-shaped key anywhere in a row, at any depth.""" + if isinstance(value, str): + return [value] if _pattern.match(value) else [] + if isinstance(value, dict): + return [key for item in value.values() for key in _keys_in(item)] + if isinstance(value, list): + return [key for item in value for key in _keys_in(item)] + return [] + + +def _write_artifact_dir(tmp_path, rows, *, schema_version, name="artifact"): + artifact_dir = tmp_path / name + artifact_dir.mkdir() + payload = "".join(json.dumps(row, sort_keys=True) + "\n" for row in rows) + (artifact_dir / "consumer_facts.jsonl").write_text(payload) + manifest = { + "schema_version": schema_version, + "fact_row_count": len(rows), + "facts_sha256": hashlib.sha256(payload.encode()).hexdigest(), + "profiles": {"us_fiscal": {"sha256": "ab" * 32, "target_count": 3}}, + } + (artifact_dir / "manifest.json").write_text( + json.dumps(manifest, indent=2, sort_keys=True) + "\n" + ) + return artifact_dir + + +def test_parse_fact_key_splits_domain_namespace_family_and_version() -> None: + identity = parse_fact_key(_LEDGER_AGGREGATE_KEY) + + assert identity is not None + assert identity.domain == "ledger.aggregate_fact.v2" + assert identity.namespace == "ledger" + assert identity.family == "aggregate_fact" + assert identity.version == "v2" + assert identity.digest == "abc123" + assert identity.epoch == LEDGER_EPOCH + + +@pytest.mark.parametrize("domain", sorted(LEDGER_FACT_KEY_DOMAINS.values())) +def test_every_observed_ledger_era_domain_resolves_to_the_ledger_epoch( + domain: str, +) -> None: + assert fact_key_epoch(f"{domain}:digest") == LEDGER_EPOCH + + +def test_observed_domain_map_covers_the_captured_feed() -> None: + """Every domain a real feed carries is one the registry declares. + + ``LEDGER_FACT_KEY_DOMAINS`` is the ledger-era half of the declaration + table, transcribed from Chronicle's own hash-domain constants. This pins + it against the captured feed: a domain the repo has actually seen and the + table has not would make that feed load with an ``undeclared`` witness, + which is a review event, not a silent pass. The version numbers differ by + family (``fact`` and ``source_cell`` are v1 where the rest are v2), which + is the concrete reason a version can never be inferred from a family. + """ + observed = { + key.split(":", 1)[0] for row in _captured_feed_rows() for key in _keys_in(row) + } + + assert observed + assert observed <= set(LEDGER_FACT_KEY_DOMAINS.values()), sorted( + observed - set(LEDGER_FACT_KEY_DOMAINS.values()) + ) + + +@pytest.mark.parametrize("domain", sorted(CHRONICLE_FACT_KEY_DOMAINS.values())) +def test_every_declared_chronicle_era_domain_resolves_to_the_chronicle_epoch( + domain: str, +) -> None: + assert fact_key_epoch(f"{domain}:digest") == CHRONICLE_EPOCH + assert fact_key_epoch_label(f"{domain}:digest") == CHRONICLE_EPOCH + assert is_chronicle_fact_key(f"{domain}:digest") + + +def test_each_declared_family_pairs_one_ledger_id_with_one_chronicle_id() -> None: + """The two eras name the same families, one version apart. + + Successor spellings are declared, so this pins the shape of the + declaration rather than deriving it at call time: the source-side + families sit at v1/v2 where the derived ones sit at v2/v3, which is + precisely why a version can never be assumed from a family name. + """ + assert set(LEDGER_FACT_KEY_DOMAINS) == set(CHRONICLE_FACT_KEY_DOMAINS) + for family, ledger_domain in LEDGER_FACT_KEY_DOMAINS.items(): + chronicle_domain = CHRONICLE_FACT_KEY_DOMAINS[family] + ledger_version = int(ledger_domain.rsplit(".v", 1)[1]) + chronicle_version = int(chronicle_domain.rsplit(".v", 1)[1]) + assert chronicle_domain == f"chronicle.{family}.v{chronicle_version}" + assert chronicle_version == ledger_version + 1, family + assert LEDGER_FACT_KEY_DOMAINS["source_cell"] == "ledger.source_cell.v1" + assert CHRONICLE_FACT_KEY_DOMAINS["source_cell"] == "chronicle.source_cell.v2" + assert LEDGER_FACT_KEY_DOMAINS["aggregate_fact"] == "ledger.aggregate_fact.v2" + assert CHRONICLE_FACT_KEY_DOMAINS["aggregate_fact"] == ( + "chronicle.aggregate_fact.v3" + ) + + +@pytest.mark.parametrize( + "key", + [ + # source_cell's declared successor is v2; v3 is nobody's spelling. + "chronicle.source_cell.v3:cell", + "chronicle.source_cell.v7:cell", + "chronicle.some_family_nobody_has_declared_yet.v11:digest", + "chronicle.aggregate_fact.v4:digest", + # A ledger-era namespace can go undeclared the same way. + "ledger.aggregate_fact.v9:digest", + "ledger.invented_family.v1:digest", + # Declared, but as a *schema* id — never as a fact-key domain. + "ledger.consumer_fact.v1:digest", + ], +) +def test_undeclared_chronicle_namespace_keys_are_reported_not_issued(key: str) -> None: + """An undeclared spelling is undeclared, not Chronicle-issued identity. + + Reading the namespace segment tells you where a key *claims* to come + from, not that Chronicle issued it. Treating the claim as identity would + let ``chronicle.anything.vN`` pass as a witnessed Chronicle key, which is + the opposite of what pinning a fact key is for. The claim is still worth + surfacing, so it is labelled ``undeclared`` and reported in provenance. + """ + assert fact_key_epoch(key) is None + assert not is_chronicle_fact_key(key) + assert fact_key_epoch_label(key) == UNDECLARED + + identity = parse_fact_key(key) + assert identity is not None + assert not identity.declared + assert identity.namespace_epoch == CHRONICLE_NAMESPACES[identity.namespace] + + +def test_undeclared_domains_are_named_in_the_feed_report() -> None: + row = _fact_row(semantic_fact_key="chronicle.semantic_fact.v9:abc123") + + assert row_fact_key_epochs(row) == frozenset({LEDGER_EPOCH, UNDECLARED}) + assert feed_fact_key_epochs([row]) == (LEDGER_EPOCH, UNDECLARED) + assert feed_undeclared_fact_key_domains([row]) == ("chronicle.semantic_fact.v9",) + assert feed_undeclared_fact_key_domains([_fact_row()]) == () + + +def test_undeclared_domains_reach_the_artifact_provenance(tmp_path) -> None: + """A release manifest has to name what the build did not recognise.""" + rows = [_fact_row(), _fact_row(source_series_key="chronicle.source_series.v9:s")] + artifact_dir = _write_artifact_dir( + tmp_path, rows, schema_version=PUBLISHED_CONSUMER_ARTIFACT_SCHEMA_VERSION + ) + + provenance = load_ledger_consumer_artifact(artifact_dir).provenance() + + assert provenance["fact_key_epochs"] == [LEDGER_EPOCH, UNDECLARED] + assert provenance["undeclared_fact_key_domains"] == ["chronicle.source_series.v9"] + + +def test_the_registry_is_keyed_by_namespace_family_and_version() -> None: + """Every accepted identity comes from one reviewable table. + + The registry, not a parser, is what makes an identity Chronicle's. This + walks it end to end: each declaration round-trips through its + ``(namespace, family, version)`` key, and the epoch a namespace belongs + to always agrees with the epoch the declaration claims. + """ + assert DECLARED_IDENTITIES + for declaration in DECLARED_IDENTITIES: + key = (declaration.namespace, declaration.family, declaration.version) + assert DECLARED_IDENTITY_EPOCHS[key] == declaration.epoch + assert declared_identity(declaration.identity) is declaration + assert CHRONICLE_NAMESPACES[declaration.namespace] == declaration.epoch + assert declaration.epoch in EPOCHS + if declaration.kind == FACT_KEY_IDENTITY_KIND: + assert fact_key_epoch(f"{declaration.identity}:digest") == ( + declaration.epoch + ) + + assert declared_identity("arch.consumer_fact.v1") is None + assert declared_identity("not-an-identity") is None + assert declared_identity(None) is None + + +def test_microcosm_minted_namespaces_are_outside_both_epochs() -> None: + """Microcosm's own derived keys must not be mistaken for Chronicle's. + + They are frozen at v1 by microcosm#639 and carry no Chronicle epoch, so + the cutover cannot re-identify them. + """ + for key in ( + "microcosm.derived_fact.congressional_district_vintage.v1:deadbeef", + "microcosm.semantic_fact.congressional_district_state_total_proxy.v1:dead", + "populace_us_trade.aggregate_fact.v1:digest", + "populace_us_trade.semantic_fact.v1:digest", + ): + assert fact_key_epoch(key) is None + assert not is_chronicle_fact_key(key) + + +@pytest.mark.parametrize( + "key", + [ + "", + "ledger.aggregate_fact.v2", # no digest + "ledger.aggregate_fact:digest", # no version + "ledger:digest", # no family + ":digest", + "not a key at all", + None, + 42, + ], +) +def test_non_key_shapes_parse_to_none_rather_than_raising(key) -> None: + assert parse_fact_key(key) is None + assert fact_key_epoch(key) is None + + +def test_row_epochs_cover_every_key_field_including_nested_lists() -> None: + ledger_row = _fact_row() + chronicle_row = _chronicle_fact_row() + + assert row_fact_key_epochs(ledger_row) == frozenset({LEDGER_EPOCH}) + assert row_fact_key_epochs(chronicle_row) == frozenset({CHRONICLE_EPOCH}) + + # A row whose only chronicle-era key is a nested source-cell key still + # reports both eras: lineage keys are Chronicle identities too. + straddling = _fact_row() + straddling["lineage"] = dict(straddling["lineage"]) + straddling["lineage"]["source_cell_keys"] = [ + CHRONICLE_FACT_KEY_DOMAINS["source_cell"] + ":cell" + ] + assert row_fact_key_epochs(straddling) == frozenset({LEDGER_EPOCH, CHRONICLE_EPOCH}) + + +def test_row_epochs_read_the_concept_alignment_key() -> None: + row = _fact_row( + concept_alignment={ + "concept_alignment_key": "chronicle.concept_alignment.v3:aligned", + "relation": "source_label", + } + ) + + assert row_fact_key_epochs(row) == frozenset({LEDGER_EPOCH, CHRONICLE_EPOCH}) + + +def test_feed_epochs_report_a_mixed_feed_in_epoch_order() -> None: + assert feed_fact_key_epochs([_fact_row()]) == (LEDGER_EPOCH,) + assert feed_fact_key_epochs([_chronicle_fact_row()]) == (CHRONICLE_EPOCH,) + assert feed_fact_key_epochs([_chronicle_fact_row(), _fact_row()]) == EPOCHS + assert ( + feed_fact_key_epochs([{"aggregate_fact_key": "populace_us_trade.a.v1:x"}]) == () + ) + assert feed_fact_key_epochs([]) == () + + +def test_schema_id_membership_covers_every_real_producer_id() -> None: + """Acceptance is keyed to what producers actually stamp, not to a pair. + + Three artifact ids exist, not two. Microcosm's own minted artifacts + declare ``policyengine_ledger.consumer_artifact.v1``; Chronicle's ``main`` + stamps ``policyengine_ledger.consumer_artifact.v2`` today + (``policyengine_chronicle/consumer.py``); the chronicle-era successor is + ``policyengine_chronicle.consumer_artifact.v3``. A loader that accepted + only the first and the last would reject every artifact Chronicle + publishes right now. + """ + assert ACCEPTED_CONSUMER_ARTIFACT_SCHEMA_VERSIONS == { + MICROCOSM_CONSUMER_ARTIFACT_SCHEMA_VERSION, + PUBLISHED_CONSUMER_ARTIFACT_SCHEMA_VERSION, + CHRONICLE_CONSUMER_ARTIFACT_SCHEMA_VERSION, + } + assert ACCEPTED_CONSUMER_FACT_SCHEMA_VERSIONS == { + LEDGER_CONSUMER_FACT_SCHEMA_VERSION, + CHRONICLE_CONSUMER_FACT_SCHEMA_VERSION, + } + for schema_id, expected_epoch in _ARTIFACT_SCHEMA_IDS_BY_EPOCH: + assert is_accepted_consumer_artifact_schema_version(schema_id), schema_id + assert consumer_artifact_schema_epoch(schema_id) == expected_epoch, schema_id + assert consumer_fact_schema_epoch(LEDGER_CONSUMER_FACT_SCHEMA_VERSION) == ( + LEDGER_EPOCH + ) + assert consumer_fact_schema_epoch(CHRONICLE_CONSUMER_FACT_SCHEMA_VERSION) == ( + CHRONICLE_EPOCH + ) + assert not is_accepted_consumer_artifact_schema_version( + "policyengine_chronicle.other.v9" + ) + # Version numbers are declared, not inferred: the chronicle-era per-row id + # is v2, so v1 under the chronicle namespace is nobody's id. + assert not is_accepted_consumer_fact_schema_version("chronicle.consumer_fact.v1") + assert consumer_fact_schema_epoch("chronicle.consumer_fact.v1") is None + # A fact-key domain is a declared identity but not a schema id, and the + # kinds do not leak into one another. + assert not is_accepted_consumer_fact_schema_version("ledger.aggregate_fact.v2") + assert consumer_artifact_schema_epoch(LEDGER_CONSUMER_FACT_SCHEMA_VERSION) is None + + +def test_chronicle_main_manifest_loads_exactly_as_published(tmp_path) -> None: + """The manifest chronicle ``main`` writes today, field for field. + + Mirrors ``build_consumer_artifact`` in ``policyengine_chronicle/consumer.py``: + ``policyengine_ledger.consumer_artifact.v2`` over rows stamped + ``ledger.consumer_fact.v1``, with ``consumer_fact_schema_versions`` and + ``consumer_fact_schema_sha256`` beside the hashes. This is the artifact a + Microcosm build is handed right now, so it is the one that has to load. + """ + rows = [_fact_row(schema_version=LEDGER_CONSUMER_FACT_SCHEMA_VERSION)] + payload = "".join(json.dumps(row, sort_keys=True) + "\n" for row in rows) + artifact_dir = tmp_path / "chronicle-main" + artifact_dir.mkdir() + (artifact_dir / "consumer_facts.jsonl").write_text(payload) + (artifact_dir / "manifest.json").write_text( + json.dumps( + { + "schema_version": PUBLISHED_CONSUMER_ARTIFACT_SCHEMA_VERSION, + "consumer_fact_schema_versions": [LEDGER_CONSUMER_FACT_SCHEMA_VERSION], + "consumer_fact_schema_sha256": "cd" * 32, + "fact_row_count": len(rows), + "facts_sha256": hashlib.sha256(payload.encode()).hexdigest(), + }, + indent=2, + sort_keys=True, + ) + + "\n" + ) + + artifact = load_ledger_consumer_artifact(artifact_dir) + + assert artifact.schema_version == PUBLISHED_CONSUMER_ARTIFACT_SCHEMA_VERSION + assert artifact.schema_epoch == LEDGER_EPOCH + assert artifact.fact_schema_versions == (LEDGER_CONSUMER_FACT_SCHEMA_VERSION,) + assert artifact.fact_key_epochs == (LEDGER_EPOCH,) + assert artifact.undeclared_fact_key_domains == () + + +@pytest.mark.parametrize( + ("schema_version", "expected_epoch"), _ARTIFACT_SCHEMA_IDS_BY_EPOCH +) +def test_artifact_loads_under_each_schema_id_and_records_the_observed_one( + tmp_path, schema_version: str, expected_epoch: str +) -> None: + rows = [ + _fact_row() if expected_epoch == LEDGER_EPOCH else _chronicle_fact_row(), + ] + artifact_dir = _write_artifact_dir(tmp_path, rows, schema_version=schema_version) + + artifact = load_ledger_consumer_artifact(artifact_dir) + + assert artifact.fact_row_count == 1 + assert artifact.schema_version == schema_version + assert artifact.schema_epoch == expected_epoch + provenance = artifact.provenance() + # The id is recorded as observed, not as assumed. + assert provenance["schema_version"] == schema_version + assert provenance["schema_epoch"] == expected_epoch + assert provenance["fact_key_epochs"] == [expected_epoch] + assert ( + consumer_artifact_schema_epoch(provenance["schema_version"]) == expected_epoch + ) + + +def test_mixed_epoch_feed_loads_and_witnesses_both_eras(tmp_path) -> None: + """The cutover window: ledger-era history beside chronicle-era rows. + + Both must calibrate, and the manifest must say the feed straddled the + cutover rather than silently reporting one era. + """ + rows = [ + _fact_row(), + _chronicle_fact_row(), + # Microcosm-minted rows carry neither epoch and must not perturb it. + _fact_row( + aggregate_fact_key="populace_us_trade.aggregate_fact.v1:digest", + semantic_fact_key="populace_us_trade.semantic_fact.v1:digest", + lineage={"source_record_id": "us_trade.month_2025_01.national"}, + ), + ] + artifact_dir = _write_artifact_dir( + tmp_path, rows, schema_version=CHRONICLE_CONSUMER_ARTIFACT_SCHEMA_VERSION + ) + + artifact = load_ledger_consumer_artifact(artifact_dir) + + assert artifact.fact_row_count == 3 + assert artifact.fact_key_epochs == EPOCHS + assert artifact.provenance()["fact_key_epochs"] == list(EPOCHS) + + +def test_bare_feed_has_no_schema_epoch_but_still_reports_fact_key_epochs( + tmp_path, +) -> None: + facts_path = tmp_path / "consumer_facts.jsonl" + facts_path.write_text( + "".join( + json.dumps(row, sort_keys=True) + "\n" + for row in (_fact_row(), _chronicle_fact_row()) + ) + ) + + artifact = load_ledger_consumer_artifact(facts_path) + + provenance = artifact.provenance() + assert provenance["schema_version"] is None + assert provenance["schema_epoch"] is None + assert provenance["fact_key_epochs"] == list(EPOCHS) + + +def test_unknown_schema_id_is_rejected_naming_every_accepted_id(tmp_path) -> None: + artifact_dir = _write_artifact_dir( + tmp_path, [_fact_row()], schema_version="policyengine_chronicle.other.v9" + ) + + with pytest.raises(ValueError) as excinfo: + load_ledger_consumer_artifact(artifact_dir) + + message = str(excinfo.value) + assert "schema_version" in message + for schema_id, _epoch in _ARTIFACT_SCHEMA_IDS_BY_EPOCH: + assert schema_id in message, schema_id + + +@pytest.mark.parametrize( + "schema_version", + [ + ["policyengine_ledger.consumer_artifact.v1"], + {"id": "policyengine_ledger.consumer_artifact.v1"}, + {}, + [], + 1, + True, + None, + ], +) +def test_malformed_schema_version_types_raise_the_documented_error( + tmp_path, schema_version +) -> None: + """A JSON list or object where a schema id belongs is *unsupported*. + + ``schema_version`` arrives from an untrusted manifest, so its type is not + guaranteed. A membership test against a frozenset raises ``TypeError`` on + an unhashable value, which would escape the loader as a bare + ``unhashable type: 'list'`` instead of the message that names what the + loader accepts. The predicate is total over JSON so the loader's own + error is the one an operator sees. + """ + artifact_dir = _write_artifact_dir( + tmp_path, [_fact_row()], schema_version=schema_version + ) + + assert not is_accepted_consumer_artifact_schema_version(schema_version) + assert not is_accepted_consumer_fact_schema_version(schema_version) + assert consumer_artifact_schema_epoch(schema_version) is None + assert consumer_fact_schema_epoch(schema_version) is None + + with pytest.raises(ValueError) as excinfo: + load_ledger_consumer_artifact(artifact_dir) + + assert "Unsupported Chronicle consumer artifact schema_version" in str( + excinfo.value + ) + + +@pytest.mark.parametrize( + "row_schema_version", + [LEDGER_CONSUMER_FACT_SCHEMA_VERSION, CHRONICLE_CONSUMER_FACT_SCHEMA_VERSION], +) +def test_per_row_schema_id_is_accepted_in_either_era( + tmp_path, row_schema_version: str +) -> None: + facts_path = tmp_path / "consumer_facts.jsonl" + facts_path.write_text( + json.dumps(_fact_row(schema_version=row_schema_version), sort_keys=True) + "\n" + ) + + artifact = load_ledger_consumer_artifact(facts_path) + + assert artifact.facts[0]["schema_version"] == row_schema_version + + +def test_per_row_schema_id_is_optional(tmp_path) -> None: + # A row that declares no schema id must not acquire one: the loader + # reports what was published, and never fabricates the field. + bare = tmp_path / "bare.jsonl" + bare.write_text(json.dumps(_fact_row(), sort_keys=True) + "\n") + assert "schema_version" not in load_ledger_consumer_artifact(bare).facts[0] + + +def test_per_row_schema_id_outside_both_eras_still_loads(tmp_path) -> None: + """Dual acceptance widens what loads; it must not narrow it. + + Real feeds stamp rows from namespaces that are neither era. The pinned US + fiscal-refresh feed ``consumer_facts_buildn_v9_4.jsonl`` declares + ``arch.consumer_fact.v1`` on the overwhelming majority of its rows and + ``ledger.consumer_fact.v1`` on the rest, and it loads through this loader + on the release path. A consumer that gated the per-row id against the two + ids chronicle#143 names would fail the build closed on its own pinned + input — so the id is carried and reported, never gated. + """ + facts_path = tmp_path / "consumer_facts.jsonl" + rows = [ + _fact_row(schema_version="arch.consumer_fact.v1"), + _fact_row(schema_version=LEDGER_CONSUMER_FACT_SCHEMA_VERSION), + _chronicle_fact_row(schema_version=CHRONICLE_CONSUMER_FACT_SCHEMA_VERSION), + _fact_row(schema_version="ledger.consumer_fact.v99"), + ] + facts_path.write_text( + "".join(json.dumps(row, sort_keys=True) + "\n" for row in rows) + ) + + artifact = load_ledger_consumer_artifact(facts_path) + + assert artifact.fact_row_count == len(rows) + assert artifact.fact_schema_versions == ( + "arch.consumer_fact.v1", + CHRONICLE_CONSUMER_FACT_SCHEMA_VERSION, + LEDGER_CONSUMER_FACT_SCHEMA_VERSION, + "ledger.consumer_fact.v99", + ) + # The unrecognized ids are reported verbatim in provenance, so a release + # manifest still witnesses exactly what it consumed. + assert artifact.provenance()["fact_schema_versions"] == [ + "arch.consumer_fact.v1", + CHRONICLE_CONSUMER_FACT_SCHEMA_VERSION, + LEDGER_CONSUMER_FACT_SCHEMA_VERSION, + "ledger.consumer_fact.v99", + ] + + +def test_every_published_key_field_is_witnessed_for_its_epoch() -> None: + """Epoch witnessing covers every Chronicle key a published row carries. + + The captured feed fixture carries nine single-key fields and two key + lists, not just the four identifiers targets resolve by. Chronicle's + cutover moves families independently, so a row can straddle it: ledger-era + aggregate key, chronicle-era source-release key. Reading only the + resolution set would report that row as pure ledger-era. + """ + for field in ( + "observed_measure_key", + "source_release_key", + "source_series_key", + "universe_constraint_set_key", + ): + straddling = _fact_row(**{field: f"chronicle.{field[:-4]}.v3:straddle"}) + + assert row_fact_key_epochs(straddling) == frozenset( + {LEDGER_EPOCH, CHRONICLE_EPOCH} + ), field + + row_keys = _fact_row() + row_keys["lineage"] = dict(row_keys["lineage"]) + row_keys["lineage"]["source_row_keys"] = [ + CHRONICLE_FACT_KEY_DOMAINS["source_row"] + ":row" + ] + + assert row_fact_key_epochs(row_keys) == frozenset({LEDGER_EPOCH, CHRONICLE_EPOCH}) + + +def test_witnessed_key_fields_match_the_captured_feed() -> None: + """The inventory is grounded in a real feed, not in a guess. + + Every ``*_key`` field the captured UK feed rows carry is either witnessed + for its epoch or is a plain source identifier rather than a Chronicle key. + """ + rows = _captured_feed_rows() + assert rows + + for row in rows: + for field, value in row.items(): + if not field.endswith("_key") or not isinstance(value, str): + continue + assert field in FACT_KEY_FIELDS, field + assert is_chronicle_fact_key(value), (field, value) + + assert feed_fact_key_epochs(rows) == (LEDGER_EPOCH,) diff --git a/packages/microcosm-build/tests/test_ledger_targets.py b/packages/microcosm-build/tests/test_ledger_targets.py index cd46e2cad..cf441548d 100644 --- a/packages/microcosm-build/tests/test_ledger_targets.py +++ b/packages/microcosm-build/tests/test_ledger_targets.py @@ -2939,3 +2939,92 @@ def test_exact_period_contract_sum_keeps_equivalent_untyped_annual_range_cells() ).specs assert spec.value == 30.0 # Both synthetic cells belong to the same academic year. + + +def test__given_mixed_epoch_fact_feed__then_both_eras_compile_to_targets() -> None: + """Ledger-era and chronicle-era rows calibrate side by side. + + During Chronicle's rename cutover a feed carries history under + ``ledger.*`` domains beside newly emitted rows under ``chronicle.*`` + (PolicyEngine/chronicle#143). Keys are opaque to this compiler, so both + must select — and each target's name and metadata must carry its own + row's key verbatim rather than being normalised onto one epoch. + """ + # Given + ledger_era = _consumer_fact_row() + chronicle_era = _consumer_fact_row( + aggregate_fact_key="chronicle.aggregate_fact.v3:def456", + semantic_fact_key="chronicle.semantic_fact.v3:def456", + lineage={ + "source_record_id": "irs_soi.ty2024.table_1_1.all.adjusted_gross_income", + "source_cell_keys": ["chronicle.source_cell.v2:cell"], + "source_row_keys": [], + }, + ) + chronicle_era.pop("legacy_fact_key", None) + mapping = LedgerTargetMapping( + measure_by_concept={ + "us:statutes/26/62#adjusted_gross_income": "adjusted_gross_income" + }, + entity_by_ledger_entity={"tax_unit": "tax_unit"}, + filter_by_domain={"all_individual_income_tax_returns": "is_tax_return"}, + ) + + # When + selection = select_ledger_targets([ledger_era, chronicle_era], mapping) + + # Then + assert not selection.unsupported + assert [spec.name for spec in selection.specs] == [ + "ledger.aggregate_fact.v2:abc123", + "chronicle.aggregate_fact.v3:def456", + ] + chronicle_spec = selection.specs[1] + assert ( + chronicle_spec.metadata["ledger_fact_key"] + == "chronicle.aggregate_fact.v3:def456" + ) + # The diagnostic field names stay ledger-era: they are frozen at v1 + # (microcosm#639) and name a slot, not an epoch. + assert ( + chronicle_spec.metadata["ledger_aggregate_fact_key"] + == "chronicle.aggregate_fact.v3:def456" + ) + assert ( + chronicle_spec.metadata["ledger_semantic_fact_key"] + == "chronicle.semantic_fact.v3:def456" + ) + + +def test__given_chronicle_era_reference_pin__then_it_resolves_against_the_feed() -> ( + None +): + """A reference pinned to a chronicle-era key resolves without a code change.""" + # Given + reference = LedgerTargetReference( + name="nation/irs/adjusted gross income/total", + ledger_fact_key="chronicle.aggregate_fact.v3:def456", + entity="tax_unit", + measure="adjusted_gross_income", + filter="is_tax_return", + period=2024, + source="IRS SOI Table 1.1", + family="irs_soi", + ) + fact = _consumer_fact_row( + aggregate_fact_key="chronicle.aggregate_fact.v3:def456", + semantic_fact_key="chronicle.semantic_fact.v3:def456", + ) + fact.pop("legacy_fact_key", None) + + # When + registry = compile_ledger_target_references([fact], [reference], country="us") + + # Then + assert [spec.name for spec in registry.specs] == [ + "nation/irs/adjusted gross income/total" + ] + assert ( + registry.specs[0].metadata["ledger_fact_key"] + == "chronicle.aggregate_fact.v3:def456" + ) diff --git a/packages/microcosm-build/tests/test_logbook_cli.py b/packages/microcosm-build/tests/test_logbook_cli.py index 64444560f..cafaec71f 100644 --- a/packages/microcosm-build/tests/test_logbook_cli.py +++ b/packages/microcosm-build/tests/test_logbook_cli.py @@ -2,6 +2,7 @@ from __future__ import annotations +import argparse import importlib.util import json from pathlib import Path @@ -165,9 +166,7 @@ def test_export_refuses_a_directory_archive( spool.mkdir() cli = _load_cli() - assert ( - cli.main(["export", "--archive", str(tmp_path), "--source", str(spool)]) == 1 - ) + assert cli.main(["export", "--archive", str(tmp_path), "--source", str(spool)]) == 1 assert "extends exactly one scope chain" in capsys.readouterr().err @@ -256,9 +255,7 @@ def test_cli_local_export_refuses_wrong_scope_rows( _write_jsonl(source, _chain(pipeline="uk-local-rowwise")) cli = _load_cli() - exit_code = cli.main( - ["export", "--archive", str(archive), "--source", str(source)] - ) + exit_code = cli.main(["export", "--archive", str(archive), "--source", str(source)]) assert exit_code == 1 err = capsys.readouterr().err @@ -280,9 +277,7 @@ def test_cli_export_refuses_an_unratified_scope_archive( _write_jsonl(source, _chain(pipeline="uk-firms-staging")) cli = _load_cli() - exit_code = cli.main( - ["export", "--archive", str(archive), "--source", str(source)] - ) + exit_code = cli.main(["export", "--archive", str(archive), "--source", str(source)]) assert exit_code == 1 assert "not in the ratified scope list" in capsys.readouterr().err @@ -300,9 +295,7 @@ def test_cli_export_accepts_uk_local_scope_archive( _write_jsonl(source, rows) cli = _load_cli() - exit_code = cli.main( - ["export", "--archive", str(archive), "--source", str(source)] - ) + exit_code = cli.main(["export", "--archive", str(archive), "--source", str(source)]) assert exit_code == 0 assert "exported 3 new Logbook rows" in capsys.readouterr().out @@ -616,3 +609,48 @@ def test_chain_scope_matches_sql_contract( cli = _load_cli() assert cli._chain_scope(pipeline) == scope + + +def test_remote_help_asks_for_the_preferred_logbook_variable_names() -> None: + """``--help`` is the only place most operators read these names. + + It advertised ``POPULACE_LEDGER_URL`` alone, so an operator configuring + from the help text set up the very spelling the dual-read window exists to + retire — and then got a deprecation warning for following the + instructions. The preferred names come first; the ledger-era ones are + named as the fallback they are, because an operator whose environment + predates the rename still has to recognise what is being asked for. + """ + cli = _load_cli() + + remote = next( + action + for action in _export_actions(cli) + if "--remote" in getattr(action, "option_strings", ()) + ) + help_text = remote.help or "" + + for preferred in ( + cli.LOGBOOK_URL_ENV, + cli.LOGBOOK_EXPORT_KEY_ENV, + cli.LOGBOOK_API_KEY_ENV, + ): + assert preferred in help_text, preferred + for legacy in ( + cli.LEGACY_URL_ENV, + cli.REMOTE_EXPORT_KEY_ENV, + cli.REMOTE_API_KEY_ENV, + ): + assert legacy in help_text, legacy + # Preferred first: the reading order is the migration instruction. + assert help_text.index(cli.LOGBOOK_URL_ENV) < help_text.index(cli.LEGACY_URL_ENV) + + +def _export_actions(cli: ModuleType) -> list[object]: + """Every argparse action on the ``export`` subcommand.""" + subparsers = next( + action + for action in cli._parser()._actions + if isinstance(action, argparse._SubParsersAction) + ) + return list(subparsers.choices["export"]._actions) diff --git a/packages/microcosm-build/tests/test_logbook_env.py b/packages/microcosm-build/tests/test_logbook_env.py new file mode 100644 index 000000000..5357f2591 --- /dev/null +++ b/packages/microcosm-build/tests/test_logbook_env.py @@ -0,0 +1,205 @@ +"""The LOGBOOK_* / POPULACE_LEDGER_* environment dual-read window. + +Logbook (microcosm#632) gives its operational store a dual-read window so +publish flows and build scripts migrate on their own schedule: ``LOGBOOK_*`` +is preferred, the ledger-era name is still honored, and honoring it warns +once per process. These tests hold both halves — the fallback keeps working, +and the warning stays a single line rather than one per read in a build loop. +""" + +from __future__ import annotations + +import warnings + +import pytest + +from microcosm.build.logbook import _remote_config +from microcosm.build.logbook_env import ( + LEGACY_API_KEY_ENV, + LEGACY_EXPORT_KEY_ENV, + LEGACY_KEY_ENV, + LEGACY_URL_ENV, + LOGBOOK_API_KEY_ENV, + LOGBOOK_ENV_LEGACY_NAMES, + LOGBOOK_EXPORT_KEY_ENV, + LOGBOOK_KEY_ENV, + LOGBOOK_URL_ENV, + describe_logbook_env, + logbook_env, + logbook_env_names, + reset_logbook_env_deprecation_warnings, +) + +PAIRS = ( + (LOGBOOK_URL_ENV, LEGACY_URL_ENV), + (LOGBOOK_KEY_ENV, LEGACY_KEY_ENV), + (LOGBOOK_API_KEY_ENV, LEGACY_API_KEY_ENV), + (LOGBOOK_EXPORT_KEY_ENV, LEGACY_EXPORT_KEY_ENV), +) + + +@pytest.fixture(autouse=True) +def _clean_env(monkeypatch): + for preferred, legacy in PAIRS: + monkeypatch.delenv(preferred, raising=False) + monkeypatch.delenv(legacy, raising=False) + reset_logbook_env_deprecation_warnings() + yield + reset_logbook_env_deprecation_warnings() + + +@pytest.mark.parametrize(("preferred", "legacy"), PAIRS) +def test_preferred_name_wins_and_warns_about_nothing( + monkeypatch, preferred: str, legacy: str +) -> None: + monkeypatch.setenv(preferred, "logbook-value") + monkeypatch.setenv(legacy, "ledger-value") + + with warnings.catch_warnings(): + warnings.simplefilter("error", DeprecationWarning) + assert logbook_env(preferred) == "logbook-value" + + +@pytest.mark.parametrize(("preferred", "legacy"), PAIRS) +def test_legacy_name_is_honored_with_a_deprecation_warning( + monkeypatch, preferred: str, legacy: str +) -> None: + monkeypatch.setenv(legacy, "ledger-value") + + with pytest.warns(DeprecationWarning) as record: + assert logbook_env(preferred) == "ledger-value" + + message = str(record[0].message) + assert legacy in message + assert preferred in message + assert "microcosm#632" in message + + +def test_the_deprecation_warning_fires_once_per_process(monkeypatch) -> None: + """A build loop reads these repeatedly; one warning, not a storm.""" + monkeypatch.setenv(LEGACY_URL_ENV, "https://ledger.example") + + with warnings.catch_warnings(record=True) as caught: + warnings.simplefilter("always") + for _ in range(5): + assert logbook_env(LOGBOOK_URL_ENV) == "https://ledger.example" + + assert [ + w for w in caught if issubclass(w.category, DeprecationWarning) + ].__len__() == 1 + + +def test_each_legacy_name_warns_on_its_own(monkeypatch) -> None: + monkeypatch.setenv(LEGACY_URL_ENV, "https://ledger.example") + monkeypatch.setenv(LEGACY_KEY_ENV, "writer-jwt") + + with warnings.catch_warnings(record=True) as caught: + warnings.simplefilter("always") + logbook_env(LOGBOOK_URL_ENV) + logbook_env(LOGBOOK_KEY_ENV) + + warned = { + legacy + for legacy in (LEGACY_URL_ENV, LEGACY_KEY_ENV) + if any(legacy in str(w.message) for w in caught) + } + assert warned == {LEGACY_URL_ENV, LEGACY_KEY_ENV} + + +def test_unset_returns_the_default_and_empty_counts_as_unset(monkeypatch) -> None: + assert logbook_env(LOGBOOK_URL_ENV) is None + assert logbook_env(LOGBOOK_URL_ENV, "fallback") == "fallback" + + monkeypatch.setenv(LOGBOOK_URL_ENV, "") + monkeypatch.setenv(LEGACY_URL_ENV, "https://ledger.example") + with pytest.warns(DeprecationWarning): + assert logbook_env(LOGBOOK_URL_ENV) == "https://ledger.example" + + +def test_an_explicit_environ_mapping_bypasses_the_process_environment() -> None: + with pytest.warns(DeprecationWarning): + assert ( + logbook_env(LOGBOOK_KEY_ENV, environ={LEGACY_KEY_ENV: "writer-jwt"}) + == "writer-jwt" + ) + + +def test_a_name_outside_the_window_is_a_programming_error() -> None: + with pytest.raises(KeyError, match="not a Logbook environment variable"): + logbook_env("LOGBOOK_NOT_A_REAL_VARIABLE") + with pytest.raises(KeyError, match="not a Logbook environment variable"): + logbook_env_names(LEGACY_URL_ENV) + + +def test_names_and_descriptions_carry_both_spellings() -> None: + assert logbook_env_names(LOGBOOK_URL_ENV) == (LOGBOOK_URL_ENV, LEGACY_URL_ENV) + + described = describe_logbook_env(LOGBOOK_URL_ENV, LOGBOOK_EXPORT_KEY_ENV) + # An operator whose environment predates the rename must still be able to + # match the error text against what they have set. + for name in ( + LOGBOOK_URL_ENV, + LOGBOOK_EXPORT_KEY_ENV, + LEGACY_URL_ENV, + LEGACY_EXPORT_KEY_ENV, + ): + assert name in described + + +def test_every_windowed_variable_maps_to_exactly_one_legacy_name() -> None: + assert dict(LOGBOOK_ENV_LEGACY_NAMES) == { + preferred: (legacy,) for preferred, legacy in PAIRS + } + + +def test_logbook_remote_config_reads_both_eras(monkeypatch) -> None: + assert _remote_config() is None + + monkeypatch.setenv(LEGACY_URL_ENV, "https://ledger.example") + monkeypatch.setenv(LEGACY_KEY_ENV, "writer-jwt") + with pytest.warns(DeprecationWarning): + assert _remote_config() == ( + "https://ledger.example", + "writer-jwt", + "writer-jwt", + ) + + monkeypatch.setenv(LOGBOOK_URL_ENV, "https://logbook.example") + monkeypatch.setenv(LOGBOOK_KEY_ENV, "logbook-jwt") + monkeypatch.setenv(LOGBOOK_API_KEY_ENV, "project-api-key") + with warnings.catch_warnings(): + warnings.simplefilter("error", DeprecationWarning) + assert _remote_config() == ( + "https://logbook.example", + "logbook-jwt", + "project-api-key", + ) + + +def test_the_module_import_binds_the_module_not_the_reader() -> None: + """``import microcosm.build.logbook_env as env`` must give the module. + + The reader function shares its module's name. Re-exporting it from the + package barrel bound the function to ``microcosm.build.logbook_env``, and + ``import a.b as c`` returns the *attribute* when one exists — so the + submodule import silently handed back a callable, and every + ``env.LOGBOOK_URL_ENV`` after it raised ``AttributeError``. The barrel + therefore does not re-export the function; callers import it from here. + """ + import types + + import microcosm.build + import microcosm.build.logbook_env as env + + assert isinstance(env, types.ModuleType) + assert env.__name__ == "microcosm.build.logbook_env" + assert env.LOGBOOK_URL_ENV == LOGBOOK_URL_ENV + assert callable(env.logbook_env) + # The package attribute is the module too: `from microcosm.build import + # logbook_env` and the submodule import must not disagree about what the + # name means. + assert microcosm.build.logbook_env is env + assert "logbook_env" not in microcosm.build.__all__ + # The non-colliding helper is still re-exported, so removing the shadow + # did not quietly shrink the barrel further than it had to. + assert microcosm.build.logbook_env_names is logbook_env_names diff --git a/packages/microcosm-build/tests/test_logbook_isolation.py b/packages/microcosm-build/tests/test_logbook_isolation.py new file mode 100644 index 000000000..cb38af7bf --- /dev/null +++ b/packages/microcosm-build/tests/test_logbook_isolation.py @@ -0,0 +1,94 @@ +"""The suite cannot reach a live Logbook, however the shell is configured. + +The Logbook live store is append-only. A test that appends to it has written +a row nobody can retract, into the chain a real release will later have to +validate. So the guarantee this file pins is not "tests usually do not use +the network" — it is that a test *cannot* reach the store even when the +developer or runner shell is fully credentialed, under either generation of +the variable names. + +The guard itself lives in ``conftest.py`` as an autouse fixture, which is +what makes it total: nothing has to remember to ask for it. +""" + +from __future__ import annotations + +import os + +import pytest + +from microcosm.build import logbook +from microcosm.build.logbook_env import LOGBOOK_ENV_LEGACY_NAMES, logbook_env + +_ALL_LOGBOOK_ENV_NAMES = tuple( + sorted( + {name for name in LOGBOOK_ENV_LEGACY_NAMES} + | { + legacy + for legacy_names in LOGBOOK_ENV_LEGACY_NAMES.values() + for legacy in legacy_names + } + | {"POPULACE_LOGBOOK_PREV_ROW_DIGEST"} + ) +) + + +@pytest.mark.parametrize("name", _ALL_LOGBOOK_ENV_NAMES) +def test_no_logbook_variable_survives_into_a_test(name: str) -> None: + """Both generations are cleared, not just the ledger-era spelling. + + The per-module ``delenv`` lists this fixture replaced named only + ``POPULACE_LEDGER_*``. They were written before ``LOGBOOK_*`` existed, so + a shell exporting the preferred names walked straight past them. + """ + assert name not in os.environ + + +def test_remote_config_is_unconfigured_regardless_of_the_shell() -> None: + """``_remote_config`` is the gate every write to the live store passes.""" + assert logbook._remote_config() is None + for name in LOGBOOK_ENV_LEGACY_NAMES: + assert logbook_env(name) is None + + +def test_the_logbook_network_call_is_refused_by_default() -> None: + """An unstubbed request raises instead of leaving the machine.""" + with pytest.raises(AssertionError, match="real Logbook HTTP request"): + logbook.urlopen(object(), timeout=1.0) + + +def test_the_underlying_opener_is_refused_too() -> None: + """The second guard: a stale reference to the real ``urlopen``. + + ``tools/logbook.py`` binds ``urlopen`` at import time, and a future + importer could do the same. Guarding the opener the real function + delegates to closes that path without having to enumerate importers. + """ + with pytest.raises(AssertionError, match="real Logbook HTTP request"): + logbook._NO_REDIRECT_OPENER.open(object(), timeout=1.0) + + +def test_appending_a_row_to_a_polluted_environment_writes_nothing() -> None: + """The end-to-end shape of the accident this prevents. + + With credentials in the environment, ``_remote_config`` would return them + and the append would POST to Supabase. Here it is unconfigured, so the + remote leg is skipped entirely — no request is attempted, which is why + this passes rather than raising the guard's error. + """ + assert logbook._remote_config() is None + + +def test_the_opt_in_fixture_restores_the_real_callables( + allow_logbook_network, +) -> None: + """The guard is a policy with a documented exception, not a wall. + + Nothing in this suite opts in. The fixture exists so a test that genuinely + needs the network can say so in its own body, and so the restoration path + is exercised rather than assumed. No request is made here: this asserts + only that the shipped callables are back in place. + """ + assert logbook.urlopen.__module__ == logbook.__name__ + assert logbook.urlopen.__name__ == "urlopen" + assert logbook._NO_REDIRECT_OPENER.open.__self__ is logbook._NO_REDIRECT_OPENER diff --git a/packages/microcosm-build/tests/test_uk_calibration_run.py b/packages/microcosm-build/tests/test_uk_calibration_run.py index 5781eb777..08405c963 100644 --- a/packages/microcosm-build/tests/test_uk_calibration_run.py +++ b/packages/microcosm-build/tests/test_uk_calibration_run.py @@ -10,7 +10,13 @@ import pandas as pd import pytest +from microcosm.build.chronicle_epoch import ( + CHRONICLE_CONSUMER_FACT_SCHEMA_VERSION, + LEDGER_CONSUMER_FACT_SCHEMA_VERSION, + PUBLISHED_CONSUMER_ARTIFACT_SCHEMA_VERSION, +) from microcosm.build.country_spec import load_country_spec +from microcosm.build.ledger_artifact import load_ledger_consumer_artifact from microcosm.build.logbook import canonical_json_bytes from microcosm.build.uk_runtime import calibration_run from microcosm.build.uk_runtime.calibration_run import ( @@ -137,9 +143,7 @@ def _write_spine_sidecar( entity: int(len(frame.table(entity))) for entity in frame.entities }, "household_weight_kind": uk_household_weight_kind(frame).value, - "household_weight_total": float( - frame.weights_for("household").values.sum() - ), + "household_weight_total": float(frame.weights_for("household").values.sum()), } sidecar.update(overrides) input_h5.with_suffix(".build.json").write_text( @@ -178,8 +182,13 @@ def _admin_anchor_values(): def test_gate_scope_classifies_every_uk_gate(): all_ids = {entry.id for entry in load_country_spec("uk").gates.gates} - assert set(UK_CALIBRATION_GATE_SCOPE) | set(UK_CALIBRATION_GATE_SCOPE_EXCLUSIONS) == all_ids - assert set(UK_CALIBRATION_GATE_SCOPE).isdisjoint(UK_CALIBRATION_GATE_SCOPE_EXCLUSIONS) + assert ( + set(UK_CALIBRATION_GATE_SCOPE) | set(UK_CALIBRATION_GATE_SCOPE_EXCLUSIONS) + == all_ids + ) + assert set(UK_CALIBRATION_GATE_SCOPE).isdisjoint( + UK_CALIBRATION_GATE_SCOPE_EXCLUSIONS + ) assert all(UK_CALIBRATION_GATE_SCOPE_EXCLUSIONS.values()) @@ -233,9 +242,15 @@ def test_run_uk_calibration_writes_cross_pinned_outputs(monkeypatch, tmp_path: P assert paths.diagnostics_json.exists() assert paths.build_record_json.exists() assert paths.terminal_gate_json.exists() - assert result.build_record["artifacts"]["staging_h5"]["sha256"] == _sha(paths.staging_h5) - assert result.build_record["artifacts"]["diagnostics_json"]["sha256"] == _sha(paths.diagnostics_json) - assert result.build_record["artifacts"]["terminal_gate_json"]["sha256"] == _sha(paths.terminal_gate_json) + assert result.build_record["artifacts"]["staging_h5"]["sha256"] == _sha( + paths.staging_h5 + ) + assert result.build_record["artifacts"]["diagnostics_json"]["sha256"] == _sha( + paths.diagnostics_json + ) + assert result.build_record["artifacts"]["terminal_gate_json"]["sha256"] == _sha( + paths.terminal_gate_json + ) # The record makes no shippability claim of its own — the hand-written # literal retired with the #757 release-cut audit — and instead points # at the certification artifact whose verdict is authoritative. @@ -251,9 +266,13 @@ def test_run_uk_calibration_writes_cross_pinned_outputs(monkeypatch, tmp_path: P assert spine_provenance["stage_records"] == spine_sidecar["stage_records"] assert spine_provenance["stage_evidence"] == spine_sidecar["stage_evidence"] assert spine_provenance["artifact_pins"] == spine_sidecar["artifact_pins"] - assert spine_provenance["input_artifact_pins"] == spine_sidecar["input_artifact_pins"] + assert ( + spine_provenance["input_artifact_pins"] == spine_sidecar["input_artifact_pins"] + ) assert spine_provenance["resource_pins"] == spine_sidecar["resource_pins"] - assert spine_provenance["stage_artifact_pins"] == spine_sidecar["stage_artifact_pins"] + assert ( + spine_provenance["stage_artifact_pins"] == spine_sidecar["stage_artifact_pins"] + ) assert spine_provenance["declared_seeds"] == spine_sidecar["declared_seeds"] assert spine_provenance["rules_engine"] == spine_sidecar["rules_engine"] assert spine_provenance["source_vintages"] == spine_sidecar["source_vintages"] @@ -272,7 +291,10 @@ def test_run_uk_calibration_writes_cross_pinned_outputs(monkeypatch, tmp_path: P signature = attestation["signature"] attestation["signature"] = None key = b"0123456789abcdef0123456789abcdef" - assert hmac.new(key, canonical_json_bytes(report), hashlib.sha256).hexdigest() == signature + assert ( + hmac.new(key, canonical_json_bytes(report), hashlib.sha256).hexdigest() + == signature + ) assert result.logbook_spool.exists() @@ -456,10 +478,13 @@ def test_run_uk_calibration_refuses_input_sha_before_outputs(tmp_path: Path): doctrine_overrides={}, measure_resolver=None, source_pins={ - "input_h5": {"sha256": _sha(input_h5), "size_bytes": input_h5.stat().st_size} + "input_h5": { + "sha256": _sha(input_h5), + "size_bytes": input_h5.stat().st_size, + } }, run_config_extra={"calibration_year": 2025}, - release_id="bad-sha", + release_id="bad-sha", ) assert not paths.staging_h5.exists() assert not paths.diagnostics_json.exists() @@ -496,7 +521,7 @@ def test_run_uk_calibration_refuses_absent_input_sidecar(tmp_path: Path): } }, run_config_extra={"calibration_year": 2025}, - release_id="missing-sidecar", + release_id="missing-sidecar", ) assert not paths.staging_h5.exists() @@ -549,7 +574,7 @@ def test_run_uk_calibration_refuses_unbound_input_sidecar( } }, run_config_extra={"calibration_year": 2025}, - release_id="unbound-sidecar", + release_id="unbound-sidecar", ) assert not paths.staging_h5.exists() @@ -612,7 +637,10 @@ def test_seam_never_modifies_data_variables(monkeypatch, tmp_path: Path): doctrine_overrides={}, measure_resolver=None, source_pins={ - "input_h5": {"sha256": _sha(input_h5), "size_bytes": input_h5.stat().st_size} + "input_h5": { + "sha256": _sha(input_h5), + "size_bytes": input_h5.stat().st_size, + } }, run_config_extra={}, release_id="invariant-run", @@ -749,7 +777,7 @@ def test_refusal_records_a_failed_attempt_and_stages_nothing(tmp_path: Path): } }, run_config_extra={"calibration_year": 2025}, - release_id="refused-run", + release_id="refused-run", ) # Every terminal disposition is a row; a refusal that left the chain @@ -806,7 +834,7 @@ def test_attempt_ids_are_unique_across_reruns_of_one_release( measure_resolver=None, source_pins=source_pins, run_config_extra={"calibration_year": 2025}, - release_id="one-release-id", + release_id="one-release-id", ) build_ids.append(result.build_record["build_id"]) @@ -857,7 +885,10 @@ def test_verified_ledger_identity_reaches_the_run_evidence(monkeypatch, tmp_path doctrine_overrides={}, measure_resolver=None, source_pins={ - "input_h5": {"sha256": _sha(input_h5), "size_bytes": input_h5.stat().st_size} + "input_h5": { + "sha256": _sha(input_h5), + "size_bytes": input_h5.stat().st_size, + } }, run_config_extra={"calibration_year": 2025}, release_id="ledger-identity", @@ -875,3 +906,169 @@ def test_verified_ledger_identity_reaches_the_run_evidence(monkeypatch, tmp_path # A bare feed carries no manifest, and that absence is recorded rather # than invented. assert calibration_run._ledger_provenance(object())["manifest_sha256"] is None + + +def _consumer_fact_row( + *, + aggregate_fact_key: str, + semantic_fact_key: str, + schema_version: str, + source_release_key: str | None = None, +) -> dict: + row = { + "aggregate_fact_key": aggregate_fact_key, + "semantic_fact_key": semantic_fact_key, + "schema_version": schema_version, + "value": 1.0, + "period": {"type": "tax_year", "value": 2025}, + "geography": {"level": "country", "id": "K02000001"}, + "entity": {"name": "household"}, + "aggregation": {"method": "sum"}, + "observed_measure": {"source_name": "ons", "unit": "gbp"}, + "source": {"source_name": "ons"}, + "lineage": {"source_record_id": "ons.2025.total"}, + } + if source_release_key is not None: + row["source_release_key"] = source_release_key + return row + + +def _mixed_epoch_artifact_dir(tmp_path: Path) -> Path: + """A cutover-window feed: ledger-era history, chronicle-era rows, and one + Chronicle-namespace key spelling nothing declares. + + The manifest declares what Chronicle's ``main`` stamps today + (``policyengine_ledger.consumer_artifact.v2``), so this is the shape a UK + run is handed now, not a hypothetical one. + """ + rows = [ + _consumer_fact_row( + aggregate_fact_key="ledger.aggregate_fact.v2:aaa", + semantic_fact_key="ledger.semantic_fact.v2:aaa", + schema_version=LEDGER_CONSUMER_FACT_SCHEMA_VERSION, + ), + _consumer_fact_row( + aggregate_fact_key="chronicle.aggregate_fact.v3:bbb", + semantic_fact_key="chronicle.semantic_fact.v3:bbb", + schema_version=CHRONICLE_CONSUMER_FACT_SCHEMA_VERSION, + ), + _consumer_fact_row( + aggregate_fact_key="ledger.aggregate_fact.v2:ccc", + semantic_fact_key="ledger.semantic_fact.v2:ccc", + schema_version=LEDGER_CONSUMER_FACT_SCHEMA_VERSION, + source_release_key="chronicle.source_release.v9:undeclared", + ), + ] + artifact_dir = tmp_path / "chronicle-artifact" + artifact_dir.mkdir() + payload = "".join(json.dumps(row, sort_keys=True) + "\n" for row in rows) + (artifact_dir / "consumer_facts.jsonl").write_text(payload) + (artifact_dir / "manifest.json").write_text( + json.dumps( + { + "schema_version": PUBLISHED_CONSUMER_ARTIFACT_SCHEMA_VERSION, + "artifact_id": "chronicle-uk-artifact-mixed", + "profile": "uk-national", + "fact_row_count": len(rows), + "facts_sha256": hashlib.sha256(payload.encode()).hexdigest(), + }, + indent=2, + sort_keys=True, + ) + + "\n" + ) + return artifact_dir + + +def test_mixed_epoch_feed_epochs_reach_the_uk_release_evidence(monkeypatch, tmp_path): + """A UK run says which Chronicle era resolved its targets. + + The run's own provenance block used to be assembled field by field from + the artifact, which is how it came to carry the hashes but not the epoch + witnesses the loader had already computed. It now delegates to the shared + block, so a cutover-window feed is visible in the signed diagnostics and + in the build record — including the one Chronicle-namespace spelling this + build does not declare, named rather than folded into an era. + """ + pytest.importorskip("tables") # pandas HDF backend + monkeypatch.setattr( + calibration_run, + "uk_aggregate_admin_totals", + lambda frame, manifest: (_admin_anchor_values(), []), + ) + input_h5 = tmp_path / "input.h5" + frame = _frame() + write_uk_national_frame(frame, input_h5) + _write_spine_sidecar(input_h5, frame) + artifact = load_ledger_consumer_artifact(_mixed_epoch_artifact_dir(tmp_path)) + diagnostics_json = tmp_path / "diagnostics.json" + + result = run_uk_calibration( + paths=UKCalibrationRunPaths( + input_h5=input_h5, + staging_h5=tmp_path / "staged.h5", + diagnostics_json=diagnostics_json, + build_record_json=tmp_path / "build_record.json", + terminal_gate_json=tmp_path / "terminal_gates.json", + ), + input_sha256=_sha(input_h5), + ledger_artifact=artifact, + register_registry=_registry(), + band_edge_registry=_registry(), + calibration_year=2025, + exclusion_receipt={}, + doctrine=UKNationalSolveDoctrine(epochs=5), + doctrine_overrides={}, + measure_resolver=None, + source_pins={ + "input_h5": { + "sha256": _sha(input_h5), + "size_bytes": input_h5.stat().st_size, + } + }, + run_config_extra={"calibration_year": 2025}, + release_id="chronicle-mixed-epoch", + ) + + ledger = result.build_record["run_config"]["ledger"] + # The observed manifest id, verbatim, and the era it belongs to. + assert ledger["manifest"]["schema_version"] == ( + PUBLISHED_CONSUMER_ARTIFACT_SCHEMA_VERSION + ) + assert ledger["schema_epoch"] == "ledger" + # The feed straddles the cutover, and says so rather than reporting one era. + assert ledger["fact_key_epochs"] == ["ledger", "chronicle", "undeclared"] + assert ledger["undeclared_fact_key_domains"] == ["chronicle.source_release.v9"] + assert ledger["fact_schema_versions"] == [ + CHRONICLE_CONSUMER_FACT_SCHEMA_VERSION, + LEDGER_CONSUMER_FACT_SCHEMA_VERSION, + ] + # The hashes the block always carried are unchanged by the delegation. + assert ledger["facts_sha256"] == artifact.facts_sha256 + assert ledger["fact_row_count"] == 3 + + # The same block is what the signed diagnostics carry, so the evidence a + # release assembler reads witnesses the era too. + diagnostics = json.loads(diagnostics_json.read_text()) + assert diagnostics["build"]["ledger"] == ledger + + +def test_the_uk_block_delegates_rather_than_reassembling_the_shared_one(tmp_path): + """Every field of the shared provenance block reaches the UK block. + + Pinned as a property, not as a list: the failure being prevented is a + field the loader learns and this seam silently drops, and a test that + enumerated today's fields would not catch tomorrow's. + """ + artifact = load_ledger_consumer_artifact(_mixed_epoch_artifact_dir(tmp_path)) + shared = artifact.provenance() + + block = calibration_run._ledger_provenance(artifact) + + for field, value in shared.items(): + if field in {"path_name", "schema_version", "profiles"}: + # Not identity of the feed: the directory name is local, and the + # manifest id is carried in the narrower manifest sub-block. + continue + assert block[field] == value, field + assert block["manifest"]["schema_version"] == shared["schema_version"] diff --git a/packages/microcosm-build/tests/test_us_congressional_district_vintage.py b/packages/microcosm-build/tests/test_us_congressional_district_vintage.py index 364b2fcfa..cd51eeb07 100644 --- a/packages/microcosm-build/tests/test_us_congressional_district_vintage.py +++ b/packages/microcosm-build/tests/test_us_congressional_district_vintage.py @@ -1,5 +1,7 @@ """US congressional-district geography-vintage translation tests.""" +import json + import pytest from microcosm.build.us_runtime import ( @@ -732,3 +734,161 @@ def _ledger_fact_for_reference(reference, *, value: float) -> dict[str, object]: "url": "https://example.org/reference", }, } + + +def _re_epoch_to_chronicle(fact: dict[str, object]) -> dict[str, object]: + """The same source fact as Chronicle emits it after the rename cutover. + + Identical canonical payload; ``ledger..v2`` domains replaced by + their ``chronicle..v3`` siblings (PolicyEngine/chronicle#143). + """ + chronicle = dict(fact) + for field, domain in ( + ("aggregate_fact_key", "chronicle.aggregate_fact.v3"), + ("semantic_fact_key", "chronicle.semantic_fact.v3"), + ): + value = chronicle.get(field) + if isinstance(value, str): + chronicle[field] = f"{domain}:{value.split(':', 1)[1]}" + chronicle.pop("legacy_fact_key", None) + return chronicle + + +def test__given_chronicle_era_source_rows__then_derived_cd_keys_are_unchanged() -> None: + """The Chronicle rename must not re-identify Microcosm-derived facts. + + Translated and proxy facts are minted into Microcosm's own + ``microcosm.derived_fact.*`` / ``microcosm.semantic_fact.*`` namespaces, + which sit outside both Chronicle eras and are frozen at v1 + (microcosm#639). Their digests are computed from the source row's + *semantic* identity, never from its key, so a feed that crosses the + cutover must mint byte-identical derived keys — otherwise every target + pinned to one would silently stop resolving on cutover day. + """ + ledger_era_facts = [ + _soi_cd_fact( + "adjusted_gross_income", + 100.0, + geography_id="5001700US0601", + source_row_id="ca_01", + ), + _soi_cd_fact( + "adjusted_gross_income", + 60.0, + geography_id="5001700US0653", + source_row_id="ca_53", + ), + ] + chronicle_era_facts = [_re_epoch_to_chronicle(fact) for fact in ledger_era_facts] + crosswalk = [ + { + "source_geography_id": "5001700US0601", + "target_geography_id": "5001900US0601", + "weight": 1.0, + }, + { + "source_geography_id": "5001700US0653", + "target_geography_id": "5001900US0602", + "weight": 1.0, + }, + ] + + def _translate(facts): + return translate_congressional_district_facts_to_current_vintage( + facts, crosswalk, crosswalk_basis="block_population" + ) + + from_ledger = _translate(ledger_era_facts) + from_chronicle = _translate(chronicle_era_facts) + + def _keys(translated): + return sorted( + ( + fact["geography"]["id"], + fact["aggregate_fact_key"], + fact["semantic_fact_key"], + ) + for fact in translated + ) + + assert _keys(from_chronicle) == _keys(from_ledger) + for fact in from_chronicle: + assert fact["aggregate_fact_key"].startswith( + "microcosm.derived_fact.congressional_district_vintage.v1:" + ) + assert fact["semantic_fact_key"].startswith( + "microcosm.semantic_fact.congressional_district_vintage.v1:" + ) + # The derived row carries neither epoch: it is Microcosm's fact now. + assert "legacy_fact_key" not in fact + + +def test__given_chronicle_era_state_rows__then_proxy_cd_keys_are_unchanged() -> None: + """The state-total-proxy mint is epoch-independent for the same reason. + + Its digest covers the source record id, the proxy district, and the + value — never the source row's Chronicle key — so a chronicle-era state + fact produces the same proxy identity a ledger-era one does. + """ + ledger_era_facts = [ + _soi_cd_fact( + "adjusted_gross_income", + 10.0, + geography_id="5001700US0601", + source_row_id="ca_01", + ), + _soi_state_fact( + "adjusted_gross_income", + 100.0, + geography_id="0400000US30", + source_row_id="mt_total", + ), + ] + chronicle_era_facts = [_re_epoch_to_chronicle(fact) for fact in ledger_era_facts] + crosswalk = [ + { + "source_geography_id": "5001700US0601", + "target_geography_id": "5001900US0601", + "weight": 1.0, + }, + { + "source_geography_id": "5001700US3000", + "target_geography_id": "5001900US3001", + "weight": 2.0, + }, + { + "source_geography_id": "5001700US3000", + "target_geography_id": "5001900US3002", + "weight": 3.0, + }, + ] + + def _translate(facts): + return translate_congressional_district_facts_to_current_vintage( + facts, crosswalk, crosswalk_basis="block_population" + ) + + def _derived_identity(translated): + # Passthrough rows keep their own Chronicle keys verbatim — that is the + # opaque-carriage rule. Only the Microcosm-minted rows are compared. + return sorted( + ( + fact["geography"]["id"], + fact["aggregate_fact_key"], + fact["semantic_fact_key"], + json.dumps(fact["lineage"], sort_keys=True), + ) + for fact in translated + if str(fact.get("aggregate_fact_key", "")).startswith("microcosm.") + ) + + from_chronicle = _derived_identity(_translate(chronicle_era_facts)) + + assert from_chronicle == _derived_identity(_translate(ledger_era_facts)) + # The proxy digest reaches the compared identity through the lineage: the + # proxy fact is minted, then translated, so its source_record_id (digest + # included) is what the derived row records. + assert any( + "congressional_district_state_total_proxy_source_record_id" in lineage + for _, _, _, lineage in from_chronicle + ) diff --git a/packages/microcosm-build/tests/test_us_source_coverage.py b/packages/microcosm-build/tests/test_us_source_coverage.py index 9f6b3096e..996967785 100644 --- a/packages/microcosm-build/tests/test_us_source_coverage.py +++ b/packages/microcosm-build/tests/test_us_source_coverage.py @@ -4,6 +4,7 @@ from microcosm.build.gates import GateReport from microcosm.build.us_runtime.source_coverage import ( + CHRONICLE_US_SOURCE_COVERAGE_CONTRACT_COMMIT, LEDGER_US_SOURCE_COVERAGE_CONTRACT_COMMIT, hard_target_package_aliases, source_gap_family_ids, @@ -14,8 +15,34 @@ ) +def test_both_contract_commit_spellings_reach_the_us_runtime_barrel() -> None: + """The Chronicle-named alias is importable everywhere its sibling is. + + The alias was added beside the ledger-era pin in ``source_coverage`` but + not re-exported, so ``from microcosm.build.us_runtime import + CHRONICLE_US_SOURCE_COVERAGE_CONTRACT_COMMIT`` failed while the ledger-era + name resolved — an alias only half of the callers could use. Both name the + same commit; neither is a second pin. + """ + from microcosm.build import us_runtime + + assert ( + us_runtime.CHRONICLE_US_SOURCE_COVERAGE_CONTRACT_COMMIT + is us_runtime.LEDGER_US_SOURCE_COVERAGE_CONTRACT_COMMIT + ) + for name in ( + "CHRONICLE_US_SOURCE_COVERAGE_CONTRACT_COMMIT", + "LEDGER_US_SOURCE_COVERAGE_CONTRACT_COMMIT", + ): + assert name in us_runtime.__all__, name + + def test_us_source_coverage_snapshot_has_expected_roles() -> None: assert len(LEDGER_US_SOURCE_COVERAGE_CONTRACT_COMMIT) == 40 + assert ( + CHRONICLE_US_SOURCE_COVERAGE_CONTRACT_COMMIT + == LEDGER_US_SOURCE_COVERAGE_CONTRACT_COMMIT + ) assert "soi-filing-season-week47-2024-eitc-total" in hard_target_package_aliases() assert "ssa-ssi-table-7b1-2024" in hard_target_package_aliases() assert "cms-aca-oep-state-level" in hard_target_package_aliases() diff --git a/packages/microcosm-build/tests/test_us_trade_facts.py b/packages/microcosm-build/tests/test_us_trade_facts.py index f20d443f6..e7b233669 100644 --- a/packages/microcosm-build/tests/test_us_trade_facts.py +++ b/packages/microcosm-build/tests/test_us_trade_facts.py @@ -7,6 +7,11 @@ import pandas as pd import pytest +from microcosm.build.chronicle_epoch import ( + CHRONICLE_CONSUMER_ARTIFACT_SCHEMA_VERSION, + LEDGER_CONSUMER_FACT_SCHEMA_VERSION, + MICROCOSM_CONSUMER_ARTIFACT_SCHEMA_VERSION, +) from microcosm.build.ledger_artifact import load_ledger_consumer_artifact from microcosm.build.ledger_targets import ( LedgerTargetReference, @@ -813,3 +818,71 @@ def test_cbp_and_district_facts_compile_into_ledger_targets(tmp_path): by_name = {spec.name: spec for spec in registry.specs} assert by_name["cbp_total_entry_summaries_fytd2026"].value == 83_133_856 assert by_name["district70_customs_value_2026_01"].value == 2_000_000 + + +def test_minted_artifact_declares_the_ledger_era_id_and_loads_under_either(tmp_path): + """Emission stays ledger-era; acceptance covers both (chronicle#143). + + These rows are minted from Census/CBP source bytes, not derived from a + Chronicle row whose epoch they could inherit, and their bytes are pinned, + so the declared id must not move on Chronicle's cutover. The declaration + is nonetheless an argument rather than a literal, checked against both + eras, so flipping it later is a caller decision. + """ + rows = build_import_entry_fact_rows( + _margins(), + retrieval_manifest=_manifest_entries(), + extracted_at="2026-08-05T00:00:00+00:00", + ) + + default_manifest = write_consumer_artifact( + tmp_path / "default", + rows, + retrieval_manifest=_manifest_entries(), + generator=default_generator_block(months=("2026-01", "2026-02")), + ) + assert ( + default_manifest["schema_version"] == MICROCOSM_CONSUMER_ARTIFACT_SCHEMA_VERSION + ) + # Every row's key stays in Microcosm's own frozen namespace, outside both + # Chronicle eras, so the cutover cannot re-identify them (microcosm#639). + assert all( + row["aggregate_fact_key"].startswith("populace_us_trade.aggregate_fact.v1:") + for row in rows + ) + assert all( + row["schema_version"] == LEDGER_CONSUMER_FACT_SCHEMA_VERSION for row in rows + ) + + chronicle_manifest = write_consumer_artifact( + tmp_path / "chronicle", + rows, + retrieval_manifest=_manifest_entries(), + generator=default_generator_block(months=("2026-01", "2026-02")), + schema_version=CHRONICLE_CONSUMER_ARTIFACT_SCHEMA_VERSION, + ) + assert ( + chronicle_manifest["schema_version"] + == CHRONICLE_CONSUMER_ARTIFACT_SCHEMA_VERSION + ) + # Only the declaration moved: the fact bytes are identical either way. + assert chronicle_manifest["facts_sha256"] == default_manifest["facts_sha256"] + + for directory, expected_epoch in ( + (tmp_path / "default", "ledger"), + (tmp_path / "chronicle", "chronicle"), + ): + artifact = load_ledger_consumer_artifact(directory) + assert artifact.fact_row_count == len(rows) + assert artifact.schema_epoch == expected_epoch + # Microcosm-minted keys belong to neither era, so nothing is claimed. + assert artifact.provenance()["fact_key_epochs"] == [] + + with pytest.raises(ValueError, match="Refusing to write a consumer artifact"): + write_consumer_artifact( + tmp_path / "bogus", + rows, + retrieval_manifest=_manifest_entries(), + generator=default_generator_block(months=("2026-01",)), + schema_version="policyengine_chronicle.consumer_artifact.v9", + ) diff --git a/tools/logbook.py b/tools/logbook.py index 78aaee9d4..852ca3946 100644 --- a/tools/logbook.py +++ b/tools/logbook.py @@ -7,8 +7,11 @@ archive or a directory of them, reporting chain by chain. Remote export uses a distinct, read-only ``logbook_exporter`` JWT supplied -as ``POPULACE_LEDGER_EXPORT_KEY`` plus the hosted project's gateway key in -``POPULACE_LEDGER_API_KEY``. It never reuses the insert-only writer key. +as ``LOGBOOK_EXPORT_KEY`` plus the hosted project's gateway key in +``LOGBOOK_API_KEY``. It never reuses the insert-only writer key. The +ledger-era spellings (``POPULACE_LEDGER_EXPORT_KEY``, +``POPULACE_LEDGER_API_KEY``, ``POPULACE_LEDGER_URL``) stay honored for the +Logbook dual-read window (microcosm#632) and warn once per process. The live store is row-oriented and carries every attempt across all scopes; the per-scope split is an archive convention, not a database partition. """ @@ -17,7 +20,6 @@ import argparse import json -import os import re import sys from pathlib import Path @@ -36,6 +38,16 @@ render_markdown, urlopen, ) +from microcosm.build.logbook_env import ( + LEGACY_API_KEY_ENV, + LEGACY_EXPORT_KEY_ENV, + LEGACY_URL_ENV, + LOGBOOK_API_KEY_ENV, + LOGBOOK_EXPORT_KEY_ENV, + LOGBOOK_URL_ENV, + describe_logbook_env, + logbook_env, +) ROOT = Path(__file__).resolve().parents[1] #: Archives live at ``logbook//.jsonl`` — one hash chain per @@ -46,8 +58,12 @@ #: chain is born, and country is the outermost one. DEFAULT_ARCHIVE_ROOT = ROOT / "logbook" DEFAULT_SPOOL_ROOT = ROOT / "logbook-spool" -REMOTE_EXPORT_KEY_ENV = "POPULACE_LEDGER_EXPORT_KEY" -REMOTE_API_KEY_ENV = "POPULACE_LEDGER_API_KEY" +#: Ledger-era names, still honored through the Logbook dual-read window +#: (microcosm#632). Reads go through +#: :func:`microcosm.build.logbook_env.logbook_env`, which prefers +#: ``LOGBOOK_EXPORT_KEY`` / ``LOGBOOK_API_KEY``. +REMOTE_EXPORT_KEY_ENV = LEGACY_EXPORT_KEY_ENV +REMOTE_API_KEY_ENV = LEGACY_API_KEY_ENV REMOTE_PAGE_SIZE = 500 # Mirror of logbook.chain_scope() in # supabase/migrations/20260818000000_logbook_chain_scopes.sql. The legacy US @@ -102,10 +118,16 @@ def _parser() -> argparse.ArgumentParser: source.add_argument( "--remote", action="store_true", + # The help text names the preferred Logbook spellings first and the + # ledger-era ones as the fallback they now are, so an operator + # reading --help configures the environment the dual-read window + # wants rather than the one it merely still tolerates. help=( - "Read the private live store using POPULACE_LEDGER_URL and the " - f"read-only {REMOTE_EXPORT_KEY_ENV}, authenticated at the gateway " - f"by {REMOTE_API_KEY_ENV}." + f"Read the private live store using {LOGBOOK_URL_ENV} and the " + f"read-only {LOGBOOK_EXPORT_KEY_ENV}, authenticated at the " + f"gateway by {LOGBOOK_API_KEY_ENV}. The ledger-era names " + f"({LEGACY_URL_ENV}, {REMOTE_EXPORT_KEY_ENV}, " + f"{REMOTE_API_KEY_ENV}) are still honored and warn once." ), ) @@ -243,13 +265,15 @@ def _archive_scope(archive: Path) -> str: def _remote_rows(scope: str) -> tuple[LogbookRow, ...]: - ledger_url = os.environ.get("POPULACE_LEDGER_URL") - export_key = os.environ.get(REMOTE_EXPORT_KEY_ENV) - api_key = os.environ.get(REMOTE_API_KEY_ENV) + ledger_url = logbook_env(LOGBOOK_URL_ENV) + export_key = logbook_env(LOGBOOK_EXPORT_KEY_ENV) + api_key = logbook_env(LOGBOOK_API_KEY_ENV) if not ledger_url or not export_key or not api_key: raise ValueError( - "remote export requires POPULACE_LEDGER_URL, " - f"{REMOTE_EXPORT_KEY_ENV}, and {REMOTE_API_KEY_ENV}" + "remote export requires " + + describe_logbook_env( + LOGBOOK_URL_ENV, LOGBOOK_EXPORT_KEY_ENV, LOGBOOK_API_KEY_ENV + ) ) rows: list[LogbookRow] = [] @@ -288,11 +312,7 @@ def _remote_rows(scope: str) -> tuple[LogbookRow, ...]: ) break wrong_scope = sorted( - { - row.pipeline - for row in rows - if _chain_scope(row.pipeline) != scope - } + {row.pipeline for row in rows if _chain_scope(row.pipeline) != scope} ) if wrong_scope: raise ValueError(