From 1eeb3bf8bc62be867917f7738c1a1e85d3b111a4 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Wed, 2 Sep 2026 10:46:15 -0400 Subject: [PATCH 01/23] Start the chronicle dual-acceptance journal chronicle#143 consumer side: microcosm must accept both ledger-era and chronicle-era Chronicle schema ids and hash domains before Chronicle flips emit. Co-Authored-By: Claude Fable 5.1 --- PROGRESS-chronicle-dual-accept.md | 38 +++++++++++++++++++++++++++++++ 1 file changed, 38 insertions(+) create mode 100644 PROGRESS-chronicle-dual-accept.md diff --git a/PROGRESS-chronicle-dual-accept.md b/PROGRESS-chronicle-dual-accept.md new file mode 100644 index 000000000..47cd999f8 --- /dev/null +++ b/PROGRESS-chronicle-dual-accept.md @@ -0,0 +1,38 @@ +# Chronicle dual-acceptance (consumer side) — chronicle#143 + +Branch `chronicle-dual-accept`. Journal for this lane. Root journals are +history, not state (see CLAUDE.md); this one is current only while the branch +is open. + +## Goal + +Microcosm must accept BOTH ledger-era and chronicle-era Chronicle identities +*before* Chronicle flips emit: + +- schema ids: `policyengine_ledger.consumer_artifact.v1` **and** + `policyengine_chronicle.consumer_artifact.v2`; `ledger.consumer_fact.v1` + **and** `chronicle.consumer_fact.v2` +- hash domains: `ledger..v2` **and** `chronicle..v3` (same canonical + payload, new domain string) +- env names: `CHRONICLE_*` preferred, legacy honored with a once-per-process + deprecation warning + +Frozen (microcosm#639): nothing on disk or in artifacts renames. Diagnostic +field names (`ledger_aggregate_fact_key`), H5 attrs, `populace_*` ids, fact +keys, goldens and fixtures stay at v1. + +## State + +- [ ] not started + +## Done + +(nothing yet) + +## Next + +1. Read the primary files, run the epoch-sensitivity audit. +2. Land the shared epoch module. +3. Dual-accept the two consumer-artifact loaders. +4. Env dual-read helper. +5. Tests + full verification. From 413b8daf3b00198e753865cbce118c7dda099766 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Wed, 2 Sep 2026 10:55:08 -0400 Subject: [PATCH 02/23] Accept chronicle-era Chronicle schema ids and add the CHRONICLE_* env dual-read MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Chronicle (formerly Ledger) introduces chronicle-era hash domains and chronicle-named schema ids for newly emitted rows at a declared cutover; ledger-era ids stay valid history forever (PolicyEngine/chronicle#143). Microcosm is a consumer, so it must accept both eras before Chronicle flips emit. microcosm.build.chronicle_epoch is the single place that knows which identity belongs to which era. Epoch detection is structural — the namespace segment of a key's domain — rather than a lookup in a frozen list of domain strings, so a chronicle-era family whose version number the spec has not yet declared still resolves, and no validator hard-codes one epoch. Fact keys stay opaque: Microcosm never recomputes a Chronicle digest, so comparison stays exact string equality and only the 'is this a Chronicle key?' question goes dual-era. The consumer-artifact loader now tests manifest schema_version for membership in both eras instead of equality with one, validates a per-row schema id when a feed carries one (Chronicle-published rows never have), and records the observed id, its epoch, and the feed's fact-key epochs in provenance() so a release manifest witnesses which era actually resolved its targets. microcosm.build.chronicle_env is the dual-read window for the operational env names: CHRONICLE_URL/KEY/API_KEY/EXPORT_KEY preferred, POPULACE_LEDGER_* still honored with one DeprecationWarning per process per name. LEDGER_HMRC_BANDS, LEDGER_ONS_*_BANDS and LEDGER_US_SOURCE_COVERAGE_CONTRACT_COMMIT only look like env vars to a grep — they are data constants, so they take chronicle-era aliases instead. Nothing renames on disk: emitted ids, diagnostic field names, populace_* key namespaces, goldens and fixtures stay frozen at v1 (microcosm#639). Co-Authored-By: Claude Fable 5.1 --- .../src/microcosm/build/__init__.py | 26 ++ .../src/microcosm/build/chronicle_env.py | 149 +++++++++ .../src/microcosm/build/chronicle_epoch.py | 294 ++++++++++++++++++ .../src/microcosm/build/ledger_artifact.py | 106 ++++++- .../src/microcosm/build/logbook.py | 32 +- .../build/uk_runtime/firm_generation.py | 10 + .../build/us_runtime/source_coverage.py | 8 + .../us_runtime/us_trade/import_entry_facts.py | 47 ++- tools/logbook.py | 36 ++- 9 files changed, 669 insertions(+), 39 deletions(-) create mode 100644 packages/microcosm-build/src/microcosm/build/chronicle_env.py create mode 100644 packages/microcosm-build/src/microcosm/build/chronicle_epoch.py diff --git a/packages/microcosm-build/src/microcosm/build/__init__.py b/packages/microcosm-build/src/microcosm/build/__init__.py index 352f53b74..ea8dbdf72 100644 --- a/packages/microcosm-build/src/microcosm/build/__init__.py +++ b/packages/microcosm-build/src/microcosm/build/__init__.py @@ -49,6 +49,21 @@ def _assert_frame_compatible(version: str, required: tuple[int, int]) -> None: _assert_frame_compatible(_frame_version, _REQUIRED_FRAME_SERIES) +from microcosm.build.chronicle_env import ( # noqa: E402 - after the compat gate + chronicle_env, + chronicle_env_names, +) +from microcosm.build.chronicle_epoch import ( # noqa: E402 - after the compat gate + ACCEPTED_CONSUMER_ARTIFACT_SCHEMA_VERSIONS, + ACCEPTED_CONSUMER_FACT_SCHEMA_VERSIONS, + CHRONICLE_EPOCH, + EPOCHS, + LEDGER_EPOCH, + fact_key_epoch, + feed_fact_key_epochs, + is_chronicle_fact_key, + parse_fact_key, +) from microcosm.build.country_spec import ( # noqa: E402 - after the compat gate CountryResourceRow, CountrySpec, @@ -196,6 +211,11 @@ def _assert_frame_compatible(version: str, required: tuple[int, int]) -> None: "StagingTelemetry", "TargetCoverageRequirement", "TargetFitRequirement", + "ACCEPTED_CONSUMER_ARTIFACT_SCHEMA_VERSIONS", + "ACCEPTED_CONSUMER_FACT_SCHEMA_VERSIONS", + "CHRONICLE_EPOCH", + "EPOCHS", + "LEDGER_EPOCH", "LedgerConsumerArtifact", "LedgerTargetMapping", "LedgerTargetSelection", @@ -204,6 +224,12 @@ def _assert_frame_compatible(version: str, required: tuple[int, int]) -> None: "MonetaryTargetProfile", "PreparedMonetaryMeasure", "add_ledger_artifact_args", + "chronicle_env", + "chronicle_env_names", + "fact_key_epoch", + "feed_fact_key_epochs", + "is_chronicle_fact_key", + "parse_fact_key", "aggregate_admin_gate", "area_support_gate", "column_implication_gate", diff --git a/packages/microcosm-build/src/microcosm/build/chronicle_env.py b/packages/microcosm-build/src/microcosm/build/chronicle_env.py new file mode 100644 index 000000000..763032d4b --- /dev/null +++ b/packages/microcosm-build/src/microcosm/build/chronicle_env.py @@ -0,0 +1,149 @@ +"""Dual-read for the Chronicle (formerly Ledger) environment variables. + +PolicyEngine/chronicle#143 gives the operational stores — buckets, database +schema, role ids, and env names — a **dual-read window**: ``CHRONICLE_*`` is +preferred, the legacy name is still honored, and honoring it emits a +deprecation warning so publish flows and build scripts migrate on their own +schedule. This module is that window, in one place. + +Every variable here is read through :func:`chronicle_env`, which tries the +preferred ``CHRONICLE_*`` name first and falls back to the legacy name, +warning once per process per legacy name. Nothing is renamed on disk: the +legacy names keep working for as long as the window is open, and the legacy +name constants stay exported so error messages and tests can still name them. + +Note on scope: the identifiers ``LEDGER_HMRC_BANDS``, +``LEDGER_ONS_TURNOVER_BANDS``, ``LEDGER_ONS_EMPLOYMENT_BANDS``, and +``LEDGER_US_SOURCE_COVERAGE_CONTRACT_COMMIT`` look like environment variables +to a grep but are plain Python module constants (band maps and a git commit +pin). They get chronicle-named *aliases* beside their modules rather than a +dual-read, because there is no environment to read them from. +""" + +from __future__ import annotations + +import os +import threading +import warnings +from collections.abc import Mapping + +__all__ = [ + "CHRONICLE_API_KEY_ENV", + "CHRONICLE_ENV_LEGACY_NAMES", + "CHRONICLE_EXPORT_KEY_ENV", + "CHRONICLE_KEY_ENV", + "CHRONICLE_URL_ENV", + "LEGACY_API_KEY_ENV", + "LEGACY_EXPORT_KEY_ENV", + "LEGACY_KEY_ENV", + "LEGACY_URL_ENV", + "chronicle_env", + "chronicle_env_names", + "describe_chronicle_env", + "reset_chronicle_env_deprecation_warnings", +] + +#: Preferred, chronicle-era names. +CHRONICLE_URL_ENV = "CHRONICLE_URL" +CHRONICLE_KEY_ENV = "CHRONICLE_KEY" +CHRONICLE_API_KEY_ENV = "CHRONICLE_API_KEY" +CHRONICLE_EXPORT_KEY_ENV = "CHRONICLE_EXPORT_KEY" + +#: Legacy, ledger-era names. Still honored; still named in error messages so +#: an operator running the old environment recognises what is being asked for. +LEGACY_URL_ENV = "POPULACE_LEDGER_URL" +LEGACY_KEY_ENV = "POPULACE_LEDGER_KEY" +LEGACY_API_KEY_ENV = "POPULACE_LEDGER_API_KEY" +LEGACY_EXPORT_KEY_ENV = "POPULACE_LEDGER_EXPORT_KEY" + +#: Preferred name -> legacy names, most recent legacy spelling first. Adding a +#: variable to the dual-read window means adding a row here and nothing else. +CHRONICLE_ENV_LEGACY_NAMES: Mapping[str, tuple[str, ...]] = { + CHRONICLE_URL_ENV: (LEGACY_URL_ENV,), + CHRONICLE_KEY_ENV: (LEGACY_KEY_ENV,), + CHRONICLE_API_KEY_ENV: (LEGACY_API_KEY_ENV,), + CHRONICLE_EXPORT_KEY_ENV: (LEGACY_EXPORT_KEY_ENV,), +} + +_WARNED_LEGACY_NAMES: set[str] = set() +_WARNED_LOCK = threading.Lock() + + +def chronicle_env( + name: str, + default: str | None = None, + *, + environ: Mapping[str, str] | None = None, +) -> str | None: + """Read one Chronicle variable, preferring ``name`` over its legacy spelling. + + ``name`` is the preferred ``CHRONICLE_*`` name. When it is unset but a + legacy name carries a value, that value is returned and a + :class:`DeprecationWarning` is emitted **once per process per legacy + name** — repeated reads in a build loop must not turn into a warning + storm. An empty value is treated as unset, matching how the callers here + already test these variables. + """ + source = os.environ if environ is None else environ + if name not in CHRONICLE_ENV_LEGACY_NAMES: + raise KeyError( + f"{name!r} is not a Chronicle environment variable; expected one of " + f"{sorted(CHRONICLE_ENV_LEGACY_NAMES)}." + ) + value = source.get(name) + if value: + return value + for legacy_name in CHRONICLE_ENV_LEGACY_NAMES[name]: + legacy_value = source.get(legacy_name) + if legacy_value: + _warn_once(legacy_name, preferred=name) + return legacy_value + return default + + +def chronicle_env_names(name: str) -> tuple[str, ...]: + """The preferred name followed by every legacy name still honored.""" + if name not in CHRONICLE_ENV_LEGACY_NAMES: + raise KeyError( + f"{name!r} is not a Chronicle environment variable; expected one of " + f"{sorted(CHRONICLE_ENV_LEGACY_NAMES)}." + ) + return (name, *CHRONICLE_ENV_LEGACY_NAMES[name]) + + +def describe_chronicle_env(*names: str) -> str: + """Render required variables for an error message, legacy names included. + + Error text names both spellings on purpose: an operator whose environment + predates the rename must still be able to match the message against what + they have set. + """ + preferred = ", ".join(names) + legacy = ", ".join( + legacy_name + for name in names + for legacy_name in CHRONICLE_ENV_LEGACY_NAMES[name] + ) + return f"{preferred} (legacy {legacy} still honored)" + + +def reset_chronicle_env_deprecation_warnings() -> None: + """Forget which legacy names have warned. For tests only.""" + with _WARNED_LOCK: + _WARNED_LEGACY_NAMES.clear() + + +def _warn_once(legacy_name: str, *, preferred: str) -> None: + with _WARNED_LOCK: + if legacy_name in _WARNED_LEGACY_NAMES: + return + _WARNED_LEGACY_NAMES.add(legacy_name) + warnings.warn( + f"{legacy_name} is the ledger-era name for {preferred} and is " + "deprecated; PolicyEngine Ledger is now Chronicle " + "(PolicyEngine/chronicle#143). Set " + f"{preferred} instead — {legacy_name} stays honored only for the " + "dual-read window.", + DeprecationWarning, + stacklevel=3, + ) diff --git a/packages/microcosm-build/src/microcosm/build/chronicle_epoch.py b/packages/microcosm-build/src/microcosm/build/chronicle_epoch.py new file mode 100644 index 000000000..0d9d82c67 --- /dev/null +++ b/packages/microcosm-build/src/microcosm/build/chronicle_epoch.py @@ -0,0 +1,294 @@ +"""Epoch tolerance for Chronicle (formerly Ledger) identities. + +PolicyEngine Ledger is being renamed Chronicle. Per the migration spec on +PolicyEngine/chronicle#143 the rename of *identities* happens **by epoch, +never in place**: Chronicle introduces chronicle-era hash domains +(``chronicle.aggregate_fact.v3``, ``chronicle.semantic_fact.v3``, ...) and +chronicle-named schema ids (``policyengine_chronicle.consumer_artifact.v2``, +``chronicle.consumer_fact.v2``) for **newly emitted** rows at a declared +cutover release. Ledger-era ids (``ledger.aggregate_fact.v2``, +``policyengine_ledger.consumer_artifact.v1``, ...) stay valid *history* +forever: no golden regenerates and no witnessed row is rewritten. + +Microcosm is a consumer, so it must accept both eras **before** Chronicle +flips emit. This module is the single place that knows which identities are +which era. The rules it encodes: + +1. **Fact keys are opaque.** Microcosm never reconstructs a Chronicle key + from a payload, so it never needs to know how the domain string feeds the + hash. Comparison stays exact string equality, which is correct in both + eras; what changes is that a validator asking "is this a Chronicle fact + key?" must answer yes for either epoch. +2. **Never hard-code a single epoch in a validator.** Epoch detection here is + *structural* — the namespace segment of the key domain — not a lookup in a + frozen list of domain strings. Chronicle has declared the family-``v3`` + spellings for the two fact families named in the spec; the version numbers + the remaining families will take are not declared yet, and this module + deliberately does not guess them. +3. **Microcosm-minted keys are epoch-independent.** Keys Microcosm mints live + in Microcosm-owned namespaces (``microcosm.derived_fact.*``, + ``populace_us_trade.*``) and are frozen at v1 by microcosm#639. They carry + no Chronicle epoch, and a chronicle-era source row must mint the + byte-identical Microcosm key a ledger-era source row does — otherwise the + cutover would silently re-identify derived facts. +4. **Nothing renames on disk.** Diagnostic field names + (``ledger_aggregate_fact_key``, ``ledger_commit``), H5 attributes, + ``populace_*`` ids, goldens, and fixtures are frozen at v1. + +Only the identity strings the migration spec names explicitly are pinned as +literals here. Everything else is derived. +""" + +from __future__ import annotations + +import re +from collections.abc import Iterable, Mapping +from dataclasses import dataclass +from typing import Any + +__all__ = [ + "ACCEPTED_CONSUMER_ARTIFACT_SCHEMA_VERSIONS", + "ACCEPTED_CONSUMER_FACT_SCHEMA_VERSIONS", + "CHRONICLE_CONSUMER_ARTIFACT_SCHEMA_VERSION", + "CHRONICLE_CONSUMER_FACT_SCHEMA_VERSION", + "CHRONICLE_EPOCH", + "EPOCHS", + "FACT_KEY_FIELDS", + "FactKeyIdentity", + "LEDGER_CONSUMER_ARTIFACT_SCHEMA_VERSION", + "LEDGER_CONSUMER_FACT_SCHEMA_VERSION", + "LEDGER_EPOCH", + "LEDGER_FACT_KEY_DOMAINS", + "consumer_artifact_schema_epoch", + "consumer_fact_schema_epoch", + "describe_accepted_consumer_artifact_schema_versions", + "describe_accepted_consumer_fact_schema_versions", + "fact_key_epoch", + "feed_fact_key_epochs", + "is_accepted_consumer_artifact_schema_version", + "is_accepted_consumer_fact_schema_version", + "is_chronicle_fact_key", + "parse_fact_key", + "row_fact_key_epochs", +] + +#: The ledger era: everything Chronicle emitted under its former name. +LEDGER_EPOCH = "ledger" + +#: The chronicle era: everything emitted from the declared cutover forward. +CHRONICLE_EPOCH = "chronicle" + +#: Both eras, oldest first. A validator iterates this; it never names one. +EPOCHS: tuple[str, ...] = (LEDGER_EPOCH, CHRONICLE_EPOCH) + +#: Consumer-artifact manifest ``schema_version``, per epoch. Both spellings +#: are declared in the chronicle#143 migration spec. +LEDGER_CONSUMER_ARTIFACT_SCHEMA_VERSION = "policyengine_ledger.consumer_artifact.v1" +CHRONICLE_CONSUMER_ARTIFACT_SCHEMA_VERSION = ( + "policyengine_chronicle.consumer_artifact.v2" +) + +#: Per-row consumer-fact ``schema_version``, per epoch. +LEDGER_CONSUMER_FACT_SCHEMA_VERSION = "ledger.consumer_fact.v1" +CHRONICLE_CONSUMER_FACT_SCHEMA_VERSION = "chronicle.consumer_fact.v2" + +CONSUMER_ARTIFACT_SCHEMA_VERSION_BY_EPOCH: Mapping[str, str] = { + LEDGER_EPOCH: LEDGER_CONSUMER_ARTIFACT_SCHEMA_VERSION, + CHRONICLE_EPOCH: CHRONICLE_CONSUMER_ARTIFACT_SCHEMA_VERSION, +} +CONSUMER_FACT_SCHEMA_VERSION_BY_EPOCH: Mapping[str, str] = { + LEDGER_EPOCH: LEDGER_CONSUMER_FACT_SCHEMA_VERSION, + CHRONICLE_EPOCH: CHRONICLE_CONSUMER_FACT_SCHEMA_VERSION, +} + +#: Membership sets. Loaders test membership, never equality with one era. +ACCEPTED_CONSUMER_ARTIFACT_SCHEMA_VERSIONS = frozenset( + CONSUMER_ARTIFACT_SCHEMA_VERSION_BY_EPOCH.values() +) +ACCEPTED_CONSUMER_FACT_SCHEMA_VERSIONS = frozenset( + CONSUMER_FACT_SCHEMA_VERSION_BY_EPOCH.values() +) + +#: Ledger-era fact-key domains, as *observed* in the feeds and fixtures this +#: repo carries. Recorded for documentation and for the frozen-history tests; +#: epoch detection does not consult it, so a family this list has never seen +#: still resolves to its epoch. +LEDGER_FACT_KEY_DOMAINS: Mapping[str, str] = { + "aggregate_fact": "ledger.aggregate_fact.v2", + "semantic_fact": "ledger.semantic_fact.v2", + "fact": "ledger.fact.v1", + "source_cell": "ledger.source_cell.v1", + "dimension_set": "ledger.dimension_set.v2", + "concept_alignment": "ledger.concept_alignment.v2", +} + +#: Consumer-fact row fields that carry a single Chronicle key, in the order +#: :mod:`microcosm.build.ledger_targets` resolves them. +FACT_KEY_FIELDS: tuple[str, ...] = ( + "aggregate_fact_key", + "semantic_fact_key", + "fact_key", + "legacy_fact_key", + "dimension_set_key", +) + +#: Row fields carrying a *list* of Chronicle keys. +_FACT_KEY_LIST_PATHS: tuple[tuple[str, ...], ...] = (("lineage", "source_cell_keys"),) + +#: Nested single-key paths. +_FACT_KEY_NESTED_PATHS: tuple[tuple[str, ...], ...] = ( + ("concept_alignment", "concept_alignment_key"), +) + +_VERSION_PATTERN = re.compile(r"^v\d+$") + + +@dataclass(frozen=True) +class FactKeyIdentity: + """The structure of a Chronicle-shaped key, ``:``. + + ``epoch`` is ``"ledger"`` or ``"chronicle"`` for Chronicle-issued keys and + ``None`` for keys minted in some other namespace — Microcosm's own + ``microcosm.derived_fact.*`` and ``populace_us_trade.*`` derived keys, for + instance, which are deliberately outside both eras. + """ + + domain: str + namespace: str + family: str + version: str + digest: str + epoch: str | None + + +def parse_fact_key(key: object) -> FactKeyIdentity | None: + """Split ``..v:``; ``None`` if not that shape. + + Parsing is deliberately shallow. Microcosm does not recompute Chronicle + digests, so it needs only enough structure to answer "which epoch issued + this key?" — and it must answer that without a frozen list of domain + strings, so a chronicle-era family this code has never seen still resolves. + """ + if not isinstance(key, str) or not key: + return None + domain, separator, digest = key.partition(":") + if not separator or not domain or not digest: + return None + segments = domain.split(".") + if len(segments) < 3: + return None + version = segments[-1] + if not _VERSION_PATTERN.match(version): + return None + namespace = segments[0] + family = ".".join(segments[1:-1]) + if not namespace or not family: + return None + return FactKeyIdentity( + domain=domain, + namespace=namespace, + family=family, + version=version, + digest=digest, + epoch=namespace if namespace in EPOCHS else None, + ) + + +def fact_key_epoch(key: object) -> str | None: + """The epoch that issued ``key``, or ``None`` for a foreign namespace.""" + identity = parse_fact_key(key) + return None if identity is None else identity.epoch + + +def is_chronicle_fact_key(key: object) -> bool: + """True when ``key`` is a Chronicle-issued key of *either* epoch.""" + return fact_key_epoch(key) is not None + + +def row_fact_key_epochs(row: object) -> frozenset[str]: + """Every Chronicle epoch appearing in one consumer-fact row's keys. + + Empty for a row whose keys are all Microcosm-minted: those namespaces are + outside both eras by design. + """ + if not isinstance(row, Mapping): + return frozenset() + epochs: set[str] = set() + for field in FACT_KEY_FIELDS: + epoch = fact_key_epoch(row.get(field)) + if epoch is not None: + epochs.add(epoch) + for path in _FACT_KEY_NESTED_PATHS: + epoch = fact_key_epoch(_at(row, path)) + if epoch is not None: + epochs.add(epoch) + for path in _FACT_KEY_LIST_PATHS: + values = _at(row, path) + if isinstance(values, Iterable) and not isinstance(values, str | bytes): + for value in values: + epoch = fact_key_epoch(value) + if epoch is not None: + epochs.add(epoch) + return frozenset(epochs) + + +def feed_fact_key_epochs(rows: Iterable[Any]) -> tuple[str, ...]: + """Chronicle epochs observed across a whole feed, in :data:`EPOCHS` order. + + A mixed-epoch feed — ledger-era history beside chronicle-era rows — is + expected during the cutover window and is reported, not rejected. + """ + observed: set[str] = set() + for row in rows: + observed |= row_fact_key_epochs(row) + return tuple(epoch for epoch in EPOCHS if epoch in observed) + + +def is_accepted_consumer_artifact_schema_version(value: object) -> bool: + """True for the consumer-artifact manifest schema id of either epoch.""" + return value in ACCEPTED_CONSUMER_ARTIFACT_SCHEMA_VERSIONS + + +def is_accepted_consumer_fact_schema_version(value: object) -> bool: + """True for the per-row consumer-fact schema id of either epoch.""" + return value in ACCEPTED_CONSUMER_FACT_SCHEMA_VERSIONS + + +def consumer_artifact_schema_epoch(value: object) -> str | None: + """The epoch of a consumer-artifact schema id, or ``None`` if unknown.""" + return _epoch_of(value, CONSUMER_ARTIFACT_SCHEMA_VERSION_BY_EPOCH) + + +def consumer_fact_schema_epoch(value: object) -> str | None: + """The epoch of a per-row consumer-fact schema id, or ``None``.""" + return _epoch_of(value, CONSUMER_FACT_SCHEMA_VERSION_BY_EPOCH) + + +def describe_accepted_consumer_artifact_schema_versions() -> str: + """Both accepted artifact schema ids, oldest era first, for messages.""" + return _describe(CONSUMER_ARTIFACT_SCHEMA_VERSION_BY_EPOCH) + + +def describe_accepted_consumer_fact_schema_versions() -> str: + """Both accepted per-row fact schema ids, oldest era first, for messages.""" + return _describe(CONSUMER_FACT_SCHEMA_VERSION_BY_EPOCH) + + +def _describe(by_epoch: Mapping[str, str]) -> str: + return ", ".join(repr(by_epoch[epoch]) for epoch in EPOCHS) + + +def _epoch_of(value: object, by_epoch: Mapping[str, str]) -> str | None: + for epoch in EPOCHS: + if value == by_epoch[epoch]: + return epoch + return None + + +def _at(row: Mapping[str, Any], path: tuple[str, ...]) -> Any: + current: Any = row + for segment in path: + if not isinstance(current, Mapping): + return None + current = current.get(segment) + return current diff --git a/packages/microcosm-build/src/microcosm/build/ledger_artifact.py b/packages/microcosm-build/src/microcosm/build/ledger_artifact.py index ff9991a5e..57457e65d 100644 --- a/packages/microcosm-build/src/microcosm/build/ledger_artifact.py +++ b/packages/microcosm-build/src/microcosm/build/ledger_artifact.py @@ -1,16 +1,25 @@ -"""Pinned loading for PolicyEngine Ledger consumer artifacts. +"""Pinned loading for PolicyEngine Chronicle consumer artifacts. -Ledger publishes consumer artifacts: a directory with ``manifest.json`` -(schema version, content hashes, embedded profile hashes), +Chronicle (formerly Ledger) publishes consumer artifacts: a directory with +``manifest.json`` (schema version, content hashes, embedded profile hashes), ``consumer_facts.jsonl``, per-profile JSON, and coverage diagnostics. A Microcosm build should consume facts through this loader so the release -manifest can record exactly which Ledger data resolved its target values +manifest can record exactly which Chronicle data resolved its target values (PolicyEngine/microcosm#160, #271) and so tampered or mismatched feeds fail before they calibrate anything. -Like the rest of Microcosm's Ledger consumption, this module is duck-typed -against the published artifact contract (stdlib only); it does not import -the Ledger implementation package. +Like the rest of Microcosm's Chronicle consumption, this module is duck-typed +against the published artifact contract (stdlib only); it does not import the +Chronicle implementation package. + +**Both eras load.** The manifest's ``schema_version`` is checked for +membership in :data:`ACCEPTED_CONSUMER_ARTIFACT_SCHEMA_VERSIONS`, never for +equality with one era, so an artifact published after Chronicle's rename +cutover loads here without a code change — see +:mod:`microcosm.build.chronicle_epoch` and PolicyEngine/chronicle#143. The +observed id, its epoch, and the fact-key epochs present in the feed are all +recorded in :meth:`LedgerConsumerArtifact.provenance`, so a release manifest +witnesses which era it actually consumed rather than which era it assumed. """ from __future__ import annotations @@ -22,7 +31,19 @@ from pathlib import Path from typing import Any +from microcosm.build.chronicle_epoch import ( + ACCEPTED_CONSUMER_ARTIFACT_SCHEMA_VERSIONS, + LEDGER_CONSUMER_ARTIFACT_SCHEMA_VERSION, + consumer_artifact_schema_epoch, + describe_accepted_consumer_artifact_schema_versions, + describe_accepted_consumer_fact_schema_versions, + feed_fact_key_epochs, + is_accepted_consumer_artifact_schema_version, + is_accepted_consumer_fact_schema_version, +) + __all__ = [ + "ACCEPTED_CONSUMER_ARTIFACT_SCHEMA_VERSIONS", "ALLOWED_LEDGER_ASSERTIONS", "CONSUMER_ARTIFACT_SCHEMA_VERSION", "DEFAULT_LEDGER_ASSERTION", @@ -32,21 +53,28 @@ "resolve_ledger_artifact", ] -CONSUMER_ARTIFACT_SCHEMA_VERSION = "policyengine_ledger.consumer_artifact.v1" +#: The era Microcosm's own minted artifacts still declare. Frozen at v1 by +#: microcosm#639; loading is governed by the accepted *set*, not by this. +CONSUMER_ARTIFACT_SCHEMA_VERSION = LEDGER_CONSUMER_ARTIFACT_SCHEMA_VERSION ALLOWED_LEDGER_ASSERTIONS = frozenset(("observation", "source_projection")) DEFAULT_LEDGER_ASSERTION = "observation" @dataclass(frozen=True) class LedgerConsumerArtifact: - """A loaded, hash-verified Ledger consumer fact feed. + """A loaded, hash-verified Chronicle consumer fact feed. ``manifest`` and ``manifest_sha256`` are ``None`` when the feed was a bare ``consumer_facts.jsonl`` file rather than an artifact directory; bare feeds are still content-addressed by ``facts_sha256`` so builds can - pin them, but they carry no Ledger-side provenance. Rows are exactly as + pin them, but they carry no Chronicle-side provenance. Rows are exactly as published: an ``assertion`` field is validated when present and never fabricated when absent (missing means observation-by-default to readers). + + The class name is ledger-era and stays: renaming an exported symbol + Microcosm's tools and experiments import buys nothing the alias in + :mod:`microcosm.build` does not, and chronicle#143 migrates identities, + not vocabulary. """ path: Path @@ -60,8 +88,36 @@ def fact_row_count(self) -> int: """Number of consumer fact rows in the feed.""" return len(self.facts) + @property + def schema_version(self) -> str | None: + """The schema id this feed actually declared, verbatim.""" + if self.manifest is None: + return None + schema_version = self.manifest.get("schema_version") + return None if schema_version is None else str(schema_version) + + @property + def schema_epoch(self) -> str | None: + """``"ledger"`` or ``"chronicle"`` for the declared schema id.""" + return consumer_artifact_schema_epoch(self.schema_version) + + @property + def fact_key_epochs(self) -> tuple[str, ...]: + """Chronicle epochs observed across the feed's fact keys. + + Empty when the feed carries only Microcosm-minted keys, and both + epochs when a cutover-window feed mixes ledger-era history with + chronicle-era rows. + """ + return feed_fact_key_epochs(self.facts) + def provenance(self) -> dict[str, Any]: - """Ledger-artifact identity block for build and release manifests.""" + """Chronicle-artifact identity block for build and release manifests. + + Records the schema id as *observed*, plus the epoch it belongs to and + the epochs of the fact keys in the feed, so a manifest witnesses which + era of Chronicle actually resolved its targets. + """ payload: dict[str, Any] = { "path_name": self.path.name, "fact_row_count": self.fact_row_count, @@ -82,6 +138,8 @@ def provenance(self) -> dict[str, Any]: else: payload["schema_version"] = None payload["manifest_sha256"] = None + payload["schema_epoch"] = self.schema_epoch + payload["fact_key_epochs"] = list(self.fact_key_epochs) return payload @@ -123,11 +181,14 @@ def load_ledger_consumer_artifact( f"Ledger consumer artifact manifest must be an object: {manifest_path}" ) schema_version = manifest.get("schema_version") - if schema_version != CONSUMER_ARTIFACT_SCHEMA_VERSION: + # Membership, not equality: ledger-era and chronicle-era artifacts are + # the same contract under two names, and both must load through the + # rename cutover (chronicle#143). + if not is_accepted_consumer_artifact_schema_version(schema_version): raise ValueError( - "Unsupported Ledger consumer artifact schema_version " - f"{schema_version!r}; expected " - f"{CONSUMER_ARTIFACT_SCHEMA_VERSION!r}." + "Unsupported Chronicle consumer artifact schema_version " + f"{schema_version!r}; expected one of " + f"{describe_accepted_consumer_artifact_schema_versions()}." ) else: facts_path = artifact_path @@ -222,9 +283,22 @@ def _load_fact_rows(path: Path) -> tuple[dict[str, Any], ...]: ) from exc if not isinstance(row, dict): raise ValueError( - f"Invalid Ledger facts JSONL row {line_number}: expected " + f"Invalid Chronicle facts JSONL row {line_number}: expected " f"object, got {type(row).__name__}." ) + # Chronicle-published rows have never carried a per-row schema id; + # only Microcosm-minted feeds stamp one. Validate it when it is + # there — against both eras — and never demand it when it is not. + row_schema_version = row.get("schema_version") + if row_schema_version is not None and ( + not is_accepted_consumer_fact_schema_version(row_schema_version) + ): + raise ValueError( + f"Chronicle facts JSONL row {line_number} declares " + f"unsupported schema_version {row_schema_version!r}; " + "expected one of " + f"{describe_accepted_consumer_fact_schema_versions()}." + ) assertion = row.get("assertion", DEFAULT_LEDGER_ASSERTION) if assertion not in ALLOWED_LEDGER_ASSERTIONS: raise ValueError( diff --git a/packages/microcosm-build/src/microcosm/build/logbook.py b/packages/microcosm-build/src/microcosm/build/logbook.py index 5047bb77a..de7d404b4 100644 --- a/packages/microcosm-build/src/microcosm/build/logbook.py +++ b/packages/microcosm-build/src/microcosm/build/logbook.py @@ -7,12 +7,16 @@ a best-effort Supabase REST insert. Remote availability is never part of build correctness. If either -``POPULACE_LEDGER_URL`` or ``POPULACE_LEDGER_KEY`` is absent, the validated row -stays in spool-only mode without error. Network and HTTP failures are returned -as receipt data and never raised. Local validation and durable-spool failures +``CHRONICLE_URL`` or ``CHRONICLE_KEY`` is absent, the validated row stays in +spool-only mode without error. Network and HTTP failures are returned as +receipt data and never raised. Local validation and durable-spool failures remain fatal: callers must not claim an attempt was recorded if its local row was invalid or could not be persisted. +The ledger-era spellings (``POPULACE_LEDGER_URL``, ``POPULACE_LEDGER_KEY``, +``POPULACE_LEDGER_API_KEY``) are still honored for the chronicle#143 dual-read +window and warn once per process; see :mod:`microcosm.build.chronicle_env`. + The caller owns chain coordination and must provide ``prev_row_digest`` for the ledger head it extends. The row digest is @@ -28,7 +32,7 @@ The Supabase key must identify the migration's ``logbook_writer`` role, not the service role. Hosted Supabase projects should additionally provide the -project gateway key as ``POPULACE_LEDGER_API_KEY``; single-key deployments may +project gateway key as ``CHRONICLE_API_KEY``; single-key deployments may omit it. The ``logbook`` schema must also be enabled in the hosted project's PostgREST exposed-schema setting. The US stacked driver and the three UK drivers record through this seam; this module remains driver-agnostic. @@ -56,6 +60,14 @@ from urllib.parse import urlencode, urlsplit from urllib.request import HTTPRedirectHandler, Request, build_opener +from microcosm.build.chronicle_env import ( + CHRONICLE_API_KEY_ENV, + CHRONICLE_KEY_ENV, + CHRONICLE_URL_ENV, + LEGACY_API_KEY_ENV, + chronicle_env, +) + __all__ = [ "BUILD_DISPOSITIONS", "LOGBOOK_ROW_FIELDS", @@ -91,7 +103,11 @@ _DIGEST_PATTERN = re.compile(r"^[0-9a-f]{64}$") _BUILD_ID_PATTERN = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._:-]{0,254}$") LOGBOOK_RUNGS = frozenset({"f001", "f004", "f010", "f025", "f100"}) -LEDGER_API_KEY_ENV = "POPULACE_LEDGER_API_KEY" +#: Ledger-era name for the project gateway key, kept because callers and +#: operator runbooks still name it. Reads go through +#: :func:`microcosm.build.chronicle_env.chronicle_env`, which prefers +#: ``CHRONICLE_API_KEY`` and warns once when it falls back to this one. +LEDGER_API_KEY_ENV = LEGACY_API_KEY_ENV LOGBOOK_ROW_FIELDS = frozenset( { "build_id", @@ -820,11 +836,11 @@ def urlopen(request: Request, *, timeout: float) -> Any: def _remote_config() -> tuple[str, str, str] | None: - url = os.environ.get("POPULACE_LEDGER_URL") - key = os.environ.get("POPULACE_LEDGER_KEY") + url = chronicle_env(CHRONICLE_URL_ENV) + key = chronicle_env(CHRONICLE_KEY_ENV) if not url or not key: return None - api_key = os.environ.get(LEDGER_API_KEY_ENV) or key + api_key = chronicle_env(CHRONICLE_API_KEY_ENV) or key return url, key, api_key diff --git a/packages/microcosm-build/src/microcosm/build/uk_runtime/firm_generation.py b/packages/microcosm-build/src/microcosm/build/uk_runtime/firm_generation.py index 8e4da91cb..9867276a1 100644 --- a/packages/microcosm-build/src/microcosm/build/uk_runtime/firm_generation.py +++ b/packages/microcosm-build/src/microcosm/build/uk_runtime/firm_generation.py @@ -159,6 +159,16 @@ class UKFirmLedgerTargetProfile: "greater_than_10m": "Greater_than_£10m", } +# Chronicle-era spellings of the three band maps above. These identifiers only +# look like environment variables to a grep — they are plain support-layout +# data, so they take aliases rather than the dual-read window in +# :mod:`microcosm.build.chronicle_env`. The ledger-era names stay: they are what +# the generator below imports, and the value ids they translate are published +# Chronicle ids, unaffected by the rename. +CHRONICLE_ONS_TURNOVER_BANDS = LEDGER_ONS_TURNOVER_BANDS +CHRONICLE_ONS_EMPLOYMENT_BANDS = LEDGER_ONS_EMPLOYMENT_BANDS +CHRONICLE_HMRC_BANDS = LEDGER_HMRC_BANDS + HMRC_BAND_COLUMNS: tuple[str, ...] = ( "Negative_or_Zero", "£1_to_Threshold", diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/source_coverage.py b/packages/microcosm-build/src/microcosm/build/us_runtime/source_coverage.py index 76214ad50..ab0bfea06 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/source_coverage.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/source_coverage.py @@ -18,6 +18,7 @@ from microcosm.build.gates import GateResult, source_coverage_gate __all__ = [ + "CHRONICLE_US_SOURCE_COVERAGE_CONTRACT_COMMIT", "LEDGER_US_SOURCE_COVERAGE_CONTRACT_COMMIT", "CoverageRole", "SourceCoverageEntry", @@ -31,6 +32,13 @@ ] LEDGER_US_SOURCE_COVERAGE_CONTRACT_COMMIT = "e2fc882c35f9203c788a7159e7b08b0b5e5ceb58" +#: Chronicle-era spelling of the same pin. This identifier only *looks* like an +#: environment variable to a grep — it is a git commit of the source-coverage +#: contract, so it takes an alias rather than the dual-read window in +#: :mod:`microcosm.build.chronicle_env`. The ledger-era name stays exported: it +#: is what callers import, and the ``ledger_commit`` diagnostic field it feeds +#: is frozen at v1 (microcosm#639). +CHRONICLE_US_SOURCE_COVERAGE_CONTRACT_COMMIT = LEDGER_US_SOURCE_COVERAGE_CONTRACT_COMMIT CoverageRole = Literal["hard_target", "validation_only", "source_gap"] diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/us_trade/import_entry_facts.py b/packages/microcosm-build/src/microcosm/build/us_runtime/us_trade/import_entry_facts.py index fd953f178..2e84e05d6 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/us_trade/import_entry_facts.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/us_trade/import_entry_facts.py @@ -1,6 +1,6 @@ -"""Ledger consumer-artifact emission for US import-entry margins. +"""Chronicle consumer-artifact emission for US import-entry margins. -Microcosm's ledger leg consumes sha-pinned consumer artifacts +Microcosm's Chronicle leg consumes sha-pinned consumer artifacts (``manifest.json`` + ``consumer_facts.jsonl``, :mod:`microcosm.build.ledger_artifact`) whose rows follow the ``ledger.consumer_fact.v1`` shape that :mod:`microcosm.build.ledger_targets` @@ -8,6 +8,15 @@ margin series to that exact contract from the Census ingest and archived CBP statistics. +The contract has a chronicle-era spelling too (``chronicle.consumer_fact.v2`` +under ``policyengine_chronicle.consumer_artifact.v2``, chronicle#143), and the +loader accepts both. What this module *emits* stays ledger-era: the rows are +byte-pinned goldens, and they are minted from official source bytes rather +than derived from a Chronicle row whose epoch they would have to inherit. +:func:`write_consumer_artifact` takes the declared id as an argument, checked +for membership in both eras, so flipping the declaration later is a caller +change rather than a code change. + Producer identity is explicit: these rows are **microcosm-minted** from official source bytes — not an export of a PolicyEngine/ledger build — and every row's ``source.extraction_method`` and the artifact manifest's @@ -39,6 +48,13 @@ import pandas as pd +from microcosm.build.chronicle_epoch import ( + ACCEPTED_CONSUMER_ARTIFACT_SCHEMA_VERSIONS, + LEDGER_CONSUMER_ARTIFACT_SCHEMA_VERSION, + LEDGER_CONSUMER_FACT_SCHEMA_VERSION, + describe_accepted_consumer_artifact_schema_versions, + is_accepted_consumer_artifact_schema_version, +) from microcosm.build.us_runtime.us_trade.cbp_entry_stats import ( CBP_TRADE_STATS_URL, CbpEntryStats, @@ -50,6 +66,7 @@ from microcosm.build.us_runtime.us_trade.imdb_bulk import IMDB_URL_TEMPLATE __all__ = [ + "ACCEPTED_CONSUMER_ARTIFACT_SCHEMA_VERSIONS", "ALL_IMPORT_ENTRY_FACT_GRAINS", "CONSUMER_ARTIFACT_SCHEMA_VERSION", "DISTRICT_ENTRY_FACT_GRAIN", @@ -138,8 +155,17 @@ class FactSourceLeg: ), ) -CONSUMER_ARTIFACT_SCHEMA_VERSION = "policyengine_ledger.consumer_artifact.v1" -_FACT_SCHEMA_VERSION = "ledger.consumer_fact.v1" +#: The schema ids these microcosm-minted rows *declare*. Both stay ledger-era: +#: the emitted bytes are golden-pinned (microcosm#639) and these facts are +#: minted from Census/CBP source bytes, so there is no chronicle-era source row +#: whose epoch they could inherit. Chronicle's own cutover moves what Chronicle +#: emits, not what Microcosm mints. Acceptance is the separate, dual-era +#: question, and it lives in :mod:`microcosm.build.chronicle_epoch`. +CONSUMER_ARTIFACT_SCHEMA_VERSION = LEDGER_CONSUMER_ARTIFACT_SCHEMA_VERSION +_FACT_SCHEMA_VERSION = LEDGER_CONSUMER_FACT_SCHEMA_VERSION +#: Microcosm's own key namespace: outside both Chronicle eras by construction, +#: so these keys can never collide with or impersonate Chronicle-built keys — +#: and a Chronicle-era cutover cannot silently re-identify them. _KEY_NAMESPACE = "populace_us_trade" #: Margins-feed grains (the aggregations of the HTS10 × country margins @@ -583,15 +609,26 @@ def write_consumer_artifact( *, retrieval_manifest: Iterable[Mapping[str, Any]], generator: Mapping[str, Any], + schema_version: str = CONSUMER_ARTIFACT_SCHEMA_VERSION, ) -> dict[str, Any]: """Write ``consumer_facts.jsonl`` + ``manifest.json`` and return the manifest. The directory loads through :func:`microcosm.build.ledger_artifact.load_ledger_consumer_artifact`, which re-hashes the fact file against ``manifest.facts_sha256``. + + ``schema_version`` defaults to the ledger-era id these artifacts have + always declared and is validated for membership in both eras, so the + chronicle-era cutover is a caller decision rather than an edit here. """ if not fact_rows: raise ValueError("Refusing to write an empty consumer artifact.") + if not is_accepted_consumer_artifact_schema_version(schema_version): + raise ValueError( + f"Refusing to write a consumer artifact declaring schema_version " + f"{schema_version!r}; expected one of " + f"{describe_accepted_consumer_artifact_schema_versions()}." + ) retrieval_entries = [dict(entry) for entry in retrieval_manifest] if not retrieval_entries: raise ValueError( @@ -621,7 +658,7 @@ def write_consumer_artifact( facts_path.write_bytes(payload) retrievals = retrieval_entries manifest = { - "schema_version": CONSUMER_ARTIFACT_SCHEMA_VERSION, + "schema_version": schema_version, "facts_sha256": hashlib.sha256(payload).hexdigest(), "fact_row_count": len(fact_rows), "profiles": {}, diff --git a/tools/logbook.py b/tools/logbook.py index 78aaee9d4..afea050ef 100644 --- a/tools/logbook.py +++ b/tools/logbook.py @@ -7,8 +7,11 @@ archive or a directory of them, reporting chain by chain. Remote export uses a distinct, read-only ``logbook_exporter`` JWT supplied -as ``POPULACE_LEDGER_EXPORT_KEY`` plus the hosted project's gateway key in -``POPULACE_LEDGER_API_KEY``. It never reuses the insert-only writer key. +as ``CHRONICLE_EXPORT_KEY`` plus the hosted project's gateway key in +``CHRONICLE_API_KEY``. It never reuses the insert-only writer key. The +ledger-era spellings (``POPULACE_LEDGER_EXPORT_KEY``, +``POPULACE_LEDGER_API_KEY``, ``POPULACE_LEDGER_URL``) stay honored for the +chronicle#143 dual-read window and warn once per process. The live store is row-oriented and carries every attempt across all scopes; the per-scope split is an archive convention, not a database partition. """ @@ -17,7 +20,6 @@ import argparse import json -import os import re import sys from pathlib import Path @@ -25,6 +27,15 @@ from urllib.parse import urlencode from urllib.request import Request +from microcosm.build.chronicle_env import ( + CHRONICLE_API_KEY_ENV, + CHRONICLE_EXPORT_KEY_ENV, + CHRONICLE_URL_ENV, + LEGACY_API_KEY_ENV, + LEGACY_EXPORT_KEY_ENV, + chronicle_env, + describe_chronicle_env, +) from microcosm.build.logbook import ( LOGBOOK_ROW_FIELDS, LogbookRow, @@ -46,8 +57,11 @@ #: chain is born, and country is the outermost one. DEFAULT_ARCHIVE_ROOT = ROOT / "logbook" DEFAULT_SPOOL_ROOT = ROOT / "logbook-spool" -REMOTE_EXPORT_KEY_ENV = "POPULACE_LEDGER_EXPORT_KEY" -REMOTE_API_KEY_ENV = "POPULACE_LEDGER_API_KEY" +#: Ledger-era names, still honored through the chronicle#143 dual-read window. +#: Reads go through :func:`microcosm.build.chronicle_env.chronicle_env`, which +#: prefers ``CHRONICLE_EXPORT_KEY`` / ``CHRONICLE_API_KEY``. +REMOTE_EXPORT_KEY_ENV = LEGACY_EXPORT_KEY_ENV +REMOTE_API_KEY_ENV = LEGACY_API_KEY_ENV REMOTE_PAGE_SIZE = 500 # Mirror of logbook.chain_scope() in # supabase/migrations/20260818000000_logbook_chain_scopes.sql. The legacy US @@ -243,13 +257,15 @@ def _archive_scope(archive: Path) -> str: def _remote_rows(scope: str) -> tuple[LogbookRow, ...]: - ledger_url = os.environ.get("POPULACE_LEDGER_URL") - export_key = os.environ.get(REMOTE_EXPORT_KEY_ENV) - api_key = os.environ.get(REMOTE_API_KEY_ENV) + ledger_url = chronicle_env(CHRONICLE_URL_ENV) + export_key = chronicle_env(CHRONICLE_EXPORT_KEY_ENV) + api_key = chronicle_env(CHRONICLE_API_KEY_ENV) if not ledger_url or not export_key or not api_key: raise ValueError( - "remote export requires POPULACE_LEDGER_URL, " - f"{REMOTE_EXPORT_KEY_ENV}, and {REMOTE_API_KEY_ENV}" + "remote export requires " + + describe_chronicle_env( + CHRONICLE_URL_ENV, CHRONICLE_EXPORT_KEY_ENV, CHRONICLE_API_KEY_ENV + ) ) rows: list[LogbookRow] = [] From ca98a9c7617cf8a51bd24574343139d8033bcb30 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Wed, 2 Sep 2026 11:04:33 -0400 Subject: [PATCH 03/23] Test dual-era acceptance across the fact-consumption surface MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Covers the epoch module itself, the loader under each schema id, a mixed-epoch feed straddling the cutover, mixed-epoch target compilation and a chronicle-era reference pin, and the two producer surfaces: the minted US trade artifact declares the ledger-era id but loads under either, and the congressional-district vintage translation mints byte-identical microcosm.derived_fact keys whether its source rows are ledger-era or chronicle-era — otherwise cutover day would silently re-identify every derived fact a target is pinned to. Also documents the env dual-read in logbook/README.md and adds the towncrier fragment. Co-Authored-By: Claude Fable 5.1 --- PROGRESS-chronicle-dual-accept.md | 48 ++- changelog.d/chronicle-dual-accept.added.md | 1 + logbook/README.md | 6 +- .../tests/test_chronicle_env.py | 176 +++++++++ .../tests/test_chronicle_epoch.py | 344 ++++++++++++++++++ .../tests/test_ledger_targets.py | 89 +++++ .../test_us_congressional_district_vintage.py | 160 ++++++++ .../tests/test_us_trade_facts.py | 71 ++++ 8 files changed, 884 insertions(+), 11 deletions(-) create mode 100644 changelog.d/chronicle-dual-accept.added.md create mode 100644 packages/microcosm-build/tests/test_chronicle_env.py create mode 100644 packages/microcosm-build/tests/test_chronicle_epoch.py diff --git a/PROGRESS-chronicle-dual-accept.md b/PROGRESS-chronicle-dual-accept.md index 47cd999f8..118003a21 100644 --- a/PROGRESS-chronicle-dual-accept.md +++ b/PROGRESS-chronicle-dual-accept.md @@ -18,21 +18,51 @@ Microcosm must accept BOTH ledger-era and chronicle-era Chronicle identities deprecation warning Frozen (microcosm#639): nothing on disk or in artifacts renames. Diagnostic -field names (`ledger_aggregate_fact_key`), H5 attrs, `populace_*` ids, fact -keys, goldens and fixtures stay at v1. +field names (`ledger_aggregate_fact_key`, `ledger_commit`), H5 attrs, +`populace_*` ids, fact keys, goldens and fixtures stay at v1. ## State -- [ ] not started +Implementation and tests landed; verification run recorded in `out.md`. ## Done -(nothing yet) +- `microcosm/build/chronicle_epoch.py` — the single epoch authority. Epoch + detection is **structural** (the namespace segment of a key domain), not a + lookup in a frozen domain list, because chronicle#143 declares the `v3` + spelling only for the aggregate and semantic fact families. Only identity + strings the spec names explicitly are pinned as literals. +- `microcosm/build/chronicle_env.py` — the env dual-read window, one helper, + one `DeprecationWarning` per process per legacy name. +- `ledger_artifact.py` — manifest `schema_version` is a membership test over + both eras; a per-row `schema_version` is validated when present (Chronicle + rows have never carried one) and never demanded; `provenance()` records the + observed id, `schema_epoch`, and `fact_key_epochs`. +- `us_trade/import_entry_facts.py` — emission stays ledger-era (bytes are + pinned; these rows are minted from Census/CBP bytes, so there is no source + epoch to inherit), but the declared id is now an argument checked against + both eras. +- Chronicle-era aliases for the four `LEDGER_*` module constants that only + look like env vars to a grep. +- Tests: `test_chronicle_epoch.py`, `test_chronicle_env.py`, plus mixed-epoch + cases in `test_ledger_targets.py`, epoch-independence of the minted + `microcosm.derived_fact.*` keys in + `test_us_congressional_district_vintage.py`, and dual-era emission and + acceptance in `test_us_trade_facts.py`. + +## Audit result + +An 8-surface / 42-agent adversarial audit of every site that compares, +parses, or mints a Chronicle fact key or schema id found exactly three +hard-coded epoch literals in non-test source — `ledger_artifact.py:35`/`:126` +and `import_entry_facts.py:141`/`:142` — all handled here. `ledger_targets.py` +carries keys opaquely and contains no epoch literal at all; +`congressional_district_vintage.py` mints into Microcosm-owned namespaces that +sit outside both eras; the UK runtime is clean; no golden embeds a Chronicle +domain and no H5 attribute name embeds `ledger`. Every other candidate site +was refuted on verification. ## Next -1. Read the primary files, run the epoch-sensitivity audit. -2. Land the shared epoch module. -3. Dual-accept the two consumer-artifact loaders. -4. Env dual-read helper. -5. Tests + full verification. +Nothing outstanding on this branch. Chronicle's own acceptance half lands in +the parallel lane; the emit flip is a separate, later cutover. diff --git a/changelog.d/chronicle-dual-accept.added.md b/changelog.d/chronicle-dual-accept.added.md new file mode 100644 index 000000000..472213d7e --- /dev/null +++ b/changelog.d/chronicle-dual-accept.added.md @@ -0,0 +1 @@ +Accept chronicle-era Chronicle identities everywhere Microcosm consumes them: consumer-artifact manifests declaring `policyengine_chronicle.consumer_artifact.v2` and fact rows declaring `chronicle.consumer_fact.v2` load beside their ledger-era spellings, mixed-epoch feeds compile, and the loader records the observed schema id, its epoch, and the feed's fact-key epochs in provenance. Environment variables gain a `CHRONICLE_*` dual-read window with the `POPULACE_LEDGER_*` names honored under a once-per-process deprecation warning. Nothing renames on disk: emitted ids, diagnostic field names, `populace_*` key namespaces, goldens and fixtures stay frozen at v1. diff --git a/logbook/README.md b/logbook/README.md index a148dd142..47f43e6fa 100644 --- a/logbook/README.md +++ b/logbook/README.md @@ -90,8 +90,10 @@ for `published` and `certified` rows. ## The live store -The best-effort Supabase insert (`POPULACE_LEDGER_URL` + -`POPULACE_LEDGER_KEY`, the migration's insert-only `logbook_writer` role) +The best-effort Supabase insert (`CHRONICLE_URL` + `CHRONICLE_KEY`, the +migration's insert-only `logbook_writer` role; the ledger-era +`POPULACE_LEDGER_URL` / `POPULACE_LEDGER_KEY` stay honored for the +chronicle#143 dual-read window and warn once per process) uses the same scope rule as the archives after this PR's migration is applied by the project owner. The writer key is unaffected: rows are still inserted through the same role and the scope is derived from the hashed `pipeline` diff --git a/packages/microcosm-build/tests/test_chronicle_env.py b/packages/microcosm-build/tests/test_chronicle_env.py new file mode 100644 index 000000000..76697a4b2 --- /dev/null +++ b/packages/microcosm-build/tests/test_chronicle_env.py @@ -0,0 +1,176 @@ +"""The CHRONICLE_* / POPULACE_LEDGER_* environment dual-read window. + +chronicle#143 gives the operational stores a dual-read window so publish +flows and build scripts migrate on their own schedule: ``CHRONICLE_*`` is +preferred, the ledger-era name is still honored, and honoring it warns once +per process. These tests hold both halves — the fallback keeps working, and +the warning stays a single line rather than one per read in a build loop. +""" + +from __future__ import annotations + +import warnings + +import pytest + +from microcosm.build.chronicle_env import ( + CHRONICLE_API_KEY_ENV, + CHRONICLE_ENV_LEGACY_NAMES, + CHRONICLE_EXPORT_KEY_ENV, + CHRONICLE_KEY_ENV, + CHRONICLE_URL_ENV, + LEGACY_API_KEY_ENV, + LEGACY_EXPORT_KEY_ENV, + LEGACY_KEY_ENV, + LEGACY_URL_ENV, + chronicle_env, + chronicle_env_names, + describe_chronicle_env, + reset_chronicle_env_deprecation_warnings, +) +from microcosm.build.logbook import _remote_config + +PAIRS = ( + (CHRONICLE_URL_ENV, LEGACY_URL_ENV), + (CHRONICLE_KEY_ENV, LEGACY_KEY_ENV), + (CHRONICLE_API_KEY_ENV, LEGACY_API_KEY_ENV), + (CHRONICLE_EXPORT_KEY_ENV, LEGACY_EXPORT_KEY_ENV), +) + + +@pytest.fixture(autouse=True) +def _clean_env(monkeypatch): + for preferred, legacy in PAIRS: + monkeypatch.delenv(preferred, raising=False) + monkeypatch.delenv(legacy, raising=False) + reset_chronicle_env_deprecation_warnings() + yield + reset_chronicle_env_deprecation_warnings() + + +@pytest.mark.parametrize(("preferred", "legacy"), PAIRS) +def test_preferred_name_wins_and_warns_about_nothing( + monkeypatch, preferred: str, legacy: str +) -> None: + monkeypatch.setenv(preferred, "chronicle-value") + monkeypatch.setenv(legacy, "ledger-value") + + with warnings.catch_warnings(): + warnings.simplefilter("error", DeprecationWarning) + assert chronicle_env(preferred) == "chronicle-value" + + +@pytest.mark.parametrize(("preferred", "legacy"), PAIRS) +def test_legacy_name_is_honored_with_a_deprecation_warning( + monkeypatch, preferred: str, legacy: str +) -> None: + monkeypatch.setenv(legacy, "ledger-value") + + with pytest.warns(DeprecationWarning) as record: + assert chronicle_env(preferred) == "ledger-value" + + message = str(record[0].message) + assert legacy in message + assert preferred in message + assert "chronicle#143" in message + + +def test_the_deprecation_warning_fires_once_per_process(monkeypatch) -> None: + """A build loop reads these repeatedly; one warning, not a storm.""" + monkeypatch.setenv(LEGACY_URL_ENV, "https://ledger.example") + + with warnings.catch_warnings(record=True) as caught: + warnings.simplefilter("always") + for _ in range(5): + assert chronicle_env(CHRONICLE_URL_ENV) == "https://ledger.example" + + assert [ + w for w in caught if issubclass(w.category, DeprecationWarning) + ].__len__() == 1 + + +def test_each_legacy_name_warns_on_its_own(monkeypatch) -> None: + monkeypatch.setenv(LEGACY_URL_ENV, "https://ledger.example") + monkeypatch.setenv(LEGACY_KEY_ENV, "writer-jwt") + + with warnings.catch_warnings(record=True) as caught: + warnings.simplefilter("always") + chronicle_env(CHRONICLE_URL_ENV) + chronicle_env(CHRONICLE_KEY_ENV) + + warned = { + legacy + for legacy in (LEGACY_URL_ENV, LEGACY_KEY_ENV) + if any(legacy in str(w.message) for w in caught) + } + assert warned == {LEGACY_URL_ENV, LEGACY_KEY_ENV} + + +def test_unset_returns_the_default_and_empty_counts_as_unset(monkeypatch) -> None: + assert chronicle_env(CHRONICLE_URL_ENV) is None + assert chronicle_env(CHRONICLE_URL_ENV, "fallback") == "fallback" + + monkeypatch.setenv(CHRONICLE_URL_ENV, "") + monkeypatch.setenv(LEGACY_URL_ENV, "https://ledger.example") + with pytest.warns(DeprecationWarning): + assert chronicle_env(CHRONICLE_URL_ENV) == "https://ledger.example" + + +def test_an_explicit_environ_mapping_bypasses_the_process_environment() -> None: + with pytest.warns(DeprecationWarning): + assert ( + chronicle_env(CHRONICLE_KEY_ENV, environ={LEGACY_KEY_ENV: "writer-jwt"}) + == "writer-jwt" + ) + + +def test_a_name_outside_the_window_is_a_programming_error() -> None: + with pytest.raises(KeyError, match="not a Chronicle environment variable"): + chronicle_env("CHRONICLE_NOT_A_REAL_VARIABLE") + with pytest.raises(KeyError, match="not a Chronicle environment variable"): + chronicle_env_names(LEGACY_URL_ENV) + + +def test_names_and_descriptions_carry_both_spellings() -> None: + assert chronicle_env_names(CHRONICLE_URL_ENV) == (CHRONICLE_URL_ENV, LEGACY_URL_ENV) + + described = describe_chronicle_env(CHRONICLE_URL_ENV, CHRONICLE_EXPORT_KEY_ENV) + # An operator whose environment predates the rename must still be able to + # match the error text against what they have set. + for name in ( + CHRONICLE_URL_ENV, + CHRONICLE_EXPORT_KEY_ENV, + LEGACY_URL_ENV, + LEGACY_EXPORT_KEY_ENV, + ): + assert name in described + + +def test_every_windowed_variable_maps_to_exactly_one_legacy_name() -> None: + assert dict(CHRONICLE_ENV_LEGACY_NAMES) == { + preferred: (legacy,) for preferred, legacy in PAIRS + } + + +def test_logbook_remote_config_reads_both_eras(monkeypatch) -> None: + assert _remote_config() is None + + monkeypatch.setenv(LEGACY_URL_ENV, "https://ledger.example") + monkeypatch.setenv(LEGACY_KEY_ENV, "writer-jwt") + with pytest.warns(DeprecationWarning): + assert _remote_config() == ( + "https://ledger.example", + "writer-jwt", + "writer-jwt", + ) + + monkeypatch.setenv(CHRONICLE_URL_ENV, "https://chronicle.example") + monkeypatch.setenv(CHRONICLE_KEY_ENV, "chronicle-jwt") + monkeypatch.setenv(CHRONICLE_API_KEY_ENV, "project-api-key") + with warnings.catch_warnings(): + warnings.simplefilter("error", DeprecationWarning) + assert _remote_config() == ( + "https://chronicle.example", + "chronicle-jwt", + "project-api-key", + ) diff --git a/packages/microcosm-build/tests/test_chronicle_epoch.py b/packages/microcosm-build/tests/test_chronicle_epoch.py new file mode 100644 index 000000000..915b44363 --- /dev/null +++ b/packages/microcosm-build/tests/test_chronicle_epoch.py @@ -0,0 +1,344 @@ +"""Dual-era acceptance of Chronicle (formerly Ledger) identities. + +Chronicle introduces chronicle-era hash domains and chronicle-named schema +ids for newly emitted rows at a declared cutover; ledger-era ids stay valid +history forever (PolicyEngine/chronicle#143). Microcosm consumes both, and +these tests are the contract that says so: no validator may reject a row for +being on the other side of the cutover, and no minted Microcosm identity may +move because a source row crossed it. +""" + +from __future__ import annotations + +import hashlib +import json + +import pytest + +from microcosm.build.chronicle_epoch import ( + ACCEPTED_CONSUMER_ARTIFACT_SCHEMA_VERSIONS, + ACCEPTED_CONSUMER_FACT_SCHEMA_VERSIONS, + CHRONICLE_CONSUMER_ARTIFACT_SCHEMA_VERSION, + CHRONICLE_CONSUMER_FACT_SCHEMA_VERSION, + CHRONICLE_EPOCH, + EPOCHS, + LEDGER_CONSUMER_ARTIFACT_SCHEMA_VERSION, + LEDGER_CONSUMER_FACT_SCHEMA_VERSION, + LEDGER_EPOCH, + LEDGER_FACT_KEY_DOMAINS, + consumer_artifact_schema_epoch, + fact_key_epoch, + feed_fact_key_epochs, + is_accepted_consumer_artifact_schema_version, + is_accepted_consumer_fact_schema_version, + is_chronicle_fact_key, + parse_fact_key, + row_fact_key_epochs, +) +from microcosm.build.ledger_artifact import load_ledger_consumer_artifact + +#: A chronicle-era row as the migration spec describes it: identical canonical +#: payload, ``chronicle.*.v3`` domain in place of ``ledger.*.v2``. +_CHRONICLE_AGGREGATE_KEY = "chronicle.aggregate_fact.v3:abc123" +_LEDGER_AGGREGATE_KEY = "ledger.aggregate_fact.v2:abc123" + + +def _fact_row(**overrides): + row = { + "aggregate_fact_key": _LEDGER_AGGREGATE_KEY, + "semantic_fact_key": "ledger.semantic_fact.v2:abc123", + "value": 100, + "period": {"type": "tax_year", "value": 2023}, + "geography": {"level": "country", "id": "0100000US"}, + "entity": {"name": "tax_unit"}, + "aggregation": {"method": "sum"}, + "observed_measure": {"source_name": "irs_soi", "unit": "usd"}, + "source": {"source_name": "irs_soi"}, + "lineage": { + "source_record_id": "irs_soi.ty2023.t.all.agi", + "source_cell_keys": ["ledger.source_cell.v1:cell"], + }, + } + row.update(overrides) + return row + + +def _chronicle_fact_row(**overrides): + """The same row after Chronicle's cutover: every key domain re-epoched.""" + row = _fact_row( + aggregate_fact_key=_CHRONICLE_AGGREGATE_KEY, + semantic_fact_key="chronicle.semantic_fact.v3:abc123", + ) + row["lineage"] = dict(row["lineage"]) + row["lineage"]["source_cell_keys"] = ["chronicle.source_cell.v3:cell"] + row.update(overrides) + return row + + +def _write_artifact_dir(tmp_path, rows, *, schema_version, name="artifact"): + artifact_dir = tmp_path / name + artifact_dir.mkdir() + payload = "".join(json.dumps(row, sort_keys=True) + "\n" for row in rows) + (artifact_dir / "consumer_facts.jsonl").write_text(payload) + manifest = { + "schema_version": schema_version, + "fact_row_count": len(rows), + "facts_sha256": hashlib.sha256(payload.encode()).hexdigest(), + "profiles": {"us_fiscal": {"sha256": "ab" * 32, "target_count": 3}}, + } + (artifact_dir / "manifest.json").write_text( + json.dumps(manifest, indent=2, sort_keys=True) + "\n" + ) + return artifact_dir + + +def test_parse_fact_key_splits_domain_namespace_family_and_version() -> None: + identity = parse_fact_key(_LEDGER_AGGREGATE_KEY) + + assert identity is not None + assert identity.domain == "ledger.aggregate_fact.v2" + assert identity.namespace == "ledger" + assert identity.family == "aggregate_fact" + assert identity.version == "v2" + assert identity.digest == "abc123" + assert identity.epoch == LEDGER_EPOCH + + +@pytest.mark.parametrize("domain", sorted(LEDGER_FACT_KEY_DOMAINS.values())) +def test_every_observed_ledger_era_domain_resolves_to_the_ledger_epoch( + domain: str, +) -> None: + assert fact_key_epoch(f"{domain}:digest") == LEDGER_EPOCH + + +def test_chronicle_era_keys_resolve_without_a_declared_version_number() -> None: + """Epoch detection is structural, so undeclared families still resolve. + + chronicle#143 names the ``v3`` spelling for the aggregate and semantic + fact families; it does not say which version number the remaining + families take. A validator that answered "unknown" for those would + fail closed on the cutover, so detection reads the namespace segment + rather than matching a frozen list of domain strings. + """ + for key in ( + _CHRONICLE_AGGREGATE_KEY, + "chronicle.semantic_fact.v3:abc123", + "chronicle.source_cell.v7:cell", + "chronicle.some_family_nobody_has_declared_yet.v11:digest", + ): + assert fact_key_epoch(key) == CHRONICLE_EPOCH + assert is_chronicle_fact_key(key) + + +def test_microcosm_minted_namespaces_are_outside_both_epochs() -> None: + """Microcosm's own derived keys must not be mistaken for Chronicle's. + + They are frozen at v1 by microcosm#639 and carry no Chronicle epoch, so + the cutover cannot re-identify them. + """ + for key in ( + "microcosm.derived_fact.congressional_district_vintage.v1:deadbeef", + "microcosm.semantic_fact.congressional_district_state_total_proxy.v1:dead", + "populace_us_trade.aggregate_fact.v1:digest", + "populace_us_trade.semantic_fact.v1:digest", + ): + assert fact_key_epoch(key) is None + assert not is_chronicle_fact_key(key) + + +@pytest.mark.parametrize( + "key", + [ + "", + "ledger.aggregate_fact.v2", # no digest + "ledger.aggregate_fact:digest", # no version + "ledger:digest", # no family + ":digest", + "not a key at all", + None, + 42, + ], +) +def test_non_key_shapes_parse_to_none_rather_than_raising(key) -> None: + assert parse_fact_key(key) is None + assert fact_key_epoch(key) is None + + +def test_row_epochs_cover_every_key_field_including_nested_lists() -> None: + ledger_row = _fact_row() + chronicle_row = _chronicle_fact_row() + + assert row_fact_key_epochs(ledger_row) == frozenset({LEDGER_EPOCH}) + assert row_fact_key_epochs(chronicle_row) == frozenset({CHRONICLE_EPOCH}) + + # A row whose only chronicle-era key is a nested source-cell key still + # reports both eras: lineage keys are Chronicle identities too. + straddling = _fact_row() + straddling["lineage"] = dict(straddling["lineage"]) + straddling["lineage"]["source_cell_keys"] = ["chronicle.source_cell.v3:cell"] + assert row_fact_key_epochs(straddling) == frozenset({LEDGER_EPOCH, CHRONICLE_EPOCH}) + + +def test_row_epochs_read_the_concept_alignment_key() -> None: + row = _fact_row( + concept_alignment={ + "concept_alignment_key": "chronicle.concept_alignment.v3:aligned", + "relation": "source_label", + } + ) + + assert row_fact_key_epochs(row) == frozenset({LEDGER_EPOCH, CHRONICLE_EPOCH}) + + +def test_feed_epochs_report_a_mixed_feed_in_epoch_order() -> None: + assert feed_fact_key_epochs([_fact_row()]) == (LEDGER_EPOCH,) + assert feed_fact_key_epochs([_chronicle_fact_row()]) == (CHRONICLE_EPOCH,) + assert feed_fact_key_epochs([_chronicle_fact_row(), _fact_row()]) == EPOCHS + assert ( + feed_fact_key_epochs([{"aggregate_fact_key": "populace_us_trade.a.v1:x"}]) == () + ) + assert feed_fact_key_epochs([]) == () + + +def test_schema_id_membership_covers_both_eras_and_nothing_else() -> None: + assert ACCEPTED_CONSUMER_ARTIFACT_SCHEMA_VERSIONS == { + LEDGER_CONSUMER_ARTIFACT_SCHEMA_VERSION, + CHRONICLE_CONSUMER_ARTIFACT_SCHEMA_VERSION, + } + assert ACCEPTED_CONSUMER_FACT_SCHEMA_VERSIONS == { + LEDGER_CONSUMER_FACT_SCHEMA_VERSION, + CHRONICLE_CONSUMER_FACT_SCHEMA_VERSION, + } + assert is_accepted_consumer_artifact_schema_version( + CHRONICLE_CONSUMER_ARTIFACT_SCHEMA_VERSION + ) + assert is_accepted_consumer_fact_schema_version( + CHRONICLE_CONSUMER_FACT_SCHEMA_VERSION + ) + assert not is_accepted_consumer_artifact_schema_version( + "policyengine_chronicle.other.v9" + ) + assert not is_accepted_consumer_fact_schema_version("chronicle.consumer_fact.v1") + + +@pytest.mark.parametrize( + ("schema_version", "expected_epoch"), + [ + (LEDGER_CONSUMER_ARTIFACT_SCHEMA_VERSION, LEDGER_EPOCH), + (CHRONICLE_CONSUMER_ARTIFACT_SCHEMA_VERSION, CHRONICLE_EPOCH), + ], +) +def test_artifact_loads_under_each_schema_id_and_records_the_observed_one( + tmp_path, schema_version: str, expected_epoch: str +) -> None: + rows = [ + _fact_row() if expected_epoch == LEDGER_EPOCH else _chronicle_fact_row(), + ] + artifact_dir = _write_artifact_dir(tmp_path, rows, schema_version=schema_version) + + artifact = load_ledger_consumer_artifact(artifact_dir) + + assert artifact.fact_row_count == 1 + assert artifact.schema_version == schema_version + assert artifact.schema_epoch == expected_epoch + provenance = artifact.provenance() + # The id is recorded as observed, not as assumed. + assert provenance["schema_version"] == schema_version + assert provenance["schema_epoch"] == expected_epoch + assert provenance["fact_key_epochs"] == [expected_epoch] + assert ( + consumer_artifact_schema_epoch(provenance["schema_version"]) == expected_epoch + ) + + +def test_mixed_epoch_feed_loads_and_witnesses_both_eras(tmp_path) -> None: + """The cutover window: ledger-era history beside chronicle-era rows. + + Both must calibrate, and the manifest must say the feed straddled the + cutover rather than silently reporting one era. + """ + rows = [ + _fact_row(), + _chronicle_fact_row(), + # Microcosm-minted rows carry neither epoch and must not perturb it. + _fact_row( + aggregate_fact_key="populace_us_trade.aggregate_fact.v1:digest", + semantic_fact_key="populace_us_trade.semantic_fact.v1:digest", + lineage={"source_record_id": "us_trade.month_2025_01.national"}, + ), + ] + artifact_dir = _write_artifact_dir( + tmp_path, rows, schema_version=CHRONICLE_CONSUMER_ARTIFACT_SCHEMA_VERSION + ) + + artifact = load_ledger_consumer_artifact(artifact_dir) + + assert artifact.fact_row_count == 3 + assert artifact.fact_key_epochs == EPOCHS + assert artifact.provenance()["fact_key_epochs"] == list(EPOCHS) + + +def test_bare_feed_has_no_schema_epoch_but_still_reports_fact_key_epochs( + tmp_path, +) -> None: + facts_path = tmp_path / "consumer_facts.jsonl" + facts_path.write_text( + "".join( + json.dumps(row, sort_keys=True) + "\n" + for row in (_fact_row(), _chronicle_fact_row()) + ) + ) + + artifact = load_ledger_consumer_artifact(facts_path) + + provenance = artifact.provenance() + assert provenance["schema_version"] is None + assert provenance["schema_epoch"] is None + assert provenance["fact_key_epochs"] == list(EPOCHS) + + +def test_unknown_schema_id_is_rejected_naming_both_accepted_eras(tmp_path) -> None: + artifact_dir = _write_artifact_dir( + tmp_path, [_fact_row()], schema_version="policyengine_chronicle.other.v9" + ) + + with pytest.raises(ValueError) as excinfo: + load_ledger_consumer_artifact(artifact_dir) + + message = str(excinfo.value) + assert "schema_version" in message + assert LEDGER_CONSUMER_ARTIFACT_SCHEMA_VERSION in message + assert CHRONICLE_CONSUMER_ARTIFACT_SCHEMA_VERSION in message + + +@pytest.mark.parametrize( + "row_schema_version", + [LEDGER_CONSUMER_FACT_SCHEMA_VERSION, CHRONICLE_CONSUMER_FACT_SCHEMA_VERSION], +) +def test_per_row_schema_id_is_accepted_in_either_era( + tmp_path, row_schema_version: str +) -> None: + facts_path = tmp_path / "consumer_facts.jsonl" + facts_path.write_text( + json.dumps(_fact_row(schema_version=row_schema_version), sort_keys=True) + "\n" + ) + + artifact = load_ledger_consumer_artifact(facts_path) + + assert artifact.facts[0]["schema_version"] == row_schema_version + + +def test_per_row_schema_id_is_optional_but_validated_when_present(tmp_path) -> None: + # Chronicle-published rows have never carried one; demanding it would + # reject every real feed. + bare = tmp_path / "bare.jsonl" + bare.write_text(json.dumps(_fact_row(), sort_keys=True) + "\n") + assert "schema_version" not in load_ledger_consumer_artifact(bare).facts[0] + + wrong = tmp_path / "wrong.jsonl" + wrong.write_text( + json.dumps(_fact_row(schema_version="ledger.consumer_fact.v99"), sort_keys=True) + + "\n" + ) + with pytest.raises(ValueError, match="unsupported schema_version"): + load_ledger_consumer_artifact(wrong) diff --git a/packages/microcosm-build/tests/test_ledger_targets.py b/packages/microcosm-build/tests/test_ledger_targets.py index cd46e2cad..60e6e7837 100644 --- a/packages/microcosm-build/tests/test_ledger_targets.py +++ b/packages/microcosm-build/tests/test_ledger_targets.py @@ -2939,3 +2939,92 @@ def test_exact_period_contract_sum_keeps_equivalent_untyped_annual_range_cells() ).specs assert spec.value == 30.0 # Both synthetic cells belong to the same academic year. + + +def test__given_mixed_epoch_fact_feed__then_both_eras_compile_to_targets() -> None: + """Ledger-era and chronicle-era rows calibrate side by side. + + During Chronicle's rename cutover a feed carries history under + ``ledger.*`` domains beside newly emitted rows under ``chronicle.*`` + (PolicyEngine/chronicle#143). Keys are opaque to this compiler, so both + must select — and each target's name and metadata must carry its own + row's key verbatim rather than being normalised onto one epoch. + """ + # Given + ledger_era = _consumer_fact_row() + chronicle_era = _consumer_fact_row( + aggregate_fact_key="chronicle.aggregate_fact.v3:def456", + semantic_fact_key="chronicle.semantic_fact.v3:def456", + lineage={ + "source_record_id": "irs_soi.ty2024.table_1_1.all.adjusted_gross_income", + "source_cell_keys": ["chronicle.source_cell.v3:cell"], + "source_row_keys": [], + }, + ) + chronicle_era.pop("legacy_fact_key", None) + mapping = LedgerTargetMapping( + measure_by_concept={ + "us:statutes/26/62#adjusted_gross_income": "adjusted_gross_income" + }, + entity_by_ledger_entity={"tax_unit": "tax_unit"}, + filter_by_domain={"all_individual_income_tax_returns": "is_tax_return"}, + ) + + # When + selection = select_ledger_targets([ledger_era, chronicle_era], mapping) + + # Then + assert not selection.unsupported + assert [spec.name for spec in selection.specs] == [ + "ledger.aggregate_fact.v2:abc123", + "chronicle.aggregate_fact.v3:def456", + ] + chronicle_spec = selection.specs[1] + assert ( + chronicle_spec.metadata["ledger_fact_key"] + == "chronicle.aggregate_fact.v3:def456" + ) + # The diagnostic field names stay ledger-era: they are frozen at v1 + # (microcosm#639) and name a slot, not an epoch. + assert ( + chronicle_spec.metadata["ledger_aggregate_fact_key"] + == "chronicle.aggregate_fact.v3:def456" + ) + assert ( + chronicle_spec.metadata["ledger_semantic_fact_key"] + == "chronicle.semantic_fact.v3:def456" + ) + + +def test__given_chronicle_era_reference_pin__then_it_resolves_against_the_feed() -> ( + None +): + """A reference pinned to a chronicle-era key resolves without a code change.""" + # Given + reference = LedgerTargetReference( + name="nation/irs/adjusted gross income/total", + ledger_fact_key="chronicle.aggregate_fact.v3:def456", + entity="tax_unit", + measure="adjusted_gross_income", + filter="is_tax_return", + period=2024, + source="IRS SOI Table 1.1", + family="irs_soi", + ) + fact = _consumer_fact_row( + aggregate_fact_key="chronicle.aggregate_fact.v3:def456", + semantic_fact_key="chronicle.semantic_fact.v3:def456", + ) + fact.pop("legacy_fact_key", None) + + # When + registry = compile_ledger_target_references([fact], [reference], country="us") + + # Then + assert [spec.name for spec in registry.specs] == [ + "nation/irs/adjusted gross income/total" + ] + assert ( + registry.specs[0].metadata["ledger_fact_key"] + == "chronicle.aggregate_fact.v3:def456" + ) diff --git a/packages/microcosm-build/tests/test_us_congressional_district_vintage.py b/packages/microcosm-build/tests/test_us_congressional_district_vintage.py index 364b2fcfa..cd51eeb07 100644 --- a/packages/microcosm-build/tests/test_us_congressional_district_vintage.py +++ b/packages/microcosm-build/tests/test_us_congressional_district_vintage.py @@ -1,5 +1,7 @@ """US congressional-district geography-vintage translation tests.""" +import json + import pytest from microcosm.build.us_runtime import ( @@ -732,3 +734,161 @@ def _ledger_fact_for_reference(reference, *, value: float) -> dict[str, object]: "url": "https://example.org/reference", }, } + + +def _re_epoch_to_chronicle(fact: dict[str, object]) -> dict[str, object]: + """The same source fact as Chronicle emits it after the rename cutover. + + Identical canonical payload; ``ledger..v2`` domains replaced by + their ``chronicle..v3`` siblings (PolicyEngine/chronicle#143). + """ + chronicle = dict(fact) + for field, domain in ( + ("aggregate_fact_key", "chronicle.aggregate_fact.v3"), + ("semantic_fact_key", "chronicle.semantic_fact.v3"), + ): + value = chronicle.get(field) + if isinstance(value, str): + chronicle[field] = f"{domain}:{value.split(':', 1)[1]}" + chronicle.pop("legacy_fact_key", None) + return chronicle + + +def test__given_chronicle_era_source_rows__then_derived_cd_keys_are_unchanged() -> None: + """The Chronicle rename must not re-identify Microcosm-derived facts. + + Translated and proxy facts are minted into Microcosm's own + ``microcosm.derived_fact.*`` / ``microcosm.semantic_fact.*`` namespaces, + which sit outside both Chronicle eras and are frozen at v1 + (microcosm#639). Their digests are computed from the source row's + *semantic* identity, never from its key, so a feed that crosses the + cutover must mint byte-identical derived keys — otherwise every target + pinned to one would silently stop resolving on cutover day. + """ + ledger_era_facts = [ + _soi_cd_fact( + "adjusted_gross_income", + 100.0, + geography_id="5001700US0601", + source_row_id="ca_01", + ), + _soi_cd_fact( + "adjusted_gross_income", + 60.0, + geography_id="5001700US0653", + source_row_id="ca_53", + ), + ] + chronicle_era_facts = [_re_epoch_to_chronicle(fact) for fact in ledger_era_facts] + crosswalk = [ + { + "source_geography_id": "5001700US0601", + "target_geography_id": "5001900US0601", + "weight": 1.0, + }, + { + "source_geography_id": "5001700US0653", + "target_geography_id": "5001900US0602", + "weight": 1.0, + }, + ] + + def _translate(facts): + return translate_congressional_district_facts_to_current_vintage( + facts, crosswalk, crosswalk_basis="block_population" + ) + + from_ledger = _translate(ledger_era_facts) + from_chronicle = _translate(chronicle_era_facts) + + def _keys(translated): + return sorted( + ( + fact["geography"]["id"], + fact["aggregate_fact_key"], + fact["semantic_fact_key"], + ) + for fact in translated + ) + + assert _keys(from_chronicle) == _keys(from_ledger) + for fact in from_chronicle: + assert fact["aggregate_fact_key"].startswith( + "microcosm.derived_fact.congressional_district_vintage.v1:" + ) + assert fact["semantic_fact_key"].startswith( + "microcosm.semantic_fact.congressional_district_vintage.v1:" + ) + # The derived row carries neither epoch: it is Microcosm's fact now. + assert "legacy_fact_key" not in fact + + +def test__given_chronicle_era_state_rows__then_proxy_cd_keys_are_unchanged() -> None: + """The state-total-proxy mint is epoch-independent for the same reason. + + Its digest covers the source record id, the proxy district, and the + value — never the source row's Chronicle key — so a chronicle-era state + fact produces the same proxy identity a ledger-era one does. + """ + ledger_era_facts = [ + _soi_cd_fact( + "adjusted_gross_income", + 10.0, + geography_id="5001700US0601", + source_row_id="ca_01", + ), + _soi_state_fact( + "adjusted_gross_income", + 100.0, + geography_id="0400000US30", + source_row_id="mt_total", + ), + ] + chronicle_era_facts = [_re_epoch_to_chronicle(fact) for fact in ledger_era_facts] + crosswalk = [ + { + "source_geography_id": "5001700US0601", + "target_geography_id": "5001900US0601", + "weight": 1.0, + }, + { + "source_geography_id": "5001700US3000", + "target_geography_id": "5001900US3001", + "weight": 2.0, + }, + { + "source_geography_id": "5001700US3000", + "target_geography_id": "5001900US3002", + "weight": 3.0, + }, + ] + + def _translate(facts): + return translate_congressional_district_facts_to_current_vintage( + facts, crosswalk, crosswalk_basis="block_population" + ) + + def _derived_identity(translated): + # Passthrough rows keep their own Chronicle keys verbatim — that is the + # opaque-carriage rule. Only the Microcosm-minted rows are compared. + return sorted( + ( + fact["geography"]["id"], + fact["aggregate_fact_key"], + fact["semantic_fact_key"], + json.dumps(fact["lineage"], sort_keys=True), + ) + for fact in translated + if str(fact.get("aggregate_fact_key", "")).startswith("microcosm.") + ) + + from_chronicle = _derived_identity(_translate(chronicle_era_facts)) + + assert from_chronicle == _derived_identity(_translate(ledger_era_facts)) + # The proxy digest reaches the compared identity through the lineage: the + # proxy fact is minted, then translated, so its source_record_id (digest + # included) is what the derived row records. + assert any( + "congressional_district_state_total_proxy_source_record_id" in lineage + for _, _, _, lineage in from_chronicle + ) diff --git a/packages/microcosm-build/tests/test_us_trade_facts.py b/packages/microcosm-build/tests/test_us_trade_facts.py index f20d443f6..08ab5b100 100644 --- a/packages/microcosm-build/tests/test_us_trade_facts.py +++ b/packages/microcosm-build/tests/test_us_trade_facts.py @@ -7,6 +7,11 @@ import pandas as pd import pytest +from microcosm.build.chronicle_epoch import ( + CHRONICLE_CONSUMER_ARTIFACT_SCHEMA_VERSION, + LEDGER_CONSUMER_ARTIFACT_SCHEMA_VERSION, + LEDGER_CONSUMER_FACT_SCHEMA_VERSION, +) from microcosm.build.ledger_artifact import load_ledger_consumer_artifact from microcosm.build.ledger_targets import ( LedgerTargetReference, @@ -813,3 +818,69 @@ def test_cbp_and_district_facts_compile_into_ledger_targets(tmp_path): by_name = {spec.name: spec for spec in registry.specs} assert by_name["cbp_total_entry_summaries_fytd2026"].value == 83_133_856 assert by_name["district70_customs_value_2026_01"].value == 2_000_000 + + +def test_minted_artifact_declares_the_ledger_era_id_and_loads_under_either(tmp_path): + """Emission stays ledger-era; acceptance covers both (chronicle#143). + + These rows are minted from Census/CBP source bytes, not derived from a + Chronicle row whose epoch they could inherit, and their bytes are pinned, + so the declared id must not move on Chronicle's cutover. The declaration + is nonetheless an argument rather than a literal, checked against both + eras, so flipping it later is a caller decision. + """ + rows = build_import_entry_fact_rows( + _margins(), + retrieval_manifest=_manifest_entries(), + extracted_at="2026-08-05T00:00:00+00:00", + ) + + default_manifest = write_consumer_artifact( + tmp_path / "default", + rows, + retrieval_manifest=_manifest_entries(), + generator=default_generator_block(months=("2026-01", "2026-02")), + ) + assert default_manifest["schema_version"] == LEDGER_CONSUMER_ARTIFACT_SCHEMA_VERSION + # Every row's key stays in Microcosm's own frozen namespace, outside both + # Chronicle eras, so the cutover cannot re-identify them (microcosm#639). + assert all( + row["aggregate_fact_key"].startswith("populace_us_trade.aggregate_fact.v1:") + for row in rows + ) + assert all( + row["schema_version"] == LEDGER_CONSUMER_FACT_SCHEMA_VERSION for row in rows + ) + + chronicle_manifest = write_consumer_artifact( + tmp_path / "chronicle", + rows, + retrieval_manifest=_manifest_entries(), + generator=default_generator_block(months=("2026-01", "2026-02")), + schema_version=CHRONICLE_CONSUMER_ARTIFACT_SCHEMA_VERSION, + ) + assert ( + chronicle_manifest["schema_version"] + == CHRONICLE_CONSUMER_ARTIFACT_SCHEMA_VERSION + ) + # Only the declaration moved: the fact bytes are identical either way. + assert chronicle_manifest["facts_sha256"] == default_manifest["facts_sha256"] + + for directory, expected_epoch in ( + (tmp_path / "default", "ledger"), + (tmp_path / "chronicle", "chronicle"), + ): + artifact = load_ledger_consumer_artifact(directory) + assert artifact.fact_row_count == len(rows) + assert artifact.schema_epoch == expected_epoch + # Microcosm-minted keys belong to neither era, so nothing is claimed. + assert artifact.provenance()["fact_key_epochs"] == [] + + with pytest.raises(ValueError, match="Refusing to write a consumer artifact"): + write_consumer_artifact( + tmp_path / "bogus", + rows, + retrieval_manifest=_manifest_entries(), + generator=default_generator_block(months=("2026-01",)), + schema_version="policyengine_chronicle.consumer_artifact.v9", + ) From 0aff29b5e516e01a301dabb291f9ff71ef62c8e8 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Wed, 2 Sep 2026 11:21:10 -0400 Subject: [PATCH 04/23] Carry per-row Chronicle schema ids instead of gating on them MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Dual acceptance must widen what loads and narrow nothing. The loader as committed added a per-row schema_version check that rejected any id outside the two chronicle#143 names — a validation main never performed, on a field main never read. Real feeds do not honor that set. The pinned US fiscal-refresh feed consumer_facts_buildn_v9_4.jsonl, which the release path loads through this loader, declares 'arch.consumer_fact.v1' on 37,006 of its first 200,000 rows and 'ledger.consumer_fact.v1' on 399 of them; test_uk_firm_generation builds the same arch-namespace rows. The check therefore failed the build closed on its own pinned input, and PR CI could not see it: the feed is gated data outside PR CI. Verified by loading an arch-namespace row through load_ledger_consumer_artifact — ValueError before, loads after. The id is now carried and reported through fact_schema_versions, which provenance records beside the artifact schema id and the fact-key epochs, so a release manifest still witnesses exactly which producers wrote its rows. Epoch witnessing was also reading only the four key fields targets resolve a fact by. Published rows carry nine, plus two key lists: the captured UK feed fixture has observed_measure_key, source_release_key, source_series_key, universe_constraint_set_key and lineage.source_row_keys as well. Chronicle moves families independently — the spec declares v3 only for the aggregate and semantic families — so a straddling row with a ledger-era aggregate key and a chronicle-era source-release key was witnessed as pure ledger-era. Co-Authored-By: Claude Fable 5.1 --- .../src/microcosm/build/chronicle_epoch.py | 19 +++- .../src/microcosm/build/ledger_artifact.py | 52 +++++---- .../tests/test_chronicle_epoch.py | 102 ++++++++++++++++-- 3 files changed, 143 insertions(+), 30 deletions(-) diff --git a/packages/microcosm-build/src/microcosm/build/chronicle_epoch.py b/packages/microcosm-build/src/microcosm/build/chronicle_epoch.py index 0d9d82c67..0eb1eed0b 100644 --- a/packages/microcosm-build/src/microcosm/build/chronicle_epoch.py +++ b/packages/microcosm-build/src/microcosm/build/chronicle_epoch.py @@ -122,18 +122,31 @@ "concept_alignment": "ledger.concept_alignment.v2", } -#: Consumer-fact row fields that carry a single Chronicle key, in the order -#: :mod:`microcosm.build.ledger_targets` resolves them. +#: Consumer-fact row fields that carry a single Chronicle key. The first four +#: are the identifiers :mod:`microcosm.build.ledger_targets` resolves a fact +#: by; the rest are the remaining single-key fields published rows carry, all +#: of them observed on the captured feed in +#: ``tests/fixtures/uk_target_reference_feed_rows.jsonl``. The list is for +#: *witnessing* epochs, so it is deliberately wider than the resolution set: a +#: row whose aggregate key is still ledger-era but whose source-release key has +#: crossed the cutover must report both epochs, not one. FACT_KEY_FIELDS: tuple[str, ...] = ( "aggregate_fact_key", "semantic_fact_key", "fact_key", "legacy_fact_key", "dimension_set_key", + "observed_measure_key", + "source_release_key", + "source_series_key", + "universe_constraint_set_key", ) #: Row fields carrying a *list* of Chronicle keys. -_FACT_KEY_LIST_PATHS: tuple[tuple[str, ...], ...] = (("lineage", "source_cell_keys"),) +_FACT_KEY_LIST_PATHS: tuple[tuple[str, ...], ...] = ( + ("lineage", "source_cell_keys"), + ("lineage", "source_row_keys"), +) #: Nested single-key paths. _FACT_KEY_NESTED_PATHS: tuple[tuple[str, ...], ...] = ( diff --git a/packages/microcosm-build/src/microcosm/build/ledger_artifact.py b/packages/microcosm-build/src/microcosm/build/ledger_artifact.py index 57457e65d..6a08f219a 100644 --- a/packages/microcosm-build/src/microcosm/build/ledger_artifact.py +++ b/packages/microcosm-build/src/microcosm/build/ledger_artifact.py @@ -17,9 +17,16 @@ equality with one era, so an artifact published after Chronicle's rename cutover loads here without a code change — see :mod:`microcosm.build.chronicle_epoch` and PolicyEngine/chronicle#143. The -observed id, its epoch, and the fact-key epochs present in the feed are all -recorded in :meth:`LedgerConsumerArtifact.provenance`, so a release manifest -witnesses which era it actually consumed rather than which era it assumed. +observed id, its epoch, the fact-key epochs present in the feed, and the +per-row schema ids the rows themselves declare are all recorded in +:meth:`LedgerConsumerArtifact.provenance`, so a release manifest witnesses +which era it actually consumed rather than which era it assumed. + +Acceptance widened here; nothing narrowed. Only the *manifest* schema id is +gated, and only against the two ids chronicle#143 declares. Row-level ids and +fact keys are carried as published — real feeds mint rows in namespaces that +belong to neither era, and a consumer that rejected them would fail closed on +data that has always loaded. """ from __future__ import annotations @@ -36,10 +43,8 @@ LEDGER_CONSUMER_ARTIFACT_SCHEMA_VERSION, consumer_artifact_schema_epoch, describe_accepted_consumer_artifact_schema_versions, - describe_accepted_consumer_fact_schema_versions, feed_fact_key_epochs, is_accepted_consumer_artifact_schema_version, - is_accepted_consumer_fact_schema_version, ) __all__ = [ @@ -111,6 +116,21 @@ def fact_key_epochs(self) -> tuple[str, ...]: """ return feed_fact_key_epochs(self.facts) + @property + def fact_schema_versions(self) -> tuple[str, ...]: + """Distinct per-row ``schema_version`` values in the feed, sorted. + + Reported verbatim and never gated on — see :func:`_load_fact_rows`. + Empty when no row declares one; more than one entry when a feed mixes + producers, which the pinned US fiscal-refresh feed already does. + """ + observed = { + str(row["schema_version"]) + for row in self.facts + if isinstance(row, dict) and row.get("schema_version") is not None + } + return tuple(sorted(observed)) + def provenance(self) -> dict[str, Any]: """Chronicle-artifact identity block for build and release manifests. @@ -140,6 +160,7 @@ def provenance(self) -> dict[str, Any]: payload["manifest_sha256"] = None payload["schema_epoch"] = self.schema_epoch payload["fact_key_epochs"] = list(self.fact_key_epochs) + payload["fact_schema_versions"] = list(self.fact_schema_versions) return payload @@ -286,19 +307,14 @@ def _load_fact_rows(path: Path) -> tuple[dict[str, Any], ...]: f"Invalid Chronicle facts JSONL row {line_number}: expected " f"object, got {type(row).__name__}." ) - # Chronicle-published rows have never carried a per-row schema id; - # only Microcosm-minted feeds stamp one. Validate it when it is - # there — against both eras — and never demand it when it is not. - row_schema_version = row.get("schema_version") - if row_schema_version is not None and ( - not is_accepted_consumer_fact_schema_version(row_schema_version) - ): - raise ValueError( - f"Chronicle facts JSONL row {line_number} declares " - f"unsupported schema_version {row_schema_version!r}; " - "expected one of " - f"{describe_accepted_consumer_fact_schema_versions()}." - ) + # A per-row ``schema_version`` is carried, never gated on. Real + # feeds stamp ids from namespaces that are neither era: the pinned + # US fiscal-refresh feed (consumer_facts_buildn_v9_4.jsonl) mixes + # 'arch.consumer_fact.v1' with 'ledger.consumer_fact.v1'. Rejecting + # an unrecognized id would fail the build closed on its own pinned + # input, and rejecting is not what dual acceptance asks for. The + # observed ids are reported through + # :attr:`LedgerConsumerArtifact.fact_schema_versions` instead. assertion = row.get("assertion", DEFAULT_LEDGER_ASSERTION) if assertion not in ALLOWED_LEDGER_ASSERTIONS: raise ValueError( diff --git a/packages/microcosm-build/tests/test_chronicle_epoch.py b/packages/microcosm-build/tests/test_chronicle_epoch.py index 915b44363..fd8532d5f 100644 --- a/packages/microcosm-build/tests/test_chronicle_epoch.py +++ b/packages/microcosm-build/tests/test_chronicle_epoch.py @@ -12,6 +12,7 @@ import hashlib import json +from pathlib import Path import pytest @@ -22,6 +23,7 @@ CHRONICLE_CONSUMER_FACT_SCHEMA_VERSION, CHRONICLE_EPOCH, EPOCHS, + FACT_KEY_FIELDS, LEDGER_CONSUMER_ARTIFACT_SCHEMA_VERSION, LEDGER_CONSUMER_FACT_SCHEMA_VERSION, LEDGER_EPOCH, @@ -328,17 +330,99 @@ def test_per_row_schema_id_is_accepted_in_either_era( assert artifact.facts[0]["schema_version"] == row_schema_version -def test_per_row_schema_id_is_optional_but_validated_when_present(tmp_path) -> None: - # Chronicle-published rows have never carried one; demanding it would - # reject every real feed. +def test_per_row_schema_id_is_optional(tmp_path) -> None: + # A row that declares no schema id must not acquire one: the loader + # reports what was published, and never fabricates the field. bare = tmp_path / "bare.jsonl" bare.write_text(json.dumps(_fact_row(), sort_keys=True) + "\n") assert "schema_version" not in load_ledger_consumer_artifact(bare).facts[0] - wrong = tmp_path / "wrong.jsonl" - wrong.write_text( - json.dumps(_fact_row(schema_version="ledger.consumer_fact.v99"), sort_keys=True) - + "\n" + +def test_per_row_schema_id_outside_both_eras_still_loads(tmp_path) -> None: + """Dual acceptance widens what loads; it must not narrow it. + + Real feeds stamp rows from namespaces that are neither era. The pinned US + fiscal-refresh feed ``consumer_facts_buildn_v9_4.jsonl`` declares + ``arch.consumer_fact.v1`` on the overwhelming majority of its rows and + ``ledger.consumer_fact.v1`` on the rest, and it loads through this loader + on the release path. A consumer that gated the per-row id against the two + ids chronicle#143 names would fail the build closed on its own pinned + input — so the id is carried and reported, never gated. + """ + facts_path = tmp_path / "consumer_facts.jsonl" + rows = [ + _fact_row(schema_version="arch.consumer_fact.v1"), + _fact_row(schema_version=LEDGER_CONSUMER_FACT_SCHEMA_VERSION), + _chronicle_fact_row(schema_version=CHRONICLE_CONSUMER_FACT_SCHEMA_VERSION), + _fact_row(schema_version="ledger.consumer_fact.v99"), + ] + facts_path.write_text( + "".join(json.dumps(row, sort_keys=True) + "\n" for row in rows) ) - with pytest.raises(ValueError, match="unsupported schema_version"): - load_ledger_consumer_artifact(wrong) + + artifact = load_ledger_consumer_artifact(facts_path) + + assert artifact.fact_row_count == len(rows) + assert artifact.fact_schema_versions == ( + "arch.consumer_fact.v1", + CHRONICLE_CONSUMER_FACT_SCHEMA_VERSION, + LEDGER_CONSUMER_FACT_SCHEMA_VERSION, + "ledger.consumer_fact.v99", + ) + # The unrecognized ids are reported verbatim in provenance, so a release + # manifest still witnesses exactly what it consumed. + assert artifact.provenance()["fact_schema_versions"] == [ + "arch.consumer_fact.v1", + CHRONICLE_CONSUMER_FACT_SCHEMA_VERSION, + LEDGER_CONSUMER_FACT_SCHEMA_VERSION, + "ledger.consumer_fact.v99", + ] + + +def test_every_published_key_field_is_witnessed_for_its_epoch() -> None: + """Epoch witnessing covers every Chronicle key a published row carries. + + The captured feed fixture carries nine single-key fields and two key + lists, not just the four identifiers targets resolve by. Chronicle's + cutover moves families independently — the spec declares ``v3`` only for + the aggregate and semantic families — so a row can straddle it: ledger-era + aggregate key, chronicle-era source-release key. Reading only the + resolution set would report that row as pure ledger-era. + """ + for field in ( + "observed_measure_key", + "source_release_key", + "source_series_key", + "universe_constraint_set_key", + ): + straddling = _fact_row(**{field: f"chronicle.{field[:-4]}.v3:straddle"}) + + assert row_fact_key_epochs(straddling) == frozenset( + {LEDGER_EPOCH, CHRONICLE_EPOCH} + ), field + + row_keys = _fact_row() + row_keys["lineage"] = dict(row_keys["lineage"]) + row_keys["lineage"]["source_row_keys"] = ["chronicle.source_row.v3:row"] + + assert row_fact_key_epochs(row_keys) == frozenset({LEDGER_EPOCH, CHRONICLE_EPOCH}) + + +def test_witnessed_key_fields_match_the_captured_feed() -> None: + """The inventory is grounded in a real feed, not in a guess. + + Every ``*_key`` field the captured UK feed rows carry is either witnessed + for its epoch or is a plain source identifier rather than a Chronicle key. + """ + fixture = Path(__file__).parent / "fixtures" / "uk_target_reference_feed_rows.jsonl" + rows = [json.loads(line) for line in fixture.read_text().splitlines() if line] + assert rows + + for row in rows: + for field, value in row.items(): + if not field.endswith("_key") or not isinstance(value, str): + continue + assert field in FACT_KEY_FIELDS, field + assert is_chronicle_fact_key(value), (field, value) + + assert feed_fact_key_epochs(rows) == (LEDGER_EPOCH,) From cff65d9e038e65c4f3f377b757cc8013eab1f07e Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Wed, 2 Sep 2026 11:23:41 -0400 Subject: [PATCH 05/23] Ground the observed ledger-era domain map in the feed it claims to describe The map documented six families and said it recorded what this repo's feeds carry. The captured UK feed fixture carries ten, and the pinned US feed carries eleven: observed_measure, source_release, source_series, universe_constraint_set and source_row were all missing. Nothing consults the map at runtime, which is precisely how such a list rots, so a test now pins it to the fixture. The completed map also shows why detection cannot be a lookup in it: the observed version numbers differ by family, with fact and source_cell at v1 where the rest are at v2. Co-Authored-By: Claude Fable 5.1 --- .../src/microcosm/build/chronicle_epoch.py | 17 +++++-- .../tests/test_chronicle_epoch.py | 50 ++++++++++++++++++- 2 files changed, 62 insertions(+), 5 deletions(-) diff --git a/packages/microcosm-build/src/microcosm/build/chronicle_epoch.py b/packages/microcosm-build/src/microcosm/build/chronicle_epoch.py index 0eb1eed0b..c84dd31b0 100644 --- a/packages/microcosm-build/src/microcosm/build/chronicle_epoch.py +++ b/packages/microcosm-build/src/microcosm/build/chronicle_epoch.py @@ -110,16 +110,27 @@ ) #: Ledger-era fact-key domains, as *observed* in the feeds and fixtures this -#: repo carries. Recorded for documentation and for the frozen-history tests; -#: epoch detection does not consult it, so a family this list has never seen -#: still resolves to its epoch. +#: repo carries: every family in +#: ``tests/fixtures/uk_target_reference_feed_rows.jsonl``, plus +#: ``source_row``, which only the far larger pinned US feed exercises. The +#: version numbers are the observed ones and differ by family — ``fact`` and +#: ``source_cell`` are still v1 while the rest are v2 — which is why epoch +#: detection is structural and never a lookup here. Recorded for documentation +#: and for the frozen-history tests; a family this list has never seen still +#: resolves to its epoch. LEDGER_FACT_KEY_DOMAINS: Mapping[str, str] = { "aggregate_fact": "ledger.aggregate_fact.v2", "semantic_fact": "ledger.semantic_fact.v2", + # Carried by ``legacy_fact_key``, not by a ``fact_key`` field. "fact": "ledger.fact.v1", "source_cell": "ledger.source_cell.v1", + "source_row": "ledger.source_row.v1", "dimension_set": "ledger.dimension_set.v2", "concept_alignment": "ledger.concept_alignment.v2", + "observed_measure": "ledger.observed_measure.v2", + "source_release": "ledger.source_release.v2", + "source_series": "ledger.source_series.v2", + "universe_constraint_set": "ledger.universe_constraint_set.v2", } #: Consumer-fact row fields that carry a single Chronicle key. The first four diff --git a/packages/microcosm-build/tests/test_chronicle_epoch.py b/packages/microcosm-build/tests/test_chronicle_epoch.py index fd8532d5f..f7cdca835 100644 --- a/packages/microcosm-build/tests/test_chronicle_epoch.py +++ b/packages/microcosm-build/tests/test_chronicle_epoch.py @@ -12,6 +12,7 @@ import hashlib import json +import re from pathlib import Path import pytest @@ -77,6 +78,32 @@ def _chronicle_fact_row(**overrides): return row +#: The captured Chronicle feed this repo carries. Its rows are ledger-era and +#: frozen: they are the only published Chronicle rows available in-tree. +_CAPTURED_FEED = ( + Path(__file__).parent / "fixtures" / "uk_target_reference_feed_rows.jsonl" +) + + +def _captured_feed_rows() -> list[dict]: + return [ + json.loads(line) + for line in _CAPTURED_FEED.read_text().splitlines() + if line.strip() + ] + + +def _keys_in(value, _pattern=re.compile(r"^[a-z][a-z0-9_]*(?:\.[a-z0-9_]+)+\.v\d+:")): + """Every Chronicle-shaped key anywhere in a row, at any depth.""" + if isinstance(value, str): + return [value] if _pattern.match(value) else [] + if isinstance(value, dict): + return [key for item in value.values() for key in _keys_in(item)] + if isinstance(value, list): + return [key for item in value for key in _keys_in(item)] + return [] + + def _write_artifact_dir(tmp_path, rows, *, schema_version, name="artifact"): artifact_dir = tmp_path / name artifact_dir.mkdir() @@ -113,6 +140,26 @@ def test_every_observed_ledger_era_domain_resolves_to_the_ledger_epoch( assert fact_key_epoch(f"{domain}:digest") == LEDGER_EPOCH +def test_observed_domain_map_covers_the_captured_feed() -> None: + """The map documents what the repo has actually seen, not a guess. + + ``LEDGER_FACT_KEY_DOMAINS`` claims to record the ledger-era families this + repo's feeds carry, and nothing consults it at runtime — which is exactly + how such a list rots. This pins it to the captured feed. The version + numbers differ by family (``fact`` and ``source_cell`` are v1 where the + rest are v2), which is the concrete reason epoch detection cannot be a + lookup in this map. + """ + observed = { + key.split(":", 1)[0] for row in _captured_feed_rows() for key in _keys_in(row) + } + + assert observed + assert observed <= set(LEDGER_FACT_KEY_DOMAINS.values()), sorted( + observed - set(LEDGER_FACT_KEY_DOMAINS.values()) + ) + + def test_chronicle_era_keys_resolve_without_a_declared_version_number() -> None: """Epoch detection is structural, so undeclared families still resolve. @@ -414,8 +461,7 @@ def test_witnessed_key_fields_match_the_captured_feed() -> None: Every ``*_key`` field the captured UK feed rows carry is either witnessed for its epoch or is a plain source identifier rather than a Chronicle key. """ - fixture = Path(__file__).parent / "fixtures" / "uk_target_reference_feed_rows.jsonl" - rows = [json.loads(line) for line in fixture.read_text().splitlines() if line] + rows = _captured_feed_rows() assert rows for row in rows: From c2d58a97a8a0f28a830c9aff456f859592f6fcb3 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Wed, 2 Sep 2026 11:24:23 -0400 Subject: [PATCH 06/23] Say in the changelog that only the manifest id is gated Co-Authored-By: Claude Fable 5.1 --- changelog.d/chronicle-dual-accept.added.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/changelog.d/chronicle-dual-accept.added.md b/changelog.d/chronicle-dual-accept.added.md index 472213d7e..09e181db6 100644 --- a/changelog.d/chronicle-dual-accept.added.md +++ b/changelog.d/chronicle-dual-accept.added.md @@ -1 +1 @@ -Accept chronicle-era Chronicle identities everywhere Microcosm consumes them: consumer-artifact manifests declaring `policyengine_chronicle.consumer_artifact.v2` and fact rows declaring `chronicle.consumer_fact.v2` load beside their ledger-era spellings, mixed-epoch feeds compile, and the loader records the observed schema id, its epoch, and the feed's fact-key epochs in provenance. Environment variables gain a `CHRONICLE_*` dual-read window with the `POPULACE_LEDGER_*` names honored under a once-per-process deprecation warning. Nothing renames on disk: emitted ids, diagnostic field names, `populace_*` key namespaces, goldens and fixtures stay frozen at v1. +Accept chronicle-era Chronicle identities everywhere Microcosm consumes them: consumer-artifact manifests declaring `policyengine_chronicle.consumer_artifact.v2` load beside `policyengine_ledger.consumer_artifact.v1`, mixed-epoch feeds compile, and the loader records the observed manifest schema id, its epoch, the fact-key epochs present in the feed, and the per-row schema ids the rows declare in provenance. Only the manifest id is gated: per-row schema ids and fact keys are carried as published, since real feeds mint rows in namespaces belonging to neither era. Environment variables gain a `CHRONICLE_*` dual-read window with the `POPULACE_LEDGER_*` names honored under a once-per-process deprecation warning. Nothing renames on disk: emitted ids, diagnostic field names, `populace_*` key namespaces, goldens and fixtures stay frozen at v1. From 4115573c853cddccf2d7d37088a897159cbd8e3f Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Wed, 2 Sep 2026 11:25:00 -0400 Subject: [PATCH 07/23] Record the regression and the arch-namespace question in the journal Co-Authored-By: Claude Fable 5.1 --- PROGRESS-chronicle-dual-accept.md | 60 +++++++++++++++++++++++-------- 1 file changed, 46 insertions(+), 14 deletions(-) diff --git a/PROGRESS-chronicle-dual-accept.md b/PROGRESS-chronicle-dual-accept.md index 118003a21..ee477892f 100644 --- a/PROGRESS-chronicle-dual-accept.md +++ b/PROGRESS-chronicle-dual-accept.md @@ -21,9 +21,13 @@ Frozen (microcosm#639): nothing on disk or in artifacts renames. Diagnostic field names (`ledger_aggregate_fact_key`, `ledger_commit`), H5 attrs, `populace_*` ids, fact keys, goldens and fixtures stay at v1. +**Acceptance widens; nothing narrows.** This is the rule the lane's own first +pass broke and the second pass restored — see "The regression that mattered". + ## State -Implementation and tests landed; verification run recorded in `out.md`. +Implementation and tests landed; verification run recorded in the PR body and +in the lane's report. PR #849, open, do not merge. ## Done @@ -34,10 +38,10 @@ Implementation and tests landed; verification run recorded in `out.md`. strings the spec names explicitly are pinned as literals. - `microcosm/build/chronicle_env.py` — the env dual-read window, one helper, one `DeprecationWarning` per process per legacy name. -- `ledger_artifact.py` — manifest `schema_version` is a membership test over - both eras; a per-row `schema_version` is validated when present (Chronicle - rows have never carried one) and never demanded; `provenance()` records the - observed id, `schema_epoch`, and `fact_key_epochs`. +- `ledger_artifact.py` — the manifest `schema_version` is a membership test + over both eras; per-row schema ids and fact keys are carried as published; + `provenance()` records the observed manifest id, `schema_epoch`, + `fact_key_epochs`, and `fact_schema_versions`. - `us_trade/import_entry_facts.py` — emission stays ledger-era (bytes are pinned; these rows are minted from Census/CBP bytes, so there is no source epoch to inherit), but the declared id is now an argument checked against @@ -50,17 +54,45 @@ Implementation and tests landed; verification run recorded in `out.md`. `test_us_congressional_district_vintage.py`, and dual-era emission and acceptance in `test_us_trade_facts.py`. +## The regression that mattered + +The lane's first pass added a per-row `schema_version` check to +`_load_fact_rows` that rejected any id outside the two chronicle#143 names — +validation `main` never performed, on a field `main` never read. Real feeds do +not honor that set. The pinned US fiscal-refresh feed +`consumer_facts_buildn_v9_4.jsonl`, which the release path loads through this +loader, declares `arch.consumer_fact.v1` on 37,006 of its first 200,000 rows +and `ledger.consumer_fact.v1` on 399. The check failed the build closed on its +own pinned input, and **PR CI could not see it**: the feed is gated data +outside PR CI. Reproduced directly, fixed, and pinned by a test. + +The same pass witnessed fact-key epochs from only the four fields targets +resolve a fact by. Published rows carry eleven key-bearing paths; the pinned +feed exercises all of them. A row straddling the cutover — ledger-era +aggregate key, chronicle-era source-release key — was reported as pure +ledger-era. The inventory is now complete and pinned to the captured feed +fixture. + ## Audit result -An 8-surface / 42-agent adversarial audit of every site that compares, -parses, or mints a Chronicle fact key or schema id found exactly three -hard-coded epoch literals in non-test source — `ledger_artifact.py:35`/`:126` -and `import_entry_facts.py:141`/`:142` — all handled here. `ledger_targets.py` -carries keys opaquely and contains no epoch literal at all; -`congressional_district_vintage.py` mints into Microcosm-owned namespaces that -sit outside both eras; the UK runtime is clean; no golden embeds a Chronicle -domain and no H5 attribute name embeds `ledger`. Every other candidate site -was refuted on verification. +Every site that compares, parses, or mints a Chronicle fact key or schema id +was inventoried. After this branch, no non-test source file hard-codes a +single epoch in a validator. `ledger_targets.py` carries keys opaquely and +contains no epoch literal at all; `congressional_district_vintage.py` and +`us_trade/import_entry_facts.py` mint into Microcosm-owned namespaces from +digest payloads that contain no Chronicle key, so a source row's epoch cannot +move a minted key; the UK runtime is clean; no golden embeds a Chronicle +domain and no H5 attribute name embeds `ledger`. `require_pins` does not exist +in this repo. + +## Open question for the Chronicle side + +The pinned feeds carry an `arch.*` key namespace alongside `ledger.*` — 37,006 +rows against 399 in `consumer_facts_buildn_v9_4.jsonl`, across all eleven +families. Microcosm treats those keys opaquely, so nothing here depends on +what `arch` is, and the epoch module reports it as outside both declared eras. +Whether chronicle#143's cutover is meant to re-epoch `arch.*` rows too is a +question for the Chronicle lane; this branch does not guess. ## Next From 061544e1f20ff8d0119439f363d4de5f467bab7e Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Wed, 2 Sep 2026 11:47:26 -0400 Subject: [PATCH 08/23] Rename the CHRONICLE_* env dual-read window to LOGBOOK_* POPULACE_LEDGER_URL/_KEY/_API_KEY/_EXPORT_KEY are Logbook store credentials (microcosm#632), not Chronicle fact-store ones. Naming the preferred spellings CHRONICLE_* would recreate the exact collision microcosm#632 renamed Ledger away from. Per PR #849 review: rename chronicle_env.py to logbook_env.py, its symbols to LOGBOOK_*/logbook_env*, and every caller (logbook.py, tools/logbook.py, build/__init__.py exports, and the two module-path comments in firm_generation.py/source_coverage.py that named the old module). Legacy POPULACE_LEDGER_* names and the once-per-process DeprecationWarning behavior are unchanged; the warning text now cites microcosm#632 instead of chronicle#143. Co-Authored-By: Claude Fable 5.1 --- .../src/microcosm/build/__init__.py | 10 +- .../src/microcosm/build/logbook.py | 29 +++--- .../{chronicle_env.py => logbook_env.py} | 91 ++++++++--------- .../build/uk_runtime/firm_generation.py | 2 +- .../build/us_runtime/source_coverage.py | 2 +- ...t_chronicle_env.py => test_logbook_env.py} | 98 +++++++++---------- tools/logbook.py | 35 +++---- 7 files changed, 136 insertions(+), 131 deletions(-) rename packages/microcosm-build/src/microcosm/build/{chronicle_env.py => logbook_env.py} (58%) rename packages/microcosm-build/tests/{test_chronicle_env.py => test_logbook_env.py} (58%) diff --git a/packages/microcosm-build/src/microcosm/build/__init__.py b/packages/microcosm-build/src/microcosm/build/__init__.py index ea8dbdf72..431f2793f 100644 --- a/packages/microcosm-build/src/microcosm/build/__init__.py +++ b/packages/microcosm-build/src/microcosm/build/__init__.py @@ -49,9 +49,9 @@ def _assert_frame_compatible(version: str, required: tuple[int, int]) -> None: _assert_frame_compatible(_frame_version, _REQUIRED_FRAME_SERIES) -from microcosm.build.chronicle_env import ( # noqa: E402 - after the compat gate - chronicle_env, - chronicle_env_names, +from microcosm.build.logbook_env import ( # noqa: E402 - after the compat gate + logbook_env, + logbook_env_names, ) from microcosm.build.chronicle_epoch import ( # noqa: E402 - after the compat gate ACCEPTED_CONSUMER_ARTIFACT_SCHEMA_VERSIONS, @@ -224,8 +224,8 @@ def _assert_frame_compatible(version: str, required: tuple[int, int]) -> None: "MonetaryTargetProfile", "PreparedMonetaryMeasure", "add_ledger_artifact_args", - "chronicle_env", - "chronicle_env_names", + "logbook_env", + "logbook_env_names", "fact_key_epoch", "feed_fact_key_epochs", "is_chronicle_fact_key", diff --git a/packages/microcosm-build/src/microcosm/build/logbook.py b/packages/microcosm-build/src/microcosm/build/logbook.py index de7d404b4..6cfb75d12 100644 --- a/packages/microcosm-build/src/microcosm/build/logbook.py +++ b/packages/microcosm-build/src/microcosm/build/logbook.py @@ -7,15 +7,16 @@ a best-effort Supabase REST insert. Remote availability is never part of build correctness. If either -``CHRONICLE_URL`` or ``CHRONICLE_KEY`` is absent, the validated row stays in +``LOGBOOK_URL`` or ``LOGBOOK_KEY`` is absent, the validated row stays in spool-only mode without error. Network and HTTP failures are returned as receipt data and never raised. Local validation and durable-spool failures remain fatal: callers must not claim an attempt was recorded if its local row was invalid or could not be persisted. The ledger-era spellings (``POPULACE_LEDGER_URL``, ``POPULACE_LEDGER_KEY``, -``POPULACE_LEDGER_API_KEY``) are still honored for the chronicle#143 dual-read -window and warn once per process; see :mod:`microcosm.build.chronicle_env`. +``POPULACE_LEDGER_API_KEY``) are still honored for the Logbook dual-read +window (microcosm#632) and warn once per process; see +:mod:`microcosm.build.logbook_env`. The caller owns chain coordination and must provide ``prev_row_digest`` for the ledger head it extends. The row digest is @@ -32,7 +33,7 @@ The Supabase key must identify the migration's ``logbook_writer`` role, not the service role. Hosted Supabase projects should additionally provide the -project gateway key as ``CHRONICLE_API_KEY``; single-key deployments may +project gateway key as ``LOGBOOK_API_KEY``; single-key deployments may omit it. The ``logbook`` schema must also be enabled in the hosted project's PostgREST exposed-schema setting. The US stacked driver and the three UK drivers record through this seam; this module remains driver-agnostic. @@ -60,12 +61,12 @@ from urllib.parse import urlencode, urlsplit from urllib.request import HTTPRedirectHandler, Request, build_opener -from microcosm.build.chronicle_env import ( - CHRONICLE_API_KEY_ENV, - CHRONICLE_KEY_ENV, - CHRONICLE_URL_ENV, +from microcosm.build.logbook_env import ( LEGACY_API_KEY_ENV, - chronicle_env, + LOGBOOK_API_KEY_ENV, + LOGBOOK_KEY_ENV, + LOGBOOK_URL_ENV, + logbook_env, ) __all__ = [ @@ -105,8 +106,8 @@ LOGBOOK_RUNGS = frozenset({"f001", "f004", "f010", "f025", "f100"}) #: Ledger-era name for the project gateway key, kept because callers and #: operator runbooks still name it. Reads go through -#: :func:`microcosm.build.chronicle_env.chronicle_env`, which prefers -#: ``CHRONICLE_API_KEY`` and warns once when it falls back to this one. +#: :func:`microcosm.build.logbook_env.logbook_env`, which prefers +#: ``LOGBOOK_API_KEY`` and warns once when it falls back to this one. LEDGER_API_KEY_ENV = LEGACY_API_KEY_ENV LOGBOOK_ROW_FIELDS = frozenset( { @@ -836,11 +837,11 @@ def urlopen(request: Request, *, timeout: float) -> Any: def _remote_config() -> tuple[str, str, str] | None: - url = chronicle_env(CHRONICLE_URL_ENV) - key = chronicle_env(CHRONICLE_KEY_ENV) + url = logbook_env(LOGBOOK_URL_ENV) + key = logbook_env(LOGBOOK_KEY_ENV) if not url or not key: return None - api_key = chronicle_env(CHRONICLE_API_KEY_ENV) or key + api_key = logbook_env(LOGBOOK_API_KEY_ENV) or key return url, key, api_key diff --git a/packages/microcosm-build/src/microcosm/build/chronicle_env.py b/packages/microcosm-build/src/microcosm/build/logbook_env.py similarity index 58% rename from packages/microcosm-build/src/microcosm/build/chronicle_env.py rename to packages/microcosm-build/src/microcosm/build/logbook_env.py index 763032d4b..3536972b7 100644 --- a/packages/microcosm-build/src/microcosm/build/chronicle_env.py +++ b/packages/microcosm-build/src/microcosm/build/logbook_env.py @@ -1,13 +1,15 @@ -"""Dual-read for the Chronicle (formerly Ledger) environment variables. +"""Dual-read for the Logbook (formerly Ledger) environment variables. -PolicyEngine/chronicle#143 gives the operational stores — buckets, database -schema, role ids, and env names — a **dual-read window**: ``CHRONICLE_*`` is +Logbook is the build ledger's own name (microcosm#632), chosen precisely to +stop colliding with Chronicle, the fact store this package also consumes. +This module gives the Logbook store's credentials — buckets, database +schema, role ids, and env names — a **dual-read window**: ``LOGBOOK_*`` is preferred, the legacy name is still honored, and honoring it emits a deprecation warning so publish flows and build scripts migrate on their own schedule. This module is that window, in one place. -Every variable here is read through :func:`chronicle_env`, which tries the -preferred ``CHRONICLE_*`` name first and falls back to the legacy name, +Every variable here is read through :func:`logbook_env`, which tries the +preferred ``LOGBOOK_*`` name first and falls back to the legacy name, warning once per process per legacy name. Nothing is renamed on disk: the legacy names keep working for as long as the window is open, and the legacy name constants stay exported so error messages and tests can still name them. @@ -16,8 +18,10 @@ ``LEDGER_ONS_TURNOVER_BANDS``, ``LEDGER_ONS_EMPLOYMENT_BANDS``, and ``LEDGER_US_SOURCE_COVERAGE_CONTRACT_COMMIT`` look like environment variables to a grep but are plain Python module constants (band maps and a git commit -pin). They get chronicle-named *aliases* beside their modules rather than a -dual-read, because there is no environment to read them from. +pin). They translate Chronicle identities and get Chronicle-named *aliases* +beside their modules rather than a dual-read, because there is no +environment to read them from and they have nothing to do with the Logbook +store this module covers. """ from __future__ import annotations @@ -28,26 +32,26 @@ from collections.abc import Mapping __all__ = [ - "CHRONICLE_API_KEY_ENV", - "CHRONICLE_ENV_LEGACY_NAMES", - "CHRONICLE_EXPORT_KEY_ENV", - "CHRONICLE_KEY_ENV", - "CHRONICLE_URL_ENV", "LEGACY_API_KEY_ENV", "LEGACY_EXPORT_KEY_ENV", "LEGACY_KEY_ENV", "LEGACY_URL_ENV", - "chronicle_env", - "chronicle_env_names", - "describe_chronicle_env", - "reset_chronicle_env_deprecation_warnings", + "LOGBOOK_API_KEY_ENV", + "LOGBOOK_ENV_LEGACY_NAMES", + "LOGBOOK_EXPORT_KEY_ENV", + "LOGBOOK_KEY_ENV", + "LOGBOOK_URL_ENV", + "describe_logbook_env", + "logbook_env", + "logbook_env_names", + "reset_logbook_env_deprecation_warnings", ] -#: Preferred, chronicle-era names. -CHRONICLE_URL_ENV = "CHRONICLE_URL" -CHRONICLE_KEY_ENV = "CHRONICLE_KEY" -CHRONICLE_API_KEY_ENV = "CHRONICLE_API_KEY" -CHRONICLE_EXPORT_KEY_ENV = "CHRONICLE_EXPORT_KEY" +#: Preferred, Logbook-named names. +LOGBOOK_URL_ENV = "LOGBOOK_URL" +LOGBOOK_KEY_ENV = "LOGBOOK_KEY" +LOGBOOK_API_KEY_ENV = "LOGBOOK_API_KEY" +LOGBOOK_EXPORT_KEY_ENV = "LOGBOOK_EXPORT_KEY" #: Legacy, ledger-era names. Still honored; still named in error messages so #: an operator running the old environment recognises what is being asked for. @@ -58,26 +62,26 @@ #: Preferred name -> legacy names, most recent legacy spelling first. Adding a #: variable to the dual-read window means adding a row here and nothing else. -CHRONICLE_ENV_LEGACY_NAMES: Mapping[str, tuple[str, ...]] = { - CHRONICLE_URL_ENV: (LEGACY_URL_ENV,), - CHRONICLE_KEY_ENV: (LEGACY_KEY_ENV,), - CHRONICLE_API_KEY_ENV: (LEGACY_API_KEY_ENV,), - CHRONICLE_EXPORT_KEY_ENV: (LEGACY_EXPORT_KEY_ENV,), +LOGBOOK_ENV_LEGACY_NAMES: Mapping[str, tuple[str, ...]] = { + LOGBOOK_URL_ENV: (LEGACY_URL_ENV,), + LOGBOOK_KEY_ENV: (LEGACY_KEY_ENV,), + LOGBOOK_API_KEY_ENV: (LEGACY_API_KEY_ENV,), + LOGBOOK_EXPORT_KEY_ENV: (LEGACY_EXPORT_KEY_ENV,), } _WARNED_LEGACY_NAMES: set[str] = set() _WARNED_LOCK = threading.Lock() -def chronicle_env( +def logbook_env( name: str, default: str | None = None, *, environ: Mapping[str, str] | None = None, ) -> str | None: - """Read one Chronicle variable, preferring ``name`` over its legacy spelling. + """Read one Logbook variable, preferring ``name`` over its legacy spelling. - ``name`` is the preferred ``CHRONICLE_*`` name. When it is unset but a + ``name`` is the preferred ``LOGBOOK_*`` name. When it is unset but a legacy name carries a value, that value is returned and a :class:`DeprecationWarning` is emitted **once per process per legacy name** — repeated reads in a build loop must not turn into a warning @@ -85,15 +89,15 @@ def chronicle_env( already test these variables. """ source = os.environ if environ is None else environ - if name not in CHRONICLE_ENV_LEGACY_NAMES: + if name not in LOGBOOK_ENV_LEGACY_NAMES: raise KeyError( - f"{name!r} is not a Chronicle environment variable; expected one of " - f"{sorted(CHRONICLE_ENV_LEGACY_NAMES)}." + f"{name!r} is not a Logbook environment variable; expected one of " + f"{sorted(LOGBOOK_ENV_LEGACY_NAMES)}." ) value = source.get(name) if value: return value - for legacy_name in CHRONICLE_ENV_LEGACY_NAMES[name]: + for legacy_name in LOGBOOK_ENV_LEGACY_NAMES[name]: legacy_value = source.get(legacy_name) if legacy_value: _warn_once(legacy_name, preferred=name) @@ -101,17 +105,17 @@ def chronicle_env( return default -def chronicle_env_names(name: str) -> tuple[str, ...]: +def logbook_env_names(name: str) -> tuple[str, ...]: """The preferred name followed by every legacy name still honored.""" - if name not in CHRONICLE_ENV_LEGACY_NAMES: + if name not in LOGBOOK_ENV_LEGACY_NAMES: raise KeyError( - f"{name!r} is not a Chronicle environment variable; expected one of " - f"{sorted(CHRONICLE_ENV_LEGACY_NAMES)}." + f"{name!r} is not a Logbook environment variable; expected one of " + f"{sorted(LOGBOOK_ENV_LEGACY_NAMES)}." ) - return (name, *CHRONICLE_ENV_LEGACY_NAMES[name]) + return (name, *LOGBOOK_ENV_LEGACY_NAMES[name]) -def describe_chronicle_env(*names: str) -> str: +def describe_logbook_env(*names: str) -> str: """Render required variables for an error message, legacy names included. Error text names both spellings on purpose: an operator whose environment @@ -122,12 +126,12 @@ def describe_chronicle_env(*names: str) -> str: legacy = ", ".join( legacy_name for name in names - for legacy_name in CHRONICLE_ENV_LEGACY_NAMES[name] + for legacy_name in LOGBOOK_ENV_LEGACY_NAMES[name] ) return f"{preferred} (legacy {legacy} still honored)" -def reset_chronicle_env_deprecation_warnings() -> None: +def reset_logbook_env_deprecation_warnings() -> None: """Forget which legacy names have warned. For tests only.""" with _WARNED_LOCK: _WARNED_LEGACY_NAMES.clear() @@ -139,9 +143,8 @@ def _warn_once(legacy_name: str, *, preferred: str) -> None: return _WARNED_LEGACY_NAMES.add(legacy_name) warnings.warn( - f"{legacy_name} is the ledger-era name for {preferred} and is " - "deprecated; PolicyEngine Ledger is now Chronicle " - "(PolicyEngine/chronicle#143). Set " + f"{legacy_name} is the pre-rename name for {preferred}; the build " + "ledger is now Logbook (microcosm#632). Set " f"{preferred} instead — {legacy_name} stays honored only for the " "dual-read window.", DeprecationWarning, diff --git a/packages/microcosm-build/src/microcosm/build/uk_runtime/firm_generation.py b/packages/microcosm-build/src/microcosm/build/uk_runtime/firm_generation.py index 9867276a1..7df3399c8 100644 --- a/packages/microcosm-build/src/microcosm/build/uk_runtime/firm_generation.py +++ b/packages/microcosm-build/src/microcosm/build/uk_runtime/firm_generation.py @@ -162,7 +162,7 @@ class UKFirmLedgerTargetProfile: # Chronicle-era spellings of the three band maps above. These identifiers only # look like environment variables to a grep — they are plain support-layout # data, so they take aliases rather than the dual-read window in -# :mod:`microcosm.build.chronicle_env`. The ledger-era names stay: they are what +# :mod:`microcosm.build.logbook_env`. The ledger-era names stay: they are what # the generator below imports, and the value ids they translate are published # Chronicle ids, unaffected by the rename. CHRONICLE_ONS_TURNOVER_BANDS = LEDGER_ONS_TURNOVER_BANDS diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/source_coverage.py b/packages/microcosm-build/src/microcosm/build/us_runtime/source_coverage.py index ab0bfea06..d37e3878e 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/source_coverage.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/source_coverage.py @@ -35,7 +35,7 @@ #: Chronicle-era spelling of the same pin. This identifier only *looks* like an #: environment variable to a grep — it is a git commit of the source-coverage #: contract, so it takes an alias rather than the dual-read window in -#: :mod:`microcosm.build.chronicle_env`. The ledger-era name stays exported: it +#: :mod:`microcosm.build.logbook_env`. The ledger-era name stays exported: it #: is what callers import, and the ``ledger_commit`` diagnostic field it feeds #: is frozen at v1 (microcosm#639). CHRONICLE_US_SOURCE_COVERAGE_CONTRACT_COMMIT = LEDGER_US_SOURCE_COVERAGE_CONTRACT_COMMIT diff --git a/packages/microcosm-build/tests/test_chronicle_env.py b/packages/microcosm-build/tests/test_logbook_env.py similarity index 58% rename from packages/microcosm-build/tests/test_chronicle_env.py rename to packages/microcosm-build/tests/test_logbook_env.py index 76697a4b2..b06205773 100644 --- a/packages/microcosm-build/tests/test_chronicle_env.py +++ b/packages/microcosm-build/tests/test_logbook_env.py @@ -1,10 +1,10 @@ -"""The CHRONICLE_* / POPULACE_LEDGER_* environment dual-read window. +"""The LOGBOOK_* / POPULACE_LEDGER_* environment dual-read window. -chronicle#143 gives the operational stores a dual-read window so publish -flows and build scripts migrate on their own schedule: ``CHRONICLE_*`` is -preferred, the ledger-era name is still honored, and honoring it warns once -per process. These tests hold both halves — the fallback keeps working, and -the warning stays a single line rather than one per read in a build loop. +Logbook (microcosm#632) gives its operational store a dual-read window so +publish flows and build scripts migrate on their own schedule: ``LOGBOOK_*`` +is preferred, the ledger-era name is still honored, and honoring it warns +once per process. These tests hold both halves — the fallback keeps working, +and the warning stays a single line rather than one per read in a build loop. """ from __future__ import annotations @@ -13,28 +13,28 @@ import pytest -from microcosm.build.chronicle_env import ( - CHRONICLE_API_KEY_ENV, - CHRONICLE_ENV_LEGACY_NAMES, - CHRONICLE_EXPORT_KEY_ENV, - CHRONICLE_KEY_ENV, - CHRONICLE_URL_ENV, +from microcosm.build.logbook import _remote_config +from microcosm.build.logbook_env import ( LEGACY_API_KEY_ENV, LEGACY_EXPORT_KEY_ENV, LEGACY_KEY_ENV, LEGACY_URL_ENV, - chronicle_env, - chronicle_env_names, - describe_chronicle_env, - reset_chronicle_env_deprecation_warnings, + LOGBOOK_API_KEY_ENV, + LOGBOOK_ENV_LEGACY_NAMES, + LOGBOOK_EXPORT_KEY_ENV, + LOGBOOK_KEY_ENV, + LOGBOOK_URL_ENV, + describe_logbook_env, + logbook_env, + logbook_env_names, + reset_logbook_env_deprecation_warnings, ) -from microcosm.build.logbook import _remote_config PAIRS = ( - (CHRONICLE_URL_ENV, LEGACY_URL_ENV), - (CHRONICLE_KEY_ENV, LEGACY_KEY_ENV), - (CHRONICLE_API_KEY_ENV, LEGACY_API_KEY_ENV), - (CHRONICLE_EXPORT_KEY_ENV, LEGACY_EXPORT_KEY_ENV), + (LOGBOOK_URL_ENV, LEGACY_URL_ENV), + (LOGBOOK_KEY_ENV, LEGACY_KEY_ENV), + (LOGBOOK_API_KEY_ENV, LEGACY_API_KEY_ENV), + (LOGBOOK_EXPORT_KEY_ENV, LEGACY_EXPORT_KEY_ENV), ) @@ -43,21 +43,21 @@ def _clean_env(monkeypatch): for preferred, legacy in PAIRS: monkeypatch.delenv(preferred, raising=False) monkeypatch.delenv(legacy, raising=False) - reset_chronicle_env_deprecation_warnings() + reset_logbook_env_deprecation_warnings() yield - reset_chronicle_env_deprecation_warnings() + reset_logbook_env_deprecation_warnings() @pytest.mark.parametrize(("preferred", "legacy"), PAIRS) def test_preferred_name_wins_and_warns_about_nothing( monkeypatch, preferred: str, legacy: str ) -> None: - monkeypatch.setenv(preferred, "chronicle-value") + monkeypatch.setenv(preferred, "logbook-value") monkeypatch.setenv(legacy, "ledger-value") with warnings.catch_warnings(): warnings.simplefilter("error", DeprecationWarning) - assert chronicle_env(preferred) == "chronicle-value" + assert logbook_env(preferred) == "logbook-value" @pytest.mark.parametrize(("preferred", "legacy"), PAIRS) @@ -67,12 +67,12 @@ def test_legacy_name_is_honored_with_a_deprecation_warning( monkeypatch.setenv(legacy, "ledger-value") with pytest.warns(DeprecationWarning) as record: - assert chronicle_env(preferred) == "ledger-value" + assert logbook_env(preferred) == "ledger-value" message = str(record[0].message) assert legacy in message assert preferred in message - assert "chronicle#143" in message + assert "microcosm#632" in message def test_the_deprecation_warning_fires_once_per_process(monkeypatch) -> None: @@ -82,7 +82,7 @@ def test_the_deprecation_warning_fires_once_per_process(monkeypatch) -> None: with warnings.catch_warnings(record=True) as caught: warnings.simplefilter("always") for _ in range(5): - assert chronicle_env(CHRONICLE_URL_ENV) == "https://ledger.example" + assert logbook_env(LOGBOOK_URL_ENV) == "https://ledger.example" assert [ w for w in caught if issubclass(w.category, DeprecationWarning) @@ -95,8 +95,8 @@ def test_each_legacy_name_warns_on_its_own(monkeypatch) -> None: with warnings.catch_warnings(record=True) as caught: warnings.simplefilter("always") - chronicle_env(CHRONICLE_URL_ENV) - chronicle_env(CHRONICLE_KEY_ENV) + logbook_env(LOGBOOK_URL_ENV) + logbook_env(LOGBOOK_KEY_ENV) warned = { legacy @@ -107,39 +107,39 @@ def test_each_legacy_name_warns_on_its_own(monkeypatch) -> None: def test_unset_returns_the_default_and_empty_counts_as_unset(monkeypatch) -> None: - assert chronicle_env(CHRONICLE_URL_ENV) is None - assert chronicle_env(CHRONICLE_URL_ENV, "fallback") == "fallback" + assert logbook_env(LOGBOOK_URL_ENV) is None + assert logbook_env(LOGBOOK_URL_ENV, "fallback") == "fallback" - monkeypatch.setenv(CHRONICLE_URL_ENV, "") + monkeypatch.setenv(LOGBOOK_URL_ENV, "") monkeypatch.setenv(LEGACY_URL_ENV, "https://ledger.example") with pytest.warns(DeprecationWarning): - assert chronicle_env(CHRONICLE_URL_ENV) == "https://ledger.example" + assert logbook_env(LOGBOOK_URL_ENV) == "https://ledger.example" def test_an_explicit_environ_mapping_bypasses_the_process_environment() -> None: with pytest.warns(DeprecationWarning): assert ( - chronicle_env(CHRONICLE_KEY_ENV, environ={LEGACY_KEY_ENV: "writer-jwt"}) + logbook_env(LOGBOOK_KEY_ENV, environ={LEGACY_KEY_ENV: "writer-jwt"}) == "writer-jwt" ) def test_a_name_outside_the_window_is_a_programming_error() -> None: - with pytest.raises(KeyError, match="not a Chronicle environment variable"): - chronicle_env("CHRONICLE_NOT_A_REAL_VARIABLE") - with pytest.raises(KeyError, match="not a Chronicle environment variable"): - chronicle_env_names(LEGACY_URL_ENV) + with pytest.raises(KeyError, match="not a Logbook environment variable"): + logbook_env("LOGBOOK_NOT_A_REAL_VARIABLE") + with pytest.raises(KeyError, match="not a Logbook environment variable"): + logbook_env_names(LEGACY_URL_ENV) def test_names_and_descriptions_carry_both_spellings() -> None: - assert chronicle_env_names(CHRONICLE_URL_ENV) == (CHRONICLE_URL_ENV, LEGACY_URL_ENV) + assert logbook_env_names(LOGBOOK_URL_ENV) == (LOGBOOK_URL_ENV, LEGACY_URL_ENV) - described = describe_chronicle_env(CHRONICLE_URL_ENV, CHRONICLE_EXPORT_KEY_ENV) + described = describe_logbook_env(LOGBOOK_URL_ENV, LOGBOOK_EXPORT_KEY_ENV) # An operator whose environment predates the rename must still be able to # match the error text against what they have set. for name in ( - CHRONICLE_URL_ENV, - CHRONICLE_EXPORT_KEY_ENV, + LOGBOOK_URL_ENV, + LOGBOOK_EXPORT_KEY_ENV, LEGACY_URL_ENV, LEGACY_EXPORT_KEY_ENV, ): @@ -147,7 +147,7 @@ def test_names_and_descriptions_carry_both_spellings() -> None: def test_every_windowed_variable_maps_to_exactly_one_legacy_name() -> None: - assert dict(CHRONICLE_ENV_LEGACY_NAMES) == { + assert dict(LOGBOOK_ENV_LEGACY_NAMES) == { preferred: (legacy,) for preferred, legacy in PAIRS } @@ -164,13 +164,13 @@ def test_logbook_remote_config_reads_both_eras(monkeypatch) -> None: "writer-jwt", ) - monkeypatch.setenv(CHRONICLE_URL_ENV, "https://chronicle.example") - monkeypatch.setenv(CHRONICLE_KEY_ENV, "chronicle-jwt") - monkeypatch.setenv(CHRONICLE_API_KEY_ENV, "project-api-key") + monkeypatch.setenv(LOGBOOK_URL_ENV, "https://logbook.example") + monkeypatch.setenv(LOGBOOK_KEY_ENV, "logbook-jwt") + monkeypatch.setenv(LOGBOOK_API_KEY_ENV, "project-api-key") with warnings.catch_warnings(): warnings.simplefilter("error", DeprecationWarning) assert _remote_config() == ( - "https://chronicle.example", - "chronicle-jwt", + "https://logbook.example", + "logbook-jwt", "project-api-key", ) diff --git a/tools/logbook.py b/tools/logbook.py index afea050ef..ed7dd72a5 100644 --- a/tools/logbook.py +++ b/tools/logbook.py @@ -7,11 +7,11 @@ archive or a directory of them, reporting chain by chain. Remote export uses a distinct, read-only ``logbook_exporter`` JWT supplied -as ``CHRONICLE_EXPORT_KEY`` plus the hosted project's gateway key in -``CHRONICLE_API_KEY``. It never reuses the insert-only writer key. The +as ``LOGBOOK_EXPORT_KEY`` plus the hosted project's gateway key in +``LOGBOOK_API_KEY``. It never reuses the insert-only writer key. The ledger-era spellings (``POPULACE_LEDGER_EXPORT_KEY``, ``POPULACE_LEDGER_API_KEY``, ``POPULACE_LEDGER_URL``) stay honored for the -chronicle#143 dual-read window and warn once per process. +Logbook dual-read window (microcosm#632) and warn once per process. The live store is row-oriented and carries every attempt across all scopes; the per-scope split is an archive convention, not a database partition. """ @@ -27,14 +27,14 @@ from urllib.parse import urlencode from urllib.request import Request -from microcosm.build.chronicle_env import ( - CHRONICLE_API_KEY_ENV, - CHRONICLE_EXPORT_KEY_ENV, - CHRONICLE_URL_ENV, +from microcosm.build.logbook_env import ( LEGACY_API_KEY_ENV, LEGACY_EXPORT_KEY_ENV, - chronicle_env, - describe_chronicle_env, + LOGBOOK_API_KEY_ENV, + LOGBOOK_EXPORT_KEY_ENV, + LOGBOOK_URL_ENV, + describe_logbook_env, + logbook_env, ) from microcosm.build.logbook import ( LOGBOOK_ROW_FIELDS, @@ -57,9 +57,10 @@ #: chain is born, and country is the outermost one. DEFAULT_ARCHIVE_ROOT = ROOT / "logbook" DEFAULT_SPOOL_ROOT = ROOT / "logbook-spool" -#: Ledger-era names, still honored through the chronicle#143 dual-read window. -#: Reads go through :func:`microcosm.build.chronicle_env.chronicle_env`, which -#: prefers ``CHRONICLE_EXPORT_KEY`` / ``CHRONICLE_API_KEY``. +#: Ledger-era names, still honored through the Logbook dual-read window +#: (microcosm#632). Reads go through +#: :func:`microcosm.build.logbook_env.logbook_env`, which prefers +#: ``LOGBOOK_EXPORT_KEY`` / ``LOGBOOK_API_KEY``. REMOTE_EXPORT_KEY_ENV = LEGACY_EXPORT_KEY_ENV REMOTE_API_KEY_ENV = LEGACY_API_KEY_ENV REMOTE_PAGE_SIZE = 500 @@ -257,14 +258,14 @@ def _archive_scope(archive: Path) -> str: def _remote_rows(scope: str) -> tuple[LogbookRow, ...]: - ledger_url = chronicle_env(CHRONICLE_URL_ENV) - export_key = chronicle_env(CHRONICLE_EXPORT_KEY_ENV) - api_key = chronicle_env(CHRONICLE_API_KEY_ENV) + ledger_url = logbook_env(LOGBOOK_URL_ENV) + export_key = logbook_env(LOGBOOK_EXPORT_KEY_ENV) + api_key = logbook_env(LOGBOOK_API_KEY_ENV) if not ledger_url or not export_key or not api_key: raise ValueError( "remote export requires " - + describe_chronicle_env( - CHRONICLE_URL_ENV, CHRONICLE_EXPORT_KEY_ENV, CHRONICLE_API_KEY_ENV + + describe_logbook_env( + LOGBOOK_URL_ENV, LOGBOOK_EXPORT_KEY_ENV, LOGBOOK_API_KEY_ENV ) ) From ee73a62495944416d3ee823655c80c899e5df842 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Wed, 2 Sep 2026 11:48:06 -0400 Subject: [PATCH 09/23] Reframe the env dual-read window as a Logbook cleanup, not chronicle#143 The changelog fragment, logbook/README.md, and the branch journal all described CHRONICLE_URL/CHRONICLE_KEY and the env dual-read window as part of the chronicle#143 epoch migration. It is a separate Logbook naming cleanup (microcosm#632) riding along on this branch: split the changelog sentence into its own paragraph, update the README's live-store section to LOGBOOK_URL/LOGBOOK_KEY, and append a dated correction note to the journal recording why and what changed. Co-Authored-By: Claude Fable 5.1 --- PROGRESS-chronicle-dual-accept.md | 29 ++++++++++++++++++++++ changelog.d/chronicle-dual-accept.added.md | 4 ++- logbook/README.md | 4 +-- 3 files changed, 34 insertions(+), 3 deletions(-) diff --git a/PROGRESS-chronicle-dual-accept.md b/PROGRESS-chronicle-dual-accept.md index ee477892f..409a159a4 100644 --- a/PROGRESS-chronicle-dual-accept.md +++ b/PROGRESS-chronicle-dual-accept.md @@ -98,3 +98,32 @@ question for the Chronicle lane; this branch does not guess. Nothing outstanding on this branch. Chronicle's own acceptance half lands in the parallel lane; the emit flip is a separate, later cutover. + +## Correction (2026-09-02): the env half was misnamed + +PR #849 review (Fable, main) caught that `POPULACE_LEDGER_URL` / +`_KEY` / `_API_KEY` / `_EXPORT_KEY` are **Logbook** store credentials +(Supabase `logbook` schema, `logbook_writer` / `logbook_exporter` roles — +see `logbook.py`'s docstring and `logbook/README.md`), not Chronicle fact- +store ones. "Ledger" there was the generic build-ledger sense renamed to +Logbook on 2026-08-08 (microcosm#632) specifically to stop colliding with +Chronicle. Naming the preferred spellings `CHRONICLE_*` would have recreated +that exact collision. + +The epoch half above (`chronicle_epoch.py`, `ledger_artifact.py`, +`import_entry_facts.py`, `ledger_targets.py`) is unaffected and was approved +as-is. The env half only: `chronicle_env.py` renamed to `logbook_env.py`, +`CHRONICLE_*_ENV`/`chronicle_env`/`chronicle_env_names`/ +`describe_chronicle_env`/`reset_chronicle_env_deprecation_warnings` renamed +to `LOGBOOK_*_ENV`/`logbook_env`/`logbook_env_names`/`describe_logbook_env`/ +`reset_logbook_env_deprecation_warnings`, and every caller (`logbook.py`, +`tools/logbook.py`, `build/__init__.py` exports, `logbook/README.md`, the +changelog fragment, this file, and the module-path comments in +`firm_generation.py`/`source_coverage.py`) updated to match. The +`POPULACE_LEDGER_*` legacy names and the once-per-process +`DeprecationWarning` behavior are unchanged; the warning text now cites +microcosm#632 instead of chronicle#143, since this window is a Logbook +naming cleanup riding along on this branch, not part of the chronicle#143 +epoch migration. `CHRONICLE_*_BANDS` and +`CHRONICLE_US_SOURCE_COVERAGE_CONTRACT_COMMIT` are untouched — those really +do translate Chronicle ids and pin a Chronicle commit. diff --git a/changelog.d/chronicle-dual-accept.added.md b/changelog.d/chronicle-dual-accept.added.md index 09e181db6..2f388d91b 100644 --- a/changelog.d/chronicle-dual-accept.added.md +++ b/changelog.d/chronicle-dual-accept.added.md @@ -1 +1,3 @@ -Accept chronicle-era Chronicle identities everywhere Microcosm consumes them: consumer-artifact manifests declaring `policyengine_chronicle.consumer_artifact.v2` load beside `policyengine_ledger.consumer_artifact.v1`, mixed-epoch feeds compile, and the loader records the observed manifest schema id, its epoch, the fact-key epochs present in the feed, and the per-row schema ids the rows declare in provenance. Only the manifest id is gated: per-row schema ids and fact keys are carried as published, since real feeds mint rows in namespaces belonging to neither era. Environment variables gain a `CHRONICLE_*` dual-read window with the `POPULACE_LEDGER_*` names honored under a once-per-process deprecation warning. Nothing renames on disk: emitted ids, diagnostic field names, `populace_*` key namespaces, goldens and fixtures stay frozen at v1. +Accept chronicle-era Chronicle identities everywhere Microcosm consumes them: consumer-artifact manifests declaring `policyengine_chronicle.consumer_artifact.v2` load beside `policyengine_ledger.consumer_artifact.v1`, mixed-epoch feeds compile, and the loader records the observed manifest schema id, its epoch, the fact-key epochs present in the feed, and the per-row schema ids the rows declare in provenance. Only the manifest id is gated: per-row schema ids and fact keys are carried as published, since real feeds mint rows in namespaces belonging to neither era. Nothing renames on disk: emitted ids, diagnostic field names, `populace_*` key namespaces, goldens and fixtures stay frozen at v1. + +Separately, a Logbook naming cleanup (microcosm#632): the build ledger's environment variables gain a `LOGBOOK_*` dual-read window with the `POPULACE_LEDGER_*` names honored under a once-per-process deprecation warning. These are the Logbook store's own credentials, unrelated to the Chronicle fact store above. diff --git a/logbook/README.md b/logbook/README.md index 47f43e6fa..81dc42fdd 100644 --- a/logbook/README.md +++ b/logbook/README.md @@ -90,10 +90,10 @@ for `published` and `certified` rows. ## The live store -The best-effort Supabase insert (`CHRONICLE_URL` + `CHRONICLE_KEY`, the +The best-effort Supabase insert (`LOGBOOK_URL` + `LOGBOOK_KEY`, the migration's insert-only `logbook_writer` role; the ledger-era `POPULACE_LEDGER_URL` / `POPULACE_LEDGER_KEY` stay honored for the -chronicle#143 dual-read window and warn once per process) +Logbook dual-read window (microcosm#632) and warn once per process) uses the same scope rule as the archives after this PR's migration is applied by the project owner. The writer key is unaffected: rows are still inserted through the same role and the scope is derived from the hashed `pipeline` From a5121a1e0c4da8d7767fe8322bc948a957c7c5e7 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Wed, 2 Sep 2026 12:00:46 -0400 Subject: [PATCH 10/23] Apply ruff import-sort and formatting to the renamed files logbook_env's import in build/__init__.py sorted alphabetically after the rename, and tools/logbook.py's two dual-read imports needed reordering and reformatting to match. Co-Authored-By: Claude Fable 5.1 --- .../src/microcosm/build/__init__.py | 8 +++---- .../src/microcosm/build/logbook_env.py | 4 +--- tools/logbook.py | 24 ++++++++----------- 3 files changed, 15 insertions(+), 21 deletions(-) diff --git a/packages/microcosm-build/src/microcosm/build/__init__.py b/packages/microcosm-build/src/microcosm/build/__init__.py index 431f2793f..17ca54703 100644 --- a/packages/microcosm-build/src/microcosm/build/__init__.py +++ b/packages/microcosm-build/src/microcosm/build/__init__.py @@ -49,10 +49,6 @@ def _assert_frame_compatible(version: str, required: tuple[int, int]) -> None: _assert_frame_compatible(_frame_version, _REQUIRED_FRAME_SERIES) -from microcosm.build.logbook_env import ( # noqa: E402 - after the compat gate - logbook_env, - logbook_env_names, -) from microcosm.build.chronicle_epoch import ( # noqa: E402 - after the compat gate ACCEPTED_CONSUMER_ARTIFACT_SCHEMA_VERSIONS, ACCEPTED_CONSUMER_FACT_SCHEMA_VERSIONS, @@ -141,6 +137,10 @@ def _assert_frame_compatible(version: str, required: tuple[int, int]) -> None: select_ledger_targets_from_jsonl, target_spec_from_ledger_fact, ) +from microcosm.build.logbook_env import ( # noqa: E402 - after the compat gate + logbook_env, + logbook_env_names, +) from microcosm.build.monetary_profile import ( # noqa: E402 - after compat gate MonetaryTargetContract, MonetaryTargetProfile, diff --git a/packages/microcosm-build/src/microcosm/build/logbook_env.py b/packages/microcosm-build/src/microcosm/build/logbook_env.py index 3536972b7..6a7ff6f9b 100644 --- a/packages/microcosm-build/src/microcosm/build/logbook_env.py +++ b/packages/microcosm-build/src/microcosm/build/logbook_env.py @@ -124,9 +124,7 @@ def describe_logbook_env(*names: str) -> str: """ preferred = ", ".join(names) legacy = ", ".join( - legacy_name - for name in names - for legacy_name in LOGBOOK_ENV_LEGACY_NAMES[name] + legacy_name for name in names for legacy_name in LOGBOOK_ENV_LEGACY_NAMES[name] ) return f"{preferred} (legacy {legacy} still honored)" diff --git a/tools/logbook.py b/tools/logbook.py index ed7dd72a5..423ba0097 100644 --- a/tools/logbook.py +++ b/tools/logbook.py @@ -27,15 +27,6 @@ from urllib.parse import urlencode from urllib.request import Request -from microcosm.build.logbook_env import ( - LEGACY_API_KEY_ENV, - LEGACY_EXPORT_KEY_ENV, - LOGBOOK_API_KEY_ENV, - LOGBOOK_EXPORT_KEY_ENV, - LOGBOOK_URL_ENV, - describe_logbook_env, - logbook_env, -) from microcosm.build.logbook import ( LOGBOOK_ROW_FIELDS, LogbookRow, @@ -47,6 +38,15 @@ render_markdown, urlopen, ) +from microcosm.build.logbook_env import ( + LEGACY_API_KEY_ENV, + LEGACY_EXPORT_KEY_ENV, + LOGBOOK_API_KEY_ENV, + LOGBOOK_EXPORT_KEY_ENV, + LOGBOOK_URL_ENV, + describe_logbook_env, + logbook_env, +) ROOT = Path(__file__).resolve().parents[1] #: Archives live at ``logbook//.jsonl`` — one hash chain per @@ -305,11 +305,7 @@ def _remote_rows(scope: str) -> tuple[LogbookRow, ...]: ) break wrong_scope = sorted( - { - row.pipeline - for row in rows - if _chain_scope(row.pipeline) != scope - } + {row.pipeline for row in rows if _chain_scope(row.pipeline) != scope} ) if wrong_scope: raise ValueError( From b98edd886f03b4ab62662d98f02fe4a46445069e Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Wed, 2 Sep 2026 12:04:07 -0400 Subject: [PATCH 11/23] Report the CHRONICLE_* to LOGBOOK_* correction on PR #849 Co-Authored-By: Claude Fable 5.1 --- out.md | 324 ++++++++++++++++++++++++++++++++++++--------------------- 1 file changed, 208 insertions(+), 116 deletions(-) diff --git a/out.md b/out.md index a727ccd79..64335b427 100644 --- a/out.md +++ b/out.md @@ -1,119 +1,211 @@ -# Armenia country package final report (#814) - -Date: 2026-08-28 - -Branch: `armenia-country-package-814` - -## Outcome - -The Armenia package is complete as a schema-valid, spec-only, engine-free -greenfield contract. It is not represented as an executable build: the exact -`populace-us` artifact, Armenian facts, generated cell bindings, community -distribution, gate policy, and shared runtime kernels remain explicit -prerequisites. - -The closed `am/` inventory mirrors Belgium's six typed resources and five -generation-zero JSON projections. It declares: - -- a two-stage load of a to-be-authenticated public `populace-us` artifact and - target-derived marz assignment; every record remains a US donor support - record and is never described as Armenian microdata; -- a 2022-census-vintage community spine constrained to the assigned marz, with - 10 marzes plus Yerevan and 71 consolidated communities documented but no - invented code roster; -- eight Ledger-only, count/indicator calibration authoring contracts spanning - demography, household structure, consumption, labour, earnings, pensions, - and family benefits; -- real-resolver refusal of unexpanded multi-cell tables: Chronicle must generate - cell-pinned Ledger references and direct/pre-built candidate bindings before - runtime activation; -- wage/payment, raw-income diagnostic, and national-accounts facts outside the - solver manifest until their validation role or AMD-compatible pre-built bridge - is enforceable; -- greenfield aggregate-admin, per-family-fit, target-coverage, macro-realism, - support, weight-audit, ESS, and ratio gate declarations, plus active - release-blocking reference-coverage/support/output checks; and -- a public 2024 release contract for `populace_am_{year}.h5`, with ArmStat open - dissemination stated and exact ArmStat/donor licence text left as a mandatory - verification item. - -`HARVEST.md` has exactly one solver worklist row per live target-reference key, -plus separate deferred validation/amount, geography, source-authentication, and -external-oracle worklists. `NOTES.md` records the Belgium solver lessons, -permanent survey tax-benefit holdouts, the amount/currency boundary, -`populace#263/#265`, and the future Axiom-backed `rulespec-am` boundary. - -Package identity after review: - -- CountrySpec fingerprint: - `64f50fa39e68e9ba6c451e3a47a2f2adeaba5a5ccb147cb80297f942de433ca8` -- Typed spec SHA-256: - `659b6baf5ebbd71fb7786ec4c4d49df565b2bddabeb868a9385ed226c56880f9` +# PR #849 env-variable correction — final report + +## What this was + +PR #849's review comment (Fable, main) caught that `POPULACE_LEDGER_URL` / +`_KEY` / `_API_KEY` / `_EXPORT_KEY` are the **Logbook** store's credentials +(Supabase `logbook` schema, `logbook_writer` / `logbook_exporter` roles — +`packages/microcosm-build/src/microcosm/build/logbook.py`'s own docstring +and `logbook/README.md`), not Chronicle fact-store variables. "Ledger" +there is the generic build-ledger sense renamed to Logbook on 2026-08-08 +(microcosm#632) specifically to stop colliding with Chronicle. The PR as +filed introduced `CHRONICLE_*` as the preferred spelling for these — the +wrong referent, recreating the exact collision #632 fixed. This task +renamed the preferred spelling to `LOGBOOK_*` everywhere and reframed the +docs/changelog/PR body accordingly. The epoch half of the PR +(`chronicle_epoch.py`, `ledger_artifact.py`, `import_entry_facts.py`, +`ledger_targets.py`) was approved as-is and left untouched except for two +stale module-path comments that had to follow the file rename. + +## Branch state before starting + +`chronicle-dual-accept` had 2 local commits ahead of `origin/chronicle-dual-accept` +(`d6cd24ca`, `1be2f18e`) that were unrelated follow-on work (journal exact-counts +edit, provenance docstring edit) not present on the PR head. Per the assignment, +the PR head (`origin/chronicle-dual-accept` = `4115573c`) is authoritative. Those +two commits were preserved on a new branch, +`backup/chronicle-dual-accept-pre-correction-1be2f18e`, before `chronicle-dual-accept` +was hard-reset to `4115573c`. `origin/main` was already an ancestor, so no rebase +was needed. + +## New head SHA + +**`a5121a1e0c4da8d7767fe8322bc948a957c7c5e7`**, pushed as a fast-forward of +`origin/chronicle-dual-accept` (no force needed — the branch was reset to the +exact PR head before new commits were added). + +Three new commits on top of `4115573c`: + +1. `061544e1` — Rename the CHRONICLE_* env dual-read window to LOGBOOK_* (module + rename + all symbol renames + every caller) +2. `ee73a624` — Reframe the env dual-read window as a Logbook cleanup, not + chronicle#143 (changelog, README, journal correction note) +3. `a5121a1e` — Apply ruff import-sort and formatting to the renamed files + +## Files changed (relative to 4115573c) + +``` + PROGRESS-chronicle-dual-accept.md | 29 +++++ + changelog.d/chronicle-dual-accept.added.md | 4 +- + logbook/README.md | 4 +- + packages/microcosm-build/src/microcosm/build/__init__.py | 10 +- + packages/microcosm-build/src/microcosm/build/logbook.py | 29 ++--- + .../build/{chronicle_env.py => logbook_env.py} | 91 ++++++------ + packages/microcosm-build/src/microcosm/build/uk_runtime/firm_generation.py | 2 +- + packages/microcosm-build/src/microcosm/build/us_runtime/source_coverage.py | 2 +- + .../tests/{test_chronicle_env.py => test_logbook_env.py} | 98 +++++++------- + tools/logbook.py | 35 ++--- +``` + +Plus the PR body itself, edited via `gh pr edit 849` (four scoped changes: the +`chronicle_env` → `logbook_env` framing paragraph, two audit-table rows naming +`chronicle_env`/`CHRONICLE_*`, and the `test_chronicle_env.py` → `test_logbook_env.py` +test-file mention — everything else verbatim, confirmed by diffing old vs. new body +text before posting). + +### Rename map applied + +| Old | New | +|---|---| +| `chronicle_env.py` | `logbook_env.py` | +| `chronicle_env()` | `logbook_env()` | +| `chronicle_env_names()` | `logbook_env_names()` | +| `describe_chronicle_env()` | `describe_logbook_env()` | +| `reset_chronicle_env_deprecation_warnings()` | `reset_logbook_env_deprecation_warnings()` | +| `CHRONICLE_URL_ENV = "CHRONICLE_URL"` | `LOGBOOK_URL_ENV = "LOGBOOK_URL"` | +| `CHRONICLE_KEY_ENV = "CHRONICLE_KEY"` | `LOGBOOK_KEY_ENV = "LOGBOOK_KEY"` | +| `CHRONICLE_API_KEY_ENV = "CHRONICLE_API_KEY"` | `LOGBOOK_API_KEY_ENV = "LOGBOOK_API_KEY"` | +| `CHRONICLE_EXPORT_KEY_ENV = "CHRONICLE_EXPORT_KEY"` | `LOGBOOK_EXPORT_KEY_ENV = "LOGBOOK_EXPORT_KEY"` | +| `CHRONICLE_ENV_LEGACY_NAMES` | `LOGBOOK_ENV_LEGACY_NAMES` | +| `test_chronicle_env.py` | `test_logbook_env.py` | + +Unchanged (legacy names, same once-per-process `DeprecationWarning` behavior): +`LEGACY_URL_ENV = "POPULACE_LEDGER_URL"`, `LEGACY_KEY_ENV = "POPULACE_LEDGER_KEY"`, +`LEGACY_API_KEY_ENV = "POPULACE_LEDGER_API_KEY"`, +`LEGACY_EXPORT_KEY_ENV = "POPULACE_LEDGER_EXPORT_KEY"`. Warning message now +reads "...is the pre-rename name for LOGBOOK_...; the build ledger is now +Logbook (microcosm#632)..." — no chronicle#143 reference. + +Untouched by design (per the reviewer's own note — these really do translate +Chronicle ids / pin a Chronicle commit, so they correctly stay `CHRONICLE_*`): +`CHRONICLE_ONS_TURNOVER_BANDS`, `CHRONICLE_ONS_EMPLOYMENT_BANDS`, +`CHRONICLE_HMRC_BANDS` (`firm_generation.py`) and +`CHRONICLE_US_SOURCE_COVERAGE_CONTRACT_COMMIT` (`source_coverage.py`). Only +their comments' dotted-path references to the old `chronicle_env` module were +updated to `logbook_env`, since that module no longer exists under the old name. ## Verification -All commands ran offline. The package-wide command used the exact lock-required -`policyengine-us==1.819.0` already present in the local uv cache, so the two -engine-only test files ran inside the same aggregate rather than failing for a -missing optional dependency. - -Exact requested package-wide command: - -```sh -UV_NO_SYNC=1 UV_PROJECT_ENVIRONMENT=/tmp/armenia-uv-env.ME3TCl UV_CACHE_DIR=/tmp/uv-cache-armenia-814 PYTHONPATH=packages/microcosm-build/src:packages/microcosm-calibrate/src:packages/microcosm-frame/src:packages/microcosm-fit/src:packages/microcosm-data/src:/Users/maxghenis/.cache/uv/archive-v0/ewqqbcYNhWejPQ-OfsFxl:/tmp/armenia-no-engine-site.vZ5Kv8:/Users/maxghenis/PolicyEngine/chronicle/.venv/lib/python3.14/site-packages uv run pytest packages/microcosm-build -``` - -Result: **6,556 passed, 45 skipped, 2,351 warnings, 0 failed** in 3,671.74 -seconds (1:01:11). The warnings are existing numerical, pandas copy/fragmentation, -and PolicyEngine-US runtime warnings; none is Armenia-specific. - -Additional checks: - -- Focused package/golden/compiler suite: - `uv run pytest packages/microcosm-build/tests/test_spec_only_country_packages.py packages/microcosm-build/tests/test_country_spec.py packages/microcosm-build/tests/test_spec_engine_country_bundles.py -p no:cacheprovider` - — **101 passed** in 29.82 seconds. -- Authoritative `shared-spec` CI group: - `python3 tools/ci_test_groups.py --list shared-spec | xargs uv run pytest -p no:cacheprovider` - — **1,334 passed, 42 skipped, 1 warning** in 418.20 seconds. -- `ruff check .` — **passed**. -- `python3 tools/ci_test_groups.py --verify` — **verification=ok**, 313 test - files tracked. -- `git diff --check` — **passed**. - -## Deviations from Belgium and why - -1. Armenia consumes a pre-built public US donor pool instead of native, - restricted SILC. No `support_spine.json` is present because the current - vocabulary describes raw ASEC pool construction, not an existing artifact. -2. Armenia is engine-free. No target, gate, or release file requires - `rulespec-am`; any later rules leg must use Frame's `RulesEngine` protocol - through the Axiom adapter. -3. Marz is target-assigned before community cloning. The clone factor is the - compile-safe minimum of one, not Belgium's 20: collision-avoiding fanout must - wait for the 71-to-11 roster and within-marz support evidence. -4. There is no incumbent, so parity/export/target-surface gates are absent. - National accounts back a deferred macro-realism band; CEQ and World Bank - estimates remain documentation-only band candidates until harvested. -5. The live solver manifest contains eight count/indicator series contracts, - not guessed scalar cells or AMD amount targets. Those series will expand to - the reviewed cell-level profile after Chronicle harvest; the eventual - 10–16-margin selection is not fabricated in this package. - -## Top five maintainer questions - -1. Which immutable `populace-us` revision/file, hash, licence, and column - inventory certify the donor input? -2. Which exact Statbank, ILCS, LFS, SRC, pension, and benefit tables/cells define - the 2024 profile, and what generated cell-reference/binding artifact owns - their fanout? -3. Which scale-free indicator or reviewed AMD-compatible pre-built bridge makes - donor consumption bands and any future amount rows conceptually comparable? -4. What ex-ante Armenia aggregate-fit, family-fit, macro-realism, ESS, and - weight-ratio thresholds—and which shared evaluators—activate the deferred - gate declarations? -5. What is the authoritative 2022 marz/community roster and assignment table, - and when will the shared geography/source-coverage runtimes land under - `populace#263/#265`? - -No network fetch, push, PR, artifact build, release, or publication was -performed. The repository-root `PROGRESS.md` was not touched. +### `uv sync --all-packages --extra us` +``` +Resolved 125 packages in 15ms +Checked 102 packages in 68ms +``` + +### Targeted tests (exact command from the assignment) +``` +uv run pytest packages/microcosm-build/tests/test_logbook_env.py \ + packages/microcosm-build/tests/test_chronicle_epoch.py \ + packages/microcosm-build/tests/test_logbook.py \ + packages/microcosm-build/tests/test_logbook_cli.py \ + packages/microcosm-build/tests/test_logbook_backfill.py \ + packages/microcosm-build/tests/test_logbook_adoption.py \ + packages/microcosm-build/tests/test_logbook_archive.py \ + packages/microcosm-build/tests/test_logbook_prediction_seed.py \ + packages/microcosm-build/tests/test_logbook_chain_scopes_pg.py -q +``` +(`test_logbook*.py` expanded explicitly; `test_logbook_cli.py` is the only +other test file that imports `tools/logbook.py`, confirmed by grep.) + +``` +........................................................................ [ 46%] +........................................................................ [ 92%] +............ [100%] +96 passed +``` + +Three `DeprecationWarning`s fired during the run, confirming the renamed +legacy fallback path and the new message text work end-to-end: +``` +DeprecationWarning: POPULACE_LEDGER_URL is the pre-rename name for LOGBOOK_URL; +the build ledger is now Logbook (microcosm#632). Set LOGBOOK_URL instead — +POPULACE_LEDGER_URL stays honored only for the dual-read window. +``` +(and the same for `POPULACE_LEDGER_KEY`/`LOGBOOK_KEY`, +`POPULACE_LEDGER_API_KEY`/`LOGBOOK_API_KEY`, +`POPULACE_LEDGER_EXPORT_KEY`/`LOGBOOK_EXPORT_KEY`.) + +### `uv run ruff check .` +``` +All checks passed! +``` +(Two `I001` unsorted-import errors were introduced by the rename in +`build/__init__.py` and `tools/logbook.py`; fixed with `ruff check --fix` +and `ruff format`, scoped to only the files this task touched, committed +separately.) + +### `uv run ruff format --check .` +116 files "would be reformatted" — all pre-existing and unrelated to this +task (confirmed: the count was 118 before the ruff --fix/format pass above +and dropped to exactly 116 after fixing the two files this task's edits +affected; the remaining 116 are files this task never touched, e.g. +`test_uk_firm_generation.py`, `tools/spec_engine_coverage.py`). + +### `tools/ci_test_groups.py --verify` (the lint lane's own check) +``` +packages/microcosm-build/tests/test_logbook_env.py -> shared-spec +... +verification=ok +``` + +### Full `microcosm-build` package suite +`uv run pytest packages/microcosm-build -q` was launched as supplementary +verification beyond the assignment's required command list. It was still +running in the background when this report was written (long-running — +includes engine-gated US/UK tests now that `--extra us` is synced). The +required, scoped verification above is fully green; this run is extra +thoroughness, not a gate for this deliverable. + +### Grep sweep (exact command from the assignment) +``` +grep -rn "CHRONICLE_URL\|CHRONICLE_KEY\|CHRONICLE_API_KEY\|CHRONICLE_EXPORT_KEY\|chronicle_env" \ + --include=*.py --include=*.md . +``` +Result — 5 lines, all in `PROGRESS-chronicle-dual-accept.md`, all intentional: + +``` +PROGRESS-chronicle-dual-accept.md:39:- `microcosm/build/chronicle_env.py` — the env dual-read window, one helper, +PROGRESS-chronicle-dual-accept.md:51:- Tests: `test_chronicle_epoch.py`, `test_chronicle_env.py`, plus mixed-epoch +PROGRESS-chronicle-dual-accept.md:115:as-is. The env half only: `chronicle_env.py` renamed to `logbook_env.py`, +PROGRESS-chronicle-dual-accept.md:116:`CHRONICLE_*_ENV`/`chronicle_env`/`chronicle_env_names`/ +PROGRESS-chronicle-dual-accept.md:117:`describe_chronicle_env`/`reset_chronicle_env_deprecation_warnings` renamed +``` + +Lines 39/51 are in the journal's original "Done" section, which the +assignment explicitly says not to rewrite ("append a dated correction note; +do not rewrite history") — they're the historical record of what this +branch originally built, before the correction. Lines 115-117 are inside +the new "Correction (2026-09-02)" section this task appended, which by +necessity names the old symbols it renamed away from. **Zero occurrences in +any `.py` file** and **zero occurrences outside this one journal file's +history/correction-note text** — i.e., nothing outside git history in any +file that matters to running code, docs a reader would act on, or the PR +body. + +## What was NOT touched + +- The epoch half (`chronicle_epoch.py`, `ledger_artifact.py`, + `import_entry_facts.py`, `ledger_targets.py`, CD-vintage tests, band/commit + aliases) — approved as-is by the review, left alone except for the two + module-path comment fixes noted above. +- `CHRONICLE_*_BANDS` / `CHRONICLE_US_SOURCE_COVERAGE_CONTRACT_COMMIT` — real + Chronicle aliases, correctly still `CHRONICLE_*`. +- No merge was performed. PR #849 remains open. + +## Backup branch + +`backup/chronicle-dual-accept-pre-correction-1be2f18e` preserves the two +local-only commits that were on `chronicle-dual-accept` before this task +reset it to the PR head, in case that follow-on work (exact whole-feed +counts in the journal; per-row ids in the provenance docstring) is still +wanted — it was not part of PR #849 and was not re-applied. From 47df8f44f01f5ecd345f992d3b1079c9fad3b0be Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Wed, 2 Sep 2026 12:04:31 -0400 Subject: [PATCH 12/23] Correct out.md's head-sha accounting for its own commit Co-Authored-By: Claude Fable 5.1 --- out.md | 25 +++++++++++++++++++++---- 1 file changed, 21 insertions(+), 4 deletions(-) diff --git a/out.md b/out.md index 64335b427..0fbb13fb8 100644 --- a/out.md +++ b/out.md @@ -30,11 +30,15 @@ was needed. ## New head SHA -**`a5121a1e0c4da8d7767fe8322bc948a957c7c5e7`**, pushed as a fast-forward of -`origin/chronicle-dual-accept` (no force needed — the branch was reset to the -exact PR head before new commits were added). +**`a5121a1e0c4da8d7767fe8322bc948a957c7c5e7`** is the head of the substantive +code/docs/PR-body correction (all verification below was run against this +sha). One further commit, `b98edd88`, adds this report file (`out.md`, +following this repo's established per-task convention — see "out.md history" +below) on top, making `b98edd88` the actual current branch tip. Both were +pushed as fast-forwards of `origin/chronicle-dual-accept` (no force needed — +the branch was reset to the exact PR head before any new commits were added). -Three new commits on top of `4115573c`: +Three code/docs commits on top of `4115573c` (the original PR head): 1. `061544e1` — Rename the CHRONICLE_* env dual-read window to LOGBOOK_* (module rename + all symbol renames + every caller) @@ -42,6 +46,10 @@ Three new commits on top of `4115573c`: chronicle#143 (changelog, README, journal correction note) 3. `a5121a1e` — Apply ruff import-sort and formatting to the renamed files +Plus the report commit: + +4. `b98edd88` — Report the CHRONICLE_* to LOGBOOK_* correction on PR #849 (this file) + ## Files changed (relative to 4115573c) ``` @@ -202,6 +210,15 @@ body. Chronicle aliases, correctly still `CHRONICLE_*`. - No merge was performed. PR #849 remains open. +## out.md history + +Root `out.md` is this repo's established ephemeral per-task report file +(`git log --all -- out.md` shows dozens of prior merged PRs overwriting it +with their own final report, same as this one). Its prior content — an +Armenia country-package report from PR #814, merged 2026-08-28 — remains +fully recoverable at `git show 305a13ed:out.md` and was not otherwise lost; +overwriting it here follows the same convention every prior task used. + ## Backup branch `backup/chronicle-dual-accept-pre-correction-1be2f18e` preserves the two From 1f78847cbad7d9c6b258829662b6f1104819dc2b Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Wed, 2 Sep 2026 12:05:46 -0400 Subject: [PATCH 13/23] Drop the lane's root out.md; the repo keeps task journals as PROGRESS-*.md only Co-Authored-By: Claude Fable 5.1 --- out.md | 228 --------------------------------------------------------- 1 file changed, 228 deletions(-) delete mode 100644 out.md diff --git a/out.md b/out.md deleted file mode 100644 index 0fbb13fb8..000000000 --- a/out.md +++ /dev/null @@ -1,228 +0,0 @@ -# PR #849 env-variable correction — final report - -## What this was - -PR #849's review comment (Fable, main) caught that `POPULACE_LEDGER_URL` / -`_KEY` / `_API_KEY` / `_EXPORT_KEY` are the **Logbook** store's credentials -(Supabase `logbook` schema, `logbook_writer` / `logbook_exporter` roles — -`packages/microcosm-build/src/microcosm/build/logbook.py`'s own docstring -and `logbook/README.md`), not Chronicle fact-store variables. "Ledger" -there is the generic build-ledger sense renamed to Logbook on 2026-08-08 -(microcosm#632) specifically to stop colliding with Chronicle. The PR as -filed introduced `CHRONICLE_*` as the preferred spelling for these — the -wrong referent, recreating the exact collision #632 fixed. This task -renamed the preferred spelling to `LOGBOOK_*` everywhere and reframed the -docs/changelog/PR body accordingly. The epoch half of the PR -(`chronicle_epoch.py`, `ledger_artifact.py`, `import_entry_facts.py`, -`ledger_targets.py`) was approved as-is and left untouched except for two -stale module-path comments that had to follow the file rename. - -## Branch state before starting - -`chronicle-dual-accept` had 2 local commits ahead of `origin/chronicle-dual-accept` -(`d6cd24ca`, `1be2f18e`) that were unrelated follow-on work (journal exact-counts -edit, provenance docstring edit) not present on the PR head. Per the assignment, -the PR head (`origin/chronicle-dual-accept` = `4115573c`) is authoritative. Those -two commits were preserved on a new branch, -`backup/chronicle-dual-accept-pre-correction-1be2f18e`, before `chronicle-dual-accept` -was hard-reset to `4115573c`. `origin/main` was already an ancestor, so no rebase -was needed. - -## New head SHA - -**`a5121a1e0c4da8d7767fe8322bc948a957c7c5e7`** is the head of the substantive -code/docs/PR-body correction (all verification below was run against this -sha). One further commit, `b98edd88`, adds this report file (`out.md`, -following this repo's established per-task convention — see "out.md history" -below) on top, making `b98edd88` the actual current branch tip. Both were -pushed as fast-forwards of `origin/chronicle-dual-accept` (no force needed — -the branch was reset to the exact PR head before any new commits were added). - -Three code/docs commits on top of `4115573c` (the original PR head): - -1. `061544e1` — Rename the CHRONICLE_* env dual-read window to LOGBOOK_* (module - rename + all symbol renames + every caller) -2. `ee73a624` — Reframe the env dual-read window as a Logbook cleanup, not - chronicle#143 (changelog, README, journal correction note) -3. `a5121a1e` — Apply ruff import-sort and formatting to the renamed files - -Plus the report commit: - -4. `b98edd88` — Report the CHRONICLE_* to LOGBOOK_* correction on PR #849 (this file) - -## Files changed (relative to 4115573c) - -``` - PROGRESS-chronicle-dual-accept.md | 29 +++++ - changelog.d/chronicle-dual-accept.added.md | 4 +- - logbook/README.md | 4 +- - packages/microcosm-build/src/microcosm/build/__init__.py | 10 +- - packages/microcosm-build/src/microcosm/build/logbook.py | 29 ++--- - .../build/{chronicle_env.py => logbook_env.py} | 91 ++++++------ - packages/microcosm-build/src/microcosm/build/uk_runtime/firm_generation.py | 2 +- - packages/microcosm-build/src/microcosm/build/us_runtime/source_coverage.py | 2 +- - .../tests/{test_chronicle_env.py => test_logbook_env.py} | 98 +++++++------- - tools/logbook.py | 35 ++--- -``` - -Plus the PR body itself, edited via `gh pr edit 849` (four scoped changes: the -`chronicle_env` → `logbook_env` framing paragraph, two audit-table rows naming -`chronicle_env`/`CHRONICLE_*`, and the `test_chronicle_env.py` → `test_logbook_env.py` -test-file mention — everything else verbatim, confirmed by diffing old vs. new body -text before posting). - -### Rename map applied - -| Old | New | -|---|---| -| `chronicle_env.py` | `logbook_env.py` | -| `chronicle_env()` | `logbook_env()` | -| `chronicle_env_names()` | `logbook_env_names()` | -| `describe_chronicle_env()` | `describe_logbook_env()` | -| `reset_chronicle_env_deprecation_warnings()` | `reset_logbook_env_deprecation_warnings()` | -| `CHRONICLE_URL_ENV = "CHRONICLE_URL"` | `LOGBOOK_URL_ENV = "LOGBOOK_URL"` | -| `CHRONICLE_KEY_ENV = "CHRONICLE_KEY"` | `LOGBOOK_KEY_ENV = "LOGBOOK_KEY"` | -| `CHRONICLE_API_KEY_ENV = "CHRONICLE_API_KEY"` | `LOGBOOK_API_KEY_ENV = "LOGBOOK_API_KEY"` | -| `CHRONICLE_EXPORT_KEY_ENV = "CHRONICLE_EXPORT_KEY"` | `LOGBOOK_EXPORT_KEY_ENV = "LOGBOOK_EXPORT_KEY"` | -| `CHRONICLE_ENV_LEGACY_NAMES` | `LOGBOOK_ENV_LEGACY_NAMES` | -| `test_chronicle_env.py` | `test_logbook_env.py` | - -Unchanged (legacy names, same once-per-process `DeprecationWarning` behavior): -`LEGACY_URL_ENV = "POPULACE_LEDGER_URL"`, `LEGACY_KEY_ENV = "POPULACE_LEDGER_KEY"`, -`LEGACY_API_KEY_ENV = "POPULACE_LEDGER_API_KEY"`, -`LEGACY_EXPORT_KEY_ENV = "POPULACE_LEDGER_EXPORT_KEY"`. Warning message now -reads "...is the pre-rename name for LOGBOOK_...; the build ledger is now -Logbook (microcosm#632)..." — no chronicle#143 reference. - -Untouched by design (per the reviewer's own note — these really do translate -Chronicle ids / pin a Chronicle commit, so they correctly stay `CHRONICLE_*`): -`CHRONICLE_ONS_TURNOVER_BANDS`, `CHRONICLE_ONS_EMPLOYMENT_BANDS`, -`CHRONICLE_HMRC_BANDS` (`firm_generation.py`) and -`CHRONICLE_US_SOURCE_COVERAGE_CONTRACT_COMMIT` (`source_coverage.py`). Only -their comments' dotted-path references to the old `chronicle_env` module were -updated to `logbook_env`, since that module no longer exists under the old name. - -## Verification - -### `uv sync --all-packages --extra us` -``` -Resolved 125 packages in 15ms -Checked 102 packages in 68ms -``` - -### Targeted tests (exact command from the assignment) -``` -uv run pytest packages/microcosm-build/tests/test_logbook_env.py \ - packages/microcosm-build/tests/test_chronicle_epoch.py \ - packages/microcosm-build/tests/test_logbook.py \ - packages/microcosm-build/tests/test_logbook_cli.py \ - packages/microcosm-build/tests/test_logbook_backfill.py \ - packages/microcosm-build/tests/test_logbook_adoption.py \ - packages/microcosm-build/tests/test_logbook_archive.py \ - packages/microcosm-build/tests/test_logbook_prediction_seed.py \ - packages/microcosm-build/tests/test_logbook_chain_scopes_pg.py -q -``` -(`test_logbook*.py` expanded explicitly; `test_logbook_cli.py` is the only -other test file that imports `tools/logbook.py`, confirmed by grep.) - -``` -........................................................................ [ 46%] -........................................................................ [ 92%] -............ [100%] -96 passed -``` - -Three `DeprecationWarning`s fired during the run, confirming the renamed -legacy fallback path and the new message text work end-to-end: -``` -DeprecationWarning: POPULACE_LEDGER_URL is the pre-rename name for LOGBOOK_URL; -the build ledger is now Logbook (microcosm#632). Set LOGBOOK_URL instead — -POPULACE_LEDGER_URL stays honored only for the dual-read window. -``` -(and the same for `POPULACE_LEDGER_KEY`/`LOGBOOK_KEY`, -`POPULACE_LEDGER_API_KEY`/`LOGBOOK_API_KEY`, -`POPULACE_LEDGER_EXPORT_KEY`/`LOGBOOK_EXPORT_KEY`.) - -### `uv run ruff check .` -``` -All checks passed! -``` -(Two `I001` unsorted-import errors were introduced by the rename in -`build/__init__.py` and `tools/logbook.py`; fixed with `ruff check --fix` -and `ruff format`, scoped to only the files this task touched, committed -separately.) - -### `uv run ruff format --check .` -116 files "would be reformatted" — all pre-existing and unrelated to this -task (confirmed: the count was 118 before the ruff --fix/format pass above -and dropped to exactly 116 after fixing the two files this task's edits -affected; the remaining 116 are files this task never touched, e.g. -`test_uk_firm_generation.py`, `tools/spec_engine_coverage.py`). - -### `tools/ci_test_groups.py --verify` (the lint lane's own check) -``` -packages/microcosm-build/tests/test_logbook_env.py -> shared-spec -... -verification=ok -``` - -### Full `microcosm-build` package suite -`uv run pytest packages/microcosm-build -q` was launched as supplementary -verification beyond the assignment's required command list. It was still -running in the background when this report was written (long-running — -includes engine-gated US/UK tests now that `--extra us` is synced). The -required, scoped verification above is fully green; this run is extra -thoroughness, not a gate for this deliverable. - -### Grep sweep (exact command from the assignment) -``` -grep -rn "CHRONICLE_URL\|CHRONICLE_KEY\|CHRONICLE_API_KEY\|CHRONICLE_EXPORT_KEY\|chronicle_env" \ - --include=*.py --include=*.md . -``` -Result — 5 lines, all in `PROGRESS-chronicle-dual-accept.md`, all intentional: - -``` -PROGRESS-chronicle-dual-accept.md:39:- `microcosm/build/chronicle_env.py` — the env dual-read window, one helper, -PROGRESS-chronicle-dual-accept.md:51:- Tests: `test_chronicle_epoch.py`, `test_chronicle_env.py`, plus mixed-epoch -PROGRESS-chronicle-dual-accept.md:115:as-is. The env half only: `chronicle_env.py` renamed to `logbook_env.py`, -PROGRESS-chronicle-dual-accept.md:116:`CHRONICLE_*_ENV`/`chronicle_env`/`chronicle_env_names`/ -PROGRESS-chronicle-dual-accept.md:117:`describe_chronicle_env`/`reset_chronicle_env_deprecation_warnings` renamed -``` - -Lines 39/51 are in the journal's original "Done" section, which the -assignment explicitly says not to rewrite ("append a dated correction note; -do not rewrite history") — they're the historical record of what this -branch originally built, before the correction. Lines 115-117 are inside -the new "Correction (2026-09-02)" section this task appended, which by -necessity names the old symbols it renamed away from. **Zero occurrences in -any `.py` file** and **zero occurrences outside this one journal file's -history/correction-note text** — i.e., nothing outside git history in any -file that matters to running code, docs a reader would act on, or the PR -body. - -## What was NOT touched - -- The epoch half (`chronicle_epoch.py`, `ledger_artifact.py`, - `import_entry_facts.py`, `ledger_targets.py`, CD-vintage tests, band/commit - aliases) — approved as-is by the review, left alone except for the two - module-path comment fixes noted above. -- `CHRONICLE_*_BANDS` / `CHRONICLE_US_SOURCE_COVERAGE_CONTRACT_COMMIT` — real - Chronicle aliases, correctly still `CHRONICLE_*`. -- No merge was performed. PR #849 remains open. - -## out.md history - -Root `out.md` is this repo's established ephemeral per-task report file -(`git log --all -- out.md` shows dozens of prior merged PRs overwriting it -with their own final report, same as this one). Its prior content — an -Armenia country-package report from PR #814, merged 2026-08-28 — remains -fully recoverable at `git show 305a13ed:out.md` and was not otherwise lost; -overwriting it here follows the same convention every prior task used. - -## Backup branch - -`backup/chronicle-dual-accept-pre-correction-1be2f18e` preserves the two -local-only commits that were on `chronicle-dual-accept` before this task -reset it to the PR head, in case that follow-on work (exact whole-feed -counts in the journal; per-row ids in the provenance docstring) is still -wanted — it was not part of PR #849 and was not re-applied. From e76bffd27491e59bbb8ac2aacde9b3246dcac648 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Wed, 2 Sep 2026 12:48:33 -0400 Subject: [PATCH 14/23] Open a round-2 journal with the verified Chronicle identity ground truth The findings turn on which schema ids Chronicle actually emits, so the identities are recorded here with the file and line they were read from on chronicle origin/main rather than restated from the PR body. Co-Authored-By: Claude Fable 5.1 --- PROGRESS-chronicle-dual-accept-round2.md | 48 ++++++++++++++++++++++++ 1 file changed, 48 insertions(+) create mode 100644 PROGRESS-chronicle-dual-accept-round2.md diff --git a/PROGRESS-chronicle-dual-accept-round2.md b/PROGRESS-chronicle-dual-accept-round2.md new file mode 100644 index 000000000..8728c5df1 --- /dev/null +++ b/PROGRESS-chronicle-dual-accept-round2.md @@ -0,0 +1,48 @@ +# Round-2 review fixes — PR #849 (`chronicle-dual-accept`) + +Journal for the gate-findings pass on top of `1f78847c`. History, not state: +check git/GitHub for current truth. + +## Verified ground truth (read this session, 2026-09-02) + +Read from `PolicyEngine/chronicle` at `origin/main`: + +| Identity | Where | Epoch | +|---|---|---| +| `policyengine_ledger.consumer_artifact.v2` | `policyengine_chronicle/consumer.py:30`, README line 299 | ledger | +| `ledger.consumer_fact.v1` | `chronicle/consumer_contract.py:28` | ledger | +| `ledger.fact.v1` | `chronicle/core.py:103` | ledger | +| `ledger.source_cell.v1` | `chronicle/sources/cells.py:24` | ledger | +| `ledger.source_row.v1`, `ledger.source_column.v1`, `ledger.source_row_value.v1` | `chronicle/sources/rows.py:27-29` | ledger | +| `ledger.{aggregate_fact,semantic_fact,concept_alignment,dimension_set,observed_measure,source_release,source_series,universe_constraint_set}.v2` | `chronicle/consumer_contract.py`, `policyengine_chronicle/consumer.py:195` | ledger | + +`policyengine_ledger.consumer_artifact.v1` is what **Microcosm** mints +(`import_entry_facts.py:164`), never what Chronicle emits — it is ledger-era +history that must keep loading. + +There is **no** `origin/epoch-dual-domain` branch on chronicle +(`git ls-remote --heads origin | grep -i epoch` → empty), so the chronicle-era +successors are declared here by the migration rule chronicle#143 states — +new namespace, same family, version bumped by one — and the brief pins the two +that matter: `policyengine_chronicle.consumer_artifact.v3` and +`chronicle.consumer_fact.v2`. + +## State + +Round 2 complete; pushed to `chronicle-dual-accept`. **DO NOT MERGE** — +the lane is authorized to fix and push only. + +## Done + +- [x] (1) Declared identity registry keyed by `(namespace, family, version)` +- [x] (2) Autouse Logbook env isolation + fail-closed network in tests +- [x] (3) UK `_ledger_provenance` delegates to `provenance()` +- [x] (4) Undeclared chronicle-namespace identities labelled `undeclared` +- [x] (5) `microcosm.build.logbook_env` module import no longer shadowed +- [x] (6) `CHRONICLE_US_SOURCE_COVERAGE_CONTRACT_COMMIT` in the us_runtime barrel +- [x] (7) Non-string `schema_version` raises the documented ValueError +- [x] (8) `tools/logbook.py --remote` help prefers `LOGBOOK_*` + +## Next + +Nothing in this lane. Reviewer decides on merge. From dd244fb41ba4b0326b3c0bac94113b2b0dbd97db Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Wed, 2 Sep 2026 13:07:50 -0400 Subject: [PATCH 15/23] Accept the artifact ids producers actually stamp, from a declared registry The accepted set was {policyengine_ledger.consumer_artifact.v1, policyengine_chronicle.consumer_artifact.v2}, which is one generation off at both ends. Chronicle's main emits policyengine_ledger.consumer_artifact.v2 today (policyengine_chronicle/consumer.py:30, README line 299), so the loader rejected every artifact Chronicle currently publishes, and the chronicle-era successor of that id is .v3, not .v2. The v1 spelling is Microcosm's own minted artifact and stays accepted as the history it is. Epoch now comes from a declaration keyed by (namespace, family, version) rather than from the namespace segment of the key. Reading the namespace told us where a key claimed to come from, not that Chronicle issued it, so chronicle.source_cell.v3 and chronicle.anything.vN were being witnessed as Chronicle-issued identity. The ledger-era half of the table is transcribed from Chronicle's own hash-domain constants; the chronicle-era half is each one's successor under the rename rule, which puts source_cell at v2 where aggregate_fact is at v3. An undeclared spelling in a Chronicle namespace is reported as 'undeclared' in provenance, with the domain named, instead of being folded into an era nobody declared it for. The membership predicates are total over JSON. A manifest whose schema_version is a list or an object raised TypeError out of a frozenset membership test, bypassing the loader's own unsupported-schema message. Co-Authored-By: Claude Fable 5.1 --- .../src/microcosm/build/chronicle_epoch.py | 481 ++++++++++++++---- .../src/microcosm/build/ledger_artifact.py | 68 ++- .../us_runtime/us_trade/import_entry_facts.py | 8 +- .../tests/test_chronicle_epoch.py | 308 +++++++++-- .../tests/test_ledger_targets.py | 2 +- .../tests/test_us_trade_facts.py | 6 +- 6 files changed, 699 insertions(+), 174 deletions(-) diff --git a/packages/microcosm-build/src/microcosm/build/chronicle_epoch.py b/packages/microcosm-build/src/microcosm/build/chronicle_epoch.py index c84dd31b0..86fdf09a3 100644 --- a/packages/microcosm-build/src/microcosm/build/chronicle_epoch.py +++ b/packages/microcosm-build/src/microcosm/build/chronicle_epoch.py @@ -2,13 +2,10 @@ PolicyEngine Ledger is being renamed Chronicle. Per the migration spec on PolicyEngine/chronicle#143 the rename of *identities* happens **by epoch, -never in place**: Chronicle introduces chronicle-era hash domains -(``chronicle.aggregate_fact.v3``, ``chronicle.semantic_fact.v3``, ...) and -chronicle-named schema ids (``policyengine_chronicle.consumer_artifact.v2``, -``chronicle.consumer_fact.v2``) for **newly emitted** rows at a declared -cutover release. Ledger-era ids (``ledger.aggregate_fact.v2``, -``policyengine_ledger.consumer_artifact.v1``, ...) stay valid *history* -forever: no golden regenerates and no witnessed row is rewritten. +never in place**: Chronicle introduces chronicle-era hash domains and +chronicle-named schema ids for **newly emitted** rows at a declared cutover +release, and ledger-era ids stay valid *history* forever. No golden +regenerates and no witnessed row is rewritten. Microcosm is a consumer, so it must accept both eras **before** Chronicle flips emit. This module is the single place that knows which identities are @@ -19,12 +16,15 @@ hash. Comparison stays exact string equality, which is correct in both eras; what changes is that a validator asking "is this a Chronicle fact key?" must answer yes for either epoch. -2. **Never hard-code a single epoch in a validator.** Epoch detection here is - *structural* — the namespace segment of the key domain — not a lookup in a - frozen list of domain strings. Chronicle has declared the family-``v3`` - spellings for the two fact families named in the spec; the version numbers - the remaining families will take are not declared yet, and this module - deliberately does not guess them. +2. **Epoch is a declaration, not a guess.** Every identity this module calls + Chronicle-issued is listed in :data:`DECLARED_IDENTITIES` below, keyed by + ``(namespace, family, version)``. Structural parsing of + ``..v:`` still happens, but only so an + *undeclared* key in a Chronicle namespace can be reported as + :data:`UNDECLARED` in provenance — never silently promoted to issued + identity. A cutover that ships a spelling this table does not carry is a + one-row diff here, made deliberately, with the reader able to see which + spellings the build was actually prepared for. 3. **Microcosm-minted keys are epoch-independent.** Keys Microcosm mints live in Microcosm-owned namespaces (``microcosm.derived_fact.*``, ``populace_us_trade.*``) and are frozen at v1 by microcosm#639. They carry @@ -35,8 +35,16 @@ (``ledger_aggregate_fact_key``, ``ledger_commit``), H5 attributes, ``populace_*`` ids, goldens, and fixtures are frozen at v1. -Only the identity strings the migration spec names explicitly are pinned as -literals here. Everything else is derived. +Provenance for the ledger-era half of the table was read out of +``PolicyEngine/chronicle`` at ``origin/main`` rather than restated from a +plan: ``policyengine_chronicle/consumer.py`` for the artifact id, +``chronicle/consumer_contract.py`` for the per-row id and the ``.v2`` fact +domains, ``chronicle/core.py`` and ``chronicle/sources/`` for the ``.v1`` +source domains. The chronicle-era half is the successor each of those takes +under the rule chronicle#143 states for the rename — same family, new +namespace, version bumped by one — with the two ids the consumer migration +pins spelled out explicitly. Chronicle has published no per-family +enumeration of its own; when it does, this table is what changes. """ from __future__ import annotations @@ -52,19 +60,34 @@ "CHRONICLE_CONSUMER_ARTIFACT_SCHEMA_VERSION", "CHRONICLE_CONSUMER_FACT_SCHEMA_VERSION", "CHRONICLE_EPOCH", + "CHRONICLE_FACT_KEY_DOMAINS", + "CHRONICLE_NAMESPACES", + "DECLARED_IDENTITIES", + "DECLARED_IDENTITY_EPOCHS", + "DECLARED_IDENTITY_REGISTRY", "EPOCHS", + "FACT_KEY_EPOCH_LABELS", "FACT_KEY_FIELDS", + "FACT_KEY_IDENTITY_KIND", + "CONSUMER_ARTIFACT_SCHEMA_IDENTITY_KIND", + "CONSUMER_FACT_SCHEMA_IDENTITY_KIND", + "DeclaredIdentity", "FactKeyIdentity", - "LEDGER_CONSUMER_ARTIFACT_SCHEMA_VERSION", "LEDGER_CONSUMER_FACT_SCHEMA_VERSION", "LEDGER_EPOCH", "LEDGER_FACT_KEY_DOMAINS", + "MICROCOSM_CONSUMER_ARTIFACT_SCHEMA_VERSION", + "PUBLISHED_CONSUMER_ARTIFACT_SCHEMA_VERSION", + "UNDECLARED", "consumer_artifact_schema_epoch", "consumer_fact_schema_epoch", + "declared_identity", "describe_accepted_consumer_artifact_schema_versions", "describe_accepted_consumer_fact_schema_versions", "fact_key_epoch", + "fact_key_epoch_label", "feed_fact_key_epochs", + "feed_undeclared_fact_key_domains", "is_accepted_consumer_artifact_schema_version", "is_accepted_consumer_fact_schema_version", "is_chronicle_fact_key", @@ -72,65 +95,225 @@ "row_fact_key_epochs", ] -#: The ledger era: everything Chronicle emitted under its former name. +#: The ledger era: everything Chronicle issued under its former name. LEDGER_EPOCH = "ledger" -#: The chronicle era: everything emitted from the declared cutover forward. +#: The chronicle era: everything issued from the declared cutover forward. CHRONICLE_EPOCH = "chronicle" #: Both eras, oldest first. A validator iterates this; it never names one. EPOCHS: tuple[str, ...] = (LEDGER_EPOCH, CHRONICLE_EPOCH) -#: Consumer-artifact manifest ``schema_version``, per epoch. Both spellings -#: are declared in the chronicle#143 migration spec. -LEDGER_CONSUMER_ARTIFACT_SCHEMA_VERSION = "policyengine_ledger.consumer_artifact.v1" +#: The label for an identity in a Chronicle namespace that this module's +#: declaration table does not carry — an unrecognized family, or a family at +#: a version nobody declared. Such an identity is *reported*, so a release +#: manifest shows that the feed contained something the build was not +#: prepared for, but it is never counted as Chronicle-issued identity. +UNDECLARED = "undeclared" + +#: The labels :func:`feed_fact_key_epochs` can emit, in report order. +FACT_KEY_EPOCH_LABELS: tuple[str, ...] = (*EPOCHS, UNDECLARED) + +#: Namespace -> epoch, for every namespace Chronicle issues identities in. +#: Fact keys and per-row schema ids use the bare ``ledger`` / ``chronicle`` +#: namespace; the consumer-artifact manifest id uses the package-qualified +#: ``policyengine_ledger`` / ``policyengine_chronicle`` one. +CHRONICLE_NAMESPACES: Mapping[str, str] = { + "ledger": LEDGER_EPOCH, + "policyengine_ledger": LEDGER_EPOCH, + "chronicle": CHRONICLE_EPOCH, + "policyengine_chronicle": CHRONICLE_EPOCH, +} + +#: Identity kinds. An identity's kind says where it is allowed to appear: +#: a fact-key domain never gates a manifest and a manifest id is never a key. +CONSUMER_ARTIFACT_SCHEMA_IDENTITY_KIND = "consumer_artifact_schema" +CONSUMER_FACT_SCHEMA_IDENTITY_KIND = "consumer_fact_schema" +FACT_KEY_IDENTITY_KIND = "fact_key" + +_VERSION_PATTERN = re.compile(r"^v\d+$") + + +@dataclass(frozen=True) +class DeclaredIdentity: + """One identity spelling the migration declares, and the epoch that owns it.""" + + namespace: str + family: str + version: str + epoch: str + kind: str + + @property + def identity(self) -> str: + """The identity string as it appears in an artifact, ``a.b.vN``.""" + return f"{self.namespace}.{self.family}.{self.version}" + + @property + def registry_key(self) -> tuple[str, str, str]: + """The ``(namespace, family, version)`` triple this is registered under.""" + return (self.namespace, self.family, self.version) + + +def _declare(identity: str, *, epoch: str, kind: str) -> DeclaredIdentity: + """Build one registry row from its literal identity string. + + Every argument is a literal at the call site below, so the table reads as + the list of spellings it is, and a typo fails at import rather than at + cutover. + """ + namespace, _, remainder = identity.partition(".") + family, _, version = remainder.rpartition(".") + if not namespace or not family or not _VERSION_PATTERN.match(version): + raise ValueError(f"Not a Chronicle identity spelling: {identity!r}.") + if CHRONICLE_NAMESPACES.get(namespace) != epoch: + raise ValueError( + f"{identity!r} is declared for the {epoch!r} epoch but its " + f"namespace {namespace!r} belongs to " + f"{CHRONICLE_NAMESPACES.get(namespace)!r}." + ) + return DeclaredIdentity( + namespace=namespace, family=family, version=version, epoch=epoch, kind=kind + ) + + +#: The consumer-artifact manifest ``schema_version`` Microcosm's **own** +#: minted artifacts declare (``import_entry_facts.write_consumer_artifact``). +#: Frozen at v1 by microcosm#639: these bytes must not move, and they must +#: keep loading forever. +MICROCOSM_CONSUMER_ARTIFACT_SCHEMA_VERSION = "policyengine_ledger.consumer_artifact.v1" + +#: The consumer-artifact manifest ``schema_version`` **Chronicle** stamps +#: today (``policyengine_chronicle/consumer.py`` on chronicle ``main``). Still +#: ledger-era: chronicle#143 has not flipped the namespace yet. +PUBLISHED_CONSUMER_ARTIFACT_SCHEMA_VERSION = "policyengine_ledger.consumer_artifact.v2" + +#: The chronicle-era successor of the id above: same payload, chronicle +#: namespace, version bumped. CHRONICLE_CONSUMER_ARTIFACT_SCHEMA_VERSION = ( - "policyengine_chronicle.consumer_artifact.v2" + "policyengine_chronicle.consumer_artifact.v3" ) -#: Per-row consumer-fact ``schema_version``, per epoch. +#: The per-row consumer-fact ``schema_version`` Chronicle stamps today +#: (``chronicle/consumer_contract.py`` on chronicle ``main``), and its +#: chronicle-era successor. LEDGER_CONSUMER_FACT_SCHEMA_VERSION = "ledger.consumer_fact.v1" CHRONICLE_CONSUMER_FACT_SCHEMA_VERSION = "chronicle.consumer_fact.v2" -CONSUMER_ARTIFACT_SCHEMA_VERSION_BY_EPOCH: Mapping[str, str] = { - LEDGER_EPOCH: LEDGER_CONSUMER_ARTIFACT_SCHEMA_VERSION, - CHRONICLE_EPOCH: CHRONICLE_CONSUMER_ARTIFACT_SCHEMA_VERSION, +#: Fact-key domains, ledger-era spelling beside its chronicle-era successor. +#: The ledger column is what Chronicle's ``main`` hashes with today; the +#: version numbers differ by family (the source-side domains are still v1 +#: where the derived ones are v2), which is why a version can never be +#: assumed and why this table is explicit rather than computed at the call +#: site. +_FACT_KEY_DOMAIN_DECLARATIONS: tuple[tuple[str, str], ...] = ( + # chronicle/core.py + ("ledger.fact.v1", "chronicle.fact.v2"), + # chronicle/sources/cells.py, chronicle/sources/rows.py + ("ledger.source_cell.v1", "chronicle.source_cell.v2"), + ("ledger.source_row.v1", "chronicle.source_row.v2"), + ("ledger.source_column.v1", "chronicle.source_column.v2"), + ("ledger.source_row_value.v1", "chronicle.source_row_value.v2"), + # chronicle/consumer_contract.py, policyengine_chronicle/consumer.py + ("ledger.aggregate_fact.v2", "chronicle.aggregate_fact.v3"), + ("ledger.semantic_fact.v2", "chronicle.semantic_fact.v3"), + ("ledger.concept_alignment.v2", "chronicle.concept_alignment.v3"), + ("ledger.dimension_set.v2", "chronicle.dimension_set.v3"), + ("ledger.observed_measure.v2", "chronicle.observed_measure.v3"), + ("ledger.source_release.v2", "chronicle.source_release.v3"), + ("ledger.source_series.v2", "chronicle.source_series.v3"), + ("ledger.universe_constraint_set.v2", "chronicle.universe_constraint_set.v3"), +) + +#: Every identity spelling this consumer is declared to understand. The +#: registry below is keyed off it; nothing outside this tuple is Chronicle +#: identity as far as Microcosm is concerned. +DECLARED_IDENTITIES: tuple[DeclaredIdentity, ...] = ( + _declare( + MICROCOSM_CONSUMER_ARTIFACT_SCHEMA_VERSION, + epoch=LEDGER_EPOCH, + kind=CONSUMER_ARTIFACT_SCHEMA_IDENTITY_KIND, + ), + _declare( + PUBLISHED_CONSUMER_ARTIFACT_SCHEMA_VERSION, + epoch=LEDGER_EPOCH, + kind=CONSUMER_ARTIFACT_SCHEMA_IDENTITY_KIND, + ), + _declare( + CHRONICLE_CONSUMER_ARTIFACT_SCHEMA_VERSION, + epoch=CHRONICLE_EPOCH, + kind=CONSUMER_ARTIFACT_SCHEMA_IDENTITY_KIND, + ), + _declare( + LEDGER_CONSUMER_FACT_SCHEMA_VERSION, + epoch=LEDGER_EPOCH, + kind=CONSUMER_FACT_SCHEMA_IDENTITY_KIND, + ), + _declare( + CHRONICLE_CONSUMER_FACT_SCHEMA_VERSION, + epoch=CHRONICLE_EPOCH, + kind=CONSUMER_FACT_SCHEMA_IDENTITY_KIND, + ), + *( + _declare(identity, epoch=epoch, kind=FACT_KEY_IDENTITY_KIND) + for ledger_domain, chronicle_domain in _FACT_KEY_DOMAIN_DECLARATIONS + for identity, epoch in ( + (ledger_domain, LEDGER_EPOCH), + (chronicle_domain, CHRONICLE_EPOCH), + ) + ), +) + +#: The registry: ``(namespace, family, version)`` -> the declared identity. +#: One triple can only belong to one epoch, and the constructor above +#: refuses a row whose namespace disagrees with the epoch it claims. +DECLARED_IDENTITY_REGISTRY: Mapping[tuple[str, str, str], DeclaredIdentity] = { + declaration.registry_key: declaration for declaration in DECLARED_IDENTITIES } -CONSUMER_FACT_SCHEMA_VERSION_BY_EPOCH: Mapping[str, str] = { - LEDGER_EPOCH: LEDGER_CONSUMER_FACT_SCHEMA_VERSION, - CHRONICLE_EPOCH: CHRONICLE_CONSUMER_FACT_SCHEMA_VERSION, +if len(DECLARED_IDENTITY_REGISTRY) != len(DECLARED_IDENTITIES): # pragma: no cover + raise ValueError("Duplicate (namespace, family, version) in DECLARED_IDENTITIES.") + +#: The same registry reduced to what epoch resolution actually asks it: +#: ``(namespace, family, version)`` -> epoch. +DECLARED_IDENTITY_EPOCHS: Mapping[tuple[str, str, str], str] = { + registry_key: declaration.epoch + for registry_key, declaration in DECLARED_IDENTITY_REGISTRY.items() } + +def _identities_of(kind: str) -> tuple[str, ...]: + """Declared identity strings of one kind, oldest era first.""" + return tuple( + declaration.identity + for era in EPOCHS + for declaration in DECLARED_IDENTITIES + if declaration.kind == kind and declaration.epoch == era + ) + + #: Membership sets. Loaders test membership, never equality with one era. +#: The artifact set carries three spellings, not two: Microcosm's own minted +#: v1, Chronicle's published v2, and the chronicle-era v3 successor. ACCEPTED_CONSUMER_ARTIFACT_SCHEMA_VERSIONS = frozenset( - CONSUMER_ARTIFACT_SCHEMA_VERSION_BY_EPOCH.values() + _identities_of(CONSUMER_ARTIFACT_SCHEMA_IDENTITY_KIND) ) ACCEPTED_CONSUMER_FACT_SCHEMA_VERSIONS = frozenset( - CONSUMER_FACT_SCHEMA_VERSION_BY_EPOCH.values() + _identities_of(CONSUMER_FACT_SCHEMA_IDENTITY_KIND) ) -#: Ledger-era fact-key domains, as *observed* in the feeds and fixtures this -#: repo carries: every family in -#: ``tests/fixtures/uk_target_reference_feed_rows.jsonl``, plus -#: ``source_row``, which only the far larger pinned US feed exercises. The -#: version numbers are the observed ones and differ by family — ``fact`` and -#: ``source_cell`` are still v1 while the rest are v2 — which is why epoch -#: detection is structural and never a lookup here. Recorded for documentation -#: and for the frozen-history tests; a family this list has never seen still -#: resolves to its epoch. +#: Fact-key domains by family, per epoch. Nothing consults these at runtime — +#: resolution goes through the registry — but they document the table and let +#: tests pin it against the feeds this repo carries. LEDGER_FACT_KEY_DOMAINS: Mapping[str, str] = { - "aggregate_fact": "ledger.aggregate_fact.v2", - "semantic_fact": "ledger.semantic_fact.v2", - # Carried by ``legacy_fact_key``, not by a ``fact_key`` field. - "fact": "ledger.fact.v1", - "source_cell": "ledger.source_cell.v1", - "source_row": "ledger.source_row.v1", - "dimension_set": "ledger.dimension_set.v2", - "concept_alignment": "ledger.concept_alignment.v2", - "observed_measure": "ledger.observed_measure.v2", - "source_release": "ledger.source_release.v2", - "source_series": "ledger.source_series.v2", - "universe_constraint_set": "ledger.universe_constraint_set.v2", + declaration.family: declaration.identity + for declaration in DECLARED_IDENTITIES + if declaration.kind == FACT_KEY_IDENTITY_KIND and declaration.epoch == LEDGER_EPOCH +} +CHRONICLE_FACT_KEY_DOMAINS: Mapping[str, str] = { + declaration.family: declaration.identity + for declaration in DECLARED_IDENTITIES + if declaration.kind == FACT_KEY_IDENTITY_KIND + and declaration.epoch == CHRONICLE_EPOCH } #: Consumer-fact row fields that carry a single Chronicle key. The first four @@ -164,17 +347,22 @@ ("concept_alignment", "concept_alignment_key"), ) -_VERSION_PATTERN = re.compile(r"^v\d+$") - @dataclass(frozen=True) class FactKeyIdentity: """The structure of a Chronicle-shaped key, ``:``. - ``epoch`` is ``"ledger"`` or ``"chronicle"`` for Chronicle-issued keys and - ``None`` for keys minted in some other namespace — Microcosm's own - ``microcosm.derived_fact.*`` and ``populace_us_trade.*`` derived keys, for - instance, which are deliberately outside both eras. + ``epoch`` is the *declared* epoch — ``"ledger"`` or ``"chronicle"`` — and + is ``None`` for anything :data:`DECLARED_IDENTITY_REGISTRY` does not carry. + ``namespace_epoch`` is the weaker, structural answer: the epoch the key's + namespace belongs to, whatever family or version follows it. The two + differ exactly for a key in a Chronicle namespace whose spelling was never + declared, which :attr:`epoch_label` reports as :data:`UNDECLARED` rather + than promoting it to issued identity. + + Both are ``None`` for keys minted in some other namespace — Microcosm's + own ``microcosm.derived_fact.*`` and ``populace_us_trade.*`` derived keys, + for instance, which are deliberately outside both eras. """ domain: str @@ -183,15 +371,29 @@ class FactKeyIdentity: version: str digest: str epoch: str | None + namespace_epoch: str | None + + @property + def declared(self) -> bool: + """True when this exact spelling is in the declaration table.""" + return self.epoch is not None + + @property + def epoch_label(self) -> str | None: + """The declared epoch, :data:`UNDECLARED`, or ``None`` if foreign.""" + if self.epoch is not None: + return self.epoch + return UNDECLARED if self.namespace_epoch is not None else None def parse_fact_key(key: object) -> FactKeyIdentity | None: """Split ``..v:``; ``None`` if not that shape. Parsing is deliberately shallow. Microcosm does not recompute Chronicle - digests, so it needs only enough structure to answer "which epoch issued - this key?" — and it must answer that without a frozen list of domain - strings, so a chronicle-era family this code has never seen still resolves. + digests, so it needs only enough structure to look the spelling up in + :data:`DECLARED_IDENTITY_REGISTRY` — and, when the lookup misses, to say + whether the miss was in a Chronicle namespace (undeclared, worth + reporting) or in some other namespace entirely (not Chronicle's at all). """ if not isinstance(key, str) or not key: return None @@ -208,105 +410,170 @@ def parse_fact_key(key: object) -> FactKeyIdentity | None: family = ".".join(segments[1:-1]) if not namespace or not family: return None + declaration = DECLARED_IDENTITY_REGISTRY.get((namespace, family, version)) + if declaration is not None and declaration.kind != FACT_KEY_IDENTITY_KIND: + # A manifest or per-row schema id is a declared identity, but it is + # not a fact-key domain: keys never carry it. + declaration = None return FactKeyIdentity( domain=domain, namespace=namespace, family=family, version=version, digest=digest, - epoch=namespace if namespace in EPOCHS else None, + epoch=None if declaration is None else declaration.epoch, + namespace_epoch=CHRONICLE_NAMESPACES.get(namespace), ) +def declared_identity(identity: object) -> DeclaredIdentity | None: + """The declaration for an identity string, or ``None`` if undeclared.""" + if not isinstance(identity, str) or not identity: + return None + namespace, _, remainder = identity.partition(".") + family, _, version = remainder.rpartition(".") + if not namespace or not family: + return None + return DECLARED_IDENTITY_REGISTRY.get((namespace, family, version)) + + def fact_key_epoch(key: object) -> str | None: - """The epoch that issued ``key``, or ``None`` for a foreign namespace.""" + """The epoch that *declared* ``key``'s domain, or ``None``. + + ``None`` covers both a foreign namespace and a Chronicle namespace whose + exact spelling was never declared; :func:`fact_key_epoch_label` + distinguishes them. + """ identity = parse_fact_key(key) return None if identity is None else identity.epoch +def fact_key_epoch_label(key: object) -> str | None: + """``"ledger"``, ``"chronicle"``, :data:`UNDECLARED`, or ``None``.""" + identity = parse_fact_key(key) + return None if identity is None else identity.epoch_label + + def is_chronicle_fact_key(key: object) -> bool: - """True when ``key`` is a Chronicle-issued key of *either* epoch.""" + """True when ``key`` is a *declared* Chronicle key of either epoch. + + An undeclared ``chronicle.*`` spelling is deliberately false here: it may + turn out to be Chronicle's, but this build has not been told so, and a + validator that answered yes would be treating a guess as identity. + """ return fact_key_epoch(key) is not None def row_fact_key_epochs(row: object) -> frozenset[str]: - """Every Chronicle epoch appearing in one consumer-fact row's keys. + """Every epoch label appearing in one consumer-fact row's Chronicle keys. - Empty for a row whose keys are all Microcosm-minted: those namespaces are - outside both eras by design. + Contains :data:`UNDECLARED` when the row carries a key in a Chronicle + namespace whose spelling this consumer does not declare. Empty for a row + whose keys are all Microcosm-minted: those namespaces are outside both + eras by design. """ - if not isinstance(row, Mapping): - return frozenset() - epochs: set[str] = set() - for field in FACT_KEY_FIELDS: - epoch = fact_key_epoch(row.get(field)) - if epoch is not None: - epochs.add(epoch) - for path in _FACT_KEY_NESTED_PATHS: - epoch = fact_key_epoch(_at(row, path)) - if epoch is not None: - epochs.add(epoch) - for path in _FACT_KEY_LIST_PATHS: - values = _at(row, path) - if isinstance(values, Iterable) and not isinstance(values, str | bytes): - for value in values: - epoch = fact_key_epoch(value) - if epoch is not None: - epochs.add(epoch) - return frozenset(epochs) + return frozenset(label for label, _ in _row_labelled_keys(row)) + + +def feed_undeclared_fact_key_domains(rows: Iterable[Any]) -> tuple[str, ...]: + """Sorted Chronicle-namespace key domains the declaration table lacks. + + Reported alongside the epochs so a release manifest names *what* it did + not recognise, rather than only that something went unrecognised. + """ + undeclared: set[str] = set() + for row in rows: + for label, domain in _row_labelled_keys(row): + if label == UNDECLARED: + undeclared.add(domain) + return tuple(sorted(undeclared)) def feed_fact_key_epochs(rows: Iterable[Any]) -> tuple[str, ...]: - """Chronicle epochs observed across a whole feed, in :data:`EPOCHS` order. + """Epoch labels observed across a whole feed, in report order. A mixed-epoch feed — ledger-era history beside chronicle-era rows — is - expected during the cutover window and is reported, not rejected. + expected during the cutover window and is reported, not rejected. So is + an undeclared spelling: it appears as :data:`UNDECLARED`, never folded + into either era. """ observed: set[str] = set() for row in rows: observed |= row_fact_key_epochs(row) - return tuple(epoch for epoch in EPOCHS if epoch in observed) + return tuple(label for label in FACT_KEY_EPOCH_LABELS if label in observed) def is_accepted_consumer_artifact_schema_version(value: object) -> bool: - """True for the consumer-artifact manifest schema id of either epoch.""" - return value in ACCEPTED_CONSUMER_ARTIFACT_SCHEMA_VERSIONS + """True for a declared consumer-artifact manifest schema id, either epoch. + + Non-string JSON — a list or object where a schema id belongs — is simply + not accepted. It must not raise here: the loader turns a rejection into + the documented unsupported-schema ``ValueError``, and a ``TypeError`` + escaping a membership test would bypass that message entirely. + """ + return ( + isinstance(value, str) and value in ACCEPTED_CONSUMER_ARTIFACT_SCHEMA_VERSIONS + ) def is_accepted_consumer_fact_schema_version(value: object) -> bool: - """True for the per-row consumer-fact schema id of either epoch.""" - return value in ACCEPTED_CONSUMER_FACT_SCHEMA_VERSIONS + """True for a declared per-row consumer-fact schema id, either epoch.""" + return isinstance(value, str) and value in ACCEPTED_CONSUMER_FACT_SCHEMA_VERSIONS def consumer_artifact_schema_epoch(value: object) -> str | None: - """The epoch of a consumer-artifact schema id, or ``None`` if unknown.""" - return _epoch_of(value, CONSUMER_ARTIFACT_SCHEMA_VERSION_BY_EPOCH) + """The epoch of a consumer-artifact schema id, or ``None`` if undeclared.""" + return _schema_epoch(value, CONSUMER_ARTIFACT_SCHEMA_IDENTITY_KIND) def consumer_fact_schema_epoch(value: object) -> str | None: """The epoch of a per-row consumer-fact schema id, or ``None``.""" - return _epoch_of(value, CONSUMER_FACT_SCHEMA_VERSION_BY_EPOCH) + return _schema_epoch(value, CONSUMER_FACT_SCHEMA_IDENTITY_KIND) def describe_accepted_consumer_artifact_schema_versions() -> str: - """Both accepted artifact schema ids, oldest era first, for messages.""" - return _describe(CONSUMER_ARTIFACT_SCHEMA_VERSION_BY_EPOCH) + """Every accepted artifact schema id, oldest era first, for messages.""" + return _describe(CONSUMER_ARTIFACT_SCHEMA_IDENTITY_KIND) def describe_accepted_consumer_fact_schema_versions() -> str: - """Both accepted per-row fact schema ids, oldest era first, for messages.""" - return _describe(CONSUMER_FACT_SCHEMA_VERSION_BY_EPOCH) + """Every accepted per-row fact schema id, oldest era first, for messages.""" + return _describe(CONSUMER_FACT_SCHEMA_IDENTITY_KIND) + + +def _describe(kind: str) -> str: + return ", ".join(repr(identity) for identity in _identities_of(kind)) + + +def _schema_epoch(value: object, kind: str) -> str | None: + declaration = declared_identity(value) + if declaration is None or declaration.kind != kind: + return None + return declaration.epoch -def _describe(by_epoch: Mapping[str, str]) -> str: - return ", ".join(repr(by_epoch[epoch]) for epoch in EPOCHS) +def _row_labelled_keys(row: object) -> tuple[tuple[str, str], ...]: + """``(epoch label, key domain)`` for every Chronicle key in one row.""" + if not isinstance(row, Mapping): + return () + labelled: list[tuple[str, str]] = [] + def record(value: object) -> None: + identity = parse_fact_key(value) + if identity is None or identity.epoch_label is None: + return + labelled.append((identity.epoch_label, identity.domain)) -def _epoch_of(value: object, by_epoch: Mapping[str, str]) -> str | None: - for epoch in EPOCHS: - if value == by_epoch[epoch]: - return epoch - return None + for field in FACT_KEY_FIELDS: + record(row.get(field)) + for path in _FACT_KEY_NESTED_PATHS: + record(_at(row, path)) + for path in _FACT_KEY_LIST_PATHS: + values = _at(row, path) + if isinstance(values, Iterable) and not isinstance(values, str | bytes): + for value in values: + record(value) + return tuple(labelled) def _at(row: Mapping[str, Any], path: tuple[str, ...]) -> Any: diff --git a/packages/microcosm-build/src/microcosm/build/ledger_artifact.py b/packages/microcosm-build/src/microcosm/build/ledger_artifact.py index 6a08f219a..815da04a7 100644 --- a/packages/microcosm-build/src/microcosm/build/ledger_artifact.py +++ b/packages/microcosm-build/src/microcosm/build/ledger_artifact.py @@ -12,21 +12,28 @@ against the published artifact contract (stdlib only); it does not import the Chronicle implementation package. -**Both eras load.** The manifest's ``schema_version`` is checked for -membership in :data:`ACCEPTED_CONSUMER_ARTIFACT_SCHEMA_VERSIONS`, never for -equality with one era, so an artifact published after Chronicle's rename +**Every declared era loads.** The manifest's ``schema_version`` is checked +for membership in :data:`ACCEPTED_CONSUMER_ARTIFACT_SCHEMA_VERSIONS`, never +for equality with one era, so an artifact published after Chronicle's rename cutover loads here without a code change — see -:mod:`microcosm.build.chronicle_epoch` and PolicyEngine/chronicle#143. The -observed id, its epoch, the fact-key epochs present in the feed, and the +:mod:`microcosm.build.chronicle_epoch` and PolicyEngine/chronicle#143. Three +spellings are accepted, not two: Microcosm's own minted +``policyengine_ledger.consumer_artifact.v1``, the +``policyengine_ledger.consumer_artifact.v2`` Chronicle's ``main`` emits +today, and the chronicle-era +``policyengine_chronicle.consumer_artifact.v3`` successor. + +The observed id, its epoch, the fact-key epochs present in the feed, any +Chronicle-namespace key domain the declaration table does not carry, and the per-row schema ids the rows themselves declare are all recorded in :meth:`LedgerConsumerArtifact.provenance`, so a release manifest witnesses which era it actually consumed rather than which era it assumed. Acceptance widened here; nothing narrowed. Only the *manifest* schema id is -gated, and only against the two ids chronicle#143 declares. Row-level ids and -fact keys are carried as published — real feeds mint rows in namespaces that -belong to neither era, and a consumer that rejected them would fail closed on -data that has always loaded. +gated, and only against the declared ids. Row-level ids and fact keys are +carried as published — real feeds mint rows in namespaces that belong to +neither era, and a consumer that rejected them would fail closed on data that +has always loaded. """ from __future__ import annotations @@ -40,10 +47,11 @@ from microcosm.build.chronicle_epoch import ( ACCEPTED_CONSUMER_ARTIFACT_SCHEMA_VERSIONS, - LEDGER_CONSUMER_ARTIFACT_SCHEMA_VERSION, + MICROCOSM_CONSUMER_ARTIFACT_SCHEMA_VERSION, consumer_artifact_schema_epoch, describe_accepted_consumer_artifact_schema_versions, feed_fact_key_epochs, + feed_undeclared_fact_key_domains, is_accepted_consumer_artifact_schema_version, ) @@ -58,9 +66,11 @@ "resolve_ledger_artifact", ] -#: The era Microcosm's own minted artifacts still declare. Frozen at v1 by -#: microcosm#639; loading is governed by the accepted *set*, not by this. -CONSUMER_ARTIFACT_SCHEMA_VERSION = LEDGER_CONSUMER_ARTIFACT_SCHEMA_VERSION +#: The id Microcosm's own minted artifacts still declare. Frozen at v1 by +#: microcosm#639; loading is governed by the accepted *set*, not by this. It +#: is deliberately *not* the id Chronicle emits — Chronicle's ``main`` stamps +#: ``policyengine_ledger.consumer_artifact.v2``, which the set also accepts. +CONSUMER_ARTIFACT_SCHEMA_VERSION = MICROCOSM_CONSUMER_ARTIFACT_SCHEMA_VERSION ALLOWED_LEDGER_ASSERTIONS = frozenset(("observation", "source_projection")) DEFAULT_LEDGER_ASSERTION = "observation" @@ -108,14 +118,26 @@ def schema_epoch(self) -> str | None: @property def fact_key_epochs(self) -> tuple[str, ...]: - """Chronicle epochs observed across the feed's fact keys. + """Epoch labels observed across the feed's Chronicle fact keys. Empty when the feed carries only Microcosm-minted keys, and both epochs when a cutover-window feed mixes ledger-era history with - chronicle-era rows. + chronicle-era rows. A Chronicle-namespace key whose exact spelling is + undeclared appears as ``"undeclared"`` — reported, never folded into + an era it was not declared for. """ return feed_fact_key_epochs(self.facts) + @property + def undeclared_fact_key_domains(self) -> tuple[str, ...]: + """Chronicle-namespace key domains the declaration table lacks. + + Normally empty. A non-empty tuple means the feed carried a spelling + this build was not told about, and names it, so the manifest says + *what* went unrecognised instead of only that something did. + """ + return feed_undeclared_fact_key_domains(self.facts) + @property def fact_schema_versions(self) -> tuple[str, ...]: """Distinct per-row ``schema_version`` values in the feed, sorted. @@ -134,9 +156,11 @@ def fact_schema_versions(self) -> tuple[str, ...]: def provenance(self) -> dict[str, Any]: """Chronicle-artifact identity block for build and release manifests. - Records the schema id as *observed*, plus the epoch it belongs to and - the epochs of the fact keys in the feed, so a manifest witnesses which - era of Chronicle actually resolved its targets. + Records the schema id as *observed*, plus the epoch it belongs to, + the epoch labels of the fact keys in the feed, and any undeclared + Chronicle-namespace domain among them, so a manifest witnesses which + era of Chronicle actually resolved its targets — and says so when the + feed carried a spelling this build did not declare. """ payload: dict[str, Any] = { "path_name": self.path.name, @@ -160,6 +184,7 @@ def provenance(self) -> dict[str, Any]: payload["manifest_sha256"] = None payload["schema_epoch"] = self.schema_epoch payload["fact_key_epochs"] = list(self.fact_key_epochs) + payload["undeclared_fact_key_domains"] = list(self.undeclared_fact_key_domains) payload["fact_schema_versions"] = list(self.fact_schema_versions) return payload @@ -203,8 +228,11 @@ def load_ledger_consumer_artifact( ) schema_version = manifest.get("schema_version") # Membership, not equality: ledger-era and chronicle-era artifacts are - # the same contract under two names, and both must load through the - # rename cutover (chronicle#143). + # the same contract under different names, and all of them must load + # through the rename cutover (chronicle#143). The predicate is total + # over JSON — a manifest whose schema_version is a list or an object + # is unsupported, and must say so with this message rather than + # escaping as a TypeError from a membership test. if not is_accepted_consumer_artifact_schema_version(schema_version): raise ValueError( "Unsupported Chronicle consumer artifact schema_version " diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/us_trade/import_entry_facts.py b/packages/microcosm-build/src/microcosm/build/us_runtime/us_trade/import_entry_facts.py index 2e84e05d6..4851851a0 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/us_trade/import_entry_facts.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/us_trade/import_entry_facts.py @@ -50,8 +50,8 @@ from microcosm.build.chronicle_epoch import ( ACCEPTED_CONSUMER_ARTIFACT_SCHEMA_VERSIONS, - LEDGER_CONSUMER_ARTIFACT_SCHEMA_VERSION, LEDGER_CONSUMER_FACT_SCHEMA_VERSION, + MICROCOSM_CONSUMER_ARTIFACT_SCHEMA_VERSION, describe_accepted_consumer_artifact_schema_versions, is_accepted_consumer_artifact_schema_version, ) @@ -161,7 +161,7 @@ class FactSourceLeg: #: whose epoch they could inherit. Chronicle's own cutover moves what Chronicle #: emits, not what Microcosm mints. Acceptance is the separate, dual-era #: question, and it lives in :mod:`microcosm.build.chronicle_epoch`. -CONSUMER_ARTIFACT_SCHEMA_VERSION = LEDGER_CONSUMER_ARTIFACT_SCHEMA_VERSION +CONSUMER_ARTIFACT_SCHEMA_VERSION = MICROCOSM_CONSUMER_ARTIFACT_SCHEMA_VERSION _FACT_SCHEMA_VERSION = LEDGER_CONSUMER_FACT_SCHEMA_VERSION #: Microcosm's own key namespace: outside both Chronicle eras by construction, #: so these keys can never collide with or impersonate Chronicle-built keys — @@ -618,8 +618,8 @@ def write_consumer_artifact( which re-hashes the fact file against ``manifest.facts_sha256``. ``schema_version`` defaults to the ledger-era id these artifacts have - always declared and is validated for membership in both eras, so the - chronicle-era cutover is a caller decision rather than an edit here. + always declared and is validated for membership in the declared set, so + the chronicle-era cutover is a caller decision rather than an edit here. """ if not fact_rows: raise ValueError("Refusing to write an empty consumer artifact.") diff --git a/packages/microcosm-build/tests/test_chronicle_epoch.py b/packages/microcosm-build/tests/test_chronicle_epoch.py index f7cdca835..9f1fd4c0e 100644 --- a/packages/microcosm-build/tests/test_chronicle_epoch.py +++ b/packages/microcosm-build/tests/test_chronicle_epoch.py @@ -6,6 +6,12 @@ these tests are the contract that says so: no validator may reject a row for being on the other side of the cutover, and no minted Microcosm identity may move because a source row crossed it. + +The other half of the contract is that acceptance follows a *declaration*. +An identity Microcosm has not been told about is reported as undeclared, not +waved through as Chronicle's — a consumer that inferred issued identity from +the shape of a string would accept anything that happened to be spelled like +a Chronicle key. """ from __future__ import annotations @@ -23,15 +29,26 @@ CHRONICLE_CONSUMER_ARTIFACT_SCHEMA_VERSION, CHRONICLE_CONSUMER_FACT_SCHEMA_VERSION, CHRONICLE_EPOCH, + CHRONICLE_FACT_KEY_DOMAINS, + CHRONICLE_NAMESPACES, + DECLARED_IDENTITIES, + DECLARED_IDENTITY_EPOCHS, EPOCHS, FACT_KEY_FIELDS, - LEDGER_CONSUMER_ARTIFACT_SCHEMA_VERSION, + FACT_KEY_IDENTITY_KIND, LEDGER_CONSUMER_FACT_SCHEMA_VERSION, LEDGER_EPOCH, LEDGER_FACT_KEY_DOMAINS, + MICROCOSM_CONSUMER_ARTIFACT_SCHEMA_VERSION, + PUBLISHED_CONSUMER_ARTIFACT_SCHEMA_VERSION, + UNDECLARED, consumer_artifact_schema_epoch, + consumer_fact_schema_epoch, + declared_identity, fact_key_epoch, + fact_key_epoch_label, feed_fact_key_epochs, + feed_undeclared_fact_key_domains, is_accepted_consumer_artifact_schema_version, is_accepted_consumer_fact_schema_version, is_chronicle_fact_key, @@ -45,6 +62,15 @@ _CHRONICLE_AGGREGATE_KEY = "chronicle.aggregate_fact.v3:abc123" _LEDGER_AGGREGATE_KEY = "ledger.aggregate_fact.v2:abc123" +#: Every consumer-artifact manifest id a real producer stamps. Microcosm's +#: own minted v1 and Chronicle's published v2 are both ledger-era; v3 is the +#: chronicle-era successor. +_ARTIFACT_SCHEMA_IDS_BY_EPOCH = ( + (MICROCOSM_CONSUMER_ARTIFACT_SCHEMA_VERSION, LEDGER_EPOCH), + (PUBLISHED_CONSUMER_ARTIFACT_SCHEMA_VERSION, LEDGER_EPOCH), + (CHRONICLE_CONSUMER_ARTIFACT_SCHEMA_VERSION, CHRONICLE_EPOCH), +) + def _fact_row(**overrides): row = { @@ -67,13 +93,20 @@ def _fact_row(**overrides): def _chronicle_fact_row(**overrides): - """The same row after Chronicle's cutover: every key domain re-epoched.""" + """The same row after Chronicle's cutover: every key domain re-epoched. + + Each successor spelling comes from the declaration table rather than + being written out here, so a row this fixture builds is one the loader + recognises as chronicle-era rather than one it reports as undeclared. + """ row = _fact_row( aggregate_fact_key=_CHRONICLE_AGGREGATE_KEY, semantic_fact_key="chronicle.semantic_fact.v3:abc123", ) row["lineage"] = dict(row["lineage"]) - row["lineage"]["source_cell_keys"] = ["chronicle.source_cell.v3:cell"] + row["lineage"]["source_cell_keys"] = [ + CHRONICLE_FACT_KEY_DOMAINS["source_cell"] + ":cell" + ] row.update(overrides) return row @@ -141,14 +174,15 @@ def test_every_observed_ledger_era_domain_resolves_to_the_ledger_epoch( def test_observed_domain_map_covers_the_captured_feed() -> None: - """The map documents what the repo has actually seen, not a guess. - - ``LEDGER_FACT_KEY_DOMAINS`` claims to record the ledger-era families this - repo's feeds carry, and nothing consults it at runtime — which is exactly - how such a list rots. This pins it to the captured feed. The version - numbers differ by family (``fact`` and ``source_cell`` are v1 where the - rest are v2), which is the concrete reason epoch detection cannot be a - lookup in this map. + """Every domain a real feed carries is one the registry declares. + + ``LEDGER_FACT_KEY_DOMAINS`` is the ledger-era half of the declaration + table, transcribed from Chronicle's own hash-domain constants. This pins + it against the captured feed: a domain the repo has actually seen and the + table has not would make that feed load with an ``undeclared`` witness, + which is a review event, not a silent pass. The version numbers differ by + family (``fact`` and ``source_cell`` are v1 where the rest are v2), which + is the concrete reason a version can never be inferred from a family. """ observed = { key.split(":", 1)[0] for row in _captured_feed_rows() for key in _keys_in(row) @@ -160,23 +194,117 @@ def test_observed_domain_map_covers_the_captured_feed() -> None: ) -def test_chronicle_era_keys_resolve_without_a_declared_version_number() -> None: - """Epoch detection is structural, so undeclared families still resolve. +@pytest.mark.parametrize("domain", sorted(CHRONICLE_FACT_KEY_DOMAINS.values())) +def test_every_declared_chronicle_era_domain_resolves_to_the_chronicle_epoch( + domain: str, +) -> None: + assert fact_key_epoch(f"{domain}:digest") == CHRONICLE_EPOCH + assert fact_key_epoch_label(f"{domain}:digest") == CHRONICLE_EPOCH + assert is_chronicle_fact_key(f"{domain}:digest") + + +def test_each_declared_family_pairs_one_ledger_id_with_one_chronicle_id() -> None: + """The two eras name the same families, one version apart. - chronicle#143 names the ``v3`` spelling for the aggregate and semantic - fact families; it does not say which version number the remaining - families take. A validator that answered "unknown" for those would - fail closed on the cutover, so detection reads the namespace segment - rather than matching a frozen list of domain strings. + Successor spellings are declared, so this pins the shape of the + declaration rather than deriving it at call time: the source-side + families sit at v1/v2 where the derived ones sit at v2/v3, which is + precisely why a version can never be assumed from a family name. """ - for key in ( - _CHRONICLE_AGGREGATE_KEY, - "chronicle.semantic_fact.v3:abc123", + assert set(LEDGER_FACT_KEY_DOMAINS) == set(CHRONICLE_FACT_KEY_DOMAINS) + for family, ledger_domain in LEDGER_FACT_KEY_DOMAINS.items(): + chronicle_domain = CHRONICLE_FACT_KEY_DOMAINS[family] + ledger_version = int(ledger_domain.rsplit(".v", 1)[1]) + chronicle_version = int(chronicle_domain.rsplit(".v", 1)[1]) + assert chronicle_domain == f"chronicle.{family}.v{chronicle_version}" + assert chronicle_version == ledger_version + 1, family + assert LEDGER_FACT_KEY_DOMAINS["source_cell"] == "ledger.source_cell.v1" + assert CHRONICLE_FACT_KEY_DOMAINS["source_cell"] == "chronicle.source_cell.v2" + assert LEDGER_FACT_KEY_DOMAINS["aggregate_fact"] == "ledger.aggregate_fact.v2" + assert CHRONICLE_FACT_KEY_DOMAINS["aggregate_fact"] == ( + "chronicle.aggregate_fact.v3" + ) + + +@pytest.mark.parametrize( + "key", + [ + # source_cell's declared successor is v2; v3 is nobody's spelling. + "chronicle.source_cell.v3:cell", "chronicle.source_cell.v7:cell", "chronicle.some_family_nobody_has_declared_yet.v11:digest", - ): - assert fact_key_epoch(key) == CHRONICLE_EPOCH - assert is_chronicle_fact_key(key) + "chronicle.aggregate_fact.v4:digest", + # A ledger-era namespace can go undeclared the same way. + "ledger.aggregate_fact.v9:digest", + "ledger.invented_family.v1:digest", + # Declared, but as a *schema* id — never as a fact-key domain. + "ledger.consumer_fact.v1:digest", + ], +) +def test_undeclared_chronicle_namespace_keys_are_reported_not_issued(key: str) -> None: + """An undeclared spelling is undeclared, not Chronicle-issued identity. + + Reading the namespace segment tells you where a key *claims* to come + from, not that Chronicle issued it. Treating the claim as identity would + let ``chronicle.anything.vN`` pass as a witnessed Chronicle key, which is + the opposite of what pinning a fact key is for. The claim is still worth + surfacing, so it is labelled ``undeclared`` and reported in provenance. + """ + assert fact_key_epoch(key) is None + assert not is_chronicle_fact_key(key) + assert fact_key_epoch_label(key) == UNDECLARED + + identity = parse_fact_key(key) + assert identity is not None + assert not identity.declared + assert identity.namespace_epoch == CHRONICLE_NAMESPACES[identity.namespace] + + +def test_undeclared_domains_are_named_in_the_feed_report() -> None: + row = _fact_row(semantic_fact_key="chronicle.semantic_fact.v9:abc123") + + assert row_fact_key_epochs(row) == frozenset({LEDGER_EPOCH, UNDECLARED}) + assert feed_fact_key_epochs([row]) == (LEDGER_EPOCH, UNDECLARED) + assert feed_undeclared_fact_key_domains([row]) == ("chronicle.semantic_fact.v9",) + assert feed_undeclared_fact_key_domains([_fact_row()]) == () + + +def test_undeclared_domains_reach_the_artifact_provenance(tmp_path) -> None: + """A release manifest has to name what the build did not recognise.""" + rows = [_fact_row(), _fact_row(source_series_key="chronicle.source_series.v9:s")] + artifact_dir = _write_artifact_dir( + tmp_path, rows, schema_version=PUBLISHED_CONSUMER_ARTIFACT_SCHEMA_VERSION + ) + + provenance = load_ledger_consumer_artifact(artifact_dir).provenance() + + assert provenance["fact_key_epochs"] == [LEDGER_EPOCH, UNDECLARED] + assert provenance["undeclared_fact_key_domains"] == ["chronicle.source_series.v9"] + + +def test_the_registry_is_keyed_by_namespace_family_and_version() -> None: + """Every accepted identity comes from one reviewable table. + + The registry, not a parser, is what makes an identity Chronicle's. This + walks it end to end: each declaration round-trips through its + ``(namespace, family, version)`` key, and the epoch a namespace belongs + to always agrees with the epoch the declaration claims. + """ + assert DECLARED_IDENTITIES + for declaration in DECLARED_IDENTITIES: + key = (declaration.namespace, declaration.family, declaration.version) + assert DECLARED_IDENTITY_EPOCHS[key] == declaration.epoch + assert declared_identity(declaration.identity) is declaration + assert CHRONICLE_NAMESPACES[declaration.namespace] == declaration.epoch + assert declaration.epoch in EPOCHS + if declaration.kind == FACT_KEY_IDENTITY_KIND: + assert fact_key_epoch(f"{declaration.identity}:digest") == ( + declaration.epoch + ) + + assert declared_identity("arch.consumer_fact.v1") is None + assert declared_identity("not-an-identity") is None + assert declared_identity(None) is None def test_microcosm_minted_namespaces_are_outside_both_epochs() -> None: @@ -224,7 +352,9 @@ def test_row_epochs_cover_every_key_field_including_nested_lists() -> None: # reports both eras: lineage keys are Chronicle identities too. straddling = _fact_row() straddling["lineage"] = dict(straddling["lineage"]) - straddling["lineage"]["source_cell_keys"] = ["chronicle.source_cell.v3:cell"] + straddling["lineage"]["source_cell_keys"] = [ + CHRONICLE_FACT_KEY_DOMAINS["source_cell"] + ":cell" + ] assert row_fact_key_epochs(straddling) == frozenset({LEDGER_EPOCH, CHRONICLE_EPOCH}) @@ -249,33 +379,88 @@ def test_feed_epochs_report_a_mixed_feed_in_epoch_order() -> None: assert feed_fact_key_epochs([]) == () -def test_schema_id_membership_covers_both_eras_and_nothing_else() -> None: +def test_schema_id_membership_covers_every_real_producer_id() -> None: + """Acceptance is keyed to what producers actually stamp, not to a pair. + + Three artifact ids exist, not two. Microcosm's own minted artifacts + declare ``policyengine_ledger.consumer_artifact.v1``; Chronicle's ``main`` + stamps ``policyengine_ledger.consumer_artifact.v2`` today + (``policyengine_chronicle/consumer.py``); the chronicle-era successor is + ``policyengine_chronicle.consumer_artifact.v3``. A loader that accepted + only the first and the last would reject every artifact Chronicle + publishes right now. + """ assert ACCEPTED_CONSUMER_ARTIFACT_SCHEMA_VERSIONS == { - LEDGER_CONSUMER_ARTIFACT_SCHEMA_VERSION, + MICROCOSM_CONSUMER_ARTIFACT_SCHEMA_VERSION, + PUBLISHED_CONSUMER_ARTIFACT_SCHEMA_VERSION, CHRONICLE_CONSUMER_ARTIFACT_SCHEMA_VERSION, } assert ACCEPTED_CONSUMER_FACT_SCHEMA_VERSIONS == { LEDGER_CONSUMER_FACT_SCHEMA_VERSION, CHRONICLE_CONSUMER_FACT_SCHEMA_VERSION, } - assert is_accepted_consumer_artifact_schema_version( - CHRONICLE_CONSUMER_ARTIFACT_SCHEMA_VERSION + for schema_id, expected_epoch in _ARTIFACT_SCHEMA_IDS_BY_EPOCH: + assert is_accepted_consumer_artifact_schema_version(schema_id), schema_id + assert consumer_artifact_schema_epoch(schema_id) == expected_epoch, schema_id + assert consumer_fact_schema_epoch(LEDGER_CONSUMER_FACT_SCHEMA_VERSION) == ( + LEDGER_EPOCH ) - assert is_accepted_consumer_fact_schema_version( - CHRONICLE_CONSUMER_FACT_SCHEMA_VERSION + assert consumer_fact_schema_epoch(CHRONICLE_CONSUMER_FACT_SCHEMA_VERSION) == ( + CHRONICLE_EPOCH ) assert not is_accepted_consumer_artifact_schema_version( "policyengine_chronicle.other.v9" ) + # Version numbers are declared, not inferred: the chronicle-era per-row id + # is v2, so v1 under the chronicle namespace is nobody's id. assert not is_accepted_consumer_fact_schema_version("chronicle.consumer_fact.v1") + assert consumer_fact_schema_epoch("chronicle.consumer_fact.v1") is None + # A fact-key domain is a declared identity but not a schema id, and the + # kinds do not leak into one another. + assert not is_accepted_consumer_fact_schema_version("ledger.aggregate_fact.v2") + assert consumer_artifact_schema_epoch(LEDGER_CONSUMER_FACT_SCHEMA_VERSION) is None + + +def test_chronicle_main_manifest_loads_exactly_as_published(tmp_path) -> None: + """The manifest chronicle ``main`` writes today, field for field. + + Mirrors ``build_consumer_artifact`` in ``policyengine_chronicle/consumer.py``: + ``policyengine_ledger.consumer_artifact.v2`` over rows stamped + ``ledger.consumer_fact.v1``, with ``consumer_fact_schema_versions`` and + ``consumer_fact_schema_sha256`` beside the hashes. This is the artifact a + Microcosm build is handed right now, so it is the one that has to load. + """ + rows = [_fact_row(schema_version=LEDGER_CONSUMER_FACT_SCHEMA_VERSION)] + payload = "".join(json.dumps(row, sort_keys=True) + "\n" for row in rows) + artifact_dir = tmp_path / "chronicle-main" + artifact_dir.mkdir() + (artifact_dir / "consumer_facts.jsonl").write_text(payload) + (artifact_dir / "manifest.json").write_text( + json.dumps( + { + "schema_version": PUBLISHED_CONSUMER_ARTIFACT_SCHEMA_VERSION, + "consumer_fact_schema_versions": [LEDGER_CONSUMER_FACT_SCHEMA_VERSION], + "consumer_fact_schema_sha256": "cd" * 32, + "fact_row_count": len(rows), + "facts_sha256": hashlib.sha256(payload.encode()).hexdigest(), + }, + indent=2, + sort_keys=True, + ) + + "\n" + ) + + artifact = load_ledger_consumer_artifact(artifact_dir) + + assert artifact.schema_version == PUBLISHED_CONSUMER_ARTIFACT_SCHEMA_VERSION + assert artifact.schema_epoch == LEDGER_EPOCH + assert artifact.fact_schema_versions == (LEDGER_CONSUMER_FACT_SCHEMA_VERSION,) + assert artifact.fact_key_epochs == (LEDGER_EPOCH,) + assert artifact.undeclared_fact_key_domains == () @pytest.mark.parametrize( - ("schema_version", "expected_epoch"), - [ - (LEDGER_CONSUMER_ARTIFACT_SCHEMA_VERSION, LEDGER_EPOCH), - (CHRONICLE_CONSUMER_ARTIFACT_SCHEMA_VERSION, CHRONICLE_EPOCH), - ], + ("schema_version", "expected_epoch"), _ARTIFACT_SCHEMA_IDS_BY_EPOCH ) def test_artifact_loads_under_each_schema_id_and_records_the_observed_one( tmp_path, schema_version: str, expected_epoch: str @@ -346,7 +531,7 @@ def test_bare_feed_has_no_schema_epoch_but_still_reports_fact_key_epochs( assert provenance["fact_key_epochs"] == list(EPOCHS) -def test_unknown_schema_id_is_rejected_naming_both_accepted_eras(tmp_path) -> None: +def test_unknown_schema_id_is_rejected_naming_every_accepted_id(tmp_path) -> None: artifact_dir = _write_artifact_dir( tmp_path, [_fact_row()], schema_version="policyengine_chronicle.other.v9" ) @@ -356,8 +541,49 @@ def test_unknown_schema_id_is_rejected_naming_both_accepted_eras(tmp_path) -> No message = str(excinfo.value) assert "schema_version" in message - assert LEDGER_CONSUMER_ARTIFACT_SCHEMA_VERSION in message - assert CHRONICLE_CONSUMER_ARTIFACT_SCHEMA_VERSION in message + for schema_id, _epoch in _ARTIFACT_SCHEMA_IDS_BY_EPOCH: + assert schema_id in message, schema_id + + +@pytest.mark.parametrize( + "schema_version", + [ + ["policyengine_ledger.consumer_artifact.v1"], + {"id": "policyengine_ledger.consumer_artifact.v1"}, + {}, + [], + 1, + True, + None, + ], +) +def test_malformed_schema_version_types_raise_the_documented_error( + tmp_path, schema_version +) -> None: + """A JSON list or object where a schema id belongs is *unsupported*. + + ``schema_version`` arrives from an untrusted manifest, so its type is not + guaranteed. A membership test against a frozenset raises ``TypeError`` on + an unhashable value, which would escape the loader as a bare + ``unhashable type: 'list'`` instead of the message that names what the + loader accepts. The predicate is total over JSON so the loader's own + error is the one an operator sees. + """ + artifact_dir = _write_artifact_dir( + tmp_path, [_fact_row()], schema_version=schema_version + ) + + assert not is_accepted_consumer_artifact_schema_version(schema_version) + assert not is_accepted_consumer_fact_schema_version(schema_version) + assert consumer_artifact_schema_epoch(schema_version) is None + assert consumer_fact_schema_epoch(schema_version) is None + + with pytest.raises(ValueError) as excinfo: + load_ledger_consumer_artifact(artifact_dir) + + assert "Unsupported Chronicle consumer artifact schema_version" in str( + excinfo.value + ) @pytest.mark.parametrize( @@ -450,7 +676,9 @@ def test_every_published_key_field_is_witnessed_for_its_epoch() -> None: row_keys = _fact_row() row_keys["lineage"] = dict(row_keys["lineage"]) - row_keys["lineage"]["source_row_keys"] = ["chronicle.source_row.v3:row"] + row_keys["lineage"]["source_row_keys"] = [ + CHRONICLE_FACT_KEY_DOMAINS["source_row"] + ":row" + ] assert row_fact_key_epochs(row_keys) == frozenset({LEDGER_EPOCH, CHRONICLE_EPOCH}) diff --git a/packages/microcosm-build/tests/test_ledger_targets.py b/packages/microcosm-build/tests/test_ledger_targets.py index 60e6e7837..cf441548d 100644 --- a/packages/microcosm-build/tests/test_ledger_targets.py +++ b/packages/microcosm-build/tests/test_ledger_targets.py @@ -2957,7 +2957,7 @@ def test__given_mixed_epoch_fact_feed__then_both_eras_compile_to_targets() -> No semantic_fact_key="chronicle.semantic_fact.v3:def456", lineage={ "source_record_id": "irs_soi.ty2024.table_1_1.all.adjusted_gross_income", - "source_cell_keys": ["chronicle.source_cell.v3:cell"], + "source_cell_keys": ["chronicle.source_cell.v2:cell"], "source_row_keys": [], }, ) diff --git a/packages/microcosm-build/tests/test_us_trade_facts.py b/packages/microcosm-build/tests/test_us_trade_facts.py index 08ab5b100..e7b233669 100644 --- a/packages/microcosm-build/tests/test_us_trade_facts.py +++ b/packages/microcosm-build/tests/test_us_trade_facts.py @@ -9,8 +9,8 @@ from microcosm.build.chronicle_epoch import ( CHRONICLE_CONSUMER_ARTIFACT_SCHEMA_VERSION, - LEDGER_CONSUMER_ARTIFACT_SCHEMA_VERSION, LEDGER_CONSUMER_FACT_SCHEMA_VERSION, + MICROCOSM_CONSUMER_ARTIFACT_SCHEMA_VERSION, ) from microcosm.build.ledger_artifact import load_ledger_consumer_artifact from microcosm.build.ledger_targets import ( @@ -841,7 +841,9 @@ def test_minted_artifact_declares_the_ledger_era_id_and_loads_under_either(tmp_p retrieval_manifest=_manifest_entries(), generator=default_generator_block(months=("2026-01", "2026-02")), ) - assert default_manifest["schema_version"] == LEDGER_CONSUMER_ARTIFACT_SCHEMA_VERSION + assert ( + default_manifest["schema_version"] == MICROCOSM_CONSUMER_ARTIFACT_SCHEMA_VERSION + ) # Every row's key stays in Microcosm's own frozen namespace, outside both # Chronicle eras, so the cutover cannot re-identify them (microcosm#639). assert all( From 05cd32e78e5bedaa03d65aea2ee9a4cf3ee72beb Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Wed, 2 Sep 2026 13:10:21 -0400 Subject: [PATCH 16/23] Cut every test in the build shard off from a live Logbook MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The per-module delenv lists named POPULACE_LEDGER_* only, because they were written before the LOGBOOK_* spelling existed. A shell exporting the preferred names — a developer who just ran an export, a runner where the variables are set for a later job — walked straight past them: logbook._remote_config() returns ('https://example.invalid', 'x', 'x') under LOGBOOK_URL/LOGBOOK_KEY alone, and the append path would then POST to it. The store is append-only, so a row that lands cannot be retracted. An autouse conftest fixture now clears both generations of every variable in the dual-read window, taken from the window itself so a variable added there cannot leave a hole here, plus POPULACE_LOGBOOK_PREV_ROW_DIGEST. The second, independent guard replaces the Logbook urlopen with one that raises — on every module currently holding it, since tools/logbook.py binds the name at import, and on the opener it delegates to, which closes the path for any importer this does not find. allow_logbook_network restores the real callables for a test that means it; nothing in the suite asks for it. Co-Authored-By: Claude Fable 5.1 --- packages/microcosm-build/tests/conftest.py | 112 +++++++++++++++++- .../tests/test_logbook_isolation.py | 94 +++++++++++++++ 2 files changed, 205 insertions(+), 1 deletion(-) create mode 100644 packages/microcosm-build/tests/test_logbook_isolation.py diff --git a/packages/microcosm-build/tests/conftest.py b/packages/microcosm-build/tests/conftest.py index ac3f6b9dc..cffd6547f 100644 --- a/packages/microcosm-build/tests/conftest.py +++ b/packages/microcosm-build/tests/conftest.py @@ -1,5 +1,7 @@ -"""Shared fixtures: a small person+household frame, and an ODS writer.""" +"""Shared fixtures: a small person+household frame, an ODS writer, and the +autouse guard that keeps every test in this shard away from a live Logbook.""" +import sys from pathlib import Path from zipfile import ZIP_DEFLATED, ZIP_STORED, ZipFile @@ -104,3 +106,111 @@ class ODSBuilder: def ods() -> ODSBuilder: """Helpers for writing a small ODS file to a temporary path.""" return ODSBuilder() + + +# --- Logbook isolation -------------------------------------------------- +# +# The Logbook live store is append-only: a row that reaches it cannot be +# taken back. These tests must never be able to reach it, and "must never" +# has to hold in a shell that already has an operator's credentials +# exported — a developer who has just run an export, or a runner where the +# variables are set for a later job. Per-module ``delenv`` lists did not +# hold: they were written before the ``LOGBOOK_*`` spelling existed, so a +# shell carrying the new names walked straight past them into +# ``_remote_config()`` and out to Supabase. +# +# Two independent guards, because either one alone is a single point of +# failure: the environment is cleared for every test, and the Logbook +# network call is replaced with one that raises. A test that means to +# exercise the remote path stubs ``urlopen`` itself, as several already do; +# a test that means to reach the real network asks for +# ``allow_logbook_network`` and says so in its own body. + + +def _logbook_environment_names() -> tuple[str, ...]: + """Every environment variable that can point tests at a live Logbook. + + Both generations of the dual-read window, taken from the window itself + rather than restated, so adding a variable there cannot leave a hole + here. ``POPULACE_LOGBOOK_PREV_ROW_DIGEST`` is not in the window (it has + only ever had one spelling) but it is read from the environment and it + changes which chain a row claims to extend, so it is cleared too. + """ + from microcosm.build.logbook_env import LOGBOOK_ENV_LEGACY_NAMES + + names = {"POPULACE_LOGBOOK_PREV_ROW_DIGEST"} + for preferred, legacy_names in LOGBOOK_ENV_LEGACY_NAMES.items(): + names.add(preferred) + names.update(legacy_names) + return tuple(sorted(names)) + + +class UnstubbedLogbookNetworkError(AssertionError): + """Raised when a test reaches the Logbook live store for real.""" + + +def _refuse_logbook_network(*_args: object, **_kwargs: object) -> None: + raise UnstubbedLogbookNetworkError( + "A test tried to open a real Logbook HTTP request. The live store is " + "append-only, so this is never a harmless mistake. Stub the module's " + "'urlopen' (see test_logbook.py) to exercise the remote path, or " + "request the 'allow_logbook_network' fixture if the network is " + "genuinely the thing under test." + ) + + +def _logbook_urlopen_holders(real_urlopen: object) -> list[object]: + """Every imported module whose ``urlopen`` is the Logbook one. + + ``tools/logbook.py`` binds the function at import + (``from microcosm.build.logbook import urlopen``), so patching the + defining module alone leaves the CLI's copy live. Rather than name the + importers — the next one would be missed — this finds every module + currently holding the same object. + """ + return [ + module + for module in list(sys.modules.values()) + if getattr(module, "urlopen", None) is real_urlopen + ] + + +@pytest.fixture(autouse=True) +def _isolate_logbook(monkeypatch: pytest.MonkeyPatch) -> dict[str, object]: + """Detach every test from any live Logbook the shell may point at. + + Returns the callables it displaced, so the opt-in fixture below can put + the real ones back without having to guess what they were. + """ + from microcosm.build import logbook + + for name in _logbook_environment_names(): + monkeypatch.delenv(name, raising=False) + + displaced = { + "urlopen": logbook.urlopen, + "open": logbook._NO_REDIRECT_OPENER.open, + } + for module in _logbook_urlopen_holders(displaced["urlopen"]): + monkeypatch.setattr(module, "urlopen", _refuse_logbook_network) + # The opener is what a surviving reference to the real ``urlopen`` would + # ultimately call, so guarding it closes the path even for a caller this + # fixture did not find. + monkeypatch.setattr(logbook._NO_REDIRECT_OPENER, "open", _refuse_logbook_network) + return displaced + + +@pytest.fixture +def allow_logbook_network( + monkeypatch: pytest.MonkeyPatch, _isolate_logbook: dict[str, object] +) -> None: + """Opt back in to real Logbook HTTP. Nothing in this suite should need it. + + It exists so the guard above is a policy with a documented exception + rather than a wall, and so the guard itself can be tested. + """ + from microcosm.build import logbook + + for module in _logbook_urlopen_holders(_refuse_logbook_network): + monkeypatch.setattr(module, "urlopen", _isolate_logbook["urlopen"]) + monkeypatch.setattr(logbook._NO_REDIRECT_OPENER, "open", _isolate_logbook["open"]) diff --git a/packages/microcosm-build/tests/test_logbook_isolation.py b/packages/microcosm-build/tests/test_logbook_isolation.py new file mode 100644 index 000000000..cb38af7bf --- /dev/null +++ b/packages/microcosm-build/tests/test_logbook_isolation.py @@ -0,0 +1,94 @@ +"""The suite cannot reach a live Logbook, however the shell is configured. + +The Logbook live store is append-only. A test that appends to it has written +a row nobody can retract, into the chain a real release will later have to +validate. So the guarantee this file pins is not "tests usually do not use +the network" — it is that a test *cannot* reach the store even when the +developer or runner shell is fully credentialed, under either generation of +the variable names. + +The guard itself lives in ``conftest.py`` as an autouse fixture, which is +what makes it total: nothing has to remember to ask for it. +""" + +from __future__ import annotations + +import os + +import pytest + +from microcosm.build import logbook +from microcosm.build.logbook_env import LOGBOOK_ENV_LEGACY_NAMES, logbook_env + +_ALL_LOGBOOK_ENV_NAMES = tuple( + sorted( + {name for name in LOGBOOK_ENV_LEGACY_NAMES} + | { + legacy + for legacy_names in LOGBOOK_ENV_LEGACY_NAMES.values() + for legacy in legacy_names + } + | {"POPULACE_LOGBOOK_PREV_ROW_DIGEST"} + ) +) + + +@pytest.mark.parametrize("name", _ALL_LOGBOOK_ENV_NAMES) +def test_no_logbook_variable_survives_into_a_test(name: str) -> None: + """Both generations are cleared, not just the ledger-era spelling. + + The per-module ``delenv`` lists this fixture replaced named only + ``POPULACE_LEDGER_*``. They were written before ``LOGBOOK_*`` existed, so + a shell exporting the preferred names walked straight past them. + """ + assert name not in os.environ + + +def test_remote_config_is_unconfigured_regardless_of_the_shell() -> None: + """``_remote_config`` is the gate every write to the live store passes.""" + assert logbook._remote_config() is None + for name in LOGBOOK_ENV_LEGACY_NAMES: + assert logbook_env(name) is None + + +def test_the_logbook_network_call_is_refused_by_default() -> None: + """An unstubbed request raises instead of leaving the machine.""" + with pytest.raises(AssertionError, match="real Logbook HTTP request"): + logbook.urlopen(object(), timeout=1.0) + + +def test_the_underlying_opener_is_refused_too() -> None: + """The second guard: a stale reference to the real ``urlopen``. + + ``tools/logbook.py`` binds ``urlopen`` at import time, and a future + importer could do the same. Guarding the opener the real function + delegates to closes that path without having to enumerate importers. + """ + with pytest.raises(AssertionError, match="real Logbook HTTP request"): + logbook._NO_REDIRECT_OPENER.open(object(), timeout=1.0) + + +def test_appending_a_row_to_a_polluted_environment_writes_nothing() -> None: + """The end-to-end shape of the accident this prevents. + + With credentials in the environment, ``_remote_config`` would return them + and the append would POST to Supabase. Here it is unconfigured, so the + remote leg is skipped entirely — no request is attempted, which is why + this passes rather than raising the guard's error. + """ + assert logbook._remote_config() is None + + +def test_the_opt_in_fixture_restores_the_real_callables( + allow_logbook_network, +) -> None: + """The guard is a policy with a documented exception, not a wall. + + Nothing in this suite opts in. The fixture exists so a test that genuinely + needs the network can say so in its own body, and so the restoration path + is exercised rather than assumed. No request is made here: this asserts + only that the shipped callables are back in place. + """ + assert logbook.urlopen.__module__ == logbook.__name__ + assert logbook.urlopen.__name__ == "urlopen" + assert logbook._NO_REDIRECT_OPENER.open.__self__ is logbook._NO_REDIRECT_OPENER From 1f5619dea7a939ba2525f11305fb725de6799855 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Wed, 2 Sep 2026 13:11:41 -0400 Subject: [PATCH 17/23] Stop the barrel shadowing logbook_env, and export the Chronicle commit alias MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit microcosm.build re-exported the reader function logbook_env under the same name as its module. 'import a.b as c' returns the attribute when the package has one, so 'import microcosm.build.logbook_env as env' handed back the function and every env.LOGBOOK_URL_ENV after it raised AttributeError. The barrel now re-exports only logbook_env_names, which collides with nothing; the function is imported from its own module, as logbook.py and tools/logbook.py already do. CHRONICLE_US_SOURCE_COVERAGE_CONTRACT_COMMIT was added beside the ledger-era pin but never re-exported, so it was importable from source_coverage and not from the us_runtime barrel its sibling reaches — an alias only half the callers could use. Co-Authored-By: Claude Fable 5.1 --- .../src/microcosm/build/__init__.py | 18 ++++++++++-- .../microcosm/build/us_runtime/__init__.py | 2 ++ .../microcosm-build/tests/test_logbook_env.py | 29 +++++++++++++++++++ .../tests/test_us_source_coverage.py | 27 +++++++++++++++++ 4 files changed, 74 insertions(+), 2 deletions(-) diff --git a/packages/microcosm-build/src/microcosm/build/__init__.py b/packages/microcosm-build/src/microcosm/build/__init__.py index 17ca54703..b50754703 100644 --- a/packages/microcosm-build/src/microcosm/build/__init__.py +++ b/packages/microcosm-build/src/microcosm/build/__init__.py @@ -53,10 +53,15 @@ def _assert_frame_compatible(version: str, required: tuple[int, int]) -> None: ACCEPTED_CONSUMER_ARTIFACT_SCHEMA_VERSIONS, ACCEPTED_CONSUMER_FACT_SCHEMA_VERSIONS, CHRONICLE_EPOCH, + DECLARED_IDENTITIES, + DECLARED_IDENTITY_EPOCHS, EPOCHS, LEDGER_EPOCH, + UNDECLARED, fact_key_epoch, + fact_key_epoch_label, feed_fact_key_epochs, + feed_undeclared_fact_key_domains, is_chronicle_fact_key, parse_fact_key, ) @@ -137,8 +142,13 @@ def _assert_frame_compatible(version: str, required: tuple[int, int]) -> None: select_ledger_targets_from_jsonl, target_spec_from_ledger_fact, ) + +# Only ``logbook_env_names`` is re-exported here. The reader function is +# spelled ``logbook_env`` — the same name as its module — so binding it on +# the package would make ``import microcosm.build.logbook_env as env`` hand +# back the function instead of the module. Import the function from its +# module: ``from microcosm.build.logbook_env import logbook_env``. from microcosm.build.logbook_env import ( # noqa: E402 - after the compat gate - logbook_env, logbook_env_names, ) from microcosm.build.monetary_profile import ( # noqa: E402 - after compat gate @@ -214,8 +224,11 @@ def _assert_frame_compatible(version: str, required: tuple[int, int]) -> None: "ACCEPTED_CONSUMER_ARTIFACT_SCHEMA_VERSIONS", "ACCEPTED_CONSUMER_FACT_SCHEMA_VERSIONS", "CHRONICLE_EPOCH", + "DECLARED_IDENTITIES", + "DECLARED_IDENTITY_EPOCHS", "EPOCHS", "LEDGER_EPOCH", + "UNDECLARED", "LedgerConsumerArtifact", "LedgerTargetMapping", "LedgerTargetSelection", @@ -224,10 +237,11 @@ def _assert_frame_compatible(version: str, required: tuple[int, int]) -> None: "MonetaryTargetProfile", "PreparedMonetaryMeasure", "add_ledger_artifact_args", - "logbook_env", "logbook_env_names", "fact_key_epoch", + "fact_key_epoch_label", "feed_fact_key_epochs", + "feed_undeclared_fact_key_domains", "is_chronicle_fact_key", "parse_fact_key", "aggregate_admin_gate", diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/__init__.py b/packages/microcosm-build/src/microcosm/build/us_runtime/__init__.py index 518baacb6..9437b99e8 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/__init__.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/__init__.py @@ -885,6 +885,7 @@ with_us_snap_take_up_inputs, ) from microcosm.build.us_runtime.source_coverage import ( + CHRONICLE_US_SOURCE_COVERAGE_CONTRACT_COMMIT, LEDGER_US_SOURCE_COVERAGE_CONTRACT_COMMIT, US_SOURCE_COVERAGE, hard_target_package_aliases, @@ -1107,6 +1108,7 @@ "SimpleTaxExpenditureReform", "ReformValidationSpec", "REFORM_VALIDATION_SCHEMA_VERSION", + "CHRONICLE_US_SOURCE_COVERAGE_CONTRACT_COMMIT", "LEDGER_US_SOURCE_COVERAGE_CONTRACT_COMMIT", "AgeBand", "AGE_BANDS", diff --git a/packages/microcosm-build/tests/test_logbook_env.py b/packages/microcosm-build/tests/test_logbook_env.py index b06205773..5357f2591 100644 --- a/packages/microcosm-build/tests/test_logbook_env.py +++ b/packages/microcosm-build/tests/test_logbook_env.py @@ -174,3 +174,32 @@ def test_logbook_remote_config_reads_both_eras(monkeypatch) -> None: "logbook-jwt", "project-api-key", ) + + +def test_the_module_import_binds_the_module_not_the_reader() -> None: + """``import microcosm.build.logbook_env as env`` must give the module. + + The reader function shares its module's name. Re-exporting it from the + package barrel bound the function to ``microcosm.build.logbook_env``, and + ``import a.b as c`` returns the *attribute* when one exists — so the + submodule import silently handed back a callable, and every + ``env.LOGBOOK_URL_ENV`` after it raised ``AttributeError``. The barrel + therefore does not re-export the function; callers import it from here. + """ + import types + + import microcosm.build + import microcosm.build.logbook_env as env + + assert isinstance(env, types.ModuleType) + assert env.__name__ == "microcosm.build.logbook_env" + assert env.LOGBOOK_URL_ENV == LOGBOOK_URL_ENV + assert callable(env.logbook_env) + # The package attribute is the module too: `from microcosm.build import + # logbook_env` and the submodule import must not disagree about what the + # name means. + assert microcosm.build.logbook_env is env + assert "logbook_env" not in microcosm.build.__all__ + # The non-colliding helper is still re-exported, so removing the shadow + # did not quietly shrink the barrel further than it had to. + assert microcosm.build.logbook_env_names is logbook_env_names diff --git a/packages/microcosm-build/tests/test_us_source_coverage.py b/packages/microcosm-build/tests/test_us_source_coverage.py index 9f6b3096e..996967785 100644 --- a/packages/microcosm-build/tests/test_us_source_coverage.py +++ b/packages/microcosm-build/tests/test_us_source_coverage.py @@ -4,6 +4,7 @@ from microcosm.build.gates import GateReport from microcosm.build.us_runtime.source_coverage import ( + CHRONICLE_US_SOURCE_COVERAGE_CONTRACT_COMMIT, LEDGER_US_SOURCE_COVERAGE_CONTRACT_COMMIT, hard_target_package_aliases, source_gap_family_ids, @@ -14,8 +15,34 @@ ) +def test_both_contract_commit_spellings_reach_the_us_runtime_barrel() -> None: + """The Chronicle-named alias is importable everywhere its sibling is. + + The alias was added beside the ledger-era pin in ``source_coverage`` but + not re-exported, so ``from microcosm.build.us_runtime import + CHRONICLE_US_SOURCE_COVERAGE_CONTRACT_COMMIT`` failed while the ledger-era + name resolved — an alias only half of the callers could use. Both name the + same commit; neither is a second pin. + """ + from microcosm.build import us_runtime + + assert ( + us_runtime.CHRONICLE_US_SOURCE_COVERAGE_CONTRACT_COMMIT + is us_runtime.LEDGER_US_SOURCE_COVERAGE_CONTRACT_COMMIT + ) + for name in ( + "CHRONICLE_US_SOURCE_COVERAGE_CONTRACT_COMMIT", + "LEDGER_US_SOURCE_COVERAGE_CONTRACT_COMMIT", + ): + assert name in us_runtime.__all__, name + + def test_us_source_coverage_snapshot_has_expected_roles() -> None: assert len(LEDGER_US_SOURCE_COVERAGE_CONTRACT_COMMIT) == 40 + assert ( + CHRONICLE_US_SOURCE_COVERAGE_CONTRACT_COMMIT + == LEDGER_US_SOURCE_COVERAGE_CONTRACT_COMMIT + ) assert "soi-filing-season-week47-2024-eitc-total" in hard_target_package_aliases() assert "ssa-ssi-table-7b1-2024" in hard_target_package_aliases() assert "cms-aca-oep-state-level" in hard_target_package_aliases() From e5111760de8272e3d2bc6ac96211976938e3c974 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Wed, 2 Sep 2026 13:17:07 -0400 Subject: [PATCH 18/23] Carry the Chronicle epoch witnesses into the UK run's own evidence MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit _ledger_provenance rebuilt the block from the artifact field by field, which is how it came to record facts_sha256, fact_row_count and manifest_sha256 but not schema_epoch, fact_key_epochs or fact_schema_versions. The loader had already worked out which era resolved the targets; the UK run reported only the hashes, so a signed diagnostics file could not say whether a cutover-window feed had straddled the rename. It now delegates to LedgerConsumerArtifact.provenance() and keeps only the manifest sub-block UK-shaped, because that one feeds the run's identity digest and is deliberately narrow. A stand-in without the shared method still records what it carries rather than fabricating the rest. Coverage is a real mixed-epoch artifact through run_uk_calibration: a Chronicle-main manifest id over ledger-era and chronicle-era rows plus one undeclared Chronicle-namespace key, asserted in the build record and in the diagnostics 'build' block. The delegation itself is pinned as a property — every shared field reaches the UK block — rather than as a list that would go stale the next time the shared block grows. Co-Authored-By: Claude Fable 5.1 --- .../build/uk_runtime/calibration_run.py | 51 +++- .../tests/test_uk_calibration_run.py | 235 ++++++++++++++++-- 2 files changed, 261 insertions(+), 25 deletions(-) diff --git a/packages/microcosm-build/src/microcosm/build/uk_runtime/calibration_run.py b/packages/microcosm-build/src/microcosm/build/uk_runtime/calibration_run.py index 6864a9eff..527961aa0 100644 --- a/packages/microcosm-build/src/microcosm/build/uk_runtime/calibration_run.py +++ b/packages/microcosm-build/src/microcosm/build/uk_runtime/calibration_run.py @@ -919,23 +919,62 @@ def uk_aggregate_admin_totals( return totals, receipt +#: The manifest fields the UK run seals into ``run_config``. Narrower than +#: the artifact's whole manifest on purpose: only the fields that identify +#: *which* published artifact was compiled belong in the identity digest. +_LEDGER_MANIFEST_IDENTITY_FIELDS = ( + "artifact_id", + "profile", + "schema_version", + "generated_at", +) + +#: Epoch witnesses :meth:`LedgerConsumerArtifact.provenance` supplies, split +#: by shape. Named here so the delegation below cannot quietly drop one: a +#: run that compiled a chronicle-era or mixed-epoch feed has to say so in its +#: own evidence, not only in the loader's return value +#: (PolicyEngine/chronicle#143). +_LEDGER_EPOCH_WITNESS_SCALARS = ("schema_epoch",) +_LEDGER_EPOCH_WITNESS_LISTS = ( + "fact_key_epochs", + "undeclared_fact_key_domains", + "fact_schema_versions", +) + + def _ledger_provenance(artifact: Any) -> dict[str, object]: - """The verified identity of the Ledger consumer feed this run compiled. + """The verified identity of the Chronicle consumer feed this run compiled. + + Delegates to :meth:`microcosm.build.ledger_artifact.LedgerConsumerArtifact.provenance` + rather than rebuilding the block field by field. Rebuilding it is how the + epoch witnesses went missing here in the first place: the loader learned + which era resolved the targets and the UK run kept reporting only the + hashes. Anything the shared block gains, this block gains. + + Only the manifest sub-block is UK-shaped, and it stays narrow because it + feeds the run's identity digest. A bare ``consumer_facts.jsonl`` feed carries no manifest, so its - Ledger-side provenance is recorded as absent rather than invented. + Chronicle-side provenance is recorded as absent rather than invented. So + is a stand-in that predates the shared block: the fields it cannot supply + are recorded ``None``/empty rather than fabricated. """ + shared = getattr(artifact, "provenance", None) + block: Mapping[str, Any] = shared() if callable(shared) else {} provenance: dict[str, object] = { - "facts_sha256": getattr(artifact, "facts_sha256", None), - "fact_row_count": getattr(artifact, "fact_row_count", None), - "manifest_sha256": getattr(artifact, "manifest_sha256", None), + field: block.get(field, getattr(artifact, field, None)) + for field in ("facts_sha256", "fact_row_count", "manifest_sha256") } + for field in _LEDGER_EPOCH_WITNESS_SCALARS: + provenance[field] = block.get(field) + for field in _LEDGER_EPOCH_WITNESS_LISTS: + provenance[field] = list(block.get(field) or ()) manifest = getattr(artifact, "manifest", None) if isinstance(manifest, Mapping): provenance["manifest"] = { key: manifest.get(key) - for key in ("artifact_id", "profile", "schema_version", "generated_at") + for key in _LEDGER_MANIFEST_IDENTITY_FIELDS if manifest.get(key) is not None } return provenance diff --git a/packages/microcosm-build/tests/test_uk_calibration_run.py b/packages/microcosm-build/tests/test_uk_calibration_run.py index 5781eb777..08405c963 100644 --- a/packages/microcosm-build/tests/test_uk_calibration_run.py +++ b/packages/microcosm-build/tests/test_uk_calibration_run.py @@ -10,7 +10,13 @@ import pandas as pd import pytest +from microcosm.build.chronicle_epoch import ( + CHRONICLE_CONSUMER_FACT_SCHEMA_VERSION, + LEDGER_CONSUMER_FACT_SCHEMA_VERSION, + PUBLISHED_CONSUMER_ARTIFACT_SCHEMA_VERSION, +) from microcosm.build.country_spec import load_country_spec +from microcosm.build.ledger_artifact import load_ledger_consumer_artifact from microcosm.build.logbook import canonical_json_bytes from microcosm.build.uk_runtime import calibration_run from microcosm.build.uk_runtime.calibration_run import ( @@ -137,9 +143,7 @@ def _write_spine_sidecar( entity: int(len(frame.table(entity))) for entity in frame.entities }, "household_weight_kind": uk_household_weight_kind(frame).value, - "household_weight_total": float( - frame.weights_for("household").values.sum() - ), + "household_weight_total": float(frame.weights_for("household").values.sum()), } sidecar.update(overrides) input_h5.with_suffix(".build.json").write_text( @@ -178,8 +182,13 @@ def _admin_anchor_values(): def test_gate_scope_classifies_every_uk_gate(): all_ids = {entry.id for entry in load_country_spec("uk").gates.gates} - assert set(UK_CALIBRATION_GATE_SCOPE) | set(UK_CALIBRATION_GATE_SCOPE_EXCLUSIONS) == all_ids - assert set(UK_CALIBRATION_GATE_SCOPE).isdisjoint(UK_CALIBRATION_GATE_SCOPE_EXCLUSIONS) + assert ( + set(UK_CALIBRATION_GATE_SCOPE) | set(UK_CALIBRATION_GATE_SCOPE_EXCLUSIONS) + == all_ids + ) + assert set(UK_CALIBRATION_GATE_SCOPE).isdisjoint( + UK_CALIBRATION_GATE_SCOPE_EXCLUSIONS + ) assert all(UK_CALIBRATION_GATE_SCOPE_EXCLUSIONS.values()) @@ -233,9 +242,15 @@ def test_run_uk_calibration_writes_cross_pinned_outputs(monkeypatch, tmp_path: P assert paths.diagnostics_json.exists() assert paths.build_record_json.exists() assert paths.terminal_gate_json.exists() - assert result.build_record["artifacts"]["staging_h5"]["sha256"] == _sha(paths.staging_h5) - assert result.build_record["artifacts"]["diagnostics_json"]["sha256"] == _sha(paths.diagnostics_json) - assert result.build_record["artifacts"]["terminal_gate_json"]["sha256"] == _sha(paths.terminal_gate_json) + assert result.build_record["artifacts"]["staging_h5"]["sha256"] == _sha( + paths.staging_h5 + ) + assert result.build_record["artifacts"]["diagnostics_json"]["sha256"] == _sha( + paths.diagnostics_json + ) + assert result.build_record["artifacts"]["terminal_gate_json"]["sha256"] == _sha( + paths.terminal_gate_json + ) # The record makes no shippability claim of its own — the hand-written # literal retired with the #757 release-cut audit — and instead points # at the certification artifact whose verdict is authoritative. @@ -251,9 +266,13 @@ def test_run_uk_calibration_writes_cross_pinned_outputs(monkeypatch, tmp_path: P assert spine_provenance["stage_records"] == spine_sidecar["stage_records"] assert spine_provenance["stage_evidence"] == spine_sidecar["stage_evidence"] assert spine_provenance["artifact_pins"] == spine_sidecar["artifact_pins"] - assert spine_provenance["input_artifact_pins"] == spine_sidecar["input_artifact_pins"] + assert ( + spine_provenance["input_artifact_pins"] == spine_sidecar["input_artifact_pins"] + ) assert spine_provenance["resource_pins"] == spine_sidecar["resource_pins"] - assert spine_provenance["stage_artifact_pins"] == spine_sidecar["stage_artifact_pins"] + assert ( + spine_provenance["stage_artifact_pins"] == spine_sidecar["stage_artifact_pins"] + ) assert spine_provenance["declared_seeds"] == spine_sidecar["declared_seeds"] assert spine_provenance["rules_engine"] == spine_sidecar["rules_engine"] assert spine_provenance["source_vintages"] == spine_sidecar["source_vintages"] @@ -272,7 +291,10 @@ def test_run_uk_calibration_writes_cross_pinned_outputs(monkeypatch, tmp_path: P signature = attestation["signature"] attestation["signature"] = None key = b"0123456789abcdef0123456789abcdef" - assert hmac.new(key, canonical_json_bytes(report), hashlib.sha256).hexdigest() == signature + assert ( + hmac.new(key, canonical_json_bytes(report), hashlib.sha256).hexdigest() + == signature + ) assert result.logbook_spool.exists() @@ -456,10 +478,13 @@ def test_run_uk_calibration_refuses_input_sha_before_outputs(tmp_path: Path): doctrine_overrides={}, measure_resolver=None, source_pins={ - "input_h5": {"sha256": _sha(input_h5), "size_bytes": input_h5.stat().st_size} + "input_h5": { + "sha256": _sha(input_h5), + "size_bytes": input_h5.stat().st_size, + } }, run_config_extra={"calibration_year": 2025}, - release_id="bad-sha", + release_id="bad-sha", ) assert not paths.staging_h5.exists() assert not paths.diagnostics_json.exists() @@ -496,7 +521,7 @@ def test_run_uk_calibration_refuses_absent_input_sidecar(tmp_path: Path): } }, run_config_extra={"calibration_year": 2025}, - release_id="missing-sidecar", + release_id="missing-sidecar", ) assert not paths.staging_h5.exists() @@ -549,7 +574,7 @@ def test_run_uk_calibration_refuses_unbound_input_sidecar( } }, run_config_extra={"calibration_year": 2025}, - release_id="unbound-sidecar", + release_id="unbound-sidecar", ) assert not paths.staging_h5.exists() @@ -612,7 +637,10 @@ def test_seam_never_modifies_data_variables(monkeypatch, tmp_path: Path): doctrine_overrides={}, measure_resolver=None, source_pins={ - "input_h5": {"sha256": _sha(input_h5), "size_bytes": input_h5.stat().st_size} + "input_h5": { + "sha256": _sha(input_h5), + "size_bytes": input_h5.stat().st_size, + } }, run_config_extra={}, release_id="invariant-run", @@ -749,7 +777,7 @@ def test_refusal_records_a_failed_attempt_and_stages_nothing(tmp_path: Path): } }, run_config_extra={"calibration_year": 2025}, - release_id="refused-run", + release_id="refused-run", ) # Every terminal disposition is a row; a refusal that left the chain @@ -806,7 +834,7 @@ def test_attempt_ids_are_unique_across_reruns_of_one_release( measure_resolver=None, source_pins=source_pins, run_config_extra={"calibration_year": 2025}, - release_id="one-release-id", + release_id="one-release-id", ) build_ids.append(result.build_record["build_id"]) @@ -857,7 +885,10 @@ def test_verified_ledger_identity_reaches_the_run_evidence(monkeypatch, tmp_path doctrine_overrides={}, measure_resolver=None, source_pins={ - "input_h5": {"sha256": _sha(input_h5), "size_bytes": input_h5.stat().st_size} + "input_h5": { + "sha256": _sha(input_h5), + "size_bytes": input_h5.stat().st_size, + } }, run_config_extra={"calibration_year": 2025}, release_id="ledger-identity", @@ -875,3 +906,169 @@ def test_verified_ledger_identity_reaches_the_run_evidence(monkeypatch, tmp_path # A bare feed carries no manifest, and that absence is recorded rather # than invented. assert calibration_run._ledger_provenance(object())["manifest_sha256"] is None + + +def _consumer_fact_row( + *, + aggregate_fact_key: str, + semantic_fact_key: str, + schema_version: str, + source_release_key: str | None = None, +) -> dict: + row = { + "aggregate_fact_key": aggregate_fact_key, + "semantic_fact_key": semantic_fact_key, + "schema_version": schema_version, + "value": 1.0, + "period": {"type": "tax_year", "value": 2025}, + "geography": {"level": "country", "id": "K02000001"}, + "entity": {"name": "household"}, + "aggregation": {"method": "sum"}, + "observed_measure": {"source_name": "ons", "unit": "gbp"}, + "source": {"source_name": "ons"}, + "lineage": {"source_record_id": "ons.2025.total"}, + } + if source_release_key is not None: + row["source_release_key"] = source_release_key + return row + + +def _mixed_epoch_artifact_dir(tmp_path: Path) -> Path: + """A cutover-window feed: ledger-era history, chronicle-era rows, and one + Chronicle-namespace key spelling nothing declares. + + The manifest declares what Chronicle's ``main`` stamps today + (``policyengine_ledger.consumer_artifact.v2``), so this is the shape a UK + run is handed now, not a hypothetical one. + """ + rows = [ + _consumer_fact_row( + aggregate_fact_key="ledger.aggregate_fact.v2:aaa", + semantic_fact_key="ledger.semantic_fact.v2:aaa", + schema_version=LEDGER_CONSUMER_FACT_SCHEMA_VERSION, + ), + _consumer_fact_row( + aggregate_fact_key="chronicle.aggregate_fact.v3:bbb", + semantic_fact_key="chronicle.semantic_fact.v3:bbb", + schema_version=CHRONICLE_CONSUMER_FACT_SCHEMA_VERSION, + ), + _consumer_fact_row( + aggregate_fact_key="ledger.aggregate_fact.v2:ccc", + semantic_fact_key="ledger.semantic_fact.v2:ccc", + schema_version=LEDGER_CONSUMER_FACT_SCHEMA_VERSION, + source_release_key="chronicle.source_release.v9:undeclared", + ), + ] + artifact_dir = tmp_path / "chronicle-artifact" + artifact_dir.mkdir() + payload = "".join(json.dumps(row, sort_keys=True) + "\n" for row in rows) + (artifact_dir / "consumer_facts.jsonl").write_text(payload) + (artifact_dir / "manifest.json").write_text( + json.dumps( + { + "schema_version": PUBLISHED_CONSUMER_ARTIFACT_SCHEMA_VERSION, + "artifact_id": "chronicle-uk-artifact-mixed", + "profile": "uk-national", + "fact_row_count": len(rows), + "facts_sha256": hashlib.sha256(payload.encode()).hexdigest(), + }, + indent=2, + sort_keys=True, + ) + + "\n" + ) + return artifact_dir + + +def test_mixed_epoch_feed_epochs_reach_the_uk_release_evidence(monkeypatch, tmp_path): + """A UK run says which Chronicle era resolved its targets. + + The run's own provenance block used to be assembled field by field from + the artifact, which is how it came to carry the hashes but not the epoch + witnesses the loader had already computed. It now delegates to the shared + block, so a cutover-window feed is visible in the signed diagnostics and + in the build record — including the one Chronicle-namespace spelling this + build does not declare, named rather than folded into an era. + """ + pytest.importorskip("tables") # pandas HDF backend + monkeypatch.setattr( + calibration_run, + "uk_aggregate_admin_totals", + lambda frame, manifest: (_admin_anchor_values(), []), + ) + input_h5 = tmp_path / "input.h5" + frame = _frame() + write_uk_national_frame(frame, input_h5) + _write_spine_sidecar(input_h5, frame) + artifact = load_ledger_consumer_artifact(_mixed_epoch_artifact_dir(tmp_path)) + diagnostics_json = tmp_path / "diagnostics.json" + + result = run_uk_calibration( + paths=UKCalibrationRunPaths( + input_h5=input_h5, + staging_h5=tmp_path / "staged.h5", + diagnostics_json=diagnostics_json, + build_record_json=tmp_path / "build_record.json", + terminal_gate_json=tmp_path / "terminal_gates.json", + ), + input_sha256=_sha(input_h5), + ledger_artifact=artifact, + register_registry=_registry(), + band_edge_registry=_registry(), + calibration_year=2025, + exclusion_receipt={}, + doctrine=UKNationalSolveDoctrine(epochs=5), + doctrine_overrides={}, + measure_resolver=None, + source_pins={ + "input_h5": { + "sha256": _sha(input_h5), + "size_bytes": input_h5.stat().st_size, + } + }, + run_config_extra={"calibration_year": 2025}, + release_id="chronicle-mixed-epoch", + ) + + ledger = result.build_record["run_config"]["ledger"] + # The observed manifest id, verbatim, and the era it belongs to. + assert ledger["manifest"]["schema_version"] == ( + PUBLISHED_CONSUMER_ARTIFACT_SCHEMA_VERSION + ) + assert ledger["schema_epoch"] == "ledger" + # The feed straddles the cutover, and says so rather than reporting one era. + assert ledger["fact_key_epochs"] == ["ledger", "chronicle", "undeclared"] + assert ledger["undeclared_fact_key_domains"] == ["chronicle.source_release.v9"] + assert ledger["fact_schema_versions"] == [ + CHRONICLE_CONSUMER_FACT_SCHEMA_VERSION, + LEDGER_CONSUMER_FACT_SCHEMA_VERSION, + ] + # The hashes the block always carried are unchanged by the delegation. + assert ledger["facts_sha256"] == artifact.facts_sha256 + assert ledger["fact_row_count"] == 3 + + # The same block is what the signed diagnostics carry, so the evidence a + # release assembler reads witnesses the era too. + diagnostics = json.loads(diagnostics_json.read_text()) + assert diagnostics["build"]["ledger"] == ledger + + +def test_the_uk_block_delegates_rather_than_reassembling_the_shared_one(tmp_path): + """Every field of the shared provenance block reaches the UK block. + + Pinned as a property, not as a list: the failure being prevented is a + field the loader learns and this seam silently drops, and a test that + enumerated today's fields would not catch tomorrow's. + """ + artifact = load_ledger_consumer_artifact(_mixed_epoch_artifact_dir(tmp_path)) + shared = artifact.provenance() + + block = calibration_run._ledger_provenance(artifact) + + for field, value in shared.items(): + if field in {"path_name", "schema_version", "profiles"}: + # Not identity of the feed: the directory name is local, and the + # manifest id is carried in the narrower manifest sub-block. + continue + assert block[field] == value, field + assert block["manifest"]["schema_version"] == shared["schema_version"] From edac801ec666222a7b0ca36f57382e6ccb8eaffd Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Wed, 2 Sep 2026 13:18:24 -0400 Subject: [PATCH 19/23] Advertise the preferred Logbook variable names in the CLI help 'logbook export --remote --help' named POPULACE_LEDGER_URL alone, so an operator configuring from the help text set up the exact spelling the dual-read window exists to retire, then got a deprecation warning for following the instructions. The preferred LOGBOOK_* names come first and the ledger-era ones are named as the fallback they are, since an operator whose environment predates the rename still has to recognise what is being asked for. Co-Authored-By: Claude Fable 5.1 --- .../microcosm-build/tests/test_logbook_cli.py | 62 +++++++++++++++---- tools/logbook.py | 13 +++- 2 files changed, 60 insertions(+), 15 deletions(-) diff --git a/packages/microcosm-build/tests/test_logbook_cli.py b/packages/microcosm-build/tests/test_logbook_cli.py index 64444560f..cafaec71f 100644 --- a/packages/microcosm-build/tests/test_logbook_cli.py +++ b/packages/microcosm-build/tests/test_logbook_cli.py @@ -2,6 +2,7 @@ from __future__ import annotations +import argparse import importlib.util import json from pathlib import Path @@ -165,9 +166,7 @@ def test_export_refuses_a_directory_archive( spool.mkdir() cli = _load_cli() - assert ( - cli.main(["export", "--archive", str(tmp_path), "--source", str(spool)]) == 1 - ) + assert cli.main(["export", "--archive", str(tmp_path), "--source", str(spool)]) == 1 assert "extends exactly one scope chain" in capsys.readouterr().err @@ -256,9 +255,7 @@ def test_cli_local_export_refuses_wrong_scope_rows( _write_jsonl(source, _chain(pipeline="uk-local-rowwise")) cli = _load_cli() - exit_code = cli.main( - ["export", "--archive", str(archive), "--source", str(source)] - ) + exit_code = cli.main(["export", "--archive", str(archive), "--source", str(source)]) assert exit_code == 1 err = capsys.readouterr().err @@ -280,9 +277,7 @@ def test_cli_export_refuses_an_unratified_scope_archive( _write_jsonl(source, _chain(pipeline="uk-firms-staging")) cli = _load_cli() - exit_code = cli.main( - ["export", "--archive", str(archive), "--source", str(source)] - ) + exit_code = cli.main(["export", "--archive", str(archive), "--source", str(source)]) assert exit_code == 1 assert "not in the ratified scope list" in capsys.readouterr().err @@ -300,9 +295,7 @@ def test_cli_export_accepts_uk_local_scope_archive( _write_jsonl(source, rows) cli = _load_cli() - exit_code = cli.main( - ["export", "--archive", str(archive), "--source", str(source)] - ) + exit_code = cli.main(["export", "--archive", str(archive), "--source", str(source)]) assert exit_code == 0 assert "exported 3 new Logbook rows" in capsys.readouterr().out @@ -616,3 +609,48 @@ def test_chain_scope_matches_sql_contract( cli = _load_cli() assert cli._chain_scope(pipeline) == scope + + +def test_remote_help_asks_for_the_preferred_logbook_variable_names() -> None: + """``--help`` is the only place most operators read these names. + + It advertised ``POPULACE_LEDGER_URL`` alone, so an operator configuring + from the help text set up the very spelling the dual-read window exists to + retire — and then got a deprecation warning for following the + instructions. The preferred names come first; the ledger-era ones are + named as the fallback they are, because an operator whose environment + predates the rename still has to recognise what is being asked for. + """ + cli = _load_cli() + + remote = next( + action + for action in _export_actions(cli) + if "--remote" in getattr(action, "option_strings", ()) + ) + help_text = remote.help or "" + + for preferred in ( + cli.LOGBOOK_URL_ENV, + cli.LOGBOOK_EXPORT_KEY_ENV, + cli.LOGBOOK_API_KEY_ENV, + ): + assert preferred in help_text, preferred + for legacy in ( + cli.LEGACY_URL_ENV, + cli.REMOTE_EXPORT_KEY_ENV, + cli.REMOTE_API_KEY_ENV, + ): + assert legacy in help_text, legacy + # Preferred first: the reading order is the migration instruction. + assert help_text.index(cli.LOGBOOK_URL_ENV) < help_text.index(cli.LEGACY_URL_ENV) + + +def _export_actions(cli: ModuleType) -> list[object]: + """Every argparse action on the ``export`` subcommand.""" + subparsers = next( + action + for action in cli._parser()._actions + if isinstance(action, argparse._SubParsersAction) + ) + return list(subparsers.choices["export"]._actions) diff --git a/tools/logbook.py b/tools/logbook.py index 423ba0097..852ca3946 100644 --- a/tools/logbook.py +++ b/tools/logbook.py @@ -41,6 +41,7 @@ from microcosm.build.logbook_env import ( LEGACY_API_KEY_ENV, LEGACY_EXPORT_KEY_ENV, + LEGACY_URL_ENV, LOGBOOK_API_KEY_ENV, LOGBOOK_EXPORT_KEY_ENV, LOGBOOK_URL_ENV, @@ -117,10 +118,16 @@ def _parser() -> argparse.ArgumentParser: source.add_argument( "--remote", action="store_true", + # The help text names the preferred Logbook spellings first and the + # ledger-era ones as the fallback they now are, so an operator + # reading --help configures the environment the dual-read window + # wants rather than the one it merely still tolerates. help=( - "Read the private live store using POPULACE_LEDGER_URL and the " - f"read-only {REMOTE_EXPORT_KEY_ENV}, authenticated at the gateway " - f"by {REMOTE_API_KEY_ENV}." + f"Read the private live store using {LOGBOOK_URL_ENV} and the " + f"read-only {LOGBOOK_EXPORT_KEY_ENV}, authenticated at the " + f"gateway by {LOGBOOK_API_KEY_ENV}. The ledger-era names " + f"({LEGACY_URL_ENV}, {REMOTE_EXPORT_KEY_ENV}, " + f"{REMOTE_API_KEY_ENV}) are still honored and warn once." ), ) From 5f14424f01f297bd344470af1e74d7c37861324e Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Wed, 2 Sep 2026 13:19:26 -0400 Subject: [PATCH 20/23] Historicize the first-pass journal's identity and design claims The lane journal is history once written, but four of its claims read as current design and are now wrong: the two-id accepted set, the uniform v2-to-v3 domain bump, CHRONICLE_* env names, and structural epoch detection. Each is annotated in place with the correction and its date so the file cannot mislead a later reader, and the changelog fragments now describe the behavior that is actually on the branch. Co-Authored-By: Claude Fable 5.1 --- PROGRESS-chronicle-dual-accept.md | 34 ++++++++++++++++++++ changelog.d/chronicle-dual-accept.added.md | 4 +-- changelog.d/chronicle-dual-accept.changed.md | 1 + 3 files changed, 36 insertions(+), 3 deletions(-) create mode 100644 changelog.d/chronicle-dual-accept.changed.md diff --git a/PROGRESS-chronicle-dual-accept.md b/PROGRESS-chronicle-dual-accept.md index 409a159a4..604b9ec9a 100644 --- a/PROGRESS-chronicle-dual-accept.md +++ b/PROGRESS-chronicle-dual-accept.md @@ -4,6 +4,15 @@ Branch `chronicle-dual-accept`. Journal for this lane. Root journals are history, not state (see CLAUDE.md); this one is current only while the branch is open. +> **Corrected 2026-09-02 by the round-2 review pass.** Several identity and +> design claims below were accurate to the first pass and are wrong now. The +> accepted artifact set is three ids, not two; the chronicle-era successor of +> the artifact id is `.v3`; epoch resolution is a declared registry, not +> namespace parsing; and the env half is `LOGBOOK_*`, not `CHRONICLE_*`. Each +> is annotated in place below. See `PROGRESS-chronicle-dual-accept-round2.md` +> and the PR's "Review fixes (round 2)" section for what is actually in the +> branch. + ## Goal Microcosm must accept BOTH ledger-era and chronicle-era Chronicle identities @@ -12,11 +21,25 @@ Microcosm must accept BOTH ledger-era and chronicle-era Chronicle identities - schema ids: `policyengine_ledger.consumer_artifact.v1` **and** `policyengine_chronicle.consumer_artifact.v2`; `ledger.consumer_fact.v1` **and** `chronicle.consumer_fact.v2` + + *(Corrected 2026-09-02: wrong at both ends. Chronicle's `main` emits + `policyengine_ledger.consumer_artifact.v2` today, so that pair rejected + every artifact Chronicle publishes; and the chronicle-era successor of the + v2 id is `policyengine_chronicle.consumer_artifact.v3`. The branch now + accepts all three.)* - hash domains: `ledger..v2` **and** `chronicle..v3` (same canonical payload, new domain string) + + *(Corrected 2026-09-02: the version numbers differ by family. The + source-side domains are ledger `v1` → chronicle `v2`; only the derived + families are `v2` → `v3`.)* - env names: `CHRONICLE_*` preferred, legacy honored with a once-per-process deprecation warning + *(Corrected on the PR before merge, and again here: these are the + **Logbook** store's credentials, so the preferred spellings are + `LOGBOOK_*`. See the PR review comment of 2026-09-01.)* + Frozen (microcosm#639): nothing on disk or in artifacts renames. Diagnostic field names (`ledger_aggregate_fact_key`, `ledger_commit`), H5 attrs, `populace_*` ids, fact keys, goldens and fixtures stay at v1. @@ -36,8 +59,19 @@ in the lane's report. PR #849, open, do not merge. lookup in a frozen domain list, because chronicle#143 declares the `v3` spelling only for the aggregate and semantic fact families. Only identity strings the spec names explicitly are pinned as literals. + + *(Corrected 2026-09-02: structural detection reported + `chronicle..vN` as Chronicle-**issued** identity, which is a + guess dressed as a witness. Resolution is now a declared registry keyed by + `(namespace, family, version)`; an undeclared spelling in a Chronicle + namespace is reported as `undeclared`. The claim that chronicle#143 + "declares the `v3` spelling" for two families also overstated the issue, + which lists `dual-hash window or v3 domains` as open options.)* - `microcosm/build/chronicle_env.py` — the env dual-read window, one helper, one `DeprecationWarning` per process per legacy name. + + *(Renamed before merge to `microcosm/build/logbook_env.py`; there is no + `chronicle_env.py` on the branch.)* - `ledger_artifact.py` — the manifest `schema_version` is a membership test over both eras; per-row schema ids and fact keys are carried as published; `provenance()` records the observed manifest id, `schema_epoch`, diff --git a/changelog.d/chronicle-dual-accept.added.md b/changelog.d/chronicle-dual-accept.added.md index 2f388d91b..0af1451c3 100644 --- a/changelog.d/chronicle-dual-accept.added.md +++ b/changelog.d/chronicle-dual-accept.added.md @@ -1,3 +1 @@ -Accept chronicle-era Chronicle identities everywhere Microcosm consumes them: consumer-artifact manifests declaring `policyengine_chronicle.consumer_artifact.v2` load beside `policyengine_ledger.consumer_artifact.v1`, mixed-epoch feeds compile, and the loader records the observed manifest schema id, its epoch, the fact-key epochs present in the feed, and the per-row schema ids the rows declare in provenance. Only the manifest id is gated: per-row schema ids and fact keys are carried as published, since real feeds mint rows in namespaces belonging to neither era. Nothing renames on disk: emitted ids, diagnostic field names, `populace_*` key namespaces, goldens and fixtures stay frozen at v1. - -Separately, a Logbook naming cleanup (microcosm#632): the build ledger's environment variables gain a `LOGBOOK_*` dual-read window with the `POPULACE_LEDGER_*` names honored under a once-per-process deprecation warning. These are the Logbook store's own credentials, unrelated to the Chronicle fact store above. +Accept chronicle-era Chronicle identities everywhere Microcosm consumes them, from a declared identity registry keyed by `(namespace, family, version)`. Consumer-artifact manifests load under every id a real producer stamps — Microcosm's own minted `policyengine_ledger.consumer_artifact.v1`, the `policyengine_ledger.consumer_artifact.v2` Chronicle emits today, and the chronicle-era `policyengine_chronicle.consumer_artifact.v3` successor — and per-row `ledger.consumer_fact.v1` / `chronicle.consumer_fact.v2` alike. Mixed-epoch feeds compile, and provenance records the observed manifest id, its epoch, the fact-key epoch labels present in the feed, any undeclared Chronicle-namespace key domain among them, and the per-row schema ids the rows declare. An identity in a Chronicle namespace that the registry does not carry is reported as `undeclared`, never counted as Chronicle-issued. Only the manifest id is gated: per-row schema ids and fact keys are carried as published, since real feeds mint rows in namespaces belonging to neither era. Nothing renames on disk: emitted ids, diagnostic field names, `populace_*` key namespaces, goldens and fixtures stay frozen at v1. diff --git a/changelog.d/chronicle-dual-accept.changed.md b/changelog.d/chronicle-dual-accept.changed.md new file mode 100644 index 000000000..ec894c9c6 --- /dev/null +++ b/changelog.d/chronicle-dual-accept.changed.md @@ -0,0 +1 @@ +Give the Logbook store's environment variables a `LOGBOOK_*` dual-read window, with the `POPULACE_LEDGER_*` names honored under a once-per-process deprecation warning and preferred in the `tools/logbook.py --remote` help text (microcosm#632). These are the Logbook store's own credentials, unrelated to the Chronicle fact store. Every test in the build shard is now detached from a live Logbook by an autouse fixture that clears both generations of the variables and refuses unstubbed Logbook HTTP. From 3c4ddf2ca1713a39699c0f646748d305ee672c57 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Wed, 2 Sep 2026 13:20:30 -0400 Subject: [PATCH 21/23] Drop two prose claims the declared registry falsified Both said chronicle#143 declares the v3 spelling for exactly two fact families, which the issue does not: it lists 'dual-hash window or v3 domains' as open options. The import_entry_facts header also named the chronicle-era artifact id as .v2 and did not say which of the ledger-era ids this module emits, which is the distinction the whole finding turned on. Co-Authored-By: Claude Fable 5.1 --- .../build/us_runtime/us_trade/import_entry_facts.py | 12 ++++++++---- .../microcosm-build/tests/test_chronicle_epoch.py | 3 +-- 2 files changed, 9 insertions(+), 6 deletions(-) diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/us_trade/import_entry_facts.py b/packages/microcosm-build/src/microcosm/build/us_runtime/us_trade/import_entry_facts.py index 4851851a0..3525b9234 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/us_trade/import_entry_facts.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/us_trade/import_entry_facts.py @@ -9,13 +9,17 @@ CBP statistics. The contract has a chronicle-era spelling too (``chronicle.consumer_fact.v2`` -under ``policyengine_chronicle.consumer_artifact.v2``, chronicle#143), and the -loader accepts both. What this module *emits* stays ledger-era: the rows are +under ``policyengine_chronicle.consumer_artifact.v3``, chronicle#143), and the +loader accepts every declared spelling. What this module *emits* stays +ledger-era, and specifically stays at +``policyengine_ledger.consumer_artifact.v1`` — the id Microcosm's own minted +artifacts have always declared, which is a generation behind the +``...consumer_artifact.v2`` Chronicle itself publishes. The rows are byte-pinned goldens, and they are minted from official source bytes rather than derived from a Chronicle row whose epoch they would have to inherit. :func:`write_consumer_artifact` takes the declared id as an argument, checked -for membership in both eras, so flipping the declaration later is a caller -change rather than a code change. +for membership in the declared set, so flipping the declaration later is a +caller change rather than a code change. Producer identity is explicit: these rows are **microcosm-minted** from official source bytes — not an export of a PolicyEngine/ledger build — and diff --git a/packages/microcosm-build/tests/test_chronicle_epoch.py b/packages/microcosm-build/tests/test_chronicle_epoch.py index 9f1fd4c0e..53d5516bb 100644 --- a/packages/microcosm-build/tests/test_chronicle_epoch.py +++ b/packages/microcosm-build/tests/test_chronicle_epoch.py @@ -657,8 +657,7 @@ def test_every_published_key_field_is_witnessed_for_its_epoch() -> None: The captured feed fixture carries nine single-key fields and two key lists, not just the four identifiers targets resolve by. Chronicle's - cutover moves families independently — the spec declares ``v3`` only for - the aggregate and semantic families — so a row can straddle it: ledger-era + cutover moves families independently, so a row can straddle it: ledger-era aggregate key, chronicle-era source-release key. Reading only the resolution set would report that row as pure ledger-era. """ From 65a25d87d0485301288e643d2e50cbd3e460a562 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Wed, 2 Sep 2026 13:23:54 -0400 Subject: [PATCH 22/23] Record the round-2 state and the structural change for the next reader Co-Authored-By: Claude Fable 5.1 --- PROGRESS-chronicle-dual-accept-round2.md | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/PROGRESS-chronicle-dual-accept-round2.md b/PROGRESS-chronicle-dual-accept-round2.md index 8728c5df1..99a513e9d 100644 --- a/PROGRESS-chronicle-dual-accept-round2.md +++ b/PROGRESS-chronicle-dual-accept-round2.md @@ -29,9 +29,23 @@ that matter: `policyengine_chronicle.consumer_artifact.v3` and ## State -Round 2 complete; pushed to `chronicle-dual-accept`. **DO NOT MERGE** — +Round 2 code, tests, journal and changelog pushed to `chronicle-dual-accept` +at `3c4ddf2c`. Two full `packages/microcosm-build` suites (clean env and the +polluted-Logbook env from finding 2) were running at the time of writing; +their results and the PR-body update are the last steps. **DO NOT MERGE** — the lane is authorized to fix and push only. +## What changed structurally, for the next reader + +The first pass treated epoch as something you could *read off* a key's +namespace. That is why it accepted a two-id set that excluded the id +Chronicle actually emits, and why `chronicle.anything.vN` was witnessed as +Chronicle-issued identity. Round 2 replaces it with a declared registry: +`DECLARED_IDENTITIES` in `chronicle_epoch.py`, keyed by +`(namespace, family, version)`. Anything not in that table is `undeclared` +and is reported as such. If Chronicle publishes its own successor +enumeration, that table is the one place that changes. + ## Done - [x] (1) Declared identity registry keyed by `(namespace, family, version)` From a15f153decc80214c7f2349777518fc53fb058b9 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Wed, 2 Sep 2026 15:09:27 -0400 Subject: [PATCH 23/23] Record the round-2 verification, including the two pre-existing failures Both microcosm-build suites are outcome-identical, which is the guarantee the Logbook isolation finding asked for. The two red tests are recorded with the evidence that they reproduce at origin/main, because the earlier lane reported this suite as all green and on a machine holding the pinned feed it is not. Co-Authored-By: Claude Fable 5.1 --- PROGRESS-chronicle-dual-accept-round2.md | 37 ++++++++++++++++++++---- 1 file changed, 32 insertions(+), 5 deletions(-) diff --git a/PROGRESS-chronicle-dual-accept-round2.md b/PROGRESS-chronicle-dual-accept-round2.md index 99a513e9d..8406d02f9 100644 --- a/PROGRESS-chronicle-dual-accept-round2.md +++ b/PROGRESS-chronicle-dual-accept-round2.md @@ -29,11 +29,38 @@ that matter: `policyengine_chronicle.consumer_artifact.v3` and ## State -Round 2 code, tests, journal and changelog pushed to `chronicle-dual-accept` -at `3c4ddf2c`. Two full `packages/microcosm-build` suites (clean env and the -polluted-Logbook env from finding 2) were running at the time of writing; -their results and the PR-body update are the last steps. **DO NOT MERGE** — -the lane is authorized to fix and push only. +Round 2 complete and pushed to `chronicle-dual-accept`; the PR body carries a +"Review fixes (round 2)" section with the same content as this journal's +verification block. **DO NOT MERGE** — the lane was authorized to fix and +push only. + +## Verification (2026-09-02) + +Both full `packages/microcosm-build` suites — clean shell and the +polluted-Logbook shell finding 2 specifies — returned **outcome-identical** +results: 7116 passed, 47 skipped, 2 failed, out of 7165. + +The 2 failures are `test_release_target_parity.py::TestRegeneration` and are +**not this branch's**. They reproduce identically at `1f78847c` (this +branch's previous head) and at `origin/main` `47c74225`, both verified in a +throwaway worktree this session: + + ValueError: Ledger target reference + 'cbo.revenue_projection.ty2024.income_by_source.adjusted_gross_income.projected_amount' + assertion_policy='observed_only' does not allow resolved fact assertion + 'source_projection'. + +They run only where the pinned feed exists — +`~/PolicyEngine/_buildh-runtime/inputs/consumer_facts_buildn_v9_4.jsonl`, +outside the repo — and skip everywhere else, PR CI included. Worth a separate +issue: on a machine holding that feed, the committed +`cbo.revenue_projection.*` reference and the feed row it resolves to disagree +about assertion class. + +`uv run ruff format --check .` is red repo-wide (116 files) at this branch +*and* at its merge base: ruff 0.15.16's formatter against a repo pinned to +`ruff>=0.8`, whose CI lint lane runs `ruff check` only. Every file this +branch touches is format-clean before and after. ## What changed structurally, for the next reader