diff --git a/changelog.d/graph-amendments-11-13.added.md b/changelog.d/graph-amendments-11-13.added.md
new file mode 100644
index 000000000..3ba203486
--- /dev/null
+++ b/changelog.d/graph-amendments-11-13.added.md
@@ -0,0 +1 @@
+Graph interface amendments 11–14 (declared entrants, partitioned mass, declared tolerance, entrant-person strata) and their executor semantics: charter properties B6, B7, C5, and D6 go green; `compile_graph` refuses any later owner of the mass-partition column; readers see a column's producer tolerance through structural carriers; the whole `Capabilities` projection joins node identity and is compared on cache hits; the acceptance ratchet compares property identities against the baseline rather than per-file counts.
diff --git a/docs/graph-acceptance.md b/docs/graph-acceptance.md
index e5d422fad..d119576cf 100644
--- a/docs/graph-acceptance.md
+++ b/docs/graph-acceptance.md
@@ -45,6 +45,8 @@ its owner.
| B3 | **Storage-preserving patch.** Patching owned positions preserves the incumbent column's dtype (nullable `boolean` stays nullable `boolean`; float bits including negative zero survive) and leaves every non-owned position byte-identical. This is the WIC guard, made structural. | leg 2 §3.3 | same |
| B4 | **Inputs are immutable.** A kernel receives read-only views; an in-place write raises inside the kernel and the node fails. | leg 1 finding 5 | same |
| B5 | **Null means absence.** A node declares each owned cell as *produced* or *absent*. A kernel writing a non-null value into an absent-declared cell is rejected. | `DESIGN.md:128-134` | same |
+| B6 | **Entrants are declared.** An `EXPAND` node with `entrants=True` may return rows with null lineage; the executor requires the kernel to materialize every carried column for such a row (dtype-checked), records them as entrants rather than copies in the lineage receipt, and refuses null lineage on a node without the declaration. `entrants=True` with `mass='conserve'` is a compile error. | Dynamics: immigrant cohorts (microcosm-dynamics#412, #218) | Max's session; amendment 11 |
+| B7 | **Entrant persons carry their stratum.** An entrant row on the person entity takes its stratum from `KernelResult.strata` (indexed by its new id); an entrant person absent from it, a label for a copied or incumbent person, or a label for an unknown id rejects the node. Entrant persons join incumbent or entrant groups through the materialized membership columns, and the mass ledger counts them from the node that admits them. | Dynamics: immigrant cohorts are persons (microcosm-dynamics#412, #218) | Max's session; amendment 14 |
## C. Seeds and factorization
@@ -54,16 +56,18 @@ its owner.
| C2 | **Removal invariance.** Removing a node that nothing depends on, or adding a new leaf node, changes no other node's key or output. This is the `0347a009` replay: five targets removed, zero survivors re-modeled. | F5 | same |
| C3 | **Declared predecessors only.** A chained target's predictors are exactly its declared predecessors. The executor hands a kernel only its declared slices, so an undeclared read is impossible rather than merely detected. | F5, leg 3 §legibility | same |
| C4 | **Seed from identity.** A node's RNG seed is a pure function of its node key. Two nodes with identical declarations, inputs, and kernels in different graphs draw identical values. No positional RNG consumption exists anywhere in the shard (static check). | F4, `docs/spec-engine.md:254-282` | same |
+| C5 | **Tolerance is declared.** A kernel claiming `tolerance_bound` numerics without a `Tolerance` is refused at registration, and a bitwise kernel may not carry one. The tolerance is recorded in every receipt, and a kernel reading a cell sees the declared tolerance of the node that produced it in `KernelContext.tolerances`: a structural version (`FILTER`, `EXPAND`, `REWEIGHT`) carries a column's tolerance through unchanged, so a bitwise carrier neither tightens nor erases a producer's bound, and a rewrite sees the incumbent producer's. Where more than one node wrote rows of a column in a version — a producer plus an `EXPAND` kernel that materialized entrant rows, or a claimant that took them over — the reader sees the loosest declared tolerance among those writers (componentwise maximum of `rtol`, `atol`, `ulps`; a bitwise writer contributes none), and a claimant's `KernelContext.tolerances` includes the coordinates it claims; a gate comparing against anything else says so in its evidence. | H2 (arm64/x86 one-ulp weights); microcosm-dynamics#412 | Max's session; amendment 13 |
## D. Weights and mass
| Id | Property | Closes | Owner |
|---|---|---|---|
| D1 | **Weight transitions are typed nodes.** `design → importance → calibrated` are the only legal transitions; the executor rejects a regression and rejects a transition declared on inherited (non-explicit) weights. | F9 (leg 1 finding 1) | María / Max's session |
-| D2 | **Mass ledger.** Every population-changing node (select, concat, clone, reweight) emits a mass record with before/after totals and per-stratum mass. Under `conserve`, a stratum losing mass fails the node. `select` cannot drop mass silently. | F9 | same |
+| D2 | **Mass ledger.** Every population-changing node (select, concat, clone, reweight) emits a mass record with before/after totals and per-stratum mass. Mass is weighted person mass per stratum, within each declared partition (amendment 12). Under `conserve`, a stratum losing mass fails the node. An expansion that conserves its weight entity's mass while changing group composition changes person mass and must say so: `declared`, with a receipt stating the invariant it does hold (ruled 2026-09-02 on #844). `select` cannot drop mass silently. | F9 | same |
| D3 | **Cap anchored to design.** A calibration node's `max_weight_ratio` is asserted against the declared anchor across composed stages; a selection-then-refit chain that ships a record above `R × design` fails. | F9 (#493) | same |
| D4 | **Filters are binary.** A target filter containing NaN or a non-binary value is rejected at compile. | F9 | same |
| D5 | **Uncertainty travels.** A target's declared standard error reaches the calibration kernel's inputs; a kernel that ignores a declared `se` must say so in its capability record. | scoreboard row 5 (leg 1 finding 7) | same |
+| D6 | **Mass is partitioned.** With `Graph.mass_partition` set, the ledger reports per stratum within each partition value, `conserve` holds within each partition, and a node that moves mass between partitions under `conserve` fails. Every `CREATE` node declares the partition column with a partition dtype, and no later node may own it (write or rewrite), or compilation fails: a partition value is fixed when the row is created, because a reassignment with the total unchanged is invisible to every mass policy. A row contributes mass only to the partitions it exists in. | Dynamics: person-period residency (microcosm-dynamics#412) | Max's session; amendment 12 |
## E. Store and resume
@@ -201,6 +205,45 @@ Amendments so far (each re-locked):
(python storage); a population entering the graph with `object`
strings is normalized at `CREATE`. Parity fixtures compare identities
after the same normalization on the legacy side, and say so.
+11. **Entrants are declared.** `Node.entrants=True` (EXPAND only) lets a
+ kernel add rows that copy no base row: their lineage is null, the
+ kernel materializes every carried column for them, the executor
+ records them as entrants, and the node's mass policy cannot be
+ `conserve`. Raised by the dynamics program (immigrant cohorts through
+ the scheduled-entries seam, microcosm-dynamics#412 / #218); Max ruled
+ go 2026-09-02; adopted 2026-09-02.
+12. **Mass is partitioned.** `Graph.mass_partition = (entity, column)`
+ partitions mass accounting (per stratum within each partition value;
+ `conserve` per partition). Every `CREATE` node declares the column
+ with a dtype in `PARTITION_DTYPES`, and `compile_graph` refuses any
+ later owner of it, rewrite or not: a partition value is fixed when
+ the row is created (review finding, 2026-09-02). The field is
+ normative: the executor folds it into every structural node's key, so
+ structural keys move once when a graph adopts it. Raised by the
+ dynamics program for person-period residency; adopted 2026-09-02.
+13. **Tolerance is declared.** `Capabilities.tolerance: Tolerance | None`
+ (`rtol`, `atol`, `ulps`) is required for `tolerance_bound` kernels and
+ forbidden for bitwise ones; `KernelContext.tolerances` hands each
+ reader the declared tolerance of every input cell's producer, resolved
+ through structural carriers to the node that wrote the values (a
+ rewrite sees the incumbent producer's; where several nodes wrote rows
+ of one column — entrant materialization, claims — the loosest declared
+ tolerance among them). The whole `Capabilities`
+ projection, tolerance included, is part of a node's identity and is
+ compared on every cache hit. Raised by the H2 parity finding (root
+ weights differ by one ulp between arm64 and x86) and the dynamics
+ review; adopted 2026-09-02.
+
+14. **Entrant persons carry their stratum.** `KernelResult.strata` (EXPAND
+ kernels on an `entrants=True` node only) names the stratum of every
+ entrant person by its new id; the executor requires exactly the entrant
+ persons there. Raised by the implementation of amendment 11, which
+ found the frozen result had no channel for a new person's mandatory
+ stratum and left person entrants fail-closed; adopted 2026-09-02.
+
+Adding a normative field with a default changes the canonical projection
+of every node that carries it, so node keys moved with amendments 11 and
+13's sibling field `entrants`; no released artifact pins a graph key yet.
## Ownership
diff --git a/docs/graph-interface.lock b/docs/graph-interface.lock
index 994c05b9d..17f183245 100644
--- a/docs/graph-interface.lock
+++ b/docs/graph-interface.lock
@@ -1,2 +1,2 @@
-b6fbf50e0253e6f04a2684fcac295fb27a4608f6a272bedb86c43f087fd36b8e decl.py
-8ac8328c086bfbea0e36032e9c7fa9cb1ed62f63a1036bca4eb528c11b9b1890 kernel.py
+d3d6ee7e5a2b758f71268f35d1c267e67781bf4be40fd9593f7332e15a27280f decl.py
+7a038cc6d64292524f0340ecaf94cdc73bfc48155a68b7b679b52e3548fd9e6c kernel.py
diff --git a/packages/microcosm-fit/src/microcosm/fit/kernels.py b/packages/microcosm-fit/src/microcosm/fit/kernels.py
index c39c9a83e..308f9e4b7 100644
--- a/packages/microcosm-fit/src/microcosm/fit/kernels.py
+++ b/packages/microcosm-fit/src/microcosm/fit/kernels.py
@@ -27,6 +27,7 @@
KernelResult,
Numeric,
SeedSource,
+ Tolerance,
source_hash,
)
@@ -48,6 +49,18 @@
)
"""Distributions whose versions form part of ``fit.qrf@1``'s identity."""
+#: Provisional relative budget for ``fit.qrf@1``, declared because the kernel
+#: is not bitwise across platforms (charter C5, H2's one-ulp root-weight
+#: finding) and a ``tolerance_bound`` kernel must declare something. It is
+#: NOT a measured bound: the only measurement so far (2026-09-02) is one
+#: 12-output, positive-only H1 fixture that was bit-identical between native
+#: arm64 and x86_64 under Rosetta, which supports nothing tighter than "small".
+#: Tightening this (for example to ``Tolerance(ulps=1)``) requires native-x86,
+#: multi-seed, multi-regime, near-tie, and mixed-sign coverage first; until
+#: then the declared budget errs loose so no gate treats an unmeasured claim
+#: as a guarantee. Changing it moves the kernel's identity (H1 pins).
+FIT_QRF_TOLERANCE = Tolerance(rtol=1e-6)
+
QRF_EXECUTOR_SEED_HIGH = 2**31 - 1
"""Exclusive upper bound for the one seed drawn from ``KernelContext.rng``."""
@@ -114,6 +127,7 @@ def __init__(self, seed_source: SeedSource = SeedSource.EXECUTOR) -> None:
numeric=Numeric.TOLERANCE_BOUND,
seed_source=seed_source,
dependencies=FIT_QRF_DEPENDENCIES,
+ tolerance=FIT_QRF_TOLERANCE,
)
def implementation_hash(self) -> str:
diff --git a/packages/microcosm-fit/tests/test_kernels.py b/packages/microcosm-fit/tests/test_kernels.py
index 356705a7b..a90e0cd34 100644
--- a/packages/microcosm-fit/tests/test_kernels.py
+++ b/packages/microcosm-fit/tests/test_kernels.py
@@ -15,6 +15,7 @@
from microcosm.fit import fit as fit_qrf
from microcosm.fit.kernels import (
FIT_QRF_DEPENDENCIES,
+ FIT_QRF_TOLERANCE,
QRF_EXECUTOR_KERNEL,
QRF_EXECUTOR_SEED_HIGH,
QRF_PARAM_KERNEL,
@@ -233,6 +234,7 @@ def test_capabilities_protocol_and_wrapped_source_hash() -> None:
numeric=Numeric.TOLERANCE_BOUND,
seed_source=SeedSource.PARAM,
dependencies=FIT_QRF_DEPENDENCIES,
+ tolerance=FIT_QRF_TOLERANCE,
)
assert QRF_EXECUTOR_KERNEL.capabilities.seed_source is SeedSource.EXECUTOR
assert QRF_PARAM_KERNEL.implementation_hash() == source_hash(
diff --git a/packages/microcosm-graph/src/microcosm/graph/__init__.py b/packages/microcosm-graph/src/microcosm/graph/__init__.py
index 696158e73..81e919c80 100644
--- a/packages/microcosm-graph/src/microcosm/graph/__init__.py
+++ b/packages/microcosm-graph/src/microcosm/graph/__init__.py
@@ -13,6 +13,7 @@
DTYPES,
GATE_OUTCOMES,
MASS_POLICIES,
+ PARTITION_DTYPES,
ROWS_ALL,
WEIGHT_KINDS,
CompiledGraph,
@@ -46,6 +47,7 @@
KernelRole,
Numeric,
SeedSource,
+ Tolerance,
source_hash,
)
@@ -54,6 +56,7 @@
"DTYPES",
"GATE_OUTCOMES",
"MASS_POLICIES",
+ "PARTITION_DTYPES",
"ROWS_ALL",
"WEIGHT_KINDS",
"Capabilities",
@@ -80,11 +83,13 @@
"Ownership",
"Param",
"Population",
+ "PopulationView",
"PopulationError",
"ResumePolicy",
"RunManifest",
"SOURCE_CODECS",
"SeedSource",
+ "Tolerance",
"Slice",
"SourceCodec",
"SourceCodecRegistry",
@@ -133,7 +138,7 @@ def _check_frame_version() -> None:
)
from .executor import NodeRejected, run_graph # noqa: E402
from .explain import explain_html # noqa: E402
-from .manifest import Decision, NodeReceipt, RunManifest # noqa: E402
+from .manifest import Decision, NodeReceipt, PopulationView, RunManifest # noqa: E402
from .population import MassRecord, Population, PopulationError # noqa: E402
from .serialize import graph_from_json, graph_to_json # noqa: E402
from .store import ( # noqa: E402
diff --git a/packages/microcosm-graph/src/microcosm/graph/decl.py b/packages/microcosm-graph/src/microcosm/graph/decl.py
index b7460730b..4fb953ac8 100644
--- a/packages/microcosm-graph/src/microcosm/graph/decl.py
+++ b/packages/microcosm-graph/src/microcosm/graph/decl.py
@@ -24,6 +24,17 @@
known at compile time: a mask that is not ``bool`` or ``boolean`` is a
compile error (charter D4). Nulls inside a nullable mask are a run-time
rejection by the executor.
+- An ``EXPAND`` node copies rows: every new row names the base row it
+ copies, so lineage is total. A node that declares ``entrants=True`` may
+ also add rows that copy nothing (births not patterned on a parent,
+ immigrant cohorts); the kernel materializes every carried column for
+ such a row, the executor records them as entrants rather than copies,
+ and the node cannot claim to conserve mass (amendment 11).
+- Mass is accounted as weighted person mass per stratum. A graph may name
+ a partition column (:attr:`Graph.mass_partition`, e.g. a period on a
+ person-period population); the ledger then reports per stratum within
+ each partition and ``conserve`` holds within each partition, so a row
+ contributes mass only to the partitions it exists in (amendment 12).
This file is a frozen interface (see ``docs/graph-acceptance.md``).
"""
@@ -42,6 +53,7 @@
"GATE_OUTCOMES",
"MASK_DTYPES",
"MASS_POLICIES",
+ "PARTITION_DTYPES",
"ROWS_ALL",
"WEIGHT_KINDS",
"CompiledGraph",
@@ -82,6 +94,9 @@
#: Mass policies a weight transition or structural node may declare.
MASS_POLICIES = frozenset({"conserve", "free", "declared"})
+#: The dtypes a mass-partition column may have.
+PARTITION_DTYPES = frozenset({"int32", "int64", "string"})
+
#: The closed set of gate outcomes (charter F4). ``unreached`` is also the
#: outcome of a release whose required human decisions are absent.
GATE_OUTCOMES = ("pass", "fail", "evidence_absent", "not_applicable", "unreached")
@@ -269,6 +284,11 @@ class Node:
sources: Names of :class:`SourceRef` entries this node reads.
weights: A declared weight-kind transition, if any.
mass: Mass policy for structural nodes that change rows or weights.
+ entrants: ``EXPAND`` nodes only: the kernel may add rows that copy
+ no base row. Such a row has null lineage, the kernel supplies
+ every carried column for it, and the executor records it as an
+ entrant. Entrants add mass, so the node's mass policy cannot be
+ ``conserve``.
description: Descriptive; never hashed.
citation: Descriptive; never hashed.
"""
@@ -286,6 +306,7 @@ class Node:
mass: str = "conserve"
description: str = ""
citation: str = ""
+ entrants: bool = False
def __post_init__(self) -> None:
_nonempty("Node.id", self.id)
@@ -333,6 +354,17 @@ def __post_init__(self) -> None:
)
elif self.base is not None:
raise GraphError(f"Node {self.id!r}: only structural nodes have a base.")
+ if not isinstance(self.entrants, bool):
+ raise GraphError(f"Node {self.id!r}: entrants must be a boolean.")
+ if self.entrants and self.structural is not StructuralDelta.EXPAND:
+ raise GraphError(
+ f"Node {self.id!r}: only an EXPAND node may admit entrants."
+ )
+ if self.entrants and self.mass == "conserve":
+ raise GraphError(
+ f"Node {self.id!r}: entrants add mass, so an entrant-admitting "
+ "node cannot declare mass='conserve'."
+ )
if self.weights is not None and self.structural is not StructuralDelta.REWEIGHT:
raise GraphError(
f"Node {self.id!r}: a weight transition changes the population "
@@ -386,11 +418,18 @@ class Graph:
into node keys (a node's identity is its computation).
sources: External inputs by name.
nodes: Every node. Declaration order carries no meaning.
+ mass_partition: ``(entity, column)`` of a column that partitions
+ mass accounting, or ``None``. When set, every ``CREATE`` node
+ declares the column with a dtype in :data:`PARTITION_DTYPES`,
+ the executor's ledger reports per stratum within each partition
+ value, and ``conserve`` holds within each partition. Normative:
+ it enters the key of every structural node.
"""
country: str
sources: tuple[SourceRef, ...]
nodes: tuple[Node, ...]
+ mass_partition: tuple[str, str] | None = None
def __post_init__(self) -> None:
_nonempty("Graph.country", self.country)
@@ -398,6 +437,21 @@ def __post_init__(self) -> None:
raise GraphError("Graph repeats a source name.")
if len({n.id for n in self.nodes}) != len(self.nodes):
raise GraphError("Graph repeats a node id.")
+ if self.mass_partition is not None:
+ if (
+ not isinstance(self.mass_partition, tuple)
+ or len(self.mass_partition) != 2
+ or not all(isinstance(part, str) for part in self.mass_partition)
+ ):
+ raise GraphError(
+ "Graph.mass_partition must be an (entity, column) pair of strings."
+ )
+ _nonempty("Graph.mass_partition entity", self.mass_partition[0])
+ _nonempty("Graph.mass_partition column", self.mass_partition[1])
+
+ def normative(self) -> dict[str, object]:
+ """The graph-level facts that enter every structural node's key."""
+ return {"mass_partition": self.mass_partition}
def node(self, node_id: str) -> Node:
for node in self.nodes:
@@ -436,8 +490,9 @@ def compile_graph(graph: Graph) -> CompiledGraph:
GraphError: A cell with two owners or none (ownership is total and
exclusive), an unknown source or population, a structural node
whose base is not structural, a row mask whose declared dtype is
- not boolean, a cycle, or a graph with several structural nodes
- and a node that omits ``population``.
+ not boolean, a mass-partition column that a ``CREATE`` node does
+ not declare with a partition dtype, a cycle, or a graph with
+ several structural nodes and a node that omits ``population``.
"""
by_id = {node.id: node for node in graph.nodes}
@@ -494,6 +549,37 @@ def compile_graph(graph: Graph) -> CompiledGraph:
owners[key] = node.id
dtypes[key] = owned.dtype
+ if graph.mass_partition is not None:
+ entity, column = graph.mass_partition
+ for node in structural:
+ if node.structural is not StructuralDelta.CREATE:
+ continue
+ dtype = dtypes.get((node.id, entity, column))
+ if dtype is None:
+ raise GraphError(
+ f"Graph.mass_partition names {entity}.{column}, which CREATE "
+ f"node {node.id!r} does not declare; partitions must exist "
+ "from the first version."
+ )
+ if dtype not in PARTITION_DTYPES:
+ raise GraphError(
+ f"Graph.mass_partition {entity}.{column} is declared {dtype!r}; "
+ f"a partition column must be one of {sorted(PARTITION_DTYPES)}."
+ )
+ # A partition value is fixed when a row is created. Any later owner of
+ # the column, rewrite or not, could move mass between partitions with
+ # the total unchanged, which no mass policy can see; refuse it here.
+ for (version, owner_entity, owner_column), owner_id in sorted(owners.items()):
+ if (owner_entity, owner_column) != (entity, column):
+ continue
+ if by_id[owner_id].structural is not StructuralDelta.CREATE:
+ raise GraphError(
+ f"Node {owner_id!r} owns mass partition {entity}.{column} in "
+ f"version {version!r}; a partition value is fixed by the CREATE "
+ "node that admits the row, and no later node may write or "
+ "rewrite it."
+ )
+
def declared_dtype(version: str, entity: str, column: str) -> str | None:
"""The owner-declared dtype of a column as visible in ``version``."""
while True:
diff --git a/packages/microcosm-graph/src/microcosm/graph/executor.py b/packages/microcosm-graph/src/microcosm/graph/executor.py
index 6bb035966..c645e9d71 100644
--- a/packages/microcosm-graph/src/microcosm/graph/executor.py
+++ b/packages/microcosm-graph/src/microcosm/graph/executor.py
@@ -34,8 +34,10 @@
KernelRegistry,
KernelResult,
KernelRole,
+ Tolerance,
)
from .keys import (
+ _capabilities_projection,
artifact_key,
frame_key,
node_key,
@@ -46,7 +48,9 @@
from .manifest import Decision, NodeReceipt, RunManifest
from .population import (
Population,
+ entrant_strata_receipt,
expand_lineage_receipt,
+ mass_record_receipt,
patch,
restore_cached_expand,
weight_cap_receipt,
@@ -80,18 +84,6 @@ def _opaque_artifact_key(key: str, name: str) -> str:
return sha256_domain("node-artifact", canonical_json((key, name)))
-def _capabilities_payload(capabilities: Capabilities) -> dict[str, object]:
- return {
- "determinism": capabilities.determinism.value,
- "numeric": capabilities.numeric.value,
- "seed_source": capabilities.seed_source.value,
- "structural": capabilities.structural.value,
- "role": capabilities.role.value,
- "consumes_se": capabilities.consumes_se,
- "dependencies": list(capabilities.dependencies),
- }
-
-
def _normal_json_mapping(value: Mapping[str, object], label: str) -> dict[str, object]:
"""Validate and detach a descriptive mapping through canonical JSON."""
@@ -400,6 +392,7 @@ def _project_context(
*,
key: str,
sources: Mapping[str, Path],
+ tolerances: Mapping[tuple[str, str], Tolerance | None],
) -> KernelContext:
if population is None:
return KernelContext(
@@ -410,6 +403,7 @@ def _project_context(
params=node.params,
rng=np.random.default_rng(seed(key)),
sources=MappingProxyType({name: sources[name] for name in node.sources}),
+ tolerances=tolerances,
)
frame = population.frame
@@ -519,9 +513,70 @@ def _project_context(
params=node.params,
rng=np.random.default_rng(seed(key)),
sources=MappingProxyType({name: sources[name] for name in node.sources}),
+ tolerances=tolerances,
)
+def _input_tolerances(
+ compiled: CompiledGraph,
+ node_id: str,
+ kernels: KernelRegistry,
+) -> Mapping[tuple[str, str], Tolerance | None]:
+ """Resolve explicit inputs and rewrite incumbents as compilation does."""
+
+ node = compiled.graph.node(node_id)
+ if node.structural is StructuralDelta.CREATE:
+ return MappingProxyType({})
+ input_version = (
+ compiled.versions[node_id]
+ if node.structural is StructuralDelta.NONE
+ else node.base
+ )
+ assert input_version is not None
+ rewritten = {
+ (owned.entity, owned.column) for owned in node.outputs if owned.rewrite
+ }
+ coordinates = rewritten | {
+ (slice_.entity, column) for slice_ in node.inputs for column in slice_.columns
+ }
+ resolved: dict[tuple[str, str], Tolerance | None] = {}
+ for coordinate in sorted(coordinates):
+ entity, column = coordinate
+ if coordinate in rewritten:
+ # The incumbent a rewrite receives was produced somewhere in the
+ # version's base chain, never in the version the rewrite opens.
+ start = (
+ compiled.graph.node(input_version).base
+ if node.structural is StructuralDelta.NONE
+ else input_version
+ )
+ else:
+ start = input_version
+ assert start is not None
+ producer = compiled.graph.node(_producer_of(compiled, start, entity, column))
+ resolved[coordinate] = kernels.get(producer.kernel).capabilities.tolerance
+ return MappingProxyType(resolved)
+
+
+def _producer_of(
+ compiled: CompiledGraph, version: str, entity: str, column: str
+) -> str:
+ """The node whose kernel wrote ``entity.column`` as seen from ``version``.
+
+ A structural version carries the columns it does not own from its base,
+ so the tolerance a reader sees is the producer's, not the carrier's: a
+ bitwise ``FILTER`` in between neither tightens nor erases it (C5).
+ """
+ while True:
+ owner = compiled.owners.get((version, entity, column))
+ if owner is not None:
+ return owner
+ holder = compiled.graph.node(version)
+ if holder.structural is StructuralDelta.CREATE or holder.base is None:
+ return version
+ version = holder.base
+
+
def _validate_series(
node: Node,
owned: Owned,
@@ -645,6 +700,15 @@ def _validate_result(
raise NodeRejected(f"Node {node.id!r} result.artifacts is not a mapping.")
if not isinstance(result.receipt, Mapping):
raise NodeRejected(f"Node {node.id!r} result.receipt is not a mapping.")
+ if result.strata is not None and not isinstance(result.strata, pd.Series):
+ raise NodeRejected(f"Node {node.id!r} result.strata is not a Series.")
+ if result.strata is not None and (
+ cache_hit or node.structural is not StructuralDelta.EXPAND or not node.entrants
+ ):
+ raise NodeRejected(
+ f"Node {node.id!r} returned entrant strata outside a fresh "
+ "entrants=True EXPAND."
+ )
if kernel_capabilities.structural is not node.structural:
raise NodeRejected(
f"Node {node.id!r} declares structural={node.structural.value!r}, but "
@@ -763,10 +827,18 @@ def _validate_result(
assert result.expand is not None
try:
receipt["expand"] = expand_lineage_receipt(result.expand)
+ assert population is not None
+ strata_receipt = entrant_strata_receipt(
+ population.frame, node, result.expand, result.strata
+ )
except (TypeError, ValueError) as error:
raise NodeRejected(
- f"EXPAND node {node.id!r} returned malformed lineage: {error}"
+ f"EXPAND node {node.id!r} returned malformed lineage or "
+ f"entrant strata: {error}"
) from error
+ receipt.pop("entrant_strata", None)
+ if strata_receipt is not None:
+ receipt["entrant_strata"] = strata_receipt
if kernel_capabilities.role is KernelRole.GATE:
outcome = receipt.get("outcome")
if outcome not in GATE_OUTCOMES:
@@ -777,6 +849,86 @@ def _validate_result(
return receipt, artifacts
+def _validate_entrant_materialization_contract(
+ compiled: CompiledGraph,
+ node: Node,
+ population: Population | None,
+ receipt: Mapping[str, object],
+) -> None:
+ """Require every entrant's carried data cells to have downstream claims."""
+
+ if not node.entrants or population is None:
+ return
+ raw_expand = receipt.get("expand")
+ if not isinstance(raw_expand, Mapping):
+ return # the ordinary EXPAND validation reports the malformed receipt
+ entrant_entities: set[str] = set()
+ for entity, entries in raw_expand.items():
+ if not isinstance(entity, str) or not isinstance(entries, list):
+ continue
+ if any(
+ isinstance(entry, list) and len(entry) == 2 and entry[1] is None
+ for entry in entries
+ ):
+ entrant_entities.add(entity)
+
+ frame = population.frame
+ for entity in sorted(entrant_entities):
+ if entity not in frame.entities:
+ continue # lineage validation supplies the node-naming rejection
+ structural = set(_structural_columns(frame, entity))
+ if (
+ compiled.graph.mass_partition is not None
+ and compiled.graph.mass_partition[0] == entity
+ ):
+ structural.add(compiled.graph.mass_partition[1])
+ for column in frame.table(entity).columns:
+ column = str(column)
+ if column in structural:
+ continue
+ coordinate = (entity, column)
+ claimant_id = compiled.owners.get((node.id, entity, column))
+ if claimant_id is None:
+ raise NodeRejected(
+ f"EXPAND node {node.id!r} entrant cell {entity}.{column} "
+ "has no materialized_expand_outputs ownership claim."
+ )
+ claimant = compiled.graph.node(claimant_id)
+ claimed = claimant.params.get("materialized_expand_outputs", ())
+ spelling = f"{entity}.{column}"
+ output = next(
+ (
+ owned
+ for owned in claimant.outputs
+ if (owned.entity, owned.column) == coordinate
+ ),
+ None,
+ )
+ if (
+ not isinstance(claimed, tuple)
+ or spelling not in claimed
+ or output is None
+ or output.rewrite
+ ):
+ raise NodeRejected(
+ f"EXPAND node {node.id!r} entrant cell {spelling} is not "
+ f"declared through node {claimant_id!r}'s "
+ "materialized_expand_outputs."
+ )
+ if output.rows != ROWS_ALL:
+ raise NodeRejected(
+ f"EXPAND node {node.id!r} entrant cell {spelling} is claimed "
+ f"through masked rows {output.rows!r}; materialization bridge "
+ "claims must use rows='all'."
+ )
+ carried_dtype = _dtype_token(frame.table(entity)[column])
+ if output.dtype != carried_dtype:
+ raise NodeRejected(
+ f"EXPAND node {node.id!r} entrant cell {spelling} is claimed "
+ f"as {output.dtype!r}; its carried dtype is {carried_dtype!r}."
+ )
+
+
def _create_population(node: Node, frame: Frame) -> Population:
# Entity ids and membership columns are structural Frame columns rather
# than declaration-owned data cells, but Population ownership is total
@@ -816,7 +968,20 @@ def _apply_result(
population: Population | None,
*,
cache_hit: bool = False,
+ mass_partition: tuple[str, str] | None = None,
) -> Population:
+ if (
+ mass_partition is not None
+ and node.structural is StructuralDelta.NONE
+ and any(
+ (owned.entity, owned.column) == mass_partition for owned in node.outputs
+ )
+ ):
+ entity, column = mass_partition
+ raise NodeRejected(
+ f"Node {node.id!r} cannot own mass partition {entity}.{column}; "
+ "partition values are fixed by the structural population."
+ )
if node.structural is StructuralDelta.CREATE:
assert result.frame is not None
return _create_population(node, result.frame)
@@ -827,7 +992,9 @@ def _apply_result(
and result.frame is not None
):
try:
- return restore_cached_expand(population, node, result)
+ return restore_cached_expand(
+ population, node, result, mass_partition=mass_partition
+ )
except (TypeError, ValueError) as error:
raise NodeRejected(
f"Node {node.id!r} cached EXPAND rejected: {error}"
@@ -860,7 +1027,7 @@ def _apply_result(
receipt=result.receipt,
)
try:
- return patch(population, node, result)
+ return patch(population, node, result, mass_partition=mass_partition)
except NodeRejected:
raise
except (TypeError, ValueError) as error:
@@ -979,7 +1146,7 @@ def _write_node(
"node_key": key,
"kernel_ref": node.kernel,
"kernel_impl_hash": kernel_impl_hash,
- "capabilities": _capabilities_payload(capabilities),
+ "capabilities": _capabilities_projection(capabilities),
"receipt": dict(receipt),
"columns": column_entries,
"frame_key": stored_frame_key,
@@ -996,7 +1163,12 @@ def _write_node(
def _require_record_shape(
- raw: object, node: Node, *, key: str, kernel_impl_hash: str
+ raw: object,
+ node: Node,
+ *,
+ key: str,
+ kernel_impl_hash: str,
+ capabilities: Capabilities,
) -> dict[str, object]:
if not isinstance(raw, dict):
raise StoreCorrupt(f"Cached receipt for node {node.id!r} is not an object.")
@@ -1035,6 +1207,12 @@ def _require_record_shape(
f"Cached receipt identity for node {node.id!r} is {actual!r}, "
f"not {expected!r}."
)
+ expected_capabilities = _capabilities_projection(capabilities)
+ if raw["capabilities"] != expected_capabilities:
+ raise StoreMiss(
+ f"Cached receipt capabilities for node {node.id!r} disagree with "
+ "the registered kernel contract."
+ )
return raw
@@ -1053,9 +1231,16 @@ def _load_record(
*,
key: str,
kernel_impl_hash: str,
+ capabilities: Capabilities,
) -> dict[str, object]:
raw = store.load_json(_cache_record_key(key))
- return _require_record_shape(raw, node, key=key, kernel_impl_hash=kernel_impl_hash)
+ return _require_record_shape(
+ raw,
+ node,
+ key=key,
+ kernel_impl_hash=kernel_impl_hash,
+ capabilities=capabilities,
+ )
def _preflight_record(store: ContentStore, record: Mapping[str, object]) -> None:
@@ -1252,6 +1437,7 @@ def _all_node_keys(
keys,
implementation,
source_keys,
+ kernel_capabilities=kernel.capabilities,
)
return keys, implementations
@@ -1261,6 +1447,7 @@ def _preflight_require(
store: ContentStore,
keys: Mapping[str, str],
implementations: Mapping[str, str],
+ kernels: KernelRegistry,
) -> None:
missing: list[str] = []
for node_id in compiled.order:
@@ -1271,6 +1458,7 @@ def _preflight_require(
node,
key=keys[node_id],
kernel_impl_hash=implementations[node_id],
+ capabilities=kernels.get(node.kernel).capabilities,
)
_preflight_record(store, record)
except StoreMiss:
@@ -1309,7 +1497,7 @@ def run_graph(
source_paths, source_keys = _source_paths_and_keys(compiled, sources, store)
keys, implementations = _all_node_keys(compiled, kernels, source_keys)
if resume == "require":
- _preflight_require(compiled, store, keys, implementations)
+ _preflight_require(compiled, store, keys, implementations, kernels)
populations: dict[str, Population] = {}
receipts: dict[str, NodeReceipt] = {}
@@ -1345,6 +1533,7 @@ def run_graph(
node,
key=key,
kernel_impl_hash=implementation,
+ capabilities=kernel.capabilities,
)
result, manifest_artifacts = _load_cached_result(
store, node, incumbent, record
@@ -1355,7 +1544,13 @@ def run_graph(
raise
if result is None:
- context = _project_context(node, incumbent, key=key, sources=source_paths)
+ context = _project_context(
+ node,
+ incumbent,
+ key=key,
+ sources=source_paths,
+ tolerances=_input_tolerances(compiled, node_id, kernels),
+ )
before = _context_digest(context)
try:
result = kernel.run(context)
@@ -1385,6 +1580,9 @@ def run_graph(
incumbent,
cache_hit=hit,
)
+ _validate_entrant_materialization_contract(
+ compiled, node, incumbent, normalized_receipt
+ )
if kernel.capabilities.role is KernelRole.RELEASE:
derived_tier, gate_ids = _release_tier(compiled, node_id, receipts)
_validate_release_tier(node, result, derived_tier)
@@ -1393,8 +1591,37 @@ def run_graph(
"pass" if derived_tier == "certified" else "fail"
)
normalized_receipt["gate_ancestry"] = list(gate_ids)
- normalized_receipt["capabilities"] = _capabilities_payload(kernel.capabilities)
- updated = _apply_result(node, result, incumbent, cache_hit=hit)
+ normalized_receipt["capabilities"] = _capabilities_projection(
+ kernel.capabilities
+ )
+ updated = _apply_result(
+ node,
+ result,
+ incumbent,
+ cache_hit=hit,
+ mass_partition=compiled.graph.mass_partition,
+ )
+ author_mass = compiled.graph.mass_partition is not None or (
+ node.structural is StructuralDelta.EXPAND
+ and node.entrants
+ and "entrant_strata" in normalized_receipt
+ )
+ if author_mass and node.structural not in {
+ StructuralDelta.NONE,
+ StructuralDelta.CREATE,
+ }:
+ existing_mass = normalized_receipt.get("mass", {})
+ if not isinstance(existing_mass, Mapping): # defended by mass validation
+ raise NodeRejected(
+ f"Node {node.id!r} receipt['mass'] is not a mapping."
+ )
+ try:
+ authored_mass = mass_record_receipt(updated.mass_ledger[-1])
+ except (TypeError, ValueError) as error:
+ raise NodeRejected(
+ f"Node {node.id!r} mass receipt rejected: {error}"
+ ) from error
+ normalized_receipt["mass"] = {**existing_mass, **authored_mass}
normalized_receipt.update(weight_cap_receipt(updated, node))
cache_receipt = normalized_receipt
run_receipt = dict(cache_receipt)
diff --git a/packages/microcosm-graph/src/microcosm/graph/explain.py b/packages/microcosm-graph/src/microcosm/graph/explain.py
index 2bc4f8996..3eb8292bb 100644
--- a/packages/microcosm-graph/src/microcosm/graph/explain.py
+++ b/packages/microcosm-graph/src/microcosm/graph/explain.py
@@ -460,6 +460,7 @@ def _render_graph(
def _capabilities(receipt: NodeReceipt) -> dict[str, object]:
capabilities = receipt.capabilities
+ tolerance = capabilities.tolerance
return {
"determinism": _value(capabilities.determinism),
"numeric": _value(capabilities.numeric),
@@ -468,6 +469,15 @@ def _capabilities(receipt: NodeReceipt) -> dict[str, object]:
"role": _value(capabilities.role),
"consumes_se": capabilities.consumes_se,
"dependencies": capabilities.dependencies,
+ "tolerance": (
+ None
+ if tolerance is None
+ else {
+ "rtol": tolerance.rtol,
+ "atol": tolerance.atol,
+ "ulps": tolerance.ulps,
+ }
+ ),
}
@@ -995,13 +1005,16 @@ def _mass_payload(
) -> dict[str, object] | None:
raw = receipt.receipt.get("mass")
if isinstance(raw, Mapping):
- return {
+ payload = {
"before": raw.get("before"),
"after": raw.get("after"),
"stratum_before": raw.get("stratum_before", {}),
"stratum_after": raw.get("stratum_after", {}),
"policy": raw.get("policy", node.mass),
}
+ if isinstance(raw.get("partition"), Mapping):
+ payload["partition"] = raw["partition"]
+ return payload
for record in reversed(manifest.mass_ledgers.get(node.id, ())):
if record.node_id == node.id:
return {
diff --git a/packages/microcosm-graph/src/microcosm/graph/kernel.py b/packages/microcosm-graph/src/microcosm/graph/kernel.py
index fbe66b2a5..a90afa680 100644
--- a/packages/microcosm-graph/src/microcosm/graph/kernel.py
+++ b/packages/microcosm-graph/src/microcosm/graph/kernel.py
@@ -17,7 +17,10 @@
- ``EXPAND`` returns the clone lineage as :attr:`KernelResult.expand` (per
entity, new ids to the source ids they copy) plus the new weights; the
executor carries every column from the source rows, records the lineage
- in the receipt, and records mass.
+ in the receipt, and records mass. A node declared ``entrants=True`` may
+ also add rows with null lineage; the kernel then materializes their
+ columns, and for entrant persons their stratum through
+ :attr:`KernelResult.strata` (amendments 11 and 14).
- ``REWEIGHT`` (and any node with a declared weight transition) returns
:attr:`KernelResult.weights`; the executor validates the kind transition
and the mass policy.
@@ -30,6 +33,12 @@
is derived from the gate verdicts in its ancestry, and its receipt reports
``unreached`` when a required human decision is absent from the run).
+Numbers carry their own contract. A kernel whose :class:`Numeric` claim is
+``tolerance_bound`` declares a :class:`Tolerance`; the executor records it
+in the receipt and hands every reader the declared tolerance of each input
+cell's owner through :attr:`KernelContext.tolerances`, so a gate compares
+against a declaration rather than a guess (amendment 13).
+
This file is a frozen interface (see ``docs/graph-acceptance.md``).
"""
@@ -63,6 +72,7 @@
"KernelRole",
"Numeric",
"SeedSource",
+ "Tolerance",
"source_hash",
]
@@ -82,6 +92,51 @@ class Numeric(StrEnum):
TOLERANCE_BOUND = "tolerance_bound"
+@dataclass(frozen=True)
+class Tolerance:
+ """How far a ``tolerance_bound`` kernel's numbers may move between runs.
+
+ Two values agree when they are within ``atol`` absolutely, or within
+ ``rtol`` relatively, or within ``ulps`` last-place units of each other.
+ A bitwise kernel declares no tolerance at all.
+
+ Attributes:
+ rtol: Relative tolerance; non-negative and finite.
+ atol: Absolute tolerance; non-negative and finite.
+ ulps: Units in the last place; non-negative.
+ """
+
+ rtol: float = 0.0
+ atol: float = 0.0
+ ulps: int = 0
+
+ def __post_init__(self) -> None:
+ for name in ("rtol", "atol"):
+ value = getattr(self, name)
+ if isinstance(value, bool) or not isinstance(value, int | float):
+ raise ValueError(f"Tolerance.{name} must be a number.")
+ try:
+ # Keys and manifests carry the float; an integer too large for
+ # one would only fail later, at identity time.
+ as_float = float(value)
+ except OverflowError as error:
+ raise ValueError(
+ f"Tolerance.{name} must be representable as a finite float."
+ ) from error
+ if not (as_float >= 0.0) or as_float == float("inf"):
+ raise ValueError(f"Tolerance.{name} must be non-negative and finite.")
+ object.__setattr__(self, name, as_float)
+ if isinstance(self.ulps, bool) or not isinstance(self.ulps, int):
+ raise ValueError("Tolerance.ulps must be an integer.")
+ if self.ulps < 0:
+ raise ValueError("Tolerance.ulps must be non-negative.")
+ if self.rtol == 0.0 and self.atol == 0.0 and self.ulps == 0:
+ raise ValueError(
+ "Tolerance must allow some movement; a bitwise kernel declares "
+ "no tolerance instead."
+ )
+
+
class SeedSource(StrEnum):
"""Where a seeded kernel takes its randomness from."""
@@ -113,6 +168,9 @@ class Capabilities:
standard errors. A kernel that ignores them says so here.
dependencies: Installed distributions whose versions enter the
implementation hash.
+ tolerance: Required when ``numeric`` is ``tolerance_bound`` and
+ forbidden otherwise: how far the kernel's numbers may move
+ between runs or machines.
"""
determinism: Determinism
@@ -122,6 +180,37 @@ class Capabilities:
role: KernelRole = KernelRole.COMPUTE
consumes_se: bool = False
dependencies: tuple[str, ...] = ()
+ tolerance: Tolerance | None = None
+
+ def __post_init__(self) -> None:
+ # Every field is validated here, so a registered contract is a real
+ # one: a string that spells an enum member does not pass as the member.
+ for name, kind in (
+ ("determinism", Determinism),
+ ("numeric", Numeric),
+ ("seed_source", SeedSource),
+ ("structural", StructuralDelta),
+ ("role", KernelRole),
+ ):
+ if not isinstance(getattr(self, name), kind):
+ raise TypeError(f"Capabilities.{name} must be a {kind.__name__}.")
+ if not isinstance(self.consumes_se, bool):
+ raise TypeError("Capabilities.consumes_se must be a boolean.")
+ if not isinstance(self.dependencies, tuple) or any(
+ not isinstance(name, str) or not name for name in self.dependencies
+ ):
+ raise TypeError(
+ "Capabilities.dependencies must be a tuple of distribution names."
+ )
+ if self.tolerance is not None and not isinstance(self.tolerance, Tolerance):
+ raise TypeError("Capabilities.tolerance must be a Tolerance or None.")
+ if self.numeric is Numeric.TOLERANCE_BOUND and self.tolerance is None:
+ raise ValueError(
+ "A tolerance_bound kernel must declare its Tolerance; a claim of "
+ "bounded movement without a bound is not a claim."
+ )
+ if self.numeric is Numeric.BITWISE and self.tolerance is not None:
+ raise ValueError("A bitwise kernel declares no Tolerance.")
@dataclass(frozen=True)
@@ -144,6 +233,9 @@ class KernelContext:
kernel may use.
sources: Source name to a content-verified path, for declared
sources only.
+ tolerances: ``(entity, column)`` of each declared input column to
+ the :class:`Tolerance` its owning kernel declared, or ``None``
+ for a bitwise owner. A gate compares against these.
"""
node: Node
@@ -153,6 +245,7 @@ class KernelContext:
params: Mapping[str, Param]
rng: np.random.Generator
sources: Mapping[str, Path] = field(default_factory=dict)
+ tolerances: Mapping[tuple[str, str], Tolerance | None] = field(default_factory=dict)
@dataclass(frozen=True)
@@ -176,6 +269,10 @@ class KernelResult:
``weights``.
weights: ``REWEIGHT`` kernels and declared weight transitions only:
the new explicit weights of the transition's entity.
+ strata: ``EXPAND`` kernels on a node with ``entrants=True`` only: the
+ stratum label of every entrant person, indexed by its new id.
+ Copied persons inherit their source's stratum and must not
+ appear here; an entrant person absent from it rejects the node.
artifacts: Opaque bytes stored beside the node's outputs (a fitted
model, a diagnostic table), keyed by name.
receipt: Descriptive facts for the manifest. Never hashed into a
@@ -192,6 +289,7 @@ class KernelResult:
weights: Weights | None = None
artifacts: Mapping[str, bytes] = field(default_factory=dict)
receipt: Mapping[str, object] = field(default_factory=dict)
+ strata: pd.Series | None = None
@runtime_checkable
@@ -281,6 +379,11 @@ def __init__(self) -> None:
def register(self, kernel: Kernel) -> Kernel:
if not isinstance(kernel, Kernel):
raise TypeError(f"{kernel!r} does not satisfy the Kernel protocol.")
+ if not isinstance(kernel.capabilities, Capabilities):
+ raise TypeError(
+ f"Kernel {getattr(kernel, 'ref', kernel)!r} must carry a Capabilities "
+ "instance, not a look-alike."
+ )
if kernel.ref in self._kernels and self._kernels[kernel.ref] is not kernel:
raise ValueError(f"Kernel {kernel.ref!r} is already registered.")
self._kernels[kernel.ref] = kernel
diff --git a/packages/microcosm-graph/src/microcosm/graph/keys.py b/packages/microcosm-graph/src/microcosm/graph/keys.py
index cab41c700..8f59ddb0d 100644
--- a/packages/microcosm-graph/src/microcosm/graph/keys.py
+++ b/packages/microcosm-graph/src/microcosm/graph/keys.py
@@ -8,6 +8,7 @@
from .canonical import canonical_json, normative, sha256_domain
from .decl import CompiledGraph, StructuralDelta
+from .kernel import Capabilities
__all__ = [
"artifact_key",
@@ -94,12 +95,43 @@ def _required_key(keys: Mapping[str, str], node_id: str, consumer: str) -> str:
) from error
+def _canonical_tolerance_float(value: int | float) -> float:
+ number = float(value)
+ return 0.0 if number == 0.0 else number
+
+
+def _capabilities_projection(capabilities: Capabilities) -> dict[str, object]:
+ """Return the complete canonical payload for a kernel contract."""
+
+ tolerance = capabilities.tolerance
+ return {
+ "determinism": capabilities.determinism.value,
+ "numeric": capabilities.numeric.value,
+ "seed_source": capabilities.seed_source.value,
+ "structural": capabilities.structural.value,
+ "role": capabilities.role.value,
+ "consumes_se": capabilities.consumes_se,
+ "dependencies": list(capabilities.dependencies),
+ "tolerance": (
+ None
+ if tolerance is None
+ else {
+ "rtol": _canonical_tolerance_float(tolerance.rtol),
+ "atol": _canonical_tolerance_float(tolerance.atol),
+ "ulps": tolerance.ulps,
+ }
+ ),
+ }
+
+
def node_key(
compiled: CompiledGraph,
node_id: str,
input_keys: Mapping[str, str],
kernel_impl_hash: str,
source_keys: Mapping[str, str],
+ *,
+ kernel_capabilities: Capabilities,
) -> str:
"""Derive a node key from its declaration and resolved input identities.
@@ -119,12 +151,19 @@ def node_key(
input_version = node.base
resolved: dict[tuple[str, str], str] = {}
+ rewritten = {
+ (owned.entity, owned.column) for owned in node.outputs if owned.rewrite
+ }
if input_version is not None:
for slice_ in node.inputs:
for column in slice_.columns:
coordinate = (slice_.entity, column)
- producer = compiled.owners.get(
- (input_version, slice_.entity, column), input_version
+ producer = (
+ input_version
+ if coordinate in rewritten
+ else compiled.owners.get(
+ (input_version, slice_.entity, column), input_version
+ )
)
producer_key = _required_key(input_keys, producer, node_id)
resolved[coordinate] = artifact_key(producer_key, slice_.entity, column)
@@ -169,6 +208,15 @@ def node_key(
else:
resolved_sources = {}
+ # Graph-level mass semantics (amendment 12) change what a structural node
+ # computes, so they enter its key; an ordinary node's key is unaffected.
+ graph_facts = (
+ {} if node.structural is StructuralDelta.NONE else compiled.graph.normative()
+ )
+ # Capabilities are executable contract, independent of implementation
+ # bytes. Bind the complete declaration so a cache entry produced under one
+ # contract cannot satisfy another kernel with the same ref and code hash.
+ capabilities = _capabilities_projection(kernel_capabilities)
return _hash_parts(
"node",
normative(node),
@@ -176,6 +224,8 @@ def node_key(
population_input,
kernel_impl_hash,
resolved_sources,
+ graph_facts,
+ capabilities,
)
diff --git a/packages/microcosm-graph/src/microcosm/graph/manifest.py b/packages/microcosm-graph/src/microcosm/graph/manifest.py
index 8f0a9857b..b6065f8f7 100644
--- a/packages/microcosm-graph/src/microcosm/graph/manifest.py
+++ b/packages/microcosm-graph/src/microcosm/graph/manifest.py
@@ -11,23 +11,53 @@
from types import MappingProxyType
from typing import TYPE_CHECKING, Self
+from microcosm.frame import Frame
+
from .canonical import canonical_json, sha256_domain
from .decl import GATE_OUTCOMES, StructuralDelta
from .errors import NodeRejectedError, StoreCorruptError
-from .kernel import Capabilities, Determinism, KernelRole, Numeric, SeedSource
+from .kernel import (
+ Capabilities,
+ Determinism,
+ KernelRole,
+ Numeric,
+ SeedSource,
+ Tolerance,
+)
from .population import MassRecord
if TYPE_CHECKING:
- from microcosm.frame import Frame
-
from .store import ContentStore
-__all__ = ["Decision", "NodeReceipt", "RunManifest"]
+__all__ = ["Decision", "NodeReceipt", "PopulationView", "RunManifest"]
_SCHEMA_VERSION = 1
_CERTIFYING_GATE_OUTCOMES = frozenset({"pass", "not_applicable"})
+class PopulationView(Frame):
+ """Zero-copy manifest view with entity-name table access.
+
+ All attached populations use this type. Existing :class:`Frame` accessors
+ remain available, and a group entity can also be read by name (for example,
+ `view.household` is equivalent to `view.table("household")`). The source
+ Frame keeps its original type.
+ """
+
+ __slots__ = ()
+
+ def __init__(self, frame: Frame) -> None:
+ if not isinstance(frame, Frame):
+ raise TypeError("PopulationView requires a Frame")
+ for slot in Frame.__slots__:
+ object.__setattr__(self, slot, getattr(frame, slot))
+
+ def __getattr__(self, name: str) -> object:
+ if name in self.entities:
+ return self.table(name)
+ raise AttributeError(f"{type(self).__name__!s} has no attribute {name!r}")
+
+
def _freeze_json(value: object) -> object:
"""Copy JSON-like receipt data into immutable containers."""
@@ -55,6 +85,16 @@ def _enum_value(value: object) -> object:
return value.value if isinstance(value, Enum) else value
+def _tolerance_payload(tolerance: Tolerance | None) -> dict[str, object] | None:
+ if tolerance is None:
+ return None
+ return {
+ "rtol": float(tolerance.rtol),
+ "atol": float(tolerance.atol),
+ "ulps": tolerance.ulps,
+ }
+
+
@dataclass(frozen=True)
class Decision(Mapping[str, str]):
"""A signed human decision carried as provenance, never as a node input."""
@@ -241,6 +281,7 @@ def _payload(self) -> dict[str, object]:
"role": _enum_value(capabilities.role),
"consumes_se": capabilities.consumes_se,
"dependencies": capabilities.dependencies,
+ "tolerance": _tolerance_payload(capabilities.tolerance),
},
"receipt": self.receipt,
"artifacts": tuple(
@@ -301,8 +342,17 @@ def __post_init__(self) -> None:
for name in ("started_at", "finished_at", "host"):
if not isinstance(getattr(self, name), str):
raise TypeError(f"RunManifest.{name} must be a string")
+ populations: dict[str, PopulationView] = {}
+ for version_id, frame in self.populations.items():
+ if not isinstance(version_id, str):
+ raise TypeError("RunManifest.populations keys must be strings")
+ if not isinstance(frame, Frame):
+ raise TypeError("RunManifest.populations values must be Frame")
+ populations[version_id] = PopulationView(frame)
object.__setattr__(
- self, "populations", MappingProxyType(dict(self.populations))
+ self,
+ "populations",
+ MappingProxyType(populations),
)
mass_ledgers: dict[str, tuple[MassRecord, ...]] = {}
for version_id, records in self.mass_ledgers.items():
@@ -416,7 +466,7 @@ def node(self, node_id: str) -> NodeReceipt:
def receipt(self, node_id: str) -> NodeReceipt:
return self.node(node_id)
- def population(self, version_id: str) -> Frame:
+ def population(self, version_id: str) -> PopulationView:
"""Return an attached final population version.
Population frames are deliberately not serialized in manifest JSON;
@@ -425,11 +475,14 @@ def population(self, version_id: str) -> Frame:
"""
try:
- return self.populations[version_id]
+ population = self.populations[version_id]
except KeyError as error:
raise KeyError(
f"Population {version_id!r} is not attached to this manifest."
) from error
+ if not isinstance(population, PopulationView): # __post_init__ invariant
+ raise RuntimeError("attached population was not normalized")
+ return population
def mass_ledger(self, version_id: str) -> tuple[MassRecord, ...]:
"""Return the transient mass audit trail for one attached version."""
@@ -687,6 +740,35 @@ def _capabilities_from_payload(value: object) -> Capabilities:
raise ValueError("capabilities.dependencies must be an array")
if not all(isinstance(item, str) for item in dependencies):
raise ValueError("capabilities.dependencies must contain strings")
+ raw_tolerance = value.get("tolerance")
+ if raw_tolerance is None:
+ tolerance = None
+ else:
+ if not isinstance(raw_tolerance, Mapping) or set(raw_tolerance) != {
+ "rtol",
+ "atol",
+ "ulps",
+ }:
+ raise ValueError(
+ "capabilities.tolerance must be null or an object containing "
+ "rtol, atol, and ulps"
+ )
+ rtol = raw_tolerance["rtol"]
+ atol = raw_tolerance["atol"]
+ ulps = raw_tolerance["ulps"]
+ if (
+ isinstance(rtol, bool)
+ or not isinstance(rtol, int | float)
+ or isinstance(atol, bool)
+ or not isinstance(atol, int | float)
+ or isinstance(ulps, bool)
+ or not isinstance(ulps, int)
+ ):
+ raise ValueError(
+ "capabilities.tolerance rtol/atol must be numeric and ulps "
+ "must be an integer"
+ )
+ tolerance = Tolerance(rtol=float(rtol), atol=float(atol), ulps=ulps)
return Capabilities(
determinism=Determinism(_string_field(value, "determinism")),
numeric=Numeric(_string_field(value, "numeric")),
@@ -695,6 +777,7 @@ def _capabilities_from_payload(value: object) -> Capabilities:
role=KernelRole(str(value.get("role", KernelRole.COMPUTE.value))),
consumes_se=consumes_se,
dependencies=tuple(dependencies),
+ tolerance=tolerance,
)
diff --git a/packages/microcosm-graph/src/microcosm/graph/population.py b/packages/microcosm-graph/src/microcosm/graph/population.py
index 98a551303..e16606bde 100644
--- a/packages/microcosm-graph/src/microcosm/graph/population.py
+++ b/packages/microcosm-graph/src/microcosm/graph/population.py
@@ -16,6 +16,7 @@
from microcosm.frame import Frame, MassChangeRecord, WeightKind, Weights
+from .canonical import canonical_json
from .decl import (
MASS_POLICIES,
ROWS_ALL,
@@ -34,6 +35,8 @@
"dtype_for_token",
"dtype_matches",
"expand_lineage_receipt",
+ "entrant_strata_receipt",
+ "mass_record_receipt",
"owned_ids",
"patch",
"population_from_frame",
@@ -119,6 +122,14 @@ class MassRecord:
before_by_stratum: tuple[tuple[object, float], ...]
after_by_stratum: tuple[tuple[object, float], ...]
entity: str | None = None
+ partition_entity: str | None = None
+ partition_column: str | None = None
+ before_by_partition_stratum: tuple[
+ tuple[object, tuple[tuple[object, float], ...]], ...
+ ] = ()
+ after_by_partition_stratum: tuple[
+ tuple[object, tuple[tuple[object, float], ...]], ...
+ ] = ()
@property
def before_strata(self) -> Mapping[object, float]:
@@ -128,6 +139,24 @@ def before_strata(self) -> Mapping[object, float]:
def after_strata(self) -> Mapping[object, float]:
return MappingProxyType(dict(self.after_by_stratum))
+ @property
+ def before_partitions(self) -> Mapping[object, Mapping[object, float]]:
+ return MappingProxyType(
+ {
+ partition: MappingProxyType(dict(strata))
+ for partition, strata in self.before_by_partition_stratum
+ }
+ )
+
+ @property
+ def after_partitions(self) -> Mapping[object, Mapping[object, float]]:
+ return MappingProxyType(
+ {
+ partition: MappingProxyType(dict(strata))
+ for partition, strata in self.after_by_partition_stratum
+ }
+ )
+
@property
def old_total(self) -> float:
"""Compatibility spelling used by :mod:`microcosm.frame`."""
@@ -141,6 +170,66 @@ def new_total(self) -> float:
return self.after_total
+def _receipt_key(value: object) -> str:
+ """Return the JSON-object-key spelling of a partition or stratum value."""
+
+ if isinstance(value, np.generic):
+ value = value.item()
+ return str(value)
+
+
+def _partition_receipt_mapping(
+ values: tuple[tuple[object, tuple[tuple[object, float], ...]], ...],
+) -> dict[str, dict[str, float]]:
+ result: dict[str, dict[str, float]] = {}
+ for partition, strata in values:
+ partition_key = _receipt_key(partition)
+ if partition_key in result:
+ raise PopulationError(
+ f"Partition values collide as JSON key {partition_key!r}."
+ )
+ converted: dict[str, float] = {}
+ for stratum, mass in strata:
+ stratum_key = _receipt_key(stratum)
+ if stratum_key in converted:
+ raise PopulationError(
+ f"Strata collide as JSON key {stratum_key!r} inside partition "
+ f"{partition_key!r}."
+ )
+ converted[stratum_key] = float(mass)
+ result[partition_key] = converted
+ return result
+
+
+def mass_record_receipt(record: MassRecord) -> dict[str, object]:
+ """Return executor-authored public mass accounting for one ledger record."""
+
+ payload: dict[str, object] = {
+ "policy": record.policy,
+ "before": record.before_total,
+ "after": record.after_total,
+ "stratum_before": _receipt_mass_mapping(
+ dict(record.before_by_stratum), label=f"Node {record.node_id!r} mass"
+ ),
+ "stratum_after": _receipt_mass_mapping(
+ dict(record.after_by_stratum), label=f"Node {record.node_id!r} mass"
+ ),
+ }
+ if record.partition_entity is not None:
+ assert record.partition_column is not None
+ payload["partition"] = {
+ "entity": record.partition_entity,
+ "column": record.partition_column,
+ "stratum_before": _partition_receipt_mapping(
+ record.before_by_partition_stratum
+ ),
+ "stratum_after": _partition_receipt_mapping(
+ record.after_by_partition_stratum
+ ),
+ }
+ return payload
+
+
@dataclass(frozen=True)
class Population:
"""One immutable graph view over a validated :class:`Frame`."""
@@ -272,9 +361,15 @@ def population_from_frame(
)
-def _lineage_json_scalar(value: object) -> str | int | float | bool:
+def _lineage_json_scalar(
+ value: object, *, allow_null: bool = False
+) -> str | int | float | bool | None:
"""Detach one entity id into the scalar vocabulary accepted by receipts."""
+ if pd.isna(value):
+ if allow_null:
+ return None
+ raise PopulationError(f"EXPAND lineage id {value!r} is not a JSON scalar.")
if isinstance(value, np.generic):
value = value.item()
if not isinstance(value, str | int | float | bool):
@@ -284,6 +379,46 @@ def _lineage_json_scalar(value: object) -> str | int | float | bool:
return value
+def _stratum_receipt_scalar(value: object) -> object:
+ """Encode one cache-safe entrant stratum label for a JSON receipt."""
+
+ missing = pd.isna(value)
+ if isinstance(missing, bool | np.bool_) and bool(missing):
+ raise PopulationError("EXPAND entrant stratum labels cannot be null.")
+ if isinstance(value, np.generic):
+ value = value.item()
+ if isinstance(value, bytes):
+ return {"bytes_hex": value.hex()}
+ if not isinstance(value, str | int | float | bool):
+ raise PopulationError(
+ f"EXPAND entrant stratum label {value!r} is not a cache-safe scalar."
+ )
+ if isinstance(value, float) and not np.isfinite(value):
+ raise PopulationError(f"EXPAND entrant stratum label {value!r} is not finite.")
+ return value
+
+
+def _stratum_from_receipt_scalar(value: object) -> object:
+ """Decode one executor-authored entrant stratum label from a receipt."""
+
+ if isinstance(value, Mapping):
+ if set(value) != {"bytes_hex"} or not isinstance(value["bytes_hex"], str):
+ raise PopulationError("EXPAND entrant stratum receipt label is malformed.")
+ encoded = value["bytes_hex"]
+ try:
+ decoded = bytes.fromhex(encoded)
+ except ValueError as error:
+ raise PopulationError(
+ "EXPAND entrant stratum receipt bytes are malformed."
+ ) from error
+ if encoded != decoded.hex():
+ raise PopulationError(
+ "EXPAND entrant stratum receipt bytes are not canonical."
+ )
+ return decoded
+ return _stratum_receipt_scalar(value)
+
+
def expand_lineage_receipt(
expand: Mapping[str, pd.Series],
) -> dict[str, list[list[object]]]:
@@ -301,7 +436,10 @@ def expand_lineage_receipt(
f"EXPAND lineage for {entity!r} is not a pandas Series."
)
payload[entity] = [
- [_lineage_json_scalar(target), _lineage_json_scalar(source)]
+ [
+ _lineage_json_scalar(target),
+ _lineage_json_scalar(source, allow_null=True),
+ ]
for target, source in zip(
lineage.index.tolist(), lineage.tolist(), strict=True
)
@@ -309,6 +447,195 @@ def expand_lineage_receipt(
return payload
+def _entrant_person_ids(frame: Frame, lineage: Mapping[str, pd.Series]) -> pd.Index:
+ """Return null-lineage person targets in declared lineage order."""
+
+ person = frame.schema.person_entity
+ id_column = frame.schema.entity_id_column(person)
+ person_lineage = lineage[person]
+ entrant_positions = np.flatnonzero(
+ person_lineage.isna().to_numpy(dtype=np.bool_, copy=False)
+ )
+ return pd.Index(person_lineage.index.take(entrant_positions), name=id_column)
+
+
+def _validated_entrant_strata(
+ frame: Frame,
+ node: Node,
+ lineage: Mapping[str, pd.Series],
+ raw: object,
+) -> pd.Series | None:
+ """Validate and align the iff contract for entrant-person strata."""
+
+ person = frame.schema.person_entity
+ id_column = frame.schema.entity_id_column(person)
+ id_dtype = frame.table(person)[id_column].dtype
+ entrant_ids = _entrant_person_ids(frame, lineage)
+ if not len(entrant_ids):
+ if raw is not None:
+ raise PopulationError(
+ f"EXPAND node {node.id!r} returned strata without entrant persons."
+ )
+ return None
+ if raw is None:
+ raise PopulationError(
+ f"EXPAND node {node.id!r} omitted strata for entrant persons "
+ f"{entrant_ids[:5].tolist()}."
+ )
+ if not isinstance(raw, pd.Series):
+ raise PopulationError(
+ f"EXPAND node {node.id!r} entrant strata is not a Series."
+ )
+ labels_index = pd.Index(raw.index, name=id_column)
+ if labels_index.nlevels != 1 or labels_index.dtype != id_dtype:
+ raise PopulationError(
+ f"EXPAND node {node.id!r} entrant strata index must use "
+ f"{id_dtype!s} person ids."
+ )
+ if not labels_index.is_unique:
+ raise PopulationError(
+ f"EXPAND node {node.id!r} repeats entrant strata person ids."
+ )
+ if labels_index.isna().any():
+ raise PopulationError(
+ f"EXPAND node {node.id!r} entrant strata contains null person ids."
+ )
+ missing = entrant_ids[~entrant_ids.isin(labels_index)]
+ extra = labels_index[~labels_index.isin(entrant_ids)]
+ if len(missing) or len(extra):
+ raise PopulationError(
+ f"EXPAND node {node.id!r} entrant strata must name exactly the entrant "
+ f"persons; missing={missing[:5].tolist()}, extra={extra[:5].tolist()}."
+ )
+ if not (
+ pd.api.types.is_object_dtype(raw.dtype) or isinstance(raw.dtype, pd.StringDtype)
+ ):
+ raise PopulationError(
+ f"EXPAND node {node.id!r} entrant strata must use object or string "
+ f"labels, got {raw.dtype!s}."
+ )
+ if raw.isna().any():
+ raise PopulationError(
+ f"EXPAND node {node.id!r} entrant strata contains missing labels."
+ )
+ aligned = raw.reindex(entrant_ids).copy()
+ for value in aligned.array:
+ _stratum_receipt_scalar(value)
+ return aligned
+
+
+def entrant_strata_receipt(
+ frame: Frame,
+ node: Node,
+ expand: Mapping[str, pd.Series],
+ strata: pd.Series | None,
+) -> list[list[object]] | None:
+ """Return executor-authored entrant-person strata in lineage order."""
+
+ lineage = _validate_expand_lineage(frame, node, expand)
+ aligned = _validated_entrant_strata(frame, node, lineage, strata)
+ if aligned is None:
+ return None
+ return [
+ [_lineage_json_scalar(target), _stratum_receipt_scalar(label)]
+ for target, label in zip(aligned.index, aligned.array, strict=True)
+ ]
+
+
+def _cached_entrant_strata(
+ frame: Frame,
+ node: Node,
+ lineage: Mapping[str, pd.Series],
+ receipt: Mapping[str, object],
+) -> pd.Series | None:
+ """Parse the executor-authored entrant-strata cache attestation."""
+
+ person = frame.schema.person_entity
+ id_column = frame.schema.entity_id_column(person)
+ id_dtype = frame.table(person)[id_column].dtype
+ entrant_ids = _entrant_person_ids(frame, lineage)
+ if not len(entrant_ids):
+ if "entrant_strata" in receipt:
+ raise PopulationError(
+ f"Cached EXPAND node {node.id!r} has entrant strata without "
+ "entrant persons."
+ )
+ return None
+ raw = receipt.get("entrant_strata")
+ if not isinstance(raw, list):
+ raise PopulationError(
+ f"Cached EXPAND node {node.id!r} has no entrant-strata receipt."
+ )
+ targets: list[object] = []
+ labels: list[object] = []
+ for entry in raw:
+ if not isinstance(entry, list) or len(entry) != 2:
+ raise PopulationError(
+ f"Cached EXPAND node {node.id!r} has malformed entrant strata."
+ )
+ targets.append(entry[0])
+ labels.append(_stratum_from_receipt_scalar(entry[1]))
+ try:
+ target_index = pd.Index(
+ pd.Series(targets, dtype=id_dtype).array, name=id_column
+ )
+ except (TypeError, ValueError) as error:
+ raise PopulationError(
+ f"Cached EXPAND node {node.id!r} entrant strata contain invalid "
+ f"{id_dtype!s} person ids."
+ ) from error
+ if not target_index.equals(entrant_ids):
+ raise PopulationError(
+ f"Cached EXPAND node {node.id!r} entrant strata do not name its "
+ "entrant persons in lineage order."
+ )
+ return pd.Series(labels, index=entrant_ids, dtype=object)
+
+
+def _assert_cached_expand_strata(
+ before: Frame,
+ after: Frame,
+ node: Node,
+ lineage: Mapping[str, pd.Series],
+ receipt: Mapping[str, object],
+) -> None:
+ """Verify cached incumbent, copied, and entrant person strata by lineage."""
+
+ person = before.schema.person_entity
+ id_column = before.schema.entity_id_column(person)
+ before_ids = pd.Index(before.table(person)[id_column], name=id_column)
+ person_lineage = lineage[person]
+ expected_ids = before_ids.append(pd.Index(person_lineage.index, name=id_column))
+ after_ids = pd.Index(after.table(person)[id_column], name=id_column)
+ if not after_ids.equals(expected_ids):
+ raise PopulationError(f"Cached EXPAND node {node.id!r} reordered person ids.")
+ entrant_strata = _cached_entrant_strata(before, node, lineage, receipt)
+ expected = before.strata.astype(object).tolist()
+ entrant_positions: list[int] = []
+ for target, source in zip(person_lineage.index, person_lineage.array, strict=True):
+ if pd.isna(source):
+ assert entrant_strata is not None
+ entrant_positions.append(len(expected))
+ expected.append(entrant_strata.loc[target])
+ continue
+ source_position = before_ids.get_loc(source)
+ expected.append(before.strata.iloc[source_position])
+ actual = after.strata.astype(object).reset_index(drop=True)
+ if not actual.equals(pd.Series(expected, dtype=object)):
+ raise PopulationError(
+ f"Cached EXPAND node {node.id!r} strata disagree with its lineage "
+ "and entrant-strata receipt."
+ )
+ for position in entrant_positions:
+ actual_label = _stratum_receipt_scalar(actual.iloc[position])
+ expected_label = _stratum_receipt_scalar(expected[position])
+ if canonical_json(actual_label) != canonical_json(expected_label):
+ raise PopulationError(
+ f"Cached EXPAND node {node.id!r} entrant stratum label "
+ "disagrees with its receipt."
+ )
+
+
def _expand_lineage_from_receipt(
frame: Frame,
node: Node,
@@ -340,11 +667,21 @@ def _expand_lineage_from_receipt(
)
id_column = frame.schema.entity_id_column(entity)
dtype = frame.table(entity)[id_column].dtype
+ source_dtype: object = dtype
+ has_null_source = any(pd.isna(value) for value in sources)
+ if has_null_source and pd.api.types.is_bool_dtype(dtype):
+ source_dtype = pd.BooleanDtype()
+ elif has_null_source and pd.api.types.is_integer_dtype(dtype):
+ numpy_dtype = np.dtype(getattr(dtype, "numpy_dtype", dtype))
+ prefix = "UInt" if np.issubdtype(numpy_dtype, np.unsignedinteger) else "Int"
+ source_dtype = pd.api.types.pandas_dtype(
+ f"{prefix}{numpy_dtype.itemsize * 8}"
+ )
lineage[entity] = pd.Series(
sources,
index=pd.Index(pd.Series(targets, dtype=dtype).array, name=id_column),
name=id_column,
- dtype=dtype,
+ dtype=source_dtype,
)
return lineage
@@ -378,7 +715,17 @@ def _validate_expand_lineage(
id_column = frame.schema.entity_id_column(entity)
source_ids = pd.Index(frame.table(entity)[id_column], name=id_column)
targets = pd.Index(lineage.index, name=id_column)
- if targets.dtype != source_ids.dtype or lineage.dtype != source_ids.dtype:
+ source_is_null = lineage.isna().to_numpy(dtype=np.bool_, copy=False)
+ nullable_sources = bool(source_is_null.any())
+ nullable_source_dtype = getattr(lineage.dtype, "numpy_dtype", None)
+ if targets.dtype != source_ids.dtype or (
+ lineage.dtype != source_ids.dtype
+ and not (
+ nullable_sources
+ and nullable_source_dtype is not None
+ and np.dtype(nullable_source_dtype) == np.dtype(source_ids.dtype)
+ )
+ ):
raise PopulationError(
f"EXPAND node {node.id!r} lineage for {entity!r} must use "
f"{source_ids.dtype!s} ids for both targets and sources."
@@ -387,9 +734,15 @@ def _validate_expand_lineage(
raise PopulationError(
f"EXPAND node {node.id!r} repeats new target {entity!r} ids."
)
- if targets.isna().any() or lineage.isna().any():
+ if targets.isna().any():
raise PopulationError(
- f"EXPAND node {node.id!r} lineage for {entity!r} contains null ids."
+ f"EXPAND node {node.id!r} lineage for {entity!r} contains null "
+ "target ids."
+ )
+ if nullable_sources and not node.entrants:
+ raise PopulationError(
+ f"EXPAND node {node.id!r} lineage for {entity!r} contains null "
+ "source ids without entrants=True."
)
collisions = targets.intersection(source_ids)
if len(collisions):
@@ -397,31 +750,41 @@ def _validate_expand_lineage(
f"EXPAND node {node.id!r} lineage target {entity!r} ids collide "
f"with incumbents {collisions[:5].tolist()}."
)
- source_positions = source_ids.get_indexer(lineage.to_numpy(copy=False))
+ source_positions = np.full(len(lineage), -1, dtype=np.int64)
+ copied = ~source_is_null
+ source_positions[copied] = source_ids.get_indexer(
+ lineage.iloc[np.flatnonzero(copied)].to_numpy(copy=False)
+ )
if (source_positions < 0).any():
- bad = lineage.iloc[np.flatnonzero(source_positions < 0)[:5]].tolist()
- raise PopulationError(
- f"EXPAND node {node.id!r} lineage names unknown {entity!r} "
- f"source ids {bad}."
- )
+ unknown = copied & (source_positions < 0)
+ if unknown.any():
+ bad = lineage.iloc[np.flatnonzero(unknown)[:5]].tolist()
+ raise PopulationError(
+ f"EXPAND node {node.id!r} lineage names unknown {entity!r} "
+ f"source ids {bad}."
+ )
if after is not None:
after_ids = pd.Index(after.table(entity)[id_column], name=id_column)
if not source_ids.isin(after_ids).all():
raise PopulationError(
f"Cached EXPAND node {node.id!r} dropped incumbent {entity!r} ids."
)
- additions = after_ids[~after_ids.isin(source_ids)]
- if not additions.equals(targets):
+ expected_ids = source_ids.append(targets)
+ if not after_ids.equals(expected_ids):
raise PopulationError(
- f"Cached EXPAND node {node.id!r} frame additions for "
- f"{entity!r} disagree with its lineage receipt."
+ f"Cached EXPAND node {node.id!r} final {entity!r} ids "
+ "disagree with its lineage receipt."
)
validated[entity] = lineage
return validated
def restore_cached_expand(
- population: Population, node: Node, result: KernelResult
+ population: Population,
+ node: Node,
+ result: KernelResult,
+ *,
+ mass_partition: tuple[str, str] | None = None,
) -> Population:
"""Restore a previously validated EXPAND frame against its keyed base.
@@ -434,6 +797,11 @@ def restore_cached_expand(
if node.structural is not StructuralDelta.EXPAND or result.frame is None:
raise PopulationError("restore_cached_expand requires an EXPAND Frame.")
+ if result.strata is not None:
+ raise PopulationError(
+ f"Cached EXPAND node {node.id!r} returned kernel strata instead of "
+ "its executor frame artifact."
+ )
frame = result.frame
if frame.schema != population.frame.schema:
raise PopulationError(f"Cached EXPAND node {node.id!r} changed schema.")
@@ -443,6 +811,7 @@ def restore_cached_expand(
lineage = _validate_expand_lineage(
population.frame, node, receipt_lineage, after=frame
)
+ _assert_cached_expand_strata(population.frame, frame, node, lineage, result.receipt)
_assert_expand_weights(population, frame, node, result)
design_weights: dict[str, np.ndarray] = {}
@@ -457,18 +826,46 @@ def restore_cached_expand(
values[retained] = old_anchor[positions[retained]]
if not retained.all():
sources = lineage[entity].reindex(after_ids[~retained])
- source_positions = before_ids.get_indexer(sources.to_numpy(copy=False))
+ source_is_null = sources.isna().to_numpy(dtype=np.bool_, copy=False)
+ introduced_positions = np.flatnonzero(~retained)
+ copied_positions = introduced_positions[~source_is_null]
+ source_positions = before_ids.get_indexer(
+ sources.iloc[np.flatnonzero(~source_is_null)].to_numpy(copy=False)
+ )
if (source_positions < 0).any(): # defended by lineage validation
raise PopulationError(
f"Cached EXPAND node {node.id!r} has unknown design lineage "
f"for new {entity!r} ids."
)
- values[~retained] = old_anchor[source_positions]
+ values[copied_positions] = old_anchor[source_positions]
+ entrant_positions = introduced_positions[source_is_null]
+ if len(entrant_positions):
+ try:
+ current = frame.weights_for(entity)
+ except ValueError as error:
+ raise PopulationError(
+ f"Cached EXPAND node {node.id!r} has no design anchor for "
+ f"entrant {entity!r} ids."
+ ) from error
+ if current.kind is not WeightKind.DESIGN:
+ raise PopulationError(
+ f"Cached EXPAND node {node.id!r} cannot anchor entrant "
+ f"{entity!r} ids from {current.kind.value!r} weights; "
+ "explicit design weights are required."
+ )
+ values[entrant_positions] = current.values[entrant_positions]
design_weights[entity] = values
ledger = (
*population.mass_ledger,
- _mass_record(population.frame, frame, node, result, _mass_policy(node)),
+ _mass_record(
+ population.frame,
+ frame,
+ node,
+ result,
+ _mass_policy(node),
+ mass_partition=mass_partition,
+ ),
)
owners = {
(entity, str(column)): node.id
@@ -531,7 +928,13 @@ def storage_equal(
return _storage_parts(left, selected) == _storage_parts(right, selected)
-def patch(population: Population, node: Node, result: KernelResult) -> Population:
+def patch(
+ population: Population,
+ node: Node,
+ result: KernelResult,
+ *,
+ mass_partition: tuple[str, str] | None = None,
+) -> Population:
"""Validate and apply one node result without mutating ``population``.
``EXPAND`` kernels return only the new-id to source-id mapping through
@@ -554,6 +957,10 @@ def patch(population: Population, node: Node, result: KernelResult) -> Populatio
_assert_no_ordinary_structural_outputs(population, node)
expected_columns = {(owned.entity, owned.column) for owned in node.outputs}
lineage_expand = node.structural is StructuralDelta.EXPAND and result.frame is None
+ if result.strata is not None and not lineage_expand:
+ raise PopulationError(
+ f"Node {node.id!r} returned entrant strata outside a lineage EXPAND."
+ )
if not lineage_expand and set(result.columns) != expected_columns:
raise PopulationError(
f"Node {node.id!r} returned columns {sorted(result.columns)}; "
@@ -602,7 +1009,14 @@ def patch(population: Population, node: Node, result: KernelResult) -> Populatio
ledger = population.mass_ledger
if node.structural is not StructuralDelta.NONE or node.weights is not None:
policy = _mass_policy(node)
- record = _mass_record(before, frame, node, result, policy)
+ record = _mass_record(
+ before,
+ frame,
+ node,
+ result,
+ policy,
+ mass_partition=mass_partition,
+ )
ledger = (*ledger, record)
return Population.from_frame(
@@ -658,6 +1072,8 @@ def _targets_by_source(lineage: pd.Series) -> dict[object, list[object]]:
grouped: dict[object, list[object]] = {}
for target, source in zip(lineage.index, lineage.array, strict=True):
+ if pd.isna(source):
+ continue
grouped.setdefault(source, []).append(target)
return grouped
@@ -680,8 +1096,10 @@ def _remap_expand_memberships(
source_person = before.table(person)
person_id = before.schema.entity_id_column(person)
source_person_ids = pd.Index(source_person[person_id])
+ entrant_mask = person_lineage.isna().to_numpy(dtype=np.bool_, copy=False)
+ copied_indices = np.flatnonzero(~entrant_mask)
source_positions = source_person_ids.get_indexer(
- person_lineage.to_numpy(copy=False)
+ person_lineage.iloc[copied_indices].to_numpy(copy=False)
)
if (source_positions < 0).any(): # defended by lineage validation
raise PopulationError(
@@ -708,16 +1126,24 @@ def _remap_expand_memberships(
)
seen: dict[object, int] = {}
- remapped: list[object] = []
- for source_position, source_person_id in zip(
- source_positions, person_lineage.array, strict=True
+ remapped = (
+ tables[person][membership]
+ .iloc[len(source_person) :]
+ .reset_index(drop=True)
+ .copy()
+ )
+ for addition_position, source_position, source_person_id in zip(
+ copied_indices,
+ source_positions,
+ person_lineage.iloc[copied_indices].array,
+ strict=True,
):
# Select the membership Series directly. Selecting a mixed-type
# DataFrame row can coerce a large integer group id through float.
source_group = source_person[membership].iloc[source_position]
candidates = group_targets.get(source_group, [])
if not candidates:
- remapped.append(source_group)
+ remapped.iloc[addition_position] = source_group
continue
ordinal = seen.get(source_person_id, 0)
if ordinal >= len(candidates):
@@ -725,13 +1151,12 @@ def _remap_expand_memberships(
f"EXPAND node {node.id!r} cannot align {membership!r} for "
f"copied person {source_person_id!r}."
)
- remapped.append(candidates[ordinal])
+ remapped.iloc[addition_position] = candidates[ordinal]
seen[source_person_id] = ordinal + 1
carried = source_person[membership].reset_index(drop=True)
- additions = pd.Series(remapped, dtype=source_person[membership].dtype)
tables[person][membership] = pd.concat(
- [carried, additions], ignore_index=True
+ [carried, remapped], ignore_index=True
).array
@@ -746,11 +1171,17 @@ def _patch_expand(
f"EXPAND node {node.id!r} cannot yet carry association link tables."
)
cells = _expand_cells(node)
- for entity, _, _ in cells:
+ for entity, column, _ in cells:
if entity not in before.entities:
raise PopulationError(
f"EXPAND node {node.id!r} names unknown entity {entity!r}."
)
+ id_column = before.schema.entity_id_column(entity)
+ if column == id_column:
+ raise PopulationError(
+ f"EXPAND node {node.id!r} cannot overlay entity id column "
+ f"{entity}.{column}; lineage supplies final ids."
+ )
cell_coordinates = {(entity, column) for entity, column, _ in cells}
if set(result.columns) != cell_coordinates:
@@ -761,7 +1192,6 @@ def _patch_expand(
lineage = _validate_expand_lineage(before, node, result.expand)
- tables: dict[str, pd.DataFrame] = {}
lineage_positions: dict[str, np.ndarray] = {}
target_ids: dict[str, pd.Index] = {}
for entity in before.entities:
@@ -772,29 +1202,40 @@ def _patch_expand(
source_table[id_column].to_numpy(copy=True), name=id_column
)
new_targets = pd.Index(entity_lineage.index, name=id_column)
- targets = source_ids.append(new_targets)
- source_positions = source_ids.get_indexer(entity_lineage.to_numpy(copy=False))
- positions = np.concatenate(
- [np.arange(len(source_ids), dtype=np.int64), source_positions]
+ target_ids[entity] = source_ids.append(new_targets)
+ entrants = entity_lineage.isna().to_numpy(dtype=np.bool_, copy=False)
+ source_positions = np.full(len(entity_lineage), -1, dtype=np.int64)
+ copied = ~entrants
+ source_positions[copied] = source_ids.get_indexer(
+ entity_lineage.iloc[np.flatnonzero(copied)].to_numpy(copy=False)
)
- carried = source_table.iloc[positions].reset_index(drop=True)
- replacement_ids = pd.Series(
- targets.to_numpy(copy=True), dtype=source_table[id_column].dtype
+ lineage_positions[entity] = np.concatenate(
+ [np.arange(len(source_ids), dtype=np.int64), source_positions]
)
- if len(replacement_ids) != len(carried):
- raise PopulationError(
- f"EXPAND node {node.id!r} lineage index/value lengths disagree "
- f"for {entity!r}."
- )
- carried[id_column] = replacement_ids.array
- tables[entity] = carried
- lineage_positions[entity] = positions
- target_ids[entity] = targets
- _remap_expand_memberships(before, tables, lineage, node)
+ if entrants.any():
+ carried = {
+ (entity, str(column))
+ for column in source_table.columns
+ if column != id_column
+ }
+ missing = sorted(carried - cell_coordinates)
+ if missing:
+ names = [
+ f"{carried_entity}.{column}" for carried_entity, column in missing
+ ]
+ raise PopulationError(
+ f"EXPAND node {node.id!r} entrant rows do not materialize "
+ f"carried columns {names}."
+ )
+ person = before.schema.person_entity
+ entrant_strata = _validated_entrant_strata(before, node, lineage, result.strata)
+
+ aligned_cells: dict[tuple[str, str], pd.Series] = {}
for entity, column, dtype in cells:
- incoming = result.columns[(entity, column)]
+ coordinate = (entity, column)
+ incoming = result.columns[coordinate]
if not isinstance(incoming, pd.Series):
raise PopulationError(
f"EXPAND node {node.id!r} cell {entity}.{column} is not a Series."
@@ -815,8 +1256,63 @@ def _patch_expand(
dtype,
label=f"EXPAND node {node.id!r} cell {entity}.{column}",
)
+ source_table = before.table(entity)
+ if column in source_table:
+ carried_dtype = token_for_dtype(source_table[column].dtype)
+ if dtype != carried_dtype:
+ raise PopulationError(
+ f"EXPAND node {node.id!r} carried cell {entity}.{column} "
+ f"declares {dtype!r}; its incumbent dtype is {carried_dtype!r}."
+ )
+ incumbent = aligned.iloc[: len(source_table)].reset_index(drop=True)
+ if not storage_equal(source_table[column], incumbent):
+ raise PopulationError(
+ f"EXPAND node {node.id!r} changed carried storage in "
+ f"{entity}.{column} for incumbent rows."
+ )
+ aligned_cells[coordinate] = aligned
+
+ tables: dict[str, pd.DataFrame] = {}
+ for entity in before.entities:
+ id_column = before.schema.entity_id_column(entity)
+ source_table = before.table(entity)
+ positions = lineage_positions[entity]
+ addition_positions = positions[len(source_table) :]
+ if len(source_table):
+ additions = source_table.iloc[
+ np.maximum(addition_positions, 0)
+ ].reset_index(drop=True)
+ else:
+ additions = source_table.reindex(range(len(addition_positions))).copy()
+ carried = pd.concat(
+ [source_table.reset_index(drop=True), additions], ignore_index=True
+ )
+ replacement_ids = pd.Series(
+ target_ids[entity].to_numpy(copy=True),
+ dtype=source_table[id_column].dtype,
+ )
+ if len(replacement_ids) != len(carried):
+ raise PopulationError(
+ f"EXPAND node {node.id!r} lineage index/value lengths disagree "
+ f"for {entity!r}."
+ )
+ carried[id_column] = replacement_ids.array
+ tables[entity] = carried
+
+ _remap_expand_memberships(before, tables, lineage, node)
+
+ for (entity, column), aligned in aligned_cells.items():
tables[entity][column] = aligned.array
+ for entity, expected_ids in target_ids.items():
+ id_column = before.schema.entity_id_column(entity)
+ final_ids = pd.Index(tables[entity][id_column], name=id_column)
+ if not final_ids.equals(expected_ids):
+ raise PopulationError(
+ f"EXPAND node {node.id!r} final {entity!r} ids disagree with "
+ "its lineage targets after cell overlays."
+ )
+
weight_entity = _expand_weight_entity(node)
assert weight_entity is not None
if weight_entity not in before.weighted_entities:
@@ -832,16 +1328,39 @@ def _patch_expand(
weights[entity] = result.weights
continue
old = before.weights_for(entity)
- weights[entity] = Weights(old.values[lineage_positions[entity]], kind=old.kind)
+ positions = lineage_positions[entity]
+ if (positions < 0).any():
+ raise PopulationError(
+ f"EXPAND node {node.id!r} cannot admit entrants on weighted "
+ f"entity {entity!r}; only {weight_entity!r} has materialized weights."
+ )
+ weights[entity] = Weights(old.values[positions], kind=old.kind)
- person = before.schema.person_entity
person_positions = lineage_positions[person]
- strata = pd.Series(
- before.strata.iloc[person_positions].array.copy(),
- index=tables[person].index,
- name=before.strata.name,
- dtype=before.strata.dtype,
- )
+ if entrant_strata is None:
+ strata = pd.Series(
+ before.strata.iloc[person_positions].array.copy(),
+ index=tables[person].index,
+ name=before.strata.name,
+ dtype=before.strata.dtype,
+ )
+ else:
+ additions: list[object] = []
+ entrant_values = iter(entrant_strata.array)
+ for source_position in person_positions[len(before.table(person)) :]:
+ additions.append(
+ next(entrant_values)
+ if source_position < 0
+ else before.strata.iloc[source_position]
+ )
+ strata = pd.concat(
+ [
+ before.strata.astype(object).reset_index(drop=True),
+ pd.Series(additions, dtype=object),
+ ],
+ ignore_index=True,
+ )
+ strata.name = before.strata.name
frame = Frame(
tables,
before.schema,
@@ -1210,13 +1729,11 @@ def _carry_design_weights(
introduced = before_positions < 0
if introduced.any():
sources = lineage.reindex(after_ids[introduced])
- if sources.isna().any():
- raise PopulationError(
- f"EXPAND node {node.id!r} has incomplete design lineage "
- f"for {entity!r}."
- )
- before_positions[introduced] = before_ids.get_indexer(
- sources.to_numpy(copy=False)
+ source_is_null = sources.isna().to_numpy(dtype=np.bool_, copy=False)
+ introduced_positions = np.flatnonzero(introduced)
+ copied_positions = introduced_positions[~source_is_null]
+ before_positions[copied_positions] = before_ids.get_indexer(
+ sources.iloc[np.flatnonzero(~source_is_null)].to_numpy(copy=False)
)
values = np.empty(len(after_ids), dtype=np.float64)
retained = before_positions >= 0
@@ -1455,6 +1972,8 @@ def _mass_record(
node: Node,
result: KernelResult,
policy: str,
+ *,
+ mass_partition: tuple[str, str] | None = None,
) -> MassRecord:
if policy not in MASS_POLICIES:
raise PopulationError(f"Node {node.id!r} has unknown mass policy {policy!r}.")
@@ -1464,7 +1983,18 @@ def _mass_record(
after_pairs = tuple((key, float(value)) for key, value in after_mass.items())
before_total = float(before_mass.sum())
after_total = float(after_mass.sum())
+ before_partition: tuple[tuple[object, tuple[tuple[object, float], ...]], ...] = ()
+ after_partition: tuple[tuple[object, tuple[tuple[object, float], ...]], ...] = ()
+ if mass_partition is not None:
+ before_partition = _mass_by_partition(before, mass_partition, node.id)
+ after_partition = _mass_by_partition(after, mass_partition, node.id)
if policy == "conserve":
+ if mass_partition is not None:
+ _assert_partition_mass_mapping(
+ before_partition,
+ after_partition,
+ label=f"Node {node.id!r} mass='conserve'",
+ )
_assert_mass_mapping(
dict(before_pairs),
dict(after_pairs),
@@ -1481,6 +2011,9 @@ def _mass_record(
before=dict(before_pairs),
after=dict(after_pairs),
node_id=node.id,
+ mass_partition=mass_partition,
+ before_partition=before_partition,
+ after_partition=after_partition,
)
elif policy == "declared":
raise PopulationError(
@@ -1507,6 +2040,89 @@ def _mass_record(
and "expand_weight_entity" in node.params
else None
),
+ partition_entity=(None if mass_partition is None else mass_partition[0]),
+ partition_column=(None if mass_partition is None else mass_partition[1]),
+ before_by_partition_stratum=before_partition,
+ after_by_partition_stratum=after_partition,
+ )
+
+
+def _partition_values_on_person(
+ frame: Frame,
+ mass_partition: tuple[str, str],
+ node_id: str,
+) -> pd.Series:
+ entity, column = mass_partition
+ if entity not in frame.entities:
+ raise PopulationError(
+ f"Node {node_id!r} mass partition names unknown entity {entity!r}."
+ )
+ table = frame.table(entity)
+ if column not in table:
+ raise PopulationError(
+ f"Node {node_id!r} mass partition column {entity}.{column} is "
+ "missing at run time."
+ )
+ person = frame.schema.person_entity
+ if entity == person:
+ return table[column].reset_index(drop=True)
+ if entity not in frame.schema.group_entities:
+ raise PopulationError(
+ f"Node {node_id!r} cannot broadcast mass partition entity {entity!r} "
+ "to persons."
+ )
+ id_column = frame.schema.entity_id_column(entity)
+ membership = frame.schema.membership_column(entity)
+ positions = pd.Index(table[id_column]).get_indexer(
+ frame.table(person)[membership].to_numpy(copy=False)
+ )
+ if (positions < 0).any(): # defended by Frame linkage validation
+ raise PopulationError(
+ f"Node {node_id!r} cannot align mass partition {entity}.{column} "
+ "to person memberships."
+ )
+ return table[column].iloc[positions].reset_index(drop=True)
+
+
+def _mass_by_partition(
+ frame: Frame,
+ mass_partition: tuple[str, str],
+ node_id: str,
+) -> tuple[tuple[object, tuple[tuple[object, float], ...]], ...]:
+ partition = _partition_values_on_person(frame, mass_partition, node_id)
+ person = frame.schema.person_entity
+ weights = frame.resolve_weights(person).values
+ strata = frame.strata.reset_index(drop=True)
+ valid = partition.notna().to_numpy(dtype=np.bool_, copy=False)
+ if not valid.any():
+ return ()
+ grouped = (
+ pd.DataFrame(
+ {
+ "_partition": partition.loc[valid].reset_index(drop=True),
+ "_stratum": strata.loc[valid].reset_index(drop=True),
+ "_mass": weights[valid],
+ }
+ )
+ .groupby(["_partition", "_stratum"], observed=True, sort=False)["_mass"]
+ .sum()
+ )
+ nested: dict[object, dict[object, float]] = {}
+ for (partition_value, stratum), mass in grouped.items():
+ nested.setdefault(partition_value, {})[stratum] = float(mass)
+ return tuple(
+ (
+ partition_value,
+ tuple(
+ sorted(
+ strata_mass.items(),
+ key=lambda item: _receipt_key(item[0]),
+ )
+ ),
+ )
+ for partition_value, strata_mass in sorted(
+ nested.items(), key=lambda item: _receipt_key(item[0])
+ )
)
@@ -1519,6 +2135,9 @@ def _validate_mass_receipt(
before: Mapping[object, float],
after: Mapping[object, float],
node_id: str,
+ mass_partition: tuple[str, str] | None,
+ before_partition: tuple[tuple[object, tuple[tuple[object, float], ...]], ...],
+ after_partition: tuple[tuple[object, tuple[tuple[object, float], ...]], ...],
) -> None:
if not isinstance(raw, Mapping):
raise PopulationError(f"Node {node_id!r} receipt['mass'] must be a mapping.")
@@ -1535,6 +2154,92 @@ def _validate_mass_receipt(
_assert_receipt_mapping(
raw.get("stratum_after"), after, f"Node {node_id!r} mass.stratum_after"
)
+ if mass_partition is None and "partition" in raw:
+ raise PopulationError(
+ f"Node {node_id!r} mass.partition is present but the graph "
+ "declares no mass partition."
+ )
+ if mass_partition is not None and "partition" in raw:
+ _validate_partition_mass_receipt(
+ raw["partition"],
+ mass_partition=mass_partition,
+ before=before_partition,
+ after=after_partition,
+ node_id=node_id,
+ )
+
+
+def _validate_partition_mass_receipt(
+ raw: object,
+ *,
+ mass_partition: tuple[str, str],
+ before: tuple[tuple[object, tuple[tuple[object, float], ...]], ...],
+ after: tuple[tuple[object, tuple[tuple[object, float], ...]], ...],
+ node_id: str,
+) -> None:
+ if not isinstance(raw, Mapping) or set(raw) != {
+ "entity",
+ "column",
+ "stratum_before",
+ "stratum_after",
+ }:
+ raise PopulationError(
+ f"Node {node_id!r} mass.partition must contain entity, column, "
+ "stratum_before, and stratum_after."
+ )
+ entity, column = mass_partition
+ if raw.get("entity") != entity or raw.get("column") != column:
+ raise PopulationError(
+ f"Node {node_id!r} mass.partition names "
+ f"{raw.get('entity')}.{raw.get('column')}; expected {entity}.{column}."
+ )
+ _assert_partition_receipt_mapping(
+ raw.get("stratum_before"),
+ before,
+ label=f"Node {node_id!r} mass.partition.stratum_before",
+ )
+ _assert_partition_receipt_mapping(
+ raw.get("stratum_after"),
+ after,
+ label=f"Node {node_id!r} mass.partition.stratum_after",
+ )
+
+
+def _assert_partition_receipt_mapping(
+ observed: object,
+ expected: tuple[tuple[object, tuple[tuple[object, float], ...]], ...],
+ *,
+ label: str,
+) -> None:
+ if not isinstance(observed, Mapping):
+ raise PopulationError(f"{label} must be a mapping.")
+ converted = _partition_receipt_mapping(expected)
+ if set(observed) != set(converted):
+ raise PopulationError(
+ f"{label} changed partitions: expected {list(converted)}, "
+ f"got {list(observed)}."
+ )
+ for partition, strata in converted.items():
+ _assert_receipt_mapping(
+ observed[partition], strata, f"{label} partition {partition!r}"
+ )
+
+
+def _assert_partition_mass_mapping(
+ expected: tuple[tuple[object, tuple[tuple[object, float], ...]], ...],
+ observed: tuple[tuple[object, tuple[tuple[object, float], ...]], ...],
+ *,
+ label: str,
+) -> None:
+ before = {partition: dict(strata) for partition, strata in expected}
+ after = {partition: dict(strata) for partition, strata in observed}
+ partitions = sorted(set(before) | set(after), key=_receipt_key)
+ for partition in partitions:
+ _assert_mass_mapping(
+ before.get(partition, {}),
+ after.get(partition, {}),
+ label=f"{label} partition {_receipt_key(partition)!r}",
+ )
def _assert_close(observed: object, expected: float, label: str) -> None:
@@ -1551,8 +2256,23 @@ def _assert_receipt_mapping(
) -> None:
if not isinstance(observed, Mapping):
raise PopulationError(f"{label} must be a mapping.")
- converted = {key: float(value) for key, value in observed.items()}
- _assert_mass_mapping(expected, converted, label=label)
+ expected_json = _receipt_mass_mapping(expected, label=label)
+ observed_json = _receipt_mass_mapping(observed, label=label)
+ _assert_mass_mapping(expected_json, observed_json, label=label)
+
+
+def _receipt_mass_mapping(
+ values: Mapping[object, object], *, label: str
+) -> dict[str, float]:
+ """Normalize a mass mapping to its stable JSON-object-key representation."""
+
+ result: dict[str, float] = {}
+ for raw_key, value in values.items():
+ key = _receipt_key(raw_key)
+ if key in result:
+ raise PopulationError(f"{label} has colliding JSON key {key!r}.")
+ result[key] = float(value)
+ return result
def _assert_mass_mapping(
diff --git a/packages/microcosm-graph/src/microcosm/graph/serialize.py b/packages/microcosm-graph/src/microcosm/graph/serialize.py
index 32b78f56d..33b5437f3 100644
--- a/packages/microcosm-graph/src/microcosm/graph/serialize.py
+++ b/packages/microcosm-graph/src/microcosm/graph/serialize.py
@@ -37,6 +37,13 @@ def graph_to_json(graph: Graph) -> str:
for source in graph.sources
],
"nodes": [_node_payload(node) for node in graph.nodes],
+ # Amendment 12: present only when declared, so a declaration written
+ # before the amendment serializes byte for byte as it did.
+ **(
+ {}
+ if graph.mass_partition is None
+ else {"mass_partition": list(graph.mass_partition)}
+ ),
}
return canonical_json(payload).decode("utf-8")
@@ -51,7 +58,10 @@ def graph_from_json(text: str) -> Graph:
except json.JSONDecodeError as error:
raise ValueError("graph JSON is not valid JSON") from error
root = _mapping(raw, "graph")
- _exact_fields(root, {"country", "sources", "nodes"}, "graph")
+ fields = {"country", "sources", "nodes"}
+ if "mass_partition" in root:
+ fields.add("mass_partition")
+ _exact_fields(root, fields, "graph")
sources_raw = _array(root["sources"], "graph.sources")
nodes_raw = _array(root["nodes"], "graph.nodes")
return Graph(
@@ -63,9 +73,21 @@ def graph_from_json(text: str) -> Graph:
nodes=tuple(
_node_from_payload(value, index) for index, value in enumerate(nodes_raw)
),
+ mass_partition=_partition_from_payload(
+ root.get("mass_partition"), "graph.mass_partition"
+ ),
)
+def _partition_from_payload(value: object, label: str) -> tuple[str, str] | None:
+ if value is None:
+ return None
+ parts = _array(value, label)
+ if len(parts) != 2:
+ raise TypeError(f"{label} must be an [entity, column] pair")
+ return (_string(parts[0], f"{label}[0]"), _string(parts[1], f"{label}[1]"))
+
+
def _node_payload(node: Node) -> dict[str, object]:
return {
"id": node.id,
@@ -104,6 +126,7 @@ def _node_payload(node: Node) -> dict[str, object]:
}
),
"mass": node.mass,
+ **({"entrants": True} if node.entrants else {}),
"description": node.description,
"citation": node.citation,
}
@@ -138,7 +161,12 @@ def _node_from_payload(value: object, index: int) -> Node:
"description",
"citation",
}
+ if "entrants" in payload:
+ fields.add("entrants")
_exact_fields(payload, fields, label)
+ entrants = payload.get("entrants", False)
+ if not isinstance(entrants, bool):
+ raise TypeError(f"{label}.entrants must be a boolean")
inputs = _array(payload["inputs"], f"{label}.inputs")
outputs = _array(payload["outputs"], f"{label}.outputs")
sources = _array(payload["sources"], f"{label}.sources")
@@ -173,6 +201,7 @@ def _node_from_payload(value: object, index: int) -> Node:
),
weights=_weights_from_payload(payload["weights"], f"{label}.weights"),
mass=_string(payload["mass"], f"{label}.mass"),
+ entrants=entrants,
description=_string(payload["description"], f"{label}.description"),
citation=_string(payload["citation"], f"{label}.citation"),
)
diff --git a/packages/microcosm-graph/src/microcosm/graph/view.py b/packages/microcosm-graph/src/microcosm/graph/view.py
index 98c3f0a7b..adb8d802b 100644
--- a/packages/microcosm-graph/src/microcosm/graph/view.py
+++ b/packages/microcosm-graph/src/microcosm/graph/view.py
@@ -91,6 +91,16 @@ def describe(
'Seed: int.from_bytes(sha256(b"seed\\0" + node_key)[:8], "little")'
)
else:
+ tolerance = run_receipt.capabilities.tolerance
+ tolerance_text = canonical_json(
+ None
+ if tolerance is None
+ else {
+ "rtol": tolerance.rtol,
+ "atol": tolerance.atol,
+ "ulps": tolerance.ulps,
+ }
+ ).decode("utf-8")
lines.extend(
[
'Seed: int.from_bytes(sha256(b"seed\\0" + node_key)[:8], '
@@ -102,7 +112,8 @@ def describe(
f"numeric={_value(run_receipt.capabilities.numeric)}, "
f"seed={_value(run_receipt.capabilities.seed_source)}, "
f"structural={_value(run_receipt.capabilities.structural)}, "
- f"consumes_se={run_receipt.capabilities.consumes_se}",
+ f"consumes_se={run_receipt.capabilities.consumes_se}, "
+ f"tolerance={tolerance_text}",
"Receipt: " + canonical_json(run_receipt.receipt).decode("utf-8"),
]
)
diff --git a/packages/microcosm-graph/tests/_toy.py b/packages/microcosm-graph/tests/_toy.py
index f37ab7767..c8ffbe78c 100644
--- a/packages/microcosm-graph/tests/_toy.py
+++ b/packages/microcosm-graph/tests/_toy.py
@@ -69,6 +69,7 @@
Slice,
SourceRef,
StructuralDelta,
+ Tolerance,
WeightTransition,
compile_graph,
run_graph,
@@ -83,6 +84,7 @@
"PUBLISH_DECISION",
"SOURCE",
"STRATA",
+ "TOY_TOLERANCE",
"ToyKernel",
"ToyRun",
"absent_node",
@@ -96,6 +98,8 @@
"descendants",
"draw",
"drop_nodes",
+ "entrant_expand_node",
+ "entrant_person_node",
"full_graph",
"gate_node",
"graph_source_files",
@@ -130,6 +134,9 @@
#: The one source every toy graph reads, through the ``csv-tables`` codec.
SOURCE = SourceRef("survey", "csv-tables", description="the toy country's tables")
+#: Cross-machine numeric movement declared by the C5 toy producer.
+TOY_TOLERANCE = Tolerance(rtol=1e-6)
+
def id_column(entity: str) -> str:
"""The frame convention: ``person_id`` for persons, ``{entity}_id`` elsewhere."""
@@ -432,6 +439,185 @@ def compute(self, context: KernelContext) -> KernelResult:
)
+class ExpandEntrants(ToyKernel):
+ """EXPAND: add one copied person and one materialized entrant household."""
+
+ def compute(self, context: KernelContext) -> KernelResult:
+ person = context.tables["person"]
+ person_ids = pd.Index(person["person_id"], name="person_id")
+ person_source_id = int(person_ids[0])
+ person_copy_id = int(person_ids.max()) + 1
+
+ household = context.tables["household"]
+ household_ids = pd.Index(household["household_id"], name="household_id")
+ household_entrant_id = int(household_ids.max()) + 1
+ household_target_ids = household_ids.append(
+ pd.Index([household_entrant_id], dtype="int64", name="household_id")
+ )
+ person_target_ids = person_ids.append(
+ pd.Index([person_copy_id], dtype="int64", name="person_id")
+ )
+
+ household_size = pd.concat(
+ [
+ household["household_size"].reset_index(drop=True),
+ pd.Series([1], dtype="int64"),
+ ],
+ ignore_index=True,
+ )
+ materialized_size = pd.Series(
+ household_size.array, index=household_target_ids, dtype="int64"
+ )
+ if context.params.get("missing_entrant_column") == "household_size":
+ materialized_size = materialized_size.drop(index=household_entrant_id)
+ memberships = pd.concat(
+ [
+ person["person_household_id"].reset_index(drop=True),
+ pd.Series([household_entrant_id], dtype="int64"),
+ ],
+ ignore_index=True,
+ )
+ materialized_memberships = pd.Series(
+ memberships.array, index=person_target_ids, dtype="int64"
+ )
+
+ empty_releases = pd.Series(
+ [],
+ index=pd.Index([], dtype="int64", name="release_id"),
+ dtype="int64",
+ )
+ household_weights = context.weights["household"]
+ expanded_weights = np.append(
+ household_weights.values, float(context.params["entrant_weight"])
+ )
+ return KernelResult(
+ expand={
+ "person": pd.Series(
+ [person_source_id],
+ index=pd.Index([person_copy_id], dtype="int64", name="person_id"),
+ dtype="int64",
+ ),
+ "household": pd.Series(
+ pd.array([pd.NA], dtype="Int64"),
+ index=pd.Index(
+ [household_entrant_id],
+ dtype="int64",
+ name="household_id",
+ ),
+ ),
+ "release": empty_releases,
+ },
+ columns={
+ ("household", "household_size"): materialized_size,
+ ("person", "person_household_id"): materialized_memberships,
+ },
+ weights=Weights(expanded_weights, kind=household_weights.kind),
+ )
+
+
+class ExpandEntrantPerson(ToyKernel):
+ """EXPAND: admit one entrant person into an incumbent household.
+
+ The entrant copies nothing: every person column is materialized from a
+ template row, its memberships name incumbent groups, and its stratum
+ arrives through ``KernelResult.strata`` (amendment 14). ``strata_mode``
+ exercises the refusals: ``missing`` omits the field, ``unknown_id``
+ labels an id the node never adds, ``labels_incumbent`` labels an
+ incumbent person as well, and ``labels_copied`` also copies the
+ template person to a second new id and labels that copy, which takes
+ its stratum from lineage and may not be labelled.
+ """
+
+ def compute(self, context: KernelContext) -> KernelResult:
+ person = context.tables["person"]
+ person_ids = pd.Index(person["person_id"], name="person_id")
+ template = person.iloc[0]
+ entrant_id = int(person_ids.max()) + 1
+ mode = str(context.params.get("strata_mode", "ok"))
+ copy_id = entrant_id + 1
+ added_ids = [entrant_id, copy_id] if mode == "labels_copied" else [entrant_id]
+ target_ids = person_ids.append(
+ pd.Index(added_ids, dtype="int64", name="person_id")
+ )
+
+ def overlay(column: str, dtype: str, value: object) -> pd.Series:
+ added = [value] + ([template[column]] if mode == "labels_copied" else [])
+ values = pd.concat(
+ [person[column].reset_index(drop=True), pd.Series(added)],
+ ignore_index=True,
+ )
+ return pd.Series(pd.array(values, dtype=dtype), index=target_ids)
+
+ columns = {
+ ("person", "age"): overlay("age", "int64", 30),
+ ("person", "income"): overlay("income", "float64", 12_500.0),
+ ("person", "is_adult"): overlay("is_adult", "boolean", True),
+ ("person", "receives_x"): overlay("receives_x", "boolean", False),
+ ("person", "person_household_id"): overlay(
+ "person_household_id", "int64", int(template["person_household_id"])
+ ),
+ ("person", "person_release_id"): overlay(
+ "person_release_id", "int64", int(template["person_release_id"])
+ ),
+ }
+ labelled = {
+ "ok": [entrant_id],
+ "unknown_id": [entrant_id + 1],
+ "labels_incumbent": [int(person_ids[0]), entrant_id],
+ "labels_copied": [entrant_id, copy_id],
+ }
+ strata = (
+ None
+ if mode == "missing"
+ else pd.Series(
+ ["urban"] * len(labelled[mode]),
+ index=pd.Index(labelled[mode], dtype="int64", name="person_id"),
+ dtype=object,
+ name="stratum",
+ )
+ )
+ empty = {
+ entity: pd.Series(
+ [],
+ index=pd.Index([], dtype="int64", name=id_column(entity)),
+ dtype="int64",
+ )
+ for entity in ("household", "release")
+ }
+ household_weights = context.weights["household"]
+ lineage = [pd.NA] + ([int(person_ids[0])] if mode == "labels_copied" else [])
+ return KernelResult(
+ expand={
+ "person": pd.Series(
+ pd.array(lineage, dtype="Int64"),
+ index=pd.Index(added_ids, dtype="int64", name="person_id"),
+ ),
+ **empty,
+ },
+ columns=columns,
+ weights=Weights(
+ household_weights.values.copy(), kind=household_weights.kind
+ ),
+ strata=strata,
+ )
+
+
+class ClaimMaterializedExpand(ToyKernel):
+ """Claim kernel-supplied EXPAND columns through the ownership surface."""
+
+ def compute(self, context: KernelContext) -> KernelResult:
+ columns: dict[tuple[str, str], pd.Series] = {}
+ for item in context.params["claim_cells"]:
+ entity, column, dtype = (str(value) for value in item)
+ table = context.tables[entity]
+ columns[(entity, column)] = pd.Series(
+ table[column].array.copy(),
+ index=pd.Index(table[id_column(entity)], name=id_column(entity)),
+ dtype=dtype,
+ )
+ return KernelResult(columns=columns)
+
+
class CalibrateToy(ToyKernel):
"""An ``importance -> calibrated`` transition hitting one target exactly."""
@@ -483,6 +669,37 @@ def compute(self, context: KernelContext) -> KernelResult:
)
+class GateReportsTolerance(ToyKernel):
+ """A gate that reports the input owner's declared numeric tolerance."""
+
+ def compute(self, context: KernelContext) -> KernelResult:
+ entity = str(context.params["entity"])
+ column = str(context.params["column"])
+ observed = float(context.tables[entity][column].astype("float64").mean())
+ declared = context.tolerances[(entity, column)]
+ tolerance = (
+ None
+ if declared is None
+ else {
+ "rtol": declared.rtol,
+ "atol": declared.atol,
+ "ulps": declared.ulps,
+ }
+ )
+ verdict_column = str(context.params["verdict_column"])
+ return KernelResult(
+ columns={
+ ("release", verdict_column): pd.Series(
+ ["pass"], index=_owned_ids(context, "release"), dtype="string"
+ )
+ },
+ receipt={
+ "outcome": "pass",
+ "evidence": {"observed": observed, "tolerance": tolerance},
+ },
+ )
+
+
class ReleaseTier(ToyKernel):
"""Derive a release tier from the gate verdicts declared as its inputs."""
@@ -626,6 +843,9 @@ def compute(self, context: KernelContext) -> KernelResult:
_FILTER = Capabilities(
determinism=Determinism.DETERMINISTIC, structural=StructuralDelta.FILTER
)
+_EXPAND = Capabilities(
+ determinism=Determinism.DETERMINISTIC, structural=StructuralDelta.EXPAND
+)
_REWEIGHT = Capabilities(
determinism=Determinism.DETERMINISTIC, structural=StructuralDelta.REWEIGHT
)
@@ -644,12 +864,23 @@ def toy_registry(*, variants: Mapping[str, str] | None = None) -> KernelRegistry
kernels = (
SourceCsv("source.csv@1", _CREATE),
DeriveAdd("derive.add@1", _DETERMINISTIC),
+ DeriveAdd(
+ "derive.tolerant@1",
+ Capabilities(
+ determinism=Determinism.DETERMINISTIC,
+ numeric=Numeric.TOLERANCE_BOUND,
+ tolerance=TOY_TOLERANCE,
+ ),
+ ),
DrawUniform("draw.uniform@1", _SEEDED),
ImputeChain("impute.chain@1", _SEEDED),
SimulateStub("simulate.stub@1", _DETERMINISTIC),
PatchColumn("patch.column@1", _DETERMINISTIC),
AbsentColumn("absent.column@1", _DETERMINISTIC),
SelectRows("select.rows@1", _FILTER),
+ ExpandEntrants("expand.entrants@1", _EXPAND),
+ ExpandEntrantPerson("expand.entrant_person@1", _EXPAND),
+ ClaimMaterializedExpand("claim.expand@1", _DETERMINISTIC),
ReweightScale("reweight.scale@1", _REWEIGHT),
CalibrateToy(
"calibrate.toy@1",
@@ -664,6 +895,10 @@ def toy_registry(*, variants: Mapping[str, str] | None = None) -> KernelRegistry
"gate.threshold@1",
Capabilities(determinism=Determinism.DETERMINISTIC, role=KernelRole.GATE),
),
+ GateReportsTolerance(
+ "gate.tolerance@1",
+ Capabilities(determinism=Determinism.DETERMINISTIC, role=KernelRole.GATE),
+ ),
ReleaseTier(
"release.tier@1",
Capabilities(
@@ -974,6 +1209,106 @@ def select_node(
)
+def entrant_expand_node(
+ node_id: str = "scheduled_entries",
+ *,
+ entrants: bool = True,
+ missing_entrant_column: str | None = None,
+) -> tuple[Node, Node]:
+ """An EXPAND plus the ownership claim for its materialized person fields."""
+ overlays = (
+ ("household", "household_size", "int64"),
+ ("person", "person_household_id", "int64"),
+ )
+ claim_cells = (overlays[0],)
+ expand = Node(
+ node_id,
+ "expand.entrants@1",
+ structural=StructuralDelta.EXPAND,
+ base="survey",
+ inputs=(
+ Slice("person", ("age",)),
+ Slice("household", ("household_size",)),
+ ),
+ params={
+ "expand_cells": overlays,
+ "expand_weight_entity": "household",
+ "expand_weight_kind": "design",
+ "entrant_weight": 125.0,
+ "missing_entrant_column": missing_entrant_column,
+ },
+ mass="free",
+ entrants=entrants,
+ )
+ claim = Node(
+ f"claim_{node_id}",
+ "claim.expand@1",
+ outputs=tuple(
+ Owned(entity, column, dtype) for entity, column, dtype in claim_cells
+ ),
+ params={
+ "claim_cells": claim_cells,
+ "materialized_expand_outputs": tuple(
+ f"{entity}.{column}" for entity, column, _ in claim_cells
+ ),
+ },
+ population=node_id,
+ )
+ return expand, claim
+
+
+def entrant_person_node(
+ node_id: str = "immigrant_cohort",
+ *,
+ strata_mode: str = "ok",
+) -> tuple[Node, Node]:
+ """An EXPAND admitting one entrant person, plus the claim of its cells."""
+ data_cells = (
+ ("person", "age", "int64"),
+ ("person", "income", "float64"),
+ ("person", "is_adult", "boolean"),
+ ("person", "receives_x", "boolean"),
+ )
+ overlays = (
+ *data_cells,
+ ("person", "person_household_id", "int64"),
+ ("person", "person_release_id", "int64"),
+ )
+ expand = Node(
+ node_id,
+ "expand.entrant_person@1",
+ structural=StructuralDelta.EXPAND,
+ base="survey",
+ inputs=(
+ Slice("person", ("age", "income", "is_adult", "receives_x")),
+ Slice("household", ("household_size",)),
+ ),
+ params={
+ "expand_cells": overlays,
+ "expand_weight_entity": "household",
+ "expand_weight_kind": "design",
+ "strata_mode": strata_mode,
+ },
+ mass="free",
+ entrants=True,
+ )
+ claim = Node(
+ f"claim_{node_id}",
+ "claim.expand@1",
+ outputs=tuple(
+ Owned(entity, column, dtype) for entity, column, dtype in data_cells
+ ),
+ params={
+ "claim_cells": data_cells,
+ "materialized_expand_outputs": tuple(
+ f"{entity}.{column}" for entity, column, _ in data_cells
+ ),
+ },
+ population=node_id,
+ )
+ return expand, claim
+
+
POOL = Node(
"pool",
"reweight.scale@1",
diff --git a/packages/microcosm-graph/tests/fixtures/parity/kernels/fit.qrf/pins.json b/packages/microcosm-graph/tests/fixtures/parity/kernels/fit.qrf/pins.json
index ebfa035a4..8a2b410e5 100644
--- a/packages/microcosm-graph/tests/fixtures/parity/kernels/fit.qrf/pins.json
+++ b/packages/microcosm-graph/tests/fixtures/parity/kernels/fit.qrf/pins.json
@@ -1 +1 @@
-{"dependencies":{"numpy":"2.4.6","pandas":"3.0.3","quantile-forest":"1.4.2","scikit-learn":"1.8.0"},"implementation_hash":"8434e3e7e6333a4bb093f5d54a93e38d393553d3b27d811bec7671828688d707","kernel":"fit.qrf@1","node":"fit_qrf","seed":947}
+{"dependencies":{"numpy":"2.4.6","pandas":"3.0.3","quantile-forest":"1.4.2","scikit-learn":"1.8.0"},"implementation_hash":"0ac1272b574a46d1407c734d9edaaee4951d35d56b06f3efe778b72e4779c82e","kernel":"fit.qrf@1","node":"fit_qrf","seed":947}
diff --git a/packages/microcosm-graph/tests/test_acceptance_b_ownership.py b/packages/microcosm-graph/tests/test_acceptance_b_ownership.py
index 72836472c..0b0426c0a 100644
--- a/packages/microcosm-graph/tests/test_acceptance_b_ownership.py
+++ b/packages/microcosm-graph/tests/test_acceptance_b_ownership.py
@@ -22,7 +22,16 @@
import pandas as pd
import pytest
-from microcosm.graph import KernelContext, Node, Owned, Ownership, Slice
+from microcosm.graph import (
+ GraphError,
+ KernelContext,
+ Node,
+ NodeRejectedError,
+ Owned,
+ Ownership,
+ Slice,
+ StructuralDelta,
+)
if "_toy" not in sys.modules:
_SPEC = importlib.util.spec_from_file_location(
@@ -107,6 +116,7 @@ def test_b2_executor_enforces_ownership(tmp_path: Path) -> None:
"params",
"rng",
"sources",
+ "tolerances", # amendment 13: declared tolerances of the inputs' owners
}
graph = toy.small_graph(
@@ -232,3 +242,115 @@ def test_b5_null_means_absence(tmp_path: Path) -> None:
with pytest.raises(NodeRejectedError, match="no_data"):
toy.run_toy(liar, tmp_path / "liar")
+
+
+def test_b6_entrants_are_declared(tmp_path: Path) -> None:
+ """Null lineage is an explicit, complete, and receipted entrant contract.
+
+ The backwards-compatible lineage shape keeps ``(new, source)`` pairs under
+ ``expand`` and uses a null source for entrants. The test admits a household
+ entrant because the frozen result interface has no separate output for a
+ new person's stratum. Its complete carried data surface is materialized by
+ the EXPAND kernel and passed through ``materialized_expand_outputs``.
+ """
+ with pytest.raises(GraphError, match="conserved_entrants"):
+ Node(
+ "conserved_entrants",
+ "expand.entrants@1",
+ structural=StructuralDelta.EXPAND,
+ base="survey",
+ entrants=True,
+ mass="conserve",
+ )
+
+ expand, claim = toy.entrant_expand_node()
+ run = toy.run_toy(
+ toy.small_graph(nodes=(toy.CREATE, expand, claim)), tmp_path / "declared"
+ )
+ before = run.manifest.population("survey")
+ after = run.manifest.population(expand.id)
+ person_copy_id = int(before.person["person_id"].max()) + 1
+ household_entrant_id = int(before.household["household_id"].max()) + 1
+
+ assert len(after.person) == len(before.person) + 1
+ assert len(after.household) == len(before.household) + 1
+ copied = after.person.set_index("person_id").loc[person_copy_id]
+ source = before.person.set_index("person_id").loc[1]
+ assert copied["person_household_id"] == household_entrant_id
+ pd.testing.assert_series_equal(
+ copied.drop(labels="person_household_id"),
+ source.drop(labels="person_household_id"),
+ check_names=False,
+ )
+ entrant = after.household.set_index("household_id").loc[household_entrant_id]
+ assert entrant["household_size"] == 1
+ assert after.household["household_size"].dtype == np.dtype("int64")
+ assert after.weights_for("household").values[-1] == 125.0
+ lineage = run.manifest.nodes[expand.id].receipt
+ assert lineage["expand"]["person"] == ((person_copy_id, 1),)
+ assert lineage["expand"]["household"] == ((household_entrant_id, None),)
+
+ undeclared, undeclared_claim = toy.entrant_expand_node(
+ "undeclared_entrants", entrants=False
+ )
+ with pytest.raises(NodeRejectedError, match="undeclared_entrants"):
+ toy.run_toy(
+ toy.small_graph(nodes=(toy.CREATE, undeclared, undeclared_claim)),
+ tmp_path / "undeclared",
+ )
+
+ incomplete, incomplete_claim = toy.entrant_expand_node(
+ "incomplete_entrant", missing_entrant_column="household_size"
+ )
+ with pytest.raises(NodeRejectedError, match="incomplete_entrant") as error:
+ toy.run_toy(
+ toy.small_graph(nodes=(toy.CREATE, incomplete, incomplete_claim)),
+ tmp_path / "incomplete",
+ )
+ assert "household_size" in str(error.value)
+
+
+def test_b7_entrant_persons_carry_their_stratum(tmp_path: Path) -> None:
+ """An entrant person's stratum arrives through ``KernelResult.strata``.
+
+ Immigrant cohorts are persons, so an EXPAND admitting entrants must be
+ able to add a person that copies nobody: every column materialized,
+ memberships naming incumbent groups, and its stratum declared by id.
+ The ledger counts the entrant from the node that admits it; a missing
+ label, a label for an id the node never adds, or a label for an
+ incumbent person rejects the node by name.
+ """
+ expand, claim = toy.entrant_person_node()
+ run = toy.run_toy(
+ toy.small_graph(nodes=(toy.CREATE, expand, claim)), tmp_path / "cohort"
+ )
+ before = run.manifest.population("survey")
+ after = run.manifest.population(expand.id)
+ entrant_id = int(before.person["person_id"].max()) + 1
+
+ assert len(after.person) == len(before.person) + 1
+ assert len(after.household) == len(before.household)
+ entrant = after.person.set_index("person_id").loc[entrant_id]
+ assert entrant["age"] == 30 and entrant["income"] == 12_500.0
+ assert (
+ entrant["person_household_id"] == before.person["person_household_id"].iloc[0]
+ )
+ assert after.strata.iloc[-1] == "urban"
+ assert after.strata.iloc[: len(before.person)].tolist() == before.strata.tolist()
+
+ receipt = run.manifest.nodes[expand.id].receipt
+ assert receipt["expand"]["person"] == ((entrant_id, None),)
+ mass = receipt["mass"]
+ assert mass["after"] > mass["before"]
+ assert mass["stratum_after"]["urban"] > mass["stratum_before"]["urban"]
+ assert mass["stratum_after"]["rural"] == mass["stratum_before"]["rural"]
+
+ # A copied person takes its stratum from lineage, so a label for it is
+ # refused exactly like a label for an incumbent or an unknown id.
+ for mode in ("missing", "unknown_id", "labels_incumbent", "labels_copied"):
+ bad, bad_claim = toy.entrant_person_node(f"cohort_{mode}", strata_mode=mode)
+ with pytest.raises(NodeRejectedError, match=f"cohort_{mode}"):
+ toy.run_toy(
+ toy.small_graph(nodes=(toy.CREATE, bad, bad_claim)),
+ tmp_path / mode,
+ )
diff --git a/packages/microcosm-graph/tests/test_acceptance_c_seeds.py b/packages/microcosm-graph/tests/test_acceptance_c_seeds.py
index 07165e789..54f398fee 100644
--- a/packages/microcosm-graph/tests/test_acceptance_c_seeds.py
+++ b/packages/microcosm-graph/tests/test_acceptance_c_seeds.py
@@ -18,9 +18,23 @@
import ast
import importlib.util
import sys
+from dataclasses import replace
from pathlib import Path
-from microcosm.graph import ContentStore, Graph, compile_graph
+import pytest
+
+from microcosm.graph import (
+ Capabilities,
+ ContentStore,
+ Determinism,
+ Graph,
+ Node,
+ Numeric,
+ Owned,
+ Slice,
+ Tolerance,
+ compile_graph,
+)
if "_toy" not in sys.modules:
_SPEC = importlib.util.spec_from_file_location(
@@ -167,3 +181,99 @@ def test_c4_seed_from_identity(tmp_path: Path) -> None:
assert elsewhere.keys()["draw_z"] != here.keys()["draw_a"]
assert elsewhere.seeds()["draw_z"] != here.seeds()["draw_a"]
assert len(set(elsewhere.seeds().values())) == len(elsewhere.seeds())
+
+
+def test_c5_tolerance_is_declared(tmp_path: Path) -> None:
+ """Receipts and readers carry an owner's exact declared tolerance.
+
+ Capability receipts encode a tolerance as ``rtol``, ``atol``, and ``ulps``;
+ bitwise owners encode it as ``None``. A gate reads that same owner mapping
+ by coordinate and reports the JSON-safe value in its evidence.
+ """
+ with pytest.raises(ValueError, match="must declare its Tolerance"):
+ Capabilities(
+ determinism=Determinism.DETERMINISTIC,
+ numeric=Numeric.TOLERANCE_BOUND,
+ )
+ with pytest.raises(ValueError, match="bitwise kernel declares no Tolerance"):
+ Capabilities(
+ determinism=Determinism.DETERMINISTIC,
+ numeric=Numeric.BITWISE,
+ tolerance=Tolerance(rtol=1e-6),
+ )
+
+ bounded = Node(
+ "bounded",
+ "derive.tolerant@1",
+ inputs=(Slice("person", ("age",)),),
+ outputs=(Owned("person", "bounded_value", "float64"),),
+ params={
+ "entity": "person",
+ "columns": ("age",),
+ "target": "bounded_value",
+ "scale": 1.0,
+ },
+ population="survey",
+ )
+ bitwise = toy.derive("bitwise", ("age",), "bitwise_value")
+
+ def tolerance_gate(node_id: str, column: str) -> Node:
+ verdict = f"{node_id}_verdict"
+ return Node(
+ node_id,
+ "gate.tolerance@1",
+ inputs=(Slice("person", (column,)),),
+ outputs=(Owned("release", verdict, "string"),),
+ params={
+ "entity": "person",
+ "column": column,
+ "verdict_column": verdict,
+ },
+ population="survey",
+ )
+
+ bounded_gate = tolerance_gate("bounded_gate", "bounded_value")
+ bitwise_gate = tolerance_gate("bitwise_gate", "bitwise_value")
+ graph = Graph(
+ "toy",
+ (toy.SOURCE,),
+ (toy.CREATE, bounded, bitwise, bounded_gate, bitwise_gate),
+ )
+ run = toy.run_toy(graph, tmp_path / "run")
+ bound = {"rtol": 1e-6, "atol": 0.0, "ulps": 0}
+
+ bounded_receipt = run.manifest.nodes[bounded.id].receipt
+ assert bounded_receipt["capabilities"]["tolerance"] == bound
+ bounded_evidence = run.manifest.nodes[bounded_gate.id].receipt
+ assert bounded_evidence["outcome"] == "pass"
+ assert bounded_evidence["evidence"]["tolerance"] == bound
+
+ bitwise_receipt = run.manifest.nodes[bitwise.id].receipt
+ assert bitwise_receipt["capabilities"]["tolerance"] is None
+ bitwise_evidence = run.manifest.nodes[bitwise_gate.id].receipt
+ assert bitwise_evidence["outcome"] == "pass"
+ assert bitwise_evidence["evidence"]["tolerance"] is None
+
+ # A carried column keeps its producer's tolerance through a structural
+ # version: the bitwise FILTER that carries ``bounded_value`` into
+ # ``adults_view`` neither tightens nor erases the bound a reader sees.
+ carrier = toy.select_node("adults_view", base="survey", policy="free")
+ carried_gate = replace(
+ tolerance_gate("carried_gate", "bounded_value"), population=carrier.id
+ )
+ carried_bitwise_gate = replace(
+ tolerance_gate("carried_bitwise_gate", "bitwise_value"),
+ population=carrier.id,
+ )
+ carried = Graph(
+ "toy",
+ (toy.SOURCE,),
+ (toy.CREATE, bounded, bitwise, carrier, carried_gate, carried_bitwise_gate),
+ )
+ carried_run = toy.run_toy(carried, tmp_path / "carried")
+ carried_evidence = carried_run.manifest.nodes[carried_gate.id].receipt
+ assert carried_evidence["outcome"] == "pass"
+ assert carried_evidence["evidence"]["tolerance"] == bound
+ carried_bitwise = carried_run.manifest.nodes[carried_bitwise_gate.id].receipt
+ assert carried_bitwise["outcome"] == "pass"
+ assert carried_bitwise["evidence"]["tolerance"] is None
diff --git a/packages/microcosm-graph/tests/test_acceptance_d_weights.py b/packages/microcosm-graph/tests/test_acceptance_d_weights.py
index 4b254bd78..baf5c45c1 100644
--- a/packages/microcosm-graph/tests/test_acceptance_d_weights.py
+++ b/packages/microcosm-graph/tests/test_acceptance_d_weights.py
@@ -15,15 +15,21 @@
from __future__ import annotations
import importlib.util
+import json
import sys
+from dataclasses import replace
from pathlib import Path
+import numpy as np
+import pandas as pd
import pytest
from microcosm.frame import WeightKind
from microcosm.graph import (
+ Graph,
GraphError,
Node,
+ NodeRejectedError,
Owned,
Slice,
StructuralDelta,
@@ -215,6 +221,135 @@ def test_d5_uncertainty_travels(tmp_path: Path) -> None:
assert toy.calibrated_node(kernel="calibrate.blind@1").params["target_se"] == 2500.0
+def test_d6_mass_is_partitioned(tmp_path: Path) -> None:
+ """Mass is conserved and receipted inside every partition value.
+
+ The receipt retains D2's flat totals and stratum maps and adds
+ ``mass["partition"]`` with ``entity``, ``column``, and nested
+ ``stratum_before``/``stratum_after`` maps keyed first by the JSON-string
+ partition value and then by stratum. This test fixes that public shape for
+ the implementation lane.
+ """
+ missing = Graph(
+ "toy",
+ (toy.SOURCE,),
+ (toy.CREATE,),
+ mass_partition=("person", "period"),
+ )
+ with pytest.raises(GraphError, match=r"person\.period.*survey"):
+ compile_graph(missing)
+
+ source_path = toy.copy_source(tmp_path / "period-source")
+ person_path = source_path / "person.csv"
+ person = pd.read_csv(person_path)
+ is_adult = person["is_adult"].astype("boolean").fillna(False).to_numpy(bool)
+ person["period"] = np.where(
+ ~is_adult | person["person_id"].mod(2).eq(0), 2024, 2025
+ ).astype("int64")
+ person.to_csv(person_path, index=False)
+
+ schema_path = source_path / "schema.json"
+ schema = json.loads(schema_path.read_text())
+ schema["data_columns"]["person"].append("period")
+ schema["dtypes"]["period"] = "int64"
+ schema_path.write_text(json.dumps(schema, indent=2, sort_keys=True) + "\n")
+
+ create = replace(
+ toy.CREATE,
+ outputs=(*toy.CREATE.outputs, Owned("person", "period", "int64")),
+ )
+
+ # A partition value is fixed when the row is created: a later node that
+ # writes or rewrites the column could move mass between partitions with
+ # the total unchanged, which no policy can see, so compilation refuses it.
+ reassigning = Graph(
+ "toy",
+ (toy.SOURCE,),
+ (
+ create,
+ toy.select_node("period_view", base="survey", policy="free"),
+ Node(
+ "reassign_period",
+ "derive.rewrite@1",
+ inputs=(Slice("person", ("age",)),),
+ outputs=(Owned("person", "period", "int64", rewrite=True),),
+ population="period_view",
+ ),
+ ),
+ mass_partition=("person", "period"),
+ )
+ with pytest.raises(GraphError, match=r"reassign_period.*mass partition"):
+ compile_graph(reassigning)
+
+ sources = {"survey": source_path}
+ conserving = Graph(
+ "toy",
+ (toy.SOURCE,),
+ (
+ create,
+ toy.select_node("period_conserve", base="survey", policy="conserve"),
+ ),
+ mass_partition=("person", "period"),
+ )
+ with pytest.raises(NodeRejectedError, match="period_conserve") as rejected:
+ toy.run_toy(conserving, tmp_path / "conserve", sources=sources)
+ assert "2024" in str(rejected.value)
+
+ free = Graph(
+ "toy",
+ (toy.SOURCE,),
+ (create, toy.select_node("period_free", base="survey", policy="free")),
+ mass_partition=("person", "period"),
+ )
+ run = toy.run_toy(free, tmp_path / "free", sources=sources)
+ mass = run.manifest.nodes["period_free"].receipt["mass"]
+ partition = mass["partition"]
+ assert set(partition) == {
+ "entity",
+ "column",
+ "stratum_before",
+ "stratum_after",
+ }
+ assert partition["entity"] == "person"
+ assert partition["column"] == "period"
+
+ weights = pd.read_csv(source_path / "weights.csv").set_index("household_id")[
+ "design_weight"
+ ]
+ weighted = person.assign(
+ _mass=person["person_household_id"].map(weights).astype("float64")
+ )
+
+ def expected(frame: pd.DataFrame) -> dict[str, dict[str, float]]:
+ grouped = frame.groupby(["period", "stratum"], observed=True)["_mass"].sum()
+ return {
+ str(period): {
+ str(stratum): float(grouped.loc[(period, stratum)])
+ for stratum in grouped.loc[period].index
+ }
+ for period in grouped.index.get_level_values("period").unique()
+ }
+
+ before = expected(weighted)
+ after = expected(weighted.loc[is_adult])
+ for key, wanted in (("stratum_before", before), ("stratum_after", after)):
+ actual = partition[key]
+ assert set(actual) == set(wanted) == {"2024", "2025"}
+ for period, strata in wanted.items():
+ assert actual[period] == pytest.approx(strata)
+
+ before_total = sum(sum(strata.values()) for strata in before.values())
+ after_total = sum(sum(strata.values()) for strata in after.values())
+ assert mass["before"] == pytest.approx(before_total)
+ assert mass["after"] == pytest.approx(after_total)
+ assert partition["stratum_after"]["2025"] == pytest.approx(
+ partition["stratum_before"]["2025"]
+ )
+ assert sum(partition["stratum_after"]["2024"].values()) < sum(
+ partition["stratum_before"]["2024"].values()
+ )
+
+
def test_the_toy_country_declares_the_weight_lineage_the_charter_names() -> None:
"""The toy graph really does compose the three kinds, in order.
diff --git a/packages/microcosm-graph/tests/test_graph_acceptance_burndown.py b/packages/microcosm-graph/tests/test_graph_acceptance_burndown.py
index dc1cc330d..53c07444d 100644
--- a/packages/microcosm-graph/tests/test_graph_acceptance_burndown.py
+++ b/packages/microcosm-graph/tests/test_graph_acceptance_burndown.py
@@ -273,7 +273,9 @@ def test_the_real_suite_is_all_strict_and_all_accounted_for() -> None:
root = burndown.ROOT
assert data["total"] == 0
assert not [entry for entry in data["properties"] if entry["state"] == "missing"]
- assert all(entry["state"] == "green" for entry in data["properties"])
+ states = {entry["id"]: entry["state"] for entry in data["properties"]}
+ assert not {identifier for identifier, state in states.items() if state == "red"}
+ assert all(state == "green" for state in states.values())
for entry in data["files"]:
source = (root / entry["file"]).read_text()
for marker in markers_in(source, entry["file"]):
@@ -291,3 +293,134 @@ def test_the_tool_runs_from_the_command_line(flag: str) -> None:
)
assert completed.returncode == 0, completed.stderr
assert completed.stdout
+
+
+NEW_PROPERTY_STARTS_RED = (
+ ONE_RED_PROPERTY
+ + """
+
+@pytest.mark.xfail(strict=True, reason="charter A9: pending")
+def test_a9_nine() -> None:
+ assert False
+"""
+)
+
+
+A1_GREEN_A3_RED = """
+import pytest
+
+
+def test_a1_one() -> None:
+ assert True
+
+
+@pytest.mark.xfail(strict=True, reason="charter A3: pending")
+def test_a3_three() -> None:
+ assert False
+"""
+
+
+def test_verify_ratchets_on_property_identity_not_on_counts(tmp_path: Path) -> None:
+ """A1 going green cannot pay for A3 going red: the count is 1 -> 1, and
+ the ratchet still refuses, naming the property and the file."""
+ root = _repository(tmp_path, {"test_acceptance_a.py": ONE_RED_PROPERTY})
+ target = root / "packages" / "microcosm-graph" / "tests" / "test_acceptance_a.py"
+ target.write_text(A1_GREEN_A3_RED)
+ swapped = _run(root, "--verify")
+ assert swapped.returncode == 1
+ assert "1 -> 1" in swapped.stdout
+ assert "re-reds 1 property: A3" in swapped.stdout
+ assert "test_acceptance_a.py" in swapped.stdout
+ assert "verification=failed" in swapped.stdout
+
+
+def test_verify_counts_markers_in_new_files(tmp_path: Path) -> None:
+ """A file the baseline lacks constrains nothing by itself, but a marker
+ inside it on a property that was green is still a re-red."""
+ root = _repository(tmp_path, {"test_acceptance_a.py": ONE_RED_PROPERTY})
+ tests = root / "packages" / "microcosm-graph" / "tests"
+ (tests / "test_acceptance_b.py").write_text(
+ 'import pytest\n\n\n@pytest.mark.xfail(strict=True, reason="charter A3: moved")\n'
+ "def test_a3_three_again() -> None:\n assert False\n"
+ )
+ moved = _run(root, "--verify")
+ assert moved.returncode == 1
+ assert "[new] packages/microcosm-graph/tests/test_acceptance_b.py" in moved.stdout
+ assert "re-reds 1 property: A3" in moved.stdout
+ assert "test_acceptance_b.py" in moved.stdout
+
+
+def test_verify_refuses_a_marker_on_an_id_the_charter_does_not_list(
+ tmp_path: Path,
+) -> None:
+ """An invented id is not a new property; it is a marker on nothing."""
+ root = _repository(tmp_path, {"test_acceptance_a.py": ONE_RED_PROPERTY})
+ target = root / "packages" / "microcosm-graph" / "tests" / "test_acceptance_a.py"
+ target.write_text(NEW_PROPERTY_STARTS_RED) # A9 is not in the charter here
+ invented = _run(root, "--verify")
+ assert invented.returncode == 1
+ assert "names A9, which docs/graph-acceptance.md does not list" in invented.stdout
+
+
+MISLABELLED_MARKER = """
+import pytest
+
+
+@pytest.mark.xfail(strict=True, reason="charter A3: pending")
+def test_a1_one() -> None:
+ assert False
+
+
+@pytest.mark.xfail(strict=True, reason="charter A3: pending")
+def test_a3_three() -> None:
+ assert False
+"""
+
+
+def test_verify_binds_a_marker_to_the_test_it_decorates(tmp_path: Path) -> None:
+ """The reason is free text; the test name is the binding.
+
+ A marker on ``test_a1_*`` that claims A3 would let A1 go red while the
+ set of red ids stays {A3}; the tool refuses the mismatch and the
+ duplicate, so the ratchet compares one marker per property.
+ """
+ root = _repository(tmp_path, {"test_acceptance_a.py": ONE_RED_PROPERTY})
+ target = root / "packages" / "microcosm-graph" / "tests" / "test_acceptance_a.py"
+ target.write_text(MISLABELLED_MARKER)
+ mislabelled = _run(root, "--verify")
+ assert mislabelled.returncode == 1
+ assert "test_a1_one claims charter A3 but is not that property's test" in (
+ mislabelled.stdout
+ )
+ assert "A3 carries two markers" in mislabelled.stdout
+
+
+def test_verify_lets_a_property_new_to_the_charter_start_red(tmp_path: Path) -> None:
+ """The charter's meta-TDD rule: a new property is committed red first.
+
+ A marker on an id the baseline charter never listed is not a re-red; a
+ marker on an id it did list still is.
+ """
+ root = _repository(tmp_path, {"test_acceptance_a.py": ONE_RED_PROPERTY})
+ (root / "docs" / "graph-acceptance.md").write_text(
+ "| Id | Property |\n|---|---|\n| A1 | one |\n| A3 | three |\n| A9 | nine |\n"
+ )
+ target = root / "packages" / "microcosm-graph" / "tests" / "test_acceptance_a.py"
+ target.write_text(NEW_PROPERTY_STARTS_RED)
+ admitted = _run(root, "--verify")
+ assert admitted.returncode == 0, admitted.stdout
+ assert "1 -> 1 (+1 new: A9)" in admitted.stdout
+ assert "verification=ok" in admitted.stdout
+
+ # The same marker on a property the baseline charter already listed
+ # (A3, green there) is a re-red and still fails.
+ target.write_text(
+ NEW_PROPERTY_STARTS_RED.replace(
+ "def test_a3_three() -> None:\n assert True",
+ '@pytest.mark.xfail(strict=True, reason="charter A3: pending")\n'
+ "def test_a3_three() -> None:\n assert False",
+ )
+ )
+ refused = _run(root, "--verify")
+ assert refused.returncode == 1
+ assert "re-reds 1 property" in refused.stdout
diff --git a/packages/microcosm-graph/tests/test_graph_decl.py b/packages/microcosm-graph/tests/test_graph_decl.py
index c5065ae6f..37bb7b2e0 100644
--- a/packages/microcosm-graph/tests/test_graph_decl.py
+++ b/packages/microcosm-graph/tests/test_graph_decl.py
@@ -270,3 +270,59 @@ def test_a_filter_cannot_read_a_column_nobody_defines() -> None:
)
with pytest.raises(GraphError, match="no node owns"):
compile_graph(Graph("toy", (SRC,), (CREATE, subset)))
+
+
+def _expand(node_id: str, **overrides: object) -> Node:
+ settings: dict[str, object] = {"mass": "free", **overrides}
+ return Node(
+ node_id,
+ "clone.persons@1",
+ base="survey",
+ structural=StructuralDelta.EXPAND,
+ **settings, # type: ignore[arg-type]
+ )
+
+
+def test_entrants_are_an_expand_declaration_that_cannot_conserve_mass() -> None:
+ """Amendment 11: entrants add rows without lineage, so they add mass."""
+ admitted = _expand("cohort", entrants=True)
+ assert admitted.entrants is True
+ assert "entrants" in admitted.normative()
+ with pytest.raises(GraphError, match="cannot declare mass='conserve'"):
+ _expand("cohort", entrants=True, mass="conserve")
+ with pytest.raises(GraphError, match="only an EXPAND node"):
+ Node("fit", "fit.qrf@1", entrants=True)
+ with pytest.raises(GraphError, match="must be a boolean"):
+ _expand("cohort", entrants="yes")
+
+
+def test_mass_partition_must_be_declared_by_every_create_node() -> None:
+ """Amendment 12: partitions exist from the first version, with a partition dtype."""
+ periodic = Node(
+ "survey",
+ "source.frame@1",
+ sources=("survey",),
+ structural=StructuralDelta.CREATE,
+ outputs=(Owned("person", "age", "int64"), Owned("person", "period", "int64")),
+ )
+ graph = Graph("toy", (SRC,), (periodic,), mass_partition=("person", "period"))
+ compiled = compile_graph(graph)
+ assert compiled.graph.normative() == {"mass_partition": ("person", "period")}
+ assert Graph("toy", (SRC,), (periodic,)).normative() == {"mass_partition": None}
+ with pytest.raises(GraphError, match="does not declare"):
+ compile_graph(
+ Graph("toy", (SRC,), (CREATE,), mass_partition=("person", "period"))
+ )
+ floating = Node(
+ "survey",
+ "source.frame@1",
+ sources=("survey",),
+ structural=StructuralDelta.CREATE,
+ outputs=(Owned("person", "period", "float64"),),
+ )
+ with pytest.raises(GraphError, match="partition column must be one of"):
+ compile_graph(
+ Graph("toy", (SRC,), (floating,), mass_partition=("person", "period"))
+ )
+ with pytest.raises(GraphError, match="pair of strings"):
+ Graph("toy", (SRC,), (periodic,), mass_partition=("person",)) # type: ignore[arg-type]
diff --git a/packages/microcosm-graph/tests/test_graph_executor.py b/packages/microcosm-graph/tests/test_graph_executor.py
index 8da32936b..6a615c370 100644
--- a/packages/microcosm-graph/tests/test_graph_executor.py
+++ b/packages/microcosm-graph/tests/test_graph_executor.py
@@ -11,9 +11,11 @@
import pandas as pd
import pytest
+import microcosm.graph.executor as graph_executor
from microcosm.frame import EntitySchema, Frame, WeightKind, Weights
from microcosm.graph.decl import (
Graph,
+ GraphError,
Node,
Owned,
Ownership,
@@ -31,6 +33,8 @@
KernelRegistry,
KernelResult,
KernelRole,
+ Numeric,
+ Tolerance,
)
from microcosm.graph.manifest import Decision
from microcosm.graph.store import (
@@ -667,10 +671,13 @@ def cross_entity(context: KernelContext) -> KernelResult:
)
+@pytest.mark.parametrize("explicit_rewrite_input", [False, True])
def test_rewrite_incumbent_is_projected_from_its_owned_declaration(
tmp_path: Path,
+ explicit_rewrite_input: bool,
) -> None:
source = _source_path(tmp_path / "source")
+ boundary_tolerance = Tolerance(atol=1e-6)
def keep_all(context: KernelContext) -> KernelResult:
person = context.tables["person"]
@@ -686,6 +693,15 @@ def rewrite(context: KernelContext) -> KernelResult:
"age",
"income",
}
+ # The FILTER that opens this version carries ``age`` and ``income``
+ # from CREATE without writing them, so the tolerances a reader (and
+ # a rewrite, for its incumbent) sees are the producer's — bitwise
+ # here — not the tolerance-bound carrier's (charter C5).
+ assert context.tolerances == {
+ ("person", "age"): None,
+ ("person", "income"): None,
+ }
+
return KernelResult(
columns={
("person", "income"): pd.Series(
@@ -706,7 +722,9 @@ def rewrite(context: KernelContext) -> KernelResult:
rewriter = Node(
"rewrite_income",
"rewrite.income@1",
- inputs=(Slice("person", ("age",)),),
+ inputs=(
+ Slice("person", ("age", "income") if explicit_rewrite_input else ("age",)),
+ ),
outputs=(Owned("person", "income", "float64", rewrite=True),),
population=boundary.id,
)
@@ -714,7 +732,12 @@ def rewrite(context: KernelContext) -> KernelResult:
registry.register(
_Kernel(
boundary.kernel,
- Capabilities(Determinism.DETERMINISTIC, structural=StructuralDelta.FILTER),
+ Capabilities(
+ Determinism.DETERMINISTIC,
+ numeric=Numeric.TOLERANCE_BOUND,
+ structural=StructuralDelta.FILTER,
+ tolerance=boundary_tolerance,
+ ),
keep_all,
)
)
@@ -729,7 +752,7 @@ def rewrite(context: KernelContext) -> KernelResult:
manifest = _run(
Graph("toy", (SOURCE,), (CREATE, boundary, rewriter)),
source,
- ContentStore(tmp_path / "store"),
+ ContentStore(tmp_path / f"store-{explicit_rewrite_input}"),
registry,
)
@@ -889,6 +912,239 @@ def claim(context: KernelContext) -> KernelResult:
assert claim_kernel.calls == 1
+def test_expand_id_overlay_is_rejected_without_committing_cache(tmp_path: Path) -> None:
+ source = _source_path(tmp_path / "source")
+
+ def replace_lineage_id(context: KernelContext) -> KernelResult:
+ return KernelResult(
+ expand={
+ "person": pd.Series(
+ [1], index=pd.Index([4], name="person_id"), dtype="int64"
+ ),
+ "household": pd.Series(
+ [10], index=pd.Index([30], name="household_id"), dtype="int64"
+ ),
+ },
+ columns={
+ ("person", "person_household_id"): pd.Series(
+ [10, 10, 20, 40],
+ index=pd.Index([1, 2, 3, 4], name="person_id"),
+ dtype="int64",
+ ),
+ ("household", "household_id"): pd.Series(
+ [10, 20, 40],
+ index=pd.Index([10, 20, 30], name="household_id"),
+ dtype="int64",
+ ),
+ },
+ weights=Weights(
+ np.array([1.0, 2.0, 1.0], dtype=np.float64), WeightKind.DESIGN
+ ),
+ )
+
+ expand = Node(
+ "replace_lineage_id",
+ "bad.expand@1",
+ structural=StructuralDelta.EXPAND,
+ base="survey",
+ params={
+ "expand_cells": (
+ ("person", "person_household_id", "int64"),
+ ("household", "household_id", "int64"),
+ ),
+ "expand_weight_entity": "household",
+ "expand_weight_kind": "design",
+ },
+ mass="free",
+ )
+ kernel = _Kernel(
+ expand.kernel,
+ Capabilities(Determinism.DETERMINISTIC, structural=StructuralDelta.EXPAND),
+ replace_lineage_id,
+ )
+ registry = _registry(extra=kernel)
+ graph = Graph("toy", (SOURCE,), (CREATE, expand))
+ store = ContentStore(tmp_path / "store")
+
+ for _ in range(2):
+ with pytest.raises(NodeRejected, match="cannot overlay entity id column"):
+ _run(graph, source, store, registry)
+ if kernel.calls == 1:
+ first_store_bytes = _object_bytes(store)
+ else:
+ assert _object_bytes(store) == first_store_bytes
+ assert kernel.calls == 2
+
+
+def test_partitioned_graph_accepts_structural_entrant_partition_values(
+ tmp_path: Path,
+) -> None:
+ source = _source_path(tmp_path / "source")
+
+ def create_partitioned(context: KernelContext) -> KernelResult:
+ original = _source_frame(context.sources["survey"])
+ tables = {entity: original.table(entity).copy() for entity in original.entities}
+ tables["household"]["period"] = np.array([2024, 2025], dtype=np.int64)
+ return KernelResult(
+ frame=Frame(
+ tables,
+ original.schema,
+ {"household": original.weights_for("household")},
+ original.strata,
+ )
+ )
+
+ def admit_household(context: KernelContext) -> KernelResult:
+ return KernelResult(
+ expand={
+ "person": pd.Series(
+ [1], index=pd.Index([4], name="person_id"), dtype="int64"
+ ),
+ "household": pd.Series(
+ [pd.NA],
+ index=pd.Index([30], name="household_id"),
+ dtype="Int64",
+ ),
+ },
+ columns={
+ ("person", "person_household_id"): pd.Series(
+ [10, 10, 20, 30],
+ index=pd.Index([1, 2, 3, 4], name="person_id"),
+ dtype="int64",
+ ),
+ ("household", "size"): pd.Series(
+ [2, 1, 1],
+ index=pd.Index([10, 20, 30], name="household_id"),
+ dtype="int64",
+ ),
+ ("household", "period"): pd.Series(
+ [2024, 2025, 2026],
+ index=pd.Index([10, 20, 30], name="household_id"),
+ dtype="int64",
+ ),
+ },
+ weights=Weights(
+ np.array([1.0, 2.0, 1.0], dtype=np.float64), WeightKind.DESIGN
+ ),
+ )
+
+ def pass_through(column: str) -> Callable[[KernelContext], KernelResult]:
+ def run(context: KernelContext) -> KernelResult:
+ household = context.tables["household"]
+ return KernelResult(
+ columns={
+ ("household", column): pd.Series(
+ household[column].array.copy(),
+ index=pd.Index(household["household_id"], name="household_id"),
+ dtype="int64",
+ )
+ }
+ )
+
+ return run
+
+ create = replace(
+ CREATE,
+ kernel="partition.source@1",
+ outputs=(*CREATE.outputs, Owned("household", "period", "int64")),
+ )
+ expand = Node(
+ "admit_household",
+ "partition.expand@1",
+ structural=StructuralDelta.EXPAND,
+ base="survey",
+ params={
+ "expand_cells": (
+ ("person", "person_household_id", "int64"),
+ ("household", "size", "int64"),
+ ("household", "period", "int64"),
+ ),
+ "expand_weight_entity": "household",
+ "expand_weight_kind": "design",
+ },
+ mass="free",
+ entrants=True,
+ )
+ claim_size = Node(
+ "claim_size",
+ "claim.size@1",
+ outputs=(Owned("household", "size", "int64"),),
+ params={"materialized_expand_outputs": ("household.size",)},
+ population=expand.id,
+ )
+ kernels = (
+ _Kernel(
+ create.kernel,
+ Capabilities(Determinism.DETERMINISTIC, structural=StructuralDelta.CREATE),
+ create_partitioned,
+ ),
+ _Kernel(
+ expand.kernel,
+ Capabilities(Determinism.DETERMINISTIC, structural=StructuralDelta.EXPAND),
+ admit_household,
+ ),
+ _Kernel(
+ claim_size.kernel,
+ Capabilities(Determinism.DETERMINISTIC),
+ pass_through("size"),
+ ),
+ )
+
+ def registry(*extra: _Kernel) -> KernelRegistry:
+ result = _registry()
+ for kernel in (*kernels, *extra):
+ result.register(kernel)
+ return result
+
+ graph = Graph(
+ "toy",
+ (SOURCE,),
+ (create, expand, claim_size),
+ mass_partition=("household", "period"),
+ )
+ store = ContentStore(tmp_path / "store")
+ first_registry = registry()
+ cold = _run(graph, source, store, first_registry)
+ warm = _run(graph, source, store, first_registry)
+
+ for manifest in (cold, warm):
+ assert manifest.population(expand.id).table("household")["period"].tolist() == [
+ 2024,
+ 2025,
+ 2026,
+ ]
+ partition = manifest.nodes[expand.id].receipt["mass"]["partition"] # type: ignore[index]
+ assert (partition["entity"], partition["column"]) == ("household", "period")
+ assert warm.nodes[expand.id].hit
+ assert warm.nodes[claim_size.id].hit
+
+ claim_period = Node(
+ "claim_period",
+ "claim.period@1",
+ outputs=(Owned("household", "period", "int64"),),
+ params={"materialized_expand_outputs": ("household.period",)},
+ population=expand.id,
+ )
+ period_kernel = _Kernel(
+ claim_period.kernel,
+ Capabilities(Determinism.DETERMINISTIC),
+ pass_through("period"),
+ )
+ # Compilation refuses the owner before the executor's own guard can.
+ with pytest.raises(GraphError, match="owns mass partition"):
+ _run(
+ Graph(
+ "toy",
+ (SOURCE,),
+ (create, expand, claim_size, claim_period),
+ mass_partition=("household", "period"),
+ ),
+ source,
+ ContentStore(tmp_path / "ordinary-owner"),
+ registry(period_kernel),
+ )
+
+
def test_create_rejects_undeclared_frame_columns(tmp_path: Path) -> None:
source = _source_path(tmp_path / "source")
@@ -1287,3 +1543,212 @@ def explode(context: KernelContext) -> KernelResult:
ContentStore(tmp_path / "compute"),
compute_registry,
)
+
+
+def test_cache_load_misses_when_stored_capabilities_disagree(
+ tmp_path: Path,
+) -> None:
+ source = _source_path(tmp_path / "source")
+ store = ContentStore(tmp_path / "store")
+ graph = _graph()
+ registry = _registry()
+ manifest = _run(graph, source, store, registry)
+
+ node = graph.node("a")
+ key = manifest.nodes["a"].key
+ record_key = graph_executor._cache_record_key(key)
+ record = store.load_json(record_key)
+ stored_capabilities = record["capabilities"]
+ assert isinstance(stored_capabilities, dict)
+ record["capabilities"] = {
+ **stored_capabilities,
+ "seed_source": "param",
+ }
+ store.put_json(record_key, record, node_key=key, verify_existing=False)
+
+ kernel = registry.get(node.kernel)
+ with pytest.raises(StoreMiss, match="capabilities"):
+ graph_executor._load_record(
+ store,
+ node,
+ key=key,
+ kernel_impl_hash=kernel.implementation_hash(),
+ capabilities=kernel.capabilities,
+ )
+
+
+def test_fit_qrf_seed_source_change_misses_a_shared_store(
+ tmp_path: Path,
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ from microcosm.fit.kernels import QRF_EXECUTOR_KERNEL, QRF_PARAM_KERNEL
+ from microcosm.graph import graph_from_json
+ from tools.graph_parity_fixtures import FIXTURES, ParityCsvSource
+
+ monkeypatch.setenv("POPULACE_FIT_N_JOBS", "1")
+ monkeypatch.setenv("POPULACE_FIT_PREDICT_WORKERS", "1")
+ case = FIXTURES / "fit.qrf"
+ compiled = compile_graph(graph_from_json((case / "graph.json").read_text()))
+ store = ContentStore(tmp_path / "store")
+
+ param_registry = KernelRegistry()
+ param_registry.register(ParityCsvSource())
+ param_registry.register(QRF_PARAM_KERNEL)
+ cold = run_graph(
+ compiled,
+ sources={"fixture": case},
+ store=store,
+ kernels=param_registry,
+ )
+ assert not cold.nodes["fit_qrf"].hit
+
+ assert QRF_PARAM_KERNEL.ref == QRF_EXECUTOR_KERNEL.ref
+ assert (
+ QRF_PARAM_KERNEL.implementation_hash()
+ == QRF_EXECUTOR_KERNEL.implementation_hash()
+ )
+ assert (
+ QRF_PARAM_KERNEL.capabilities.seed_source
+ is not QRF_EXECUTOR_KERNEL.capabilities.seed_source
+ )
+
+ executor_registry = KernelRegistry()
+ executor_registry.register(ParityCsvSource())
+ executor_registry.register(QRF_EXECUTOR_KERNEL)
+ with pytest.raises(NodeRejected, match="EXECUTOR-seeded.*must omit"):
+ run_graph(
+ compiled,
+ sources={"fixture": case},
+ store=store,
+ kernels=executor_registry,
+ )
+
+
+def test_fit_qrf_tolerance_source_hash_pin_is_current() -> None:
+ import json
+
+ from microcosm.fit.kernels import FIT_QRF_TOLERANCE, QRF_PARAM_KERNEL, QRFKernel
+ from tools.graph_parity_fixtures import FIXTURES
+
+ pins = json.loads((FIXTURES / "fit.qrf" / "pins.json").read_text())
+ # The declared budget is provisional (see the constant's comment); the
+ # pin tracks whatever it is, and the identity must not depend on it.
+ assert QRF_PARAM_KERNEL.capabilities.tolerance == FIT_QRF_TOLERANCE
+ assert pins["implementation_hash"] == QRF_PARAM_KERNEL.implementation_hash()
+ changed_tolerance = QRFKernel(QRF_PARAM_KERNEL.capabilities.seed_source)
+ changed_tolerance.capabilities = replace(
+ changed_tolerance.capabilities, tolerance=Tolerance(ulps=2)
+ )
+ assert changed_tolerance.implementation_hash() == pins["implementation_hash"]
+
+
+def test_entrant_materialization_rejects_a_masked_claimant(
+ tmp_path: Path,
+) -> None:
+ source = _source_path(tmp_path / "source")
+
+ def admit_household(context: KernelContext) -> KernelResult:
+ return KernelResult(
+ expand={
+ "person": pd.Series(
+ [1],
+ index=pd.Index([4], name="person_id"),
+ dtype="int64",
+ ),
+ "household": pd.Series(
+ [pd.NA],
+ index=pd.Index([30], name="household_id"),
+ dtype="Int64",
+ ),
+ },
+ columns={
+ ("person", "person_household_id"): pd.Series(
+ [10, 10, 20, 30],
+ index=pd.Index([1, 2, 3, 4], name="person_id"),
+ dtype="int64",
+ ),
+ ("household", "size"): pd.Series(
+ [2, 1, 1],
+ index=pd.Index([10, 20, 30], name="household_id"),
+ dtype="int64",
+ ),
+ ("household", "claim_mask"): pd.Series(
+ [True, True, False],
+ index=pd.Index([10, 20, 30], name="household_id"),
+ dtype="boolean",
+ ),
+ },
+ weights=Weights(
+ np.array([1.0, 2.0, 1.0], dtype=np.float64),
+ WeightKind.DESIGN,
+ ),
+ )
+
+ expand = Node(
+ "admit_household",
+ "masked.expand@1",
+ structural=StructuralDelta.EXPAND,
+ base="survey",
+ params={
+ "expand_cells": (
+ ("person", "person_household_id", "int64"),
+ ("household", "size", "int64"),
+ ("household", "claim_mask", "boolean"),
+ ),
+ "expand_weight_entity": "household",
+ "expand_weight_kind": "design",
+ },
+ mass="free",
+ entrants=True,
+ )
+ claim_mask = Node(
+ "claim_mask",
+ "claim.mask@1",
+ outputs=(Owned("household", "claim_mask", "boolean"),),
+ params={"materialized_expand_outputs": ("household.claim_mask",)},
+ population=expand.id,
+ )
+ claim_size = Node(
+ "claim_size",
+ "claim.masked-size@1",
+ inputs=(Slice("household", ("claim_mask",)),),
+ outputs=(Owned("household", "size", "int64", rows="claim_mask"),),
+ params={"materialized_expand_outputs": ("household.size",)},
+ population=expand.id,
+ )
+
+ def must_not_run(context: KernelContext) -> KernelResult:
+ raise AssertionError(f"claimant {context.node.id} should not run")
+
+ registry = _registry()
+ registry.register(
+ _Kernel(
+ expand.kernel,
+ Capabilities(
+ Determinism.DETERMINISTIC,
+ structural=StructuralDelta.EXPAND,
+ ),
+ admit_household,
+ )
+ )
+ mask_kernel = _Kernel(
+ claim_mask.kernel,
+ Capabilities(Determinism.DETERMINISTIC),
+ must_not_run,
+ )
+ size_kernel = _Kernel(
+ claim_size.kernel,
+ Capabilities(Determinism.DETERMINISTIC),
+ must_not_run,
+ )
+ registry.register(mask_kernel)
+ registry.register(size_kernel)
+
+ with pytest.raises(NodeRejected, match="household.size.*rows='all'"):
+ _run(
+ Graph("toy", (SOURCE,), (CREATE, expand, claim_mask, claim_size)),
+ source,
+ ContentStore(tmp_path / "store"),
+ registry,
+ )
+ assert mask_kernel.calls == size_kernel.calls == 0
diff --git a/packages/microcosm-graph/tests/test_graph_explain.py b/packages/microcosm-graph/tests/test_graph_explain.py
index 62d9b6aa8..30ab061dc 100644
--- a/packages/microcosm-graph/tests/test_graph_explain.py
+++ b/packages/microcosm-graph/tests/test_graph_explain.py
@@ -158,6 +158,36 @@ def test_explain_html_is_the_public_export() -> None:
assert graph_api.explain_html is explain_html
+def test_entrant_person_strata_survive_cache_and_are_explained(tmp_path: Path) -> None:
+ expand, claim = toy.entrant_person_node()
+ graph = toy.small_graph(nodes=(toy.CREATE, expand, claim))
+ cold = toy.run_toy(graph, tmp_path / "cold")
+ warm = toy.run_toy(
+ graph,
+ tmp_path / "warm",
+ sources=cold.sources,
+ registry=cold.registry,
+ store=cold.store,
+ )
+ entrant_id = int(cold.manifest.population("survey").person["person_id"].max()) + 1
+
+ assert warm.manifest.nodes[expand.id].hit
+ assert warm.manifest.population(expand.id).strata.equals(
+ cold.manifest.population(expand.id).strata
+ )
+ assert cold.manifest.nodes[expand.id].receipt["entrant_strata"] == (
+ (entrant_id, "urban"),
+ )
+ assert (
+ warm.manifest.nodes[expand.id].receipt["entrant_strata"]
+ == cold.manifest.nodes[expand.id].receipt["entrant_strata"]
+ )
+ detail = describe(cold.compiled, expand.id, cold.manifest)
+ rendered = explain_html(cold.compiled, cold.manifest)
+ assert "entrant_strata" in detail and "urban" in detail
+ assert "entrant_strata" in rendered and "urban" in rendered
+
+
def test_page_contains_every_node_and_its_click_detail(explanation) -> None:
run, _charter, rendered = explanation
for node_id in run.compiled.order:
@@ -172,11 +202,13 @@ def test_page_contains_every_node_and_its_click_detail(explanation) -> None:
def test_page_contains_every_charter_property(explanation) -> None:
_run, charter, rendered = explanation
identifiers = re.findall(r"^\|\s*([A-Z]\d+)\s*\|", charter, re.MULTILINE)
- assert len(dict.fromkeys(identifiers)) == 41
+ assert (
+ len(dict.fromkeys(identifiers)) == 45
+ ) # 41 + B6, C5, D6, B7 (amendments 11-14)
for identifier in identifiers:
assert f"{identifier}" in rendered
assert "35 green" not in rendered # V1-V4 are also represented.
- assert "41 green" in rendered
+ assert "45 green" in rendered
assert "0 red" in rendered
assert "Flip PR | " in rendered
assert "Not recorded" in rendered
diff --git a/packages/microcosm-graph/tests/test_graph_kernel_contract.py b/packages/microcosm-graph/tests/test_graph_kernel_contract.py
new file mode 100644
index 000000000..fd447e57d
--- /dev/null
+++ b/packages/microcosm-graph/tests/test_graph_kernel_contract.py
@@ -0,0 +1,157 @@
+"""Kernel-protocol contracts of the frozen interface (amendment 13).
+
+A kernel that claims bounded numeric movement declares the bound; a bitwise
+kernel declares none; the context hands readers their inputs' declared
+tolerances; and the two new declaration fields round-trip through JSON.
+"""
+
+from __future__ import annotations
+
+import numpy as np
+import pandas as pd
+import pytest
+
+from microcosm.graph import (
+ Capabilities,
+ Determinism,
+ Graph,
+ KernelBase,
+ KernelContext,
+ KernelRegistry,
+ Node,
+ Numeric,
+ Owned,
+ SourceRef,
+ StructuralDelta,
+ Tolerance,
+ graph_from_json,
+ graph_to_json,
+)
+
+
+def test_tolerance_must_allow_some_movement_and_stay_finite() -> None:
+ assert Tolerance(rtol=1e-9).ulps == 0
+ assert Tolerance(ulps=2) == Tolerance(0.0, 0.0, 2)
+ with pytest.raises(ValueError, match="allow some movement"):
+ Tolerance()
+ with pytest.raises(ValueError, match="non-negative and finite"):
+ Tolerance(rtol=-1e-9)
+ with pytest.raises(ValueError, match="non-negative and finite"):
+ Tolerance(atol=float("inf"))
+ with pytest.raises(ValueError, match="non-negative and finite"):
+ Tolerance(rtol=float("nan"))
+ with pytest.raises(ValueError, match="must be an integer"):
+ Tolerance(ulps=1.5) # type: ignore[arg-type]
+
+
+def test_tolerance_bound_kernels_declare_a_bound_and_bitwise_kernels_none() -> None:
+ bounded = Capabilities(
+ determinism=Determinism.SEEDED,
+ numeric=Numeric.TOLERANCE_BOUND,
+ tolerance=Tolerance(rtol=1e-6),
+ )
+ assert bounded.tolerance == Tolerance(rtol=1e-6)
+ with pytest.raises(ValueError, match="must declare its Tolerance"):
+ Capabilities(determinism=Determinism.SEEDED, numeric=Numeric.TOLERANCE_BOUND)
+ with pytest.raises(ValueError, match="bitwise kernel declares no Tolerance"):
+ Capabilities(determinism=Determinism.DETERMINISTIC, tolerance=Tolerance(ulps=1))
+ with pytest.raises(TypeError, match="must be a Tolerance or None"):
+ Capabilities(
+ determinism=Determinism.DETERMINISTIC,
+ numeric=Numeric.TOLERANCE_BOUND,
+ tolerance=1e-6, # type: ignore[arg-type]
+ )
+
+
+def test_context_carries_declared_tolerances_and_defaults_to_none() -> None:
+ node = Node("gate", "gate.check@1")
+ context = KernelContext(
+ node=node,
+ tables={},
+ weights={},
+ strata=pd.Series([], dtype=object, name="stratum"),
+ params={},
+ rng=np.random.default_rng(0),
+ )
+ assert dict(context.tolerances) == {}
+ carried = KernelContext(
+ node=node,
+ tables={},
+ weights={},
+ strata=pd.Series([], dtype=object, name="stratum"),
+ params={},
+ rng=np.random.default_rng(0),
+ tolerances={
+ ("person", "income"): Tolerance(rtol=1e-6),
+ ("person", "age"): None,
+ },
+ )
+ assert carried.tolerances[("person", "income")] == Tolerance(rtol=1e-6)
+ assert carried.tolerances[("person", "age")] is None
+
+
+def test_entrants_and_mass_partition_round_trip_through_canonical_json() -> None:
+ source = SourceRef("survey", "frame-h5")
+ create = Node(
+ "survey",
+ "source.frame@1",
+ sources=("survey",),
+ structural=StructuralDelta.CREATE,
+ outputs=(Owned("person", "age", "int64"), Owned("person", "period", "int64")),
+ )
+ cohort = Node(
+ "cohort",
+ "enter.immigrants@1",
+ base="survey",
+ structural=StructuralDelta.EXPAND,
+ mass="declared",
+ entrants=True,
+ )
+ graph = Graph(
+ "toy", (source,), (create, cohort), mass_partition=("person", "period")
+ )
+ text = graph_to_json(graph)
+ assert '"entrants":true' in text and '"mass_partition":["person","period"]' in text
+ assert graph_from_json(text) == graph
+ # A declaration without either field serializes exactly as it did before
+ # amendments 11 and 12, so every pinned graph JSON still matches.
+ plain = Graph("toy", (source,), (create,))
+ plain_text = graph_to_json(plain)
+ assert "entrants" not in plain_text and "mass_partition" not in plain_text
+ assert graph_from_json(plain_text) == plain
+
+
+def test_capabilities_reject_look_alike_fields_and_registration_needs_the_real_thing() -> (
+ None
+):
+ """A string spelling an enum member is not the member (review of #851, finding 5)."""
+ with pytest.raises(TypeError, match="Capabilities.numeric must be a Numeric"):
+ Capabilities(determinism=Determinism.DETERMINISTIC, numeric="tolerance_bound") # type: ignore[arg-type]
+ with pytest.raises(
+ TypeError, match="Capabilities.determinism must be a Determinism"
+ ):
+ Capabilities(determinism="deterministic") # type: ignore[arg-type]
+ with pytest.raises(TypeError, match="consumes_se must be a boolean"):
+ Capabilities(determinism=Determinism.DETERMINISTIC, consumes_se=1) # type: ignore[arg-type]
+ with pytest.raises(TypeError, match="dependencies must be a tuple"):
+ Capabilities(determinism=Determinism.DETERMINISTIC, dependencies=["numpy"]) # type: ignore[arg-type]
+
+ class LookAlike:
+ determinism = Determinism.DETERMINISTIC
+ numeric = "tolerance_bound"
+ seed_source = "none"
+ structural = "none"
+ role = "compute"
+ consumes_se = False
+ dependencies = ()
+ tolerance = None
+
+ class Impostor(KernelBase):
+ ref = "impostor@1"
+ capabilities = LookAlike() # type: ignore[assignment]
+
+ def run(self, context): # pragma: no cover - never reached
+ raise AssertionError
+
+ with pytest.raises(TypeError, match="must carry a Capabilities instance"):
+ KernelRegistry().register(Impostor())
diff --git a/packages/microcosm-graph/tests/test_graph_keys.py b/packages/microcosm-graph/tests/test_graph_keys.py
index 4b1acdd25..b5342394d 100644
--- a/packages/microcosm-graph/tests/test_graph_keys.py
+++ b/packages/microcosm-graph/tests/test_graph_keys.py
@@ -17,6 +17,14 @@
StructuralDelta,
compile_graph,
)
+from microcosm.graph.kernel import (
+ Capabilities,
+ Determinism,
+ KernelRole,
+ Numeric,
+ SeedSource,
+ Tolerance,
+)
from microcosm.graph.keys import (
artifact_key,
frame_key,
@@ -39,6 +47,12 @@
)
+def _capabilities(
+ structural: StructuralDelta = StructuralDelta.NONE,
+) -> Capabilities:
+ return Capabilities(Determinism.DETERMINISTIC, structural=structural)
+
+
def _ordinary(
node_id: str,
inputs: tuple[str, ...],
@@ -87,6 +101,7 @@ def _all_keys(
keys,
implementation_hashes[node.kernel],
{"survey": source_key},
+ kernel_capabilities=_capabilities(node.structural),
)
return compiled, keys
@@ -199,13 +214,21 @@ def test_carried_columns_resolve_to_the_structural_version() -> None:
graph = Graph("toy", (SOURCE,), (CREATE, subset, model))
compiled = compile_graph(graph)
keys = {"survey": "a" * 64, "adults": "b" * 64}
- baseline = node_key(compiled, "model", keys, "c" * 64, {})
+ baseline = node_key(
+ compiled,
+ "model",
+ keys,
+ "c" * 64,
+ {},
+ kernel_capabilities=_capabilities(),
+ )
changed_unreachable_base = node_key(
compiled,
"model",
{"survey": "d" * 64, "adults": "b" * 64},
"c" * 64,
{},
+ kernel_capabilities=_capabilities(),
)
assert baseline == changed_unreachable_base
@@ -227,6 +250,7 @@ def test_structural_key_binds_every_patch_in_its_base_version() -> None:
{"survey": "a" * 64, "patched": "b" * 64},
"c" * 64,
{},
+ kernel_capabilities=_capabilities(StructuralDelta.FILTER),
)
changed_patch = node_key(
compiled,
@@ -234,6 +258,7 @@ def test_structural_key_binds_every_patch_in_its_base_version() -> None:
{"survey": "a" * 64, "patched": "d" * 64},
"c" * 64,
{},
+ kernel_capabilities=_capabilities(StructuralDelta.FILTER),
)
assert baseline != changed_patch
@@ -248,6 +273,7 @@ def test_non_create_source_consumers_bind_their_declared_source_bytes() -> None:
{"survey": "a" * 64},
"b" * 64,
{"survey": "c" * 64},
+ kernel_capabilities=_capabilities(),
)
changed = node_key(
compiled,
@@ -255,5 +281,46 @@ def test_non_create_source_consumers_bind_their_declared_source_bytes() -> None:
{"survey": "a" * 64},
"b" * 64,
{"survey": "d" * 64},
+ kernel_capabilities=_capabilities(),
)
assert baseline != changed
+
+
+def test_every_capability_field_changes_the_node_key() -> None:
+ compiled = compile_graph(_graph())
+ base = Capabilities(
+ determinism=Determinism.SEEDED,
+ numeric=Numeric.TOLERANCE_BOUND,
+ seed_source=SeedSource.EXECUTOR,
+ role=KernelRole.COMPUTE,
+ consumes_se=False,
+ dependencies=("numpy",),
+ tolerance=Tolerance(rtol=1e-6, atol=2e-6, ulps=1),
+ )
+
+ def key(capabilities: Capabilities) -> str:
+ return node_key(
+ compiled,
+ "a",
+ {"survey": "a" * 64},
+ "b" * 64,
+ {},
+ kernel_capabilities=capabilities,
+ )
+
+ baseline = key(base)
+ variants = (
+ replace(base, determinism=Determinism.DETERMINISTIC),
+ replace(base, numeric=Numeric.BITWISE, tolerance=None),
+ replace(base, seed_source=SeedSource.PARAM),
+ replace(base, structural=StructuralDelta.FILTER),
+ replace(base, role=KernelRole.GATE),
+ replace(base, consumes_se=True),
+ replace(base, dependencies=("numpy", "pandas")),
+ replace(base, tolerance=Tolerance(rtol=3e-6, atol=2e-6, ulps=1)),
+ )
+ assert all(key(capabilities) != baseline for capabilities in variants)
+
+ positive_zero = replace(base, tolerance=Tolerance(rtol=0.0, atol=2e-6, ulps=1))
+ negative_zero = replace(base, tolerance=Tolerance(rtol=-0.0, atol=2e-6, ulps=1))
+ assert key(positive_zero) == key(negative_zero)
diff --git a/packages/microcosm-graph/tests/test_graph_manifest.py b/packages/microcosm-graph/tests/test_graph_manifest.py
index 8d6a50621..e90570d14 100644
--- a/packages/microcosm-graph/tests/test_graph_manifest.py
+++ b/packages/microcosm-graph/tests/test_graph_manifest.py
@@ -6,13 +6,15 @@
from dataclasses import FrozenInstanceError, replace
from pathlib import Path
+import numpy as np
import pandas as pd
import pytest
import microcosm.graph as graph_api
+from microcosm.frame import EntitySchema, Frame, WeightKind, Weights
from microcosm.graph.decl import StructuralDelta
from microcosm.graph.kernel import Capabilities, Determinism, KernelRole, SeedSource
-from microcosm.graph.manifest import Decision, NodeReceipt, RunManifest
+from microcosm.graph.manifest import Decision, NodeReceipt, PopulationView, RunManifest
from microcosm.graph.population import MassRecord
@@ -26,6 +28,32 @@ def _capabilities(role: KernelRole = KernelRole.COMPUTE) -> Capabilities:
)
+def _frame() -> Frame:
+ person = pd.DataFrame(
+ {
+ "person_id": np.asarray([1, 2], dtype=np.int64),
+ "person_household_id": np.asarray([10, 20], dtype=np.int64),
+ }
+ )
+ household = pd.DataFrame(
+ {
+ "household_id": np.asarray([10, 20], dtype=np.int64),
+ "size": np.asarray([1, 1], dtype=np.int64),
+ }
+ )
+ return Frame(
+ {"person": person, "household": household},
+ EntitySchema(group_entities=("household",)),
+ {
+ "household": Weights(
+ np.asarray([1.0, 2.0], dtype=np.float64),
+ WeightKind.DESIGN,
+ )
+ },
+ pd.Series(["a", "b"], name="stratum"),
+ )
+
+
def _receipt(key: str, *, hit: bool = False, wall_time: float = 0.2) -> NodeReceipt:
return NodeReceipt(
key=key,
@@ -88,8 +116,8 @@ def _persisted_manifest(
return RunManifest("toy", {"release": release, "gate": gate})
-def test_manifest_json_round_trip_and_convenient_lookup() -> None:
- population = object()
+def test_manifest_json_round_trip_and_population_view() -> None:
+ raw = _frame()
manifest = RunManifest(
country="toy",
nodes={"b": _receipt("b" * 64), "a": _receipt("a" * 64)},
@@ -97,7 +125,7 @@ def test_manifest_json_round_trip_and_convenient_lookup() -> None:
started_at="2026-09-01T12:00:00Z",
finished_at="2026-09-01T12:00:01Z",
host="runner-1",
- populations={"survey": population}, # type: ignore[dict-item]
+ populations={"survey": raw, "filtered": raw},
)
restored = RunManifest.from_json(manifest.to_json())
assert restored == manifest
@@ -105,9 +133,30 @@ def test_manifest_json_round_trip_and_convenient_lookup() -> None:
assert manifest.nodes["a"] is manifest.node("a")
assert manifest.receipts["a"] is manifest.receipt("a")
assert manifest["a"].artifacts[("person", "x")] == "d" * 64
- assert manifest.population("survey") is population
+
+ survey = manifest.population("survey")
+ filtered = manifest.population("filtered")
+ assert type(survey) is type(filtered) is PopulationView
+ assert isinstance(survey, Frame)
+ assert manifest.population("survey") is survey
+ assert type(raw) is Frame
+ assert not hasattr(raw, "household")
+ assert survey.person is raw.person
+ assert survey.household is raw.table("household")
+ assert survey.table("household") is raw.table("household")
+ assert survey.weights_for("household") is raw.weights_for("household")
+ assert survey.strata is raw.strata
+ with pytest.raises(AttributeError, match="PopulationView.*missing"):
+ _ = survey.missing
+
with pytest.raises(KeyError, match="not attached"):
restored.population("survey")
+ with pytest.raises(TypeError, match="values must be Frame"):
+ RunManifest(
+ "toy",
+ {"a": _receipt("a" * 64)},
+ populations={"survey": object()}, # type: ignore[dict-item]
+ )
def test_manifest_key_excludes_every_operational_field() -> None:
@@ -397,6 +446,7 @@ def test_load_requires_every_manifest_artifact(
def test_package_exports_runtime_implementations_and_failures() -> None:
assert graph_api.ContentStore.__module__.endswith(".store")
assert graph_api.RunManifest is RunManifest
+ assert graph_api.PopulationView is PopulationView
assert graph_api.NodeReceipt is NodeReceipt
assert graph_api.Decision is Decision
assert graph_api.run_graph.__module__.endswith(".executor")
diff --git a/packages/microcosm-graph/tests/test_graph_population.py b/packages/microcosm-graph/tests/test_graph_population.py
index 13bb682c9..6c3c7fca1 100644
--- a/packages/microcosm-graph/tests/test_graph_population.py
+++ b/packages/microcosm-graph/tests/test_graph_population.py
@@ -21,6 +21,7 @@
PopulationError,
dtype_for_token,
dtype_matches,
+ entrant_strata_receipt,
expand_lineage_receipt,
owned_ids,
patch,
@@ -393,6 +394,35 @@ def test_declared_mass_validates_the_kernel_receipt() -> None:
patch(population, node, result)
+def test_mass_receipt_rejects_partition_when_graph_has_none() -> None:
+ population = _population()
+ node = Node(
+ "importance",
+ "test@1",
+ structural=StructuralDelta.REWEIGHT,
+ base="source",
+ weights=WeightTransition("household", "importance", mass="declared"),
+ mass="declared",
+ )
+ receipt = _mass_receipt(
+ policy="declared",
+ before=7.0,
+ after=14.0,
+ stratum_before={"a": 2.0, "b": 5.0},
+ stratum_after={"a": 4.0, "b": 10.0},
+ )
+ mass = receipt["mass"]
+ assert isinstance(mass, dict)
+ mass["partition"] = {}
+ result = KernelResult(
+ weights=Weights(np.array([2.0, 4.0, 6.0]), WeightKind.IMPORTANCE),
+ receipt=receipt,
+ )
+
+ with pytest.raises(PopulationError, match="declares no mass partition"):
+ patch(population, node, result)
+
+
def test_filter_requires_subset_ids_and_records_free_mass() -> None:
population = _population()
filtered = population.frame.select(
@@ -475,6 +505,84 @@ def _lineage_expand_result(*, bad_source: bool = False) -> KernelResult:
)
+def _entrant_person_expand_node(*, membership_dtype: str = "int64") -> Node:
+ return Node(
+ "entrant_person",
+ "test@1",
+ structural=StructuralDelta.EXPAND,
+ base="source",
+ params={
+ "expand_cells": (
+ ("person", "person_household_id", membership_dtype),
+ ("person", "keep", "bool"),
+ ("person", "owned", "boolean"),
+ ("person", "nullable", "boolean"),
+ ("person", "amount", "float64"),
+ ),
+ "expand_weight_entity": "household",
+ "expand_weight_kind": "design",
+ },
+ mass="free",
+ entrants=True,
+ )
+
+
+def _entrant_person_expand_result(
+ strata: object, *, frame: Frame | None = None
+) -> KernelResult:
+ frame = _frame() if frame is None else frame
+ person = frame.table("person")
+ person_id_dtype = person["person_id"].dtype
+ household_id_dtype = frame.table("household")["household_id"].dtype
+ entrant_id = int(person["person_id"].max()) + 1
+ ids = pd.Index(
+ pd.Series([*person["person_id"], entrant_id], dtype=person_id_dtype).array,
+ name="person_id",
+ )
+ additions = {
+ "person_household_id": 10,
+ "keep": True,
+ "owned": False,
+ "nullable": pd.NA,
+ "amount": 3.0,
+ }
+ tokens = {
+ "person_household_id": token_for_dtype(person["person_household_id"].dtype),
+ "keep": "bool",
+ "owned": "boolean",
+ "nullable": "boolean",
+ "amount": "float64",
+ }
+ columns = {
+ ("person", column): pd.Series(
+ pd.array([*person[column], value], dtype=tokens[column]), index=ids
+ )
+ for column, value in additions.items()
+ }
+ return KernelResult(
+ expand={
+ "person": pd.Series(
+ pd.array(
+ [pd.NA],
+ dtype=f"Int{np.dtype(person_id_dtype).itemsize * 8}",
+ ),
+ index=pd.Index(
+ pd.Series([entrant_id], dtype=person_id_dtype).array,
+ name="person_id",
+ ),
+ ),
+ "household": pd.Series(
+ [],
+ index=pd.Index([], dtype=household_id_dtype, name="household_id"),
+ dtype=household_id_dtype,
+ ),
+ },
+ columns=columns,
+ weights=frame.weights_for("household"),
+ strata=strata, # type: ignore[arg-type]
+ )
+
+
def test_expand_lineage_carries_rows_remaps_memberships_and_restores_cache() -> None:
population = _population()
node = _lineage_expand_node()
@@ -507,6 +615,265 @@ def test_expand_lineage_carries_rows_remaps_memberships_and_restores_cache() ->
assert cached.mass_ledger == expanded.mass_ledger
+def test_cached_expand_requires_exact_lineage_id_sequence() -> None:
+ population = _population()
+ node = Node(
+ "cached_midpoint",
+ "test@1",
+ structural=StructuralDelta.EXPAND,
+ base="source",
+ params={
+ "expand_cells": (),
+ "expand_weight_entity": "household",
+ "expand_weight_kind": "design",
+ },
+ mass="free",
+ )
+ before = population.frame
+ person = before.table("person")
+ household = before.table("household")
+ added_person = person.iloc[[0]].copy()
+ added_person["person_id"] = np.array([5], dtype=np.int64)
+ added_person["person_household_id"] = np.array([15], dtype=np.int64)
+ final_person = pd.concat([person, added_person], ignore_index=True)
+ added_household = household.iloc[[0]].copy()
+ added_household["household_id"] = np.array([15], dtype=np.int64)
+ final_household = (
+ pd.concat([household, added_household], ignore_index=True)
+ .sort_values("household_id")
+ .reset_index(drop=True)
+ )
+ final_weights = Weights(
+ np.array([1.0, 1.0, 2.0, 3.0], dtype=np.float64), WeightKind.DESIGN
+ )
+ cached_frame = Frame(
+ {"person": final_person, "household": final_household},
+ before.schema,
+ {"household": final_weights},
+ pd.concat([before.strata, before.strata.iloc[[0]]], ignore_index=True),
+ )
+ lineage = {
+ "person": pd.Series([1], index=pd.Index([5], name="person_id"), dtype="int64"),
+ "household": pd.Series(
+ [10], index=pd.Index([15], name="household_id"), dtype="int64"
+ ),
+ }
+
+ with pytest.raises(PopulationError, match="final 'household' ids"):
+ restore_cached_expand(
+ population,
+ node,
+ KernelResult(
+ frame=cached_frame,
+ weights=final_weights,
+ receipt={"expand": expand_lineage_receipt(lineage)},
+ ),
+ )
+
+
+def test_entrant_person_strata_materialize_and_attest_cached_replay() -> None:
+ population = _population()
+ node = _entrant_person_expand_node()
+ result = _entrant_person_expand_result(
+ pd.Series(
+ ["new"],
+ index=pd.Index([5], dtype="int64", name="ignored"),
+ dtype=object,
+ name="ignored",
+ )
+ )
+
+ expanded = patch(population, node, result)
+
+ assert expanded.frame.table("person")["person_household_id"].tolist()[-1] == 10
+ assert expanded.frame.strata.tolist() == ["a", "a", "b", "b", "new"]
+ assert expanded.mass_ledger[-1].before_total == 7.0
+ assert expanded.mass_ledger[-1].after_total == 8.0
+ assert result.expand is not None
+ entrant_receipt = entrant_strata_receipt(
+ population.frame, node, result.expand, result.strata
+ )
+ receipt = {
+ "expand": expand_lineage_receipt(result.expand),
+ "entrant_strata": entrant_receipt,
+ }
+ cached = restore_cached_expand(
+ population,
+ node,
+ KernelResult(
+ frame=expanded.frame,
+ weights=result.weights,
+ receipt=receipt,
+ ),
+ )
+ pd.testing.assert_series_equal(cached.frame.strata, expanded.frame.strata)
+ assert cached.mass_ledger == expanded.mass_ledger
+
+ with pytest.raises(PopulationError, match="entrant-strata receipt"):
+ restore_cached_expand(
+ population,
+ node,
+ KernelResult(
+ frame=expanded.frame,
+ weights=result.weights,
+ receipt={"expand": expand_lineage_receipt(result.expand)},
+ ),
+ )
+
+
+def test_cached_entrant_strata_rehydrate_the_base_id_dtype() -> None:
+ source = _frame()
+ person = source.table("person").copy()
+ household = source.table("household").copy()
+ for column in ("person_id", "person_household_id"):
+ person[column] = person[column].astype("int32")
+ household["household_id"] = household["household_id"].astype("int32")
+ frame = Frame(
+ {"person": person, "household": household},
+ source.schema,
+ {"household": source.weights_for("household")},
+ source.strata.copy(),
+ )
+ population = Population.from_frame(frame, "source")
+ node = _entrant_person_expand_node(membership_dtype="int32")
+ result = _entrant_person_expand_result(
+ pd.Series(["new"], index=pd.Index([5], dtype="int32"), dtype=object),
+ frame=frame,
+ )
+
+ expanded = patch(population, node, result)
+ assert result.expand is not None
+ receipt = {
+ "expand": expand_lineage_receipt(result.expand),
+ "entrant_strata": entrant_strata_receipt(
+ frame, node, result.expand, result.strata
+ ),
+ }
+ cached = restore_cached_expand(
+ population,
+ node,
+ KernelResult(
+ frame=expanded.frame,
+ weights=result.weights,
+ receipt=receipt,
+ ),
+ )
+
+ assert cached.frame.table("person")["person_id"].dtype == np.dtype("int32")
+ pd.testing.assert_series_equal(cached.frame.strata, expanded.frame.strata)
+
+
+@pytest.mark.parametrize(
+ ("receipt_label", "changed_label"),
+ [(1, True), (1, 1.0), (-0.0, 0.0)],
+ ids=["bool", "float", "signed-zero"],
+)
+def test_cached_entrant_strata_preserve_label_scalar(
+ receipt_label: object, changed_label: object
+) -> None:
+ population = _population()
+ node = _entrant_person_expand_node()
+ result = _entrant_person_expand_result(
+ pd.Series([receipt_label], index=pd.Index([5], dtype="int64"), dtype=object)
+ )
+ expanded = patch(population, node, result)
+ changed_strata = expanded.frame.strata.copy()
+ changed_strata.iloc[-1] = changed_label
+ changed_frame = _replace_person_table(
+ expanded.frame,
+ expanded.frame.table("person").copy(),
+ changed_strata,
+ )
+ assert result.expand is not None
+ receipt = {
+ "expand": expand_lineage_receipt(result.expand),
+ "entrant_strata": entrant_strata_receipt(
+ population.frame, node, result.expand, result.strata
+ ),
+ }
+
+ with pytest.raises(PopulationError, match="label"):
+ restore_cached_expand(
+ population,
+ node,
+ KernelResult(
+ frame=changed_frame,
+ weights=result.weights,
+ receipt=receipt,
+ ),
+ )
+
+
+def test_cached_entrant_strata_encode_bytes_labels() -> None:
+ population = _population()
+ node = _entrant_person_expand_node()
+ result = _entrant_person_expand_result(
+ pd.Series([b"new\x00stratum"], index=pd.Index([5], dtype="int64"), dtype=object)
+ )
+ expanded = patch(population, node, result)
+ assert result.expand is not None
+ receipt = {
+ "expand": expand_lineage_receipt(result.expand),
+ "entrant_strata": entrant_strata_receipt(
+ population.frame, node, result.expand, result.strata
+ ),
+ }
+
+ assert receipt["entrant_strata"] == [[5, {"bytes_hex": "6e6577007374726174756d"}]]
+ cached = restore_cached_expand(
+ population,
+ node,
+ KernelResult(
+ frame=expanded.frame,
+ weights=result.weights,
+ receipt=receipt,
+ ),
+ )
+ pd.testing.assert_series_equal(cached.frame.strata, expanded.frame.strata)
+
+
+@pytest.mark.parametrize(
+ "strata",
+ [
+ None,
+ pd.Series(["new"], index=pd.Index([6], dtype="int64"), dtype=object),
+ pd.Series(["old", "new"], index=pd.Index([1, 5], dtype="int64"), dtype=object),
+ pd.Series(["new"], index=pd.Index([5], dtype="int32"), dtype=object),
+ pd.Series([pd.NA], index=pd.Index([5], dtype="int64"), dtype=object),
+ pd.Series([1], index=pd.Index([5], dtype="int64"), dtype="int64"),
+ ],
+ ids=[
+ "missing",
+ "unknown-id",
+ "incumbent-id",
+ "wrong-id-dtype",
+ "missing-label",
+ "wrong-label-dtype",
+ ],
+)
+def test_entrant_person_strata_reject_malformed_exact_set(strata: object) -> None:
+ with pytest.raises(PopulationError, match="strata"):
+ patch(
+ _population(),
+ _entrant_person_expand_node(),
+ _entrant_person_expand_result(strata),
+ )
+
+
+def test_strata_are_rejected_without_entrant_persons() -> None:
+ result = _lineage_expand_result()
+ with pytest.raises(PopulationError, match="without entrant persons"):
+ patch(
+ _population(),
+ _lineage_expand_node(),
+ KernelResult(
+ expand=result.expand,
+ weights=result.weights,
+ strata=pd.Series([], dtype=object),
+ ),
+ )
+
+
def test_expand_lineage_rejects_an_unknown_source_id() -> None:
with pytest.raises(PopulationError, match="unknown 'person' source ids"):
patch(
diff --git a/tools/graph_acceptance_burndown.py b/tools/graph_acceptance_burndown.py
index 923b8ffc4..96225d16a 100644
--- a/tools/graph_acceptance_burndown.py
+++ b/tools/graph_acceptance_burndown.py
@@ -12,18 +12,22 @@
marker on a helper would all fool a text search, and the number this prints is
the number the charter is scored on.
-``--verify`` compares against ``origin/main``, file by file, and exits 1
-if any file's count rose. A file that does not exist on the baseline is
-reported as new and constrains nothing; a file that does is a ratchet. It also
-refuses a marker that is not ``strict=True`` (a non-strict marker hides an
-``xpass``, so a property could go green without anybody noticing), a marker
-whose reason names no charter id, and a charter id with no test at all.
+``--verify`` compares against the baseline branch by property identity and
+exits 1 if any property that was green there is red now, whichever file the
+marker sits in and whatever else went green (counts can offset; identities
+cannot). A property the charter gained since the baseline may start red. It
+also refuses a marker that is not ``strict=True`` (a non-strict marker hides
+an ``xpass``, so a property could go green without anybody noticing), a marker
+whose reason names no charter id, an id the charter does not list, or an id
+other than the one in its own test name, two markers on one property, and a
+charter id with no test at all.
"""
from __future__ import annotations
import argparse
import ast
+import fnmatch
import json
import re
import subprocess
@@ -156,6 +160,37 @@ def counts(files: tuple[str, ...]) -> dict[str, tuple[Marker, ...]]:
}
+def baseline_suite_files(ref: str) -> tuple[str, ...]:
+ """The acceptance files as of ``ref``, including ones since deleted."""
+ # ``git ls-tree`` takes literal paths, not globs: list the directory and
+ # match the file pattern here.
+ result = subprocess.run(
+ [
+ "git",
+ "ls-tree",
+ "-r",
+ "--name-only",
+ ref,
+ "--",
+ str(Path(SUITE_GLOB).parent),
+ ],
+ cwd=ROOT,
+ text=True,
+ stdout=subprocess.PIPE,
+ stderr=subprocess.DEVNULL,
+ )
+ if result.returncode != 0:
+ return ()
+ pattern = Path(SUITE_GLOB).name
+ return tuple(
+ sorted(
+ line
+ for line in result.stdout.splitlines()
+ if line and fnmatch.fnmatch(Path(line).name, pattern)
+ )
+ )
+
+
def baseline_source(ref: str, file: str) -> str | None:
"""``file`` as of ``ref``, or ``None`` when it did not exist there."""
result = subprocess.run(
@@ -291,23 +326,86 @@ def verify(ref: str = BASELINE_REF) -> int:
if entry["state"] == "missing":
problems.append(f"charter {entry['id']} has no test in the suite")
+ declared = set(charter_ids((ROOT / CHARTER).read_text()))
+ seen: dict[str, str] = {}
+ for file in sorted(current):
+ for marker in current[file]:
+ if not marker.charter_id:
+ continue
+ if marker.charter_id not in declared:
+ problems.append(
+ f"{file}::{marker.test} names {marker.charter_id}, which "
+ f"{CHARTER} does not list"
+ )
+ # The reason is free text; the test name is the binding. A marker
+ # whose reason names one property while sitting on another's test
+ # would let a re-red hide behind a known red, so the two must agree
+ # and each property may carry one marker.
+ named = TEST_ID.match(marker.test)
+ if named is None or named.group(1).upper() != marker.charter_id:
+ problems.append(
+ f"{file}::{marker.test} claims charter {marker.charter_id} "
+ "but is not that property's test"
+ )
+ if marker.charter_id in seen:
+ problems.append(
+ f"{marker.charter_id} carries two markers: {seen[marker.charter_id]} "
+ f"and {file}::{marker.test}"
+ )
+ seen.setdefault(marker.charter_id, f"{file}::{marker.test}")
+
if not fetch_baseline(ref):
print(f"baseline={ref} unavailable; the ratchet did not run")
else:
print(f"baseline={ref}")
+ # The ratchet is on property identities, not counts: a property that
+ # is green on the baseline and red now is a re-red, whatever else
+ # went green, whichever file the marker sits in. A property the
+ # charter gained since the baseline is committed red first (the
+ # charter's meta-TDD rule), so its marker is not a re-red.
+ baseline_charter = baseline_source(ref, CHARTER)
+ known = set(charter_ids(baseline_charter)) if baseline_charter else set()
+ new_to_charter = declared - known
+ was_red: set[str] = set()
+ for file in baseline_suite_files(ref):
+ source = baseline_source(ref, file)
+ if source is not None:
+ was_red |= {m.charter_id for m in markers_in(source, file)}
for file in sorted(current):
source = baseline_source(ref, file)
+ ids = {m.charter_id for m in current[file] if m.charter_id}
+ new_reds = sorted(ids & new_to_charter)
+ suffix = (
+ f" (+{len(new_reds)} new: {', '.join(new_reds)})" if new_reds else ""
+ )
if source is None:
- print(f" [new] {file}: {len(current[file])}")
+ print(f" [new] {file}: {len(current[file])}{suffix}")
continue
was = len(markers_in(source, file))
- now = len(current[file])
- print(f" {'rose' if now > was else 'ok':<6} {file}: {was} -> {now}")
- if now > was:
- problems.append(
- f"{file} re-reds {now - was} propert"
- f"{'y' if now - was == 1 else 'ies'} ({was} -> {now})"
+ now = len(current[file]) - len(new_reds)
+ print(
+ f" {'rose' if now > was else 'ok':<6} {file}: {was} -> {now}{suffix}"
+ )
+ now_red = {m.charter_id for file in current for m in current[file]}
+ re_reds = sorted((now_red & declared) - was_red - new_to_charter)
+ if re_reds:
+ where = {
+ identifier: sorted(
+ file
+ for file in current
+ if any(m.charter_id == identifier for m in current[file])
)
+ for identifier in re_reds
+ }
+ problems.append(
+ f"re-reds {len(re_reds)} propert"
+ f"{'y' if len(re_reds) == 1 else 'ies'}: "
+ + ", ".join(
+ f"{identifier} (green on {ref}; red in "
+ f"{', '.join(where[identifier])})"
+ for identifier in re_reds
+ )
+ )
if problems:
print()