From 6bb344a44513c0ab456bd73e7a8d1e3babaa46fa Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Wed, 2 Sep 2026 09:24:28 -0400 Subject: [PATCH 1/6] Interface amendments 11-13: entrants, mass partition, declared tolerance - decl.py: Node.entrants (EXPAND only; cannot conserve mass) and Graph.mass_partition (an (entity, column) every CREATE node declares with a partition dtype; normative, folded into structural node keys). - kernel.py: Tolerance(rtol, atol, ulps); Capabilities.tolerance is required for tolerance_bound kernels and forbidden for bitwise ones; KernelContext.tolerances hands readers their inputs' owners' bounds. - serialize.py and keys.py carry the new fields; older JSON still loads. - fit.qrf@1 declares a provisional Tolerance(rtol=1e-6) pending measurement across arm64 and x86_64. - Charter: amendments 11-13, D2 clarified (weighted person mass per stratum; composition-changing expansions declare), rows B6, C5, D6. - Lock re-recorded; B2's KernelContext field pin gains tolerances. Max ruled go on the two dynamics decisions (microcosm-dynamics#412); the B6/C5/D6 acceptance tests follow from the acceptance lane. Co-Authored-By: Claude Fable 5 --- docs/graph-acceptance.md | 29 ++++- docs/graph-interface.lock | 4 +- .../src/microcosm/fit/kernels.py | 9 ++ packages/microcosm-fit/tests/test_kernels.py | 2 + .../src/microcosm/graph/__init__.py | 4 + .../src/microcosm/graph/decl.py | 77 ++++++++++- .../src/microcosm/graph/kernel.py | 62 +++++++++ .../src/microcosm/graph/keys.py | 6 + .../src/microcosm/graph/serialize.py | 27 +++- .../tests/test_acceptance_b_ownership.py | 1 + .../microcosm-graph/tests/test_graph_decl.py | 56 ++++++++ .../tests/test_graph_kernel_contract.py | 121 ++++++++++++++++++ 12 files changed, 392 insertions(+), 6 deletions(-) create mode 100644 packages/microcosm-graph/tests/test_graph_kernel_contract.py diff --git a/docs/graph-acceptance.md b/docs/graph-acceptance.md index ef1ea00fe..78fbc97a5 100644 --- a/docs/graph-acceptance.md +++ b/docs/graph-acceptance.md @@ -45,6 +45,7 @@ its owner. | B3 | **Storage-preserving patch.** Patching owned positions preserves the incumbent column's dtype (nullable `boolean` stays nullable `boolean`; float bits including negative zero survive) and leaves every non-owned position byte-identical. This is the WIC guard, made structural. | leg 2 §3.3 | same | | B4 | **Inputs are immutable.** A kernel receives read-only views; an in-place write raises inside the kernel and the node fails. | leg 1 finding 5 | same | | B5 | **Null means absence.** A node declares each owned cell as *produced* or *absent*. A kernel writing a non-null value into an absent-declared cell is rejected. | `DESIGN.md:128-134` | same | +| B6 | **Entrants are declared.** An `EXPAND` node with `entrants=True` may return rows with null lineage; the executor requires the kernel to materialize every carried column for such a row (dtype-checked), records them as entrants rather than copies in the lineage receipt, and refuses null lineage on a node without the declaration. `entrants=True` with `mass='conserve'` is a compile error. | Dynamics: immigrant cohorts (microcosm-dynamics#412, #218) | Max's session; amendment 11 | ## C. Seeds and factorization @@ -54,16 +55,18 @@ its owner. | C2 | **Removal invariance.** Removing a node that nothing depends on, or adding a new leaf node, changes no other node's key or output. This is the `0347a009` replay: five targets removed, zero survivors re-modeled. | F5 | same | | C3 | **Declared predecessors only.** A chained target's predictors are exactly its declared predecessors. The executor hands a kernel only its declared slices, so an undeclared read is impossible rather than merely detected. | F5, leg 3 §legibility | same | | C4 | **Seed from identity.** A node's RNG seed is a pure function of its node key. Two nodes with identical declarations, inputs, and kernels in different graphs draw identical values. No positional RNG consumption exists anywhere in the shard (static check). | F4, `docs/spec-engine.md:254-282` | same | +| C5 | **Tolerance is declared.** A kernel claiming `tolerance_bound` numerics without a `Tolerance` is refused at registration, and a bitwise kernel may not carry one. The tolerance is recorded in every receipt, and a kernel reading a cell sees its owner's declared tolerance in `KernelContext.tolerances`; a gate comparing against anything else says so in its evidence. | H2 (arm64/x86 one-ulp weights); microcosm-dynamics#412 | Max's session; amendment 13 | ## D. Weights and mass | Id | Property | Closes | Owner | |---|---|---|---| | D1 | **Weight transitions are typed nodes.** `design → importance → calibrated` are the only legal transitions; the executor rejects a regression and rejects a transition declared on inherited (non-explicit) weights. | F9 (leg 1 finding 1) | María / Max's session | -| D2 | **Mass ledger.** Every population-changing node (select, concat, clone, reweight) emits a mass record with before/after totals and per-stratum mass. Under `conserve`, a stratum losing mass fails the node. `select` cannot drop mass silently. | F9 | same | +| D2 | **Mass ledger.** Every population-changing node (select, concat, clone, reweight) emits a mass record with before/after totals and per-stratum mass. Mass is weighted person mass per stratum, within each declared partition (amendment 12). Under `conserve`, a stratum losing mass fails the node. An expansion that conserves its weight entity's mass while changing group composition changes person mass and must say so: `declared`, with a receipt stating the invariant it does hold (ruled 2026-09-02 on #844). `select` cannot drop mass silently. | F9 | same | | D3 | **Cap anchored to design.** A calibration node's `max_weight_ratio` is asserted against the declared anchor across composed stages; a selection-then-refit chain that ships a record above `R × design` fails. | F9 (#493) | same | | D4 | **Filters are binary.** A target filter containing NaN or a non-binary value is rejected at compile. | F9 | same | | D5 | **Uncertainty travels.** A target's declared standard error reaches the calibration kernel's inputs; a kernel that ignores a declared `se` must say so in its capability record. | scoreboard row 5 (leg 1 finding 7) | same | +| D6 | **Mass is partitioned.** With `Graph.mass_partition` set, the ledger reports per stratum within each partition value, `conserve` holds within each partition, and a node that moves mass between partitions under `conserve` fails. Every `CREATE` node declares the partition column with a partition dtype, or compilation fails. A row contributes mass only to the partitions it exists in. | Dynamics: person-period residency (microcosm-dynamics#412) | Max's session; amendment 12 | ## E. Store and resume @@ -201,6 +204,30 @@ Amendments so far (each re-locked): (python storage); a population entering the graph with `object` strings is normalized at `CREATE`. Parity fixtures compare identities after the same normalization on the legacy side, and say so. +11. **Entrants are declared.** `Node.entrants=True` (EXPAND only) lets a + kernel add rows that copy no base row: their lineage is null, the + kernel materializes every carried column for them, the executor + records them as entrants, and the node's mass policy cannot be + `conserve`. Raised by the dynamics program (immigrant cohorts through + the scheduled-entries seam, microcosm-dynamics#412 / #218); Max ruled + go 2026-09-02; adopted 2026-09-02. +12. **Mass is partitioned.** `Graph.mass_partition = (entity, column)` + partitions mass accounting (per stratum within each partition value; + `conserve` per partition). Every `CREATE` node declares the column + with a dtype in `PARTITION_DTYPES`. The field is normative: the + executor folds it into every structural node's key, so structural keys + move once when a graph adopts it. Raised by the dynamics program for + person-period residency; adopted 2026-09-02. +13. **Tolerance is declared.** `Capabilities.tolerance: Tolerance | None` + (`rtol`, `atol`, `ulps`) is required for `tolerance_bound` kernels and + forbidden for bitwise ones; `KernelContext.tolerances` hands each + reader the declared tolerance of every input cell's owner. Raised by + the H2 parity finding (root weights differ by one ulp between arm64 + and x86) and the dynamics review; adopted 2026-09-02. + +Adding a normative field with a default changes the canonical projection +of every node that carries it, so node keys moved with amendments 11 and +13's sibling field `entrants`; no released artifact pins a graph key yet. ## Ownership diff --git a/docs/graph-interface.lock b/docs/graph-interface.lock index 994c05b9d..7f954afe4 100644 --- a/docs/graph-interface.lock +++ b/docs/graph-interface.lock @@ -1,2 +1,2 @@ -b6fbf50e0253e6f04a2684fcac295fb27a4608f6a272bedb86c43f087fd36b8e decl.py -8ac8328c086bfbea0e36032e9c7fa9cb1ed62f63a1036bca4eb528c11b9b1890 kernel.py +bdd74c8598a8537f7ed95ffa712726c8b01d8b8b840e5d9e7006b2cc705bd0ff decl.py +ec2f9184450edd8bca6835e63d927fa9bbcf5dcf21acd808794a8a1a520ae8a9 kernel.py diff --git a/packages/microcosm-fit/src/microcosm/fit/kernels.py b/packages/microcosm-fit/src/microcosm/fit/kernels.py index c39c9a83e..1df027289 100644 --- a/packages/microcosm-fit/src/microcosm/fit/kernels.py +++ b/packages/microcosm-fit/src/microcosm/fit/kernels.py @@ -27,6 +27,7 @@ KernelResult, Numeric, SeedSource, + Tolerance, source_hash, ) @@ -48,6 +49,13 @@ ) """Distributions whose versions form part of ``fit.qrf@1``'s identity.""" +#: How far ``fit.qrf@1`` numbers may move between machines. The forest stack +#: promises no cross-platform bit stability (charter H1 records the claim as +#: ``tolerance_bound``); this bound is provisional until measured on the H1 +#: fixture across arm64 and x86_64 (amendment 13 follow-up), and parity in +#: the locked environment is still asserted byte for byte. +FIT_QRF_TOLERANCE = Tolerance(rtol=1e-6) + QRF_EXECUTOR_SEED_HIGH = 2**31 - 1 """Exclusive upper bound for the one seed drawn from ``KernelContext.rng``.""" @@ -114,6 +122,7 @@ def __init__(self, seed_source: SeedSource = SeedSource.EXECUTOR) -> None: numeric=Numeric.TOLERANCE_BOUND, seed_source=seed_source, dependencies=FIT_QRF_DEPENDENCIES, + tolerance=FIT_QRF_TOLERANCE, ) def implementation_hash(self) -> str: diff --git a/packages/microcosm-fit/tests/test_kernels.py b/packages/microcosm-fit/tests/test_kernels.py index 356705a7b..930694005 100644 --- a/packages/microcosm-fit/tests/test_kernels.py +++ b/packages/microcosm-fit/tests/test_kernels.py @@ -31,6 +31,7 @@ Owned, SeedSource, Slice, + Tolerance, source_hash, ) @@ -233,6 +234,7 @@ def test_capabilities_protocol_and_wrapped_source_hash() -> None: numeric=Numeric.TOLERANCE_BOUND, seed_source=SeedSource.PARAM, dependencies=FIT_QRF_DEPENDENCIES, + tolerance=Tolerance(rtol=1e-6), ) assert QRF_EXECUTOR_KERNEL.capabilities.seed_source is SeedSource.EXECUTOR assert QRF_PARAM_KERNEL.implementation_hash() == source_hash( diff --git a/packages/microcosm-graph/src/microcosm/graph/__init__.py b/packages/microcosm-graph/src/microcosm/graph/__init__.py index 696158e73..a090e96cb 100644 --- a/packages/microcosm-graph/src/microcosm/graph/__init__.py +++ b/packages/microcosm-graph/src/microcosm/graph/__init__.py @@ -13,6 +13,7 @@ DTYPES, GATE_OUTCOMES, MASS_POLICIES, + PARTITION_DTYPES, ROWS_ALL, WEIGHT_KINDS, CompiledGraph, @@ -46,6 +47,7 @@ KernelRole, Numeric, SeedSource, + Tolerance, source_hash, ) @@ -54,6 +56,7 @@ "DTYPES", "GATE_OUTCOMES", "MASS_POLICIES", + "PARTITION_DTYPES", "ROWS_ALL", "WEIGHT_KINDS", "Capabilities", @@ -85,6 +88,7 @@ "RunManifest", "SOURCE_CODECS", "SeedSource", + "Tolerance", "Slice", "SourceCodec", "SourceCodecRegistry", diff --git a/packages/microcosm-graph/src/microcosm/graph/decl.py b/packages/microcosm-graph/src/microcosm/graph/decl.py index b7460730b..b88b4bd23 100644 --- a/packages/microcosm-graph/src/microcosm/graph/decl.py +++ b/packages/microcosm-graph/src/microcosm/graph/decl.py @@ -24,6 +24,17 @@ known at compile time: a mask that is not ``bool`` or ``boolean`` is a compile error (charter D4). Nulls inside a nullable mask are a run-time rejection by the executor. +- An ``EXPAND`` node copies rows: every new row names the base row it + copies, so lineage is total. A node that declares ``entrants=True`` may + also add rows that copy nothing (births not patterned on a parent, + immigrant cohorts); the kernel materializes every carried column for + such a row, the executor records them as entrants rather than copies, + and the node cannot claim to conserve mass (amendment 11). +- Mass is accounted as weighted person mass per stratum. A graph may name + a partition column (:attr:`Graph.mass_partition`, e.g. a period on a + person-period population); the ledger then reports per stratum within + each partition and ``conserve`` holds within each partition, so a row + contributes mass only to the partitions it exists in (amendment 12). This file is a frozen interface (see ``docs/graph-acceptance.md``). """ @@ -42,6 +53,7 @@ "GATE_OUTCOMES", "MASK_DTYPES", "MASS_POLICIES", + "PARTITION_DTYPES", "ROWS_ALL", "WEIGHT_KINDS", "CompiledGraph", @@ -82,6 +94,9 @@ #: Mass policies a weight transition or structural node may declare. MASS_POLICIES = frozenset({"conserve", "free", "declared"}) +#: The dtypes a mass-partition column may have. +PARTITION_DTYPES = frozenset({"int32", "int64", "string"}) + #: The closed set of gate outcomes (charter F4). ``unreached`` is also the #: outcome of a release whose required human decisions are absent. GATE_OUTCOMES = ("pass", "fail", "evidence_absent", "not_applicable", "unreached") @@ -269,6 +284,11 @@ class Node: sources: Names of :class:`SourceRef` entries this node reads. weights: A declared weight-kind transition, if any. mass: Mass policy for structural nodes that change rows or weights. + entrants: ``EXPAND`` nodes only: the kernel may add rows that copy + no base row. Such a row has null lineage, the kernel supplies + every carried column for it, and the executor records it as an + entrant. Entrants add mass, so the node's mass policy cannot be + ``conserve``. description: Descriptive; never hashed. citation: Descriptive; never hashed. """ @@ -284,6 +304,7 @@ class Node: sources: tuple[str, ...] = () weights: WeightTransition | None = None mass: str = "conserve" + entrants: bool = False description: str = "" citation: str = "" @@ -333,6 +354,17 @@ def __post_init__(self) -> None: ) elif self.base is not None: raise GraphError(f"Node {self.id!r}: only structural nodes have a base.") + if not isinstance(self.entrants, bool): + raise GraphError(f"Node {self.id!r}: entrants must be a boolean.") + if self.entrants and self.structural is not StructuralDelta.EXPAND: + raise GraphError( + f"Node {self.id!r}: only an EXPAND node may admit entrants." + ) + if self.entrants and self.mass == "conserve": + raise GraphError( + f"Node {self.id!r}: entrants add mass, so an entrant-admitting " + "node cannot declare mass='conserve'." + ) if self.weights is not None and self.structural is not StructuralDelta.REWEIGHT: raise GraphError( f"Node {self.id!r}: a weight transition changes the population " @@ -386,11 +418,18 @@ class Graph: into node keys (a node's identity is its computation). sources: External inputs by name. nodes: Every node. Declaration order carries no meaning. + mass_partition: ``(entity, column)`` of a column that partitions + mass accounting, or ``None``. When set, every ``CREATE`` node + declares the column with a dtype in :data:`PARTITION_DTYPES`, + the executor's ledger reports per stratum within each partition + value, and ``conserve`` holds within each partition. Normative: + it enters the key of every structural node. """ country: str sources: tuple[SourceRef, ...] nodes: tuple[Node, ...] + mass_partition: tuple[str, str] | None = None def __post_init__(self) -> None: _nonempty("Graph.country", self.country) @@ -398,6 +437,21 @@ def __post_init__(self) -> None: raise GraphError("Graph repeats a source name.") if len({n.id for n in self.nodes}) != len(self.nodes): raise GraphError("Graph repeats a node id.") + if self.mass_partition is not None: + if ( + not isinstance(self.mass_partition, tuple) + or len(self.mass_partition) != 2 + or not all(isinstance(part, str) for part in self.mass_partition) + ): + raise GraphError( + "Graph.mass_partition must be an (entity, column) pair of strings." + ) + _nonempty("Graph.mass_partition entity", self.mass_partition[0]) + _nonempty("Graph.mass_partition column", self.mass_partition[1]) + + def normative(self) -> dict[str, object]: + """The graph-level facts that enter every structural node's key.""" + return {"mass_partition": self.mass_partition} def node(self, node_id: str) -> Node: for node in self.nodes: @@ -436,8 +490,9 @@ def compile_graph(graph: Graph) -> CompiledGraph: GraphError: A cell with two owners or none (ownership is total and exclusive), an unknown source or population, a structural node whose base is not structural, a row mask whose declared dtype is - not boolean, a cycle, or a graph with several structural nodes - and a node that omits ``population``. + not boolean, a mass-partition column that a ``CREATE`` node does + not declare with a partition dtype, a cycle, or a graph with + several structural nodes and a node that omits ``population``. """ by_id = {node.id: node for node in graph.nodes} @@ -494,6 +549,24 @@ def compile_graph(graph: Graph) -> CompiledGraph: owners[key] = node.id dtypes[key] = owned.dtype + if graph.mass_partition is not None: + entity, column = graph.mass_partition + for node in structural: + if node.structural is not StructuralDelta.CREATE: + continue + dtype = dtypes.get((node.id, entity, column)) + if dtype is None: + raise GraphError( + f"Graph.mass_partition names {entity}.{column}, which CREATE " + f"node {node.id!r} does not declare; partitions must exist " + "from the first version." + ) + if dtype not in PARTITION_DTYPES: + raise GraphError( + f"Graph.mass_partition {entity}.{column} is declared {dtype!r}; " + f"a partition column must be one of {sorted(PARTITION_DTYPES)}." + ) + def declared_dtype(version: str, entity: str, column: str) -> str | None: """The owner-declared dtype of a column as visible in ``version``.""" while True: diff --git a/packages/microcosm-graph/src/microcosm/graph/kernel.py b/packages/microcosm-graph/src/microcosm/graph/kernel.py index fbe66b2a5..68424479f 100644 --- a/packages/microcosm-graph/src/microcosm/graph/kernel.py +++ b/packages/microcosm-graph/src/microcosm/graph/kernel.py @@ -30,6 +30,12 @@ is derived from the gate verdicts in its ancestry, and its receipt reports ``unreached`` when a required human decision is absent from the run). +Numbers carry their own contract. A kernel whose :class:`Numeric` claim is +``tolerance_bound`` declares a :class:`Tolerance`; the executor records it +in the receipt and hands every reader the declared tolerance of each input +cell's owner through :attr:`KernelContext.tolerances`, so a gate compares +against a declaration rather than a guess (amendment 13). + This file is a frozen interface (see ``docs/graph-acceptance.md``). """ @@ -63,6 +69,7 @@ "KernelRole", "Numeric", "SeedSource", + "Tolerance", "source_hash", ] @@ -82,6 +89,42 @@ class Numeric(StrEnum): TOLERANCE_BOUND = "tolerance_bound" +@dataclass(frozen=True) +class Tolerance: + """How far a ``tolerance_bound`` kernel's numbers may move between runs. + + Two values agree when they are within ``atol`` absolutely, or within + ``rtol`` relatively, or within ``ulps`` last-place units of each other. + A bitwise kernel declares no tolerance at all. + + Attributes: + rtol: Relative tolerance; non-negative and finite. + atol: Absolute tolerance; non-negative and finite. + ulps: Units in the last place; non-negative. + """ + + rtol: float = 0.0 + atol: float = 0.0 + ulps: int = 0 + + def __post_init__(self) -> None: + for name in ("rtol", "atol"): + value = getattr(self, name) + if isinstance(value, bool) or not isinstance(value, int | float): + raise ValueError(f"Tolerance.{name} must be a number.") + if not (value >= 0.0) or value == float("inf"): + raise ValueError(f"Tolerance.{name} must be non-negative and finite.") + if isinstance(self.ulps, bool) or not isinstance(self.ulps, int): + raise ValueError("Tolerance.ulps must be an integer.") + if self.ulps < 0: + raise ValueError("Tolerance.ulps must be non-negative.") + if self.rtol == 0.0 and self.atol == 0.0 and self.ulps == 0: + raise ValueError( + "Tolerance must allow some movement; a bitwise kernel declares " + "no tolerance instead." + ) + + class SeedSource(StrEnum): """Where a seeded kernel takes its randomness from.""" @@ -113,6 +156,9 @@ class Capabilities: standard errors. A kernel that ignores them says so here. dependencies: Installed distributions whose versions enter the implementation hash. + tolerance: Required when ``numeric`` is ``tolerance_bound`` and + forbidden otherwise: how far the kernel's numbers may move + between runs or machines. """ determinism: Determinism @@ -122,6 +168,18 @@ class Capabilities: role: KernelRole = KernelRole.COMPUTE consumes_se: bool = False dependencies: tuple[str, ...] = () + tolerance: Tolerance | None = None + + def __post_init__(self) -> None: + if self.tolerance is not None and not isinstance(self.tolerance, Tolerance): + raise TypeError("Capabilities.tolerance must be a Tolerance or None.") + if self.numeric is Numeric.TOLERANCE_BOUND and self.tolerance is None: + raise ValueError( + "A tolerance_bound kernel must declare its Tolerance; a claim of " + "bounded movement without a bound is not a claim." + ) + if self.numeric is Numeric.BITWISE and self.tolerance is not None: + raise ValueError("A bitwise kernel declares no Tolerance.") @dataclass(frozen=True) @@ -144,6 +202,9 @@ class KernelContext: kernel may use. sources: Source name to a content-verified path, for declared sources only. + tolerances: ``(entity, column)`` of each declared input column to + the :class:`Tolerance` its owning kernel declared, or ``None`` + for a bitwise owner. A gate compares against these. """ node: Node @@ -153,6 +214,7 @@ class KernelContext: params: Mapping[str, Param] rng: np.random.Generator sources: Mapping[str, Path] = field(default_factory=dict) + tolerances: Mapping[tuple[str, str], Tolerance | None] = field(default_factory=dict) @dataclass(frozen=True) diff --git a/packages/microcosm-graph/src/microcosm/graph/keys.py b/packages/microcosm-graph/src/microcosm/graph/keys.py index cab41c700..daf3f5bc3 100644 --- a/packages/microcosm-graph/src/microcosm/graph/keys.py +++ b/packages/microcosm-graph/src/microcosm/graph/keys.py @@ -169,6 +169,11 @@ def node_key( else: resolved_sources = {} + # Graph-level mass semantics (amendment 12) change what a structural node + # computes, so they enter its key; an ordinary node's key is unaffected. + graph_facts = ( + {} if node.structural is StructuralDelta.NONE else compiled.graph.normative() + ) return _hash_parts( "node", normative(node), @@ -176,6 +181,7 @@ def node_key( population_input, kernel_impl_hash, resolved_sources, + graph_facts, ) diff --git a/packages/microcosm-graph/src/microcosm/graph/serialize.py b/packages/microcosm-graph/src/microcosm/graph/serialize.py index 32b78f56d..3ecc3ea54 100644 --- a/packages/microcosm-graph/src/microcosm/graph/serialize.py +++ b/packages/microcosm-graph/src/microcosm/graph/serialize.py @@ -37,6 +37,9 @@ def graph_to_json(graph: Graph) -> str: for source in graph.sources ], "nodes": [_node_payload(node) for node in graph.nodes], + "mass_partition": ( + None if graph.mass_partition is None else list(graph.mass_partition) + ), } return canonical_json(payload).decode("utf-8") @@ -51,7 +54,10 @@ def graph_from_json(text: str) -> Graph: except json.JSONDecodeError as error: raise ValueError("graph JSON is not valid JSON") from error root = _mapping(raw, "graph") - _exact_fields(root, {"country", "sources", "nodes"}, "graph") + fields = {"country", "sources", "nodes"} + if "mass_partition" in root: + fields.add("mass_partition") + _exact_fields(root, fields, "graph") sources_raw = _array(root["sources"], "graph.sources") nodes_raw = _array(root["nodes"], "graph.nodes") return Graph( @@ -63,9 +69,21 @@ def graph_from_json(text: str) -> Graph: nodes=tuple( _node_from_payload(value, index) for index, value in enumerate(nodes_raw) ), + mass_partition=_partition_from_payload( + root.get("mass_partition"), "graph.mass_partition" + ), ) +def _partition_from_payload(value: object, label: str) -> tuple[str, str] | None: + if value is None: + return None + parts = _array(value, label) + if len(parts) != 2: + raise TypeError(f"{label} must be an [entity, column] pair") + return (_string(parts[0], f"{label}[0]"), _string(parts[1], f"{label}[1]")) + + def _node_payload(node: Node) -> dict[str, object]: return { "id": node.id, @@ -104,6 +122,7 @@ def _node_payload(node: Node) -> dict[str, object]: } ), "mass": node.mass, + **({"entrants": True} if node.entrants else {}), "description": node.description, "citation": node.citation, } @@ -138,7 +157,12 @@ def _node_from_payload(value: object, index: int) -> Node: "description", "citation", } + if "entrants" in payload: + fields.add("entrants") _exact_fields(payload, fields, label) + entrants = payload.get("entrants", False) + if not isinstance(entrants, bool): + raise TypeError(f"{label}.entrants must be a boolean") inputs = _array(payload["inputs"], f"{label}.inputs") outputs = _array(payload["outputs"], f"{label}.outputs") sources = _array(payload["sources"], f"{label}.sources") @@ -173,6 +197,7 @@ def _node_from_payload(value: object, index: int) -> Node: ), weights=_weights_from_payload(payload["weights"], f"{label}.weights"), mass=_string(payload["mass"], f"{label}.mass"), + entrants=entrants, description=_string(payload["description"], f"{label}.description"), citation=_string(payload["citation"], f"{label}.citation"), ) diff --git a/packages/microcosm-graph/tests/test_acceptance_b_ownership.py b/packages/microcosm-graph/tests/test_acceptance_b_ownership.py index 72836472c..8447bb9e4 100644 --- a/packages/microcosm-graph/tests/test_acceptance_b_ownership.py +++ b/packages/microcosm-graph/tests/test_acceptance_b_ownership.py @@ -107,6 +107,7 @@ def test_b2_executor_enforces_ownership(tmp_path: Path) -> None: "params", "rng", "sources", + "tolerances", # amendment 13: declared tolerances of the inputs' owners } graph = toy.small_graph( diff --git a/packages/microcosm-graph/tests/test_graph_decl.py b/packages/microcosm-graph/tests/test_graph_decl.py index c5065ae6f..d75567fa2 100644 --- a/packages/microcosm-graph/tests/test_graph_decl.py +++ b/packages/microcosm-graph/tests/test_graph_decl.py @@ -270,3 +270,59 @@ def test_a_filter_cannot_read_a_column_nobody_defines() -> None: ) with pytest.raises(GraphError, match="no node owns"): compile_graph(Graph("toy", (SRC,), (CREATE, subset))) + + +def _expand(node_id: str, **overrides: object) -> Node: + return Node( + node_id, + "clone.persons@1", + base="survey", + structural=StructuralDelta.EXPAND, + mass="free", + **overrides, # type: ignore[arg-type] + ) + + +def test_entrants_are_an_expand_declaration_that_cannot_conserve_mass() -> None: + """Amendment 11: entrants add rows without lineage, so they add mass.""" + admitted = _expand("cohort", entrants=True) + assert admitted.entrants is True + assert "entrants" in admitted.normative() + with pytest.raises(GraphError, match="cannot declare mass='conserve'"): + _expand("cohort", entrants=True, mass="conserve") + with pytest.raises(GraphError, match="only an EXPAND node"): + Node("fit", "fit.qrf@1", entrants=True) + with pytest.raises(GraphError, match="must be a boolean"): + _expand("cohort", entrants="yes") + + +def test_mass_partition_must_be_declared_by_every_create_node() -> None: + """Amendment 12: partitions exist from the first version, with a partition dtype.""" + periodic = Node( + "survey", + "source.frame@1", + sources=("survey",), + structural=StructuralDelta.CREATE, + outputs=(Owned("person", "age", "int64"), Owned("person", "period", "int64")), + ) + graph = Graph("toy", (SRC,), (periodic,), mass_partition=("person", "period")) + compiled = compile_graph(graph) + assert compiled.graph.normative() == {"mass_partition": ("person", "period")} + assert Graph("toy", (SRC,), (periodic,)).normative() == {"mass_partition": None} + with pytest.raises(GraphError, match="does not declare"): + compile_graph( + Graph("toy", (SRC,), (CREATE,), mass_partition=("person", "period")) + ) + floating = Node( + "survey", + "source.frame@1", + sources=("survey",), + structural=StructuralDelta.CREATE, + outputs=(Owned("person", "period", "float64"),), + ) + with pytest.raises(GraphError, match="partition column must be one of"): + compile_graph( + Graph("toy", (SRC,), (floating,), mass_partition=("person", "period")) + ) + with pytest.raises(GraphError, match="pair of strings"): + Graph("toy", (SRC,), (periodic,), mass_partition=("person",)) # type: ignore[arg-type] diff --git a/packages/microcosm-graph/tests/test_graph_kernel_contract.py b/packages/microcosm-graph/tests/test_graph_kernel_contract.py new file mode 100644 index 000000000..48f197935 --- /dev/null +++ b/packages/microcosm-graph/tests/test_graph_kernel_contract.py @@ -0,0 +1,121 @@ +"""Kernel-protocol contracts of the frozen interface (amendment 13). + +A kernel that claims bounded numeric movement declares the bound; a bitwise +kernel declares none; the context hands readers their inputs' declared +tolerances; and the two new declaration fields round-trip through JSON. +""" + +from __future__ import annotations + +import numpy as np +import pandas as pd +import pytest + +from microcosm.graph import ( + Capabilities, + Determinism, + Graph, + KernelContext, + Node, + Numeric, + Owned, + SourceRef, + StructuralDelta, + Tolerance, + graph_from_json, + graph_to_json, +) + + +def test_tolerance_must_allow_some_movement_and_stay_finite() -> None: + assert Tolerance(rtol=1e-9).ulps == 0 + assert Tolerance(ulps=2) == Tolerance(0.0, 0.0, 2) + with pytest.raises(ValueError, match="allow some movement"): + Tolerance() + with pytest.raises(ValueError, match="non-negative and finite"): + Tolerance(rtol=-1e-9) + with pytest.raises(ValueError, match="non-negative and finite"): + Tolerance(atol=float("inf")) + with pytest.raises(ValueError, match="non-negative and finite"): + Tolerance(rtol=float("nan")) + with pytest.raises(ValueError, match="must be an integer"): + Tolerance(ulps=1.5) # type: ignore[arg-type] + + +def test_tolerance_bound_kernels_declare_a_bound_and_bitwise_kernels_none() -> None: + bounded = Capabilities( + determinism=Determinism.SEEDED, + numeric=Numeric.TOLERANCE_BOUND, + tolerance=Tolerance(rtol=1e-6), + ) + assert bounded.tolerance == Tolerance(rtol=1e-6) + with pytest.raises(ValueError, match="must declare its Tolerance"): + Capabilities(determinism=Determinism.SEEDED, numeric=Numeric.TOLERANCE_BOUND) + with pytest.raises(ValueError, match="bitwise kernel declares no Tolerance"): + Capabilities(determinism=Determinism.DETERMINISTIC, tolerance=Tolerance(ulps=1)) + with pytest.raises(TypeError, match="must be a Tolerance or None"): + Capabilities( + determinism=Determinism.DETERMINISTIC, + numeric=Numeric.TOLERANCE_BOUND, + tolerance=1e-6, # type: ignore[arg-type] + ) + + +def test_context_carries_declared_tolerances_and_defaults_to_none() -> None: + node = Node("gate", "gate.check@1") + context = KernelContext( + node=node, + tables={}, + weights={}, + strata=pd.Series([], dtype=object, name="stratum"), + params={}, + rng=np.random.default_rng(0), + ) + assert dict(context.tolerances) == {} + carried = KernelContext( + node=node, + tables={}, + weights={}, + strata=pd.Series([], dtype=object, name="stratum"), + params={}, + rng=np.random.default_rng(0), + tolerances={ + ("person", "income"): Tolerance(rtol=1e-6), + ("person", "age"): None, + }, + ) + assert carried.tolerances[("person", "income")] == Tolerance(rtol=1e-6) + assert carried.tolerances[("person", "age")] is None + + +def test_entrants_and_mass_partition_round_trip_through_canonical_json() -> None: + source = SourceRef("survey", "frame-h5") + create = Node( + "survey", + "source.frame@1", + sources=("survey",), + structural=StructuralDelta.CREATE, + outputs=(Owned("person", "age", "int64"), Owned("person", "period", "int64")), + ) + cohort = Node( + "cohort", + "enter.immigrants@1", + base="survey", + structural=StructuralDelta.EXPAND, + mass="declared", + entrants=True, + ) + graph = Graph( + "toy", (source,), (create, cohort), mass_partition=("person", "period") + ) + text = graph_to_json(graph) + assert '"entrants":true' in text and '"mass_partition":["person","period"]' in text + assert graph_from_json(text) == graph + plain = Graph("toy", (source,), (create,)) + plain_text = graph_to_json(plain) + assert "entrants" not in plain_text + assert graph_from_json(plain_text) == plain + # A declaration written before amendments 11 and 12 still loads. + legacy = plain_text.replace(',"mass_partition":null', "") + assert "mass_partition" not in legacy + assert graph_from_json(legacy) == plain From 416bfab5747a6f6c29bdb3b16f1cb1d21f3412a8 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Wed, 2 Sep 2026 09:31:36 -0400 Subject: [PATCH 2/6] Amendments 11-13: keep pre-amendment JSON byte-identical; re-pin H1; count 44 properties graph_to_json emits mass_partition only when declared, so every pinned graph JSON still round-trips unchanged. fit.qrf@1's implementation hash moved with its declared tolerance; the H1 pins are regenerated. The explorer test counts the charter's 44 properties. Co-Authored-By: Claude Fable 5 --- packages/microcosm-graph/src/microcosm/graph/serialize.py | 8 ++++++-- .../tests/fixtures/parity/kernels/fit.qrf/pins.json | 2 +- packages/microcosm-graph/tests/test_graph_decl.py | 4 ++-- packages/microcosm-graph/tests/test_graph_explain.py | 2 +- .../microcosm-graph/tests/test_graph_kernel_contract.py | 8 +++----- 5 files changed, 13 insertions(+), 11 deletions(-) diff --git a/packages/microcosm-graph/src/microcosm/graph/serialize.py b/packages/microcosm-graph/src/microcosm/graph/serialize.py index 3ecc3ea54..33b5437f3 100644 --- a/packages/microcosm-graph/src/microcosm/graph/serialize.py +++ b/packages/microcosm-graph/src/microcosm/graph/serialize.py @@ -37,8 +37,12 @@ def graph_to_json(graph: Graph) -> str: for source in graph.sources ], "nodes": [_node_payload(node) for node in graph.nodes], - "mass_partition": ( - None if graph.mass_partition is None else list(graph.mass_partition) + # Amendment 12: present only when declared, so a declaration written + # before the amendment serializes byte for byte as it did. + **( + {} + if graph.mass_partition is None + else {"mass_partition": list(graph.mass_partition)} ), } return canonical_json(payload).decode("utf-8") diff --git a/packages/microcosm-graph/tests/fixtures/parity/kernels/fit.qrf/pins.json b/packages/microcosm-graph/tests/fixtures/parity/kernels/fit.qrf/pins.json index ebfa035a4..3b8d76dad 100644 --- a/packages/microcosm-graph/tests/fixtures/parity/kernels/fit.qrf/pins.json +++ b/packages/microcosm-graph/tests/fixtures/parity/kernels/fit.qrf/pins.json @@ -1 +1 @@ -{"dependencies":{"numpy":"2.4.6","pandas":"3.0.3","quantile-forest":"1.4.2","scikit-learn":"1.8.0"},"implementation_hash":"8434e3e7e6333a4bb093f5d54a93e38d393553d3b27d811bec7671828688d707","kernel":"fit.qrf@1","node":"fit_qrf","seed":947} +{"dependencies":{"numpy":"2.4.6","pandas":"3.0.3","quantile-forest":"1.4.2","scikit-learn":"1.8.0"},"implementation_hash":"3a0fabca2f9bedf98c33846e7b6d59825f17007933b78ec5aeb31d3f0a2706e7","kernel":"fit.qrf@1","node":"fit_qrf","seed":947} diff --git a/packages/microcosm-graph/tests/test_graph_decl.py b/packages/microcosm-graph/tests/test_graph_decl.py index d75567fa2..37bb7b2e0 100644 --- a/packages/microcosm-graph/tests/test_graph_decl.py +++ b/packages/microcosm-graph/tests/test_graph_decl.py @@ -273,13 +273,13 @@ def test_a_filter_cannot_read_a_column_nobody_defines() -> None: def _expand(node_id: str, **overrides: object) -> Node: + settings: dict[str, object] = {"mass": "free", **overrides} return Node( node_id, "clone.persons@1", base="survey", structural=StructuralDelta.EXPAND, - mass="free", - **overrides, # type: ignore[arg-type] + **settings, # type: ignore[arg-type] ) diff --git a/packages/microcosm-graph/tests/test_graph_explain.py b/packages/microcosm-graph/tests/test_graph_explain.py index 62d9b6aa8..0f1a91135 100644 --- a/packages/microcosm-graph/tests/test_graph_explain.py +++ b/packages/microcosm-graph/tests/test_graph_explain.py @@ -172,7 +172,7 @@ def test_page_contains_every_node_and_its_click_detail(explanation) -> None: def test_page_contains_every_charter_property(explanation) -> None: _run, charter, rendered = explanation identifiers = re.findall(r"^\|\s*([A-Z]\d+)\s*\|", charter, re.MULTILINE) - assert len(dict.fromkeys(identifiers)) == 41 + assert len(dict.fromkeys(identifiers)) == 44 # 41 + B6, C5, D6 (amendments 11-13) for identifier in identifiers: assert f"{identifier}" in rendered assert "35 green" not in rendered # V1-V4 are also represented. diff --git a/packages/microcosm-graph/tests/test_graph_kernel_contract.py b/packages/microcosm-graph/tests/test_graph_kernel_contract.py index 48f197935..51373ac41 100644 --- a/packages/microcosm-graph/tests/test_graph_kernel_contract.py +++ b/packages/microcosm-graph/tests/test_graph_kernel_contract.py @@ -111,11 +111,9 @@ def test_entrants_and_mass_partition_round_trip_through_canonical_json() -> None text = graph_to_json(graph) assert '"entrants":true' in text and '"mass_partition":["person","period"]' in text assert graph_from_json(text) == graph + # A declaration without either field serializes exactly as it did before + # amendments 11 and 12, so every pinned graph JSON still matches. plain = Graph("toy", (source,), (create,)) plain_text = graph_to_json(plain) - assert "entrants" not in plain_text + assert "entrants" not in plain_text and "mass_partition" not in plain_text assert graph_from_json(plain_text) == plain - # A declaration written before amendments 11 and 12 still loads. - legacy = plain_text.replace(',"mass_partition":null', "") - assert "mass_partition" not in legacy - assert graph_from_json(legacy) == plain From 6d0f05601cd722115907d51c0fc814ad9676d477 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Wed, 2 Sep 2026 10:15:40 -0400 Subject: [PATCH 3/6] Commit charter B6, C5, D6 red; let the ratchet admit a property new to the charter The acceptance lane (sol, 20260902-092431-acceptance-11-13) added the three strict-xfail tests and the toy machinery they need: a toy EXPAND kernel with declared entrants, a tolerance-bound producer and a gate that reads context.tolerances, and a partitioned toy source. Each fails today at the absent feature, not before it (evidence in the lane report). tools/graph_acceptance_burndown.py now treats a marker on a property the baseline charter never listed as a new red rather than a re-red, with a unit test for both directions. The suite pins report 3 red until the implementation lane flips them. Co-Authored-By: Claude Fable 5 --- packages/microcosm-graph/tests/_toy.py | 194 ++++++++++++++++++ .../tests/test_acceptance_b_ownership.py | 78 ++++++- .../tests/test_acceptance_c_seeds.py | 88 +++++++- .../tests/test_acceptance_d_weights.py | 113 ++++++++++ .../tests/test_graph_acceptance_burndown.py | 56 ++++- .../tests/test_graph_explain.py | 2 +- tools/graph_acceptance_burndown.py | 26 ++- 7 files changed, 549 insertions(+), 8 deletions(-) diff --git a/packages/microcosm-graph/tests/_toy.py b/packages/microcosm-graph/tests/_toy.py index f37ab7767..f096bb878 100644 --- a/packages/microcosm-graph/tests/_toy.py +++ b/packages/microcosm-graph/tests/_toy.py @@ -69,6 +69,7 @@ Slice, SourceRef, StructuralDelta, + Tolerance, WeightTransition, compile_graph, run_graph, @@ -83,6 +84,7 @@ "PUBLISH_DECISION", "SOURCE", "STRATA", + "TOY_TOLERANCE", "ToyKernel", "ToyRun", "absent_node", @@ -96,6 +98,7 @@ "descendants", "draw", "drop_nodes", + "entrant_expand_node", "full_graph", "gate_node", "graph_source_files", @@ -130,6 +133,9 @@ #: The one source every toy graph reads, through the ``csv-tables`` codec. SOURCE = SourceRef("survey", "csv-tables", description="the toy country's tables") +#: Cross-machine numeric movement declared by the C5 toy producer. +TOY_TOLERANCE = Tolerance(rtol=1e-6) + def id_column(entity: str) -> str: """The frame convention: ``person_id`` for persons, ``{entity}_id`` elsewhere.""" @@ -432,6 +438,98 @@ def compute(self, context: KernelContext) -> KernelResult: ) +class ExpandEntrants(ToyKernel): + """EXPAND: add one copied person and one materialized entrant household.""" + + def compute(self, context: KernelContext) -> KernelResult: + person = context.tables["person"] + person_ids = pd.Index(person["person_id"], name="person_id") + person_source_id = int(person_ids[0]) + person_copy_id = int(person_ids.max()) + 1 + + household = context.tables["household"] + household_ids = pd.Index(household["household_id"], name="household_id") + household_entrant_id = int(household_ids.max()) + 1 + household_target_ids = household_ids.append( + pd.Index([household_entrant_id], dtype="int64", name="household_id") + ) + person_target_ids = person_ids.append( + pd.Index([person_copy_id], dtype="int64", name="person_id") + ) + + household_size = pd.concat( + [ + household["household_size"].reset_index(drop=True), + pd.Series([1], dtype="int64"), + ], + ignore_index=True, + ) + materialized_size = pd.Series( + household_size.array, index=household_target_ids, dtype="int64" + ) + if context.params.get("missing_entrant_column") == "household_size": + materialized_size = materialized_size.drop(index=household_entrant_id) + memberships = pd.concat( + [ + person["person_household_id"].reset_index(drop=True), + pd.Series([household_entrant_id], dtype="int64"), + ], + ignore_index=True, + ) + materialized_memberships = pd.Series( + memberships.array, index=person_target_ids, dtype="int64" + ) + + empty_releases = pd.Series( + [], + index=pd.Index([], dtype="int64", name="release_id"), + dtype="int64", + ) + household_weights = context.weights["household"] + expanded_weights = np.append( + household_weights.values, float(context.params["entrant_weight"]) + ) + return KernelResult( + expand={ + "person": pd.Series( + [person_source_id], + index=pd.Index([person_copy_id], dtype="int64", name="person_id"), + dtype="int64", + ), + "household": pd.Series( + pd.array([pd.NA], dtype="Int64"), + index=pd.Index( + [household_entrant_id], + dtype="int64", + name="household_id", + ), + ), + "release": empty_releases, + }, + columns={ + ("household", "household_size"): materialized_size, + ("person", "person_household_id"): materialized_memberships, + }, + weights=Weights(expanded_weights, kind=household_weights.kind), + ) + + +class ClaimMaterializedExpand(ToyKernel): + """Claim kernel-supplied EXPAND columns through the ownership surface.""" + + def compute(self, context: KernelContext) -> KernelResult: + columns: dict[tuple[str, str], pd.Series] = {} + for item in context.params["claim_cells"]: + entity, column, dtype = (str(value) for value in item) + table = context.tables[entity] + columns[(entity, column)] = pd.Series( + table[column].array.copy(), + index=pd.Index(table[id_column(entity)], name=id_column(entity)), + dtype=dtype, + ) + return KernelResult(columns=columns) + + class CalibrateToy(ToyKernel): """An ``importance -> calibrated`` transition hitting one target exactly.""" @@ -483,6 +581,37 @@ def compute(self, context: KernelContext) -> KernelResult: ) +class GateReportsTolerance(ToyKernel): + """A gate that reports the input owner's declared numeric tolerance.""" + + def compute(self, context: KernelContext) -> KernelResult: + entity = str(context.params["entity"]) + column = str(context.params["column"]) + observed = float(context.tables[entity][column].astype("float64").mean()) + declared = context.tolerances[(entity, column)] + tolerance = ( + None + if declared is None + else { + "rtol": declared.rtol, + "atol": declared.atol, + "ulps": declared.ulps, + } + ) + verdict_column = str(context.params["verdict_column"]) + return KernelResult( + columns={ + ("release", verdict_column): pd.Series( + ["pass"], index=_owned_ids(context, "release"), dtype="string" + ) + }, + receipt={ + "outcome": "pass", + "evidence": {"observed": observed, "tolerance": tolerance}, + }, + ) + + class ReleaseTier(ToyKernel): """Derive a release tier from the gate verdicts declared as its inputs.""" @@ -626,6 +755,9 @@ def compute(self, context: KernelContext) -> KernelResult: _FILTER = Capabilities( determinism=Determinism.DETERMINISTIC, structural=StructuralDelta.FILTER ) +_EXPAND = Capabilities( + determinism=Determinism.DETERMINISTIC, structural=StructuralDelta.EXPAND +) _REWEIGHT = Capabilities( determinism=Determinism.DETERMINISTIC, structural=StructuralDelta.REWEIGHT ) @@ -644,12 +776,22 @@ def toy_registry(*, variants: Mapping[str, str] | None = None) -> KernelRegistry kernels = ( SourceCsv("source.csv@1", _CREATE), DeriveAdd("derive.add@1", _DETERMINISTIC), + DeriveAdd( + "derive.tolerant@1", + Capabilities( + determinism=Determinism.DETERMINISTIC, + numeric=Numeric.TOLERANCE_BOUND, + tolerance=TOY_TOLERANCE, + ), + ), DrawUniform("draw.uniform@1", _SEEDED), ImputeChain("impute.chain@1", _SEEDED), SimulateStub("simulate.stub@1", _DETERMINISTIC), PatchColumn("patch.column@1", _DETERMINISTIC), AbsentColumn("absent.column@1", _DETERMINISTIC), SelectRows("select.rows@1", _FILTER), + ExpandEntrants("expand.entrants@1", _EXPAND), + ClaimMaterializedExpand("claim.expand@1", _DETERMINISTIC), ReweightScale("reweight.scale@1", _REWEIGHT), CalibrateToy( "calibrate.toy@1", @@ -664,6 +806,10 @@ def toy_registry(*, variants: Mapping[str, str] | None = None) -> KernelRegistry "gate.threshold@1", Capabilities(determinism=Determinism.DETERMINISTIC, role=KernelRole.GATE), ), + GateReportsTolerance( + "gate.tolerance@1", + Capabilities(determinism=Determinism.DETERMINISTIC, role=KernelRole.GATE), + ), ReleaseTier( "release.tier@1", Capabilities( @@ -974,6 +1120,54 @@ def select_node( ) +def entrant_expand_node( + node_id: str = "scheduled_entries", + *, + entrants: bool = True, + missing_entrant_column: str | None = None, +) -> tuple[Node, Node]: + """An EXPAND plus the ownership claim for its materialized person fields.""" + overlays = ( + ("household", "household_size", "int64"), + ("person", "person_household_id", "int64"), + ) + claim_cells = (overlays[0],) + expand = Node( + node_id, + "expand.entrants@1", + structural=StructuralDelta.EXPAND, + base="survey", + inputs=( + Slice("person", ("age",)), + Slice("household", ("household_size",)), + ), + params={ + "expand_cells": overlays, + "expand_weight_entity": "household", + "expand_weight_kind": "design", + "entrant_weight": 125.0, + "missing_entrant_column": missing_entrant_column, + }, + mass="free", + entrants=entrants, + ) + claim = Node( + f"claim_{node_id}", + "claim.expand@1", + outputs=tuple( + Owned(entity, column, dtype) for entity, column, dtype in claim_cells + ), + params={ + "claim_cells": claim_cells, + "materialized_expand_outputs": tuple( + f"{entity}.{column}" for entity, column, _ in claim_cells + ), + }, + population=node_id, + ) + return expand, claim + + POOL = Node( "pool", "reweight.scale@1", diff --git a/packages/microcosm-graph/tests/test_acceptance_b_ownership.py b/packages/microcosm-graph/tests/test_acceptance_b_ownership.py index 8447bb9e4..f652b0df7 100644 --- a/packages/microcosm-graph/tests/test_acceptance_b_ownership.py +++ b/packages/microcosm-graph/tests/test_acceptance_b_ownership.py @@ -22,7 +22,16 @@ import pandas as pd import pytest -from microcosm.graph import KernelContext, Node, Owned, Ownership, Slice +from microcosm.graph import ( + GraphError, + KernelContext, + Node, + NodeRejectedError, + Owned, + Ownership, + Slice, + StructuralDelta, +) if "_toy" not in sys.modules: _SPEC = importlib.util.spec_from_file_location( @@ -233,3 +242,70 @@ def test_b5_null_means_absence(tmp_path: Path) -> None: with pytest.raises(NodeRejectedError, match="no_data"): toy.run_toy(liar, tmp_path / "liar") + + +@pytest.mark.xfail(strict=True, reason="charter B6: entrant execution pending") +def test_b6_entrants_are_declared(tmp_path: Path) -> None: + """Null lineage is an explicit, complete, and receipted entrant contract. + + The backwards-compatible lineage shape keeps ``(new, source)`` pairs under + ``expand`` and uses a null source for entrants. The test admits a household + entrant because the frozen result interface has no separate output for a + new person's stratum. Its complete carried data surface is materialized by + the EXPAND kernel and passed through ``materialized_expand_outputs``. + """ + with pytest.raises(GraphError, match="conserved_entrants"): + Node( + "conserved_entrants", + "expand.entrants@1", + structural=StructuralDelta.EXPAND, + base="survey", + entrants=True, + mass="conserve", + ) + + expand, claim = toy.entrant_expand_node() + run = toy.run_toy( + toy.small_graph(nodes=(toy.CREATE, expand, claim)), tmp_path / "declared" + ) + before = run.manifest.population("survey") + after = run.manifest.population(expand.id) + person_copy_id = int(before.person["person_id"].max()) + 1 + household_entrant_id = int(before.household["household_id"].max()) + 1 + + assert len(after.person) == len(before.person) + 1 + assert len(after.household) == len(before.household) + 1 + copied = after.person.set_index("person_id").loc[person_copy_id] + source = before.person.set_index("person_id").loc[1] + assert copied["person_household_id"] == household_entrant_id + pd.testing.assert_series_equal( + copied.drop(labels="person_household_id"), + source.drop(labels="person_household_id"), + check_names=False, + ) + entrant = after.household.set_index("household_id").loc[household_entrant_id] + assert entrant["household_size"] == 1 + assert after.household["household_size"].dtype == np.dtype("int64") + assert after.weights_for("household").values[-1] == 125.0 + lineage = run.manifest.nodes[expand.id].receipt + assert lineage["expand"]["person"] == ((person_copy_id, 1),) + assert lineage["expand"]["household"] == ((household_entrant_id, None),) + + undeclared, undeclared_claim = toy.entrant_expand_node( + "undeclared_entrants", entrants=False + ) + with pytest.raises(NodeRejectedError, match="undeclared_entrants"): + toy.run_toy( + toy.small_graph(nodes=(toy.CREATE, undeclared, undeclared_claim)), + tmp_path / "undeclared", + ) + + incomplete, incomplete_claim = toy.entrant_expand_node( + "incomplete_entrant", missing_entrant_column="household_size" + ) + with pytest.raises(NodeRejectedError, match="incomplete_entrant") as error: + toy.run_toy( + toy.small_graph(nodes=(toy.CREATE, incomplete, incomplete_claim)), + tmp_path / "incomplete", + ) + assert "household_size" in str(error.value) diff --git a/packages/microcosm-graph/tests/test_acceptance_c_seeds.py b/packages/microcosm-graph/tests/test_acceptance_c_seeds.py index 07165e789..75f3eb62c 100644 --- a/packages/microcosm-graph/tests/test_acceptance_c_seeds.py +++ b/packages/microcosm-graph/tests/test_acceptance_c_seeds.py @@ -20,7 +20,20 @@ import sys from pathlib import Path -from microcosm.graph import ContentStore, Graph, compile_graph +import pytest + +from microcosm.graph import ( + Capabilities, + ContentStore, + Determinism, + Graph, + Node, + Numeric, + Owned, + Slice, + Tolerance, + compile_graph, +) if "_toy" not in sys.modules: _SPEC = importlib.util.spec_from_file_location( @@ -167,3 +180,76 @@ def test_c4_seed_from_identity(tmp_path: Path) -> None: assert elsewhere.keys()["draw_z"] != here.keys()["draw_a"] assert elsewhere.seeds()["draw_z"] != here.seeds()["draw_a"] assert len(set(elsewhere.seeds().values())) == len(elsewhere.seeds()) + + +@pytest.mark.xfail(strict=True, reason="charter C5: tolerance propagation pending") +def test_c5_tolerance_is_declared(tmp_path: Path) -> None: + """Receipts and readers carry an owner's exact declared tolerance. + + Capability receipts encode a tolerance as ``rtol``, ``atol``, and ``ulps``; + bitwise owners encode it as ``None``. A gate reads that same owner mapping + by coordinate and reports the JSON-safe value in its evidence. + """ + with pytest.raises(ValueError, match="must declare its Tolerance"): + Capabilities( + determinism=Determinism.DETERMINISTIC, + numeric=Numeric.TOLERANCE_BOUND, + ) + with pytest.raises(ValueError, match="bitwise kernel declares no Tolerance"): + Capabilities( + determinism=Determinism.DETERMINISTIC, + numeric=Numeric.BITWISE, + tolerance=Tolerance(rtol=1e-6), + ) + + bounded = Node( + "bounded", + "derive.tolerant@1", + inputs=(Slice("person", ("age",)),), + outputs=(Owned("person", "bounded_value", "float64"),), + params={ + "entity": "person", + "columns": ("age",), + "target": "bounded_value", + "scale": 1.0, + }, + population="survey", + ) + bitwise = toy.derive("bitwise", ("age",), "bitwise_value") + + def tolerance_gate(node_id: str, column: str) -> Node: + verdict = f"{node_id}_verdict" + return Node( + node_id, + "gate.tolerance@1", + inputs=(Slice("person", (column,)),), + outputs=(Owned("release", verdict, "string"),), + params={ + "entity": "person", + "column": column, + "verdict_column": verdict, + }, + population="survey", + ) + + bounded_gate = tolerance_gate("bounded_gate", "bounded_value") + bitwise_gate = tolerance_gate("bitwise_gate", "bitwise_value") + graph = Graph( + "toy", + (toy.SOURCE,), + (toy.CREATE, bounded, bitwise, bounded_gate, bitwise_gate), + ) + run = toy.run_toy(graph, tmp_path / "run") + bound = {"rtol": 1e-6, "atol": 0.0, "ulps": 0} + + bounded_receipt = run.manifest.nodes[bounded.id].receipt + assert bounded_receipt["capabilities"]["tolerance"] == bound + bounded_evidence = run.manifest.nodes[bounded_gate.id].receipt + assert bounded_evidence["outcome"] == "pass" + assert bounded_evidence["evidence"]["tolerance"] == bound + + bitwise_receipt = run.manifest.nodes[bitwise.id].receipt + assert bitwise_receipt["capabilities"]["tolerance"] is None + bitwise_evidence = run.manifest.nodes[bitwise_gate.id].receipt + assert bitwise_evidence["outcome"] == "pass" + assert bitwise_evidence["evidence"]["tolerance"] is None diff --git a/packages/microcosm-graph/tests/test_acceptance_d_weights.py b/packages/microcosm-graph/tests/test_acceptance_d_weights.py index 4b254bd78..cc4817e62 100644 --- a/packages/microcosm-graph/tests/test_acceptance_d_weights.py +++ b/packages/microcosm-graph/tests/test_acceptance_d_weights.py @@ -15,15 +15,21 @@ from __future__ import annotations import importlib.util +import json import sys +from dataclasses import replace from pathlib import Path +import numpy as np +import pandas as pd import pytest from microcosm.frame import WeightKind from microcosm.graph import ( + Graph, GraphError, Node, + NodeRejectedError, Owned, Slice, StructuralDelta, @@ -215,6 +221,113 @@ def test_d5_uncertainty_travels(tmp_path: Path) -> None: assert toy.calibrated_node(kernel="calibrate.blind@1").params["target_se"] == 2500.0 +@pytest.mark.xfail(strict=True, reason="charter D6: partitioned mass pending") +def test_d6_mass_is_partitioned(tmp_path: Path) -> None: + """Mass is conserved and receipted inside every partition value. + + The receipt retains D2's flat totals and stratum maps and adds + ``mass["partition"]`` with ``entity``, ``column``, and nested + ``stratum_before``/``stratum_after`` maps keyed first by the JSON-string + partition value and then by stratum. This test fixes that public shape for + the implementation lane. + """ + missing = Graph( + "toy", + (toy.SOURCE,), + (toy.CREATE,), + mass_partition=("person", "period"), + ) + with pytest.raises(GraphError, match=r"person\.period.*survey"): + compile_graph(missing) + + source_path = toy.copy_source(tmp_path / "period-source") + person_path = source_path / "person.csv" + person = pd.read_csv(person_path) + is_adult = person["is_adult"].astype("boolean").fillna(False).to_numpy(bool) + person["period"] = np.where( + ~is_adult | person["person_id"].mod(2).eq(0), 2024, 2025 + ).astype("int64") + person.to_csv(person_path, index=False) + + schema_path = source_path / "schema.json" + schema = json.loads(schema_path.read_text()) + schema["data_columns"]["person"].append("period") + schema["dtypes"]["period"] = "int64" + schema_path.write_text(json.dumps(schema, indent=2, sort_keys=True) + "\n") + + create = replace( + toy.CREATE, + outputs=(*toy.CREATE.outputs, Owned("person", "period", "int64")), + ) + sources = {"survey": source_path} + conserving = Graph( + "toy", + (toy.SOURCE,), + ( + create, + toy.select_node("period_conserve", base="survey", policy="conserve"), + ), + mass_partition=("person", "period"), + ) + with pytest.raises(NodeRejectedError, match="period_conserve") as rejected: + toy.run_toy(conserving, tmp_path / "conserve", sources=sources) + assert "2024" in str(rejected.value) + + free = Graph( + "toy", + (toy.SOURCE,), + (create, toy.select_node("period_free", base="survey", policy="free")), + mass_partition=("person", "period"), + ) + run = toy.run_toy(free, tmp_path / "free", sources=sources) + mass = run.manifest.nodes["period_free"].receipt["mass"] + partition = mass["partition"] + assert set(partition) == { + "entity", + "column", + "stratum_before", + "stratum_after", + } + assert partition["entity"] == "person" + assert partition["column"] == "period" + + weights = pd.read_csv(source_path / "weights.csv").set_index("household_id")[ + "design_weight" + ] + weighted = person.assign( + _mass=person["person_household_id"].map(weights).astype("float64") + ) + + def expected(frame: pd.DataFrame) -> dict[str, dict[str, float]]: + grouped = frame.groupby(["period", "stratum"], observed=True)["_mass"].sum() + return { + str(period): { + str(stratum): float(grouped.loc[(period, stratum)]) + for stratum in grouped.loc[period].index + } + for period in grouped.index.get_level_values("period").unique() + } + + before = expected(weighted) + after = expected(weighted.loc[is_adult]) + for key, wanted in (("stratum_before", before), ("stratum_after", after)): + actual = partition[key] + assert set(actual) == set(wanted) == {"2024", "2025"} + for period, strata in wanted.items(): + assert actual[period] == pytest.approx(strata) + + before_total = sum(sum(strata.values()) for strata in before.values()) + after_total = sum(sum(strata.values()) for strata in after.values()) + assert mass["before"] == pytest.approx(before_total) + assert mass["after"] == pytest.approx(after_total) + assert partition["stratum_after"]["2025"] == pytest.approx( + partition["stratum_before"]["2025"] + ) + assert sum(partition["stratum_after"]["2024"].values()) < sum( + partition["stratum_before"]["2024"].values() + ) + + def test_the_toy_country_declares_the_weight_lineage_the_charter_names() -> None: """The toy graph really does compose the three kinds, in order. diff --git a/packages/microcosm-graph/tests/test_graph_acceptance_burndown.py b/packages/microcosm-graph/tests/test_graph_acceptance_burndown.py index 91fa3d9b6..528542116 100644 --- a/packages/microcosm-graph/tests/test_graph_acceptance_burndown.py +++ b/packages/microcosm-graph/tests/test_graph_acceptance_burndown.py @@ -271,9 +271,19 @@ def test_the_real_suite_is_all_strict_and_all_accounted_for() -> None: """The tool's own checks, run against the suite it exists to score.""" data = burndown.report(burndown.counts(burndown.suite_files())) root = burndown.ROOT - assert data["total"] == 0 + assert data["total"] == 3 assert not [entry for entry in data["properties"] if entry["state"] == "missing"] - assert all(entry["state"] == "green" for entry in data["properties"]) + states = {entry["id"]: entry["state"] for entry in data["properties"]} + assert {identifier for identifier, state in states.items() if state == "red"} == { + "B6", + "C5", + "D6", + } + assert all( + state == "green" + for identifier, state in states.items() + if identifier not in {"B6", "C5", "D6"} + ) for entry in data["files"]: source = (root / entry["file"]).read_text() for marker in markers_in(source, entry["file"]): @@ -291,3 +301,45 @@ def test_the_tool_runs_from_the_command_line(flag: str) -> None: ) assert completed.returncode == 0, completed.stderr assert completed.stdout + + +NEW_PROPERTY_STARTS_RED = ( + ONE_RED_PROPERTY + + """ + +@pytest.mark.xfail(strict=True, reason="charter A9: pending") +def test_a9_nine() -> None: + assert False +""" +) + + +def test_verify_lets_a_property_new_to_the_charter_start_red(tmp_path: Path) -> None: + """The charter's meta-TDD rule: a new property is committed red first. + + A marker on an id the baseline charter never listed is not a re-red; a + marker on an id it did list still is. + """ + root = _repository(tmp_path, {"test_acceptance_a.py": ONE_RED_PROPERTY}) + (root / "docs" / "graph-acceptance.md").write_text( + "| Id | Property |\n|---|---|\n| A1 | one |\n| A3 | three |\n| A9 | nine |\n" + ) + target = root / "packages" / "microcosm-graph" / "tests" / "test_acceptance_a.py" + target.write_text(NEW_PROPERTY_STARTS_RED) + admitted = _run(root, "--verify") + assert admitted.returncode == 0, admitted.stdout + assert "1 -> 1 (+1 new: A9)" in admitted.stdout + assert "verification=ok" in admitted.stdout + + # The same marker on a property the baseline charter already listed + # (A3, green there) is a re-red and still fails. + target.write_text( + NEW_PROPERTY_STARTS_RED.replace( + "def test_a3_three() -> None:\n assert True", + '@pytest.mark.xfail(strict=True, reason="charter A3: pending")\n' + "def test_a3_three() -> None:\n assert False", + ) + ) + refused = _run(root, "--verify") + assert refused.returncode == 1 + assert "re-reds 1 property" in refused.stdout diff --git a/packages/microcosm-graph/tests/test_graph_explain.py b/packages/microcosm-graph/tests/test_graph_explain.py index 0f1a91135..074b64117 100644 --- a/packages/microcosm-graph/tests/test_graph_explain.py +++ b/packages/microcosm-graph/tests/test_graph_explain.py @@ -177,7 +177,7 @@ def test_page_contains_every_charter_property(explanation) -> None: assert f"{identifier}" in rendered assert "35 green" not in rendered # V1-V4 are also represented. assert "41 green" in rendered - assert "0 red" in rendered + assert "3 red" in rendered assert "Flip PR" in rendered assert "Not recorded" in rendered diff --git a/tools/graph_acceptance_burndown.py b/tools/graph_acceptance_burndown.py index 141a932c2..31d99536e 100644 --- a/tools/graph_acceptance_burndown.py +++ b/tools/graph_acceptance_burndown.py @@ -295,14 +295,34 @@ def verify(ref: str = BASELINE_REF) -> int: print(f"baseline={ref} unavailable; the ratchet did not run") else: print(f"baseline={ref}") + # A property the charter gained since the baseline is committed red + # first (the charter's meta-TDD rule), so its marker is not a re-red. + baseline_charter = baseline_source(ref, CHARTER) + known = set(charter_ids(baseline_charter)) if baseline_charter else set() for file in sorted(current): source = baseline_source(ref, file) if source is None: print(f" [new] {file}: {len(current[file])}") continue - was = len(markers_in(source, file)) - now = len(current[file]) - print(f" {'rose' if now > was else 'ok':<6} {file}: {was} -> {now}") + was_markers = markers_in(source, file) + was_ids = {marker.charter_id for marker in was_markers} + new_reds = sorted( + { + marker.charter_id + for marker in current[file] + if marker.charter_id + and marker.charter_id not in was_ids + and marker.charter_id not in known + } + ) + was = len(was_markers) + now = len(current[file]) - len(new_reds) + suffix = ( + f" (+{len(new_reds)} new: {', '.join(new_reds)})" if new_reds else "" + ) + print( + f" {'rose' if now > was else 'ok':<6} {file}: {was} -> {now}{suffix}" + ) if now > was: problems.append( f"{file} re-reds {now - was} propert" From 602c0f81146e655278fe2948c78faa81680818f8 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Wed, 2 Sep 2026 12:19:16 -0400 Subject: [PATCH 4/6] Amendment 14: entrant persons carry their stratum (charter B7, red) Implementing amendment 11 showed the frozen KernelResult had no channel for a new person's mandatory stratum, so entrant persons were fail-closed while immigrant cohorts are persons. KernelResult.strata (EXPAND kernels on an entrants=True node) names every entrant person's stratum by id; the executor requires exactly the entrant persons there. Charter row B7 is committed red with a toy kernel that admits one entrant person into an incumbent household; it fails today at the executor's null-lineage refusal. Lock re-recorded; suite pins count 45 properties, 4 red. Co-Authored-By: Claude Fable 5 --- docs/graph-acceptance.md | 8 ++ docs/graph-interface.lock | 2 +- .../src/microcosm/graph/kernel.py | 10 +- packages/microcosm-graph/tests/_toy.py | 134 ++++++++++++++++++ .../tests/test_acceptance_b_ownership.py | 45 ++++++ .../tests/test_graph_acceptance_burndown.py | 13 +- .../tests/test_graph_explain.py | 6 +- 7 files changed, 207 insertions(+), 11 deletions(-) diff --git a/docs/graph-acceptance.md b/docs/graph-acceptance.md index 78fbc97a5..50e63b7e8 100644 --- a/docs/graph-acceptance.md +++ b/docs/graph-acceptance.md @@ -46,6 +46,7 @@ its owner. | B4 | **Inputs are immutable.** A kernel receives read-only views; an in-place write raises inside the kernel and the node fails. | leg 1 finding 5 | same | | B5 | **Null means absence.** A node declares each owned cell as *produced* or *absent*. A kernel writing a non-null value into an absent-declared cell is rejected. | `DESIGN.md:128-134` | same | | B6 | **Entrants are declared.** An `EXPAND` node with `entrants=True` may return rows with null lineage; the executor requires the kernel to materialize every carried column for such a row (dtype-checked), records them as entrants rather than copies in the lineage receipt, and refuses null lineage on a node without the declaration. `entrants=True` with `mass='conserve'` is a compile error. | Dynamics: immigrant cohorts (microcosm-dynamics#412, #218) | Max's session; amendment 11 | +| B7 | **Entrant persons carry their stratum.** An entrant row on the person entity takes its stratum from `KernelResult.strata` (indexed by its new id); an entrant person absent from it, a label for a copied or incumbent person, or a label for an unknown id rejects the node. Entrant persons join incumbent or entrant groups through the materialized membership columns, and the mass ledger counts them from the node that admits them. | Dynamics: immigrant cohorts are persons (microcosm-dynamics#412, #218) | Max's session; amendment 14 | ## C. Seeds and factorization @@ -225,6 +226,13 @@ Amendments so far (each re-locked): the H2 parity finding (root weights differ by one ulp between arm64 and x86) and the dynamics review; adopted 2026-09-02. +14. **Entrant persons carry their stratum.** `KernelResult.strata` (EXPAND + kernels on an `entrants=True` node only) names the stratum of every + entrant person by its new id; the executor requires exactly the entrant + persons there. Raised by the implementation of amendment 11, which + found the frozen result had no channel for a new person's mandatory + stratum and left person entrants fail-closed; adopted 2026-09-02. + Adding a normative field with a default changes the canonical projection of every node that carries it, so node keys moved with amendments 11 and 13's sibling field `entrants`; no released artifact pins a graph key yet. diff --git a/docs/graph-interface.lock b/docs/graph-interface.lock index 7f954afe4..3f2187c0e 100644 --- a/docs/graph-interface.lock +++ b/docs/graph-interface.lock @@ -1,2 +1,2 @@ bdd74c8598a8537f7ed95ffa712726c8b01d8b8b840e5d9e7006b2cc705bd0ff decl.py -ec2f9184450edd8bca6835e63d927fa9bbcf5dcf21acd808794a8a1a520ae8a9 kernel.py +6ff42d1696c1bd35ae667dbcf109cc0e17341728510fd8e8c16a2f63fce7cab9 kernel.py diff --git a/packages/microcosm-graph/src/microcosm/graph/kernel.py b/packages/microcosm-graph/src/microcosm/graph/kernel.py index 68424479f..425da8e59 100644 --- a/packages/microcosm-graph/src/microcosm/graph/kernel.py +++ b/packages/microcosm-graph/src/microcosm/graph/kernel.py @@ -17,7 +17,10 @@ - ``EXPAND`` returns the clone lineage as :attr:`KernelResult.expand` (per entity, new ids to the source ids they copy) plus the new weights; the executor carries every column from the source rows, records the lineage - in the receipt, and records mass. + in the receipt, and records mass. A node declared ``entrants=True`` may + also add rows with null lineage; the kernel then materializes their + columns, and for entrant persons their stratum through + :attr:`KernelResult.strata` (amendments 11 and 14). - ``REWEIGHT`` (and any node with a declared weight transition) returns :attr:`KernelResult.weights`; the executor validates the kind transition and the mass policy. @@ -238,6 +241,10 @@ class KernelResult: ``weights``. weights: ``REWEIGHT`` kernels and declared weight transitions only: the new explicit weights of the transition's entity. + strata: ``EXPAND`` kernels on a node with ``entrants=True`` only: the + stratum label of every entrant person, indexed by its new id. + Copied persons inherit their source's stratum and must not + appear here; an entrant person absent from it rejects the node. artifacts: Opaque bytes stored beside the node's outputs (a fitted model, a diagnostic table), keyed by name. receipt: Descriptive facts for the manifest. Never hashed into a @@ -252,6 +259,7 @@ class KernelResult: keep: pd.Series | None = None expand: Mapping[str, pd.Series] | None = None weights: Weights | None = None + strata: pd.Series | None = None artifacts: Mapping[str, bytes] = field(default_factory=dict) receipt: Mapping[str, object] = field(default_factory=dict) diff --git a/packages/microcosm-graph/tests/_toy.py b/packages/microcosm-graph/tests/_toy.py index f096bb878..bbfb413e4 100644 --- a/packages/microcosm-graph/tests/_toy.py +++ b/packages/microcosm-graph/tests/_toy.py @@ -99,6 +99,7 @@ "draw", "drop_nodes", "entrant_expand_node", + "entrant_person_node", "full_graph", "gate_node", "graph_source_files", @@ -514,6 +515,86 @@ def compute(self, context: KernelContext) -> KernelResult: ) +class ExpandEntrantPerson(ToyKernel): + """EXPAND: admit one entrant person into an incumbent household. + + The entrant copies nothing: every person column is materialized from a + template row, its memberships name incumbent groups, and its stratum + arrives through ``KernelResult.strata`` (amendment 14). ``strata_mode`` + exercises the refusals: ``missing`` omits the field, ``unknown_id`` + labels an id the node never adds, ``labels_incumbent`` labels an + incumbent person as well. + """ + + def compute(self, context: KernelContext) -> KernelResult: + person = context.tables["person"] + person_ids = pd.Index(person["person_id"], name="person_id") + template = person.iloc[0] + entrant_id = int(person_ids.max()) + 1 + target_ids = person_ids.append( + pd.Index([entrant_id], dtype="int64", name="person_id") + ) + + def overlay(column: str, dtype: str, value: object) -> pd.Series: + values = pd.concat( + [person[column].reset_index(drop=True), pd.Series([value])], + ignore_index=True, + ) + return pd.Series(pd.array(values, dtype=dtype), index=target_ids) + + columns = { + ("person", "age"): overlay("age", "int64", 30), + ("person", "income"): overlay("income", "float64", 12_500.0), + ("person", "is_adult"): overlay("is_adult", "boolean", True), + ("person", "receives_x"): overlay("receives_x", "boolean", False), + ("person", "person_household_id"): overlay( + "person_household_id", "int64", int(template["person_household_id"]) + ), + ("person", "person_release_id"): overlay( + "person_release_id", "int64", int(template["person_release_id"]) + ), + } + mode = str(context.params.get("strata_mode", "ok")) + labelled = { + "ok": [entrant_id], + "unknown_id": [entrant_id + 1], + "labels_incumbent": [int(person_ids[0]), entrant_id], + } + strata = ( + None + if mode == "missing" + else pd.Series( + ["urban"] * len(labelled[mode]), + index=pd.Index(labelled[mode], dtype="int64", name="person_id"), + dtype=object, + name="stratum", + ) + ) + empty = { + entity: pd.Series( + [], + index=pd.Index([], dtype="int64", name=id_column(entity)), + dtype="int64", + ) + for entity in ("household", "release") + } + household_weights = context.weights["household"] + return KernelResult( + expand={ + "person": pd.Series( + pd.array([pd.NA], dtype="Int64"), + index=pd.Index([entrant_id], dtype="int64", name="person_id"), + ), + **empty, + }, + columns=columns, + weights=Weights( + household_weights.values.copy(), kind=household_weights.kind + ), + strata=strata, + ) + + class ClaimMaterializedExpand(ToyKernel): """Claim kernel-supplied EXPAND columns through the ownership surface.""" @@ -791,6 +872,7 @@ def toy_registry(*, variants: Mapping[str, str] | None = None) -> KernelRegistry AbsentColumn("absent.column@1", _DETERMINISTIC), SelectRows("select.rows@1", _FILTER), ExpandEntrants("expand.entrants@1", _EXPAND), + ExpandEntrantPerson("expand.entrant_person@1", _EXPAND), ClaimMaterializedExpand("claim.expand@1", _DETERMINISTIC), ReweightScale("reweight.scale@1", _REWEIGHT), CalibrateToy( @@ -1168,6 +1250,58 @@ def entrant_expand_node( return expand, claim +def entrant_person_node( + node_id: str = "immigrant_cohort", + *, + strata_mode: str = "ok", +) -> tuple[Node, Node]: + """An EXPAND admitting one entrant person, plus the claim of its cells.""" + data_cells = ( + ("person", "age", "int64"), + ("person", "income", "float64"), + ("person", "is_adult", "boolean"), + ("person", "receives_x", "boolean"), + ) + overlays = ( + *data_cells, + ("person", "person_household_id", "int64"), + ("person", "person_release_id", "int64"), + ) + expand = Node( + node_id, + "expand.entrant_person@1", + structural=StructuralDelta.EXPAND, + base="survey", + inputs=( + Slice("person", ("age", "income", "is_adult", "receives_x")), + Slice("household", ("household_size",)), + ), + params={ + "expand_cells": overlays, + "expand_weight_entity": "household", + "expand_weight_kind": "design", + "strata_mode": strata_mode, + }, + mass="free", + entrants=True, + ) + claim = Node( + f"claim_{node_id}", + "claim.expand@1", + outputs=tuple( + Owned(entity, column, dtype) for entity, column, dtype in data_cells + ), + params={ + "claim_cells": data_cells, + "materialized_expand_outputs": tuple( + f"{entity}.{column}" for entity, column, _ in data_cells + ), + }, + population=node_id, + ) + return expand, claim + + POOL = Node( "pool", "reweight.scale@1", diff --git a/packages/microcosm-graph/tests/test_acceptance_b_ownership.py b/packages/microcosm-graph/tests/test_acceptance_b_ownership.py index f652b0df7..2d0d6415a 100644 --- a/packages/microcosm-graph/tests/test_acceptance_b_ownership.py +++ b/packages/microcosm-graph/tests/test_acceptance_b_ownership.py @@ -309,3 +309,48 @@ def test_b6_entrants_are_declared(tmp_path: Path) -> None: tmp_path / "incomplete", ) assert "household_size" in str(error.value) + + +@pytest.mark.xfail(strict=True, reason="charter B7: entrant person strata pending") +def test_b7_entrant_persons_carry_their_stratum(tmp_path: Path) -> None: + """An entrant person's stratum arrives through ``KernelResult.strata``. + + Immigrant cohorts are persons, so an EXPAND admitting entrants must be + able to add a person that copies nobody: every column materialized, + memberships naming incumbent groups, and its stratum declared by id. + The ledger counts the entrant from the node that admits it; a missing + label, a label for an id the node never adds, or a label for an + incumbent person rejects the node by name. + """ + expand, claim = toy.entrant_person_node() + run = toy.run_toy( + toy.small_graph(nodes=(toy.CREATE, expand, claim)), tmp_path / "cohort" + ) + before = run.manifest.population("survey") + after = run.manifest.population(expand.id) + entrant_id = int(before.person["person_id"].max()) + 1 + + assert len(after.person) == len(before.person) + 1 + assert len(after.household) == len(before.household) + entrant = after.person.set_index("person_id").loc[entrant_id] + assert entrant["age"] == 30 and entrant["income"] == 12_500.0 + assert ( + entrant["person_household_id"] == before.person["person_household_id"].iloc[0] + ) + assert after.strata.iloc[-1] == "urban" + assert after.strata.iloc[: len(before.person)].tolist() == before.strata.tolist() + + receipt = run.manifest.nodes[expand.id].receipt + assert receipt["expand"]["person"] == ((entrant_id, None),) + mass = receipt["mass"] + assert mass["after"] > mass["before"] + assert mass["stratum_after"]["urban"] > mass["stratum_before"]["urban"] + assert mass["stratum_after"]["rural"] == mass["stratum_before"]["rural"] + + for mode in ("missing", "unknown_id", "labels_incumbent"): + bad, bad_claim = toy.entrant_person_node(f"cohort_{mode}", strata_mode=mode) + with pytest.raises(NodeRejectedError, match=f"cohort_{mode}"): + toy.run_toy( + toy.small_graph(nodes=(toy.CREATE, bad, bad_claim)), + tmp_path / mode, + ) diff --git a/packages/microcosm-graph/tests/test_graph_acceptance_burndown.py b/packages/microcosm-graph/tests/test_graph_acceptance_burndown.py index 528542116..172acb70e 100644 --- a/packages/microcosm-graph/tests/test_graph_acceptance_burndown.py +++ b/packages/microcosm-graph/tests/test_graph_acceptance_burndown.py @@ -271,18 +271,17 @@ def test_the_real_suite_is_all_strict_and_all_accounted_for() -> None: """The tool's own checks, run against the suite it exists to score.""" data = burndown.report(burndown.counts(burndown.suite_files())) root = burndown.ROOT - assert data["total"] == 3 + assert data["total"] == 4 assert not [entry for entry in data["properties"] if entry["state"] == "missing"] states = {entry["id"]: entry["state"] for entry in data["properties"]} - assert {identifier for identifier, state in states.items() if state == "red"} == { - "B6", - "C5", - "D6", - } + pending = {"B6", "B7", "C5", "D6"} # amendments 11-14, flipped by their lanes + assert { + identifier for identifier, state in states.items() if state == "red" + } == pending assert all( state == "green" for identifier, state in states.items() - if identifier not in {"B6", "C5", "D6"} + if identifier not in pending ) for entry in data["files"]: source = (root / entry["file"]).read_text() diff --git a/packages/microcosm-graph/tests/test_graph_explain.py b/packages/microcosm-graph/tests/test_graph_explain.py index 074b64117..4e2b8ddba 100644 --- a/packages/microcosm-graph/tests/test_graph_explain.py +++ b/packages/microcosm-graph/tests/test_graph_explain.py @@ -172,12 +172,14 @@ def test_page_contains_every_node_and_its_click_detail(explanation) -> None: def test_page_contains_every_charter_property(explanation) -> None: _run, charter, rendered = explanation identifiers = re.findall(r"^\|\s*([A-Z]\d+)\s*\|", charter, re.MULTILINE) - assert len(dict.fromkeys(identifiers)) == 44 # 41 + B6, C5, D6 (amendments 11-13) + assert ( + len(dict.fromkeys(identifiers)) == 45 + ) # 41 + B6, C5, D6, B7 (amendments 11-14) for identifier in identifiers: assert f"{identifier}" in rendered assert "35 green" not in rendered # V1-V4 are also represented. assert "41 green" in rendered - assert "3 red" in rendered + assert "4 red" in rendered assert "Flip PR" in rendered assert "Not recorded" in rendered From 721fcf32ff1c468718b81ea606a861de39ea328e Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Wed, 2 Sep 2026 13:07:39 -0400 Subject: [PATCH 5/6] Capabilities validates every field; the registry requires a real instance The #851 review showed a string spelling an enum member passed as the member, so a kernel could claim tolerance_bound without a bound. Every Capabilities field is now checked at construction (enums by type, consumes_se a boolean, dependencies a tuple of names) and KernelRegistry.register refuses a look-alike. Lock re-recorded. Co-Authored-By: Claude Fable 5 --- .impl-b7-link | 1 + docs/graph-interface.lock | 2 +- .../src/microcosm/graph/kernel.py | 24 ++++++++++++ .../tests/test_graph_kernel_contract.py | 38 +++++++++++++++++++ 4 files changed, 64 insertions(+), 1 deletion(-) create mode 120000 .impl-b7-link diff --git a/.impl-b7-link b/.impl-b7-link new file mode 120000 index 000000000..f49c784f9 --- /dev/null +++ b/.impl-b7-link @@ -0,0 +1 @@ +/private/tmp/microcosm-impl-b7 \ No newline at end of file diff --git a/docs/graph-interface.lock b/docs/graph-interface.lock index 3f2187c0e..aed00bcc1 100644 --- a/docs/graph-interface.lock +++ b/docs/graph-interface.lock @@ -1,2 +1,2 @@ bdd74c8598a8537f7ed95ffa712726c8b01d8b8b840e5d9e7006b2cc705bd0ff decl.py -6ff42d1696c1bd35ae667dbcf109cc0e17341728510fd8e8c16a2f63fce7cab9 kernel.py +b4315d44f7e54a845d7efd7ac1239a813ddd312ce642869b9e15a2b66f62ea45 kernel.py diff --git a/packages/microcosm-graph/src/microcosm/graph/kernel.py b/packages/microcosm-graph/src/microcosm/graph/kernel.py index 425da8e59..dd02d5a0e 100644 --- a/packages/microcosm-graph/src/microcosm/graph/kernel.py +++ b/packages/microcosm-graph/src/microcosm/graph/kernel.py @@ -174,6 +174,25 @@ class Capabilities: tolerance: Tolerance | None = None def __post_init__(self) -> None: + # Every field is validated here, so a registered contract is a real + # one: a string that spells an enum member does not pass as the member. + for name, kind in ( + ("determinism", Determinism), + ("numeric", Numeric), + ("seed_source", SeedSource), + ("structural", StructuralDelta), + ("role", KernelRole), + ): + if not isinstance(getattr(self, name), kind): + raise TypeError(f"Capabilities.{name} must be a {kind.__name__}.") + if not isinstance(self.consumes_se, bool): + raise TypeError("Capabilities.consumes_se must be a boolean.") + if not isinstance(self.dependencies, tuple) or any( + not isinstance(name, str) or not name for name in self.dependencies + ): + raise TypeError( + "Capabilities.dependencies must be a tuple of distribution names." + ) if self.tolerance is not None and not isinstance(self.tolerance, Tolerance): raise TypeError("Capabilities.tolerance must be a Tolerance or None.") if self.numeric is Numeric.TOLERANCE_BOUND and self.tolerance is None: @@ -351,6 +370,11 @@ def __init__(self) -> None: def register(self, kernel: Kernel) -> Kernel: if not isinstance(kernel, Kernel): raise TypeError(f"{kernel!r} does not satisfy the Kernel protocol.") + if not isinstance(kernel.capabilities, Capabilities): + raise TypeError( + f"Kernel {getattr(kernel, 'ref', kernel)!r} must carry a Capabilities " + "instance, not a look-alike." + ) if kernel.ref in self._kernels and self._kernels[kernel.ref] is not kernel: raise ValueError(f"Kernel {kernel.ref!r} is already registered.") self._kernels[kernel.ref] = kernel diff --git a/packages/microcosm-graph/tests/test_graph_kernel_contract.py b/packages/microcosm-graph/tests/test_graph_kernel_contract.py index 51373ac41..fd447e57d 100644 --- a/packages/microcosm-graph/tests/test_graph_kernel_contract.py +++ b/packages/microcosm-graph/tests/test_graph_kernel_contract.py @@ -15,7 +15,9 @@ Capabilities, Determinism, Graph, + KernelBase, KernelContext, + KernelRegistry, Node, Numeric, Owned, @@ -117,3 +119,39 @@ def test_entrants_and_mass_partition_round_trip_through_canonical_json() -> None plain_text = graph_to_json(plain) assert "entrants" not in plain_text and "mass_partition" not in plain_text assert graph_from_json(plain_text) == plain + + +def test_capabilities_reject_look_alike_fields_and_registration_needs_the_real_thing() -> ( + None +): + """A string spelling an enum member is not the member (review of #851, finding 5).""" + with pytest.raises(TypeError, match="Capabilities.numeric must be a Numeric"): + Capabilities(determinism=Determinism.DETERMINISTIC, numeric="tolerance_bound") # type: ignore[arg-type] + with pytest.raises( + TypeError, match="Capabilities.determinism must be a Determinism" + ): + Capabilities(determinism="deterministic") # type: ignore[arg-type] + with pytest.raises(TypeError, match="consumes_se must be a boolean"): + Capabilities(determinism=Determinism.DETERMINISTIC, consumes_se=1) # type: ignore[arg-type] + with pytest.raises(TypeError, match="dependencies must be a tuple"): + Capabilities(determinism=Determinism.DETERMINISTIC, dependencies=["numpy"]) # type: ignore[arg-type] + + class LookAlike: + determinism = Determinism.DETERMINISTIC + numeric = "tolerance_bound" + seed_source = "none" + structural = "none" + role = "compute" + consumes_se = False + dependencies = () + tolerance = None + + class Impostor(KernelBase): + ref = "impostor@1" + capabilities = LookAlike() # type: ignore[assignment] + + def run(self, context): # pragma: no cover - never reached + raise AssertionError + + with pytest.raises(TypeError, match="must carry a Capabilities instance"): + KernelRegistry().register(Impostor()) From 1c86caf0d23f73960efb45c583defcbb9fe68302 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Wed, 2 Sep 2026 13:41:19 -0400 Subject: [PATCH 6/6] Drop a stray lane worktree link file Co-Authored-By: Claude Fable 5 --- .impl-b7-link | 1 - 1 file changed, 1 deletion(-) delete mode 120000 .impl-b7-link diff --git a/.impl-b7-link b/.impl-b7-link deleted file mode 120000 index f49c784f9..000000000 --- a/.impl-b7-link +++ /dev/null @@ -1 +0,0 @@ -/private/tmp/microcosm-impl-b7 \ No newline at end of file