Amendment 20: keyed draw streams (SeedSource.KEYED, keyed_uniform, fit.qrf uniform-driven draws)

PROGRESS-amendment-20-keyed-draws.md

@@ -0,0 +1,165 @@
+# Amendment 20 — keyed draw streams (lane journal)
+
+Branch: `amend-keyed-seed-and-uniform-draws`, cut from `origin/main` at `3094bfe84`.
+Worktree: `~/PolicyEngine/_worktrees/microcosm-amend-keyed`. No push, no new branches.
+Platform of record for H1 pins: **arm64/darwin/py3.14** (Python 3.14.4) — the
+fixture's authoring platform.
+
+*Journal, not state: accurate as written on 2026-09-11. Check git and the
+tracking issue for what is true later.*
+
+Full receipts — every identity, command and exit code, and the three open
+decisions with their evidence — are in
+[`experiments/amendment-20-keyed-draws-receipts.md`](experiments/amendment-20-keyed-draws-receipts.md).
+
+## State
+
+Implementation, tests, pins, charter and changelog are landed and committed.
+`packages/microcosm-graph/tests` and `packages/microcosm-fit/tests` are green.
+
+> Historicized 2026-09-12: the paragraph below described the branch as the lane
+> left it. The seed digests were re-pinned when the branch was stacked on
+> amendment 19 and pushed as PR #912, so nothing here is red any more; the
+> `-0.0` question in item 7.3 of the receipts was settled by normalising
+> signed zero in `_coordinate`.
+
+**The branch cannot go green in CI as it stands**, and that is deliberate: the
+`qrf.py` edit moves the spec-engine seed digests, leaving seven tests red in a
+lane that runs on every PR. The lane brief said report the drift, not re-pin
+it, so the branch carries it and the decision goes to the merge owner — see
+**Open for decision** item 1, which carries a tested recipe.
+
+## Done
+
+1. `graph/kernel.py`: `SeedSource.KEYED` plus the `KernelContext.rng` docstring.
+   `ArtifactValue` / `KernelContext.artifacts` deliberately NOT brought over.
+2. `graph/randomness.py` (new, 69 lines): `keyed_uniform`, taken verbatim from
+   `origin/microcosm-us-launch-integration-20260909`.
+3. `graph/__init__.py`: `keyed_uniform` exported, inserted in sorted position
+   among the lowercase callables (`graph_to_json`, `keyed_uniform`, `load_source`).
+4. `fit/qrf.py`: `_draw_target_from_uniforms` + `predict_from_uniforms`, taken
+   verbatim from the same branch (`6e3907f86`); the diff applied with no conflict.
+5. `docs/graph-interface.lock`: `kernel.py` re-recorded
+   `eaf07da2… → 3483d091b03b19ae35c0268c01cb9e0f76c4cd63742083130567321d70da6048`.
+   The lock is plain `shasum -a 256` of the file bytes (confirmed against the
+   unchanged `decl.py` line).
+6. Tests: `test_graph_randomness.py` (new), `SeedSource.KEYED` contracts in
+   `test_graph_kernel_contract.py`, `test_qrf_stateless.py` (new, verbatim from
+   the integration branch), a `QRFKernel` non-widening guard in microcosm-fit's
+   `test_kernels.py`, and `test_graph_parity_pins.py` (new).
+7. `tools/graph_parity_repin.py` (new) + the re-pinned `fit.qrf` fixture.
+8. `docs/graph-acceptance.md` amendment 20 + a `changelog.d` fragment.
+
+**No `test_acceptance_*.py` file was edited at all**, so the "acceptance-suite
+edit is its own commit" rule never had to be exercised.
+
+### Second pass, after an adversarial review of the branch (same day)
+
+Six independent reviewers read the working tree; each finding was then put to
+two adversarial verifiers. Four findings survived and were fixed:
+
+9. **The re-pin now checks the environment it derives foreign keys from**
+   (`test_graph_parity_repin.py`, new, 8 tests). The guard added in the first
+   pass reproduced every pinned key from `pins["implementation_hash"]` — which
+   substitutes away the only input carrying dependency versions, so *any*
+   environment reproduced the pins and the check could not see the drift it
+   existed to catch. `pins.json` already recorded the versions the keys were
+   taken under; nothing read them. `repin` now refuses unless they equal this
+   machine's installed versions, before deriving anything.
+10. **The re-pin docstring no longer claims what the code does not do.** It had
+    said the reproduction loop "proves the assumption rather than asserting it"
+    and that "only one locked environment can do that". Neither was true. It now
+    states what each of the two checks establishes.
+11. **The final-bin closure is pinned.** Amendment 20 claims two deliberate
+    differences from the generator path; only the strict comparison was tested.
+    Deleting `cumulative[:, -1] = 1.0` left all fourteen stateless tests green.
+12. **The keyed-kernel contract test asserts the half it only asserted in
+    prose.** Its toy body never referenced `context.rng`, so "a keyed kernel
+    simply does not spend it" held by construction. It now deep-copies the bit
+    generator state across the call; adding `context.rng.random(1)` to the body
+    makes it fail.
+
+Each fix was mutation-tested: the mutation that breaks the behaviour makes the
+new assertion, and only it, go red.
+
+## Key findings (verified this session)
+
+- The integration branch carries **no** executor change for `KEYED` and **no**
+  `fit/kernels.py` change. `grep` over `origin/main`'s `executor.py` finds no
+  `seed_source` branch. Honouring `KEYED` therefore required no executor edit.
+- `QRFKernel.implementation_hash()` hashes `microcosm.fit.qrf`'s module bytes,
+  so editing `qrf.py` moved `fit.qrf@1`'s implementation hash
+  (`02db8f5c… → d1f8b192…`) and all three pinned platform node keys.
+- **Foreign-platform node keys are derivable here because the environments
+  agree — not because the fingerprint is the only channel.** A platform reaches
+  a key two ways: the fingerprint string, and the implementation hash, into
+  which `source_hash` folds `f"{distribution}=={version}"` for every declared
+  dependency. Re-deriving the three OLD pinned keys with the OLD implementation
+  hash substituted and only the fingerprint varied reproduces all three — which
+  establishes that each pin is the key its platform computed *under that hash*,
+  and establishes nothing about the dependency channel, because substituting
+  the hash is exactly what removes it. That channel is now checked separately
+  and explicitly. (The first pass wrote "so the platform reaches a key as that
+  string and nothing else"; the experiment could not support it.)
+- **`tools/graph_parity_fixtures.py` must not be edited.** `ParityCsvSource`
+  and `ParityRulesEngine` are defined there, so its bytes are inside
+  `ParityCsvSource.implementation_hash()` and
+  `SimulateRulesKernel.implementation_hash()`. Measured: adding the re-pin code
+  there moved the calibrate node key `184ccd0a → f8c9ed4b` with its
+  implementation hash unchanged. The re-pin logic therefore lives in a sibling
+  module. Reverted.
+- `generate()` resets `pins["platforms"]` to the local platform alone, so a bare
+  regeneration would have dropped both `x86_64/linux` pins and silently put H1
+  on its off-platform branch there (which asserts no bytes).
+- `direct.csv` is byte-identical before and after on every platform
+  (`7b8dbd56c91ee71552ff6d892a42c56494b1813fb5d4b11553a8a8ccc9b90dca`).
+- **`origin/main` moved during the lane** (to `e6d362b7e`, PR #909, object-dtype
+  storage hashing). It touches `graph/store.py` and `graph/population.py`;
+  neither enters `QRFKernel.implementation_hash()`. Merging is clean
+  (`git merge-tree`, no file touched by both), and running H1 parity, the pin
+  tests and the tolerance pin with main's post-#909 versions of those two files
+  in place is green, so main's advance does not disturb this lane's pins.
+
+## Open for decision (Max / the merge owner)
+
+1. **The spec-engine seed digests are stale on this branch, and that is
+   merge-blocking.** `tools/spec_engine_coverage.py --check` exits 1 (0 on
+   `origin/main`). The cause is `packages/microcosm-fit/src/microcosm/fit/qrf.py`
+   alone, bisected: `microcosm.fit.qrf` is in `_QRF_KERNEL_MODULES`
+   (`spec_engine/seeds.py:353-362`), whose `source_inventory_sha256` hashes each
+   module's exact installed source bytes. `SeedSource.KEYED`, `randomness.py`
+   and the export move none of it. **Seven tests are red**: one failure in
+   `test_spec_engine_inventory_coverage.py` and six fixture errors in
+   `test_spec_engine_coverage_tool.py`. Both files run in the **`engine-shared`**
+   CI lane, which has no `if:` condition — it runs on every PR and the aggregate
+   gate requires it — so this cannot be merged as it stands. (The integration
+   branch re-pinned these same two digests itself, in `1734b9e90`; its values
+   cannot be copied here because they also fold in its own `acs_transfer` and
+   housing changes.) The lane brief said
+   report, do not re-pin, so the branch carries the drift. The re-pin is two
+   pin values plus a regenerated evidence file; it was applied, verified green,
+   and reverted this session, so the recipe in the lane report is tested rather
+   than proposed.
+2. **Amendment numbering.** `main`'s list ends at 18, so 19 is the next free
+   number by its own arithmetic. The brief assigns 20 (the artifacts lane owns
+   19). Two unmerged commits on the
+   `candidate-quality-producer-integration-20260905` family already claim both:
+   `3ff92b0ae` = 19 ("Typed artifacts and stable draw coordinates", which
+   bundles this lane's subject), `d2043d85e` = 20 ("Failed typed evidence
+   remains a failed gate"). Neither is on `main` or on the integration branch.
+   Concrete consequence of leaving the gap: CommonMark renumbers an ordered
+   list from its first item, so the entry *renders* as 19 however it is
+   written — colliding with the number the artifacts lane expects.
+3. **`keyed_uniform` gives `-0.0` and `0.0` different draws** (verified:
+   `0.3464688…` vs `0.3251263…`), though they are `==` and hash-equal in Python.
+   `canonical_json` emits them distinctly and `_coordinate` does not normalise
+   the sign, so a float coordinate arriving as `-0.0` on one run re-randomises
+   that row. Not fixed here on purpose: `randomness.py` is byte-identical to the
+   integration branch that owns it, and `"sha256-u53-v1"` is a *versioned*
+   normative algorithm — changing which uniform a coordinate draws is a version
+   bump, not a drive-by fix. For that lane to decide.
+
+## Next
+
+- Nothing blocking in this lane's own work. A reviewer should decide (1), (2)
+  and (3) above; (1) must be resolved before the branch can go green in CI.

changelog.d/amend-keyed-seed-and-uniform-draws.added.md

@@ -0,0 +1 @@
+Graph interface amendment 20, keyed draw streams: `SeedSource.KEYED` declares that a kernel's randomness comes from normative stream parameters (`("sha256-u53-v1", experiment_id, replicate, base_seed)`) and one stable coordinate per draw rather than from a position in the executor's generator, and `microcosm.graph.randomness.keyed_uniform` computes those draws as the top 53 bits of a SHA-256 over the canonical stream and canonically tagged coordinates, reading and advancing no numpy RNG state. Order, chunking, and unrelated inserted identities cannot reach a draw. `FittedRegimeGatedQRF.predict_from_uniforms` is the first consumer surface: it draws from caller-supplied per-row uniforms, so pairing them with stable entity ids makes a batch invariant to recipient ordering and batching, and it advances no model RNG. `fit.qrf@1`'s existing outputs are unchanged — `predict` and its consumption order are untouched and the method is additive, and the kernel still declares `PARAM` or `EXECUTOR` — but its implementation hash moves with its module's source, so the H1 `fit.qrf` parity pins are re-recorded on all three pinned platforms while every `direct.csv` stays byte-identical. Adding an enum member rather than a normative field gains no node a projection field, so the member itself moves no node key — every kernel keeps projecting the seed source it already declared, and the `fit.qrf` keys move because its implementation hash moved. `tools/graph_parity_repin.py` re-pins one parity case in place, keeping every platform it carries instead of dropping the ones the local machine cannot run; it writes a key for a platform it is not only after refusing unless this environment's dependency versions are the ones the pins were taken under and every pinned key reproduces from the implementation hash recorded beside it.

changelog.d/keyed-draw-input-validation.fixed.md

@@ -0,0 +1,3 @@
+Reject temporal random coordinates before NumPy scalar conversion and reject
+non-real QRF uniform arrays before float conversion. Refuse inconsistent
+authoring keys when re-pinning graph parity fixtures.

docs/evidence/spec-engine/us-f0-coverage.json

@@ -1656,13 +1656,13 @@
           "compiler_ir.node_slices"
         ],
         "expected": {
-          "map_sha256": "87ba50531d9fa6683096ecb39a31655b331ab8acff3a5876c2f62b33562a0885",
-          "protocol_sha256": "fd3e4b06f11be4e8c13ea19fef9469ab95cbbe3e2dcfce351e860dd3e00709e4"
+          "map_sha256": "20058e544f6034cee2e76d3b864cf86b6230c8dce042931dfe9247d8ac1329c4",
+          "protocol_sha256": "553d5e0bd5afa93146dfbf8dca684926ee773dbfc89a559434528af5b4eb1d97"
         },
         "failures": [],
         "observed": {
-          "map_sha256": "87ba50531d9fa6683096ecb39a31655b331ab8acff3a5876c2f62b33562a0885",
-          "protocol_sha256": "fd3e4b06f11be4e8c13ea19fef9469ab95cbbe3e2dcfce351e860dd3e00709e4"
+          "map_sha256": "20058e544f6034cee2e76d3b864cf86b6230c8dce042931dfe9247d8ac1329c4",
+          "protocol_sha256": "553d5e0bd5afa93146dfbf8dca684926ee773dbfc89a559434528af5b4eb1d97"
         },
         "status": "covered"
       },
@@ -1677,7 +1677,7 @@
           "compiler_ir.seed_stream_map"
         ],
         "expected": {
-          "implementation_sha256": "fd3e4b06f11be4e8c13ea19fef9469ab95cbbe3e2dcfce351e860dd3e00709e4",
+          "implementation_sha256": "553d5e0bd5afa93146dfbf8dca684926ee773dbfc89a559434528af5b4eb1d97",
           "protocol": "legacy-v1",
           "streams": [
             "build_model",
@@ -1698,7 +1698,7 @@
         },
         "failures": [],
         "observed": {
-          "implementation_sha256": "fd3e4b06f11be4e8c13ea19fef9469ab95cbbe3e2dcfce351e860dd3e00709e4",
+          "implementation_sha256": "553d5e0bd5afa93146dfbf8dca684926ee773dbfc89a559434528af5b4eb1d97",
           "protocol": "legacy-v1",
           "streams": [
             "build_model",
@@ -2599,7 +2599,7 @@
       "country": "us",
       "schema_id": "country_spec",
       "schema_version": 1,
-      "spec_sha256": "35a02b6b19c921faba1407d441e0b9d9623c496e2cd5b711be014def281a95c6"
+      "spec_sha256": "1eeca53aa80da949a292fbd8cb0afefde95c68888ed35f3f477f3c962e6bc644"
     }
   },
   "report_schema_version": 3,
@@ -2609,7 +2609,7 @@
     "country": "us",
     "schema_id": "country_spec",
     "schema_version": 1,
-    "spec_sha256": "35a02b6b19c921faba1407d441e0b9d9623c496e2cd5b711be014def281a95c6"
+    "spec_sha256": "1eeca53aa80da949a292fbd8cb0afefde95c68888ed35f3f477f3c962e6bc644"
   },
   "status": "pass"
 }

docs/graph-acceptance.md

@@ -377,6 +377,42 @@ Amendments so far (each re-locked):
     Raised by the US launch integration branch
     (`microcosm-us-launch-integration-20260909`), which extended both
     frozen files without an amendment; extracted and adopted 2026-09-11.
+20. **Keyed draw streams.** `SeedSource.KEYED` and
+    `microcosm.graph.randomness.keyed_uniform`: a keyed kernel's draws are a
+    pure function of normative stream parameters — `("sha256-u53-v1",
+    experiment_id, replicate, base_seed)` — and one stable coordinate per
+    draw, conventionally `(person_id, process, period, draw_index)`, with a
+    float coordinate's signed zero normalised to one identity, as the
+    capabilities projection already does for tolerances. It
+    reads and advances no generator, so a row's draw stops depending on how
+    many rows were drawn before it: the invariance to packing that C1 and C2
+    already gave a node's key and seed now reaches each individual draw, and
+    an inserted or removed identity leaves every other row's value alone. C4
+    is neither weakened nor edited — `randomness.py` consumes no RNG at all,
+    positionally or otherwise, so its static check still holds over the
+    whole shard. `microcosm-fit` carries the first consumer surface,
+    `FittedRegimeGatedQRF.predict_from_uniforms`, which draws from
+    caller-supplied per-row uniforms; pairing those with stable entity ids
+    makes a batch's results invariant to recipient ordering and chunking.
+    **`fit.qrf@1`'s existing outputs are unchanged.** `predict`, its RNG
+    consumption order, and every value it draws are untouched; the new
+    method is additive, and the kernel still declares `PARAM` or `EXECUTOR`,
+    never `KEYED`. Its *implementation* identity moves all the same, because
+    `QRFKernel.implementation_hash()` hashes `microcosm.fit.qrf`'s module
+    bytes — so H1's `fit.qrf` pins are re-recorded on every pinned platform
+    while `direct.csv` stays byte-identical on each, and that byte-identity
+    is the evidence for the additivity claim rather than a restatement of
+    it. Unlike amendments 11 and 13 this adds an enum member, not a
+    normative field: no node's canonical projection gains a field, so the
+    member itself moves no node key — every kernel keeps projecting the seed
+    source it already declared. The `fit.qrf` keys move because the
+    implementation hash moved, not because `KEYED` exists. One deliberate
+    difference from the generator path, commented where it lives: the sign
+    gate's inverse CDF compares strictly and closes its final bin at 1.0,
+    so a uniform of exactly zero skips a zero-probability class instead of
+    selecting it, and a CDF that sums to just under one can no longer
+    silently select the first class. Raised by the US launch integration
+    branch, which carried the code without an amendment; adopted 2026-09-11.
 
 Adding a normative field with a default changes the canonical projection
 of every node that carries it, so node keys moved with amendments 11 and

docs/graph-interface.lock

@@ -1,2 +1,2 @@
 ed0a859adcae12510d5ba74d51c694617201f7b448b108a3f602410f5da44876  decl.py
-97c3ec9fc38aacb58f99e69c5a913ffca72c897cea4ca1002659ef06198b2b33  kernel.py
+dbf57c137330f0f12744c557ee594586a1b308b6d6adaba1938e2b6efded21ca  kernel.py