docs/plans: graph-port plans (full rev 3; shadow-only Plan A rev 4, gate-blocked)

docs/plans/graph-port-2026-09-03.md

@@ -0,0 +1,84 @@
+# Graph-native rearchitecture of the covered-earnings campaign
+
+Status: PROPOSED 2026-09-03, revision 3, for the main/peer agreement gate (Fable main + Sol peer; gate 20260903-093201-plan-342fa24d). Revision 1 received REQUEST CHANGES on all seven questions (round 2); revision 2 folded them verbatim and received six further findings (round 3), all folded below. Max's directive of 2026-09-03 08:1x EDT: "lets rearchitect before more lanes unless we can parallelize. i dont know enough to answer these qs, defer to you+sol."
+
+This plan decides the questions the two 2026-09-03 reviews left open and sequences the port of the campaign's evidence arms onto `microcosm-graph` (PolicyEngine/microcosm#836, merged 2026-09-02). It changes no gate threshold, no ledger entry, no design text, no lane brief and no harvest rule until the step that says so. Claims about the graph were checked against the code snapshot of `origin/main` by an adversarial refuter and corrected by the peer in rounds 2 and 3.
+
+## 1. Answer to "can we parallelize"
+
+Yes. The port needs no sol-lane hours, no brief or template change, and no change to `#405`, `e20-amend20`, `docs/design/` or the registry before the revision-22 repin. The purpose census (final tranche R17889–R20815, ten blocks), the in-flight Q5 pairs and the A4/C20 chain continue on the bespoke path, with one exception the peer required: a C20-only correction of the ceremony kit (step 1a) before any freeze. Capacity policy:
+
+Preserve the standing fuel rule: reserve `~/.codex-6` and first-call reset capacity for the remaining purpose work, one A4 draft, two independent C20 referees, and one retry margin. Route Q5 to `~/.codex-4` and newly added accounts, requesting additional campaign capacity before the active home caps. If capacity is not supplied, Q5 may idle only as an explicit fallback whose approximately 79-hour delay is recorded. When no codex home accepts sessions, held campaign briefs may run on Claude Opus under the ledger's engine-assumptions clause, engine-tagged and measured, recorded on the board at dispatch. Graph-port build agents use separate Claude capacity and pause if harvest, A4/C20, or referee capacity is threatened.
+
+## 2. Decisions (recommended answers; the gate's agreement adopts them)
+
+D1. Port only the per-document Q5 harvest/transport pipeline mid-flight under the source model, shadow first. Lane and reconcile reports are byte-hashed sources, never kernel outputs; deterministic post-lane transformations are kernels. Name the release `q5.harvest_seal@1` and mark every shadow manifest NONAUTHORITY: it certifies transport and seal-record integrity only and does not instantiate §19 Q5, satisfy §19.3.3 deep equality, perform the unique Q5 Git first-add, or instantiate A20 lifecycle row 14. Switch only after the immutable-through-switch cohort defined in step 5 passes complete parity with zero discrepancies or formally signed typed exceptions.
+
+D2. After a real revision-22 repin, graph runs compute and certify candidate evidence only; graph node keys, receipts, tiers, fail, and unreached are never A20 identities, lifecycle statuses, nonemission proofs, or repository first-adds. For every stage, a fail-closed design controller authenticates the exact predecessor and ordered input identities, validates the certified graph manifest with the campaign verifier of D8, derives `output_identity_id` from the enacted canonical five-member preimage, and atomically first-adds the canonical output and pass transition only when every design condition passes. Every lifecycle record carries exactly `lifecycle_stage_id`, `schema_id`, `predecessor_stage_ids`, `input_identity_ids`, `output_identity_id`, `first_add_index`, `selection_enabled`, and `status`. A missing or nonpassing predecessor records `blocked_predecessor` with null output and dispatches no descendant; any other failure records `fail_atomic_nonemission` with null output, no canonical first-add, and no descendant. If these semantics cannot be demonstrated exactly, retain the design-pinned bespoke controller and use the graph only for computation and provenance. A21 remains closed.
+
+D3. Only after milestone B's publication merge, stage `ss.oracle@1`, anchors, the candidate protocol, and a floor-only build that scores no candidate. Benefit/AIME/PIA cells may be proposed; the 14 circular claim-age cells remain report-only unless a non-circular candidate protocol is separately ratified. Before any candidate run, freeze sources, scored surface, baselines, and thresholds. Complete public proposal → adversarial review → fixes → independent verification → maintainer ratification by merge, creating a fresh `locked: true`, `status: locked` top-level gate. No candidate is run or scored before the locked gate exists; only a fresh, newly registered post-ratification run may score. The gate neither grades nor changes ledger entry 22 or milestone B; register a new forecast only for its own downstream deliverable if needed.
+
+D4. Place the store outside e8-ops in a dedicated private location. Treat copied source payloads as NONAUTHORITY caches and preserve each enacted canonical path and storage identity. Before selecting ordinary Git, inventory source bytes plus every column, node-key duplicate, attempt, metadata object, and projected growth; check source-by-source access, licensing, storage, and repository binary limits. Git-track only manifests and small objects unless the measured corpus passes that policy; otherwise use approved Git LFS or private object storage. Name the backup owner and off-host replica, and require a clean-store restore-and-rehash drill before acceptance. No capacity ceiling is asserted until measured (the 81 existing `annotation_v1` JSON files alone total 686,695,310 bytes).
+
+D5. Fable owns design, parity criteria, and review; isolated build agents own kernel and ingestion work. Sol continues the running campaign and performs parity review only from surplus capacity. The port must neither cancel account additions nor consume the milestone-A reserve.
+
+D6. Dynamics reviews an exact #847 commit SHA and its complete amendment set before merge. Any accepted change requires a fresh upstream interface lock; after merge, Dynamics re-snapshots merged main and verifies `decl.py` and `kernel.py` against `graph-interface.lock`. Immediate Q5/A3/purpose exact-byte graphs need no semantic use of amendments 11–14. Entrants, mass partition, and entrant strata remain required only for graphs opting into those semantics; `ss.oracle@1`, `benefit.level_gate@1`, and any tolerance-bound successor kernel require Amendment 13 and may not be declared until #847 merges and the lock is re-recorded.
+
+D7. Certification trusted computing base. The Q5 execution and any cutover are pinned to a complete, immutable dependency surface: the exact `microcosm-graph` and `microcosm-frame` commits, every Python dependency version, the interpreter, the campaign's kernels and codecs, and the verifier of D8, recorded as a TCB manifest whose digest appears in every campaign manifest. Because #847 changes normative defaults, node keys, the manifest schema and certification behavior, the cutover requires either the reviewed merged #847 revision or an explicitly frozen, vetted pre-#847 implementation; in either case the entire parity cohort is rerun under the final bytes before step 5, and step 2 acceptance is mandatory before step 5.
+
+D8. Decision authority. Kernels never receive Decision records, and release processing matches decision names only after kernels execute, so no ancestral gate can authenticate a ruling. Therefore Decision records are nonselecting provenance. Selection of the active canonical record is carried in authenticated, source-derived owned columns (the commit-pinned ruling bytes are declared sources; a gate derives the disposition from those bytes), and an external fail-closed campaign verifier checks every certified manifest for the expected graph identity, release node, document, exact gate set, active SHA, and pass-only gate outcomes (rejecting empty ancestry and `not_applicable` as certifying), against immutable per-ruling bytes, with substitution and tampering tests in its suite.
+
+## 3. Interface facts the build relies on (origin/main as of 2026-09-03; corrected by the peer; superseded by the TCB pin of D7 once recorded)
+
+- Any node may declare sources; CREATE must. A regular-file source identity hashes its name, exact bytes, and size path-independently. All supplied node parameters are normative, including engine/model ID and attempt index.
+- Determinism is declared, not verified. `auto` and `require` reuse a valid result for an unchanged key; `forbid` re-executes and may displace the stored object. Campaign policy keeps LLM calls out of kernels and supplies reports as source bytes; deterministic rules-engine wrappers remain supported by the interface.
+- Opaque artifacts are terminal within a run. Downstream contexts receive declared column slices plus structural IDs/membership, weights, strata, parameters, RNG, and declared sources. Same-run T evidence required by a seal gate must be materialized as owned string/digest columns.
+- In the campaign's flat post-CREATE graphs, every gate owns a column in release ancestry. Structural nodes can create additional ancestry. Empty gate ancestry certifies vacuously, so every campaign release requires at least one ancestral gate and nonempty `requires_decisions`, and the D8 verifier independently rejects empty or `not_applicable` ancestry.
+- A portable manifest contains at most one RELEASE; zero is allowed. Cross-run dependencies are declared sources. A manifest source remains bytes unless campaign code explicitly validates it with the D8 verifier.
+- Main ships `frame-store` and `csv-tables` codecs; every additional Markdown, JSON, PDF, text, or manifest codec used by the campaign must be explicitly registered and tested. Non-CREATE source handling verifies registration and content identity but does not invoke the loader automatically.
+- Campaign CREATE frames use occurrence or rank as the person entity, document as a group entity, required ID/membership structural columns, unit design weights on one entity, and Owned declarations for every nonstructural column. Merged main has no `mass_partition` field; immediate graphs omit it and later adopt only the exact merged interface.
+- Graph output keys derive from producer node keys and output coordinates, so equal payloads under distinct node keys remain distinct objects.
+- Decisions are unauthenticated manifest provenance records matched only by name/kind; they select nothing (D8).
+
+## 4. Sequence, owners and acceptance
+
+| Step | What | When | Owner | Accepted when |
+|---|---|---|---|---|
+| 1 | Finish purpose census and the A4 freeze / C20 chain on the bespoke path; no brief/template/harvest/fold change | now → repin | orchestrator + lanes | contiguous R1–R20815; step 1a accepted; a4_readiness green under the corrected script; revision-22 repin |
+| 1a | C20-only correction of the ceremony kit before any freeze: generate the complete ceremony from one pinned commit; the v2 receipt is a fixed tracked mode-100644 file first-added by closure per §34.8; arm and successor identities use their distinct seven-field schemas; `--commit` design bytes are never mixed with live-worktree validator code; referee B is fully specified; `a4_readiness` validates the completed purpose census and instantiated identities and exits nonzero on any pending check; a fail-closed freeze/receipt/closure validator is added | before the freeze | orchestrator (kit owner) + Sol review | corrected kit generated from one pinned commit; validator passes a dry run; Sol review AGREE |
+| 2 | Establish the store per D4; record the TCB manifest per D7; register codecs; ingest every governing byte as a NONAUTHORITY source cache (index, rubric, every ledger and library version, seals-v2, every staged report, every brief, Q5 artifact JSONs and PDFs) and the exact commit-pinned board ruling bytes used by every seal, void, supersession, replication, operator-merge, and switch decision | days 0–3 | this session + build agents | every ingested source re-hashes to the pinned SHA cited in the campaign record; canonical paths and storage identities preserved; nothing in e8-ops changed; restore-and-rehash drill passed; TCB manifest recorded. MANDATORY before step 5 |
+| 3 | Q5 harvest/transport subgraph (CREATE occurrence universe; `q5.decode_lane@1`; `q5.transport_gate@1`; `q5.reconstruct_T@1`; `q5.seal_gate@1`; `q5.harvest_seal@1` RELEASE) with a parity harness | days 1–6 | build agents; this session reviews | For each cohort document: two independent cold executions in separate processes and separate stores with byte and object-digest equality, then an auto-resume warm-cache round-trip; compare exact source bytes, ordered decoded rows/counts/digests, transport and seal outcomes and complete evidence, reconstructed T bytes/count/digest, release ancestry/tier/outcome, manifest round-trip, and the D8 verifier result. A source-consumption matrix proves every governing source is declared and consumed by the intended nodes, and one-byte mutation tests prove each governing source changes or invalidates the intended descendant identities. `q5.decode_lane@1` and `q5.reconstruct_T@1` must be total over declared bytes and emit typed status/evidence columns, or all fallible parsing must occur inside GATE kernels. Malformed, truncated, noncanonical, wrong-sentinel, wrong-count, and wrong-digest fixtures must persist a gate fail rather than abort. |
+| 4 | Backfill the historical documents, including every void, supersession, and replication disposition | days 4–8 | build agents; this session reviews | The step-3 contract passes for every historical document; any residue blocks cutover for that document. Every document has exactly one active canonical record selected by authenticated source-derived columns (D8); all superseded, void, and replication-only records remain byte-reachable through authenticated disposition edges. For doc_058, active SHA is 38eea5db57118df598e2f2ea1d68466df58d9e1ff1615de38d36c4609da2fef3 and preserved superseded SHA is b3a36967ffdc934521c5459f7bf71b46cb5741f2a089a4a89b5d8d750de89fc8; filenames confer no authority. Decision records select nothing. |
+| 5 | Harvest switch with an atomic authority handoff | after steps 2, 3 and 4 pass under the final TCB bytes and the agreement gate records the switch | orchestrator | Cohort is immutable through the switch: a high-water mark is declared; every document completed before the barrier is shadowed (catch-up for intervening documents); dispatches drain; a write fence stops bespoke seal records; a single-writer handoff is tested for no gaps, duplicates, or competing active records; a rollback trigger and a post-switch soak period are defined; zero discrepancies or formally signed typed exceptions only; from then on the board's append-only record may cite the graph manifest; lanes, briefs, templates and ladder law unchanged; harvest latency not worse than the measured 0.9–1.9 min |
+| 6 | Dynamics review of #847 per D6; TCB pin per D7 | days 3–10 | this session (+ microcosm-graph owner) | Review names the exact #847 commit and complete amendment set; requested changes are reflected and re-locked upstream; the post-merge snapshot hashes match merged-main `decl.py` and `kernel.py`; TCB manifest updated and the parity cohort rerun |
+| 7 | Backfill A3 (37 library-pass blocks, 37 audits, 26 patch files, 28 finals, 31 library folds) and, after the census closes, the purpose arm | days 6–12 | build agents | For each A3/purpose unit, exact source identities, ordered rows and domain digests, audit verdicts, patch/final ancestry, library-fold decisions, and active/superseded/void dispositions equal the bespoke record; any residue blocks retirement only for that unit |
+| 8 | Successor construction per D2: graph runs for computation and provenance under a fail-closed design controller | post-repin | build agents; this session reviews | All 26 ordered pass vectors reproduce their complete identity and envelope preimages; every stage also has forced-failure, blocked-predecessor, no-descendant, restart/crash-atomicity, and ordered Git first-add tests. A19, row 2, and R05 spot checks are necessary but not sufficient. |
+| 9 | Milestone B close-out on the graph; entry 23 only if its unresolved forecast changes | ~2026-10-21 p50 | this session + Max (merge) | publication PR merged; the design validator passes and the D8 verifier accepts the publication manifest |
+| 10 | Stage `ss.oracle@1` and `benefit.level_gate@1` and complete the prospective lock ceremony described in D3 | only after step 9 publication merge | this session + independent referee/verifier | floor-only build passed; amendment reviewed, fixed, independently verified and ratified; frozen lock present; proof that no candidate was run or scored before the locked gate existed and that only a fresh, newly registered post-ratification run was scored |
+| 11 | Append a canonical manifest-pointer index only for units with complete parity; never rewrite or delete seals-v2 artifacts or historical board lines | per unit after complete step 4 or step 7 parity | orchestrator | every pointer resolves to one certified manifest and one active canonical record selected per D8; every predecessor remains byte-reachable; each non-active record has an authenticated superseded, void, or replication_only disposition; failed or residue units retain the historical record as sole authority |
+
+Effort remains ESTIMATED (no kernel-authoring baseline exists): dynamics 20–34 person-days plus the D7/D8 verifier and step 1a correction (unpriced), of which 8–14 are the successor build the ledger already prices; microcosm-graph 3–9.
+
+## 5. Invariants and do-nots
+
+- Do not pause, re-brief or re-dispatch the census, the Q5 pairs or the A4/C20 chain to fit the graph; the only permitted pre-freeze change is the C20-only correction of step 1a.
+- Do not wrap a sol lane as a kernel or let any kernel call an engine.
+- Do not move gates.yaml thresholds into node params; declare the yaml as a source of each gate node.
+- Do not add graph code, stores or kernels to `#405`, `e20-amend20`, `docs/design/` or the registry before the revision-22 repin.
+- Do not use a node key or tier as an A20 identity or status.
+- Do not treat a Decision record, receipt, graph tier/outcome, or per-document manifest as authenticated authority, a design identity/status, §19 Q5 passage, or first-add; Decision records select nothing.
+- Do not cut over on any bytes other than the recorded TCB, and do not cut over before step 2 acceptance and a full-cohort rerun under those bytes.
+- Do not use `resume=forbid` against the shared campaign store; cold re-execution uses a new isolated store.
+- Do not edit ledger entries, gates.yaml, committed `runs/` artifacts or the ledger/library files; append only.
+- Do not put the store in e8-ops or rely on `/tmp` lane work dirs.
+- Do not build a second registry beside the graph.
+- Do not declare `Tolerance` or entrants before #847 merges and the lock is re-recorded.
+- Do not retire, rewrite or delete any old record; pointer indexes are appended only after complete per-unit parity, and failed or residue units keep the historical record as sole authority.
+- Do not hand-repair anything that fails a gate during backfill.
+- Do not cancel or defer account additions on the port's account; do not consume the milestone-A reserve.
+- Do not run or score any benefit-gate candidate before the locked gate exists.
+- Do not report effort or date effects as measured.
+
+## 6. What the gate is asked to agree to
+
+Agreement adopts D1–D8 and authorizes steps 1a, 2, 3, 4, 6 and 7 to start immediately in parallel with the running campaign, step 5 on its acceptance criteria and a recorded switch decision, and steps 8–10 after the revision-22 repin. Requested changes are folded and the gate re-run with the new fingerprint.