Consolidate the US survey graph build and enrichment pipeline

[MaxGhenis]

The US builder needs a reproducible path from original ACS and ASEC sources to one complete population, imputed inputs, calibration and verified analysis files. This draft consolidates that implementation with shared graph, atomic-geography, artifact/replay and calibration changes. It is an integration branch; its full completion is not a prerequisite for independent UK work.

The intended order, corrected on 10 September, is ACS + ASEC → complete initial multispine including support/PUF clones → assign one Census block per resulting household → derive larger geographies → remaining imputation and derivation → national/CD calibration → full and pruned exports. Distinct initial clones can receive different constrained draws, keyed by stable source identity plus a clone discriminator. Their locations then remain fixed. The corrected nine-node prefix now clones before block assignment, with nineteen-node financial and twelve-node age extensions. Its 70 inventoried controls plus a separate default-path compatibility check pass on invented inputs; the small native post-clone pilot did not complete final verification within its reviewed resource budget and remains unaccepted. Earlier native acceptance remains explicitly scoped to the prior order.

The PUF operator acts on the combined survey population. Social Security values and unknownness remain source-owned; the two PUF55 routes use nine predictors when the selected total is known and eight when it is unknown. Prior wages are excluded from the launch input profile. National and local exports must preserve the same constructed record values and relationships; calibration weights, scope and sparsity may vary.

Parallel review and dependencies

Reviewer	Useful review now	Work that should continue
Anthony	Shared graph/Frame and calibration compatibility; target hierarchy/diagnostics and dashboard metadata, including #855	Merged #900 clears target/dashboard changes; UK staging #896 can proceed independently
Maria	Register every existing UK pipeline stage as a graph node using this branch's machinery; verify UK behavior and coverage	With #900 merged, UK graph-node registration deliberately stacks on #893, followed by stage reordering in a separate PR; source/imputation and candidate validation continue
US integration owner	Resolve overlap, demonstrate compatibility, extract agreed shared prerequisites and complete the US build/release checks	US source preparation, imputation, quality diagnostics and progressive local builds

Coordination corrected after the 10 September team meeting: Maria intentionally stacks her UK graph-node PR on #893; pin its parent revision and coordinate shared-interface changes. Her #900 source/target reconciliation merged on 10 September at 14:54 UTC, clearing that dependency for Anthony's #855 hierarchy/diagnostics and dashboard work. Anthony's staging #896 proceeds independently. Anthony considers #885 superseded; it is reference material, not another runtime implementation to retain or merge. This does not claim the PR is closed or that he has approved #893.

The US integration owner will support Maria's graph with the shared atomic-area adapter, not build a competing UK graph. Keep UK graph-node registration and stage reordering as separate PRs. Record shared dependencies as specific changes and pinned revisions, extracting smaller prerequisites where useful while preserving Maria's intentional stack. Full native US PUF completion does not gate UK source/target repairs, staging or candidate validation. Full graph coverage in both countries remains a requirement for the broader advertised launch.

Verification and remaining work

This branch incorporates main through 6f7571e1ab8c516154289bd8ebf75cb2646f03b5, including Maria's merged #900 target/lookup reconciliation, UK CGT observation-period work #889 and PR CI scheduling #899. The integration preserves both parent histories and introduces exactly the upstream changed paths; dependency metadata parses and the updated worker lock digest matches. No dependency installation or new runtime acceptance is implied. The shared solver retains UK ordinary/informed behavior and the US grouped-bound path; unsupported combinations fail explicitly.

• The small native survey/block/financial pilot passed cold execution and required replay: 1,584 source households / 3,464 people became 3,168 / 6,928 after cloning. All 20 nodes replayed; four non-manifest exports matched cold bytes and the portable manifest retained its content. This evidence uses the earlier pre-clone geography order.
• Corrected post-clone geography preserves stable origin/clone draw identity, source weights, missingness and retained populations, with a typed geography validation dependency before financial enrichment. The 70 scoped controls are bound to the adopted source through exact selected-function/fixture AST evidence; maintained-module-path CI remains separate. Fable reviewed the production/test change, and root verified the applied hashes. Ten corrected artifact-reader controls also pass. The native nineteen-node runner returned and export readback ran, but final validation did not complete; the last sample recorded 7,216 CPU seconds against a 7,200-second cap and about 11.5 GB peak RSS. There is no final acceptance receipt. Repeated source/catalogue qualification is being optimized under separate controls; intermediate exports are not release candidates.
• All 28 focused PUF cross-store replay controls pass with independent source/resource verification. The comparison now uses directional Frame/ledger replay for independently decoded stores while retaining physical mutation fences for live objects. Two positive controls of the actual 245-node host now pass on a separately frozen source snapshot, using 64 invented donors, both Social Security routes, 110 fits and 110 applications, and complete cold/required runs. Its 18 adversarial controls are running separately; maintained adoption, native donor fitting and release remain unverified.
• The two-route numerical layer passed 46 invented controls with real QRF fitting/applications and whole-cohort-finalizer comparisons. Its receipt binds the finalized candidate and rejects mutation during encoding. All 47 canonical donor CREATE/converter controls passed, including cold execution and required replay. Independent source/resource checks passed; both source changes are included here. These controls do not qualify native donor/recipient fitting or the complete PUF attachment.
• Public recipient checks cover 28 distinct controls across the original 27 passing cases and a corrected five-case run with four overlaps, not one 28-case invocation. Earlier failed runs remain preserved. Donor-helper and full65/PUF55 cold/required chains have separate frozen-snapshot evidence.
• All 149 selected invented calibration controls passed on the combined US/UK solver snapshot, with independent source and unique-result checks. This verifies solver behavior, not native national/CD calibration.
• Changed-file Ruff, workflow actionlint and tracked test inventory checks passed on their identified revisions. Independent review scopes and corrections are recorded in the review guide. Current-head CI remains separate.

The CI diagnostic preserves missing event merge metadata as empty while requiring checkout/event equality and head/base identities. Forty-one environment-only controls were added; the bootstrap test step passed on a preceding commit, while spec-identity derivation still encounters restricted dependency probes. No source-access or bootstrap permissions were broadened.

Remaining release requirements include native execution/replay of the corrected geography order, current-head CI/spec identities, complete native PUF and other model-input coverage, a small full candidate, model/holdout quality, national/CD calibration, progressive scaling, full/pruned readback and a dashboard tied to the exact release file. No dataset release, default consumer promotion or deployment is claimed.

See the review and evidence index, shared US/UK release-path reference, historical native replay record, and companion site/methods draft #72. Typed-model #873 still needs reconciliation; #885 is retained as historical reference following Anthony's supersession decision.

11 September: continuation and CI repair (123650f)

Continued from the 10 September owner's last local state. Landed in order: the two verified but unpushed commits (survey replay-dependency binding; input-coverage profile/diagnostic, calibration-diagnostic option reconstruction, catalogue immutable memo, with their 25/27/20 controls); the uncommitted CI fixes (ASEC v3→v4 exact-source restoration helper carrying the Frame's mass log and metadata, plus its parametrized checkpoint regression; the two ASEC source test modules the coverage-authentication test imports; the strict-leaf audit stub's include_consumers keyword; the 21-node PUF55 recipient count after post-clone geography; the wheels gate installing microcosm-frame[us] after the base boundary check); and the incomplete native pilot observation.

Then the failures CI had not yet surfaced (collection errors had masked them):

• Runtime inventory: the consolidation added 113 graph-native modules the spine-blindness classification did not list. They are recorded from their docstrings in a separate reviewed set; the pool build tool's import graph is re-pinned 70→73 with the three newly reached modules added to the required set.
• Source-spine scan: every one of the 238 flagged sites was read. Seventeen modules that select or validate ACS/ASEC origin rows for the composed two-survey population (and the PUF55 routes' source-owned Social Security conditioning) join the reviewed provenance owners with their reason; twenty-one modules whose only findings were the fail-closed dynamic-selector class (tuple/state/bytes/profile-roster indexing, never a provenance column) are accepted for that class alone, with a regression proving accessor calls and contraband columns still fail.
• Spec pins: the seed protocol and compiled seed map digests are re-pinned; the drift traces to three attested kernel modules this branch changed (acs_transfer's column contract, housing_inputs' rent preparation, fit.qrf's predict_from_uniforms). Main is fully merged and spec_engine is unchanged.
• Seed-identity diagnostic (new on this branch): its boundary now pins the eight public parameter CSVs the US engine reads at import (RECORD-verified), tolerates psutil's /proc/stat layout read and urllib3's loopback IPv6 probe from _has_ipv6 only, and its kernel-module roster includes microcosm.calibrate.gates and .initialization, which the seed protocol has always listed.

Local verification on this head: test_us_spine_blindness.py complete file green; the seven test_spec_seed_* files green (122); the touched ASEC/ACS/PUF55 suites green; tools/spec_engine_coverage.py --check clean; ruff check/format clean. The diagnostic refuses non-Linux hosts, so its end-to-end proof is the CI job itself.

11 September, later: native pilot result, ASEC records memo, main merged (9fba910)

Native pilot v2 (guarded, local, 1/1000 sample, same seeds/estimator count/limits as 10 September, snapshot 123650f with the ACS catalogue memo): all nineteen nodes completed inside the caps for the first time (financial runner 4,744 s CPU; 5,615 s CPU / 5,665 s wall of the 7,200 / 10,800 limits; peak RSS 11.6 GB; no unexpected boundary refusals). The run then failed in the harness's own final seal (geography_pins == stored_geography_artifacts, status INCOMPLETE). Reproduced on the invented fixture: the harness pinned each geography stage with survey_atomic_geography._population_stamp, which is an in-process mutation seal — microcosm.graph.population._storage_parts hashes object-dtype storage (the strata axis) as raw ndarray.tobytes(), i.e. PyObject pointers, so two independent reconstructions of equal content (frame identity, mass ledger and owners all equal) never agree. Every in-repo use of that stamp compares the same objects, which is why the graph itself ran end to end; the harness helper now pins populations by content identity (_frame_identity plus version/owners/weight kind/mass ledger/design weights), verified equal back-to-back and across check_atomic_survey_financial_run on the fixture. Worth remembering repo-side: _population_stamp must never be persisted or compared across reconstructions.

ASEC records memo (c2c6b37): the pilot's retained phase transitions attributed about a fifth of its CPU to asec_population_catalogue._records_identity, which every checked borrow of the catalogue re-ran twice and the preparation's nested seal a third time (about 2 s per pass on the real catalogue). The catalogue now binds a leaf snapshot to the identity in the same streaming pass at issuance; later borrows reuse the identity only when the issued root tuples are the same objects, the memo identity equals the owner's retained one, and every projected leaf (literals, keys, person rosters and members' fields) is the very object seen while streaming. Frozen dataclasses stay writable through object.__setattr__, so the leaves carry the proof; subclass leaves, replaced containers or foreign types keep the full canonical encoding and its refusals. Seventeen controls in test_us_asec_catalogue_records_memo.py, including real issuance: zero identity recomputations per borrow, RECORDS_CHANGED and NESTED_EVIDENCE_CHANGED on an in-place leaf change. The module's inventory unbound_uses digest is re-pinned (only new uses: survey_population_domains type checks in _eligible_key/_eligible_leaves; no new imports or resource accesses). The larger cost — each checked borrow re-verifying the whole ancestor chain, including asec_current_money_source._frame_signature — is structural and left as is.

Main merged (9fba910; #855 schema-7 geography dimensions, #902 state tax metadata). The only conflict was microcosm-calibrate/__init__.py, resolved by keeping both import blocks. tools/spec_engine_coverage.py --check and tools/ci_test_groups.py --verify are clean on the merged tree.

Native pilot v3 (snapshot 9fba910, same sample/seeds/limits, helper pin fix) is running locally under the guard; its receipt will be recorded here when it completes, accepted or not.

11 September, evening: schema-8 hierarchies, a sealed-module test seam, memo hardening (2ca11c8)

• Calibration hierarchy on every national demographic count target (d872960): main's diagnostics schema 8 requires a complete hierarchy for every registry-backed target, and national.demographic_calibration failed on the merge ref with none. Each activated S0101/B01001 cell now carries the only hierarchy it may have (provider census_acs; category census_acs.population_by_age / population_by_sex_and_age, both labels added to the reviewed US variable-label catalog; the activation's own national publisher row; one publisher_cell dimension; the cell as target). Both spec builders attach it, the registry declaration round-trips it, the validator refuses anything but the hierarchy rebuilt from table, name, label and geography, and the survey diagnostics verifier's defensive pin moves 6→8.
• Terminal-manifest mutation controls (72fbe7b): a pre-existing failure on this branch, reproduced on the previous head in a fresh worktree — survey_origin_budget seals its own functions, so the tests' monkeypatch of verify_survey_weight_only_successor refused with PRODUCER_CHANGED before the seven-node graph ran. The seam is now injected through the stage's budgets alias; the sealed module is untouched and the mutation lands where it did before.
• ASEC records memo hardening (2ca11c8), from an independent Opus read of c2c6b37: the Source.ASEC member's value object and every record/key/person class are now projected leaves, so _value_ tampering and __class__ reassignment miss the memo and reach the full path's refusals; two regressions.
• main is fully merged (Add native SPM role source enrichment and release verification #894 included); spec_engine_coverage --check, ci_test_groups --verify, ruff check clean.

11 September, 14:52 ET: native pilot v4 accepted (snapshot 2ca11c8)

Guarded local rerun of the 1/1000 native post-clone nineteen-node pilot on this head (same sample, seeds 20260908, n_estimators=2, demographic conditioning, limits 7,200 s CPU / 10,800 s wall / 32 GB): passed — guard result 0, 1 collected, outcomes ["passed"]; 5,279 s CPU, 5,362 s wall, peak RSS 9.3 GB; source, resource, owned and native pins unchanged; 23 known boundary refusals, none unexpected; native-financial-manifest.json status ACTUAL_ARTIFACTS_RECORDED, release_eligible: false, guard_completion_required: true, cross_process_source_authority_granted: false. Entity rows: 1,584 households / 3,464 people at create and raw, 3,168 / 6,928 after expansion and through geography and financial. Nine prefix nodes (175 s wall) and nineteen extension nodes (968 s wall; the three predictor sources ≈ 260–300 s each); financial runner 4,686 s wall including the requalifying prefix borrows; export read-back 211 s; final seals passed. Compared with v2 (snapshot 123650f): 336 s less CPU with the ASEC records memo, and the harness's cross-reconstruction geography pin now holds. A v3 attempt on 9fba910 died at 65 min with no receipt or refusal (no memory pressure, no app restart; treated as an external process kill), so this run is the first accepted receipt. Evidence: ~/PolicyEngine/_recovered/scratch-backup/893/pilot-v4-20260911/ (receipt, manifest, resource progress, exports). Development scope only; no release certification claimed.

11 September, 15:10 ET: main #903 merged, two identities re-pinned (77c9e78)

Main's #903 (UK household composition, frs_relationships spine stage) is merged; the only conflict was the UK country-bundle digest in test_spec_engine_country_bundles.py, re-pinned to the value the merged tree computes (neither side's pin survives a merge that changes attested modules on both sides). The engine-us us-am failure on 2ca11c8 was the multispine pool tool's constants-adapter country_spec digest, still asserted at main's value while docs/evidence/spec-engine/us-f0-coverage.json already carried this branch's attested value from the seed re-pin; the test now asserts that value. _population_stamp in survey_atomic_geography.py documents that it is an in-process seal (see #907 / #909 for the repo-side fix on main).