Consolidate the US survey graph build and enrichment pipeline

.gitattributes

@@ -0,0 +1,2 @@
+# Preserve exact unified-diff context markers in the archived measured experiment.
+experiments/us-acs-code-memo-not-adopted-20260910.patch -whitespace

.github/workflows/test.yml

@@ -207,7 +207,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        python-version: ["3.13", "3.14"]
+        python-version: ["3.13", "3.14.4"]
     steps:
       - uses: actions/checkout@v4
       - uses: astral-sh/setup-uv@v6
@@ -220,6 +220,70 @@ jobs:
           mapfile -t files < <(uv run --no-sync python tools/ci_test_groups.py --list shared-spec)
           /usr/bin/time -v uv run --no-sync pytest "${files[@]}" -p no:cacheprovider
 
+  spec-seed-diagnostics:
+    # Produce bounded diagnosis immediately, independently of the full suite.
+    # This remains required by ci-ok and never updates assertion goldens.
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: astral-sh/setup-uv@v6
+        with:
+          python-version: "3.14.4"
+      - name: Sync workspace with US and UK engines
+        run: uv sync --all-packages --locked --extra us --extra uk
+      - name: Check diagnostic bootstrap and failure reporting
+        timeout-minutes: 2
+        env:
+          PYTHONDONTWRITEBYTECODE: "1"
+          PYTEST_DISABLE_PLUGIN_AUTOLOAD: "1"
+        run: >-
+          uv run --no-sync pytest --noconftest -p no:cacheprovider -q
+          packages/microcosm-build/tests/test_spec_seed_identity_cpu_bootstrap.py
+          packages/microcosm-build/tests/test_spec_seed_identity_outer_failure.py
+          packages/microcosm-build/tests/test_spec_seed_identity_owned_temp.py
+          packages/microcosm-build/tests/test_spec_seed_identity_system_metadata.py
+          packages/microcosm-build/tests/test_spec_seed_identity_per_code_context.py
+      - name: Derive candidate spec and seed identities (diagnostic only)
+        id: spec_seed_diagnostics
+        timeout-minutes: 15
+        env:
+          DIAG_WORKFLOW_SHA: ${{ github.workflow_sha }}
+          DIAG_PR_HEAD_SHA: ${{ github.event.pull_request.head.sha }}
+          DIAG_PR_BASE_SHA: ${{ github.event.pull_request.base.sha }}
+          DIAG_MERGE_SHA: ${{ github.event.pull_request.merge_commit_sha }}
+          PYTHONDONTWRITEBYTECODE: "1"
+          PYTEST_DISABLE_PLUGIN_AUTOLOAD: "1"
+          OMP_NUM_THREADS: "1"
+          OPENBLAS_NUM_THREADS: "1"
+          MKL_NUM_THREADS: "1"
+          VECLIB_MAXIMUM_THREADS: "1"
+          NUMEXPR_NUM_THREADS: "1"
+          BLIS_NUM_THREADS: "1"
+          POPULACE_FIT_N_JOBS: "1"
+          POPULACE_FIT_PREDICT_WORKERS: "1"
+        run: |
+          DIAG_CHECKOUT_SHA="$(git rev-parse --verify HEAD)"
+          DIAG_UV_VERSION="$(uv --version)"
+          export DIAG_CHECKOUT_SHA DIAG_UV_VERSION
+          uv run --no-sync python tools/spec_seed_identity_diagnostics.py \
+            --output-dir "$RUNNER_TEMP/spec-seed-diagnostics"
+      - name: Retain bounded candidate identity evidence
+        if: ${{ always() && steps.spec_seed_diagnostics.outcome != 'skipped' }}
+        uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
+        with:
+          name: spec-seed-diagnostics-${{ github.run_id }}-${{ github.run_attempt }}-py3.14.4
+          path: |
+            ${{ runner.temp }}/spec-seed-diagnostics/candidate-digests.json
+            ${{ runner.temp }}/spec-seed-diagnostics/seed-protocol.json
+            ${{ runner.temp }}/spec-seed-diagnostics/seed-map.json
+            ${{ runner.temp }}/spec-seed-diagnostics/seed-bindings.json
+            ${{ runner.temp }}/spec-seed-diagnostics/environment-and-source.json
+            ${{ runner.temp }}/spec-seed-diagnostics/diagnostic-status.json
+          if-no-files-found: error
+          retention-days: 3
+          compression-level: 0
+          include-hidden-files: false
+
   engine-us:
     needs: changes
     if: github.event_name == 'push' || needs.changes.outputs.us == 'true' || needs.changes.outputs.shared == 'true'
@@ -336,6 +400,43 @@ jobs:
           PY
       - name: Spec identity section digests (cross-environment diffing aid)
         run: env -u PYTHONPATH /tmp/wheels-venv/bin/python -I tools/spec_envelope_digests.py be uk
+      - name: Add the engine-free US extra the source runtimes require
+        # microunit is microcosm-frame's `us` extra: the sanctioned tax-unit
+        # constructor, and the only optional dependency the US source runtimes
+        # need without a country engine. Every ACS/CREATE stage names it in
+        # us_runtime.graph_implementation.STAGE_DEPENDENCIES, so an installed
+        # constellation without it cannot hash those stages: the suite then
+        # fails closed (SOURCE_PREPARATION_REFUSED / CATALOGUE_ISSUANCE_REFUSED)
+        # instead of skipping, which is a false packaging verdict.
+        #
+        # This runs after the boundary check above, so that check still proves
+        # the base wheels import with no optional dependency installed at all.
+        # Requesting the extra against the built wheel — rather than naming
+        # `microunit` here — keeps the extra declared only in
+        # packages/microcosm-frame/pyproject.toml and makes this gate prove the
+        # wheel carries resolvable extra metadata. policyengine-us stays absent:
+        # it is declared by microcosm-build[us] and microcosm-frame[policyengine],
+        # and neither is requested.
+        run: |
+          frame_wheel="$(ls dist/microcosm_frame-*.whl)"
+          uv pip install --python /tmp/wheels-venv/bin/python \
+            --constraint /tmp/wheel-constraints.txt \
+            "microcosm-frame[us] @ file://$(realpath "$frame_wheel")"
+          env -u PYTHONPATH /tmp/wheels-venv/bin/python -I - <<'PY'
+          from importlib import metadata, util
+
+          assert util.find_spec("microunit") is not None, (
+              "microcosm-frame[us] must install microunit into the wheel venv"
+          )
+          assert util.find_spec("policyengine_us") is None
+          try:
+              metadata.distribution("policyengine-us")
+          except metadata.PackageNotFoundError:
+              pass
+          else:
+              raise AssertionError("policyengine-us must be absent from the base wheel gate")
+          print("microunit", metadata.version("microunit"))
+          PY
       - name: Run the suite against the installed wheels
         run: |
           /usr/bin/time -v env -u PYTHONPATH /tmp/wheels-venv/bin/python -I -m pytest \
@@ -350,7 +451,7 @@ jobs:
     # dependency-skip lets a failed `changes` take the engine lanes down without
     # any required check going red: a lane skip is accepted only when
     # classification itself succeeded.
-    needs: [changes, lint, fast, engine-shared, engine-us, engine-uk, wheels]
+    needs: [changes, lint, fast, engine-shared, spec-seed-diagnostics, engine-us, engine-uk, wheels]
     if: always()
     runs-on: ubuntu-latest
     steps:
@@ -365,6 +466,7 @@ jobs:
           require_success lint           "${{ needs.lint.result }}"
           require_success fast           "${{ needs.fast.result }}"
           require_success engine-shared  "${{ needs['engine-shared'].result }}"
+          require_success spec-seed-diagnostics "${{ needs['spec-seed-diagnostics'].result }}"
           require_success wheels         "${{ needs.wheels.result }}"
           for lane in "engine-us:${{ needs['engine-us'].result }}" "engine-uk:${{ needs['engine-uk'].result }}"; do
             name="${lane%%:*}"; result="${lane#*:}"

CLAUDE.md

@@ -39,6 +39,12 @@ shard's real wheel, install into a clean uv-export-constrained venv, assert
 the wheel/import boundary and spec digests, and run the suite against installed
 wheels.
 
+The shared engine lane also runs a bounded source/seed identity diagnostic on
+Python 3.14.4 after its normal tests, including when those tests fail. Its six
+JSON artifacts retain candidate digests and their complete canonical seed
+records for review. They never replace the test assertions or certify coverage;
+review the captured source and dependency identities before updating any pins.
+
 New commits to a PR cancel older unfinished CI runs for that same PR.
 Each main-push run has a unique concurrency group, so all main-push runs
 remain independent and can finish validating their merged changes.

PROGRESS.md

@@ -726,3 +726,65 @@ The still-earlier PolicyEngine-US 1.819.0 lock-bump lane merged into
 `origin/main` at `7b90bb18` on 2026-08-24; its final state remains at commit
 `05d254aa` and its detailed receipts remain in the historical section of
 `_LANE-NOTES.md`.
+
+## US launch integration staging — 2026-09-09
+
+### State
+Source-only staging in progress. Execution and source/data admission remain root-owned.
+
+### Done
+Verified requested clean branch, base HEAD, main ancestry and preservation pins.
+
+### Next
+Apply thirteen explicit source layers, commit each, then separately review isolated ordinary execution. Existing journal history above is retained.
+
+Layer 1: SAFE-ADDITIVE.patch applied; all declared postimages and preservation hashes verified. No tests executed.
+
+Layer 2: GRAPH-RESTORE.patch applied; all declared postimages and preservation hashes verified. No tests executed.
+
+Layer 3: ACCEPTED-SHARED-RESTORE.patch applied; all declared postimages and preservation hashes verified. No tests executed.
+
+Layer 4: PUF-SUPPORT-MERGE.patch applied; all declared postimages and preservation hashes verified. No tests executed.
+
+Layer 5: SOLVE-MERGE-PROPOSAL.patch applied; all declared postimages and preservation hashes verified. No tests executed.
+
+Layer 6: J-GRAPH-COMPATIBILITY.patch applied; all declared postimages and preservation hashes verified. No tests executed.
+
+Layer 7: F-CATALOGUE-OPTIMIZATION.patch applied; all declared postimages and preservation hashes verified. No tests executed.
+
+Layer 8: GRAPH-ATTACHMENT-METADATA.patch applied; all declared postimages and preservation hashes verified. No tests executed.
+
+Layer 9: F-JOINT-GEOGRAPHY-GATE.patch applied; all declared postimages and preservation hashes verified. No tests executed.
+
+Layer 10: SOURCE-CLOSURE.patch applied; all declared postimages and preservation hashes verified. No tests executed.
+
+Layer 11: PLACEMENT-ADDITIONS.patch applied; all declared postimages and preservation hashes verified. No tests executed.
+
+Layer 12: ORDINARY-CLOSURE.patch applied; all declared postimages and preservation hashes verified. No tests executed.
+
+Layer 13: INTEGRATION-REGRESSIONS.patch applied; all declared postimages and preservation hashes verified. No tests executed.
+
+### State
+All thirteen source layers staged and committed; behavioral qualification pending.
+
+### Done
+Per-layer pins and actual commit messages checked; store/current-main preservation retained.
+
+### Next
+Root reviews exact ordinary guard/source/resource admissions before execution. Full65 findings and survey/SCF lanes remain separately owned. No remote action is authorized.
+
+### Integration source-resource closure — 2026-09-09
+
+Root preflight found seven JSON source definitions declared by the ordinary guard but omitted by the staged patch delivery. Added the exact previously reviewed resource bytes from the accepted full65 source projection; no new resource admission or genuine payload. Preserved source staging and earlier evidence.
+
+## Layer14 full65 replay correction — 2026-09-09
+
+State: exact accepted two-file correction staged; successor57 integration execution pending.
+Done: verified clean84243 preimages, exact r2 postimages, fixed store/current-main sources and frozen36 evidence.
+Next: root admits the separate exact-allowlist57-case guard and final source identities before execution; no new resources.
+
+## Source review publication — 2026-09-09
+
+The user explicitly authorized pushing the current source work and creating PRs for Anthony to review. This supersedes earlier source-only local restrictions for source publication; it does not authorize a data release, merge or deployment.
+
+Integration controls passed 36 cases; the exact replay correction subsequently passed all 57 cases at dfa7f872cd3eba3c42adf5758cde8b3ca38f3d17. Source/control, model-declaration and resource hashes matched externally after both runs. A final formatting/import cleanup and explicit test-observer loop binding are included for CI; no release result is claimed. See docs/us-launch-review.md for current scope, evidence and related PRs.

changelog.d/20260906-frame-metadata-storage.fixed.md

@@ -0,0 +1 @@
+Preserve complete Frame metadata in graph content-store format v2, including nested source evidence, and refuse older frame caches that cannot restore it or reuse of one frame key with different metadata.

changelog.d/893-age-count-fixture.internal.md

@@ -0,0 +1 @@
+Use the maintained Frame store API in the national age-count cold and fresh-process replay tests after removal of the transfer example module.

changelog.d/893-diagnostic-current-lock.fixed.md

@@ -0,0 +1 @@
+Align the source/seed diagnostic with the lockfile already approved for the primary QRF worker after the UK openpyxl dependency declaration.

changelog.d/893-diagnostic-refusal-context.internal.md

@@ -0,0 +1 @@
+Record bounded code-path context for refused CI spec diagnostics while preserving the existing read restrictions and incomplete result status.

changelog.d/893-historical-zero-provenance.internal.md

@@ -0,0 +1 @@
+Keep the exact historical ASEC zero-origin citations in the dependency scan while continuing to reject other incumbent data-package references in that module.

changelog.d/893-population-only-block-source.added.md

@@ -0,0 +1 @@
+Normalize pinned population-only Census API responses into atomic block support. Reconcile every positive block against independent state totals and CD/PUMA mappings, retain exact source bytes and request descriptions, and keep publisher qualification separate from byte integrity.

changelog.d/893-source-demographic-conditioning.added.md

@@ -0,0 +1 @@
+Add optional source-qualified sex and state predictors to survey financial imputation. Preserve observed-year distinctions and default behavior, refuse unresolved source demographics, and verify complete population preservation through fitting, attachment and required replay.

changelog.d/893-spec-identity-diagnostics.internal.md

@@ -0,0 +1 @@
+Capture bounded spec and seed identity evidence after shared CI tests so changed source fingerprints can be reviewed without weakening coverage assertions.

changelog.d/893-survey-atomic-prefix.added.md

@@ -0,0 +1 @@
+Connect qualified survey geography to shared Census-block assignment before cloning and age calibration. Preserve raw sampling allocation separately, reconstruct geography in calibration budgets, and verify complete geography inheritance through cold execution and required replay. Add bounded source-byte normalization controls and record the remaining native-source and release checks.

changelog.d/893-uk-atomic-area-and-lazy-exports.added.md

@@ -0,0 +1 @@
+Add supplied-source UK atomic-area assignment keyed to stable post-clone household identity, and load UK runtime exports only when requested.

changelog.d/893-worker-resource-trace.fixed.md

@@ -0,0 +1 @@
+Update the worker resource identity regression to follow the actual lazy import closure, and verify that opened JSON resources affect identity while unlisted resources do not.

changelog.d/common-frame-export-contract.added.md

@@ -0,0 +1 @@
+Add supplied-parent comparison for US full, pruned and local frame exports, preserving retained inputs and missingness through checkpoint readback.

changelog.d/puf55-canonical-donor.added.md

@@ -0,0 +1 @@
+Add a canonical59 artifact adapter for the explicit PUF55 survey Social Security profile, retaining return identity and weights and validating the total-carrier convention. Record the scoped passing native seven-node age-development run.

changelog.d/puf55-survey-social-security.added.md

@@ -0,0 +1 @@
+Add an explicit 55-output PUF profile that preserves survey Social Security components and requires a separate total as its ninth conditioning predictor. Validate canonical donor projection and whole-Population attachment/replay while retaining source identities, weights and Social Security unknownness.

changelog.d/shared-atomic-geography.added.md

@@ -0,0 +1 @@
+Add shared graph operators for atomic-area assignment and geographic derivation. Country declarations choose observed constraints, stable draw identities, sampling stages and versioned mapping relations. Preserve observed locations, reject ambiguous support, and verify mapping consistency on cloned or pruned populations. Existing country builds retain their current geography paths pending native support integration. Normalize US block mappings, qualify observed ACS/ASEC geography against source receipts, and compose shared assignment before the combined-survey clone, with an inherited-mapping gate afterward.

changelog.d/survey-social-security-source.added.md

@@ -0,0 +1 @@
+Add current survey Social Security source qualification with explicit reporting units, question-universe unknownness, allocation literals and component mapping judgments.

changelog.d/us-atomic-financial-composition.added.md

@@ -0,0 +1 @@
+Compose survey atomic geography and current financial imputation in a checked twenty-node graph, retaining geography through cloning and financial attachment. Add native Delaware population-only source normalization and complete support readback controls.

changelog.d/us-launch-integration.fixed.md

@@ -0,0 +1 @@
+Preserve grouped calibration closing-state results alongside ordinary best-iterate selection, retain the combined US runtime facade and joint geography validation, and publish complete metadata for lazy graph snapshots.

changelog.d/us-launch-store-grouped-validation.fixed.md

@@ -0,0 +1 @@
+Reject nonfinite stored JSON as a store-corruption error and normalize the deprecated calibration method alias before validating grouped constraints.

changelog.d/us-national-atomic-support.added.md

@@ -0,0 +1,4 @@
+Add an opt-in national Census population-only source control covering the fifty
+states and DC, with independent block/state reconciliation, complete CD119 and
+PUMA joins, and full support serialization/readback. Record the accepted native
+normalization separately from household assignment and dataset release checks.

changelog.d/us-puf55-canonical-and-output-seals.added.md

@@ -0,0 +1,4 @@
+Add a source-bound PUF55 canonical donor graph node with cold/replay controls,
+stable live-code identity checks, and finalized-candidate seals for the shared
+two-route numerical finalizer. Complete native survey attachment and release
+validation remain separate build steps.

changelog.d/us-puf55-survey-ss-measurement.added.md

@@ -0,0 +1,5 @@
+Add an authenticated filer/joint-spouse Social Security report-sum measurement
+for PUF55 conditioning, with an explicit eight-predictor fallback for incomplete
+reports and preservation of all survey Social Security cells and masks. Record
+31 passing invented controls and the remaining recipient integration and
+measurement limitations.

changelog.d/us-survey-catalogue-digest.changed.md

@@ -0,0 +1 @@
+Stream eligible ACS catalogue seals through bounded record JSON encoding while preserving canonical digest bytes, full-value fallback, and repeated source and owner validation. Bind the additional JSON provider calls into the existing preparation authority checks.

changelog.d/us-survey-copy-and-identity.fixed.md

@@ -0,0 +1 @@
+Keep frame metadata copying and pickling from changing producer identity on Python 3.14. Refresh ACS source pins after verified formatting-only changes, and reduce per-cell survey identity encoding overhead while preserving canonical bytes, mutation checks, and the current-wage projection.

changelog.d/us-survey-diagnostic-options.fixed.md

@@ -0,0 +1 @@
+Reconstruct the grouped calibration solver's closing-state selection options during survey diagnostic verification, preserving exact diagnostic comparison and late-mutation checks.