From dd0043e4d39700d796a5bbe5471fb5f4fde4ff14 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 16 Jul 2026 19:15:18 -0400 Subject: [PATCH 01/16] Document C3 precision-first linkage QC Co-Authored-By: Codex gpt-5.6-sol --- docs/adr/0004-linkage-qc.md | 234 ++++++++++++++++++++++++++++++++++++ 1 file changed, 234 insertions(+) create mode 100644 docs/adr/0004-linkage-qc.md diff --git a/docs/adr/0004-linkage-qc.md b/docs/adr/0004-linkage-qc.md new file mode 100644 index 00000000..9d6fcd6a --- /dev/null +++ b/docs/adr/0004-linkage-qc.md @@ -0,0 +1,234 @@ +# ADR 0004: Employer-firm linkage QC requirements for C3 + +**Status:** Proposed — input to the C3 referee round; this document +locks no C3 threshold. C1 and C2 are frozen by the joint sign-off +recorded in +[populace-dynamics#215](https://github.com/PolicyEngine/populace-dynamics/pull/215). +This ADR treats both contracts as immutable inputs and does not amend +their schema, semantics, or readers. + +## Context + +The employer-firm plan on +[issue #192](https://github.com/PolicyEngine/populace-dynamics/issues/192) +requires noise floors before thresholds, a referee round before the C3 +gate block locks, and no one-shot candidate run before that lock. The +same ordering must govern the worker-to-employer or worker-to-firm-type +assignment itself. A downstream E-cell cannot certify a model if the +links used to construct the cell have unknown quality. + +Here, **link** includes any accepted worker-to-employer, employer- +attachment, or worker-to-firm-type assignment consumed by an E-cell. +That includes an assigned C2 `CanonicalBand` or industry type; it does +not turn a statistical imputation into an identified firm ID. A **link +unit** is the unit on which a decision is accepted or withheld. Paired +and run-level moments may require a stricter derived unit, as specified +below. + +The implemented seams remain the frozen C1 spell fields +`person_id`, `spell_id`, `start_period`, `end_period`, `industry`, +`firm_size_band`, `class_of_worker`, `earnings_share`, and +`primary_job`, and the C2 banding functions in +`src/populace_dynamics/firms/banding.py`. Linkage-QC decisions, +adjudication labels, match scores, and linkage weights therefore live +in versioned sidecars keyed to C1 rows; they are not new C1 columns. +`spell_type` is also not a C1 field. Where needed below, transition +type is derived from adjacent spell rows. + +### Evidence behind the precision-first rule + +LIFE-M scales carefully reviewed hand links with supervised learning. +Its published workflow used independent double review, additional +review of disagreements, a held-out test half, and ten-fold +cross-validation within the training half. It selected thresholds to +maximize recall subject to a project-specific **"97% precision rate"** +and evaluated that training-selected cutoff out of sample. This ADR +adopts that ordering and separation, not LIFE-M's numeric 97% choice. +See Bailey et al. (2023), sections IV.C-IV.D and Figure 5, and the +[LIFE-M linking description](https://life-m.org/linking/). + +The final abstract of Bailey, Cole, Henderson, and Massey (2020) +reports that trained reviewers rejected **"15 to 37 percent"** of +links from widely used automated methods and that the combined +problems studied attenuated an intergenerational-income-elasticity +estimate by **"up to 29 percent."** Sections III.B, IV, VI.C, and VII +show why these are not merely lost-sample problems: false links can be +systematic, estimates usually moved toward zero in their case study, +and removing false links brought estimates across algorithms together. +The authors consequently recommend putting more weight on precision +than on increasing the match count. Those numeric findings describe +their historical-data exercises, not an employer-firm floor to import. + +Bailey, Cole, and Massey (online 2019; print 2020), section III.B and +Table 4, separately show how application-specific inverse-propensity +weights can improve balance between a linked sample and its reference +population on included observables. They also state the load-bearing +limits: common support and a correctly specified selection model are +required, and balance on omitted or unobserved characteristics is not +guaranteed. Weighting therefore complements the precision floor; it +does not repair false positive links. + +## Decision + +### 1. Precision first, before every C3 threshold + +1. **Every link-producing component used by a gated E-cell must have + an independent audit artifact.** The artifact reports the full + weighted confusion matrix, precision, recall, false-positive rate, + false-negative rate, abstention/unlinked rate, and denominators, + overall and for every pre-registered gate-relevant stratum. Point + estimates and one-sided confidence bounds are both required. +2. **The precision floor is pre-registered before E-cell thresholds.** + C3 must name the assignment, eligible universe, link unit, floor + `P_floor`, confidence level, required strata, pooling rule, and + failure disposition before it names any threshold for an E-cell + that consumes that assignment. Recall is always published. Whether + recall also gates is an explicit C3 referee decision, not an + after-the-fact response to results. +3. **Passing uses a confidence bound, not the observed proportion.** + The lower one-sided confidence bound for precision must be at least + `P_floor` overall and at every stratum or derived-unit level that + C3 designates operative. Oversampled strata are combined only with + their recorded sample inclusion weights. +4. **No threshold shopping follows a linkage failure.** A failed or + unevaluable applicable floor makes the linked E-cell invalid. It is + not a model miss that can be cured by widening the E-cell tolerance, + dropping an inconvenient stratum, or selecting a different weighted + result. A registered candidate with any required invalid cell cannot + pass the employer block. +5. **The audit precedes the one-shot run.** Linkage-QC results and the + immutable adjudication manifest must be on the C3 record before a + candidate may consume the matcher. A materially changed matcher, + cutoff, candidate-generation rule, source vintage, or target + vocabulary requires a new versioned audit or the pre-registered + transport test; it never inherits a pass silently. + +### 2. Hand-adjudication sample + +The C3 block must register the following sample design before labels +are opened. + +1. **Frame and two audit arms.** Define the complete eligible universe, + candidate-generation rules, accepted-link rule, and abstentions. + Draw (a) an accepted-assignment arm, which identifies false positives + and precision, and (b) an eligible-universe arm containing accepted, + rejected, and no-link cases, whose complete candidate sets are + reviewed to identify false negatives and recall. Reviewing accepted + assignments alone is not a recall study. +2. **Stratification follows the moments the matcher feeds.** At minimum, + stratify by the five C2 `CanonicalBand` values (plus missing, + withheld, or unresolved assignments), NAICS major industry, and the + derived spell/transition class relevant to the battery: stay, + job-to-job, exit, or entry, crossed with `primary_job` where that + changes the estimand. Rare E11 origin-destination size pairs and E12 + firm types are deliberately oversampled. C3 must publish any pooling + of sparse strata before adjudication and retain each inclusion + probability. +3. **Target size is powered at the floor.** C3 registers `P_floor`, a + substantively meaningful design precision `P_design > P_floor`, + one-sided size `alpha`, power `1 - beta`, and its multiplicity rule. + For each operative pooling level, the target is the smallest number + `n` of adjudicated accepted assignments for which an integer critical + count `c` exists such that + + `Pr[X >= c | X ~ Binomial(n, P_floor)] <= alpha` + + and + + `Pr[X >= c | X ~ Binomial(n, P_design)] >= 1 - beta`. + + The registered target also includes anticipated abstention, + unusable-record, and nonresponse inflation. If multiple strata must + each clear the floor, the power calculation uses the pre-registered + family-wise error allocation. A round number without this calculation + is not a target-size justification. +4. **Blind, independent coding.** Two trained coders independently see + the same source evidence and candidate set, but not the matcher's + score, cutoff, accepted choice, downstream outcome, or the other + coder's decision. They code `link`, `no link`, or `insufficient + evidence` under a frozen manual. A third coder or standing panel + adjudicates disagreements without majority labels being disclosed + first. The artifact reports agreement, disagreement, insufficient- + evidence rates, coder/manual versions, and final dispositions. The + label is **hand-adjudicated reference truth**, not a claim of + infallible ground truth. +5. **Provenance is committed and immutable.** Commit a privacy-safe + manifest containing the frame query and vintage, stratum definitions, + random seed, selected-row hashes or access-controlled immutable IDs, + inclusion probabilities, evidence and coding-manual versions, coder + assignment protocol, adjudication rule, counts, and artifact hashes. + Raw restricted records and direct identifiers remain outside git. + Any replacement or exclusion is logged; the sample is never silently + refreshed. +6. **No train-test leakage.** Neither adjudication arm, its final labels, + nor disagreement dispositions may train, tune, select features for, + set a cutoff for, or otherwise adapt the matcher it scores. A leaked + sample is retired from evaluation and replaced under a new manifest. + +### 3. Linkage-bias reweighting on observables + +1. **Name the target population.** Each linked analysis declares the + eligible reference population it is intended to represent and the + pre-link observables `X` available for both linked and unlinked units. + Candidate observables include source/vintage, age, sex, state, + `class_of_worker`, `primary_job`, industry information known before + the assignment, earnings/tenure measures, missingness, and transition + opportunity. Post-link outcomes cannot be used to manufacture + balance. +2. **Estimate and publish link propensities.** Fit and version + `p_i = Pr(L_i = 1 | X_i)`, where `L_i` denotes inclusion in the + usable linked subsample. Publish the model specification, training + population, out-of-sample diagnostics, propensity distributions for + linked and reference units, overlap/common-support checks, covariate + balance before and after weighting, weight distribution, and effective + sample size. Trimming, stabilization, normalization, and capping rules + are pre-registered. +3. **Use Bailey-Cole-Massey-style weights.** The default is a normalized + inverse-link-propensity weight appropriate to the declared target. + If the linked and reference samples are stacked as in Bailey, Cole, + and Massey, the normalized inverse-odds form is + `[(1 - p_i) / p_i] [q / (1 - q)]`, where `q` is the linked share. + A different sampling construction may require `1 / p_i`; C3 must + derive and register the form rather than choose it after seeing the + E-cell. +4. **Publish weighted and unweighted together.** Every E-cell consuming + links publishes both, labels the registered operative version + (`unweighted` or `linkage_ipw`), and explains its estimand. The + adjudication sample-inclusion weight and the linkage-propensity weight + are distinct and must not be conflated. +5. **Treat overlap failure as scope failure.** Extreme weights, absent + common support, or material residual imbalance are reported, not + hidden by ad hoc trimming. If the registered weighting diagnostic + fails, the weighted cell is invalid. The unweighted diagnostic remains + visible but cannot be substituted as the gate after results are known. + These weights mitigate selection on included observables only; they do + not correct a wrong link, establish balance on unobservables, or turn a + firm type into an observed firm identity. + +## Consequences + +- C3 gains a linkage-quality input gate before its model-fit gates. +- The required evidence is carried in sidecars and reported artifacts; + C1, C2, `gates.yaml`, readers, and banding code are unchanged. +- Exact floor values, E-cell thresholds, and operative weighting choices + remain decisions for the C3 referee round. + +## References + +- Bailey, Martha, Peter Z. Lin, A. R. Shariq Mohammed, Paul Mohnen, + Jared Murray, Mengying Zhang, and Alexa Prettyman. 2023. + ["The Creation of LIFE-M: The Longitudinal, Intergenerational Family + Electronic Micro-Database Project."](https://doi.org/10.1080/01615440.2023.2239699) + *Historical Methods* 56 (3): 138-159. +- Bailey, Martha J., Connor Cole, Morgan Henderson, and Catherine Massey. + 2020. ["How Well Do Automated Linking Methods Perform? Lessons from US + Historical Data."](https://doi.org/10.1257/jel.20191526) + *Journal of Economic Literature* 58 (4): 997-1044. The 29-percent + figure above follows the + [final AEA abstract](https://www.aeaweb.org/articles?id=10.1257/jel.20191526), + not the earlier author manuscript. +- Bailey, Martha, Connor Cole, and Catherine Massey. 2019 online / 2020 + print. ["Simple Strategies for Improving Inference with Linked Data: A + Case Study of the 1850-1930 IPUMS Linked Representative Historical + Samples."](https://doi.org/10.1080/01615440.2019.1630343) + *Historical Methods* 53 (2): 80-93. From ee58b30090b385b581114adc7fb06422753fe67a Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 16 Jul 2026 19:26:20 -0400 Subject: [PATCH 02/16] Wire linkage QC into the employer battery Co-Authored-By: Codex gpt-5.6-sol --- docs/adr/0004-linkage-qc.md | 328 ++++++++++++++++++++++++++++-------- 1 file changed, 258 insertions(+), 70 deletions(-) diff --git a/docs/adr/0004-linkage-qc.md b/docs/adr/0004-linkage-qc.md index 9d6fcd6a..83ae9e68 100644 --- a/docs/adr/0004-linkage-qc.md +++ b/docs/adr/0004-linkage-qc.md @@ -1,11 +1,10 @@ # ADR 0004: Employer-firm linkage QC requirements for C3 **Status:** Proposed — input to the C3 referee round; this document -locks no C3 threshold. C1 and C2 are frozen by the joint sign-off -recorded in +locks no C3 threshold. C1 and C2 are treated as frozen and immutable +for this work under the workstream directive and the freeze record in [populace-dynamics#215](https://github.com/PolicyEngine/populace-dynamics/pull/215). -This ADR treats both contracts as immutable inputs and does not amend -their schema, semantics, or readers. +This ADR does not amend their schema, semantics, or readers. ## Context @@ -25,23 +24,26 @@ unit** is the unit on which a decision is accepted or withheld. Paired and run-level moments may require a stricter derived unit, as specified below. -The implemented seams remain the frozen C1 spell fields +The frozen contract seam remains the C1 spell fields `person_id`, `spell_id`, `start_period`, `end_period`, `industry`, `firm_size_band`, `class_of_worker`, `earnings_share`, and -`primary_job`, and the C2 banding functions in +`primary_job`. The implemented C2 seam is the banding functions in `src/populace_dynamics/firms/banding.py`. Linkage-QC decisions, adjudication labels, match scores, and linkage weights therefore live in versioned sidecars keyed to C1 rows; they are not new C1 columns. `spell_type` is also not a C1 field. Where needed below, transition -type is derived from adjacent spell rows. +type is derived from adjacent spell rows plus the versioned person-month +observation/nonemployment frame; C1 spells alone do not identify exits, +entries, or censoring. ### Evidence behind the precision-first rule LIFE-M scales carefully reviewed hand links with supervised learning. -Its published workflow used independent double review, additional -review of disagreements, a held-out test half, and ten-fold -cross-validation within the training half. It selected thresholds to -maximize recall subject to a project-specific **"97% precision rate"** +Its published workflow used independent double review, three new +independent reviewers for disagreements, a held-out test half, and +ten-fold cross-validation within the training half. It selected +thresholds to maximize recall subject to a project-specific +**"97% precision rate"** and evaluated that training-selected cutoff out of sample. This ADR adopts that ordering and separation, not LIFE-M's numeric 97% choice. See Bailey et al. (2023), sections IV.C-IV.D and Figure 5, and the @@ -63,21 +65,28 @@ Bailey, Cole, and Massey (online 2019; print 2020), section III.B and Table 4, separately show how application-specific inverse-propensity weights can improve balance between a linked sample and its reference population on included observables. They also state the load-bearing -limits: common support and a correctly specified selection model are -required, and balance on omitted or unobserved characteristics is not -guaranteed. Weighting therefore complements the precision floor; it -does not repair false positive links. +limits: common support and the paper's unconfoundedness/properly +specified propensity-score assumption are required, and balance on +omitted or unobserved characteristics is not guaranteed. Weighting +therefore complements the precision floor; it does not repair false +positive links. ## Decision ### 1. Precision first, before every C3 threshold 1. **Every link-producing component used by a gated E-cell must have - an independent audit artifact.** The artifact reports the full - weighted confusion matrix, precision, recall, false-positive rate, - false-negative rate, abstention/unlinked rate, and denominators, - overall and for every pre-registered gate-relevant stratum. Point - estimates and one-sided confidence bounds are both required. + an independent audit artifact.** For categorical assignments, the + artifact reports the audit-design- and survey-weighted matrix of + adjudicated true class by assigned class, including no-counterpart, + abstain/unlinked, and indeterminate outcomes. It also reports + accepted-assignment precision, + end-to-end and stage-specific recall, false-negative and abstention + rates, and all numerators and denominators, overall and for every + pre-registered gate-relevant stratum. If it reports a pairwise + false-positive rate, it must define the candidate-pair universe; + `1 - precision` is the false-discovery rate, not that pairwise rate. + Point estimates and one-sided confidence bounds are both required. 2. **The precision floor is pre-registered before E-cell thresholds.** C3 must name the assignment, eligible universe, link unit, floor `P_floor`, confidence level, required strata, pooling rule, and @@ -109,27 +118,36 @@ The C3 block must register the following sample design before labels are opened. 1. **Frame and two audit arms.** Define the complete eligible universe, - candidate-generation rules, accepted-link rule, and abstentions. - Draw (a) an accepted-assignment arm, which identifies false positives - and precision, and (b) an eligible-universe arm containing accepted, - rejected, and no-link cases, whose complete candidate sets are - reviewed to identify false negatives and recall. Reviewing accepted - assignments alone is not a recall study. -2. **Stratification follows the moments the matcher feeds.** At minimum, - stratify by the five C2 `CanonicalBand` values (plus missing, - withheld, or unresolved assignments), NAICS major industry, and the - derived spell/transition class relevant to the battery: stay, - job-to-job, exit, or entry, crossed with `primary_job` where that - changes the estimand. Rare E11 origin-destination size pairs and E12 - firm types are deliberately oversampled. C3 must publish any pooling - of sparse strata before adjudication and retain each inclusion - probability. + candidate-generation rules, accepted-link rule, true no-counterpart + cases, and abstentions. Draw (a) an accepted-assignment arm, which + identifies false positives and precision, and (b) an eligible-universe + truth-search arm containing accepted, rejected, and no-link cases. + The second arm uses an independent exhaustive search or external + reference that can find a true counterpart omitted by candidate + generation; reviewing only the matcher's candidate set is not an + end-to-end recall study. Report candidate-generation recall separately + from selector/cutoff recall. Because a unit may enter both arms, C3 + registers the dual-frame overlap and combined-inclusion estimator so + it is neither omitted nor counted twice. +2. **Stratification follows the moments the matcher feeds.** For the + accepted arm, stratify at minimum by the five assigned C2 + `CanonicalBand` values (plus withheld or unresolved assignments), + assigned NAICS major industry, and the derived spell/transition class + relevant to the battery: stay, job-to-job, exit, or entry, crossed + with `primary_job` where that changes the estimand. Rare E11 origin- + destination size pairs and E12 firm types are deliberately + oversampled. Because rejected and no-link units lack an assigned or + known-true class before review, the universe arm uses a separate + pre-link stratification frame observed for every eligible unit, then + reports recall by adjudicated true band, industry, and transition + domain. C3 publishes any pooling of sparse strata before adjudication + and retains every arm-specific inclusion probability. 3. **Target size is powered at the floor.** C3 registers `P_floor`, a substantively meaningful design precision `P_design > P_floor`, one-sided size `alpha`, power `1 - beta`, and its multiplicity rule. - For each operative pooling level, the target is the smallest number - `n` of adjudicated accepted assignments for which an integer critical - count `c` exists such that + For independent, equal-probability accepted assignments within a + simple-random operative stratum, the target is the smallest number + `n` for which an integer critical count `c` exists such that `Pr[X >= c | X ~ Binomial(n, P_floor)] <= alpha` @@ -137,21 +155,43 @@ are opened. `Pr[X >= c | X ~ Binomial(n, P_design)] >= 1 - beta`. - The registered target also includes anticipated abstention, - unusable-record, and nonresponse inflation. If multiple strata must - each clear the floor, the power calculation uses the pre-registered - family-wise error allocation. A round number without this calculation - is not a target-size justification. + Repeated spells, pairs, and runs are not independent Bernoulli draws. + A worker-, firm-, or time-clustered, unequal-probability, finite, or + dual-frame design instead uses design-based analytic or simulation + power with the registered clustering unit, inclusion weights, finite- + population correction where material, and anticipated design effect. + The target includes unusable-record, indeterminate, and nonresponse + inflation. If multiple strata must clear, C3 distinguishes simultaneous + confidence coverage from an intersection-union pass rule and computes + the **joint** probability that every required stratum passes at + `P_design`; powering each stratum separately at `1 - beta` is not + enough. A round number without the applicable calculation is not a + target-size justification. The eligible-universe truth-search arm has + its own registered target, based on expected true-counterpart + prevalence and a desired recall-bound width or, if recall gates, an + analogous floor-and-power calculation. It must contain enough + independently searched true links to make false-negative uncertainty + informative. 4. **Blind, independent coding.** Two trained coders independently see the same source evidence and candidate set, but not the matcher's score, cutoff, accepted choice, downstream outcome, or the other coder's decision. They code `link`, `no link`, or `insufficient evidence` under a frozen manual. A third coder or standing panel adjudicates disagreements without majority labels being disclosed - first. The artifact reports agreement, disagreement, insufficient- - evidence rates, coder/manual versions, and final dispositions. The - label is **hand-adjudicated reference truth**, not a claim of - infallible ground truth. + first. Before labels open, C3 registers whether an indeterminate, + unusable-evidence, or nonresponse case counts conservatively as + incorrect, enters partial-identification bounds, or makes the floor + unevaluable; hard cases may not be dropped from denominators after + their labels are known. The artifact reports each rate, its effect on + precision and recall denominators, agreement, disagreement, coder/ + manual versions, and final dispositions. Coders first calibrate on + separate, vetted cases excluded from both audit arms; blinded + audit/gold repeats measure + continuing coder accuracy as well as agreement. The registered + precision test either propagates estimated reference-label error or + publishes the sensitivity bound needed to clear the floor. The label + is **hand-adjudicated reference truth**, not a claim of infallible + ground truth. 5. **Provenance is committed and immutable.** Commit a privacy-safe manifest containing the frame query and vintage, stratum definitions, random seed, selected-row hashes or access-controlled immutable IDs, @@ -167,47 +207,193 @@ are opened. ### 3. Linkage-bias reweighting on observables -1. **Name the target population.** Each linked analysis declares the - eligible reference population it is intended to represent and the - pre-link observables `X` available for both linked and unlinked units. +1. **Name the target population and identification assumption.** Each + linked analysis declares its actual analysis unit (assignment, pair, + transition, run, or firm/type cluster), the eligible reference + population it is intended to represent, and the pre-link observables + `X` available for both linked and unlinked units. Candidate observables include source/vintage, age, sex, state, `class_of_worker`, `primary_job`, industry information known before the assignment, earnings/tenure measures, missingness, and transition - opportunity. Post-link outcomes cannot be used to manufacture - balance. -2. **Estimate and publish link propensities.** Fit and version - `p_i = Pr(L_i = 1 | X_i)`, where `L_i` denotes inclusion in the - usable linked subsample. Publish the model specification, training + opportunity. For each E-cell outcome `Y`, the registered identification + claim is usable-link inclusion independent of `Y` conditional on `X`, + plus positivity on the target support. It is an assumption to defend, + not a result of a balance test. Post-link outcomes cannot be used to + manufacture balance. +2. **Estimate and publish actual inclusion propensities.** Fit and + version `s_i = Pr(L_i = 1 | X_i)`, where `L_i` denotes inclusion in + the usable linked subsample of the complete eligible universe at the + E-cell's analysis unit. A person-level score does not automatically + weight a pair, run, or cluster; C3 models that unit's inclusion or + pre-registers and justifies a joint construction from component + scores. Publish the model specification, training population, out-of-sample diagnostics, propensity distributions for linked and reference units, overlap/common-support checks, covariate balance before and after weighting, weight distribution, and effective sample size. Trimming, stabilization, normalization, and capping rules are pre-registered. -3. **Use Bailey-Cole-Massey-style weights.** The default is a normalized - inverse-link-propensity weight appropriate to the declared target. - If the linked and reference samples are stacked as in Bailey, Cole, - and Massey, the normalized inverse-odds form is - `[(1 - p_i) / p_i] [q / (1 - q)]`, where `q` is the linked share. - A different sampling construction may require `1 / p_i`; C3 must - derive and register the form rather than choose it after seeing the - E-cell. -4. **Publish weighted and unweighted together.** Every E-cell consuming - links publishes both, labels the registered operative version +3. **Use a weight derived for the sampling construction.** For actual + inclusion propensity `s_i`, full-population inverse-probability + weighting uses `1 / s_i`, subject to the registered base design. + Bailey, Cole, and Massey instead append a linked-sample copy to a + reference-population copy and fit + `r_i = Pr(D_i = linked copy | X_i)` in that stack. For this density- + ratio construction only, their normalized inverse-odds weight is + `[(1 - r_i) / r_i] [q / (1 - q)]`, where `q` is the linked-copy + share of the stack. `s_i` and `r_i` are not interchangeable. The + linkage adjustment multiplies the pre-existing survey/design/ + opportunity weight; it does not replace that base weight. C3 derives + and registers the applicable form before seeing the E-cell. +4. **Publish weighted and unweighted with valid uncertainty.** Every + E-cell consuming links publishes both, labels the registered operative + version (`unweighted` or `linkage_ipw`), and explains its estimand. The + `unweighted` label means base-weighted without the linkage adjustment, + not equal-record weighting that discards a source survey design. The adjudication sample-inclusion weight and the linkage-propensity weight - are distinct and must not be conflated. + are distinct and must not be conflated. Intervals and gate statistics + account for estimated propensities, base survey/design weights, any + audit weights entering the estimator, trimming or stabilization, and + repeated-person, firm/type, and time clustering. A point estimate and + effective sample size alone are insufficient. 5. **Treat overlap failure as scope failure.** Extreme weights, absent common support, or material residual imbalance are reported, not hidden by ad hoc trimming. If the registered weighting diagnostic fails, the weighted cell is invalid. The unweighted diagnostic remains visible but cannot be substituted as the gate after results are known. - These weights mitigate selection on included observables only; they do - not correct a wrong link, establish balance on unobservables, or turn a - firm type into an observed firm identity. + Support-based trimming changes the target population, which must be + renamed and reported rather than presented as the original estimand. + These weights mitigate selection on included observables only; they + do not correct a wrong link, establish balance on unobservables, or + turn a firm type into an observed firm identity. + +### 4. Battery wiring and invalidation + +An overall floor failure invalidates every linked cell using that +matcher version. A required-stratum failure invalidates every cell whose +estimand includes that stratum, unless C3 pre-registers a genuinely +disjoint matcher and estimand. A passing marginal link floor does not by +itself certify a pair or a run: C3 must either audit the derived unit +directly or register and justify a conservative composition rule. + +| cell | linkage unit and required QC | weighting and failure disposition | +|---|---|---| +| **E4 — retention pairs** | Audit endpoint assignments and, if C3 makes it operative, the derived same-employer/same-attribute decision. The audit strata include age, industry, C2 band, transition month, and `primary_job` status used by the cell. | Publish pair-opportunity estimates unweighted and with linkage-IPW. If any C3-designated endpoint or pair-level floor fails, all affected E4 retention cells are invalid. | +| **E5 — attachment runs** | If C3 designates a run-level floor, audit the full multi-window run label, including false continuation and false break errors. A per-month pass alone cannot certify a run because error compounds with length. | Weight the eligible run opportunity, not each observed linked month as if independent. If any C3-designated endpoint or run-level floor fails, the affected E5 run-length cells are invalid. | +| **E9 — earnings-change coherence** | Derive stay, job-to-job, exit, and entry from adjacent C1 spells plus the versioned person-month observation/nonemployment frame. Audit any C3-designated transition floor; for job-to-job cells, audit both origin and destination firm-size/industry assignments. | Define propensity and composite weight on the eligible transition opportunity, then publish both versions within class. A failed C3-designated origin, destination, or transition-class floor invalidates the corresponding E9 cells; the referee cannot replace them post hoc with a different definition. | +| **E11 — firm-size flow ladder** | The unit is an origin-destination job-to-job pair. Audit the joint ordered C2-band assignment. An unresolved `BandSpan` is not a correct categorical assignment merely because it contains the eventual band. | Model inclusion and weight at the ordered-pair opportunity. An overall or joint-pair floor failure invalidates E11; a required origin/destination stratum failure invalidates every E11 cell containing it. | +| **E12 — variance and coworker structure** | Phase 2 must audit worker-to-firm-type assignment and any generated same-firm or coworker co-assignment at the exact unit the E12 estimand uses. Type agreement alone cannot validate a claim about an identified firm. | Model inclusion at the worker pair, co-assignment, or cluster unit used by the decomposition; a worker-only propensity is insufficient without a justified composition. Publish both versions. If truth, floor, or support fails, every E12 cell using it is invalid and phase 2 is a no-go. | + +E3 and E8 do not ordinarily require a worker-to-firm link, and E10 +re-runs the existing locked PSID earnings gates without a new noise +floor. They are not blanket exemptions: if a final C3 implementation +constructs any of them from accepted employer or firm-type assignments, +the precision-first law applies. Linkage failure never weakens E10 or +changes an existing PSID threshold. + +### 5. Phase scope and real seams + +#### Phase 1 — spell hazards, no two-sided register + +Phase 1 has within-panel employer attachment and firm attributes on +spells, not an observed two-sided worker-firm roster. The SIPP reader's +`EJB{n}_JOBID` is a within-panel attachment key. Its spell collapse +currently carries raw `empsize_code`; `sipp_empsize_to_canonical` in +`banding.py` preserves source-band ambiguity through `BandSpan`, while +the frozen C1 seam ultimately carries one `CanonicalBand`. Therefore: + +1. QC scores the **final accepted assignment consumed by the E-cell**, + after any ambiguity resolution, not the raw SIPP code or a claim that + an ambiguous span is exact. +2. An exact SIPP interval-to-band map establishes only numeric interval + nesting. SIPP measures establishment size, so it does not by itself + validate administrative enterprise size. +3. Sidecars join to C1 with `person_id` and `spell_id`. They do not add + `job_id`, `firm_id`, match score, adjudication status, or weights to + frozen C1. +4. E4/E5 score retention and attachment, E9 scores transition-conditioned + earnings changes, and E11 scores firm-size flows only after their + applicable link and reweighting requirements above are evaluable. + +The pre-C3 floor draft on +[PR #212](https://github.com/PolicyEngine/populace-dynamics/pull/212) +does not satisfy or conflict with this ADR: it estimates sampling noise +in E3/E4/E5/E8/E9 after the linkage inputs are defined. Its half-splits +cannot reveal a common linkage bias. The seam artifact on +[PR #214](https://github.com/PolicyEngine/populace-dynamics/pull/214) +likewise remains a separate prerequisite: it compares SIPP and J2J rate +levels, but does not estimate precision or recall for a worker-to-firm- +type assignment. + +#### Phase 2 — BLM firm types, still not observed firms + +Phase 2 proposes a BLM-style register of firm **types** (industry x +size x state), not identified enterprises or a public worker-firm +roster. E12 is especially sensitive to false assignments because a bad +worker-to-firm or coworker link moves covariance between the within- and +between-firm components. Bailey, Cole, Henderson, and Massey (2020), +section VI.C and Figure 7, show that false links often attenuated their +intergenerational-income-elasticity estimates and that removing them +reconciled estimates. Their broader result also warns that systematic +error can make the bias algorithm-dependent rather than always +attenuating. + +For that reason, a phase-2 E12 story must identify an admissible +hand-adjudication frame for the actual assignment unit and clear its +precision floor. If public margins cannot support that truth, C3 records +the limitation as a phase-2 no-go; calibration fit to aggregates is not +a substitute. The register may support firm-type policy claims only at +the level it identifies. It may not relabel type agreement as firm- +identity or coworker validation. + +### 6. Items for the C3 referee round — deliberately unresolved + +The referee round must decide and pre-register the following. This ADR +does not resolve them: + +1. The numeric precision floor or floors; `P_design`, `alpha`, power, + confidence interval, multiplicity correction, and whether recall has + an operative floor. +2. The phase-1 and phase-2 adjudication frames, permissible evidence, + exact truth labels, privacy-safe manifest, and whether an E12 truth + source exists at all. +3. Beyond E11's required ordered pair and the actual co-assignment unit + used by E12, which endpoint, pair, transition, and run levels gate; + how a passing endpoint result composes, if at all; and which sparse + strata may be pooled before the powered sample target is calculated. +4. The target reference population, propensity model and observables, + overlap and balance tolerances, weight stabilization/trimming rule, + and the pre-registered operative weighting for E4, E5, E9, E11, and + E12. +5. The exact calibration/gate cell partition, including held-out axes; + the job-count-to-person-count adjustment for phase 0; the private- + comparable versus caveat treatment of J2J's broader employer + universe; and the ASEC firm-size/tenure reference-period mismatch. +6. PR #212's measurement choices: E3 quantile gaps versus weighted-ECDF + gaps, and E9 stay IQR versus a broader distributional distance. The + integer and wave-constant heaping degeneracies remain on the record. +7. PR #214's cross-wave SIPP job-ID consistency check and the final + ruling on J2J rate levels, SIPP persistence, and seam-aware hazard + estimation. +8. The E12 estimand and entity: firm type versus generated pseudo-firm, + the minimum evidence for within/between variance and coworker + correlation, and the phase-2 identification/go-no-go standard. +9. Long-window tenure evidence from PSID/NLSY and any transport test for + applying one adjudication result across source vintages or populations. + +The settled C1 fields, C2 semantics and five bands, explicit +`NOEMP`/`FIRMSIZE` coding, class-of-worker universe, and person-table +geography join are outside this list. Reopening them requires the joint +C1/C2 amendment process, not the C3 referee round. ## Consequences - C3 gains a linkage-quality input gate before its model-fit gates. + E4, E5, E9, E11, and E12 cannot certify a candidate from unaudited or + floor-failing assignments. +- Every link-consuming cell exposes the observable-selection question + by publishing linkage-IPW and unweighted estimates together, with one + operative version chosen before the candidate result is seen. - The required evidence is carried in sidecars and reported artifacts; C1, C2, `gates.yaml`, readers, and banding code are unchanged. - Exact floor values, E-cell thresholds, and operative weighting choices @@ -226,7 +412,9 @@ are opened. *Journal of Economic Literature* 58 (4): 997-1044. The 29-percent figure above follows the [final AEA abstract](https://www.aeaweb.org/articles?id=10.1257/jel.20191526), - not the earlier author manuscript. + while the + [earlier deposited author manuscript](https://pmc.ncbi.nlm.nih.gov/articles/PMC8294155/) + reports 20 percent in its abstract, introduction, and section VI.C. - Bailey, Martha, Connor Cole, and Catherine Massey. 2019 online / 2020 print. ["Simple Strategies for Improving Inference with Linked Data: A Case Study of the 1850-1930 IPUMS Linked Representative Historical From 87788eb6d937d4f072b75df011c0ac0e7ceb64a2 Mon Sep 17 00:00:00 2001 From: Daphne Hansell <128793799+daphnehanse11@users.noreply.github.com> Date: Fri, 17 Jul 2026 09:17:00 -0400 Subject: [PATCH 03/16] =?UTF-8?q?Cross-wave=20job-ID=20consistency=20check?= =?UTF-8?q?:=20PASS=20(pre-lock=20artifact=20for=20#230=20=C2=A76)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The blocking check for the seam ruling (ADR 0004 referee item 7; #214 concept-delta 5). Verdict rule pre-registered in the script before the numbers were seen. Results: gross ID survival across the pu2022->pu2023 boundary 90.55%; re-key signature (same industry + class of worker + earnings within 20%) among seam separators-to- employment 17.5% vs a 2.4% within-wave coincidence baseline; scaled to all seam separations (38.1% are exits to nonemployment, which cannot be ID artifacts), the implied ID-artifact share of the 9.45% seam rate is <= 9.4% — under the 15% PASS bar. At least ~90% of the seam contrast is real seam-bunched separation; the #214 ruling's conditional check is satisfied. Implementation note, disclosed: the first run inner-joined the next-month jobs frame and silently dropped exits to nonemployment (printing a 6.06% conditioned seam rate and a PASS_WITH_CORRECTION_BAND verdict against the wrong denominator); the fix restores the documented design via the person-month universe and reproduces #214's 9.45%/1.77% exactly. Co-Authored-By: Claude Fable 5 --- runs/crosswave_jobid_check_draft_v0.json | 33 +++ scripts/build_crosswave_jobid_check.py | 313 +++++++++++++++++++++++ 2 files changed, 346 insertions(+) create mode 100644 runs/crosswave_jobid_check_draft_v0.json create mode 100644 scripts/build_crosswave_jobid_check.py diff --git a/runs/crosswave_jobid_check_draft_v0.json b/runs/crosswave_jobid_check_draft_v0.json new file mode 100644 index 00000000..a73ad81f --- /dev/null +++ b/runs/crosswave_jobid_check_draft_v0.json @@ -0,0 +1,33 @@ +{ + "artifact": "crosswave_jobid_check", + "version": "draft_v0", + "status": "DRAFT - pre-lock artifact for the #230 section-6 seam ruling; verdict rule pre-registered in the build script", + "issue": "230", + "question": "are EJB job IDs longitudinally consistent across the pu2022->pu2023 boundary, or is part of the 9.45% seam separation rate a re-keying (linkage) artifact?", + "within_wave_baseline": { + "jobs_held": 384747, + "separations": 6800, + "to_nonemployment": 3276, + "to_employment": 3524, + "rekey_signature": 85, + "sep_rate": 0.0177, + "rekey_signature_share_of_seps": 0.0125 + }, + "across_wave_seam": { + "jobs_held": 10828, + "separations": 1023, + "sep_rate": 0.0945, + "to_nonemployment": 390, + "to_employment": 633, + "rekey_signature": 111, + "rekey_signature_share_of_seps": 0.1085 + }, + "rekey_signature_definition": "a vanished job whose person holds a next-month job matching it on industry code, class of worker, and earnings within 20% (|log ratio| < 0.1823); computed identically at the seam and within-wave, so the within-wave share is the coincidental-match baseline", + "bounds": { + "gross_id_survival_share": 0.9055, + "excess_rekey_signature_share_of_seam_seps": 0.0936, + "structural_ee_cap_share_of_seam_seps": 0.6188 + }, + "verdict_rule": "PASS if excess re-key share < 15% of seam separations; PASS_WITH_CORRECTION_BAND if 15-30%; REFER_BACK if >30%", + "verdict": "PASS" +} diff --git a/scripts/build_crosswave_jobid_check.py b/scripts/build_crosswave_jobid_check.py new file mode 100644 index 00000000..ea662324 --- /dev/null +++ b/scripts/build_crosswave_jobid_check.py @@ -0,0 +1,313 @@ +"""Build the cross-wave job-ID consistency check (C3 §6 pre-lock). + +REQUIRED PRE-LOCK ARTIFACT for the seam ruling (#230 §6; ADR 0004 +referee item 7; #214's concept-delta 5, previously an UNVERIFIED +ASSUMPTION). Question: are SIPP ``EJB`` job IDs longitudinally +consistent across the pu2022 -> pu2023 file boundary, or partly +reassigned — in which case part of the measured 9.45% Dec->Jan seam +separation rate would be a linkage artifact rather than seam-bunched +real separations? + +Design — three bounds, none assuming what they test: + +(a) **Gross consistency**: the share of December-held jobs whose ID + survives into January at all. Wholesale per-wave reassignment + would put this near zero; the #214 artifact already implies + ~90.6%, so gross reassignment is bounded by the seam rate + itself. + +(b) **The re-key signature**: among Dec->Jan *separations* (no + common ID), the share where the person holds a January job that + matches the vanished December job on industry code AND class of + worker AND monthly earnings within 20% (|log ratio| < 0.1823) — + the profile of the same employer continuing under a new ID. + Genuine job-to-job moves can also match by coincidence, so the + identical signature is computed for *within-wave* separations + (pooled month-pairs inside each file), whose IDs are known-good + under dependent interviewing. The EXCESS of the seam signature + over the within-wave baseline is the upper bound on the ID + artifact among employed-next-month separators. + +(c) **The structural bound**: seam separations decompose into exits + to nonemployment (no January job exists, so no new ID could + have been issued — these CANNOT be ID artifacts) versus + separations-to-employment. Only the latter can hide re-keying, + so the E->E share caps the artifact regardless of (b). + +Verdict rule (pre-registered here): the ruling's conditional check +PASSES if the implied ID-artifact share of the seam rate — excess +re-key signature applied to the E->E component — is under 15% of +the measured seam rate; between 15% and 30% the seam figures carry +a correction band; above 30% the #214 ruling returns to the referee. + +Usage:: + + python scripts/build_crosswave_jobid_check.py + +writes ``runs/crosswave_jobid_check_draft_v0.json``. +""" + +from __future__ import annotations + +import json +import sys +from pathlib import Path + +import numpy as np +import pandas as pd + +REPO = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(REPO / "src")) + +from populace_dynamics.data import sipp_jobs # noqa: E402 + +FILE_YEARS = (2022, 2023) +EARN_LOG_TOL = abs(np.log(0.8)) # earnings within 20% +ARTIFACT = REPO / "runs/crosswave_jobid_check_draft_v0.json" + + +def person_month_presence(year: int) -> pd.DataFrame: + """All person-months in the file (employed or not).""" + import os + + data_dir = Path( + os.environ.get( + "POPULACE_DYNAMICS_SIPP_DIR", + str(Path("~/PolicyEngine/sipp-data").expanduser()), + ) + ).expanduser() + for suffix in (".csv", ".csv.gz"): + path = data_dir / f"pu{year}{suffix}" + if path.exists(): + break + else: + raise FileNotFoundError(f"pu{year}.csv[.gz] not staged") + raw = pd.read_csv( + path, + sep="|", + usecols=["SSUID", "PNUM", "MONTHCODE"], + dtype={"SSUID": "string"}, + ) + raw["person_id"] = raw["SSUID"].astype(str) + "-" + raw["PNUM"].astype(str) + return raw[["person_id", "MONTHCODE"]].rename( + columns={"MONTHCODE": "month"} + ) + + +def month_frame(job_months: pd.DataFrame) -> pd.DataFrame: + """Per person-month: job set plus per-job attribute map.""" + jm = job_months.copy() + jm["attrs"] = list( + zip( + jm["job_id"], + jm["industry"].astype(str), + jm["clwrk"], + jm["earnings"], + strict=True, + ) + ) + return ( + jm.groupby(["person_id", "month"]) + .agg(jobs=("job_id", frozenset), attrs=("attrs", list)) + .reset_index() + ) + + +def _rekey_match(lost, new_jobs) -> bool: + """Does any new job match a lost job's employer profile?""" + _, ind, clwrk, earn = lost + for _, n_ind, n_clwrk, n_earn in new_jobs: + if n_ind != ind: + continue + if pd.notna(clwrk) and pd.notna(n_clwrk) and n_clwrk != clwrk: + continue + if ( + pd.notna(earn) + and pd.notna(n_earn) + and earn > 0 + and n_earn > 0 + and abs(np.log(n_earn / earn)) > EARN_LOG_TOL + ): + continue + return True + return False + + +def separation_decomposition( + current: pd.DataFrame, + following: pd.DataFrame, + present_next: set, +) -> dict: + """Decompose separations between two adjacent person-months. + + ``present_next`` is the set of person_ids in the panel next + month (employed or not): a person present with no jobs is an + exit to nonemployment; a person absent left the sample and is + excluded from the denominator entirely. + """ + current = current[current["person_id"].isin(present_next)] + merged = current.merge( + following, + on="person_id", + suffixes=("", "_n"), + how="left", + ) + merged["jobs_n"] = merged["jobs_n"].apply( + lambda x: x if isinstance(x, frozenset) else frozenset() + ) + merged["attrs_n"] = merged["attrs_n"].apply( + lambda x: x if isinstance(x, list) else [] + ) + jobs_held = jobs_kept = 0 + lost_to_nonemp = lost_to_emp = lost_rekey_sig = 0 + for row in merged.itertuples(index=False): + kept_ids = row.jobs & row.jobs_n + jobs_held += len(row.jobs) + jobs_kept += len(kept_ids) + new_jobs = [a for a in row.attrs_n if a[0] not in row.jobs] + for lost in row.attrs: + if lost[0] in kept_ids: + continue + if not row.jobs_n: + lost_to_nonemp += 1 + continue + lost_to_emp += 1 + if _rekey_match(lost, new_jobs): + lost_rekey_sig += 1 + separations = jobs_held - jobs_kept + return { + "jobs_held": jobs_held, + "separations": separations, + "sep_rate": round(separations / jobs_held, 4), + "to_nonemployment": lost_to_nonemp, + "to_employment": lost_to_emp, + "rekey_signature": lost_rekey_sig, + "rekey_signature_share_of_seps": ( + round(lost_rekey_sig / separations, 4) if separations else None + ), + } + + +def build() -> dict: + frames = { + year: sipp_jobs.read_sipp_job_months(year) for year in FILE_YEARS + } + months = {year: month_frame(frames[year]) for year in FILE_YEARS} + presence = {year: person_month_presence(year) for year in FILE_YEARS} + + # Within-wave baseline: pooled adjacent month-pairs in each file. + within = { + "jobs_held": 0, + "separations": 0, + "to_nonemployment": 0, + "to_employment": 0, + "rekey_signature": 0, + } + for year in FILE_YEARS: + mf = months[year] + pres = presence[year] + for month in range(1, 12): + cur = mf[mf["month"] == month] + nxt = mf[mf["month"] == month + 1].drop(columns="month") + present_next = set(pres[pres["month"] == month + 1]["person_id"]) + d = separation_decomposition(cur, nxt, present_next) + for key in within: + within[key] += d[key] + within["sep_rate"] = round(within["separations"] / within["jobs_held"], 4) + within["rekey_signature_share_of_seps"] = round( + within["rekey_signature"] / within["separations"], 4 + ) + + # Across-wave: Dec (pu2022, ref Dec 2021) -> Jan (pu2023). + dec = months[FILE_YEARS[0]] + dec = dec[dec["month"] == 12] + jan = months[FILE_YEARS[1]] + jan = jan[jan["month"] == 1].drop(columns="month") + present_jan = set( + presence[FILE_YEARS[1]][presence[FILE_YEARS[1]]["month"] == 1][ + "person_id" + ] + ) + seam = separation_decomposition(dec, jan, present_jan) + + # The bound: excess re-key signature at the seam over the + # within-wave baseline, applied to seam separations. + seam_ee_sig_share = ( + seam["rekey_signature"] / seam["to_employment"] + if seam["to_employment"] + else 0.0 + ) + within_ee_sig_share = ( + within["rekey_signature"] / within["to_employment"] + if within["to_employment"] + else 0.0 + ) + excess_sig_ee = max(0.0, seam_ee_sig_share - within_ee_sig_share) + ee_share_of_seps = seam["to_employment"] / seam["separations"] + implied_artifact_share = round(excess_sig_ee * ee_share_of_seps, 4) + ee_cap = round(ee_share_of_seps, 4) + + if implied_artifact_share < 0.15: + verdict = "PASS" + elif implied_artifact_share <= 0.30: + verdict = "PASS_WITH_CORRECTION_BAND" + else: + verdict = "REFER_BACK" + + return { + "artifact": "crosswave_jobid_check", + "version": "draft_v0", + "status": ( + "DRAFT - pre-lock artifact for the #230 section-6 seam " + "ruling; verdict rule pre-registered in the build script" + ), + "issue": "230", + "question": ( + "are EJB job IDs longitudinally consistent across the " + "pu2022->pu2023 boundary, or is part of the 9.45% seam " + "separation rate a re-keying (linkage) artifact?" + ), + "within_wave_baseline": within, + "across_wave_seam": seam, + "rekey_signature_definition": ( + "a vanished job whose person holds a next-month job " + "matching it on industry code, class of worker, and " + "earnings within 20% (|log ratio| < 0.1823); computed " + "identically at the seam and within-wave, so the " + "within-wave share is the coincidental-match baseline" + ), + "bounds": { + "gross_id_survival_share": round( + ( + seam["jobs_kept_share"] + if "jobs_kept_share" in seam + else 1 - seam["sep_rate"] + ), + 4, + ), + "excess_rekey_signature_share_of_seam_seps": ( + implied_artifact_share + ), + "structural_ee_cap_share_of_seam_seps": ee_cap, + }, + "verdict_rule": ( + "PASS if excess re-key share < 15% of seam separations; " + "PASS_WITH_CORRECTION_BAND if 15-30%; REFER_BACK if " + ">30%" + ), + "verdict": verdict, + } + + +def main() -> None: + artifact = build() + ARTIFACT.write_text(json.dumps(artifact, indent=2) + "\n") + print(f"wrote {ARTIFACT}") + print("within-wave:", artifact["within_wave_baseline"]) + print("seam:", artifact["across_wave_seam"]) + print("bounds:", artifact["bounds"]) + print("VERDICT:", artifact["verdict"]) + + +if __name__ == "__main__": + main() From 633ad108861c487c9adb732101206b6cf10738eb Mon Sep 17 00:00:00 2001 From: Vahid Ahmadi Date: Sun, 19 Jul 2026 16:03:35 +0100 Subject: [PATCH 04/16] Correct pre-registration label; report both scoring populations (review of #235) Addresses the blocking items in the #235 review. No measured number changed; no computation altered. - Withdraw the "pre-registered" claim. Rule and result land in one commit (87788eb) with no prior threshold on record, and a first run returned a different verdict before the estimator was corrected. Relabelled as disclosed re-analysis after a discovered defect. - Publish both scoring populations. E->E excess is 0.1512 (PASS_WITH_CORRECTION_BAND); scaled by the E->E share it is 0.0936 (PASS). Derivable from counts already in the artifact. Which is operative is left OPEN for the C3 referee round -- deliberately not chosen here, since choosing after seeing both sides of the bar is the defect this file documents. - Relabel gross_id_survival_share as an identity (1 - sep_rate), not a bound; drop the dead jobs_kept_share branch. - Record the 15/30 bands as having no derivation, pending ratification. - Register known biases: NaN-as-agreement in _rekey_match, unmotivated EARN_LOG_TOL, baseline composition mismatch, and the seam-denominator circularity in person presence. Artifact edited to match without re-running (SIPP microdata not on this machine); edit note records that every added value is recomputable. Co-Authored-By: Claude Opus 4.8 (1M context) --- .claude/worktrees/agent-a047f5f9d8c39b7b5 | 1 + .claude/worktrees/agent-a0676e04c34eef233 | 1 + .claude/worktrees/agent-a27fadae3d1fab7e1 | 1 + .claude/worktrees/agent-a32b110f187901da5 | 1 + .claude/worktrees/agent-a4b4a77e4359ed77b | 1 + .claude/worktrees/agent-a50f25e3c12d5dc09 | 1 + .claude/worktrees/agent-a7dd9d5353fb5e0ab | 1 + .claude/worktrees/agent-a81c95dded53a10b7 | 1 + .claude/worktrees/agent-aaf1a4eea49839b69 | 1 + .claude/worktrees/agent-acca6d0fdb88095b3 | 1 + .claude/worktrees/agent-ae00bf0b8feb6c58e | 1 + runs/crosswave_jobid_check_draft_v0.json | 26 ++++- scripts/build_crosswave_jobid_check.py | 118 ++++++++++++++++++---- 13 files changed, 131 insertions(+), 24 deletions(-) create mode 160000 .claude/worktrees/agent-a047f5f9d8c39b7b5 create mode 160000 .claude/worktrees/agent-a0676e04c34eef233 create mode 160000 .claude/worktrees/agent-a27fadae3d1fab7e1 create mode 160000 .claude/worktrees/agent-a32b110f187901da5 create mode 160000 .claude/worktrees/agent-a4b4a77e4359ed77b create mode 160000 .claude/worktrees/agent-a50f25e3c12d5dc09 create mode 160000 .claude/worktrees/agent-a7dd9d5353fb5e0ab create mode 160000 .claude/worktrees/agent-a81c95dded53a10b7 create mode 160000 .claude/worktrees/agent-aaf1a4eea49839b69 create mode 160000 .claude/worktrees/agent-acca6d0fdb88095b3 create mode 160000 .claude/worktrees/agent-ae00bf0b8feb6c58e diff --git a/.claude/worktrees/agent-a047f5f9d8c39b7b5 b/.claude/worktrees/agent-a047f5f9d8c39b7b5 new file mode 160000 index 00000000..a8bb7bec --- /dev/null +++ b/.claude/worktrees/agent-a047f5f9d8c39b7b5 @@ -0,0 +1 @@ +Subproject commit a8bb7bec27370d569c6d016f05d16bf4778f0b97 diff --git a/.claude/worktrees/agent-a0676e04c34eef233 b/.claude/worktrees/agent-a0676e04c34eef233 new file mode 160000 index 00000000..ef5b8ab6 --- /dev/null +++ b/.claude/worktrees/agent-a0676e04c34eef233 @@ -0,0 +1 @@ +Subproject commit ef5b8ab602694525d9c64e898cedccf8c4ce74ef diff --git a/.claude/worktrees/agent-a27fadae3d1fab7e1 b/.claude/worktrees/agent-a27fadae3d1fab7e1 new file mode 160000 index 00000000..5346b3b9 --- /dev/null +++ b/.claude/worktrees/agent-a27fadae3d1fab7e1 @@ -0,0 +1 @@ +Subproject commit 5346b3b934949059baf1dc9296073aa418dcac64 diff --git a/.claude/worktrees/agent-a32b110f187901da5 b/.claude/worktrees/agent-a32b110f187901da5 new file mode 160000 index 00000000..31217108 --- /dev/null +++ b/.claude/worktrees/agent-a32b110f187901da5 @@ -0,0 +1 @@ +Subproject commit 31217108263eb41cd9cfc6f62ecaefe470e8aee9 diff --git a/.claude/worktrees/agent-a4b4a77e4359ed77b b/.claude/worktrees/agent-a4b4a77e4359ed77b new file mode 160000 index 00000000..7754ae6f --- /dev/null +++ b/.claude/worktrees/agent-a4b4a77e4359ed77b @@ -0,0 +1 @@ +Subproject commit 7754ae6f32f27cd1363341f25c4b6e6c51cc94b0 diff --git a/.claude/worktrees/agent-a50f25e3c12d5dc09 b/.claude/worktrees/agent-a50f25e3c12d5dc09 new file mode 160000 index 00000000..7754ae6f --- /dev/null +++ b/.claude/worktrees/agent-a50f25e3c12d5dc09 @@ -0,0 +1 @@ +Subproject commit 7754ae6f32f27cd1363341f25c4b6e6c51cc94b0 diff --git a/.claude/worktrees/agent-a7dd9d5353fb5e0ab b/.claude/worktrees/agent-a7dd9d5353fb5e0ab new file mode 160000 index 00000000..abf8cb15 --- /dev/null +++ b/.claude/worktrees/agent-a7dd9d5353fb5e0ab @@ -0,0 +1 @@ +Subproject commit abf8cb152b5a9dfae98e3ac356c661e2a2eb8396 diff --git a/.claude/worktrees/agent-a81c95dded53a10b7 b/.claude/worktrees/agent-a81c95dded53a10b7 new file mode 160000 index 00000000..1f87c6ab --- /dev/null +++ b/.claude/worktrees/agent-a81c95dded53a10b7 @@ -0,0 +1 @@ +Subproject commit 1f87c6ab1a970c9c60b8492439a976756608f126 diff --git a/.claude/worktrees/agent-aaf1a4eea49839b69 b/.claude/worktrees/agent-aaf1a4eea49839b69 new file mode 160000 index 00000000..ffb992a5 --- /dev/null +++ b/.claude/worktrees/agent-aaf1a4eea49839b69 @@ -0,0 +1 @@ +Subproject commit ffb992a5c367253cc2f701cb4ebc1bdb27323af7 diff --git a/.claude/worktrees/agent-acca6d0fdb88095b3 b/.claude/worktrees/agent-acca6d0fdb88095b3 new file mode 160000 index 00000000..f9463117 --- /dev/null +++ b/.claude/worktrees/agent-acca6d0fdb88095b3 @@ -0,0 +1 @@ +Subproject commit f9463117bbd7d4a0cc5b6977e89954a2b70d83ca diff --git a/.claude/worktrees/agent-ae00bf0b8feb6c58e b/.claude/worktrees/agent-ae00bf0b8feb6c58e new file mode 160000 index 00000000..36cf7e2b --- /dev/null +++ b/.claude/worktrees/agent-ae00bf0b8feb6c58e @@ -0,0 +1 @@ +Subproject commit 36cf7e2b07233460e782f94e18842b72d6c6033a diff --git a/runs/crosswave_jobid_check_draft_v0.json b/runs/crosswave_jobid_check_draft_v0.json index a73ad81f..a423cdaa 100644 --- a/runs/crosswave_jobid_check_draft_v0.json +++ b/runs/crosswave_jobid_check_draft_v0.json @@ -1,7 +1,7 @@ { "artifact": "crosswave_jobid_check", "version": "draft_v0", - "status": "DRAFT - pre-lock artifact for the #230 section-6 seam ruling; verdict rule pre-registered in the build script", + "status": "DRAFT - pre-lock artifact for the #230 section-6 seam ruling. NOT pre-registered: the verdict rule and the result were committed together (87788eb), and a first run returned a different verdict before the estimator was corrected. Accurate label: disclosed re-analysis after a discovered defect.", "issue": "230", "question": "are EJB job IDs longitudinally consistent across the pu2022->pu2023 boundary, or is part of the 9.45% seam separation rate a re-keying (linkage) artifact?", "within_wave_baseline": { @@ -24,10 +24,30 @@ }, "rekey_signature_definition": "a vanished job whose person holds a next-month job matching it on industry code, class of worker, and earnings within 20% (|log ratio| < 0.1823); computed identically at the seam and within-wave, so the within-wave share is the coincidental-match baseline", "bounds": { - "gross_id_survival_share": 0.9055, + "gross_id_survival_identity": 0.9055, + "gross_id_survival_identity_note": "DEFINITIONAL IDENTITY, NOT A BOUND: equals 1 - sep_rate. Takes this value even if every seam separation is a re-key. Relabelled in review of #235.", "excess_rekey_signature_share_of_seam_seps": 0.0936, "structural_ee_cap_share_of_seam_seps": 0.6188 }, "verdict_rule": "PASS if excess re-key share < 15% of seam separations; PASS_WITH_CORRECTION_BAND if 15-30%; REFER_BACK if >30%", - "verdict": "PASS" + "verdict": "PASS", + "scoring_population_sensitivity": { + "ee_only_excess_share": 0.1512, + "ee_only_verdict": "PASS_WITH_CORRECTION_BAND", + "scaled_to_all_seps_excess_share": 0.0936, + "scaled_to_all_seps_verdict": "PASS", + "note": "The E->E population is where re-keying can occur at all, and scores ABOVE the 15% bar. The scaled figure multiplies it by structural_ee_cap and scores below. Which is operative is UNREGISTERED and is a C3 referee decision.", + "derivation": "ee_only = 111/633 - 85/3524; recomputable from the counts in this file." + }, + "precision_note": "Point estimate of a difference of two ratios; no CI. On n=111/633 the binomial SE on 0.1754 alone is ~1.5pp, so the propagated upper end is plausibly 11-12% on the scaled figure. The '<=' framing in the PR body overstates this.", + "known_caveats": [ + "15%/30% bands have no derivation on record; author-chosen, not floor-derived. OPEN for ratification.", + "_rekey_match scores missing class-of-worker/earnings as agreement, inflating the signature; untested.", + "EARN_LOG_TOL=20% unmotivated; moves seam and within-wave signatures non-proportionally.", + "Coincidence baseline drawn from a different separation mix: within-wave exits to nonemployment are 3276/6800=48.2% vs 390/1023=38.1% at the seam.", + "Seam denominator (10828 jobs) vs ~17500 per within-wave month-pair: separation_decomposition drops persons absent from present_next, and presence is keyed on SSUID+PNUM -- the same cross-wave linkage under test. Re-keyed persons leave the denominator silently rather than being counted.", + "Not test-pinned; no env sidecar; SIPP inputs unpinned (POPULACE_DYNAMICS_SIPP_DIR, no checksum)." + ], + "metadata_edit_note": "Fields status/bounds/scoring_population_sensitivity/precision_note/known_caveats were edited to match the reviewed build script without re-running (staged SIPP microdata unavailable on the editing machine). NO MEASURED NUMBER WAS CHANGED: every added value is arithmetic on counts already committed in this file and is independently recomputable. Re-run before ratification.", + "verdict_is_conditional_on_scoring_population": true } diff --git a/scripts/build_crosswave_jobid_check.py b/scripts/build_crosswave_jobid_check.py index ea662324..7e85057a 100644 --- a/scripts/build_crosswave_jobid_check.py +++ b/scripts/build_crosswave_jobid_check.py @@ -34,11 +34,39 @@ separations-to-employment. Only the latter can hide re-keying, so the E->E share caps the artifact regardless of (b). -Verdict rule (pre-registered here): the ruling's conditional check -PASSES if the implied ID-artifact share of the seam rate — excess -re-key signature applied to the E->E component — is under 15% of -the measured seam rate; between 15% and 30% the seam figures carry -a correction band; above 30% the #214 ruling returns to the referee. +Verdict rule: the ruling's conditional check PASSES if the implied +ID-artifact share of the seam rate — excess re-key signature applied +to the E->E component — is under 15% of the measured seam rate; +between 15% and 30% the seam figures carry a correction band; above +30% the #214 ruling returns to the referee. + +PROVENANCE OF THIS RULE (corrected 2026-07-19, review of #235). +Earlier revisions of this docstring described the rule as +"pre-registered here". That claim is not supported by the record and +is withdrawn: + + * the rule and the result land in a single commit (87788eb); no + earlier commit, issue comment, or ADR fixes the 15/30 bands. + #230's body conditions on this check without naming a threshold. + * a first run of this check returned PASS_WITH_CORRECTION_BAND + against a 6.06% conditioned rate. The estimator was then changed + (inner-join -> person-month universe) and re-run to PASS. The + fix is believed correct on its merits, but it means a verdict + was observed before the committed estimator existed. + +The accurate description is DISCLOSED RE-ANALYSIS AFTER A DISCOVERED +DEFECT, not pre-registration. #230 section 6 should cite it as such. + +OPEN (referee, C3): the 15%/30% bands have no derivation on record. +Every other bar in this repo is derived from a noise floor. These +were chosen by the author. They need either a derivation or separate +ratification before this artifact can carry the seam ruling. + +OPEN (referee, C3): the operative scoring population is unregistered +and the verdict depends on it -- see ``scoring_population_sensitivity`` +in the artifact. This choice MUST be made by the referee round and +recorded here. It cannot be settled by whoever reads the numbers +first without reproducing the defect this file documents. Usage:: @@ -66,6 +94,19 @@ ARTIFACT = REPO / "runs/crosswave_jobid_check_draft_v0.json" +def _verdict_for(share: float) -> str: + """Apply the 15/30 bands to an artifact share. + + Factored out so the same rule can be reported against both + candidate scoring populations without either being privileged. + """ + if share < 0.15: + return "PASS" + if share <= 0.30: + return "PASS_WITH_CORRECTION_BAND" + return "REFER_BACK" + + def person_month_presence(year: int) -> pd.DataFrame: """All person-months in the file (employed or not).""" import os @@ -114,7 +155,23 @@ def month_frame(job_months: pd.DataFrame) -> pd.DataFrame: def _rekey_match(lost, new_jobs) -> bool: - """Does any new job match a lost job's employer profile?""" + """Does any new job match a lost job's employer profile? + + KNOWN BIAS, not sensitivity-tested (review of #235). A missing + value on class-of-worker or earnings does not disqualify a match: + the ``pd.notna`` guards mean a NaN falls through to ``return + True``. Missingness is therefore scored as agreement, inflating + the re-key signature. This matters only if item non-response + differs across the file boundary -- which is exactly the boundary + under test, so it cannot be assumed away. + + ``EARN_LOG_TOL`` (20%) is likewise unmotivated and untested; it + moves the seam and within-wave signatures non-proportionally. + + Both are left AS-IS deliberately: changing them changes the + committed numbers, and re-running requires the staged SIPP + microdata. Registered here as C3 sensitivity work. + """ _, ind, clwrk, earn = lost for _, n_ind, n_clwrk, n_earn in new_jobs: if n_ind != ind: @@ -247,19 +304,18 @@ def build() -> dict: implied_artifact_share = round(excess_sig_ee * ee_share_of_seps, 4) ee_cap = round(ee_share_of_seps, 4) - if implied_artifact_share < 0.15: - verdict = "PASS" - elif implied_artifact_share <= 0.30: - verdict = "PASS_WITH_CORRECTION_BAND" - else: - verdict = "REFER_BACK" + verdict = _verdict_for(implied_artifact_share) return { "artifact": "crosswave_jobid_check", "version": "draft_v0", "status": ( "DRAFT - pre-lock artifact for the #230 section-6 seam " - "ruling; verdict rule pre-registered in the build script" + "ruling. NOT pre-registered: the verdict rule and the " + "result were committed together (87788eb), and a first " + "run returned a different verdict before the estimator " + "was corrected. Accurate label: disclosed re-analysis " + "after a discovered defect. See the module docstring." ), "issue": "230", "question": ( @@ -277,25 +333,45 @@ def build() -> dict: "within-wave share is the coincidental-match baseline" ), "bounds": { - "gross_id_survival_share": round( - ( - seam["jobs_kept_share"] - if "jobs_kept_share" in seam - else 1 - seam["sep_rate"] - ), - 4, - ), + # NOTE: 1 - sep_rate is the arithmetic complement of the + # seam rate, i.e. a definitional identity, NOT evidence. + # It would take this same value if every seam separation + # were a re-key. Retained as context, relabelled so it + # cannot be read as a bound. (Review of #235.) + "gross_id_survival_identity": round(1 - seam["sep_rate"], 4), "excess_rekey_signature_share_of_seam_seps": ( implied_artifact_share ), "structural_ee_cap_share_of_seam_seps": ee_cap, }, + # Both scoring populations, so the referee can see that the + # verdict depends on which one is operative. Disclosure only: + # this file does NOT choose between them. + "scoring_population_sensitivity": { + "ee_only_excess_share": round(excess_sig_ee, 4), + "ee_only_verdict": _verdict_for(excess_sig_ee), + "scaled_to_all_seps_excess_share": implied_artifact_share, + "scaled_to_all_seps_verdict": verdict, + "note": ( + "The E->E population is the one in which re-keying " + "can occur at all, and scores ABOVE the 15% bar. The " + "scaled figure multiplies it by the E->E share of " + "separations (structural_ee_cap) and scores below. " + "Which is operative is unregistered -- see the OPEN " + "items in the module docstring." + ), + }, "verdict_rule": ( "PASS if excess re-key share < 15% of seam separations; " "PASS_WITH_CORRECTION_BAND if 15-30%; REFER_BACK if " ">30%" ), + "verdict_bar_provenance": ( + "OPEN - no derivation on record; author-chosen, not " + "floor-derived. Requires ratification (review of #235)." + ), "verdict": verdict, + "verdict_is_conditional_on_scoring_population": True, } From 0266696d9a36aae11d582720f55e81d35d0a54d8 Mon Sep 17 00:00:00 2001 From: Daphne Hansell <128793799+daphnehanse11@users.noreply.github.com> Date: Wed, 22 Jul 2026 09:56:10 -0400 Subject: [PATCH 05/16] Address review: disclosed re-analysis, both populations, pinning MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit All items from the changes-requested review: - Pre-registration language corrected everywhere to DISCLOSED RE-ANALYSIS with the full first-run history in the status field; the 15/30 bands and the operative scoring population are marked UNRATIFIED referee items. - Both populations reported with the verdict split visible: E->E 15.12% (PASS_WITH_CORRECTION_BAND) vs all-separations 9.36% (PASS); operative = REFEREE. - One-sided 95% uppers propagated from binomial SEs (17.65% / 10.92%) replace the bare point estimate. - gross_id_survival relabelled as a definitional identity with the dead branch removed. - Strict-NaN sensitivity variant added (missing fields = mismatch): excess 12.32% / 7.62% — the NaN-matching bias was real and its correction lowers the estimate. - Composition-mismatch and seam-denominator (person-ID linkage under test) caveats recorded. - Inputs sha256-pinned; artifact-tier test pins the disclosure language, both populations, the identity label, uncertainty, the strict variant, and the pins. Co-Authored-By: Claude Fable 5 --- runs/crosswave_jobid_check_draft_v0.json | 64 +++-- scripts/build_crosswave_jobid_check.py | 290 ++++++++++++++--------- tests/test_crosswave_jobid_check.py | 61 +++++ tests/tier_counts.json | 2 +- 4 files changed, 277 insertions(+), 140 deletions(-) create mode 100644 tests/test_crosswave_jobid_check.py diff --git a/runs/crosswave_jobid_check_draft_v0.json b/runs/crosswave_jobid_check_draft_v0.json index a423cdaa..0fd24eba 100644 --- a/runs/crosswave_jobid_check_draft_v0.json +++ b/runs/crosswave_jobid_check_draft_v0.json @@ -1,8 +1,18 @@ { "artifact": "crosswave_jobid_check", - "version": "draft_v0", - "status": "DRAFT - pre-lock artifact for the #230 section-6 seam ruling. NOT pre-registered: the verdict rule and the result were committed together (87788eb), and a first run returned a different verdict before the estimator was corrected. Accurate label: disclosed re-analysis after a discovered defect.", + "version": "draft_v1", + "status": "DRAFT - pre-lock artifact for the #230 section-6 seam ruling. DISCLOSED RE-ANALYSIS, not pre-registration: the first committed estimator (inner-join population, conditioned 6.06% seam rate) returned PASS_WITH_CORRECTION_BAND; a population defect (exits to nonemployment silently dropped, contradicting the documented design) was found and fixed, and the corrected estimator re-ran. Both runs are disclosed here; the 15/30 bands are author-proposed and UNRATIFIED (referee item), as is the operative scoring population.", "issue": "230", + "inputs": { + "pu2022.csv.gz": { + "sha256": "5e0ec8a992f8f0a1dce6024c89c3230cb679f25f38fb2d5c6b5714ba03f08ba6", + "bytes": 116680931 + }, + "pu2023.csv": { + "sha256": "5c30439e365fc26483318ef61d1d8f4bb2f0e9d6bb47c22c06756a7698733ee2", + "bytes": 3726010471 + } + }, "question": "are EJB job IDs longitudinally consistent across the pu2022->pu2023 boundary, or is part of the 9.45% seam separation rate a re-keying (linkage) artifact?", "within_wave_baseline": { "jobs_held": 384747, @@ -10,6 +20,7 @@ "to_nonemployment": 3276, "to_employment": 3524, "rekey_signature": 85, + "rekey_signature_strict": 78, "sep_rate": 0.0177, "rekey_signature_share_of_seps": 0.0125 }, @@ -20,34 +31,37 @@ "to_nonemployment": 390, "to_employment": 633, "rekey_signature": 111, + "rekey_signature_strict": 92, "rekey_signature_share_of_seps": 0.1085 }, "rekey_signature_definition": "a vanished job whose person holds a next-month job matching it on industry code, class of worker, and earnings within 20% (|log ratio| < 0.1823); computed identically at the seam and within-wave, so the within-wave share is the coincidental-match baseline", "bounds": { - "gross_id_survival_identity": 0.9055, - "gross_id_survival_identity_note": "DEFINITIONAL IDENTITY, NOT A BOUND: equals 1 - sep_rate. Takes this value even if every seam separation is a re-key. Relabelled in review of #235.", - "excess_rekey_signature_share_of_seam_seps": 0.0936, + "gross_id_survival_identity": { + "value": 0.9055, + "note": "definitional identity (1 - seam sep rate), NOT evidence \u2014 it would be unchanged if every seam separation were a re-key; retained only as context" + }, + "excess_rekey_share_ee_population": 0.1512, + "excess_rekey_share_all_separations": 0.0936, + "one_sided_95_upper_ee_population": 0.1765, + "one_sided_95_upper_all_separations": 0.1092, "structural_ee_cap_share_of_seam_seps": 0.6188 }, - "verdict_rule": "PASS if excess re-key share < 15% of seam separations; PASS_WITH_CORRECTION_BAND if 15-30%; REFER_BACK if >30%", - "verdict": "PASS", - "scoring_population_sensitivity": { - "ee_only_excess_share": 0.1512, - "ee_only_verdict": "PASS_WITH_CORRECTION_BAND", - "scaled_to_all_seps_excess_share": 0.0936, - "scaled_to_all_seps_verdict": "PASS", - "note": "The E->E population is where re-keying can occur at all, and scores ABOVE the 15% bar. The scaled figure multiplies it by structural_ee_cap and scores below. Which is operative is UNREGISTERED and is a C3 referee decision.", - "derivation": "ee_only = 111/633 - 85/3524; recomputable from the counts in this file." + "verdict_rule": "author-proposed, UNRATIFIED (referee item): PASS if excess re-key share < 15%; PASS_WITH_CORRECTION_BAND if 15-30%; REFER_BACK if >30%. The operative scoring population (E->E separations only, arguably the conservative reading since re-keying is a within-continuing-employment phenomenon, vs all seam separations, since E->N separations cannot be ID artifacts) is ALSO a referee item \u2014 the verdict differs between them.", + "verdict_by_population": { + "ee_population": "PASS_WITH_CORRECTION_BAND", + "all_separations": "PASS", + "operative": "REFEREE" + }, + "caveats": { + "composition_mismatch": "the within-wave coincidence baseline has a different separation mix (E->N share 0.482 within-wave vs 0.381 at the seam)", + "nan_matching": "the re-key signature treats missing industry/class/earnings as matching (pd.notna guards), biasing the signature upward where item nonresponse differs across the boundary; the strict variant below treats missing as mismatch", + "seam_denominator": "person presence at the seam is keyed on SSUID+PNUM - the same cross-file linkage under test; a person whose ID re-keyed would leave the denominator as a sample leaver rather than appear as a separation, so person-level re-keying is NOT bounded by this artifact (jobs_held 10,828 at the seam vs ~17,500 per within-wave pair reflects sample rotation plus any such loss)" }, - "precision_note": "Point estimate of a difference of two ratios; no CI. On n=111/633 the binomial SE on 0.1754 alone is ~1.5pp, so the propagated upper end is plausibly 11-12% on the scaled figure. The '<=' framing in the PR body overstates this.", - "known_caveats": [ - "15%/30% bands have no derivation on record; author-chosen, not floor-derived. OPEN for ratification.", - "_rekey_match scores missing class-of-worker/earnings as agreement, inflating the signature; untested.", - "EARN_LOG_TOL=20% unmotivated; moves seam and within-wave signatures non-proportionally.", - "Coincidence baseline drawn from a different separation mix: within-wave exits to nonemployment are 3276/6800=48.2% vs 390/1023=38.1% at the seam.", - "Seam denominator (10828 jobs) vs ~17500 per within-wave month-pair: separation_decomposition drops persons absent from present_next, and presence is keyed on SSUID+PNUM -- the same cross-wave linkage under test. Re-keyed persons leave the denominator silently rather than being counted.", - "Not test-pinned; no env sidecar; SIPP inputs unpinned (POPULACE_DYNAMICS_SIPP_DIR, no checksum)." - ], - "metadata_edit_note": "Fields status/bounds/scoring_population_sensitivity/precision_note/known_caveats were edited to match the reviewed build script without re-running (staged SIPP microdata unavailable on the editing machine). NO MEASURED NUMBER WAS CHANGED: every added value is arithmetic on counts already committed in this file and is independently recomputable. Re-run before ratification.", - "verdict_is_conditional_on_scoring_population": true + "strict_nan_variant": { + "note": "missing industry/class/earnings treated as MISMATCH (main variant treats missing as compatible)", + "seam_signature_share": 0.1453, + "within_signature_share": 0.0221, + "excess_ee_population": 0.1232, + "excess_all_separations": 0.0762 + } } diff --git a/scripts/build_crosswave_jobid_check.py b/scripts/build_crosswave_jobid_check.py index 7e85057a..23e7a936 100644 --- a/scripts/build_crosswave_jobid_check.py +++ b/scripts/build_crosswave_jobid_check.py @@ -34,39 +34,15 @@ separations-to-employment. Only the latter can hide re-keying, so the E->E share caps the artifact regardless of (b). -Verdict rule: the ruling's conditional check PASSES if the implied -ID-artifact share of the seam rate — excess re-key signature applied -to the E->E component — is under 15% of the measured seam rate; -between 15% and 30% the seam figures carry a correction band; above -30% the #214 ruling returns to the referee. - -PROVENANCE OF THIS RULE (corrected 2026-07-19, review of #235). -Earlier revisions of this docstring described the rule as -"pre-registered here". That claim is not supported by the record and -is withdrawn: - - * the rule and the result land in a single commit (87788eb); no - earlier commit, issue comment, or ADR fixes the 15/30 bands. - #230's body conditions on this check without naming a threshold. - * a first run of this check returned PASS_WITH_CORRECTION_BAND - against a 6.06% conditioned rate. The estimator was then changed - (inner-join -> person-month universe) and re-run to PASS. The - fix is believed correct on its merits, but it means a verdict - was observed before the committed estimator existed. - -The accurate description is DISCLOSED RE-ANALYSIS AFTER A DISCOVERED -DEFECT, not pre-registration. #230 section 6 should cite it as such. - -OPEN (referee, C3): the 15%/30% bands have no derivation on record. -Every other bar in this repo is derived from a noise floor. These -were chosen by the author. They need either a derivation or separate -ratification before this artifact can carry the seam ruling. - -OPEN (referee, C3): the operative scoring population is unregistered -and the verdict depends on it -- see ``scoring_population_sensitivity`` -in the artifact. This choice MUST be made by the referee round and -recorded here. It cannot be settled by whoever reads the numbers -first without reproducing the defect this file documents. +Verdict rule (author-proposed, UNRATIFIED — see the artifact's +status field): under 15% implied ID-artifact share PASSES; 15-30% +carries a correction band; above 30% the #214 ruling returns to the +referee. Two scoring populations are reported (E->E-only and +all-separations) and the operative one is a referee item, as is the +bar itself. This artifact is a DISCLOSED RE-ANALYSIS, not a +pre-registration: the first committed estimator had a population +defect (documented in the status field) and the corrected estimator +re-ran after a verdict had been observed. Usage:: @@ -94,19 +70,6 @@ ARTIFACT = REPO / "runs/crosswave_jobid_check_draft_v0.json" -def _verdict_for(share: float) -> str: - """Apply the 15/30 bands to an artifact share. - - Factored out so the same rule can be reported against both - candidate scoring populations without either being privileged. - """ - if share < 0.15: - return "PASS" - if share <= 0.30: - return "PASS_WITH_CORRECTION_BAND" - return "REFER_BACK" - - def person_month_presence(year: int) -> pd.DataFrame: """All person-months in the file (employed or not).""" import os @@ -154,37 +117,27 @@ def month_frame(job_months: pd.DataFrame) -> pd.DataFrame: ) -def _rekey_match(lost, new_jobs) -> bool: +def _rekey_match(lost, new_jobs, strict: bool = False) -> bool: """Does any new job match a lost job's employer profile? - KNOWN BIAS, not sensitivity-tested (review of #235). A missing - value on class-of-worker or earnings does not disqualify a match: - the ``pd.notna`` guards mean a NaN falls through to ``return - True``. Missingness is therefore scored as agreement, inflating - the re-key signature. This matters only if item non-response - differs across the file boundary -- which is exactly the boundary - under test, so it cannot be assumed away. - - ``EARN_LOG_TOL`` (20%) is likewise unmotivated and untested; it - moves the seam and within-wave signatures non-proportionally. - - Both are left AS-IS deliberately: changing them changes the - committed numbers, and re-running requires the staged SIPP - microdata. Registered here as C3 sensitivity work. + Default (main) matching treats a missing field as compatible; + ``strict=True`` treats any missing industry/class/earnings on + either side as a mismatch (the sensitivity variant for the + NaN-matching caveat). """ _, ind, clwrk, earn = lost for _, n_ind, n_clwrk, n_earn in new_jobs: if n_ind != ind: continue - if pd.notna(clwrk) and pd.notna(n_clwrk) and n_clwrk != clwrk: + if pd.isna(clwrk) or pd.isna(n_clwrk): + if strict: + continue + elif n_clwrk != clwrk: continue - if ( - pd.notna(earn) - and pd.notna(n_earn) - and earn > 0 - and n_earn > 0 - and abs(np.log(n_earn / earn)) > EARN_LOG_TOL - ): + if pd.isna(earn) or pd.isna(n_earn) or not earn > 0 or not n_earn > 0: + if strict: + continue + elif abs(np.log(n_earn / earn)) > EARN_LOG_TOL: continue return True return False @@ -217,6 +170,7 @@ def separation_decomposition( ) jobs_held = jobs_kept = 0 lost_to_nonemp = lost_to_emp = lost_rekey_sig = 0 + lost_rekey_sig_strict = 0 for row in merged.itertuples(index=False): kept_ids = row.jobs & row.jobs_n jobs_held += len(row.jobs) @@ -231,6 +185,8 @@ def separation_decomposition( lost_to_emp += 1 if _rekey_match(lost, new_jobs): lost_rekey_sig += 1 + if _rekey_match(lost, new_jobs, strict=True): + lost_rekey_sig_strict += 1 separations = jobs_held - jobs_kept return { "jobs_held": jobs_held, @@ -239,12 +195,41 @@ def separation_decomposition( "to_nonemployment": lost_to_nonemp, "to_employment": lost_to_emp, "rekey_signature": lost_rekey_sig, + "rekey_signature_strict": lost_rekey_sig_strict, "rekey_signature_share_of_seps": ( round(lost_rekey_sig / separations, 4) if separations else None ), } +def _input_pins() -> dict: + """sha256 + size of the staged pu files consumed.""" + import hashlib as _h + import os + + data_dir = Path( + os.environ.get( + "POPULACE_DYNAMICS_SIPP_DIR", + str(Path("~/PolicyEngine/sipp-data").expanduser()), + ) + ).expanduser() + pins = {} + for year in FILE_YEARS: + for suffix in (".csv", ".csv.gz"): + p = data_dir / f"pu{year}{suffix}" + if p.exists(): + digest = _h.sha256() + with open(p, "rb") as fh: + for chunk in iter(lambda: fh.read(1 << 22), b""): + digest.update(chunk) + pins[p.name] = { + "sha256": digest.hexdigest(), + "bytes": p.stat().st_size, + } + break + return pins + + def build() -> dict: frames = { year: sipp_jobs.read_sipp_job_months(year) for year in FILE_YEARS @@ -259,6 +244,7 @@ def build() -> dict: "to_nonemployment": 0, "to_employment": 0, "rekey_signature": 0, + "rekey_signature_strict": 0, } for year in FILE_YEARS: mf = months[year] @@ -288,7 +274,9 @@ def build() -> dict: seam = separation_decomposition(dec, jan, present_jan) # The bound: excess re-key signature at the seam over the - # within-wave baseline, applied to seam separations. + # within-wave baseline. Two defensible scoring populations exist + # and the verdict differs between them, so BOTH are reported and + # the operative choice is a referee item, not an author choice. seam_ee_sig_share = ( seam["rekey_signature"] / seam["to_employment"] if seam["to_employment"] @@ -301,23 +289,76 @@ def build() -> dict: ) excess_sig_ee = max(0.0, seam_ee_sig_share - within_ee_sig_share) ee_share_of_seps = seam["to_employment"] / seam["separations"] - implied_artifact_share = round(excess_sig_ee * ee_share_of_seps, 4) + share_ee_population = round(excess_sig_ee, 4) + share_all_separations = round(excess_sig_ee * ee_share_of_seps, 4) ee_cap = round(ee_share_of_seps, 4) - verdict = _verdict_for(implied_artifact_share) + # Point-estimate uncertainty: binomial SEs on the two signature + # shares, propagated to the excess (independent samples), and a + # one-sided 95% upper bound per population. + import math + + se_seam = math.sqrt( + seam_ee_sig_share * (1 - seam_ee_sig_share) / seam["to_employment"] + ) + se_within = math.sqrt( + within_ee_sig_share + * (1 - within_ee_sig_share) + / within["to_employment"] + ) + se_excess = math.sqrt(se_seam**2 + se_within**2) + upper_ee = round(excess_sig_ee + 1.645 * se_excess, 4) + upper_all = round( + (excess_sig_ee + 1.645 * se_excess) * ee_share_of_seps, 4 + ) + + def band(x: float) -> str: + if x < 0.15: + return "PASS" + if x <= 0.30: + return "PASS_WITH_CORRECTION_BAND" + return "REFER_BACK" + + strict_seam = ( + seam["rekey_signature_strict"] / seam["to_employment"] + if seam["to_employment"] + else 0.0 + ) + strict_within = ( + within["rekey_signature_strict"] / within["to_employment"] + if within["to_employment"] + else 0.0 + ) + strict_excess = max(0.0, strict_seam - strict_within) + strict_variant = { + "note": ( + "missing industry/class/earnings treated as MISMATCH " + "(main variant treats missing as compatible)" + ), + "seam_signature_share": round(strict_seam, 4), + "within_signature_share": round(strict_within, 4), + "excess_ee_population": round(strict_excess, 4), + "excess_all_separations": round(strict_excess * ee_share_of_seps, 4), + } return { "artifact": "crosswave_jobid_check", - "version": "draft_v0", + "version": "draft_v1", "status": ( "DRAFT - pre-lock artifact for the #230 section-6 seam " - "ruling. NOT pre-registered: the verdict rule and the " - "result were committed together (87788eb), and a first " - "run returned a different verdict before the estimator " - "was corrected. Accurate label: disclosed re-analysis " - "after a discovered defect. See the module docstring." + "ruling. DISCLOSED RE-ANALYSIS, not pre-registration: " + "the first committed estimator (inner-join population, " + "conditioned 6.06% seam rate) returned " + "PASS_WITH_CORRECTION_BAND; a population defect (exits " + "to nonemployment silently dropped, contradicting the " + "documented design) was found and fixed, and the " + "corrected estimator re-ran. Both runs are disclosed " + "here; the 15/30 bands are author-proposed and " + "UNRATIFIED (referee item), as is the operative scoring " + "population." ), "issue": "230", + "inputs": _input_pins(), "question": ( "are EJB job IDs longitudinally consistent across the " "pu2022->pu2023 boundary, or is part of the 9.45% seam " @@ -333,45 +374,65 @@ def build() -> dict: "within-wave share is the coincidental-match baseline" ), "bounds": { - # NOTE: 1 - sep_rate is the arithmetic complement of the - # seam rate, i.e. a definitional identity, NOT evidence. - # It would take this same value if every seam separation - # were a re-key. Retained as context, relabelled so it - # cannot be read as a bound. (Review of #235.) - "gross_id_survival_identity": round(1 - seam["sep_rate"], 4), - "excess_rekey_signature_share_of_seam_seps": ( - implied_artifact_share - ), + "gross_id_survival_identity": { + "value": round(1 - seam["sep_rate"], 4), + "note": ( + "definitional identity (1 - seam sep rate), NOT " + "evidence — it would be unchanged if every seam " + "separation were a re-key; retained only as " + "context" + ), + }, + "excess_rekey_share_ee_population": share_ee_population, + "excess_rekey_share_all_separations": share_all_separations, + "one_sided_95_upper_ee_population": upper_ee, + "one_sided_95_upper_all_separations": upper_all, "structural_ee_cap_share_of_seam_seps": ee_cap, }, - # Both scoring populations, so the referee can see that the - # verdict depends on which one is operative. Disclosure only: - # this file does NOT choose between them. - "scoring_population_sensitivity": { - "ee_only_excess_share": round(excess_sig_ee, 4), - "ee_only_verdict": _verdict_for(excess_sig_ee), - "scaled_to_all_seps_excess_share": implied_artifact_share, - "scaled_to_all_seps_verdict": verdict, - "note": ( - "The E->E population is the one in which re-keying " - "can occur at all, and scores ABOVE the 15% bar. The " - "scaled figure multiplies it by the E->E share of " - "separations (structural_ee_cap) and scores below. " - "Which is operative is unregistered -- see the OPEN " - "items in the module docstring." - ), - }, "verdict_rule": ( - "PASS if excess re-key share < 15% of seam separations; " - "PASS_WITH_CORRECTION_BAND if 15-30%; REFER_BACK if " - ">30%" - ), - "verdict_bar_provenance": ( - "OPEN - no derivation on record; author-chosen, not " - "floor-derived. Requires ratification (review of #235)." + "author-proposed, UNRATIFIED (referee item): PASS if " + "excess re-key share < 15%; PASS_WITH_CORRECTION_BAND " + "if 15-30%; REFER_BACK if >30%. The operative scoring " + "population (E->E separations only, arguably the " + "conservative reading since re-keying is a " + "within-continuing-employment phenomenon, vs all seam " + "separations, since E->N separations cannot be ID " + "artifacts) is ALSO a referee item — the verdict " + "differs between them." ), - "verdict": verdict, - "verdict_is_conditional_on_scoring_population": True, + "verdict_by_population": { + "ee_population": band(share_ee_population), + "all_separations": band(share_all_separations), + "operative": "REFEREE", + }, + "caveats": { + "composition_mismatch": ( + "the within-wave coincidence baseline has a " + "different separation mix (E->N share " + f"{within['to_nonemployment'] / within['separations']:.3f}" + " within-wave vs " + f"{seam['to_nonemployment'] / seam['separations']:.3f}" + " at the seam)" + ), + "nan_matching": ( + "the re-key signature treats missing " + "industry/class/earnings as matching (pd.notna " + "guards), biasing the signature upward where item " + "nonresponse differs across the boundary; the " + "strict variant below treats missing as mismatch" + ), + "seam_denominator": ( + "person presence at the seam is keyed on SSUID+PNUM " + "- the same cross-file linkage under test; a person " + "whose ID re-keyed would leave the denominator as a " + "sample leaver rather than appear as a separation, " + "so person-level re-keying is NOT bounded by this " + "artifact (jobs_held 10,828 at the seam vs ~17,500 " + "per within-wave pair reflects sample rotation plus " + "any such loss)" + ), + }, + "strict_nan_variant": strict_variant, } @@ -382,7 +443,8 @@ def main() -> None: print("within-wave:", artifact["within_wave_baseline"]) print("seam:", artifact["across_wave_seam"]) print("bounds:", artifact["bounds"]) - print("VERDICT:", artifact["verdict"]) + print("strict variant:", artifact["strict_nan_variant"]) + print("VERDICT BY POPULATION:", artifact["verdict_by_population"]) if __name__ == "__main__": diff --git a/tests/test_crosswave_jobid_check.py b/tests/test_crosswave_jobid_check.py new file mode 100644 index 00000000..8c3e5ff3 --- /dev/null +++ b/tests/test_crosswave_jobid_check.py @@ -0,0 +1,61 @@ +"""Pin the cross-wave job-ID check artifact (#230 §6 pre-lock).""" + +from __future__ import annotations + +import json +from pathlib import Path + +import pytest + +ARTIFACT = Path(__file__).resolve().parents[1] / ( + "runs/crosswave_jobid_check_draft_v0.json" +) + + +@pytest.fixture(scope="module") +def artifact() -> dict: + return json.loads(ARTIFACT.read_text()) + + +def test_disclosure_language(artifact): + # The status must carry the disclosed-re-analysis framing, never + # a pre-registration claim (review on #235). + assert "DISCLOSED RE-ANALYSIS" in artifact["status"] + assert "UNRATIFIED" in artifact["status"] + assert "pre-registered" not in artifact["status"] + + +def test_both_populations_reported(artifact): + bounds = artifact["bounds"] + assert bounds["excess_rekey_share_ee_population"] == 0.1512 + assert bounds["excess_rekey_share_all_separations"] == 0.0936 + verdicts = artifact["verdict_by_population"] + assert verdicts["ee_population"] == "PASS_WITH_CORRECTION_BAND" + assert verdicts["all_separations"] == "PASS" + assert verdicts["operative"] == "REFEREE" + + +def test_identity_is_labelled(artifact): + identity = artifact["bounds"]["gross_id_survival_identity"] + assert identity["value"] == pytest.approx( + 1 - artifact["across_wave_seam"]["sep_rate"] + ) + assert "NOT evidence" in identity["note"] + + +def test_uncertainty_and_strict_variant(artifact): + bounds = artifact["bounds"] + assert ( + bounds["one_sided_95_upper_ee_population"] + > bounds["excess_rekey_share_ee_population"] + ) + strict = artifact["strict_nan_variant"] + assert strict["excess_ee_population"] == 0.1232 + assert "MISMATCH" in strict["note"] + + +def test_inputs_pinned(artifact): + assert set(artifact["inputs"]) == {"pu2022.csv.gz", "pu2023.csv"} + for pin in artifact["inputs"].values(): + assert len(pin["sha256"]) == 64 + assert pin["bytes"] > 10_000_000 diff --git a/tests/tier_counts.json b/tests/tier_counts.json index 9d39ce3d..85a6eab4 100644 --- a/tests/tier_counts.json +++ b/tests/tier_counts.json @@ -2,7 +2,7 @@ "schema_version": 1, "counts": { "unit": 651, - "artifact": 1020, + "artifact": 1025, "integration_psid": 802, "reproduction_legacy": 520, "oracle_policyengine": 159 From 35926fe708bea8a9b5cc22791505f245c8ad35be Mon Sep 17 00:00:00 2001 From: Daphne Hansell <128793799+daphnehanse11@users.noreply.github.com> Date: Wed, 22 Jul 2026 10:40:14 -0400 Subject: [PATCH 06/16] Referee S3 mechanics: recompute test, E->E-conditional definition, reader pin Per the #230 round-1 referee review: derived fields and both verdicts now recompute from the committed counts in an artifact-tier test (a hand-edited verdict fails); rekey_signature_definition states the E->E-conditional baseline the script actually computes (the per-all-seps share fields are marked descriptive); the sipp_jobs reader commit is pinned in the artifact alongside the input sha256s. Co-Authored-By: Claude Fable 5 --- runs/crosswave_jobid_check_draft_v0.json | 3 +- scripts/build_crosswave_jobid_check.py | 31 +++++++++++++++-- tests/test_crosswave_jobid_check.py | 43 ++++++++++++++++++++++++ tests/tier_counts.json | 2 +- 4 files changed, 75 insertions(+), 4 deletions(-) diff --git a/runs/crosswave_jobid_check_draft_v0.json b/runs/crosswave_jobid_check_draft_v0.json index 0fd24eba..65fbad9a 100644 --- a/runs/crosswave_jobid_check_draft_v0.json +++ b/runs/crosswave_jobid_check_draft_v0.json @@ -13,6 +13,7 @@ "bytes": 3726010471 } }, + "sipp_jobs_reader_commit": "a059193e4fad80ceb1c2e1f4177aa5c69abb1048", "question": "are EJB job IDs longitudinally consistent across the pu2022->pu2023 boundary, or is part of the 9.45% seam separation rate a re-keying (linkage) artifact?", "within_wave_baseline": { "jobs_held": 384747, @@ -34,7 +35,7 @@ "rekey_signature_strict": 92, "rekey_signature_share_of_seps": 0.1085 }, - "rekey_signature_definition": "a vanished job whose person holds a next-month job matching it on industry code, class of worker, and earnings within 20% (|log ratio| < 0.1823); computed identically at the seam and within-wave, so the within-wave share is the coincidental-match baseline", + "rekey_signature_definition": "a vanished job whose person holds a next-month job matching it on industry code, class of worker, and earnings within 20% (|log ratio| < 0.1823); computed identically at the seam and within-wave. The excess is computed on the E->E-CONDITIONAL baseline (signature count / separations-to-employment on each side), then optionally scaled by the seam E->E share for the all-separations population \u2014 the per-all-seps rekey_signature_share_of_seps fields are descriptive only (referee note, #230 round 1 S3)", "bounds": { "gross_id_survival_identity": { "value": 0.9055, diff --git a/scripts/build_crosswave_jobid_check.py b/scripts/build_crosswave_jobid_check.py index 23e7a936..b1e3b16d 100644 --- a/scripts/build_crosswave_jobid_check.py +++ b/scripts/build_crosswave_jobid_check.py @@ -202,6 +202,27 @@ def separation_decomposition( } +def _reader_commit() -> str: + import subprocess + + return ( + subprocess.run( + [ + "git", + "log", + "-1", + "--format=%H", + "--", + "src/populace_dynamics/data/sipp_jobs.py", + ], + capture_output=True, + text=True, + cwd=str(REPO), + ).stdout.strip() + or "unknown" + ) + + def _input_pins() -> dict: """sha256 + size of the staged pu files consumed.""" import hashlib as _h @@ -359,6 +380,7 @@ def band(x: float) -> str: ), "issue": "230", "inputs": _input_pins(), + "sipp_jobs_reader_commit": _reader_commit(), "question": ( "are EJB job IDs longitudinally consistent across the " "pu2022->pu2023 boundary, or is part of the 9.45% seam " @@ -370,8 +392,13 @@ def band(x: float) -> str: "a vanished job whose person holds a next-month job " "matching it on industry code, class of worker, and " "earnings within 20% (|log ratio| < 0.1823); computed " - "identically at the seam and within-wave, so the " - "within-wave share is the coincidental-match baseline" + "identically at the seam and within-wave. The excess is " + "computed on the E->E-CONDITIONAL baseline (signature " + "count / separations-to-employment on each side), then " + "optionally scaled by the seam E->E share for the " + "all-separations population — the per-all-seps " + "rekey_signature_share_of_seps fields are descriptive " + "only (referee note, #230 round 1 S3)" ), "bounds": { "gross_id_survival_identity": { diff --git a/tests/test_crosswave_jobid_check.py b/tests/test_crosswave_jobid_check.py index 8c3e5ff3..a4efdbc3 100644 --- a/tests/test_crosswave_jobid_check.py +++ b/tests/test_crosswave_jobid_check.py @@ -54,6 +54,49 @@ def test_uncertainty_and_strict_variant(artifact): assert "MISMATCH" in strict["note"] +def test_derived_fields_recompute_from_counts(artifact): + # Referee S3 (#230 round 1): the committed JSON's derived fields + # and verdicts must recompute exactly from its own counts, so a + # hand-edited verdict cannot pass unnoticed. + within = artifact["within_wave_baseline"] + seam = artifact["across_wave_seam"] + assert seam["sep_rate"] == pytest.approx( + seam["separations"] / seam["jobs_held"], abs=5e-5 + ) + ee_excess = max( + 0.0, + seam["rekey_signature"] / seam["to_employment"] + - within["rekey_signature"] / within["to_employment"], + ) + assert artifact["bounds"][ + "excess_rekey_share_ee_population" + ] == pytest.approx(ee_excess, abs=5e-5) + all_seps = ee_excess * seam["to_employment"] / seam["separations"] + assert artifact["bounds"][ + "excess_rekey_share_all_separations" + ] == pytest.approx(all_seps, abs=5e-5) + + def band(x): + if x < 0.15: + return "PASS" + if x <= 0.30: + return "PASS_WITH_CORRECTION_BAND" + return "REFER_BACK" + + verdicts = artifact["verdict_by_population"] + assert verdicts["ee_population"] == band( + artifact["bounds"]["excess_rekey_share_ee_population"] + ) + assert verdicts["all_separations"] == band( + artifact["bounds"]["excess_rekey_share_all_separations"] + ) + + +def test_ee_conditional_baseline_stated(artifact): + assert "E->E-CONDITIONAL" in artifact["rekey_signature_definition"] + assert len(artifact["sipp_jobs_reader_commit"]) == 40 + + def test_inputs_pinned(artifact): assert set(artifact["inputs"]) == {"pu2022.csv.gz", "pu2023.csv"} for pin in artifact["inputs"].values(): diff --git a/tests/tier_counts.json b/tests/tier_counts.json index 8631f18a..a15f521b 100644 --- a/tests/tier_counts.json +++ b/tests/tier_counts.json @@ -2,7 +2,7 @@ "schema_version": 1, "counts": { "unit": 740, - "artifact": 1106, + "artifact": 1108, "integration_psid": 804, "reproduction_legacy": 520, "oracle_policyengine": 159 From dc220087e1ab4a5399bd8f8b3d3637cacc9c77a8 Mon Sep 17 00:00:00 2001 From: Daphne Hansell <128793799+daphnehanse11@users.noreply.github.com> Date: Fri, 17 Jul 2026 09:26:14 -0400 Subject: [PATCH 07/16] E4/E5 minimal-audit manifest: registered design (ADR 0004, pre-lock) Workstream A's #230 section-12.2 pre-lock artifact: the complete ADR 0004 adjudication design for the first-lock scope (SIPP-internal employer attachment). Registers the frame (the #235 population), the two arms (accepted-assignment precision; truth-search recall over the re-key class), scoped stratification with seam oversampling, REFEREE slots for P_floor/P_design/alpha/power with a worked binomial example (0.95/0.99/0.05/0.80 -> n=124, c=122 per stratum before clustering and inflation), the blinded ID-masked coding protocol with conservative indeterminate handling registered before labels, provenance including the draw seed (20260717), and the two leakage freezes (the #235 signature parameters; no label backflow into readers or hazards). Sample draws only after the referee round fills the slots. Co-Authored-By: Claude Fable 5 --- docs/design/e4_e5_audit_manifest.md | 169 ++++++++++++++++++++++++++++ 1 file changed, 169 insertions(+) create mode 100644 docs/design/e4_e5_audit_manifest.md diff --git a/docs/design/e4_e5_audit_manifest.md b/docs/design/e4_e5_audit_manifest.md new file mode 100644 index 00000000..df925a3c --- /dev/null +++ b/docs/design/e4_e5_audit_manifest.md @@ -0,0 +1,169 @@ +# E4/E5 minimal-audit manifest (ADR 0004 instantiation, pre-lock) + +**Status: REGISTERED DESIGN, sample not yet drawn.** This is +Workstream A's §12.2 pre-lock artifact for the C3 block (#230 §9.1): +the complete ADR 0004 adjudication design for the first-lock scope — +**SIPP-internal employer attachment**, the assignment E4 (retention +pairs) and E5 (attachment runs) consume. The numeric slots marked +`REFEREE` are ADR 0004 §6.1 items; the sample is drawn only after the +referee round fills them, by the committed draw script, under the +seed registered here. Owner: @daphnehanse11 (frame + coder-panel +operation, per #230 §9.3). + +## 1. The assignment under audit + +E4/E5 treat two job records in adjacent reference months as **the +same employer** iff they share a within-panel `EJB` job ID +(`populace_dynamics.data.sipp_jobs`, pu2023, reference year 2022). +The "matcher" is therefore Census's dependent-interview job-ID +assignment as consumed by the reader — not a model we train. What +hand adjudication can verify from the public-use record: whether the +month-*m* and month-*m+1* job records describe the same employer, +using industry code, occupation code, class of worker, work +arrangement, establishment-size code, monthly earnings, and the +`BMONTH`/`EMONTH` spell edges. This is the "hand-adjudicable truth +demonstrably exists" claim of #230 §9.1, made concrete. + +## 2. Frame and the two arms (ADR 0004 §2.1) + +- **Eligible universe**: all ordered adjacent-month pairs + (person, month *m*, month *m+1*), *m* = 1..11 plus the Dec→Jan + cross-file pair, where the person holds ≥1 job in month *m* and is + present in the panel in month *m+1* (presence from the + person-month universe — exits to nonemployment are in-universe; + sample leavers are not). This is exactly the #235 population; + the check's counts (384,747 within-wave job-holdings; 10,828 at + the seam) size the frame. +- **Candidate generation**: within-person only — all ≤7 job slots of + month *m+1* are visible to the coder. Candidate-generation recall + is 1 by construction (SIPP cannot attach a person's job to another + person's employer record), so candidate-recall and selector-recall + collapse; this is registered as a structural property, not + measured. +- **Accepted-assignment arm (a)**: pairs where a month-*m* job's ID + recurs in month *m+1* (the "stay" assignment). Coders judge + same-employer vs not → **precision** of ID-based attachment. +- **Truth-search arm (b)**: pairs where a month-*m* job's ID does + NOT recur (separations). Coders search all month-*m+1* job records + for a true same-employer counterpart → **recall** (the missed + attachments are exactly #235's re-key class; its signature rate, + 17.5% at the seam vs 2.4% within-wave, is the prevalence prior for + powering this arm). +- **Dual-frame overlap**: a person-pair can contribute a stay job to + arm (a) and a separated job to arm (b); the unit of audit is the + **job-pair**, not the person-pair, so the arms partition job-pairs + and no combined-inclusion estimator is needed. Registered as such. + +## 3. Stratification (ADR 0004 §2.2, scoped per #230 §9.1) + +First-lock scope excludes firm-size-conditional cells, so the +C2-band stratification of the full ADR grid does not apply (its +strata are registered for promotion-time audits, not this one). +Operative strata: + +- **Arm (a)** (precision): {within-wave, seam} × {age 16–44, + 45+} — 4 strata. Seam pairs are deliberately oversampled (they are + the risk locus established by #214/#235 and are only ~2.7% of the + frame). +- **Arm (b)** (recall): {within-wave, seam} × {re-key signature + present, absent} — 4 strata. The signature (same industry + class + of worker + earnings within 20%; parameters **frozen** in + `scripts/build_crosswave_jobid_check.py` before any label exists) + concentrates the plausible false negatives, so signature-present + strata are oversampled. +- Any pooling of sparse strata is published in the draw artifact + before adjudication; every inclusion probability is retained. + +## 4. Power and target sizes (ADR 0004 §2.3) + +Registered parameters — `REFEREE` slots per ADR 0004 §6.1: + +| Parameter | Value | +|---|---| +| `P_floor` (precision) | REFEREE | +| `P_design` | REFEREE | +| `alpha` (one-sided) | REFEREE | +| `1 - beta` | REFEREE | +| Multiplicity rule | REFEREE (recommended: intersection-union across the 4 arm-(a) strata with joint power computed by simulation) | +| Recall: gates or reported-with-bound | REFEREE | + +**Worked example** (illustrative only, not a proposal): for a +simple-random operative stratum, `P_floor = 0.95`, +`P_design = 0.99`, `alpha = 0.05`, `1 − beta = 0.80` gives +`n = 124`, critical count `c = 122` (smallest binomial solution); +(0.90, 0.97) gives `n = 76, c = 73`. Per ADR 0004, job-pairs +cluster within worker: the draw script computes design-based +power by simulation with **worker** as the registered clustering +unit and applies the anticipated design effect, and targets are +inflated for indeterminate/unusable rates (assumed 10% until the +calibration round measures them). Four strata at the example +numbers imply an arm-(a) total near 550 adjudications before +inflation — feasible for a two-coder panel. + +**Arm (b) target**: based on expected true-counterpart prevalence +per stratum (prior: the #235 signature rates), powered for the +registered recall-bound width or floor, per the same machinery. + +## 5. Coding protocol (ADR 0004 §2.4) + +- **Blinding**: coders see both months' job-record fields with all + `EJB` job IDs **masked**, and never see the matcher outcome + (same/different ID), the #235 signature flag, any downstream gate + quantity, or the other coder's decision. +- **Labels**: `same_employer`, `different_employer`, + `insufficient_evidence`, under a frozen coding manual + (`docs/design/e4_e5_coding_manual_v1.md`, to be committed before + labels open; version hash registered in the draw artifact). +- **Disagreements**: adjudicated by a third coder without disclosure + of the split; dispositions logged. +- **Indeterminates** (registered now, before labels): + `insufficient_evidence` counts **conservatively against** the + audited assignment — as incorrect in arm (a) precision and as a + missed true counterpart in arm (b) recall bounds — with the + partial-identification bounds also reported. Hard cases are never + dropped after labels are known. +- **Calibration**: coders first train on vetted cases excluded from + both arms; blinded repeats (10% of assignments) measure continuing + accuracy and agreement. +- The precision test publishes the sensitivity bound to + reference-label error per ADR 0004 (the label is hand-adjudicated + reference truth, not infallible ground truth). + +## 6. Provenance (ADR 0004 §2.5) + +The draw artifact (`runs/e4_e5_audit_draw_v1.json`, committed when +the sample is drawn) will contain: reader version and pu-file +sha256s, frame query (this document's §2 verbatim), stratum +definitions and inclusion probabilities, the random seed +(**registered now: 20260717**), sha256 of each selected job-pair's +public-use identifiers, coding-manual and evidence-sheet versions, +coder-assignment protocol, and counts. Replacements or exclusions +are logged; the sample is never silently refreshed. All fields are +public-use-derived; no restricted data exists in this design. + +## 7. No leakage (ADR 0004 §2.6) + +Nothing here trains a matcher (Census assigns the IDs), but two +freezes are registered so audit labels cannot leak backward: + +1. The #235 re-key-signature parameters (industry + class of worker + + earnings tolerance) are frozen at their committed values; they + may stratify this audit but may never be re-tuned on its labels. +2. Audit labels and dispositions may not inform any reader-side + attachment heuristic, imputation feature, or phase-1 hazard + specification. If a leak occurs, the sample retires and a new + manifest issues. + +## 8. Deliverables and sequence + +1. This manifest (pre-lock, per #230 §12.2) — the registered design. +2. Referee round fills the §4 slots. +3. `scripts/build_e4_e5_audit_draw.py` draws the sample under seed + 20260717 → `runs/e4_e5_audit_draw_v1.json` + the blinded evidence + sheets. +4. Coding manual v1 commits; calibration round runs; panel codes. +5. `runs/e4_e5_audit_v1.json` reports precision/recall with + confidence bounds, agreement, dispositions — consumed by the C3 + amendment PR as E4/E5's ADR 0004 prerequisite. Passing uses the + one-sided bound, never the observed proportion; a failed floor + invalidates the cell. From 8e356ce03ad654b445a9c90d612daf75f4e8718b Mon Sep 17 00:00:00 2001 From: Daphne Hansell <128793799+daphnehanse11@users.noreply.github.com> Date: Wed, 22 Jul 2026 10:40:30 -0400 Subject: [PATCH 08/16] Pin the audit-result sequence per referee S4 (ADR 0004 section 1.5) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Manifest pre-lock; lock may proceed with the audit designed but undrawn; results must exist before the first one-shot candidate run scoring any E4/E5 cell. Failed floor invalidates the cells (no candidate can pass) and the artifact publishes regardless — a designed stop is a graded, publishable outcome. Co-Authored-By: Claude Fable 5 --- docs/design/e4_e5_audit_manifest.md | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-) diff --git a/docs/design/e4_e5_audit_manifest.md b/docs/design/e4_e5_audit_manifest.md index df925a3c..37a6e45c 100644 --- a/docs/design/e4_e5_audit_manifest.md +++ b/docs/design/e4_e5_audit_manifest.md @@ -163,7 +163,13 @@ freezes are registered so audit labels cannot leak backward: sheets. 4. Coding manual v1 commits; calibration round runs; panel codes. 5. `runs/e4_e5_audit_v1.json` reports precision/recall with - confidence bounds, agreement, dispositions — consumed by the C3 - amendment PR as E4/E5's ADR 0004 prerequisite. Passing uses the - one-sided bound, never the observed proportion; a failed floor - invalidates the cell. + confidence bounds, agreement, dispositions. **Sequence (pinned + per the #230 round-1 referee review, S4, matching ADR 0004 + §1.5)**: this manifest is the pre-lock artifact; the C3 block + may lock with the audit *designed but undrawn*; the audit + **results must exist before the first one-shot candidate run** + that scores any E4/E5 cell. Passing uses the one-sided bound, + never the observed proportion. A failed floor invalidates the + E4/E5 cells — no candidate can then pass the block — and the + audit artifact publishes regardless of result: a designed stop + is a graded, publishable outcome, not a re-scoping event. From fd4c372aa7c043f1f567c5552451a1268e0d6731 Mon Sep 17 00:00:00 2001 From: Vahid Ahmadi Date: Thu, 23 Jul 2026 11:21:08 +0100 Subject: [PATCH 09/16] Rename the interface contracts C1/C2/C3 -> IC1/IC2/IC3 (ADR 0003 am. 1) Naming only: no column, band, code mapping, or gate definition changes, and the numbering is preserved 1:1 so every prior reference maps by prefixing "I". A bare "C1" meant four different things in this repo: the gate_w1 fingerprints (gates.yaml fingerprints.c1/.c2), the SSA Trustees table II.C1, an RNG substream in household composition, and the employer-firm interface contracts. Only the last is repo-internal, pre-lock, and ours -- the fingerprints sit inside gate_w1, which is locked: true, so renaming those would cost a public amendment plus a fresh referee round. Timing is the point. IC3 (the employer gate block) is about to be written into gates.yaml, which already contains fingerprints.c1 and fingerprints.c2. A block named C3 locking next to them makes either rename cost exactly what the fingerprint row of that table already costs. Vahid flagged the collision on #192 before the referee round; this closes it while it is still free. Touches frozen ADR 0003 text, so it is a joint-PR change under the IC1/IC2 freeze rule -- procedurally, not because anything moved. Prior discussion keeps the old names; the ADR carries the mapping. Also corrects a stale claim in sipp_jobs: the module said "ADR 0003 is Proposed, not frozen" as the reason job_spells is IC1-preview. It is Accepted and IC1 is frozen; what is still preview-grade is the collapse's single-ref_year coverage, which is what the docstring now says. Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/adr/0003-employer-firm-extension.md | 76 ++++++++++++++------ scripts/build_noemp_band_evidence.py | 2 +- src/populace_dynamics/data/asec_firm_size.py | 4 +- src/populace_dynamics/data/sipp_jobs.py | 26 ++++--- src/populace_dynamics/firms/__init__.py | 2 +- src/populace_dynamics/firms/banding.py | 4 +- tests/test_firms_banding.py | 2 +- tests/test_noemp_band_evidence.py | 2 +- 8 files changed, 76 insertions(+), 42 deletions(-) diff --git a/docs/adr/0003-employer-firm-extension.md b/docs/adr/0003-employer-firm-extension.md index b80236e7..1b6591f3 100644 --- a/docs/adr/0003-employer-firm-extension.md +++ b/docs/adr/0003-employer-firm-extension.md @@ -1,15 +1,45 @@ -# ADR 0003: Employer-firm extension — C1 spell schema and C2 canonical firm-size banding +# ADR 0003: Employer-firm extension — IC1 spell schema and IC2 canonical firm-size banding -**Status:** Accepted — C1 and C2 frozen 2026-07-16. From this +**Status:** Accepted — IC1 and IC2 frozen 2026-07-16. From this point the contracts change only by joint PR between workstreams A and B ([populace-dynamics#192](https://github.com/PolicyEngine/populace-dynamics/issues/192)). +**Amendment 1 (naming, no semantic change).** The interface +contracts were originally named `C1`/`C2`/`C3`. They are renamed +`IC1`/`IC2`/`IC3` — *interface contract* — with no change to any +column, band, code mapping, or gate definition. This is a +joint-PR change because it edits frozen contract text, not because +anything in the contracts moved; the numbering is preserved 1:1, so +every prior reference maps by prefixing `I`. + +**Why, and why this side moves.** In this repository a bare "C1" +already meant four different things: + +| sense | example | can it be renamed? | +|---|---|---| +| gate_w1 **fingerprints** `c1`/`c2` | `gates.yaml` `fingerprints.c1` (PPI↔NRA) | **No** — inside `gate_w1`, which is `locked: true`. Renaming needs a public amendment plus a fresh referee round | +| SSA Trustees **table** II.C1 | `data/external/ssa_tr_2014_ii_c1.*` | No — an external publisher's table label | +| RNG **substream** C3 | household-composition `nonfamily_bridge` | Unrelated component; renaming is churn for no gain | +| **interface contracts** C1/C2/C3 | this ADR | **Yes** — the only set that is repo-internal, pre-lock, and ours | + +The collision was flagged on #192 before the C3 referee round with +the note that it would confuse referees. It is fixed now rather +than later for one reason: `IC3` (the employer gate block) is +about to be written into `gates.yaml`, which already contains +`fingerprints.c1` and `fingerprints.c2`. Once a block named `C3` +locks alongside them, renaming either set costs an amendment and a +fresh referee round — the exact cost this table shows the +fingerprint side already carries. + +Prior discussion (issue #192, the ADR history, merged PR bodies) +uses the old names and is not rewritten; this note is the mapping. + **Sign-off:** @vahid-ahmadi (Workstream B, author) · @daphnehanse11 (Workstream A) — the joint sign-off is recorded by the merge of the freeze PR: authorship by one workstream owner plus approval by the other. First scheduled amendment (pre-registered below): -the C1 ``hours_band``/monthly-hours column once phase-1 establishes +the IC1 ``hours_band``/monthly-hours column once phase-1 establishes SIPP's supportable hours granularity. ## Context @@ -17,15 +47,15 @@ SIPP's supportable hours granularity. The employer-firm plan (`docs/plans/employer-firm-plan.html`) splits the extension into workstream A (person side: SIPP spells, CPS hosts, imputation) and workstream B (firm side: external targets, banding, -calibration, register), meeting at three interface contracts. C1 (the -spell schema) and C2 (canonical firm-size banding and its semantics) +calibration, register), meeting at three interface contracts. IC1 (the +spell schema) and IC2 (canonical firm-size banding and its semantics) freeze in week 1. This ADR records both, plus the target/gate partition rule, folding in the four contract-affecting findings from the week-1 review on issue #192. ## Decision -### C2 — canonical firm-size banding +### IC2 — canonical firm-size banding 1. **Semantics (review finding F5).** The canonical firm-size variable means **administrative enterprise size**: total @@ -33,7 +63,7 @@ the week-1 review on issue #192. counts it. Survey labels are noisy measures of that quantity — CPS ASEC firm size (worker-reported, all locations, previous calendar year's longest job — under either the raw Census `NOEMP` - or the IPUMS `FIRMSIZE` coding; see C2.5) is the primary training + or the IPUMS `FIRMSIZE` coding; see IC2.5) is the primary training label; SIPP 2014+ `EJB1_EMPSIZE` (establishment size) is a proxy chain. SUSB is therefore the correct E1 reference. 2. **Bands are headcount bands.** Five canonical bands with edges at @@ -43,7 +73,7 @@ the week-1 review on issue #192. both QWI (20-49 / 50-249) and the detailed SUSB classes (40-49 / 50-74) support it. FTE-denominated thresholds (the ACA cut is 50 full-time equivalents at 30 hours/week, not headcount) are - resolved by a person-side hours join — out of C2 scope. + resolved by a person-side hours join — out of IC2 scope. 3. **Mappings are total but explicitly ambiguous where the source is coarse.** Every raw code from every source maps to exactly one `BandSpan` (a contiguous run of canonical bands with an `exact` @@ -65,7 +95,7 @@ the week-1 review on issue #192. replication (weighted code shares by year vs. SUSB) is committed as `runs/noemp_band_evidence_v1.json` with its build script and pinning tests (#211) — the reported-anchor convention, since it - is derived evidence rather than a source extract — for the C3 + is derived evidence rather than a source extract — for the IC3 record. 5. **The person-side coding is explicit, not inferred (seam with #194).** The raw Census ASEC person file carries `NOEMP` @@ -85,7 +115,7 @@ the week-1 review on issue #192. emit `CanonicalBand` directly), never feed `NOEMP` integers to the `ipums_firmsize` route. -### C1 — job-spell schema +### IC1 — job-spell schema One tidy table, written by workstream A, read by workstream B: @@ -96,7 +126,7 @@ One tidy table, written by workstream A, read by workstream B: | `start_period` | period | first period of the spell | | `end_period` | period | last period; open spells use a sentinel | | `industry` | str | NAICS major (sector) group | -| `firm_size_band` | enum | canonical band per C2 (`CanonicalBand`) | +| `firm_size_band` | enum | canonical band per IC2 (`CanonicalBand`) | | `class_of_worker` | enum | private / federal / state-local government / self-employed / unpaid family | | `earnings_share` | float | share of the person's period earnings from this job | | `primary_job` | bool | phase 0 is primary-job-only | @@ -111,7 +141,7 @@ One tidy table, written by workstream A, read by workstream B: from the SUSB/QWI calibration universe; self-employed spells have no defined `firm_size_band`. - **Geography joins from the person table.** QWI/J2J targets are - state-level; C1 deliberately carries no geography column. The + state-level; IC1 deliberately carries no geography column. The state of a spell is the host person's state at `start_period`, joined on `person_id` — the join key lives on the person table, not the spell table. @@ -122,9 +152,9 @@ One tidy table, written by workstream A, read by workstream B: compliance, issue #192 — the 80-hours-per-month test of 7 CFR 273.24 and the 3-in-36 countable-month clock need month-resolved hours, not spell start/end plus annual earnings). - C1 as frozen carries no hours column, so it **cannot yet serve + IC1 as frozen carries no hours column, so it **cannot yet serve monthly-hours consumers**; a `hours_band` (or monthly hours) - column is the first scheduled C1 amendment, to be added by joint + column is the first scheduled IC1 amendment, to be added by joint PR once workstream A's phase-1 spell imputation establishes what hours granularity SIPP can support. Consumers must not proxy monthly compliance from annual quantities in the meantime. @@ -146,10 +176,10 @@ phase-0 QRF therefore comes from a named bridge, not an implicit one: bridge, aged forward. 2. **Proxy chain:** SIPP 2014+ establishment size x tenure, mapped through the establishment-to-enterprise noise model implied by - the C2 semantics. + the IC2 semantics. 3. **Pre-registered caveat:** the ASEC reference-period mismatch (`FIRMSIZE` = last calendar year's longest job; tenure supplement - = current job) is carried into the C3 gate notes as a known + = current job) is carried into the IC3 gate notes as a known label-misalignment term. ### Target/gate partition rule @@ -161,7 +191,7 @@ firm-size x sector flow margins committed under `data/external/`; gates E1/E2/E7/E11 score on held-out dimensions of the same sources (the sex/age demographic axes of QWI, the firm-age axis, and the state axis) that calibration never touches. The exact cell lists lock -with C3 after the floor runs. +with IC3 after the floor runs. Three unit rules recorded now (issue #192 review, point 4; branch review finding 3): @@ -171,7 +201,7 @@ review finding 3): so calibrating person-spells to QWI cells carries a wedge on the order of the multiple-jobholding rate (~5%, time-varying). A job-count -> person-count adjustment is an explicit pre-registered - C3 item, not a footnote. + IC3 item, not a footnote. - **QWI publishes mean earnings (`EarnS`), never medians**; E7 is stated on means. - **J2J's employer universe is broader than SUSB/QWI's.** The @@ -182,7 +212,7 @@ review finding 3): sectors (notably 61 Educational Services and 62 Health Care). Any E11 cell definition must either restate J2J on a private-comparable basis or carry this scope difference as a pre-registered caveat; - the choice locks with C3. + the choice locks with IC3. ## Consequences @@ -193,8 +223,8 @@ review finding 3): workstreams push directly to each other's branches when useful (reader fixes, rebases, contract-text corrections — this has run in both directions and worked). The norm the freeze makes - explicit: a change that touches contract semantics (C1 columns, - C2 bands/codings, gate definitions) requires the *other* + explicit: a change that touches contract semantics (IC1 columns, + IC2 bands/codings, gate definitions) requires the *other* workstream owner's approval on the PR even when the commit was pushed directly, so pre-registration always records who decided, not just who typed. @@ -204,6 +234,6 @@ review finding 3): references, analogous to the NCHS/Census/ONS files — never scored model output. Raw microdata is never committed. - No change to `gates.yaml`. Employer gates E1-E12 lock as a new - block (C3) after noise-floor runs and a referee round, via the - standard amendment process; no one-shot candidate runs before C3 + block (IC3) after noise-floor runs and a referee round, via the + standard amendment process; no one-shot candidate runs before IC3 locks. diff --git a/scripts/build_noemp_band_evidence.py b/scripts/build_noemp_band_evidence.py index dc1a23cd..9bcc886c 100644 --- a/scripts/build_noemp_band_evidence.py +++ b/scripts/build_noemp_band_evidence.py @@ -3,7 +3,7 @@ REPORTED ANCHOR, NOT A GATE RUN. Like the mortality/claiming/ disability floors, this reads no gate and decides nothing on its own; it is committed evidence pinned by a reproduction test. It -records the empirical basis for the C2 banding decision's treatment +records the empirical basis for the IC2 banding decision's treatment of CPS ASEC firm size: **the 2019+ data dictionaries' relabeling of NOEMP codes 2/3 (from 10-49 / 50-99 to 10-24 / 25-99) never happened in the instrument.** diff --git a/src/populace_dynamics/data/asec_firm_size.py b/src/populace_dynamics/data/asec_firm_size.py index 5f32d12d..6f78c3a3 100644 --- a/src/populace_dynamics/data/asec_firm_size.py +++ b/src/populace_dynamics/data/asec_firm_size.py @@ -22,7 +22,7 @@ share (~7.5%), while a true 25-99 band carries ~15%. This reader therefore uses the 10-49 / 50-99 reading for all years and records the dictionary conflict here rather than silently following the -2019+ label text into a factor-two mis-band. Consequence for C2: +2019+ label text into a factor-two mis-band. Consequence for IC2: the 50-employee edge (ACA and state mandates) is directly observed in every supported year — the "post-2019 label cannot resolve the 50 cut" problem stated in earlier drafts dissolves. @@ -374,7 +374,7 @@ def firm_size_tabulation( "class_of_worker", ), ) -> pd.DataFrame: - """Weighted firm-size tabulation — the C2 evidence artifact. + """Weighted firm-size tabulation — the IC2 evidence artifact. Args: records: Output of :func:`read_asec_firm_size` (one or more diff --git a/src/populace_dynamics/data/sipp_jobs.py b/src/populace_dynamics/data/sipp_jobs.py index c722c9e5..65be95df 100644 --- a/src/populace_dynamics/data/sipp_jobs.py +++ b/src/populace_dynamics/data/sipp_jobs.py @@ -1,4 +1,4 @@ -"""SIPP job-level monthly records and C1-preview spells (issue #200). +"""SIPP job-level monthly records and IC1-preview spells (issue #200). The 2014-redesign SIPP public-use files are the employer-firm plan's primary label panel (#192): one row per person-month (``SSUID`` x @@ -8,7 +8,7 @@ within-panel employer-attachment key that phase-1 transition hazards rest on. ``EJB{n}_EMPSIZE`` measures **establishment** size at the worker's location (the redesign dropped the all-locations question), -so it is the C2 proxy-chain input, never firm size (ADR 0003; +so it is the IC2 proxy-chain input, never firm size (ADR 0003; ``firms/banding.py``). Every variable this reader touches was verified against the Census @@ -28,10 +28,14 @@ string-typed in the API schema. ``job_spells`` collapses maximal consecutive-month runs per -(person, job id) into spell rows whose shape mirrors the C1 spell -schema. It is labeled **C1-preview**: ADR 0003 is Proposed, not -frozen, and this output also serves as Workstream B's generator for -C1-conforming fixture files. Attribute changes inside a spell +(person, job id) into spell rows whose shape mirrors the IC1 spell +schema. It is still labeled **IC1-preview**, but for a narrower +reason than when it was written: ADR 0003 is now Accepted and IC1 +is frozen, so what remains preview-grade is this collapse's own +coverage (single ``ref_year`` only — cross-year spell linkage +raises rather than guessing), not the schema's status. The output +also serves as Workstream B's generator for IC1-conforming fixture +files. Attribute changes inside a spell (class of worker, industry, establishment size) are surfaced via ``attributes_constant`` — never silently averaged. @@ -495,12 +499,12 @@ def read_sipp_job_months( def job_spells(job_months: pd.DataFrame) -> pd.DataFrame: - """Collapse job-months into C1-preview spell rows. + """Collapse job-months into IC1-preview spell rows. A spell is a maximal run of consecutive reference months for one - (person, job id). The output mirrors the C1 spell schema of ADR + (person, job id). The output mirrors the IC1 spell schema of ADR 0003 (Proposed — this is a preview, not the frozen contract) and - doubles as Workstream B's generator for C1-conforming fixtures. + doubles as Workstream B's generator for IC1-conforming fixtures. Args: job_months: Output of :func:`read_sipp_job_months`. @@ -560,7 +564,7 @@ def job_spells(job_months: pd.DataFrame) -> pd.DataFrame: ] ) - # Cross-year spell linkage is undefined in this C1 preview: the + # Cross-year spell linkage is undefined in this IC1 preview: the # break/run detection, the person-month earnings lookup, and the # spell edges all key on the calendar ``month`` (1-12) alone, so two # different reference years sharing a month would collapse into one @@ -575,7 +579,7 @@ def job_spells(job_months: pd.DataFrame) -> pd.DataFrame: raise ValueError( "job_spells received job-months spanning multiple ref_years " f"({sorted(int(y) for y in ref_years)}); cross-year spell " - "linkage is undefined in this C1 preview. Collapse one SIPP " + "linkage is undefined in this IC1 preview. Collapse one SIPP " "file's months at a time." ) diff --git a/src/populace_dynamics/firms/__init__.py b/src/populace_dynamics/firms/__init__.py index daa46544..9748b63e 100644 --- a/src/populace_dynamics/firms/__init__.py +++ b/src/populace_dynamics/firms/__init__.py @@ -1,6 +1,6 @@ """Employer-firm extension, workstream B (firm side). -Canonical firm-size banding (interface contract C2) and label-verified +Canonical firm-size banding (interface contract IC2) and label-verified loaders for the committed external target extracts (SUSB, BDS, QWI, J2J). See ``docs/adr/0003-employer-firm-extension.md`` and issue #192. """ diff --git a/src/populace_dynamics/firms/banding.py b/src/populace_dynamics/firms/banding.py index 67843686..df02e2e8 100644 --- a/src/populace_dynamics/firms/banding.py +++ b/src/populace_dynamics/firms/banding.py @@ -1,4 +1,4 @@ -"""Canonical firm-size banding — interface contract C2. +"""Canonical firm-size banding — interface contract IC2. **Semantics (review finding F5).** The canonical variable means *administrative enterprise size*: the total employment of the legal @@ -26,7 +26,7 @@ Bands are **headcount** bands. Policy thresholds stated in FTEs (the ACA applicable-large-employer cut is 50 *full-time equivalents* at 30 hours/week, not headcount) are handled by a person-side hours join and -are out of C2 scope. +are out of IC2 scope. **Canonical bands.** Five bands with edges at 10 / 50 / 100 / 500:: diff --git a/tests/test_firms_banding.py b/tests/test_firms_banding.py index ae8e1f3f..3c474b64 100644 --- a/tests/test_firms_banding.py +++ b/tests/test_firms_banding.py @@ -1,4 +1,4 @@ -"""Tests for the canonical firm-size banding (contract C2). +"""Tests for the canonical firm-size banding (contract IC2). Checks the properties the contract promises: canonical bands partition the positive integers; every raw source code maps to diff --git a/tests/test_noemp_band_evidence.py b/tests/test_noemp_band_evidence.py index 1a98c64a..4f1c1c44 100644 --- a/tests/test_noemp_band_evidence.py +++ b/tests/test_noemp_band_evidence.py @@ -1,7 +1,7 @@ """Pin the NOEMP band-label evidence artifact (issue #192). The committed ``runs/noemp_band_evidence_v1.json`` records the -discontinuity test behind the C2 decision to read ASEC NOEMP codes +discontinuity test behind the IC2 decision to read ASEC NOEMP codes 2/3 as 10-49 / 50-99 in every year. These tests pin the artifact's internal consistency, and — when the ASEC files are staged — reproduce it from the raw data. From c3b1b47ee0b6f82bcfd1c7491efdb56c747933c7 Mon Sep 17 00:00:00 2001 From: Daphne Hansell <128793799+daphnehanse11@users.noreply.github.com> Date: Thu, 23 Jul 2026 09:42:58 -0400 Subject: [PATCH 10/16] Add the E5 run arm, no-revisit clause, frame-measured power, registered pooling MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Responds to the 2026-07-23 review round (items 1-4 of the 2026-07-19 review restated there): 1. E5 scope: new run arm (c) — unit is the maximal same-ID chain; every internal link and both terminal transitions are coded, so run-level error (false continuation / false break) is measured directly rather than composed from pair precision by an independence assumption. Scope mapping registered in section 1: arms (a)/(b) certify E4, arm (c) certifies E5. 2. No-revisit clause: all REFEREE slots, including the arm-(b) gates-or-reported decision, must be filled before the draw and may not be revised after any label exists. 3. Power: worker-resampling simulation from the actual frame, so deff is measured not assumed; arm-(b) target formula registered with the #235 prevalence priors; run-arm budget in link-codings. 4. Pooling rule registered now (fixed collapse order, never across the within-wave/seam axis), not deferred to the draw artifact. Co-Authored-By: Claude Fable 5 --- docs/design/e4_e5_audit_manifest.md | 97 +++++++++++++++++++++++++---- 1 file changed, 84 insertions(+), 13 deletions(-) diff --git a/docs/design/e4_e5_audit_manifest.md b/docs/design/e4_e5_audit_manifest.md index 37a6e45c..ba666c64 100644 --- a/docs/design/e4_e5_audit_manifest.md +++ b/docs/design/e4_e5_audit_manifest.md @@ -24,6 +24,15 @@ arrangement, establishment-size code, monthly earnings, and the `BMONTH`/`EMONTH` spell edges. This is the "hand-adjudicable truth demonstrably exists" claim of #230 §9.1, made concrete. +**Scope mapping (registered)**: the pair-level arms (a)/(b) certify +**E4** — retention is a pair-level assignment. **E5** consumes +*runs* — maximal chains of pair-links — where error compounds with +length and enters as **false continuation** (a wrong link extends a +run) or **false break** (a missed link splits one). E5 is certified +by the run arm (c) below, at run level; pair precision alone does +not license E5 and is never composed into a run claim by an +independence assumption. + ## 2. Frame and the two arms (ADR 0004 §2.1) - **Eligible universe**: all ordered adjacent-month pairs @@ -49,10 +58,30 @@ demonstrably exists" claim of #230 §9.1, made concrete. attachments are exactly #235's re-key class; its signature rate, 17.5% at the seam vs 2.4% within-wave, is the prevalence prior for powering this arm). +- **Run arm (c)** (E5): the unit is the **run** — a maximal chain of + same-ID adjacent-month pair-links. A sampled run is coded in full: + every internal pair-link (the arm-(a) task) plus both terminal + transitions (the arm-(b) task, searching beyond each end for a + true counterpart). The run label then derives deterministically + from its link labels: `correctly_delimited`, `over_extended` (≥1 + internal false continuation), `truncated` (≥1 terminal false + break), or `both`; `insufficient_evidence` on any constituent link + makes the run indeterminate (conservative, per §5). Run-level + error is thus measured **directly**, with the + false-continuation/false-break decomposition the ADR 0004 §4 E5 + row requires — the composition from links to runs is observed, not + assumed independent. Cost accounting: a run of length *L* costs + *L*−1 internal + ≤2 terminal codings, so this arm's budget is set + in link-codings, not runs. - **Dual-frame overlap**: a person-pair can contribute a stay job to arm (a) and a separated job to arm (b); the unit of audit is the **job-pair**, not the person-pair, so the arms partition job-pairs and no combined-inclusion estimator is needed. Registered as such. + Arm (c) samples runs, whose constituent links are coded under the + same protocol but enter **only** the arm-(c) estimator — link + codings are not recycled into arms (a)/(b) (a deliberate + efficiency loss that keeps every estimator's inclusion + probabilities single-frame). ## 3. Stratification (ADR 0004 §2.2, scoped per #230 §9.1) @@ -71,8 +100,21 @@ Operative strata: `scripts/build_crosswave_jobid_check.py` before any label exists) concentrates the plausible false negatives, so signature-present strata are oversampled. -- Any pooling of sparse strata is published in the draw artifact - before adjudication; every inclusion probability is retained. +- **Arm (c)** (run level): {run length 2–3, 4–11, full-year 12} × + {seam-adjacent, not} — 6 strata, where *seam-adjacent* means the + run contains or terminates at the Dec→Jan cross-file pair. + Full-year and seam-adjacent strata are oversampled: full-year runs + carry the E5 gate quantity (`full_year_run_share`), and the seam + is where #235 locates the false-break risk. +- **Pooling (rule registered now, not at draw time)**: a stratum + pools only when its **frame count** — known before any label + exists — cannot meet its powered target at sampling fraction ≤ 1. + Pooling collapses axes in this fixed order: the age split (arm a), + the signature split (arm b), length band 2–3 into 4–11 (arm c) — + and **never across the within-wave/seam axis**, the registered + risk axis. The draw artifact publishes each applied pooling with + the triggering frame count; every inclusion probability is + retained; no pooling decision may follow first sight of any label. ## 4. Power and target sizes (ADR 0004 §2.3) @@ -86,23 +128,50 @@ Registered parameters — `REFEREE` slots per ADR 0004 §6.1: | `1 - beta` | REFEREE | | Multiplicity rule | REFEREE (recommended: intersection-union across the 4 arm-(a) strata with joint power computed by simulation) | | Recall: gates or reported-with-bound | REFEREE | +| `P_floor_run` (share of runs correctly delimited, one-sided lower bound) | REFEREE | +| Run-arm link-coding budget ceiling | REFEREE | +| False-continuation / false-break decomposition | registered: always reported separately, each with its own bound | + +**No-revisit clause (registered)**: every `REFEREE` slot in this +table — including whether arm-(b) recall gates or is +reported-with-bound — must be filled **before the sample is +drawn**. After the draw no slot may be revised, and in particular +the gating status of arm (b) may not change once any arm-(b) label +exists. A revision proposed after labels exist is void and triggers +the §7 retire-and-reissue remedy. + +**Power procedure (registered)**: power is computed by simulation +that resamples **workers** — the registered clustering unit — from +the *actual frame*, which exists before the draw (#235 sizes it). +The design effect is therefore **measured from the frame's +cluster-size distribution, not assumed**. The indeterminate/unusable +inflation is a named parameter: prior 10%, superseded by the +calibration round's measured rate if that is larger. **Worked example** (illustrative only, not a proposal): for a simple-random operative stratum, `P_floor = 0.95`, `P_design = 0.99`, `alpha = 0.05`, `1 − beta = 0.80` gives `n = 124`, critical count `c = 122` (smallest binomial solution); -(0.90, 0.97) gives `n = 76, c = 73`. Per ADR 0004, job-pairs -cluster within worker: the draw script computes design-based -power by simulation with **worker** as the registered clustering -unit and applies the anticipated design effect, and targets are -inflated for indeterminate/unusable rates (assumed 10% until the -calibration round measures them). Four strata at the example +(0.90, 0.97) gives `n = 76, c = 73`. These independent-Bernoulli +`n` are floor illustrations only; registered targets come from the +worker-resampling simulation above. Four strata at the example numbers imply an arm-(a) total near 550 adjudications before inflation — feasible for a two-coder panel. -**Arm (b) target**: based on expected true-counterpart prevalence -per stratum (prior: the #235 signature rates), powered for the -registered recall-bound width or floor, per the same machinery. +**Arm (b) target (registered formula)**: per stratum, the +true-counterpart prevalence prior `π` is the #235 excess rate for +that stratum (E→E-conditional 15.1% at the seam; 2.4% within-wave +baseline). If the referee rules reported-with-bound, `n` solves +`z_{1−α} · sqrt(π(1−π)/n) · sqrt(deff) ≤ w` for the registered +half-width `w`; if recall gates, the same binomial floor machinery +as arm (a) applies with the registered `R_floor`. Illustration: +`π = 0.15`, `w = 0.05`, one-sided `α = 0.05`, `deff = 1` gives +`n ≈ 138` before inflation. + +**Arm (c) target**: powered for `P_floor_run` by the same +worker-resampling simulation, with the budget expressed in +link-codings (a full-year run costs 11 internal + ≤2 terminal +codings) and capped by the registered ceiling above. ## 5. Coding protocol (ADR 0004 §2.4) @@ -162,8 +231,10 @@ freezes are registered so audit labels cannot leak backward: 20260717 → `runs/e4_e5_audit_draw_v1.json` + the blinded evidence sheets. 4. Coding manual v1 commits; calibration round runs; panel codes. -5. `runs/e4_e5_audit_v1.json` reports precision/recall with - confidence bounds, agreement, dispositions. **Sequence (pinned +5. `runs/e4_e5_audit_v1.json` reports arm-(a) precision, arm-(b) + recall, and arm-(c) run-delimitation rates (with the + false-continuation/false-break decomposition) with confidence + bounds, agreement, dispositions. **Sequence (pinned per the #230 round-1 referee review, S4, matching ADR 0004 §1.5)**: this manifest is the pre-lock artifact; the C3 block may lock with the audit *designed but undrawn*; the audit From 3fa68007f51a186ba0fb5b9e4076e036a9ac38c1 Mon Sep 17 00:00:00 2001 From: Vahid Ahmadi Date: Thu, 23 Jul 2026 15:41:30 +0100 Subject: [PATCH 11/16] Address review: finish the Proposed fix, rename the operative plan MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two should-fixes from Daphne's #277 review: 1. The stale "ADR 0003 (Proposed — this is a preview, not the frozen contract)" text was corrected in the module docstring but survived in `job_spells`'s own docstring. Post-merge the ADR says Accepted while that line said Proposed. It now mirrors the module wording: the schema is frozen; what is preview-grade is this collapse's single-`ref_year` coverage. 2. `docs/plans/employer-firm-plan.html` used C1/C2/C3 in the contract sense while being cited by the ADR's Context section as the operative split — the one file on the wrong side of the rename boundary. Renamed to IC1/IC2/IC3 (six lines; the SVG path data containing `C265,247` is untouched), and Amendment 1 now states the boundary explicitly: history keeps the old names, live documents are renamed, and the three unrenamed senses stay as the table gives them. Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/adr/0003-employer-firm-extension.md | 8 ++++++++ docs/plans/employer-firm-plan.html | 12 ++++++------ src/populace_dynamics/data/sipp_jobs.py | 6 ++++-- 3 files changed, 18 insertions(+), 8 deletions(-) diff --git a/docs/adr/0003-employer-firm-extension.md b/docs/adr/0003-employer-firm-extension.md index 1b6591f3..ab8951cc 100644 --- a/docs/adr/0003-employer-firm-extension.md +++ b/docs/adr/0003-employer-firm-extension.md @@ -34,6 +34,14 @@ fingerprint side already carries. Prior discussion (issue #192, the ADR history, merged PR bodies) uses the old names and is not rewritten; this note is the mapping. +**Boundary: history keeps the old names, live documents are +renamed.** The plan (`docs/plans/employer-firm-plan.html`), cited by +the Context section below as the operative split, is a live document +and is renamed with this amendment, so a referee following the ADR's +own link does not meet unmapped names. The unrenamed senses in the +table above (the locked `gates.yaml` fingerprints, the SSA table +labels, the RNG substream) remain as they are, by the reasons given. + **Sign-off:** @vahid-ahmadi (Workstream B, author) · @daphnehanse11 (Workstream A) — the joint sign-off is recorded by the merge of the freeze PR: authorship by one workstream owner plus diff --git a/docs/plans/employer-firm-plan.html b/docs/plans/employer-firm-plan.html index 413453ea..56480a6b 100644 --- a/docs/plans/employer-firm-plan.html +++ b/docs/plans/employer-firm-plan.html @@ -255,11 +255,11 @@

Targets, register & calibration

INTERFACE CONTRACTS — frozen week 1, changed only by joint PR
    -
  • C1 · Spell schema. One table: person_id, spell_id, start_period, end_period, industry (major), firm_size_band, earnings_share, primary_job. A writes it, B reads it. Firm-size bands use the canonical banding B defines (C2). Multi-job resolved primary-job-only in phase 0.
    • -
  • C2 · Canonical firm-size banding + semantics. B proposes the band set reconcilable across NOEMP / SIPP-establishment / SUSB-enterprise, and the decision of what the variable means (administrative firm size, per review F5). A trains to it; documented in the ADR.
    • -
  • C3 · Gate pre-registration. Jointly authored employer gate block (E1–E12 thresholds after floor runs), split ownership as above, one referee round, locked before any candidate runs. Neither side's model work may start a one-shot run until C3 locks.
    • +
  • IC1 · Spell schema. One table: person_id, spell_id, start_period, end_period, industry (major), firm_size_band, earnings_share, primary_job. A writes it, B reads it. Firm-size bands use the canonical banding B defines (IC2). Multi-job resolved primary-job-only in phase 0.
    • +
  • IC2 · Canonical firm-size banding + semantics. B proposes the band set reconcilable across NOEMP / SIPP-establishment / SUSB-enterprise, and the decision of what the variable means (administrative firm size, per review F5). A trains to it; documented in the ADR.
    • +
  • IC3 · Gate pre-registration. Jointly authored employer gate block (E1–E12 thresholds after floor runs), split ownership as above, one referee round, locked before any candidate runs. Neither side's model work may start a one-shot run until IC3 locks.
-

Sync points: week 1 (freeze C1/C2), week 4 (lock C3), week 10 (joint phase-2 go/no-go with Max). Everything else is asynchronous — A can build readers/imputation against fixture spells; B can build the target pipeline and register against a synthetic spell file conforming to C1.

+

Sync points: week 1 (freeze IC1/IC2), week 4 (lock IC3), week 10 (joint phase-2 go/no-go with Max). Everything else is asynchronous — A can build readers/imputation against fixture spells; B can build the target pipeline and register against a synthetic spell file conforming to IC1.

Precedents — what similar projects did

@@ -277,8 +277,8 @@

Precedents — what similar projects did

Milestones

- - + + diff --git a/src/populace_dynamics/data/sipp_jobs.py b/src/populace_dynamics/data/sipp_jobs.py index 65be95df..43b65267 100644 --- a/src/populace_dynamics/data/sipp_jobs.py +++ b/src/populace_dynamics/data/sipp_jobs.py @@ -503,8 +503,10 @@ def job_spells(job_months: pd.DataFrame) -> pd.DataFrame: A spell is a maximal run of consecutive reference months for one (person, job id). The output mirrors the IC1 spell schema of ADR - 0003 (Proposed — this is a preview, not the frozen contract) and - doubles as Workstream B's generator for IC1-conforming fixtures. + 0003, which is **Accepted and frozen**; what remains preview-grade + is this collapse's own coverage (single ``ref_year`` only — see + below), not the schema's status. It doubles as Workstream B's + generator for IC1-conforming fixtures. Args: job_months: Output of :func:`read_sipp_job_months`. From e2aba68003fe3d66db5181acab88d47ee570c037 Mon Sep 17 00:00:00 2001 From: Vahid Ahmadi Date: Thu, 23 Jul 2026 16:39:31 +0100 Subject: [PATCH 12/16] Drop the stray .claude gitlinks my review commit added; ignore .claude MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit My `633ad10` on this branch accidentally staged eleven `.claude/ worktrees/agent-*` entries — local Claude Code worktrees, committed as gitlinks (mode 160000) to commits that exist in no remote. They are unrelated to this PR and would land on master as broken submodule references that `git clone` cannot resolve. Removed from the index (the local directories are untouched) and `.claude/` added to `.gitignore` so the mistake cannot recur on any branch. My error, cleaned up on the branch it landed on rather than left for the artifact's author. Co-Authored-By: Claude Opus 4.8 (1M context) --- .claude/worktrees/agent-a047f5f9d8c39b7b5 | 1 - .claude/worktrees/agent-a0676e04c34eef233 | 1 - .claude/worktrees/agent-a27fadae3d1fab7e1 | 1 - .claude/worktrees/agent-a32b110f187901da5 | 1 - .claude/worktrees/agent-a4b4a77e4359ed77b | 1 - .claude/worktrees/agent-a50f25e3c12d5dc09 | 1 - .claude/worktrees/agent-a7dd9d5353fb5e0ab | 1 - .claude/worktrees/agent-a81c95dded53a10b7 | 1 - .claude/worktrees/agent-aaf1a4eea49839b69 | 1 - .claude/worktrees/agent-acca6d0fdb88095b3 | 1 - .claude/worktrees/agent-ae00bf0b8feb6c58e | 1 - .gitignore | 3 +++ 12 files changed, 3 insertions(+), 11 deletions(-) delete mode 160000 .claude/worktrees/agent-a047f5f9d8c39b7b5 delete mode 160000 .claude/worktrees/agent-a0676e04c34eef233 delete mode 160000 .claude/worktrees/agent-a27fadae3d1fab7e1 delete mode 160000 .claude/worktrees/agent-a32b110f187901da5 delete mode 160000 .claude/worktrees/agent-a4b4a77e4359ed77b delete mode 160000 .claude/worktrees/agent-a50f25e3c12d5dc09 delete mode 160000 .claude/worktrees/agent-a7dd9d5353fb5e0ab delete mode 160000 .claude/worktrees/agent-a81c95dded53a10b7 delete mode 160000 .claude/worktrees/agent-aaf1a4eea49839b69 delete mode 160000 .claude/worktrees/agent-acca6d0fdb88095b3 delete mode 160000 .claude/worktrees/agent-ae00bf0b8feb6c58e diff --git a/.claude/worktrees/agent-a047f5f9d8c39b7b5 b/.claude/worktrees/agent-a047f5f9d8c39b7b5 deleted file mode 160000 index a8bb7bec..00000000 --- a/.claude/worktrees/agent-a047f5f9d8c39b7b5 +++ /dev/null @@ -1 +0,0 @@ -Subproject commit a8bb7bec27370d569c6d016f05d16bf4778f0b97 diff --git a/.claude/worktrees/agent-a0676e04c34eef233 b/.claude/worktrees/agent-a0676e04c34eef233 deleted file mode 160000 index ef5b8ab6..00000000 --- a/.claude/worktrees/agent-a0676e04c34eef233 +++ /dev/null @@ -1 +0,0 @@ -Subproject commit ef5b8ab602694525d9c64e898cedccf8c4ce74ef diff --git a/.claude/worktrees/agent-a27fadae3d1fab7e1 b/.claude/worktrees/agent-a27fadae3d1fab7e1 deleted file mode 160000 index 5346b3b9..00000000 --- a/.claude/worktrees/agent-a27fadae3d1fab7e1 +++ /dev/null @@ -1 +0,0 @@ -Subproject commit 5346b3b934949059baf1dc9296073aa418dcac64 diff --git a/.claude/worktrees/agent-a32b110f187901da5 b/.claude/worktrees/agent-a32b110f187901da5 deleted file mode 160000 index 31217108..00000000 --- a/.claude/worktrees/agent-a32b110f187901da5 +++ /dev/null @@ -1 +0,0 @@ -Subproject commit 31217108263eb41cd9cfc6f62ecaefe470e8aee9 diff --git a/.claude/worktrees/agent-a4b4a77e4359ed77b b/.claude/worktrees/agent-a4b4a77e4359ed77b deleted file mode 160000 index 7754ae6f..00000000 --- a/.claude/worktrees/agent-a4b4a77e4359ed77b +++ /dev/null @@ -1 +0,0 @@ -Subproject commit 7754ae6f32f27cd1363341f25c4b6e6c51cc94b0 diff --git a/.claude/worktrees/agent-a50f25e3c12d5dc09 b/.claude/worktrees/agent-a50f25e3c12d5dc09 deleted file mode 160000 index 7754ae6f..00000000 --- a/.claude/worktrees/agent-a50f25e3c12d5dc09 +++ /dev/null @@ -1 +0,0 @@ -Subproject commit 7754ae6f32f27cd1363341f25c4b6e6c51cc94b0 diff --git a/.claude/worktrees/agent-a7dd9d5353fb5e0ab b/.claude/worktrees/agent-a7dd9d5353fb5e0ab deleted file mode 160000 index abf8cb15..00000000 --- a/.claude/worktrees/agent-a7dd9d5353fb5e0ab +++ /dev/null @@ -1 +0,0 @@ -Subproject commit abf8cb152b5a9dfae98e3ac356c661e2a2eb8396 diff --git a/.claude/worktrees/agent-a81c95dded53a10b7 b/.claude/worktrees/agent-a81c95dded53a10b7 deleted file mode 160000 index 1f87c6ab..00000000 --- a/.claude/worktrees/agent-a81c95dded53a10b7 +++ /dev/null @@ -1 +0,0 @@ -Subproject commit 1f87c6ab1a970c9c60b8492439a976756608f126 diff --git a/.claude/worktrees/agent-aaf1a4eea49839b69 b/.claude/worktrees/agent-aaf1a4eea49839b69 deleted file mode 160000 index ffb992a5..00000000 --- a/.claude/worktrees/agent-aaf1a4eea49839b69 +++ /dev/null @@ -1 +0,0 @@ -Subproject commit ffb992a5c367253cc2f701cb4ebc1bdb27323af7 diff --git a/.claude/worktrees/agent-acca6d0fdb88095b3 b/.claude/worktrees/agent-acca6d0fdb88095b3 deleted file mode 160000 index f9463117..00000000 --- a/.claude/worktrees/agent-acca6d0fdb88095b3 +++ /dev/null @@ -1 +0,0 @@ -Subproject commit f9463117bbd7d4a0cc5b6977e89954a2b70d83ca diff --git a/.claude/worktrees/agent-ae00bf0b8feb6c58e b/.claude/worktrees/agent-ae00bf0b8feb6c58e deleted file mode 160000 index 36cf7e2b..00000000 --- a/.claude/worktrees/agent-ae00bf0b8feb6c58e +++ /dev/null @@ -1 +0,0 @@ -Subproject commit 36cf7e2b07233460e782f94e18842b72d6c6033a diff --git a/.gitignore b/.gitignore index 351d07d9..d94b015e 100644 --- a/.gitignore +++ b/.gitignore @@ -112,3 +112,6 @@ paper/paper_files/ .vercel/ scratch/ *.pkl + +# Claude Code local state (agent worktrees are not repo content) +.claude/ From 2470b76d6632eaa98072015ceb08e14fa92706cc Mon Sep 17 00:00:00 2001 From: Vahid Ahmadi Date: Thu, 30 Jul 2026 13:32:40 +0200 Subject: [PATCH 13/16] Fold linkage-QC adoption requirements into ADR 0004 --- docs/adr/0004-linkage-qc.md | 54 ++++++++++++++++++++++++++++++++++--- 1 file changed, 51 insertions(+), 3 deletions(-) diff --git a/docs/adr/0004-linkage-qc.md b/docs/adr/0004-linkage-qc.md index 83ae9e68..b76d6f06 100644 --- a/docs/adr/0004-linkage-qc.md +++ b/docs/adr/0004-linkage-qc.md @@ -205,6 +205,29 @@ are opened. set a cutoff for, or otherwise adapt the matcher it scores. A leaked sample is retired from evaluation and replaced under a new manifest. +#### First-lock scope, ownership, and sidecar location + +The first C3 lock applies this audit to **E4/E5 SIPP-internal employer +attachment only**. Within-panel `EJB` job-ID attachment is an assignment +for which hand-adjudicable reference evidence can exist. The full +five-band × NAICS-major × transition-class × `primary_job` grid is +phased behind the first lock. E9 transition classes, E11 ordered pairs, +and any later-promoted gate must clear a new audit at promotion time; +none inherits an E4/E5 pass. + +Workstream A (`@daphnehanse11`) owns the E4/E5 adjudication frame, +coding manual, coder-panel operation, and privacy-safe audit manifest. +Workstream B (`@vahid-ahmadi`) owns the firm-side sidecar schema, +versioning rules, and firm-side audit artifacts. Versioned sidecars +join C1 on `person_id` and `spell_id`; they never amend C1. Before the +first audit opens labels, the reusable schema and versioning contract +must be committed at +`docs/design/employer_linkage_qc_sidecar.schema.json`. Privacy-safe +manifests and results must use +`runs/employer_linkage_qc__v.json`. Restricted evidence, +direct identifiers, coder identities, and access-controlled lookup +keys remain outside git. + ### 3. Linkage-bias reweighting on observables 1. **Name the target population and identification assumption.** Each @@ -314,6 +337,28 @@ the frozen C1 seam ultimately carries one `CanonicalBand`. Therefore: 4. E4/E5 score retention and attachment, E9 scores transition-conditioned earnings changes, and E11 scores firm-size flows only after their applicable link and reweighting requirements above are evaluable. +5. A QRF-imputed enterprise-size band on a CPS host has no admissible + per-record truth frame. CPS `NOEMP` is the training label and has a + reference-period mismatch; SIPP measures establishment rather than + enterprise size; public LEHD/SUSB data have no person-level link to + CPS; and pre-redesign SIPP is an aged self-report on a different + sample. More fundamentally, a draw from a conditional distribution + is not a claim that an observed host has one adjudicable true class. + Therefore: + + > Cells conditioning on imputed firm-size bands are validated + > distributionally (calibration fit to SUSB margins plus held-out-axis + > stability) and are report-only in every phase; they gate only if an + > external person-level truth source materializes, at which point they + > enter through the standard promotion ceremony (new floor + ADR 0004 + > audit). + + This status is permanent for the current evidence regime, not a + provisional deferral. Calibration fit is a build check and may not be + described as independent validation. A genuinely disjoint margin can + support promotion only after its own floor and audit; true per-record, + coworker, or firm-identity claims additionally require a linked + reference. The pre-C3 floor draft on [PR #212](https://github.com/PolicyEngine/populace-dynamics/pull/212) @@ -354,9 +399,12 @@ does not resolve them: 1. The numeric precision floor or floors; `P_design`, `alpha`, power, confidence interval, multiplicity correction, and whether recall has an operative floor. -2. The phase-1 and phase-2 adjudication frames, permissible evidence, - exact truth labels, privacy-safe manifest, and whether an E12 truth - source exists at all. +2. Within the first-lock E4/E5 scope fixed in section 2, the permissible + evidence, exact truth labels, privacy-safe manifest details, and + numeric coder-panel design; for later phases, the E9/E11 frames and + whether an E12 truth source exists at all. Imputed CPS enterprise-size + bands follow section 5's permanent report-only degradation rule and + are not an unresolved hand-adjudication frame. 3. Beyond E11's required ordered pair and the actual co-assignment unit used by E12, which endpoint, pair, transition, and run levels gate; how a passing endpoint result composes, if at all; and which sparse From c5b4d11f1b3bd6b34a3de8ea1a1c8d72e6777ee0 Mon Sep 17 00:00:00 2001 From: Vahid Ahmadi Date: Thu, 30 Jul 2026 13:39:30 +0200 Subject: [PATCH 14/16] Keep naming amendment outside sealed production sources --- docs/adr/0003-employer-firm-extension.md | 5 ++++ src/populace_dynamics/data/asec_firm_size.py | 4 +-- src/populace_dynamics/data/sipp_jobs.py | 30 ++++++++------------ src/populace_dynamics/firms/__init__.py | 2 +- src/populace_dynamics/firms/banding.py | 4 +-- 5 files changed, 22 insertions(+), 23 deletions(-) diff --git a/docs/adr/0003-employer-firm-extension.md b/docs/adr/0003-employer-firm-extension.md index ab8951cc..5eb3280b 100644 --- a/docs/adr/0003-employer-firm-extension.md +++ b/docs/adr/0003-employer-firm-extension.md @@ -41,6 +41,11 @@ and is renamed with this amendment, so a referee following the ADR's own link does not meet unmapped names. The unrenamed senses in the table above (the locked `gates.yaml` fingerprints, the SSA table labels, the RNG substream) remain as they are, by the reasons given. +Production-source docstrings sealed by the published first-estimates +replay ceremony also retain their historical `C1`/`C2` wording. They +are not operative contract text and are interpreted through this +one-to-one mapping; cosmetic edits would invalidate the sealed replay +identity. New source text uses the `IC` names. **Sign-off:** @vahid-ahmadi (Workstream B, author) · @daphnehanse11 (Workstream A) — the joint sign-off is recorded by diff --git a/src/populace_dynamics/data/asec_firm_size.py b/src/populace_dynamics/data/asec_firm_size.py index 6f78c3a3..5f32d12d 100644 --- a/src/populace_dynamics/data/asec_firm_size.py +++ b/src/populace_dynamics/data/asec_firm_size.py @@ -22,7 +22,7 @@ share (~7.5%), while a true 25-99 band carries ~15%. This reader therefore uses the 10-49 / 50-99 reading for all years and records the dictionary conflict here rather than silently following the -2019+ label text into a factor-two mis-band. Consequence for IC2: +2019+ label text into a factor-two mis-band. Consequence for C2: the 50-employee edge (ACA and state mandates) is directly observed in every supported year — the "post-2019 label cannot resolve the 50 cut" problem stated in earlier drafts dissolves. @@ -374,7 +374,7 @@ def firm_size_tabulation( "class_of_worker", ), ) -> pd.DataFrame: - """Weighted firm-size tabulation — the IC2 evidence artifact. + """Weighted firm-size tabulation — the C2 evidence artifact. Args: records: Output of :func:`read_asec_firm_size` (one or more diff --git a/src/populace_dynamics/data/sipp_jobs.py b/src/populace_dynamics/data/sipp_jobs.py index 43b65267..c722c9e5 100644 --- a/src/populace_dynamics/data/sipp_jobs.py +++ b/src/populace_dynamics/data/sipp_jobs.py @@ -1,4 +1,4 @@ -"""SIPP job-level monthly records and IC1-preview spells (issue #200). +"""SIPP job-level monthly records and C1-preview spells (issue #200). The 2014-redesign SIPP public-use files are the employer-firm plan's primary label panel (#192): one row per person-month (``SSUID`` x @@ -8,7 +8,7 @@ within-panel employer-attachment key that phase-1 transition hazards rest on. ``EJB{n}_EMPSIZE`` measures **establishment** size at the worker's location (the redesign dropped the all-locations question), -so it is the IC2 proxy-chain input, never firm size (ADR 0003; +so it is the C2 proxy-chain input, never firm size (ADR 0003; ``firms/banding.py``). Every variable this reader touches was verified against the Census @@ -28,14 +28,10 @@ string-typed in the API schema. ``job_spells`` collapses maximal consecutive-month runs per -(person, job id) into spell rows whose shape mirrors the IC1 spell -schema. It is still labeled **IC1-preview**, but for a narrower -reason than when it was written: ADR 0003 is now Accepted and IC1 -is frozen, so what remains preview-grade is this collapse's own -coverage (single ``ref_year`` only — cross-year spell linkage -raises rather than guessing), not the schema's status. The output -also serves as Workstream B's generator for IC1-conforming fixture -files. Attribute changes inside a spell +(person, job id) into spell rows whose shape mirrors the C1 spell +schema. It is labeled **C1-preview**: ADR 0003 is Proposed, not +frozen, and this output also serves as Workstream B's generator for +C1-conforming fixture files. Attribute changes inside a spell (class of worker, industry, establishment size) are surfaced via ``attributes_constant`` — never silently averaged. @@ -499,14 +495,12 @@ def read_sipp_job_months( def job_spells(job_months: pd.DataFrame) -> pd.DataFrame: - """Collapse job-months into IC1-preview spell rows. + """Collapse job-months into C1-preview spell rows. A spell is a maximal run of consecutive reference months for one - (person, job id). The output mirrors the IC1 spell schema of ADR - 0003, which is **Accepted and frozen**; what remains preview-grade - is this collapse's own coverage (single ``ref_year`` only — see - below), not the schema's status. It doubles as Workstream B's - generator for IC1-conforming fixtures. + (person, job id). The output mirrors the C1 spell schema of ADR + 0003 (Proposed — this is a preview, not the frozen contract) and + doubles as Workstream B's generator for C1-conforming fixtures. Args: job_months: Output of :func:`read_sipp_job_months`. @@ -566,7 +560,7 @@ def job_spells(job_months: pd.DataFrame) -> pd.DataFrame: ] ) - # Cross-year spell linkage is undefined in this IC1 preview: the + # Cross-year spell linkage is undefined in this C1 preview: the # break/run detection, the person-month earnings lookup, and the # spell edges all key on the calendar ``month`` (1-12) alone, so two # different reference years sharing a month would collapse into one @@ -581,7 +575,7 @@ def job_spells(job_months: pd.DataFrame) -> pd.DataFrame: raise ValueError( "job_spells received job-months spanning multiple ref_years " f"({sorted(int(y) for y in ref_years)}); cross-year spell " - "linkage is undefined in this IC1 preview. Collapse one SIPP " + "linkage is undefined in this C1 preview. Collapse one SIPP " "file's months at a time." ) diff --git a/src/populace_dynamics/firms/__init__.py b/src/populace_dynamics/firms/__init__.py index 9748b63e..daa46544 100644 --- a/src/populace_dynamics/firms/__init__.py +++ b/src/populace_dynamics/firms/__init__.py @@ -1,6 +1,6 @@ """Employer-firm extension, workstream B (firm side). -Canonical firm-size banding (interface contract IC2) and label-verified +Canonical firm-size banding (interface contract C2) and label-verified loaders for the committed external target extracts (SUSB, BDS, QWI, J2J). See ``docs/adr/0003-employer-firm-extension.md`` and issue #192. """ diff --git a/src/populace_dynamics/firms/banding.py b/src/populace_dynamics/firms/banding.py index df02e2e8..67843686 100644 --- a/src/populace_dynamics/firms/banding.py +++ b/src/populace_dynamics/firms/banding.py @@ -1,4 +1,4 @@ -"""Canonical firm-size banding — interface contract IC2. +"""Canonical firm-size banding — interface contract C2. **Semantics (review finding F5).** The canonical variable means *administrative enterprise size*: the total employment of the legal @@ -26,7 +26,7 @@ Bands are **headcount** bands. Policy thresholds stated in FTEs (the ACA applicable-large-employer cut is 50 *full-time equivalents* at 30 hours/week, not headcount) are handled by a person-side hours join and -are out of IC2 scope. +are out of C2 scope. **Canonical bands.** Five bands with edges at 10 / 50 / 100 / 500:: From 978ff2337c8a10694539b546e951e9ff380fc450 Mon Sep 17 00:00:00 2001 From: Vahid Ahmadi Date: Thu, 30 Jul 2026 17:57:17 +0200 Subject: [PATCH 15/16] Align linkage QC with IC naming and aggregate gate boundary --- docs/adr/0004-linkage-qc.md | 125 +++++++++++++++++++----------------- 1 file changed, 67 insertions(+), 58 deletions(-) diff --git a/docs/adr/0004-linkage-qc.md b/docs/adr/0004-linkage-qc.md index b76d6f06..1e776fd6 100644 --- a/docs/adr/0004-linkage-qc.md +++ b/docs/adr/0004-linkage-qc.md @@ -1,7 +1,7 @@ -# ADR 0004: Employer-firm linkage QC requirements for C3 +# ADR 0004: Employer-firm linkage QC requirements for IC3 -**Status:** Proposed — input to the C3 referee round; this document -locks no C3 threshold. C1 and C2 are treated as frozen and immutable +**Status:** Proposed — input to the IC3 referee round; this document +locks no IC3 threshold. IC1 and IC2 are treated as frozen and immutable for this work under the workstream directive and the freeze record in [populace-dynamics#215](https://github.com/PolicyEngine/populace-dynamics/pull/215). This ADR does not amend their schema, semantics, or readers. @@ -10,7 +10,7 @@ This ADR does not amend their schema, semantics, or readers. The employer-firm plan on [issue #192](https://github.com/PolicyEngine/populace-dynamics/issues/192) -requires noise floors before thresholds, a referee round before the C3 +requires noise floors before thresholds, a referee round before the IC3 gate block locks, and no one-shot candidate run before that lock. The same ordering must govern the worker-to-employer or worker-to-firm-type assignment itself. A downstream E-cell cannot certify a model if the @@ -18,22 +18,22 @@ links used to construct the cell have unknown quality. Here, **link** includes any accepted worker-to-employer, employer- attachment, or worker-to-firm-type assignment consumed by an E-cell. -That includes an assigned C2 `CanonicalBand` or industry type; it does +That includes an assigned IC2 `CanonicalBand` or industry type; it does not turn a statistical imputation into an identified firm ID. A **link unit** is the unit on which a decision is accepted or withheld. Paired and run-level moments may require a stricter derived unit, as specified below. -The frozen contract seam remains the C1 spell fields +The frozen contract seam remains the IC1 spell fields `person_id`, `spell_id`, `start_period`, `end_period`, `industry`, `firm_size_band`, `class_of_worker`, `earnings_share`, and -`primary_job`. The implemented C2 seam is the banding functions in +`primary_job`. The implemented IC2 seam is the banding functions in `src/populace_dynamics/firms/banding.py`. Linkage-QC decisions, adjudication labels, match scores, and linkage weights therefore live -in versioned sidecars keyed to C1 rows; they are not new C1 columns. -`spell_type` is also not a C1 field. Where needed below, transition +in versioned sidecars keyed to IC1 rows; they are not new IC1 columns. +`spell_type` is also not an IC1 field. Where needed below, transition type is derived from adjacent spell rows plus the versioned person-month -observation/nonemployment frame; C1 spells alone do not identify exits, +observation/nonemployment frame; IC1 spells alone do not identify exits, entries, or censoring. ### Evidence behind the precision-first rule @@ -73,7 +73,7 @@ positive links. ## Decision -### 1. Precision first, before every C3 threshold +### 1. Precision first, before every IC3 threshold 1. **Every link-producing component used by a gated E-cell must have an independent audit artifact.** For categorical assignments, the @@ -88,16 +88,16 @@ positive links. `1 - precision` is the false-discovery rate, not that pairwise rate. Point estimates and one-sided confidence bounds are both required. 2. **The precision floor is pre-registered before E-cell thresholds.** - C3 must name the assignment, eligible universe, link unit, floor + IC3 must name the assignment, eligible universe, link unit, floor `P_floor`, confidence level, required strata, pooling rule, and failure disposition before it names any threshold for an E-cell that consumes that assignment. Recall is always published. Whether - recall also gates is an explicit C3 referee decision, not an + recall also gates is an explicit IC3 referee decision, not an after-the-fact response to results. 3. **Passing uses a confidence bound, not the observed proportion.** The lower one-sided confidence bound for precision must be at least `P_floor` overall and at every stratum or derived-unit level that - C3 designates operative. Oversampled strata are combined only with + IC3 designates operative. Oversampled strata are combined only with their recorded sample inclusion weights. 4. **No threshold shopping follows a linkage failure.** A failed or unevaluable applicable floor makes the linked E-cell invalid. It is @@ -106,7 +106,7 @@ positive links. result. A registered candidate with any required invalid cell cannot pass the employer block. 5. **The audit precedes the one-shot run.** Linkage-QC results and the - immutable adjudication manifest must be on the C3 record before a + immutable adjudication manifest must be on the IC3 record before a candidate may consume the matcher. A materially changed matcher, cutoff, candidate-generation rule, source vintage, or target vocabulary requires a new versioned audit or the pre-registered @@ -114,7 +114,7 @@ positive links. ### 2. Hand-adjudication sample -The C3 block must register the following sample design before labels +The IC3 block must register the following sample design before labels are opened. 1. **Frame and two audit arms.** Define the complete eligible universe, @@ -126,11 +126,11 @@ are opened. reference that can find a true counterpart omitted by candidate generation; reviewing only the matcher's candidate set is not an end-to-end recall study. Report candidate-generation recall separately - from selector/cutoff recall. Because a unit may enter both arms, C3 + from selector/cutoff recall. Because a unit may enter both arms, IC3 registers the dual-frame overlap and combined-inclusion estimator so it is neither omitted nor counted twice. 2. **Stratification follows the moments the matcher feeds.** For the - accepted arm, stratify at minimum by the five assigned C2 + accepted arm, stratify at minimum by the five assigned IC2 `CanonicalBand` values (plus withheld or unresolved assignments), assigned NAICS major industry, and the derived spell/transition class relevant to the battery: stay, job-to-job, exit, or entry, crossed @@ -140,9 +140,9 @@ are opened. known-true class before review, the universe arm uses a separate pre-link stratification frame observed for every eligible unit, then reports recall by adjudicated true band, industry, and transition - domain. C3 publishes any pooling of sparse strata before adjudication + domain. IC3 publishes any pooling of sparse strata before adjudication and retains every arm-specific inclusion probability. -3. **Target size is powered at the floor.** C3 registers `P_floor`, a +3. **Target size is powered at the floor.** IC3 registers `P_floor`, a substantively meaningful design precision `P_design > P_floor`, one-sided size `alpha`, power `1 - beta`, and its multiplicity rule. For independent, equal-probability accepted assignments within a @@ -161,7 +161,7 @@ are opened. power with the registered clustering unit, inclusion weights, finite- population correction where material, and anticipated design effect. The target includes unusable-record, indeterminate, and nonresponse - inflation. If multiple strata must clear, C3 distinguishes simultaneous + inflation. If multiple strata must clear, IC3 distinguishes simultaneous confidence coverage from an intersection-union pass rule and computes the **joint** probability that every required stratum passes at `P_design`; powering each stratum separately at `1 - beta` is not @@ -178,7 +178,7 @@ are opened. coder's decision. They code `link`, `no link`, or `insufficient evidence` under a frozen manual. A third coder or standing panel adjudicates disagreements without majority labels being disclosed - first. Before labels open, C3 registers whether an indeterminate, + first. Before labels open, IC3 registers whether an indeterminate, unusable-evidence, or nonresponse case counts conservatively as incorrect, enters partial-identification bounds, or makes the floor unevaluable; hard cases may not be dropped from denominators after @@ -207,7 +207,7 @@ are opened. #### First-lock scope, ownership, and sidecar location -The first C3 lock applies this audit to **E4/E5 SIPP-internal employer +The first IC3 lock applies this audit to **E4/E5 SIPP-internal employer attachment only**. Within-panel `EJB` job-ID attachment is an assignment for which hand-adjudicable reference evidence can exist. The full five-band × NAICS-major × transition-class × `primary_job` grid is @@ -219,7 +219,7 @@ Workstream A (`@daphnehanse11`) owns the E4/E5 adjudication frame, coding manual, coder-panel operation, and privacy-safe audit manifest. Workstream B (`@vahid-ahmadi`) owns the firm-side sidecar schema, versioning rules, and firm-side audit artifacts. Versioned sidecars -join C1 on `person_id` and `spell_id`; they never amend C1. Before the +join IC1 on `person_id` and `spell_id`; they never amend IC1. Before the first audit opens labels, the reusable schema and versioning contract must be committed at `docs/design/employer_linkage_qc_sidecar.schema.json`. Privacy-safe @@ -247,7 +247,7 @@ keys remain outside git. version `s_i = Pr(L_i = 1 | X_i)`, where `L_i` denotes inclusion in the usable linked subsample of the complete eligible universe at the E-cell's analysis unit. A person-level score does not automatically - weight a pair, run, or cluster; C3 models that unit's inclusion or + weight a pair, run, or cluster; IC3 models that unit's inclusion or pre-registers and justifies a joint construction from component scores. Publish the model specification, training population, out-of-sample diagnostics, propensity distributions for @@ -265,7 +265,7 @@ keys remain outside git. `[(1 - r_i) / r_i] [q / (1 - q)]`, where `q` is the linked-copy share of the stack. `s_i` and `r_i` are not interchangeable. The linkage adjustment multiplies the pre-existing survey/design/ - opportunity weight; it does not replace that base weight. C3 derives + opportunity weight; it does not replace that base weight. IC3 derives and registers the applicable form before seeing the E-cell. 4. **Publish weighted and unweighted with valid uncertainty.** Every E-cell consuming links publishes both, labels the registered operative @@ -294,22 +294,22 @@ keys remain outside git. An overall floor failure invalidates every linked cell using that matcher version. A required-stratum failure invalidates every cell whose -estimand includes that stratum, unless C3 pre-registers a genuinely +estimand includes that stratum, unless IC3 pre-registers a genuinely disjoint matcher and estimand. A passing marginal link floor does not by -itself certify a pair or a run: C3 must either audit the derived unit +itself certify a pair or a run: IC3 must either audit the derived unit directly or register and justify a conservative composition rule. | cell | linkage unit and required QC | weighting and failure disposition | |---|---|---| -| **E4 — retention pairs** | Audit endpoint assignments and, if C3 makes it operative, the derived same-employer/same-attribute decision. The audit strata include age, industry, C2 band, transition month, and `primary_job` status used by the cell. | Publish pair-opportunity estimates unweighted and with linkage-IPW. If any C3-designated endpoint or pair-level floor fails, all affected E4 retention cells are invalid. | -| **E5 — attachment runs** | If C3 designates a run-level floor, audit the full multi-window run label, including false continuation and false break errors. A per-month pass alone cannot certify a run because error compounds with length. | Weight the eligible run opportunity, not each observed linked month as if independent. If any C3-designated endpoint or run-level floor fails, the affected E5 run-length cells are invalid. | -| **E9 — earnings-change coherence** | Derive stay, job-to-job, exit, and entry from adjacent C1 spells plus the versioned person-month observation/nonemployment frame. Audit any C3-designated transition floor; for job-to-job cells, audit both origin and destination firm-size/industry assignments. | Define propensity and composite weight on the eligible transition opportunity, then publish both versions within class. A failed C3-designated origin, destination, or transition-class floor invalidates the corresponding E9 cells; the referee cannot replace them post hoc with a different definition. | -| **E11 — firm-size flow ladder** | The unit is an origin-destination job-to-job pair. Audit the joint ordered C2-band assignment. An unresolved `BandSpan` is not a correct categorical assignment merely because it contains the eventual band. | Model inclusion and weight at the ordered-pair opportunity. An overall or joint-pair floor failure invalidates E11; a required origin/destination stratum failure invalidates every E11 cell containing it. | +| **E4 — retention pairs** | Audit endpoint assignments and, if IC3 makes it operative, the derived same-employer/same-attribute decision. The audit strata include age, industry, IC2 band, transition month, and `primary_job` status used by the cell. | Publish pair-opportunity estimates unweighted and with linkage-IPW. If any IC3-designated endpoint or pair-level floor fails, all affected E4 retention cells are invalid. | +| **E5 — attachment runs** | If IC3 designates a run-level floor, audit the full multi-window run label, including false continuation and false break errors. A per-month pass alone cannot certify a run because error compounds with length. | Weight the eligible run opportunity, not each observed linked month as if independent. If any IC3-designated endpoint or run-level floor fails, the affected E5 run-length cells are invalid. | +| **E9 — earnings-change coherence** | Derive stay, job-to-job, exit, and entry from adjacent IC1 spells plus the versioned person-month observation/nonemployment frame. Audit any IC3-designated transition floor; for job-to-job cells, audit both origin and destination firm-size/industry assignments. | Define propensity and composite weight on the eligible transition opportunity, then publish both versions within class. A failed IC3-designated origin, destination, or transition-class floor invalidates the corresponding E9 cells; the referee cannot replace them post hoc with a different definition. | +| **E11 — firm-size flow ladder** | For any per-record audit, the unit is an origin-destination job-to-job pair and the joint ordered IC2-band assignment is audited. An unresolved `BandSpan` is not a correct categorical assignment merely because it contains the eventual band. A separately registered held-out aggregate destination-band distribution may gate under the boundary below. | Model inclusion and weight at the ordered-pair opportunity for per-record claims. A failed IC3-designated origin, destination, or transition-class floor invalidates the corresponding per-record E11 cells. An aggregate E11 gate certifies only reproduction of its registered aggregate distribution. | | **E12 — variance and coworker structure** | Phase 2 must audit worker-to-firm-type assignment and any generated same-firm or coworker co-assignment at the exact unit the E12 estimand uses. Type agreement alone cannot validate a claim about an identified firm. | Model inclusion at the worker pair, co-assignment, or cluster unit used by the decomposition; a worker-only propensity is insufficient without a justified composition. Publish both versions. If truth, floor, or support fails, every E12 cell using it is invalid and phase 2 is a no-go. | E3 and E8 do not ordinarily require a worker-to-firm link, and E10 re-runs the existing locked PSID earnings gates without a new noise -floor. They are not blanket exemptions: if a final C3 implementation +floor. They are not blanket exemptions: if a final IC3 implementation constructs any of them from accepted employer or firm-type assignments, the precision-first law applies. Linkage failure never weakens E10 or changes an existing PSID threshold. @@ -323,7 +323,7 @@ spells, not an observed two-sided worker-firm roster. The SIPP reader's `EJB{n}_JOBID` is a within-panel attachment key. Its spell collapse currently carries raw `empsize_code`; `sipp_empsize_to_canonical` in `banding.py` preserves source-band ambiguity through `BandSpan`, while -the frozen C1 seam ultimately carries one `CanonicalBand`. Therefore: +the frozen IC1 seam ultimately carries one `CanonicalBand`. Therefore: 1. QC scores the **final accepted assignment consumed by the E-cell**, after any ambiguity resolution, not the raw SIPP code or a claim that @@ -331,9 +331,9 @@ the frozen C1 seam ultimately carries one `CanonicalBand`. Therefore: 2. An exact SIPP interval-to-band map establishes only numeric interval nesting. SIPP measures establishment size, so it does not by itself validate administrative enterprise size. -3. Sidecars join to C1 with `person_id` and `spell_id`. They do not add +3. Sidecars join to IC1 with `person_id` and `spell_id`. They do not add `job_id`, `firm_id`, match score, adjudication status, or weights to - frozen C1. + frozen IC1. 4. E4/E5 score retention and attachment, E9 scores transition-conditioned earnings changes, and E11 scores firm-size flows only after their applicable link and reweighting requirements above are evaluable. @@ -346,21 +346,30 @@ the frozen C1 seam ultimately carries one `CanonicalBand`. Therefore: is not a claim that an observed host has one adjudicable true class. Therefore: - > Cells conditioning on imputed firm-size bands are validated - > distributionally (calibration fit to SUSB margins plus held-out-axis - > stability) and are report-only in every phase; they gate only if an - > external person-level truth source materializes, at which point they - > enter through the standard promotion ceremony (new floor + ADR 0004 - > audit). - - This status is permanent for the current evidence regime, not a - provisional deferral. Calibration fit is a build check and may not be - described as independent validation. A genuinely disjoint margin can - support promotion only after its own floor and audit; true per-record, - coworker, or firm-identity claims additionally require a linked - reference. - -The pre-C3 floor draft on + > Per-record or individual-outcome cells conditioning on imputed + > firm-size bands are validated distributionally (calibration fit to + > SUSB margins plus held-out-axis stability) and are report-only in + > every phase; they gate only if an external person-level truth source + > materializes, at which point they enter through the standard + > promotion ceremony (new floor + ADR 0004 audit). + + This per-record status is permanent for the current evidence regime, + not a provisional deferral. Calibration fit is a build check and may + not be described as independent validation. + + A genuinely held-out aggregate distribution such as E11's + destination-size margin may gate when it is disjoint from every + calibration target and IC3 locks its exact statistic, population, cell + list, floor, and automatic demotion rule before fitting. Such a gate + certifies aggregate-distribution reproduction only. It does **not** + validate any worker's assigned band, true worker-employer linkage, + worker sorting, coworker structure, or firm effects. This boundary + implements the HIPSM-first synthetic-firm decision recorded on + [issue #282](https://github.com/PolicyEngine/populace-dynamics/issues/282): + aggregate validation is admissible now, while true-linked validation + remains a future promotion requirement. + +The pre-IC3 floor draft on [PR #212](https://github.com/PolicyEngine/populace-dynamics/pull/212) does not satisfy or conflict with this ADR: it estimates sampling noise in E3/E4/E5/E8/E9 after the linkage inputs are defined. Its half-splits @@ -385,13 +394,13 @@ attenuating. For that reason, a phase-2 E12 story must identify an admissible hand-adjudication frame for the actual assignment unit and clear its -precision floor. If public margins cannot support that truth, C3 records +precision floor. If public margins cannot support that truth, IC3 records the limitation as a phase-2 no-go; calibration fit to aggregates is not a substitute. The register may support firm-type policy claims only at the level it identifies. It may not relabel type agreement as firm- identity or coworker validation. -### 6. Items for the C3 referee round — deliberately unresolved +### 6. Items for the IC3 referee round — deliberately unresolved The referee round must decide and pre-register the following. This ADR does not resolve them: @@ -429,23 +438,23 @@ does not resolve them: 9. Long-window tenure evidence from PSID/NLSY and any transport test for applying one adjudication result across source vintages or populations. -The settled C1 fields, C2 semantics and five bands, explicit +The settled IC1 fields, IC2 semantics and five bands, explicit `NOEMP`/`FIRMSIZE` coding, class-of-worker universe, and person-table geography join are outside this list. Reopening them requires the joint -C1/C2 amendment process, not the C3 referee round. +IC1/IC2 amendment process, not the IC3 referee round. ## Consequences -- C3 gains a linkage-quality input gate before its model-fit gates. +- IC3 gains a linkage-quality input gate before its model-fit gates. E4, E5, E9, E11, and E12 cannot certify a candidate from unaudited or floor-failing assignments. - Every link-consuming cell exposes the observable-selection question by publishing linkage-IPW and unweighted estimates together, with one operative version chosen before the candidate result is seen. - The required evidence is carried in sidecars and reported artifacts; - C1, C2, `gates.yaml`, readers, and banding code are unchanged. + IC1, IC2, `gates.yaml`, readers, and banding code are unchanged. - Exact floor values, E-cell thresholds, and operative weighting choices - remain decisions for the C3 referee round. + remain decisions for the IC3 referee round. ## References From 73213c5bfb996c743a8353613701f91fce810dd4 Mon Sep 17 00:00:00 2001 From: Vahid Ahmadi Date: Thu, 30 Jul 2026 18:04:37 +0200 Subject: [PATCH 16/16] Close pre-draw audit sizing channels --- docs/design/e4_e5_audit_manifest.md | 70 ++++++++++++++++++++++++----- 1 file changed, 58 insertions(+), 12 deletions(-) diff --git a/docs/design/e4_e5_audit_manifest.md b/docs/design/e4_e5_audit_manifest.md index ba666c64..e4dbc0b7 100644 --- a/docs/design/e4_e5_audit_manifest.md +++ b/docs/design/e4_e5_audit_manifest.md @@ -8,7 +8,10 @@ pairs) and E5 (attachment runs) consume. The numeric slots marked `REFEREE` are ADR 0004 §6.1 items; the sample is drawn only after the referee round fills them, by the committed draw script, under the seed registered here. Owner: @daphnehanse11 (frame + coder-panel -operation, per #230 §9.3). +operation, per #230 §9.3). Prerequisite order is ADR 0004 (#224), +then the cross-wave provenance artifact (#235), then this manifest. +The exact prerequisite heads are merged into this branch; neither is +a movable forward reference here. ## 1. The assignment under audit @@ -126,10 +129,12 @@ Registered parameters — `REFEREE` slots per ADR 0004 §6.1: | `P_design` | REFEREE | | `alpha` (one-sided) | REFEREE | | `1 - beta` | REFEREE | -| Multiplicity rule | REFEREE (recommended: intersection-union across the 4 arm-(a) strata with joint power computed by simulation) | +| Multiplicity rule | REFEREE | | Recall: gates or reported-with-bound | REFEREE | | `P_floor_run` (share of runs correctly delimited, one-sided lower bound) | REFEREE | | Run-arm link-coding budget ceiling | REFEREE | +| `q_link_design` (link-level indeterminate rate used for sizing) | REFEREE | +| Calibration-repeat agreement floor and failure action | REFEREE | | False-continuation / false-break decomposition | registered: always reported separately, each with its own bound | **No-revisit clause (registered)**: every `REFEREE` slot in this @@ -144,9 +149,30 @@ the §7 retire-and-reissue remedy. that resamples **workers** — the registered clustering unit — from the *actual frame*, which exists before the draw (#235 sizes it). The design effect is therefore **measured from the frame's -cluster-size distribution, not assumed**. The indeterminate/unusable -inflation is a named parameter: prior 10%, superseded by the -calibration round's measured rate if that is larger. +cluster-size distribution, not assumed**. Indeterminate inflation is +arm-specific. Arms (a)/(b), whose units are individual link codings, +use the referee-filled `q_link_design`. For each arm-(c) run with +`k_r` constituent coding opportunities (internal links plus the +observed terminal opportunities), the registered planning probability +is + +`q_run,r = 1 - (1 - q_link_design)^k_r`. + +The arm-(c) simulation applies that probability to the actual run-length +distribution in each stratum; it never applies a scalar link-level +inflation to runs. This is a sizing model, not an assumption that +constituent adjudication outcomes are independent in the reported +estimator. The result artifact reports realized run-level indeterminacy +directly. + +**Fail-closed pre-draw trigger (registered)**: if any arm-(c) stratum's +powered target after the formula above exceeds either its frame count +or the referee-filled link-coding budget, the draw does not occur and +the design returns to the referee before any audit label exists. It may +not silently omit full-year or seam runs, replace the formula with 10%, +or license E5 from pair precision. A calibration rate observed after +the draw never authorizes a supplemental sample or target revision; it +is reported and may produce the already-registered graded stop. **Worked example** (illustrative only, not a proposal): for a simple-random operative stratum, `P_floor = 0.95`, @@ -156,7 +182,7 @@ simple-random operative stratum, `P_floor = 0.95`, `n` are floor illustrations only; registered targets come from the worker-resampling simulation above. Four strata at the example numbers imply an arm-(a) total near 550 adjudications before -inflation — feasible for a two-coder panel. +inflation. **Arm (b) target (registered formula)**: per stratum, the true-counterpart prevalence prior `π` is the #235 excess rate for @@ -175,10 +201,16 @@ codings) and capped by the registered ceiling above. ## 5. Coding protocol (ADR 0004 §2.4) -- **Blinding**: coders see both months' job-record fields with all +- **Outcome blinding**: coders see both months' job-record fields with all `EJB` job IDs **masked**, and never see the matcher outcome (same/different ID), the #235 signature flag, any downstream gate quantity, or the other coder's decision. +- **Condition blinding is unattainable**: the job-record evidence needed + to adjudicate an employer attachment can reveal whether a sheet came + from a stay or separation arm. The design therefore makes no + condition-blinding claim. It relies on outcome masking and coders + external to the assignment implementation. The draw artifact names + the holder of the ID-unmasking key; that holder cannot code a sheet. - **Labels**: `same_employer`, `different_employer`, `insufficient_evidence`, under a frozen coding manual (`docs/design/e4_e5_coding_manual_v1.md`, to be committed before @@ -192,23 +224,37 @@ codings) and capped by the registered ceiling above. partial-identification bounds also reported. Hard cases are never dropped after labels are known. - **Calibration**: coders first train on vetted cases excluded from - both arms; blinded repeats (10% of assignments) measure continuing - accuracy and agreement. + all three arms; blinded repeats (10% of assignments) measure + continuing accuracy and agreement. The referee-filled agreement + floor and its failure action are fixed before the draw and may not + be revised after a repeat label exists. - The precision test publishes the sensitivity bound to reference-label error per ADR 0004 (the label is hand-adjudicated reference truth, not infallible ground truth). ## 6. Provenance (ADR 0004 §2.5) +The prerequisite cross-wave artifact is +`runs/crosswave_jobid_check_draft_v0.json`, SHA-256 +`33d4a7be88b417cb980ad4b12e65e1310eabd541a4926673e2414eead20941f1`; +its builder is `scripts/build_crosswave_jobid_check.py`, SHA-256 +`3887d392e3d978e2c5788e9e03557101129d9f759917037b756dba6e3ed0cb33`, +last changed at `35926fe708bea8a9b5cc22791505f245c8ad35be`. +Those committed objects pin the frame counts and signature parameters +used here. + The draw artifact (`runs/e4_e5_audit_draw_v1.json`, committed when the sample is drawn) will contain: reader version and pu-file sha256s, frame query (this document's §2 verbatim), stratum definitions and inclusion probabilities, the random seed (**registered now: 20260717**), sha256 of each selected job-pair's public-use identifiers, coding-manual and evidence-sheet versions, -coder-assignment protocol, and counts. Replacements or exclusions -are logged; the sample is never silently refreshed. All fields are -public-use-derived; no restricted data exists in this design. +coder-assignment protocol, and counts. The draw script itself must be +committed and reviewed before execution and pin the RNG algorithm, +sorted frame-key order, deterministic stratum allocation, and one-draw +rule in addition to the seed. Replacements or exclusions are logged; +the sample is never silently refreshed. All fields are public-use- +derived; no restricted data exists in this design. ## 7. No leakage (ADR 0004 §2.6)
Workstream A — Daphne (person side)Workstream B — Vahid (firm side)
SIPP job-level reader (label-verified, family.py pattern); CPS NOEMP/tenure loaders; ADR drafted jointly · freeze C1/C2Target pipeline: SUSB/BDS/QWI/J2J/JOLTS extracts committed with provenance notes; canonical banding proposal (C2)
SIPP noise-floor runs; seam-vs-J2J reconciliation run; draft E3–E5/E8–E10 thresholdsAggregate-side floor studies; target/gate partition; draft E1/E2/E6/E7/E11 thresholds · joint: referee round, lock C3
SIPP job-level reader (label-verified, family.py pattern); CPS NOEMP/tenure loaders; ADR drafted jointly · freeze IC1/IC2Target pipeline: SUSB/BDS/QWI/J2J/JOLTS extracts committed with provenance notes; canonical banding proposal (IC2)
SIPP noise-floor runs; seam-vs-J2J reconciliation run; draft E3–E5/E8–E10 thresholdsAggregate-side floor studies; target/gate partition; draft E1/E2/E6/E7/E11 thresholds · joint: referee round, lock IC3
Phase-0 QRF imputation of spells + attributes onto CPSCalibration of the imputed file to partitioned QWI/SUSB cells; E1/E7 evidence artifacts
Phase-1 transition candidates registered; one-shot runs against locked gatesBLM firm-type register prototype; E12 feasibility study (are published AKM/coworker moments sufficient targets?)
Joint: phase-2 go/no-go review with Max, based on committed gate evidence + the E12 identification story