From 908456c492e1cf0d24a8d05d8bd0b29ac31ce671 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Tue, 18 Aug 2026 22:11:47 -0400 Subject: [PATCH 1/7] Add stability measurement spec (post-adversarial-review v1) Three layers: answer stability (repeats + variance decomposition against household-sampling SE), reasoning stability (mechanism-extraction judge with gold-set/cross-judge validation), counterfactual consistency (+$1,000 head-wage twins vs PolicyEngine true deltas). Spec adopted after a 5-reviewer adversarial workflow (36 findings); load-bearing findings re-verified directly: LiteLLM cache replay would have faked repeat stability, +$1k produces zero binary flips and 247 nonzero amount deltas, judge is Tier-2 classification not >90% extraction. Co-Authored-By: Claude Fable 5 --- docs/stability_spec.md | 567 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 567 insertions(+) create mode 100644 docs/stability_spec.md diff --git a/docs/stability_spec.md b/docs/stability_spec.md new file mode 100644 index 0000000..d67e96c --- /dev/null +++ b/docs/stability_spec.md @@ -0,0 +1,567 @@ +# Stability measurement spec + +Version: `2026-08-18-v1`. Status: **spec adopted after adversarial review** +(see the review section at the end); implementation follows this document, +and every paid run is gated on explicit approval. + +## Why + +"The Stability Trap: Evaluating the Reliability of LLM-Based Instruction +Adherence Auditing" (Shergadwala, FAccT '26, arXiv:2601.11783) shows LLM +judges can reach >99% verdict agreement while the reasoning behind those +verdicts is unstable — as low as ≈19% reasoning stability on the paper's +worst quantitative rubric check (a judge-side word-count task; its +quantitative checks cluster roughly 19–30%), with >90% stability observed +only on verbatim closed-list extraction tasks. The paper's instability is +measured on the *judge* side — judges doing arithmetic — which is why this +spec's own judge is never allowed to compute or compare anything; the trap +it names, high verdict agreement masking fragile reasoning, is the risk +this spec measures in PolicyBench's *benchmarked models*. + +PolicyBench is exposed in three places: + +1. **Answer stability is asserted, not measured.** The manuscript's + Uncertainty section states that the benchmark sample, not run-to-run + model variation, dominates uncertainty (`paper/index.qmd`, "These + intervals are a manuscript artifact…"), with no committed repeated-run + artifacts behind it. The machinery exists (`eval-no-tools-repeated`, + `analyze --runs-dir`, `run_stability_by_model`) and has never been run + for the record. +2. **Explanations are collected but unexamined for stability.** Every + scored response carries a required per-output explanation. The paper + correctly disclaims them as faithful reasoning traces — but if a model + gives the same right answer twice citing two different mechanisms, that + is measurable evidence about its reliability as a policy explainer, + independent of trace faithfulness. +3. **Counterfactual behavior is untested.** Stewart ("Beyond Explanation", + arXiv:2603.22716) proposes counterfactual interrogation rights: + affected parties should be able to probe a deciding system with + modified inputs and observe whether outcomes change — benefit + determinations are a central example, and the interrogation must hit + the pinned system version that made the decision. PolicyEngine answers + such probes exactly; models' implied counterfactual responses have + never been compared to it. Here the models are interrogated as if they + were the deciding system, with PolicyEngine computing the true + counterfactual and per-arm provider fingerprints standing in for the + pinned-version requirement. + +Everything ships under the frozen-board discipline of +`sensitivity/claude-thinking-2026-08.md` and +[policybench#139](https://github.com/PolicyEngine/policybench/issues/139): +the 30-model board is never edited; stability results publish as labeled +sensitivity artifacts (a `sensitivity/*.md` doc plus release-attached +CSVs), or fold into the v2 board protocol as a versioned re-run. + +## Shared definitions + +- **Row**: one `(model, scenario_id, variable)` cell; `variable` is an + expanded output id (person-level coverage flags expand per person). The + frozen US manifest yields 1,984 rows per model: 1,000 amount rows and + 984 binary rows. +- **Run**: one complete pass over a fixed scenario manifest for one model + under one serving treatment, tagged `run_id` (`run_000`, …), produced by + `eval-no-tools-repeated`, which routes through `run_single_no_tools`, so + per-model cards (answer contract, chunk size, timeouts, completion + budgets, in-request repair rounds) apply per request exactly as in board + runs. Board *pipelines* additionally ran chunk-completeness gating, + post-run response retries, and row repairs; repeats deliberately run the + single-shot sync path, so repeat coverage is a lower bound on board + coverage and layer-1 claims are scoped to within-repeat-set comparisons. +- **Repeat set**: exactly K runs (K pinned per repeat set; default 3, + ladder to 5 — never mixed within one table) of the same manifest, model + set, programs, prompt contract, and condition. The serving condition + (effective `tool_choice`, chunk override) is recorded in each run's + resume-metadata sidecar **(new in this change** — today's sidecar + records neither `POLICYBENCH_TOOL_CHOICE` nor + `POLICYBENCH_CHUNK_OVERRIDE`**)**, and `stability-report` refuses to + pool runs whose fingerprints mismatch. +- **Cache discipline (hard requirement)**: as shipped, the CLI enables the + LiteLLM disk cache for `eval-no-tools-repeated` and every request sets + `caching: True` with no run-specific key material + (`policybench/cli.py` enable-cache set, `eval_no_tools.py` + request kwargs, `cache.py` namespace = + `policybench:{PROMPT_CONTRACT_VERSION}`), so runs 2..K would replay run + 1's cached completions and every stability metric would measure the + cache. Implementation therefore (a) stops enabling the disk cache for + `eval-no-tools-repeated`, and (b) makes `stability-report` **hard-fail** + when any run's spend ledger contains a `cache_hit: true` record, and + report the count of `provider_response_id` values shared across runs as + a residual replay diagnostic. The same guard applies to both + counterfactual arms. +- **Parse, per row type**: an amount row is parsed iff its prediction is a + non-null numeric; a binary row is parsed iff `binary_flag(prediction)` + is not `None`. Binary rows with numeric but non-0/1 predictions are + counted in their own `invalid_binary_rate` bucket — never as flips, + never as exact. +- **Mutual exact** (prediction vs prediction): headline tolerance applied + between two predictions — amounts `|p_a − p_b| ≤ $1`; binary rows equal + parsed flags. Unparsed values are never mutually exact. +- **Exact-correct** (prediction vs reference): existing headline semantics + (`row_hit_scores(...)["exact"]`). +- **Reference discipline**: the frozen snapshot's references were + generated with policyengine-us 1.755.4; the current environment installs + 1.723.0 through the `policyengine[us]==4.16.1` bundle. Regenerated + base-arm references agree with the frozen CSV on 1,983/1,984 rows within + $1 (max drift $67.74, one state-tax row, measured 2026-08-18). Rules: + layer-1 scoring uses one fixed reference file for all runs (the frozen + CSV, for board comparability); layer 3 regenerates **both** arms with + the installed engine and never mixes frozen with regenerated references + inside a delta; every export records the installed policyengine-us + version and the base-vs-frozen drift count. +- **Scenario manifest**: `paper/snapshot/20260501/us_scenarios.csv` + (US-only in this iteration). + +## Layer 1 — answer stability + +**Question**: how much does a model's score and row-level output move when +the identical request is repeated, and is that movement small relative to +household-sampling uncertainty (the manuscript's unmeasured claim)? + +### Existing machinery (unchanged) + +`eval-no-tools-repeated` → `run_000.csv…`; `analyze --runs-dir` → +`summarize_runs_by_model` → `run_stability_by_model`. These stay as-is. + +### New row-level metrics (`row_stability_by_model`) + +Computed per model. Pair metrics pool the C(K,2) unordered run pairs per +row; row metrics are K-way. All rows of a scenario come from 1–3 +completions per run, so pooled pair counts overstate precision ~20×: +every pooled rate ships with a **scenario-level cluster bootstrap CI** +(households resampled with replacement, all their rows carried along), and +`n_scenarios` publishes beside `n_rows` / `n_run_pairs`; `n_run_pairs` is +never quoted as precision. + +- `unanimous_parse_rate@K`, `coverage_flip_rate`: rows parsed in all K + runs / parsed in ≥1 and unparsed in ≥1 (per-type parse definition). +- `invalid_binary_rate`: binary rows with ≥1 numeric non-0/1 prediction. +- `answer_flip_rate`: share of run pairs, both sides parsed, that are not + mutually exact. Pair-level, so K-invariant. +- `unanimous_exact_answer_rate@K`: rows with all K runs parsed and + pairwise mutually exact (labeled with K; increases mechanically with K, + so never compared across different K). +- `verdict_flip_rate_parsed` / `verdict_flip_rate_all`: share of run + pairs whose exact-correct verdicts differ — the parsed variant requires + both sides parsed (the stability-trap headline analogue); the all-rows + variant scores unparsed as wrong (headline-consistent) and therefore + absorbs coverage flips; both denominators are stated in the export. +- `consistently_wrong_divergent_rate`: **amount rows only** (a parsed + wrong binary flag can only equal the one wrong value, so binary + divergence is impossible by construction), among rows with all K runs + parsed and all K wrong: the share where wrong answers are not mutually + exact — "consistently wrong, differently wrong." The signature pairing + is a low `verdict_flip_rate_parsed` with a high + `consistently_wrong_divergent_rate`: stable grades, unstable outputs. + +### Variance decomposition (`stability_variance_decomposition`) + +Per model: + +- `run_score_std`: sample std (ddof=1) of the K run-level headline scores + (household-impact-weighted exact rate, per `run_id`, via + `household_headline_scores(metric="exact")`). +- `run_score_std_ci_low/high`: chi-square CI on that std (df = K−1; + multipliers at K=3: [0.578, 4.415]; K=5: [0.649, 2.372]). Normality of + run scores is a reasonable CLT approximation (each is a mean over ~100 + households); the df, not normality, is the binding limitation. +- `sampling_se`: per run, `std(household scores, ddof=1) / sqrt(n)` with + `n` = the positive-weight household count after the weighting drop, + averaged across runs. +- `run_to_sampling_ratio` with an explicit decision rule: the manuscript + claim "sampling dominates run-to-run variation" is **confirmed** iff + `run_score_std_ci_high` (one-sided 95% upper bound) < `sampling_se`. At + K=3 that requires a point ratio below ≈0.23 (the detectable-effect + floor, printed in the artifact); otherwise the result is reported as + inconclusive-at-K, never as a confirmation. +- Pooled roster estimate: run-score deviations pooled across the M models + of the repeat set (M·(K−1) df) give a roster-level + `pooled_run_to_sampling_ratio` with usable power, published with a + per-model homogeneity table (each model's contribution to the pooled + sum of squares). + +### Runbook (paid; gated) + +```bash +RUN_DIR=results/local/stability_20260818 +K=3 +MANIFEST=paper/snapshot/20260501/us_scenarios.csv +FROZEN_REFS=paper/snapshot/20260501/us_reference_outputs.csv + +# One runs-subdirectory per provider group (separate invocations sharing +# one output dir would collide on run_NNN.csv resume metadata). Execution +# within an invocation is serial; schedule groups in parallel terminals. +# A provider 429 beyond in-layer retries exits the command; rerunning the +# identical command resumes from the checkpoint. +uv run policybench eval-no-tools-repeated \ + --country us \ + --scenario-manifest "$MANIFEST" \ + --output-dir "$RUN_DIR/us/runs/openai" \ + --repeats "$K" \ + --model gpt-5.4-mini + +uv run policybench eval-no-tools-repeated \ + --country us \ + --scenario-manifest "$MANIFEST" \ + --output-dir "$RUN_DIR/us/runs/gemini" \ + --repeats "$K" \ + --model gemini-3.7-flash + +# Repeated-run invocations run cache-free by design (see Cache discipline). + +uv run policybench analyze \ + -g "$FROZEN_REFS" \ + -p "$RUN_DIR/us/runs/openai/run_000.csv" \ + -s "$MANIFEST" \ + -o "$RUN_DIR/us/analysis" \ + --app-data-output "$RUN_DIR/us/analysis/data.json" \ + --runs-dir "$RUN_DIR/us/runs/openai" + +# --app-data-output must ALWAYS be re-pointed inside the run directory: +# its default is app/src/data.json, and the local app prefers that file +# over the pinned frozen artifact — writing it would violate the +# frozen-board discipline this spec operates under. + +uv run policybench stability-report \ + --runs-dir "$RUN_DIR/us/runs/openai" \ + --runs-dir "$RUN_DIR/us/runs/gemini" \ + --reference-outputs "$FROZEN_REFS" \ + --scenario-manifest "$MANIFEST" \ + --output-dir "$RUN_DIR/us/stability" +``` + +`stability-report` accepts repeated `--runs-dir` flags (model sets must be +disjoint across dirs), validates run-fingerprint consistency, enforces the +cache guard, and emits `stability_metadata.json` including each run's +effective serving config diffed against +`paper/snapshot/20260501/model_serving_config.json`. + +## Layer 2 — reasoning stability + +**Question**: when a model gives the same answer twice, does it cite the +same mechanism twice? Headline: the **"right answer, unstable reasoning" +rate** — among repeat pairs where the answer is stable and correct, the +share whose explanations invoke different mechanisms. + +### The judge's epistemic status (corrected after review) + +Mapping an explanation onto 12 mechanism labels is **semantic +classification — the Stability Trap paper's Tier-2 regime, which it +brackets at roughly 35–83% reasoning stability — not the verbatim +closed-list extraction where it observed >90%**. The design keeps the +judge as close to the stable regime as the task allows (single text in, +discrete labels out, no pairs, no comparison, no arithmetic; the paper's +judge-side instability was specifically judges doing arithmetic), but +reliability is **established empirically by the validation battery below, +never asserted by construction**. The paper itself measured reasoning +similarity with a deterministic pipeline (regex fingerprinting + +sentence-embedding clustering); this spec prefers taxonomy labels because +they are interpretable policy mechanisms aligned with PolicyBench's audit +annotation system and comparable across models — accepting, in exchange, +a judge that must itself be validated. The deterministic numeric-claim +channel below is the in-design hedge; the pre-registered fallbacks if the +gate fails are (1) reformulate as 12 independent per-label yes/no rubric +questions (the binary-verdict form the paper found most stable), then +(2) collapse to 5 super-domains +(`income_and_deductions` = {taxable_income_or_deductions, asset_resource, +period_annualization}; `rates_and_phaseouts` = {thresholds_rates, +credit_phaseout}; `eligibility` = {categorical_eligibility, +age_disability, household_unit_or_filing_status}; `program_specific` = +{health_coverage, payroll_tax_base, state_local_rule}; `other`). + +### Pair selection (deterministic) + +For each row and each of the C(K,2) run pairs: + +- **Answer-stable pair**: both runs parsed, mutually exact. +- **Stable-and-exact pair**: answer-stable AND both runs exact-correct. + +Mechanism grading applies to answer-stable pairs; both strata are +reported with their denominators and the share of all pairs they cover. + +### Mechanism labels + +`MECHANISM_LABELS` = `FAILURE_SUBTYPE_VALUES` minus `missing_output` (a +parse status, not a mechanism): 12 labels, multi-label extraction (a +payroll-tax derivation citing the 6.2% Social Security rate up to the +wage base maps to `{payroll_tax_base, thresholds_rates}`). + +### Judge (LLM, discrete extraction only) + +- Input: one explanation text plus `(variable, country, year)` context; + output strict JSON `{"labels": [...]}` validated against + `MECHANISM_LABELS`; invalid → one retry → error row (excluded from + metrics, counted in `judge_error_rate`). +- Prompt: fixed instructions, label glossary, and 3–5 static few-shot + examples drawn from the committed reference-explanation CSV, committed + verbatim in code. One prompt version grades model and reference + explanations alike; the version string rides in every output row. +- Anchoring: the reference explanation for each `(scenario_id, variable)` + (coverage: 1,984/1,984 cells) is extracted once through the same judge; + the same-row reference is never shown while grading a model explanation. +- Judge calls never enable the LiteLLM disk cache. The module's own + dedup cache is keyed by + `sha256(prompt_version ‖ variable ‖ normalized_text ‖ grade_pass)`, so + validation re-grades are genuinely independent calls while identical + texts within one pass share one call. Verbatim-identical explanation + pairs short-circuit to "same labels" with no call; the share of stable + pairs resolved by this short-circuit is published per model (see the + asymmetry note under Metrics). +- Judge models: primary `--judge-model` (default gemini-3.7-flash), + secondary `--cross-judge-model` (default gpt-5.4-mini), temperature 0, + ids + resolved versions recorded. + +### Validation battery (the load-bearing control) + +1. **Gold-set validity gate.** A committed, stratified gold set of ~100 + explanations (drawn from reference explanations across output groups), + labeled by the developers and versioned in-repo. The gate: judge + exact-set accuracy vs gold ≥ 0.80, flagged CI-aware (fail when the 95% + lower bound sits below 0.70). Per-label agreement and prevalence + tables publish alongside. Thresholds are provisional until the rung-1 + pilot and are recorded in the export either way. +2. **Cross-judge reliability.** The ~10% validation sample (deterministic + hash of the dedup key) is extracted by both judge models; + `cross_judge_set_agreement` ≥ 0.90 flagged CI-aware. This is the + reliability statistic — same-model temp-0 re-grading measures serving + determinism, not extraction validity, so it is reported only as a + `determinism_floor` and gates nothing. +3. **Agreement statistics that survive skewed prevalence**: per-label + Gwet's AC1 and raw agreement with a prevalence column (the motivating + paper's own choice for this regime), Krippendorff's alpha over label + sets with Jaccard distance; Cohen's κ only as a supplementary column + for labels with ≥20 validation occurrences (κ is prevalence-paradox + degenerate below that). +4. **Paper-comparable statistic**: per-row dominant-label-set share + across the K runs (the R_stab analogue). Pairwise agreement rates are + never numerically compared with the paper's percentages (pairwise 0.90 + corresponds to a dominant-share ≈0.95). +5. **Judge noise floor / attenuation.** From double-graded texts: + `pair_noise_floor` = P(two independent extractions of the same text + disagree). The headline publishes raw and attenuation-adjusted + (`max(0, (observed − floor) / (1 − floor))`), plus the + verbatim-short-circuit share — a model that repeats canned text incurs + zero judge noise while a paraphrasing model incurs it twice, so the + headline restricted to non-identical-text pairs is the like-for-like + companion. + +If the gate fails, layer-2 outputs still export, flagged +`judge_below_reliability_bar`, and the headline is not reported; the +pre-registered fallbacks above are the next step, not ad hoc re-prompting. + +### Metrics (deterministic, from extractions) + +Per model: + +- `mechanism_agreement_rate_stable[_exact]`: identical label sets among + answer-stable [stable-and-exact] pairs. +- **`right_answer_unstable_reasoning_rate`** = 1 − + `mechanism_agreement_rate_stable_exact`. Conditioning on + stable-and-exact is a composition-sensitive denominator, so it ships + with three companions: (a) the **joint rate** over ALL pairs (stable ∧ + exact ∧ mechanism-divergent — fixed denominator, cross-model comparable + by construction); (b) a **direct-standardized** rate reweighted to the + pooled stable-pair output-group composition of the report's models; (c) + a suppression rule — no headline below 200 stable-and-exact pairs — and + a per-model stratum-composition table. +- `mechanism_jaccard_mean_stable[_exact]`: graded softening. +- `numeric_claim_jaccard_mean`, `numeric_claim_disjoint_rate` + (deterministic channel): currency amounts (rounded to $1) and + percentages (rounded to 0.1pp) regex-extracted from each explanation, + excluding values within $1 of the row's own prediction (the restated + answer would inflate overlap); disjoint = both non-empty, empty + intersection. +- `reasoning_unstable_strict_rate`: labels differ OR claims disjoint + (sensitivity companion, not the headline). +- `reference_alignment_rate`: among exact-correct run-explanations + (row-level), share whose label set equals the reference explanation's. +- All pooled pair rates carry scenario-level cluster bootstrap CIs, as in + layer 1. + +## Layer 3 — counterfactual consistency + +**Question**: when one input changes, does the model's answer change the +way the law says it should? + +### Perturbation + +One twin per scenario: `adults[0]` (the head in all 100 manifest +scenarios) `employment_income` += **$1,000**. Twins are built at the +`Scenario` level — `scenario_from_dict` → perturb → `id = f"{base}__cf1k"` +→ `scenario_manifest()` — never by editing the frozen CSV, so the embedded +id, recomputed `total_income`, and `scenario_json` stay mutually +consistent (`load_scenarios_from_manifest` hard-fails on id mismatch). +The perturbed manifest adds `base_scenario_id`, `perturbed_field`, +`perturbation_amount` columns (ignored by the loader) and mirrors them in +`scenario_json.metadata.counterfactual`. + +An absolute increase is the only perturbation well-defined for every +household: 44/100 heads have zero wages, where percentage or decrease +perturbations are degenerate. For those 44 twins the rendered prompt gains +a wages line that the base prompt omits entirely (the template renders +head wages only when nonzero, and the prompt preamble instructs "treat any +unlisted numeric input as 0", which keeps the base well-posed); these +`first_dollar` pairs are flagged and reported as a sub-split beside the +wage-shift pairs. An optional −$1,000 arm on the 55 households with wages +≥ $1,000 is a costed rung extension (~0.55 of an arm; measured +expectation: 135/550 nonzero amount deltas, 1 eligibility flip) that +would separate schedule knowledge from a "wages up ⇒ taxes up" direction +prior; percentage shocks and demographic perturbations are future +conditions. + +### Measured truth-delta reality (installed policyengine-us 1.723.0, 2026-08-18) + +The entire design below is disciplined by the measured distribution of +true deltas on this manifest (regenerate via +`counterfactual-manifest --write-truth-deltas`): + +- 1,984 matched rows: 1,000 amount, 984 binary. **Zero** binary + eligibility flips. 247 amount rows with `|Δ| > $1`; zero-delta share + 87.55%. +- Nonzero rows by group: payroll_tax 100 (median +$76.50), federal income + tax 46, state income tax 44, federal refundable credits 21, state + refundable credits 19, snap 13 (median −$237.60), ssi 3, tanf 1; + local_income_tax and self_employment_tax 0. +- Sign split 216 positive / 31 negative. +- `|Δ| ≥ $500` selects 4 rows (two +$550 refundable-credit phase-ins, one + SSI −$500.003 that clears the cut by three tenths of a cent, one TANF + −$750) — no SNAP or Medicaid boundary anywhere near it. + +### Arms + +Truth: both arms' references computed together with the installed engine +(`true_delta = ref_perturbed − ref_base` per row; deltas rounded to cents +before any threshold comparison). Model: one run per arm under identical +serving treatment; the base arm reuses the layer-1 repeats when run in the +same session — and once repeats exist, `pred_delta` is computed against +**every** base repeat, giving a delta distribution rather than one +arbitrary pairing. Per-arm `provider_resolved_model` / +`provider_system_fingerprint` summaries are recorded, and matched pairs +whose arms resolved to different provider fingerprints are counted and +flagged (the Stewart pinned-version concern, at the level an API consumer +can observe). Nothing in any prompt names the twin relationship; this +measures *implied* counterfactual response, not in-context delta +reasoning. + +### Metrics + +Primary reporting universe: the **247 nonzero-true-delta amount rows**, +with the six groups at n ≥ 10 (payroll, federal tax, state tax, federal +refundable, state refundable, snap) pre-registered as reportable; +ssi/tanf/local/self-employment publish row counts only. Amount and binary +rows are never pooled into one rate. Pooled-1,984-row rates appear only as +a secondary column beside the zero-delta baseline (`pred_delta ≡ 0`), +which scores 87.55% on pooled delta-exact by construction. + +Per model, on matched rows with both arms parsed (`delta_coverage` +reported): + +- `delta_exact_rate`: `|pred_delta − true_delta| ≤ $1`. +- `delta_within_10pct_rate`: band = `max($1, 0.10 · |true_delta|)` — the + floor makes within-10% a strict superset of delta-exact (65/247 nonzero + true deltas sit below $76.50, where a bare relative band would invert + the two metrics; this deliberately diverges from the level metrics' + zero-floor semantics). +- `delta_sign_agreement` on nonzero-true rows, with `sign(x) = 0` for + `|x| ≤ $1`, reported **per direction** (positive-recall and + negative-recall separately — 216/31 asymmetry means a "wages up ⇒ taxes + up" prior scores ≈0.9 pooled, and the negative column is where + schedule knowledge shows). +- `delta_mae` (nonzero rows; pooled as secondary). +- **Large-response detection** (renamed from "cliff" — the $1,000 shock + crosses no observed program boundary): true cut `|Δ| ≥ $200` (n=40, + spanning 6 groups, with a margin from the nearest mass point), predicted + cut ≥ $100 in the true direction; exact binomial CI; suppressed below + n=20 qualifying rows; the qualifying-row list is committed in the + artifact so membership is auditable. +- Binary rows: the reference produces **zero flips**, so flip + recall/precision are undefined and are **not published as rates**; the + export carries raw counts instead — `n_reference_flips = 0` and each + model's `n_model_flips` (false-positive flips, informative on its own). + A flip-capable perturbation (magnitude verified to cross boundaries the + way this spec's truth-delta audit was computed, target ≥ ~20 reference + flips) is a pre-registered future rung, not a promise of this one. +- **Noise floor** (requires layer-1 repeats): every delta statistic is + paired with the same statistic computed between two base repeats + (true delta ≡ 0), on the **same row subset** (nonzero-true rows, + large-response rows, binary rows — the base-base false-flip count is + the binary floor). "Distinguishable from repeat noise" is a named test: + scenario-level paired cluster bootstrap of (signal statistic − floor + statistic); indistinguishable iff the 95% interval covers 0. + +## Artifacts and discipline + +- New CLI: `stability-report`, `reasoning-stability` (with + `--deterministic-only` to skip the judge), `counterfactual-manifest` + (twin manifest + both-arm references + optional truth-delta export), + `counterfactual-report`, `stability-cost-plan`. +- Every export carries `stability_metadata.json`: spec version, prompt + contract, condition fields, cache mode + guard result, judge + models/prompt versions, installed PE versions + base-vs-frozen drift + count, manifest hash, run fingerprints, serving-config diff vs the + snapshot registry. +- Results publish as `sensitivity/stability-.md` + release-attached + CSVs. Never an edit to frozen board scores or `app/` data; every + `analyze` invocation in the runbook re-points `--app-data-output` into + the run directory (its default writes `app/src/data.json`, which the + local app prefers over the pinned artifact). +- The v2 board plan (#139) is the natural carrier for a full-roster + repeat protocol; rungs 1–2 de-risk the metrics on the v1 manifest. + +## Cost ladder (measured basis; no spend authorized here) + +One full-roster board-condition run ≈ $316 recorded + Claude Fable 5 +≈$54.10 (run-level usage; $0.541/hh) + two unpriced models at override +prices (grok-build-0.1 ≈$4.50, gemini-3.6-flash ≈$5.50) ≈ **$380/run**. +Repeats run cache-free, so arms price at full run cost. Judge basis: +≈800 input + 30 output tokens/call at gemini-3.7-flash overrides +($0.75/$3.75 per 1M) = $0.0007125/call; 5,952 calls per model per K=3 +repeat set = **$4.24/model pre-dedup** (verbatim dedup lowers it; the +pilot reports the realized rate); one-time reference-anchor extraction +(1,984 calls) ≈ $1.41. + +| Rung | Scope | Model spend | Judge spend | Purpose | +|---|---|---|---|---| +| 0 (this PR) | mocks + fixtures; free local truth-deltas | $0 | $0 | metrics, tests, dry run | +| 1 pilot | gpt-5.4-mini ($0.58/run) + gemini-3.7-flash ($1.44/run) × (3 repeats + 1 cf arm) | ≈ $8 | ≈ $17 double-graded (2 models × 3 runs × 1,984 × 2 passes), plus $1.41 anchors | end-to-end validation; gold-set + cross-judge gates | +| 2 subset | top-8 board models as ranked (sol, kimi-k3, luna, inkling, gpt-5.5, terra, grok-4.5, fable) = $158.5/run × 4 arms | ≈ $634 (≈$446 without kimi-k3; ≈$418 without fable) | ≈ $39 (incl. 10% validation) | reportable sensitivity doc | +| 3 full | 30-model v1 roster × 4 arms ≈ $1,520, or fold into the 20-model v2 protocol | — | ≈ $141 (incl. validation + anchors) | board-grade artifact | + +Optional −$1,000 arm (55 households): ≈0.55 × one arm per model. Every +paid rung is gated on explicit approval; boring before billed. + +## Adversarial review record (2026-08-18) + +Per process, the metric definitions were adversarially reviewed before +implementation: five lens-diverse reviewers (statistical validity, +literature fidelity — both papers fetched and read, PolicyBench +integration, counterfactual design, ops/cost) produced 36 findings; the +independent verification stage was interrupted by an account rate limit, +so the maintainer-agent re-verified the load-bearing findings directly +against code and data before adopting them. Highlights that reshaped v1: + +- **Cache replay (4 reviewers independently)**: the drafted runbook would + have had repeats 2..K replay the LiteLLM disk cache, making "perfect + stability" a cache artifact — the exact stability-trap failure the spec + exists to catch. Fixed via cache-free repeats + a hard ledger guard. +- **Truth-delta audit (verified by independent recomputation)**: +$1,000 + produces zero binary flips, 247/1,000 nonzero amount rows, a 216/31 + sign split, and no $500-scale program boundary — layer 3 was rebuilt + around the measured distribution (nonzero-stratum primary reporting, + flip counts instead of flip rates, "large response" instead of "cliff", + per-direction sign recall). +- **Judge regime honesty**: 12-label extraction is the paper's Tier-2 + classification regime (35–83% observed stability), not its >90% + verbatim-extraction regime; the validation battery, not construction, + carries the reliability claim, with gold-set + cross-judge gates + replacing circular temp-0 self-agreement, AC1/Krippendorff replacing + prevalence-fragile κ, and pre-registered fallbacks. +- **Statistical decision rules**: chi-square-CI-based confirmation rule + (K=3 detectable floor ≈0.23) plus a pooled roster estimator for the + sampling-dominance claim; scenario-level cluster bootstrap CIs on all + pooled rates; K-invariant pair-level flip metrics; composition-adjusted + companions for the conditioned headline. +- **Ops corrections**: real `analyze` flags; `--app-data-output` + re-pointed away from `app/src/data.json`; per-provider-group run dirs; + serving-condition fields added to resume metadata; engine-version + discipline (1.755.4 snapshot vs 1.723.0 installed, 1,983/1,984 rows + within $1); corrected arithmetic throughout the cost ladder. From 398022d335aeb80c082cac79c578eed2ca308dd1 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Tue, 18 Aug 2026 22:21:56 -0400 Subject: [PATCH 2/7] Layers 1+3 core: stability metrics, cache guard, counterfactual deltas (TDD) - policybench/stability.py: pair/row flip metrics with scenario-cluster bootstrap CIs, chi-square variance decomposition with the CI-based sampling-dominance decision rule, pooled roster estimator, spend-ledger cache-contamination guard, multi-dir runs loading. - policybench/counterfactual.py: Scenario-level twin builder (+$1k head wages), both-arm truth deltas rounded to cents, nonzero-stratum delta metrics with the max($1, 10%) band, per-direction sign recall, binary flip counts (never rates), large-response detection with exact binomial CI and suppression, zero-delta baseline, row-matched noise floor and signal-vs-noise bootstrap test. - Harness: eval-no-tools-repeated no longer enables the LiteLLM disk cache (repeat replay would fake stability); serving condition (POLICYBENCH_TOOL_CHOICE / CHUNK_OVERRIDE) now part of the response contract fingerprint; repeat sets write runs_metadata.json and reject mixed model groups per directory. Co-Authored-By: Claude Fable 5 --- policybench/cli.py | 8 +- policybench/counterfactual.py | 571 ++++++++++++++++++++++++++ policybench/eval_no_tools.py | 77 ++++ policybench/stability.py | 687 ++++++++++++++++++++++++++++++++ tests/test_counterfactual.py | 339 ++++++++++++++++ tests/test_stability.py | 348 ++++++++++++++++ tests/test_stability_harness.py | 165 ++++++++ 7 files changed, 2193 insertions(+), 2 deletions(-) create mode 100644 policybench/counterfactual.py create mode 100644 policybench/stability.py create mode 100644 tests/test_counterfactual.py create mode 100644 tests/test_stability.py create mode 100644 tests/test_stability_harness.py diff --git a/policybench/cli.py b/policybench/cli.py index 15da8eb..00ccc76 100644 --- a/policybench/cli.py +++ b/policybench/cli.py @@ -947,10 +947,14 @@ def main(): # Enable disk cache for ordinary eval calls. Contract-failure retry/repair # commands bypass cache by default, otherwise they can replay the same bad - # provider response indefinitely. + # provider response indefinitely. eval-no-tools-repeated is deliberately + # NOT in this set: repeat requests are byte-identical across runs, so a + # shared disk cache would replay run 1's responses into runs 2..K and + # every stability metric would measure the cache, not the model + # (docs/stability_spec.md, "Cache discipline"). stability-report enforces + # this by hard-failing on cache_hit records in the runs' spend ledgers. if args.command in { "eval-no-tools", - "eval-no-tools-repeated", "eval-no-tools-chunked", } or ( args.command in {"retry-failed-responses", "repair-failed-rows"} diff --git a/policybench/counterfactual.py b/policybench/counterfactual.py new file mode 100644 index 0000000..f63a918 --- /dev/null +++ b/policybench/counterfactual.py @@ -0,0 +1,571 @@ +"""Counterfactual consistency (stability spec layer 3). + +Builds +$1,000 head-earnings twins of the frozen scenarios, computes +PolicyEngine true deltas for both arms with one engine version, and compares +each model's implied delta to the truth. Metric semantics follow +docs/stability_spec.md, which was rebuilt around the measured truth-delta +distribution (zero binary flips, 247/1,000 nonzero amount rows on the +frozen manifest): the nonzero-true stratum is primary, binary flips report +counts rather than rates, and "large response" replaces cliff language. +""" + +from __future__ import annotations + +import copy +from itertools import combinations + +import numpy as np +import pandas as pd + +from policybench.analysis import BINARY_PROGRAMS, binary_flag +from policybench.ground_truth import calculate_ground_truth +from policybench.scenarios import Scenario, scenario_manifest +from policybench.spec import metric_type_for_output, output_group_id +from policybench.stability import is_parsed_prediction + +CF_ID_SUFFIX = "__cf1k" +CF_DEFAULT_AMOUNT = 1000.0 +CF_PERTURBED_FIELD = "employment_income" + +DELTA_EXACT_TOLERANCE = 1.0 +# Sign dead zone: |delta| <= $1 counts as "no change". +SIGN_DEAD_ZONE = 1.0 +LARGE_RESPONSE_TRUE_CUT = 200.0 +LARGE_RESPONSE_PREDICTED_CUT = 100.0 +LARGE_RESPONSE_MIN_ROWS = 20 + +ZERO_DELTA_BASELINE_MODEL = "__zero_delta_baseline__" + + +def _is_binary_variable(variable: str) -> bool: + return metric_type_for_output(variable) == "binary" or variable in BINARY_PROGRAMS + + +def build_counterfactual_twin( + scenario: Scenario, amount: float = CF_DEFAULT_AMOUNT +) -> Scenario: + """Return the scenario's perturbed twin (head wages += amount). + + Twins are built at the Scenario level so the embedded id, recomputed + total_income, and scenario_json stay mutually consistent; the manifest + loader hard-fails on id mismatches, so the frozen CSV is never edited. + """ + twin = copy.deepcopy(scenario) + head = twin.adults[0] + first_dollar = float(head.employment_income) == 0.0 + head.employment_income = float(head.employment_income) + float(amount) + twin.id = f"{scenario.id}{CF_ID_SUFFIX}" + twin.metadata = dict(twin.metadata) + twin.metadata["counterfactual"] = { + "base_scenario_id": scenario.id, + "perturbed_field": CF_PERTURBED_FIELD, + "person": head.name, + "perturbation_amount": float(amount), + "first_dollar": first_dollar, + } + return twin + + +def build_counterfactual_manifest( + scenarios: list[Scenario], amount: float = CF_DEFAULT_AMOUNT +) -> tuple[list[Scenario], pd.DataFrame]: + """Build twins and their manifest (extra columns are loader-ignored).""" + twins = [build_counterfactual_twin(scenario, amount) for scenario in scenarios] + manifest = scenario_manifest(twins) + manifest["base_scenario_id"] = [ + twin.metadata["counterfactual"]["base_scenario_id"] for twin in twins + ] + manifest["perturbed_field"] = CF_PERTURBED_FIELD + manifest["perturbation_amount"] = float(amount) + manifest["first_dollar"] = [ + twin.metadata["counterfactual"]["first_dollar"] for twin in twins + ] + return twins, manifest + + +def compute_truth_deltas( + base_scenarios: list[Scenario], + twin_scenarios: list[Scenario], + programs: list[str], + year: int, +) -> pd.DataFrame: + """PolicyEngine true deltas, both arms computed with the installed engine. + + Deltas are rounded to cents before any threshold comparison so knife-edge + rows cannot enter or leave a cut on sub-cent engine jitter. + """ + base_truth = calculate_ground_truth(base_scenarios, programs=programs, year=year) + twin_truth = calculate_ground_truth(twin_scenarios, programs=programs, year=year) + first_dollar = { + twin.metadata["counterfactual"]["base_scenario_id"]: twin.metadata[ + "counterfactual" + ]["first_dollar"] + for twin in twin_scenarios + } + twin_truth = twin_truth.copy() + twin_truth["base_scenario_id"] = twin_truth["scenario_id"].str.replace( + CF_ID_SUFFIX, "", regex=False + ) + merged = base_truth.merge( + twin_truth[["base_scenario_id", "variable", "value"]], + left_on=["scenario_id", "variable"], + right_on=["base_scenario_id", "variable"], + suffixes=("_base", "_perturbed"), + ) + out = pd.DataFrame( + { + "scenario_id": merged["scenario_id"], + "variable": merged["variable"], + "base_value": merged["value_base"], + "perturbed_value": merged["value_perturbed"], + "true_delta": ( + (merged["value_perturbed"] - merged["value_base"]).round(2) + ), + "is_binary": merged["variable"].map(_is_binary_variable), + "output_group": merged["variable"].map(output_group_id), + "first_dollar": merged["scenario_id"].map(first_dollar).fillna(False), + } + ) + return out + + +def matched_delta_frame( + base_predictions: pd.DataFrame, + perturbed_predictions: pd.DataFrame, + truth_deltas: pd.DataFrame, +) -> pd.DataFrame: + """One row per (model, base run, scenario, variable) with pred/true deltas. + + When the base arm carries a run_id column (layer-1 repeats), every base + run is matched, giving a pred-delta distribution instead of one arbitrary + pairing. + """ + base = base_predictions.copy() + if "run_id" not in base.columns: + base["run_id"] = "base" + pert = perturbed_predictions.copy() + pert["base_scenario_id"] = pert["scenario_id"].str.replace( + CF_ID_SUFFIX, "", regex=False + ) + pert_lookup = { + (row.model, row.base_scenario_id, row.variable): row.prediction + for row in pert.itertuples() + } + truth_lookup = truth_deltas.set_index(["scenario_id", "variable"]) + + rows = [] + for row in base.itertuples(): + key = (row.scenario_id, row.variable) + if key not in truth_lookup.index: + continue + truth = truth_lookup.loc[key] + pred_base = row.prediction + pred_pert = pert_lookup.get((row.model, row.scenario_id, row.variable)) + parsed_base = is_parsed_prediction(row.variable, pred_base) + parsed_pert = is_parsed_prediction(row.variable, pred_pert) + both = parsed_base and parsed_pert + is_binary = bool(truth["is_binary"]) + if both and not is_binary: + pred_delta = float(pred_pert) - float(pred_base) + elif both and is_binary: + pred_delta = float(binary_flag(pred_pert) - binary_flag(pred_base)) + else: + pred_delta = float("nan") + rows.append( + { + "model": row.model, + "base_run_id": row.run_id, + "scenario_id": row.scenario_id, + "variable": row.variable, + "is_binary": is_binary, + "output_group": truth["output_group"], + "first_dollar": bool(truth["first_dollar"]), + "both_parsed": both, + "pred_delta": pred_delta, + "true_delta": float(truth["true_delta"]), + } + ) + return pd.DataFrame(rows) + + +def _sign(value: float, dead_zone: float = SIGN_DEAD_ZONE) -> int: + if abs(value) <= dead_zone: + return 0 + return 1 if value > 0 else -1 + + +def _delta_within_band(pred_delta: float, true_delta: float) -> bool: + """Band = max($1, 10% |true|): a strict superset of delta-exact. + + Deliberately diverges from the level metrics' zero-floor semantics — + counterfactual deltas concentrate under $77, where a bare relative band + would be stricter than exact. + """ + band = max(DELTA_EXACT_TOLERANCE, 0.10 * abs(true_delta)) + return abs(pred_delta - true_delta) <= band + + +def reportable_groups(truth_deltas: pd.DataFrame, min_rows: int = 10) -> list[str]: + """Amount output groups with >= min_rows nonzero-true rows (pre-registered).""" + amount = truth_deltas[~truth_deltas["is_binary"]] + nonzero = amount[amount["true_delta"].abs() > DELTA_EXACT_TOLERANCE] + counts = nonzero.groupby("output_group").size() + return sorted(counts[counts >= min_rows].index) + + +def binomial_ci_exact(successes: int, trials: int, level: float = 0.95): + """Clopper–Pearson interval via bisection on exact binomial tails.""" + if trials <= 0: + return float("nan"), float("nan") + alpha = 1.0 - level + + def tail_ge(p: float) -> float: + # P(X >= successes | p) + q = 1.0 - p + total = 0.0 + prob = q**trials + # iterate pmf via recurrence + for k in range(0, trials + 1): + if k >= successes: + total += prob + prob = prob * (trials - k) / (k + 1) * (p / q) if q > 0 else 0.0 + return total + + def tail_le(p: float) -> float: + q = 1.0 - p + total = 0.0 + prob = q**trials + for k in range(0, trials + 1): + if k <= successes: + total += prob + prob = prob * (trials - k) / (k + 1) * (p / q) if q > 0 else 0.0 + return total + + if successes == 0: + low = 0.0 + else: + lo, hi = 0.0, 1.0 + for _ in range(100): + mid = 0.5 * (lo + hi) + if tail_ge(mid) < alpha / 2: + lo = mid + else: + hi = mid + low = 0.5 * (lo + hi) + if successes == trials: + high = 1.0 + else: + lo, hi = 0.0, 1.0 + for _ in range(100): + mid = 0.5 * (lo + hi) + if tail_le(mid) > alpha / 2: + lo = mid + else: + hi = mid + high = 0.5 * (lo + hi) + return low, high + + +def _with_zero_baseline(matched: pd.DataFrame) -> pd.DataFrame: + """Append the zero-delta baseline as a pseudo-model on the same rows.""" + one_run = matched.drop_duplicates(subset=["scenario_id", "variable"]).copy() + baseline = one_run.assign( + model=ZERO_DELTA_BASELINE_MODEL, + base_run_id="baseline", + both_parsed=True, + pred_delta=0.0, + ) + return pd.concat([matched, baseline], ignore_index=True) + + +def delta_metrics_by_model( + matched: pd.DataFrame, + large_response_min_rows: int = LARGE_RESPONSE_MIN_ROWS, + group_min_rows: int = 10, +) -> dict[str, pd.DataFrame]: + """Delta metrics per docs/stability_spec.md layer 3. + + Returns ``summary`` (per model), ``per_group`` (pre-registered groups + only), ``binary_counts`` (counts, never rates: the +$1k reference + produces zero flips), ``large_response`` (with suppression flag and + exact binomial CI), and ``large_response_rows`` (the auditable + qualifying-row list). + """ + if matched.empty: + return { + "summary": pd.DataFrame(), + "per_group": pd.DataFrame(), + "binary_counts": pd.DataFrame(), + "large_response": pd.DataFrame(), + "large_response_rows": pd.DataFrame(), + } + matched = _with_zero_baseline(matched) + amount = matched[~matched["is_binary"] & matched["both_parsed"]].copy() + amount["nonzero_true"] = amount["true_delta"].abs() > DELTA_EXACT_TOLERANCE + amount["exact"] = ( + (amount["pred_delta"] - amount["true_delta"]).abs() <= DELTA_EXACT_TOLERANCE + ) + amount["within_band"] = [ + _delta_within_band(p, t) + for p, t in zip(amount["pred_delta"], amount["true_delta"], strict=True) + ] + amount["true_sign"] = amount["true_delta"].map(_sign) + amount["pred_sign"] = amount["pred_delta"].map(_sign) + amount["sign_match"] = amount["true_sign"] == amount["pred_sign"] + amount["abs_error"] = (amount["pred_delta"] - amount["true_delta"]).abs() + + summary_rows = [] + per_group_rows = [] + binary_rows = [] + large_rows = [] + + truth_rows = matched.drop_duplicates(subset=["scenario_id", "variable"]) + qualifying = truth_rows[ + ~truth_rows["is_binary"] + & (truth_rows["true_delta"].abs() >= LARGE_RESPONSE_TRUE_CUT) + ][["scenario_id", "variable", "output_group", "true_delta"]].reset_index(drop=True) + + groups = reportable_groups( + truth_rows.rename(columns={"output_group": "output_group"}), + min_rows=group_min_rows, + ) + + for model, group in matched.groupby("model"): + model_amount = amount[amount["model"] == model] + nonzero = model_amount[model_amount["nonzero_true"]] + pos = nonzero[nonzero["true_sign"] > 0] + neg = nonzero[nonzero["true_sign"] < 0] + summary_rows.append( + { + "model": model, + "n_matched_rows": int(len(group)), + "delta_coverage": float(group["both_parsed"].mean()), + "n_nonzero_true": int(len(nonzero)), + "delta_exact_rate_nonzero": ( + float(nonzero["exact"].mean()) if len(nonzero) else float("nan") + ), + "delta_within_10pct_rate_nonzero": ( + float(nonzero["within_band"].mean()) + if len(nonzero) + else float("nan") + ), + "delta_mae_nonzero": ( + float(nonzero["abs_error"].mean()) if len(nonzero) else float("nan") + ), + "sign_recall_positive": ( + float(pos["sign_match"].mean()) if len(pos) else float("nan") + ), + "sign_recall_negative": ( + float(neg["sign_match"].mean()) if len(neg) else float("nan") + ), + "delta_exact_rate_amount_pooled": ( + float(model_amount["exact"].mean()) + if len(model_amount) + else float("nan") + ), + "delta_mae_amount_pooled": ( + float(model_amount["abs_error"].mean()) + if len(model_amount) + else float("nan") + ), + } + ) + + for output_group in groups: + in_group = nonzero[nonzero["output_group"] == output_group] + if in_group.empty: + continue + per_group_rows.append( + { + "model": model, + "output_group": output_group, + "n_nonzero_true": int(len(in_group)), + "delta_exact_rate": float(in_group["exact"].mean()), + "delta_within_10pct_rate": float(in_group["within_band"].mean()), + "delta_mae": float(in_group["abs_error"].mean()), + "sign_agreement": float(in_group["sign_match"].mean()), + } + ) + + model_binary = group[group["is_binary"] & group["both_parsed"]] + reference_flips = int( + ( + model_binary.drop_duplicates(subset=["scenario_id", "variable"])[ + "true_delta" + ].abs() + > 0 + ).sum() + ) + model_flips = int((model_binary["pred_delta"].abs() > 0).sum()) + binary_rows.append( + { + "model": model, + "n_binary_rows": int(len(model_binary)), + "n_reference_flips": reference_flips, + "n_model_flips": model_flips, + } + ) + + model_large = model_amount.merge( + qualifying[["scenario_id", "variable"]], + on=["scenario_id", "variable"], + ) + detected = model_large[ + (model_large["pred_sign"] == model_large["true_sign"]) + & (model_large["pred_delta"].abs() >= LARGE_RESPONSE_PREDICTED_CUT) + ] + n_qualifying = int(len(model_large)) + ci_low, ci_high = binomial_ci_exact(len(detected), n_qualifying) + large_rows.append( + { + "model": model, + "true_cut_usd": LARGE_RESPONSE_TRUE_CUT, + "predicted_cut_usd": LARGE_RESPONSE_PREDICTED_CUT, + "n_qualifying_rows": n_qualifying, + "n_detected": int(len(detected)), + "detection_rate": ( + len(detected) / n_qualifying if n_qualifying else float("nan") + ), + "detection_rate_ci_low": ci_low, + "detection_rate_ci_high": ci_high, + "suppressed": n_qualifying < large_response_min_rows, + } + ) + + return { + "summary": pd.DataFrame(summary_rows), + "per_group": pd.DataFrame(per_group_rows), + "binary_counts": pd.DataFrame(binary_rows), + "large_response": pd.DataFrame(large_rows), + "large_response_rows": qualifying, + } + + +def noise_floor_frame( + repeated_base_predictions: pd.DataFrame, + truth_deltas: pd.DataFrame, +) -> pd.DataFrame: + """Base-vs-base null deltas from layer-1 repeats (true delta ≡ 0). + + Produces the same shape as matched_delta_frame so every signal statistic + can be recomputed on its exact row subset with a zero true delta — the + row-matched noise floor the spec requires. + """ + preds = repeated_base_predictions + if "run_id" not in preds.columns: + raise ValueError("noise floor needs repeated base runs with run_id") + run_ids = sorted(preds["run_id"].dropna().unique()) + if len(run_ids) < 2: + raise ValueError("noise floor needs at least two base runs") + lookup = { + (row.model, row.run_id, row.scenario_id, row.variable): row.prediction + for row in preds.itertuples() + } + truth_lookup = truth_deltas.set_index(["scenario_id", "variable"]) + models = sorted(preds["model"].dropna().unique()) + rows = [] + for (scenario_id, variable), truth in truth_lookup.iterrows(): + is_binary = bool(truth["is_binary"]) + for model in models: + for run_a, run_b in combinations(run_ids, 2): + pred_a = lookup.get((model, run_a, scenario_id, variable)) + pred_b = lookup.get((model, run_b, scenario_id, variable)) + parsed = is_parsed_prediction( + variable, pred_a + ) and is_parsed_prediction(variable, pred_b) + if parsed and not is_binary: + delta = float(pred_b) - float(pred_a) + elif parsed and is_binary: + delta = float(binary_flag(pred_b) - binary_flag(pred_a)) + else: + delta = float("nan") + rows.append( + { + "model": model, + "base_run_id": f"{run_a}|{run_b}", + "scenario_id": scenario_id, + "variable": variable, + "is_binary": is_binary, + "output_group": truth["output_group"], + "first_dollar": bool(truth["first_dollar"]), + "both_parsed": parsed, + "pred_delta": delta, + "true_delta": 0.0, + } + ) + return pd.DataFrame(rows) + + +def signal_vs_noise_test( + matched: pd.DataFrame, + floor: pd.DataFrame, + n_boot: int = 1000, + seed: int = 20260818, +) -> pd.DataFrame: + """Scenario-cluster bootstrap of (signal − floor) median |delta|. + + Signal: median |pred_delta| on nonzero-true amount rows of the matched + frame. Floor: median |pred_delta| on the same rows of the base-vs-base + frame. "Indistinguishable from repeat noise" iff the 95% interval of + the difference covers 0. + """ + rows = [] + rng = np.random.default_rng(seed) + key_cols = ["scenario_id", "variable"] + signal_rows = matched[ + ~matched["is_binary"] + & matched["both_parsed"] + & (matched["true_delta"].abs() > DELTA_EXACT_TOLERANCE) + & (matched["model"] != ZERO_DELTA_BASELINE_MODEL) + ] + for model, signal in signal_rows.groupby("model"): + keys = signal[key_cols].drop_duplicates() + floor_rows = floor.merge(keys, on=key_cols) + floor_rows = floor_rows[ + (floor_rows["model"] == model) & floor_rows["both_parsed"] + ] + if signal.empty or floor_rows.empty: + continue + scenarios = sorted(signal["scenario_id"].unique()) + signal_by_scenario = { + sid: group["pred_delta"].abs().to_numpy() + for sid, group in signal.groupby("scenario_id") + } + floor_by_scenario = { + sid: group["pred_delta"].abs().to_numpy() + for sid, group in floor_rows.groupby("scenario_id") + } + point_signal = float(np.median(signal["pred_delta"].abs())) + point_floor = float(np.median(floor_rows["pred_delta"].abs())) + diffs = [] + n = len(scenarios) + for _ in range(max(0, n_boot)): + draw = rng.integers(0, n, size=n) + sig_values = np.concatenate( + [signal_by_scenario[scenarios[i]] for i in draw] + ) + floor_values = np.concatenate( + [ + floor_by_scenario.get(scenarios[i], np.array([])) + for i in draw + ] + ) + if not len(floor_values): + continue + diffs.append(float(np.median(sig_values) - np.median(floor_values))) + if diffs: + ci_low, ci_high = np.percentile(diffs, [2.5, 97.5]) + else: + ci_low = ci_high = float("nan") + rows.append( + { + "model": model, + "signal_median_abs_delta": point_signal, + "noise_floor_median_abs_delta": point_floor, + "difference": point_signal - point_floor, + "difference_ci_low": float(ci_low), + "difference_ci_high": float(ci_high), + "distinguishable_from_noise": bool(ci_low > 0 or ci_high < 0), + } + ) + return pd.DataFrame(rows) diff --git a/policybench/eval_no_tools.py b/policybench/eval_no_tools.py index f665288..b158ca6 100644 --- a/policybench/eval_no_tools.py +++ b/policybench/eval_no_tools.py @@ -2781,6 +2781,14 @@ def _response_contract_metadata() -> dict: "explanation_value_contract": "terminal explanation value equals numeric value", "prompt_template_sha256": _package_file_sha256("prompts.py"), "benchmark_spec_sha256": _package_file_sha256("benchmark_specs.json"), + # Serving-condition escape hatches change what the models are asked to + # do, so they are part of the resume/pooling fingerprint: a run under + # tool_choice auto (or with chunking overridden) must never resume or + # pool with a canonical-condition run. + "tool_choice_condition": ( + "auto" if os.environ.get("POLICYBENCH_TOOL_CHOICE") == "auto" else "forced" + ), + "chunk_override": os.environ.get("POLICYBENCH_CHUNK_OVERRIDE") or None, } @@ -3438,6 +3446,66 @@ def run_no_tools_single_output_eval( return df +def _write_runs_metadata( + output_path: Path, + *, + scenarios: list[Scenario], + repeats: int, + models: dict[str, str], + programs: list[str], + include_explanations: bool, + single_output: bool, +) -> None: + """Record the repeat set's shared fingerprint for stability pooling. + + stability-report refuses to pool runs directories whose fingerprints + mismatch; the serving condition rides in the response contract. + """ + import litellm + + metadata = _build_resume_metadata( + task="eval_no_tools_repeated", + scenarios=scenarios, + models=models, + programs=programs, + run_id=None, + include_explanations=include_explanations, + ) + metadata.update( + { + "repeats": repeats, + "single_output": single_output, + "cache_enabled": litellm.cache is not None, + } + ) + metadata_path = output_path / "runs_metadata.json" + if metadata_path.exists(): + existing = json.loads(metadata_path.read_text(encoding="utf-8")) + mismatched = [ + key + for key in ( + "scenario_hash", + "programs", + "models", + "response_contract", + "include_explanations", + "single_output", + ) + if existing.get(key) != metadata.get(key) + ] + if mismatched: + raise ValueError( + f"Existing repeat set at {output_path} was produced with " + f"different settings ({', '.join(mismatched)}). Use one runs " + "directory per model group; do not mix groups or conditions " + "in one directory." + ) + metadata_path.write_text( + json.dumps(metadata, indent=2, sort_keys=True), + encoding="utf-8", + ) + + def run_repeated_no_tools_eval( scenarios: list[Scenario], repeats: int, @@ -3450,6 +3518,15 @@ def run_repeated_no_tools_eval( """Run repeated AI-alone evaluations, saving one artifact per run.""" output_path = Path(output_dir) output_path.mkdir(parents=True, exist_ok=True) + _write_runs_metadata( + output_path, + scenarios=scenarios, + repeats=repeats, + models=models if models is not None else MODELS, + programs=programs if programs is not None else PROGRAMS, + include_explanations=include_explanations, + single_output=single_output, + ) frames = [] for run_index in range(repeats): diff --git a/policybench/stability.py b/policybench/stability.py new file mode 100644 index 0000000..fe6eeef --- /dev/null +++ b/policybench/stability.py @@ -0,0 +1,687 @@ +"""Repeated-run answer-stability metrics (stability spec layer 1). + +Implements docs/stability_spec.md: row-level flip metrics with +scenario-cluster bootstrap CIs, the run-vs-sampling variance decomposition +with its chi-square decision rule, and the cache-contamination guard that +makes repeat runs meaningful (see "Cache discipline" in the spec). +""" + +from __future__ import annotations + +import json +import math +from itertools import combinations +from pathlib import Path + +import numpy as np +import pandas as pd + +from policybench.analysis import ( + BINARY_PROGRAMS, + binary_flag, + household_headline_scores, + row_hit_scores, +) +from policybench.spec import metric_type_for_output + +STABILITY_SPEC_VERSION = "2026-08-18-v1" + +# Headline mutual-exact tolerance for amount rows, matching the +# prediction-vs-reference exact tolerance in analysis.row_hit_scores. +AMOUNT_EXACT_TOLERANCE = 1.0 + +DEFAULT_BOOTSTRAP_DRAWS = 1000 +DEFAULT_BOOTSTRAP_SEED = 20260818 + +_BOOTSTRAPPED_METRICS = ( + "unanimous_parse_rate", + "coverage_flip_rate", + "invalid_binary_rate", + "answer_flip_rate", + "unanimous_exact_answer_rate", + "verdict_flip_rate_parsed", + "verdict_flip_rate_all", + "consistently_wrong_divergent_rate", +) + + +class CacheContaminationError(RuntimeError): + """Raised when repeat runs contain cache-served responses.""" + + +def _is_binary_variable(variable: str) -> bool: + return metric_type_for_output(variable) == "binary" or variable in BINARY_PROGRAMS + + +def is_parsed_prediction(variable: str, prediction) -> bool: + """Per-type parse: amounts need a numeric; binary rows need a valid flag.""" + if prediction is None: + return False + try: + if pd.isna(prediction): + return False + except (TypeError, ValueError): + return False + if _is_binary_variable(variable): + return binary_flag(prediction) is not None + try: + float(prediction) + except (TypeError, ValueError): + return False + return True + + +def mutually_exact(variable: str, a, b) -> bool: + """Headline exact tolerance applied between two predictions.""" + if not is_parsed_prediction(variable, a) or not is_parsed_prediction(variable, b): + return False + if _is_binary_variable(variable): + return binary_flag(a) == binary_flag(b) + return abs(float(a) - float(b)) <= AMOUNT_EXACT_TOLERANCE + + +def _regularized_lower_gamma(a: float, x: float) -> float: + """Regularized lower incomplete gamma P(a, x) (series / continued fraction).""" + if x < 0 or a <= 0: + raise ValueError("Require x >= 0 and a > 0") + if x == 0: + return 0.0 + log_gamma_a = math.lgamma(a) + if x < a + 1.0: + # Series representation. + term = 1.0 / a + total = term + n = a + for _ in range(500): + n += 1.0 + term *= x / n + total += term + if abs(term) < abs(total) * 1e-14: + break + return total * math.exp(-x + a * math.log(x) - log_gamma_a) + # Continued fraction for Q(a, x) = 1 - P(a, x) (Lentz's method). + tiny = 1e-300 + b = x + 1.0 - a + c = 1.0 / tiny + d = 1.0 / b + h = d + for i in range(1, 500): + an = -i * (i - a) + b += 2.0 + d = an * d + b + if abs(d) < tiny: + d = tiny + c = b + an / c + if abs(c) < tiny: + c = tiny + d = 1.0 / d + delta = d * c + h *= delta + if abs(delta - 1.0) < 1e-14: + break + q = math.exp(-x + a * math.log(x) - log_gamma_a) * h + return 1.0 - q + + +def chi2_ppf(p: float, df: int) -> float: + """Chi-square quantile via bisection on the regularized gamma CDF. + + Self-contained because scipy is not a project dependency; accurate to + ~1e-8 on the small df used for run-count confidence intervals. + """ + if not 0.0 < p < 1.0: + raise ValueError("p must be in (0, 1)") + if df < 1: + raise ValueError("df must be >= 1") + a = df / 2.0 + + def cdf(x: float) -> float: + return _regularized_lower_gamma(a, x / 2.0) + + low, high = 0.0, float(df) + while cdf(high) < p: + high *= 2.0 + if high > 1e8: + raise RuntimeError("chi2_ppf failed to bracket the quantile") + for _ in range(200): + mid = 0.5 * (low + high) + if cdf(mid) < p: + low = mid + else: + high = mid + if high - low < 1e-10 * max(1.0, high): + break + return 0.5 * (low + high) + + +def run_std_ci(std: float, k: int, level: float = 0.90) -> tuple[float, float]: + """Chi-square CI for a std estimated from ``k`` runs (df = k - 1). + + The upper bound at two-sided ``level`` 0.90 equals the one-sided 95% + upper bound the spec's decision rule uses. + """ + if k < 2 or std is None or (isinstance(std, float) and math.isnan(std)): + return float("nan"), float("nan") + df = k - 1 + alpha = 1.0 - level + low = std * math.sqrt(df / chi2_ppf(1.0 - alpha / 2.0, df)) + high = std * math.sqrt(df / chi2_ppf(alpha / 2.0, df)) + return low, high + + +def _prediction_lookup(repeated_predictions: pd.DataFrame) -> pd.DataFrame: + required = {"run_id", "model", "scenario_id", "variable", "prediction"} + missing = required - set(repeated_predictions.columns) + if missing: + raise ValueError(f"repeated predictions missing columns: {sorted(missing)}") + return repeated_predictions + + +def run_pair_frame( + repeated_predictions: pd.DataFrame, + ground_truth: pd.DataFrame, +) -> pd.DataFrame: + """One row per (model, scenario, variable, run pair) with stability flags. + + Every expected row (from the reference frame) appears for every model and + run pair; predictions a run never produced count as unparsed, mirroring + the headline's missing-scores-zero semantics. + """ + if repeated_predictions.empty or ground_truth.empty: + return pd.DataFrame( + columns=[ + "model", + "scenario_id", + "variable", + "run_a", + "run_b", + "is_binary", + "parsed_a", + "parsed_b", + "both_parsed", + "mutually_exact", + "exact_a", + "exact_b", + ] + ) + preds = _prediction_lookup(repeated_predictions) + run_ids = sorted(preds["run_id"].dropna().unique()) + models = sorted(preds["model"].dropna().unique()) + by_key: dict[tuple, object] = { + (row.model, row.run_id, row.scenario_id, row.variable): row.prediction + for row in preds.itertuples() + } + exact_cache: dict[tuple, float] = {} + + def exact_correct(model, run_id, scenario_id, variable, value) -> bool: + key = (model, run_id, scenario_id, variable) + if key not in exact_cache: + prediction = by_key.get(key) + exact_cache[key] = row_hit_scores(variable, value, prediction)["exact"] + return bool(exact_cache[key]) + + rows = [] + for gt_row in ground_truth.itertuples(): + variable = gt_row.variable + is_binary = _is_binary_variable(variable) + for model in models: + for run_a, run_b in combinations(run_ids, 2): + pred_a = by_key.get((model, run_a, gt_row.scenario_id, variable)) + pred_b = by_key.get((model, run_b, gt_row.scenario_id, variable)) + parsed_a = is_parsed_prediction(variable, pred_a) + parsed_b = is_parsed_prediction(variable, pred_b) + rows.append( + { + "model": model, + "scenario_id": gt_row.scenario_id, + "variable": variable, + "run_a": run_a, + "run_b": run_b, + "is_binary": is_binary, + "parsed_a": parsed_a, + "parsed_b": parsed_b, + "both_parsed": parsed_a and parsed_b, + "mutually_exact": mutually_exact(variable, pred_a, pred_b), + "exact_a": exact_correct( + model, run_a, gt_row.scenario_id, variable, gt_row.value + ), + "exact_b": exact_correct( + model, run_b, gt_row.scenario_id, variable, gt_row.value + ), + } + ) + return pd.DataFrame(rows) + + +def _row_level_frame( + repeated_predictions: pd.DataFrame, + ground_truth: pd.DataFrame, +) -> pd.DataFrame: + """One row per (model, scenario, variable) with K-way flags.""" + preds = _prediction_lookup(repeated_predictions) + run_ids = sorted(preds["run_id"].dropna().unique()) + models = sorted(preds["model"].dropna().unique()) + by_key = { + (row.model, row.run_id, row.scenario_id, row.variable): row.prediction + for row in preds.itertuples() + } + rows = [] + for gt_row in ground_truth.itertuples(): + variable = gt_row.variable + is_binary = _is_binary_variable(variable) + for model in models: + predictions = [ + by_key.get((model, run_id, gt_row.scenario_id, variable)) + for run_id in run_ids + ] + parsed = [is_parsed_prediction(variable, p) for p in predictions] + numeric_non_flag = False + if is_binary: + for p in predictions: + if p is None: + continue + try: + if pd.isna(p): + continue + float(p) + except (TypeError, ValueError): + continue + if binary_flag(p) is None: + numeric_non_flag = True + all_pairwise_exact = all( + mutually_exact(variable, a, b) + for a, b in combinations(predictions, 2) + ) + exact_flags = [ + bool(row_hit_scores(variable, gt_row.value, p)["exact"]) + for p in predictions + ] + rows.append( + { + "model": model, + "scenario_id": gt_row.scenario_id, + "variable": variable, + "is_binary": is_binary, + "all_parsed": all(parsed), + "any_parsed": any(parsed), + "invalid_binary": numeric_non_flag, + "unanimous_exact_answer": all(parsed) and all_pairwise_exact, + "all_wrong": all(parsed) and not any(exact_flags), + "wrong_divergent": ( + all(parsed) and not any(exact_flags) and not all_pairwise_exact + ), + } + ) + return pd.DataFrame(rows) + + +def _scenario_aggregates( + pair_frame: pd.DataFrame, + row_frame: pd.DataFrame, +) -> pd.DataFrame: + """Numerator/denominator sums per (model, scenario) for every metric.""" + pair = pair_frame.assign( + flip_num=lambda df: df["both_parsed"] & ~df["mutually_exact"], + flip_den=lambda df: df["both_parsed"], + verdict_parsed_num=lambda df: df["both_parsed"] + & (df["exact_a"] != df["exact_b"]), + verdict_parsed_den=lambda df: df["both_parsed"], + verdict_all_num=lambda df: df["exact_a"] != df["exact_b"], + verdict_all_den=1, + ) + pair_agg = ( + pair.groupby(["model", "scenario_id"])[ + [ + "flip_num", + "flip_den", + "verdict_parsed_num", + "verdict_parsed_den", + "verdict_all_num", + "verdict_all_den", + ] + ] + .sum() + .reset_index() + ) + row = row_frame.assign( + unanimous_parse_num=lambda df: df["all_parsed"], + rows_den=1, + coverage_flip_num=lambda df: df["any_parsed"] & ~df["all_parsed"], + invalid_binary_num=lambda df: df["is_binary"] & df["invalid_binary"], + binary_den=lambda df: df["is_binary"], + unanimous_exact_num=lambda df: df["unanimous_exact_answer"], + wrong_divergent_num=lambda df: ~df["is_binary"] & df["wrong_divergent"], + wrong_den=lambda df: ~df["is_binary"] & df["all_wrong"], + ) + row_agg = ( + row.groupby(["model", "scenario_id"])[ + [ + "unanimous_parse_num", + "rows_den", + "coverage_flip_num", + "invalid_binary_num", + "binary_den", + "unanimous_exact_num", + "wrong_divergent_num", + "wrong_den", + ] + ] + .sum() + .reset_index() + ) + return pair_agg.merge(row_agg, on=["model", "scenario_id"], how="outer").fillna(0) + + +_METRIC_FRACTIONS = { + "unanimous_parse_rate": ("unanimous_parse_num", "rows_den"), + "coverage_flip_rate": ("coverage_flip_num", "rows_den"), + "invalid_binary_rate": ("invalid_binary_num", "binary_den"), + "answer_flip_rate": ("flip_num", "flip_den"), + "unanimous_exact_answer_rate": ("unanimous_exact_num", "rows_den"), + "verdict_flip_rate_parsed": ("verdict_parsed_num", "verdict_parsed_den"), + "verdict_flip_rate_all": ("verdict_all_num", "verdict_all_den"), + "consistently_wrong_divergent_rate": ("wrong_divergent_num", "wrong_den"), +} + + +def _rates_from_aggregates(aggregates: pd.DataFrame) -> pd.DataFrame: + sums = aggregates.groupby("model").sum(numeric_only=True) + out = pd.DataFrame(index=sums.index) + for metric, (num, den) in _METRIC_FRACTIONS.items(): + denominator = sums[den] + out[metric] = np.where(denominator > 0, sums[num] / denominator, np.nan) + return out + + +def row_stability_by_model( + repeated_predictions: pd.DataFrame, + ground_truth: pd.DataFrame, + n_boot: int = DEFAULT_BOOTSTRAP_DRAWS, + seed: int = DEFAULT_BOOTSTRAP_SEED, +) -> pd.DataFrame: + """Row-level stability metrics per model with scenario-cluster bootstrap CIs. + + All rows of a scenario come from a handful of completions per run, so the + scenario (household), not the row or pair, is the independent unit; CIs + resample scenarios with replacement, paired across models. + """ + if ( + repeated_predictions is None + or repeated_predictions.empty + or ground_truth.empty + or "run_id" not in getattr(repeated_predictions, "columns", []) + ): + return pd.DataFrame() + pair_frame = run_pair_frame(repeated_predictions, ground_truth) + row_frame = _row_level_frame(repeated_predictions, ground_truth) + if pair_frame.empty or row_frame.empty: + return pd.DataFrame() + aggregates = _scenario_aggregates(pair_frame, row_frame) + point = _rates_from_aggregates(aggregates) + + scenario_ids = sorted(aggregates["scenario_id"].unique()) + rng = np.random.default_rng(seed) + wide = { + column: aggregates.pivot_table( + index="scenario_id", columns="model", values=column, aggfunc="sum" + ) + .reindex(scenario_ids) + .fillna(0.0) + for column in set( + column for pair in _METRIC_FRACTIONS.values() for column in pair + ) + } + models = point.index.tolist() + boot_values: dict[str, list[np.ndarray]] = {m: [] for m in _METRIC_FRACTIONS} + n_scenarios = len(scenario_ids) + for _ in range(max(0, n_boot)): + draw = rng.integers(0, n_scenarios, size=n_scenarios) + for metric, (num, den) in _METRIC_FRACTIONS.items(): + num_sum = wide[num].values[draw].sum(axis=0) + den_sum = wide[den].values[draw].sum(axis=0) + with np.errstate(invalid="ignore", divide="ignore"): + boot_values[metric].append(np.where(den_sum > 0, num_sum / den_sum, np.nan)) + + run_ids = sorted(repeated_predictions["run_id"].dropna().unique()) + result_rows = [] + for model_index, model in enumerate(models): + model_pairs = pair_frame[pair_frame["model"] == model] + row = { + "model": model, + "k": len(run_ids), + "n_rows": int((row_frame["model"] == model).sum()), + "n_scenarios": n_scenarios, + "n_run_pairs_parsed": int(model_pairs["both_parsed"].sum()), + "n_consistently_wrong_amount_rows": int( + ( + (row_frame["model"] == model) + & ~row_frame["is_binary"] + & row_frame["all_wrong"] + ).sum() + ), + } + for metric in _METRIC_FRACTIONS: + row[metric] = float(point.loc[model, metric]) + draws = np.array([b[model_index] for b in boot_values[metric]]) + draws = draws[~np.isnan(draws)] + if len(draws): + row[f"{metric}_ci_low"] = float(np.percentile(draws, 2.5)) + row[f"{metric}_ci_high"] = float(np.percentile(draws, 97.5)) + else: + row[f"{metric}_ci_low"] = float("nan") + row[f"{metric}_ci_high"] = float("nan") + result_rows.append(row) + return pd.DataFrame(result_rows) + + +def stability_variance_decomposition( + ground_truth: pd.DataFrame, + repeated_predictions: pd.DataFrame, + market_income_by_scenario: dict[str, float] | pd.Series, + country: str | None = None, +) -> tuple[pd.DataFrame, dict]: + """Across-run std of the headline score vs household-sampling SE. + + Returns ``(per_model, pooled)``. The decision rule is CI-based: the + manuscript's sampling-dominance claim is confirmed for a model iff the + one-sided 95% upper bound on its run std sits below its sampling SE. + ``pooled`` aggregates squared run deviations across models (M·(K−1) df) + for a roster-level ratio with usable power at small K, plus each model's + share of the pooled sum of squares as a homogeneity check. + """ + empty = pd.DataFrame(), {} + if ( + repeated_predictions is None + or repeated_predictions.empty + or "run_id" not in getattr(repeated_predictions, "columns", []) + or ground_truth.empty + ): + return empty + + run_ids = sorted(repeated_predictions["run_id"].dropna().unique()) + per_run_scores: list[pd.DataFrame] = [] + for run_id in run_ids: + run_preds = repeated_predictions[repeated_predictions["run_id"] == run_id] + household = household_headline_scores( + ground_truth, + run_preds, + market_income_by_scenario, + country=country, + metric="exact", + ) + if household.empty: + continue + household = household.assign(run_id=run_id) + per_run_scores.append(household) + if not per_run_scores: + return empty + household_scores = pd.concat(per_run_scores, ignore_index=True) + + run_level = ( + household_scores.groupby(["model", "run_id"])["score"] + .agg(["mean", "std", "count"]) + .reset_index() + .rename(columns={"mean": "run_score", "std": "household_std", "count": "n_households"}) + ) + run_level["run_sampling_se"] = run_level["household_std"] / np.sqrt( + run_level["n_households"] + ) + + rows = [] + pooled_ss = 0.0 + pooled_df = 0 + ss_by_model: dict[str, float] = {} + sampling_ses = [] + for model, group in run_level.groupby("model"): + k = len(group) + run_scores = group["run_score"].to_numpy() + std = float(np.std(run_scores, ddof=1)) if k > 1 else float("nan") + ci_low, ci_high = run_std_ci(std, k) + sampling_se = float(group["run_sampling_se"].mean()) + ratio = std / sampling_se if sampling_se > 0 else float("nan") + ratio_ci_high = ci_high / sampling_se if sampling_se > 0 else float("nan") + detectable_floor = ( + math.sqrt((k - 1) / chi2_ppf(0.05, k - 1)) ** -1 if k > 1 else float("nan") + ) + if k > 1: + ss = float(((run_scores - run_scores.mean()) ** 2).sum()) + pooled_ss += ss + pooled_df += k - 1 + ss_by_model[model] = ss + sampling_ses.append(sampling_se) + rows.append( + { + "model": model, + "k": k, + "n_households": int(group["n_households"].iloc[0]), + "run_score_mean": float(np.mean(run_scores)), + "run_score_std": std, + "run_score_std_ci_low": ci_low, + "run_score_std_ci_high": ci_high, + "sampling_se": sampling_se, + "run_to_sampling_ratio": ratio, + "ratio_ci_high": ratio_ci_high, + "sampling_dominates": bool(ci_high < sampling_se) + if not math.isnan(ci_high) + else False, + "detectable_ratio_floor": detectable_floor, + } + ) + per_model = pd.DataFrame(rows) + + pooled: dict = {} + if pooled_df > 0: + pooled_std = math.sqrt(pooled_ss / pooled_df) + mean_sampling_se = float(np.mean(sampling_ses)) if sampling_ses else float("nan") + pooled_ci_low = pooled_std * math.sqrt(pooled_df / chi2_ppf(0.95, pooled_df)) + pooled_ci_high = pooled_std * math.sqrt(pooled_df / chi2_ppf(0.05, pooled_df)) + total_ss = pooled_ss if pooled_ss > 0 else float("nan") + pooled = { + "n_models": len(ss_by_model), + "pooled_df": pooled_df, + "pooled_run_score_std": pooled_std, + "pooled_run_score_std_ci_low": pooled_ci_low, + "pooled_run_score_std_ci_high": pooled_ci_high, + "mean_sampling_se": mean_sampling_se, + "pooled_run_to_sampling_ratio": ( + pooled_std / mean_sampling_se if mean_sampling_se else float("nan") + ), + "pooled_sampling_dominates": bool(pooled_ci_high < mean_sampling_se), + "homogeneity": { + model: (ss / total_ss if total_ss else float("nan")) + for model, ss in sorted(ss_by_model.items()) + }, + } + return per_model, pooled + + +def scan_runs_for_cache_hits(runs_dirs: list[str | Path]) -> dict: + """Scan run directories' spend ledgers for cache-served responses.""" + ledgers = 0 + records = 0 + cache_hits = 0 + runs_without_ledger = 0 + for runs_dir in runs_dirs: + runs_path = Path(runs_dir) + for run_csv in sorted(runs_path.glob("run_*.csv")): + ledger_path = run_csv.with_name(run_csv.name + ".spend.jsonl") + if not ledger_path.exists(): + runs_without_ledger += 1 + continue + ledgers += 1 + for line in ledger_path.read_text(encoding="utf-8").splitlines(): + line = line.strip() + if not line: + continue + try: + record = json.loads(line) + except json.JSONDecodeError: + continue + records += 1 + if record.get("cache_hit"): + cache_hits += 1 + return { + "ledgers": ledgers, + "records": records, + "cache_hits": cache_hits, + "runs_without_ledger": runs_without_ledger, + } + + +def assert_cache_free(runs_dirs: list[str | Path]) -> dict: + """Hard-fail when any repeat run served a response from the disk cache.""" + report = scan_runs_for_cache_hits(runs_dirs) + if report["cache_hits"]: + raise CacheContaminationError( + f"{report['cache_hits']} cache-served responses found across " + f"{report['ledgers']} spend ledgers. Repeat-run stability metrics " + "would measure the cache, not the model; rerun the repeats " + "cache-free (eval-no-tools-repeated no longer enables the disk " + "cache) before reporting." + ) + return report + + +def cross_run_response_id_duplicates(repeated_predictions: pd.DataFrame) -> int | None: + """Count response ids reused across different runs (residual replay check).""" + if "provider_response_id" not in repeated_predictions.columns: + return None + frame = repeated_predictions.dropna(subset=["provider_response_id"]) + if frame.empty: + return 0 + per_id = frame.groupby(["model", "provider_response_id"])["run_id"].nunique() + return int((per_id > 1).sum()) + + +def load_runs_dirs(runs_dirs: list[str | Path]) -> pd.DataFrame: + """Load and concatenate per-provider-group runs directories. + + Model sets must be disjoint across directories (each model's repeats live + in exactly one directory) and every directory must contain the same + run_id set, so K is pinned across the pooled repeat set. + """ + from policybench.eval_no_tools import load_repeated_predictions + + frames = [] + seen_models: dict[str, Path] = {} + run_id_sets: dict[Path, frozenset] = {} + for runs_dir in runs_dirs: + runs_path = Path(runs_dir) + frame = load_repeated_predictions(str(runs_path)) + for model in frame["model"].dropna().unique(): + if model in seen_models: + raise ValueError( + f"model {model!r} appears in both {seen_models[model]} and " + f"{runs_path}; per-provider-group runs directories must be " + "model-disjoint" + ) + seen_models[str(model)] = runs_path + run_id_sets[runs_path] = frozenset(frame["run_id"].dropna().unique()) + frames.append(frame) + distinct = set(run_id_sets.values()) + if len(distinct) > 1: + detail = {str(path): sorted(ids) for path, ids in run_id_sets.items()} + raise ValueError( + f"runs directories disagree on run_id sets (K must be pinned): {detail}" + ) + return pd.concat(frames, ignore_index=True) diff --git a/tests/test_counterfactual.py b/tests/test_counterfactual.py new file mode 100644 index 0000000..0975844 --- /dev/null +++ b/tests/test_counterfactual.py @@ -0,0 +1,339 @@ +"""Tests for counterfactual consistency (stability spec layer 3).""" + +from unittest.mock import patch + +import pandas as pd +import pytest + +from policybench.counterfactual import ( + ZERO_DELTA_BASELINE_MODEL, + CF_ID_SUFFIX, + binomial_ci_exact, + build_counterfactual_manifest, + build_counterfactual_twin, + compute_truth_deltas, + delta_metrics_by_model, + matched_delta_frame, + noise_floor_frame, + reportable_groups, +) +from policybench.scenarios import load_scenarios_from_manifest + + +class TestTwinBuilder: + def test_twin_perturbs_head_and_rewrites_id(self, simple_single_scenario): + twin = build_counterfactual_twin(simple_single_scenario) + assert twin.id == f"{simple_single_scenario.id}{CF_ID_SUFFIX}" + assert twin.adults[0].employment_income == pytest.approx(51_000.0) + # Base is untouched. + assert simple_single_scenario.adults[0].employment_income == pytest.approx( + 50_000.0 + ) + cf = twin.metadata["counterfactual"] + assert cf["base_scenario_id"] == simple_single_scenario.id + assert cf["perturbed_field"] == "employment_income" + assert cf["perturbation_amount"] == 1000.0 + assert cf["first_dollar"] is False + + def test_first_dollar_flag_for_zero_wage_head(self, simple_single_scenario): + simple_single_scenario.adults[0].employment_income = 0.0 + twin = build_counterfactual_twin(simple_single_scenario) + assert twin.metadata["counterfactual"]["first_dollar"] is True + + def test_manifest_round_trips_through_loader(self, tmp_path, sample_scenarios): + twins, manifest = build_counterfactual_manifest(sample_scenarios) + assert list(manifest["base_scenario_id"]) == [s.id for s in sample_scenarios] + assert set(manifest["perturbed_field"]) == {"employment_income"} + assert set(manifest["perturbation_amount"]) == {1000.0} + # total_income reflects the perturbation, not the base value. + base_incomes = [s.total_income for s in sample_scenarios] + assert list(manifest["total_income"]) == pytest.approx( + [income + 1000.0 for income in base_incomes] + ) + path = tmp_path / "cf_scenarios.csv" + manifest.to_csv(path, index=False) + loaded = load_scenarios_from_manifest(path) + assert [s.id for s in loaded] == [t.id for t in twins] + + +class TestTruthDeltas: + def test_merges_arms_and_rounds_to_cents(self, sample_scenarios): + twins, _ = build_counterfactual_manifest(sample_scenarios) + base_ids = [s.id for s in sample_scenarios] + + def fake_ground_truth(scenarios, programs=None, year=None): + rows = [] + for scenario in scenarios: + perturbed = scenario.id.endswith(CF_ID_SUFFIX) + rows.append( + { + "scenario_id": scenario.id, + "variable": "snap", + "value": 100.0 + (10.001 if perturbed else 0.0), + } + ) + rows.append( + { + "scenario_id": scenario.id, + "variable": "person_wic_eligible", + "value": 1.0, + } + ) + return pd.DataFrame(rows) + + with patch( + "policybench.counterfactual.calculate_ground_truth", + side_effect=fake_ground_truth, + ): + deltas = compute_truth_deltas(sample_scenarios, twins, ["snap"], 2026) + + assert set(deltas["scenario_id"]) == set(base_ids) + snap = deltas[deltas["variable"] == "snap"].iloc[0] + assert snap["true_delta"] == pytest.approx(10.0) # rounded to cents + assert not snap["is_binary"] + wic = deltas[deltas["variable"] == "person_wic_eligible"].iloc[0] + assert wic["true_delta"] == pytest.approx(0.0) + assert wic["is_binary"] + assert "first_dollar" in deltas.columns + + +def _truth_deltas(): + rows = [] + # 12 scenarios x payroll_tax with a nonzero delta -> reportable group. + for i in range(12): + rows.append( + { + "scenario_id": f"s{i}", + "variable": "payroll_tax", + "base_value": 1000.0, + "true_delta": 76.5, + "is_binary": False, + "output_group": "payroll_tax", + "first_dollar": False, + } + ) + # snap: 2 nonzero (one negative), below the n>=10 group floor. + rows.append( + { + "scenario_id": "s0", + "variable": "snap", + "base_value": 300.0, + "true_delta": -237.6, + "is_binary": False, + "output_group": "snap", + "first_dollar": False, + } + ) + rows.append( + { + "scenario_id": "s1", + "variable": "snap", + "base_value": 0.0, + "true_delta": 0.0, + "is_binary": False, + "output_group": "snap", + "first_dollar": False, + } + ) + # One large-response row (>= $200) beyond payroll's 76.5s. + rows.append( + { + "scenario_id": "s2", + "variable": "tanf", + "base_value": 750.0, + "true_delta": -750.0, + "is_binary": False, + "output_group": "tanf", + "first_dollar": False, + } + ) + # Binary row with no reference flip. + rows.append( + { + "scenario_id": "s0", + "variable": "person_wic_eligible", + "base_value": 1.0, + "true_delta": 0.0, + "is_binary": True, + "output_group": "person_wic_eligible", + "first_dollar": False, + } + ) + return pd.DataFrame(rows) + + +def _predictions(model="m", run_id=None, perturbed=False): + truth = _truth_deltas() + rows = [] + for row in truth.itertuples(): + base = 900.0 if row.variable == "payroll_tax" else float(row.base_value) + if row.variable == "person_wic_eligible": + prediction = 1.0 + elif perturbed: + # The model moves exactly with truth on payroll, misses tanf/snap. + prediction = base + (76.5 if row.variable == "payroll_tax" else 0.0) + else: + prediction = base + entry = { + "model": model, + "scenario_id": row.scenario_id, + "variable": row.variable, + "prediction": prediction, + } + if run_id is not None: + entry["run_id"] = run_id + rows.append(entry) + return pd.DataFrame(rows) + + +class TestMatchedDeltaFrame: + def test_matches_rows_and_computes_pred_delta(self): + matched = matched_delta_frame( + _predictions(), _predictions(perturbed=True), _truth_deltas() + ) + payroll = matched[matched["variable"] == "payroll_tax"] + assert payroll["pred_delta"].tolist() == pytest.approx([76.5] * len(payroll)) + assert payroll["both_parsed"].all() + tanf = matched[matched["variable"] == "tanf"].iloc[0] + assert tanf["pred_delta"] == pytest.approx(0.0) + + def test_multiple_base_runs_produce_one_match_per_run(self): + base = pd.concat( + [ + _predictions(run_id="run_000"), + _predictions(run_id="run_001"), + ], + ignore_index=True, + ) + matched = matched_delta_frame(base, _predictions(perturbed=True), _truth_deltas()) + assert set(matched["base_run_id"]) == {"run_000", "run_001"} + assert len(matched) == 2 * len(_truth_deltas()) + + +class TestDeltaMetrics: + def test_headline_metrics_and_baseline(self): + matched = matched_delta_frame( + _predictions(), _predictions(perturbed=True), _truth_deltas() + ) + result = delta_metrics_by_model(matched) + summary = result["summary"].set_index("model") + row = summary.loc["m"] + # Nonzero universe: 12 payroll + 1 snap + 1 tanf = 14 rows. + assert row["n_nonzero_true"] == 14 + # Model nails payroll (12), misses snap and tanf -> 12/14. + assert row["delta_exact_rate_nonzero"] == pytest.approx(12 / 14) + # within band = max($1, 10%|true|): identical hits here. + assert row["delta_within_10pct_rate_nonzero"] == pytest.approx(12 / 14) + # Positive direction: 12 payroll all correct; negative: snap+tanf missed. + assert row["sign_recall_positive"] == pytest.approx(1.0) + assert row["sign_recall_negative"] == pytest.approx(0.0) + baseline = summary.loc["__zero_delta_baseline__"] + assert baseline["delta_exact_rate_nonzero"] == pytest.approx(0.0) + # Pooled secondary includes the zero-delta snap row: 1 exact / 15 amount. + assert baseline["delta_exact_rate_amount_pooled"] == pytest.approx(1 / 15) + + def test_small_delta_band_floor_makes_within_superset_of_exact(self): + truth = pd.DataFrame( + [ + { + "scenario_id": "s0", + "variable": "state_refundable_credits", + "base_value": 100.0, + "true_delta": 3.83, + "is_binary": False, + "output_group": "state_refundable_credits", + "first_dollar": False, + } + ] + ) + base = pd.DataFrame( + [{"model": "m", "scenario_id": "s0", "variable": "state_refundable_credits", "prediction": 100.0}] + ) + pert = base.assign(prediction=104.0) # pred_delta = 4.0, error $0.17 + matched = matched_delta_frame(base, pert, truth) + summary = delta_metrics_by_model(matched)["summary"].set_index("model").loc["m"] + assert summary["delta_exact_rate_nonzero"] == pytest.approx(1.0) + # A bare 10% band would be $0.38 -- the $1 floor keeps this a superset. + assert summary["delta_within_10pct_rate_nonzero"] == pytest.approx(1.0) + + def test_binary_counts_not_rates(self): + matched = matched_delta_frame( + _predictions(), _predictions(perturbed=True), _truth_deltas() + ) + result = delta_metrics_by_model(matched) + binary = result["binary_counts"].set_index("model").loc["m"] + assert binary["n_reference_flips"] == 0 + assert binary["n_model_flips"] == 0 + assert "flip_recall" not in result["binary_counts"].columns + + def test_large_response_suppressed_below_min_rows(self): + matched = matched_delta_frame( + _predictions(), _predictions(perturbed=True), _truth_deltas() + ) + result = delta_metrics_by_model(matched, large_response_min_rows=20) + large = result["large_response"] + # tanf -750 and snap -237.6 qualify at >= $200 -> suppressed but counted. + assert large.iloc[0]["n_qualifying_rows"] == 2 + assert bool(large.iloc[0]["suppressed"]) is True + rows = result["large_response_rows"] + assert set(rows["variable"]) == {"tanf", "snap"} + + def test_reportable_groups_from_truth(self): + groups = reportable_groups(_truth_deltas(), min_rows=10) + assert groups == ["payroll_tax"] + per_group = delta_metrics_by_model( + matched_delta_frame( + _predictions(), _predictions(perturbed=True), _truth_deltas() + ) + )["per_group"] + assert set(per_group["output_group"]) == {"payroll_tax"} + + +class TestNoiseFloor: + def test_base_pairs_have_zero_true_delta(self): + base = pd.concat( + [_predictions(run_id="run_000"), _predictions(run_id="run_001")], + ignore_index=True, + ) + floor = noise_floor_frame(base, _truth_deltas()) + assert (floor["true_delta"] == 0).all() + assert set(floor["variable"]) == set(_truth_deltas()["variable"]) + # Identical repeats -> zero null deltas. + assert floor["pred_delta"].abs().max() == pytest.approx(0.0) + + +class TestSignalVsNoise: + def test_real_signal_with_silent_noise_floor_is_distinguishable(self): + from policybench.counterfactual import signal_vs_noise_test + + base = pd.concat( + [_predictions(run_id="run_000"), _predictions(run_id="run_001")], + ignore_index=True, + ) + matched = matched_delta_frame( + _predictions(run_id="run_000"), + _predictions(perturbed=True), + _truth_deltas(), + ) + floor = noise_floor_frame(base, _truth_deltas()) + result = signal_vs_noise_test(matched, floor, n_boot=100, seed=5) + row = result.set_index("model").loc["m"] + # Model moves $76.50 on payroll; identical repeats -> zero noise. + assert row["signal_median_abs_delta"] == pytest.approx(76.5) + assert row["noise_floor_median_abs_delta"] == pytest.approx(0.0) + assert bool(row["distinguishable_from_noise"]) is True + assert ZERO_DELTA_BASELINE_MODEL not in set(result["model"]) + + +class TestBinomialCI: + def test_exact_ci_known_values(self): + low, high = binomial_ci_exact(0, 20) + assert low == pytest.approx(0.0) + assert high == pytest.approx(0.1684, abs=2e-3) + low, high = binomial_ci_exact(10, 20) + assert low == pytest.approx(0.272, abs=2e-3) + assert high == pytest.approx(0.728, abs=2e-3) + + def test_degenerate_n_zero(self): + low, high = binomial_ci_exact(0, 0) + assert pd.isna(low) and pd.isna(high) diff --git a/tests/test_stability.py b/tests/test_stability.py new file mode 100644 index 0000000..ef67b1f --- /dev/null +++ b/tests/test_stability.py @@ -0,0 +1,348 @@ +"""Tests for repeated-run answer-stability metrics (stability spec layer 1).""" + +import json +import math + +import pandas as pd +import pytest + +from policybench.stability import ( + CacheContaminationError, + assert_cache_free, + chi2_ppf, + cross_run_response_id_duplicates, + is_parsed_prediction, + load_runs_dirs, + mutually_exact, + row_stability_by_model, + run_pair_frame, + run_std_ci, + scan_runs_for_cache_hits, + stability_variance_decomposition, +) + + +class TestChiSquare: + def test_known_quantiles_df2_closed_form(self): + # df=2 has the closed form ppf(p) = -2 ln(1 - p). + assert chi2_ppf(0.95, 2) == pytest.approx(5.99146, abs=1e-4) + assert chi2_ppf(0.05, 2) == pytest.approx(0.10259, abs=1e-4) + + def test_known_quantiles_other_df(self): + assert chi2_ppf(0.95, 4) == pytest.approx(9.48773, abs=1e-3) + assert chi2_ppf(0.05, 4) == pytest.approx(0.71072, abs=1e-3) + assert chi2_ppf(0.95, 1) == pytest.approx(3.84146, abs=1e-3) + + def test_run_std_ci_multipliers_match_spec(self): + # Spec: K=3 -> std CI multipliers [0.578, 4.415] at 90% two-sided. + low, high = run_std_ci(1.0, k=3, level=0.90) + assert low == pytest.approx(0.578, abs=2e-3) + assert high == pytest.approx(4.415, abs=2e-3) + low5, high5 = run_std_ci(1.0, k=5, level=0.90) + assert low5 == pytest.approx(0.649, abs=2e-3) + assert high5 == pytest.approx(2.372, abs=2e-3) + + def test_run_std_ci_scales_with_std(self): + low1, high1 = run_std_ci(1.0, k=3) + low2, high2 = run_std_ci(2.0, k=3) + assert low2 == pytest.approx(2 * low1) + assert high2 == pytest.approx(2 * high1) + + def test_run_std_ci_nan_below_two_runs(self): + low, high = run_std_ci(float("nan"), k=1) + assert math.isnan(low) and math.isnan(high) + + +class TestParseAndMutualExact: + def test_amount_parse(self): + assert is_parsed_prediction("snap", 100.0) + assert is_parsed_prediction("snap", 0.0) + assert not is_parsed_prediction("snap", None) + assert not is_parsed_prediction("snap", float("nan")) + + def test_binary_parse_requires_valid_flag(self): + assert is_parsed_prediction("person_wic_eligible", 1.0) + assert is_parsed_prediction("person_wic_eligible", 0) + # Numeric but not 0/1: coverage-parse would say parsed; flag-parse says no. + assert not is_parsed_prediction("person_wic_eligible", 0.5) + assert not is_parsed_prediction("person_wic_eligible", None) + + def test_mutual_exact_amounts_dollar_tolerance(self): + assert mutually_exact("snap", 100.0, 100.9) + assert not mutually_exact("snap", 100.0, 101.5) + assert not mutually_exact("snap", None, 100.0) + + def test_mutual_exact_binary_flags(self): + assert mutually_exact("person_wic_eligible", 1.0, 1) + assert not mutually_exact("person_wic_eligible", 1.0, 0.0) + # Identical invalid flags are neither flips nor exact. + assert not mutually_exact("person_wic_eligible", 0.5, 0.5) + + +def _ground_truth(): + return pd.DataFrame( + { + "scenario_id": ["s1", "s1", "s2", "s2"], + "variable": ["snap", "person_wic_eligible", "snap", "person_wic_eligible"], + "value": [100.0, 1.0, 0.0, 0.0], + } + ) + + +def _repeated_predictions(): + """Two runs, one model, engineered per-row behaviors. + + s1/snap: 100 vs 250 -> parsed both, flip, verdict flips (exact vs wrong). + s1/wic: 1 vs 1 -> stable and exact-correct both runs. + s2/snap: 0 vs missing -> coverage flip; verdict flips under all-rows. + s2/wic: 0.5 vs 0.5 -> invalid binary in both runs (never parsed). + """ + rows = [] + for run_id, snap1, wic1, snap2, wic2 in [ + ("run_000", 100.0, 1.0, 0.0, 0.5), + ("run_001", 250.0, 1.0, None, 0.5), + ]: + rows += [ + {"run_id": run_id, "model": "m", "scenario_id": "s1", "variable": "snap", "prediction": snap1}, + {"run_id": run_id, "model": "m", "scenario_id": "s1", "variable": "person_wic_eligible", "prediction": wic1}, + {"run_id": run_id, "model": "m", "scenario_id": "s2", "variable": "snap", "prediction": snap2}, + {"run_id": run_id, "model": "m", "scenario_id": "s2", "variable": "person_wic_eligible", "prediction": wic2}, + ] + return pd.DataFrame(rows) + + +class TestRunPairFrame: + def test_pair_flags(self): + pairs = run_pair_frame(_repeated_predictions(), _ground_truth()) + assert set(pairs["run_a"]) == {"run_000"} + assert set(pairs["run_b"]) == {"run_001"} + by_row = pairs.set_index(["scenario_id", "variable"]) + + snap1 = by_row.loc[("s1", "snap")] + assert bool(snap1["both_parsed"]) and not bool(snap1["mutually_exact"]) + assert bool(snap1["exact_a"]) and not bool(snap1["exact_b"]) + + wic1 = by_row.loc[("s1", "person_wic_eligible")] + assert bool(wic1["mutually_exact"]) and bool(wic1["exact_a"]) and bool(wic1["exact_b"]) + + snap2 = by_row.loc[("s2", "snap")] + assert not bool(snap2["both_parsed"]) + assert bool(snap2["exact_a"]) and not bool(snap2["exact_b"]) + + wic2 = by_row.loc[("s2", "person_wic_eligible")] + assert not bool(wic2["both_parsed"]) + + def test_three_runs_make_three_pairs_per_row(self): + preds = _repeated_predictions() + third = preds[preds["run_id"] == "run_000"].assign(run_id="run_002") + pairs = run_pair_frame(pd.concat([preds, third]), _ground_truth()) + assert len(pairs) == 4 * 3 # C(3,2) pairs x 4 rows + + +class TestRowStability: + def test_metrics_on_engineered_runs(self): + table = row_stability_by_model( + _repeated_predictions(), _ground_truth(), n_boot=50, seed=1 + ) + row = table.set_index("model").loc["m"] + assert row["k"] == 2 + assert row["n_rows"] == 4 + assert row["n_scenarios"] == 2 + # Parsed in all runs: s1/snap and s1/wic -> 2/4. + assert row["unanimous_parse_rate"] == pytest.approx(0.5) + # s2/snap parsed once, missing once. + assert row["coverage_flip_rate"] == pytest.approx(0.25) + # s2/wic invalid in at least one run, over 2 binary rows. + assert row["invalid_binary_rate"] == pytest.approx(0.5) + # Both-parsed pairs: s1/snap (flip), s1/wic (stable) -> 1/2. + assert row["answer_flip_rate"] == pytest.approx(0.5) + assert row["unanimous_exact_answer_rate"] == pytest.approx(0.25) + # Verdict flips among both-parsed pairs: s1/snap only -> 1/2. + assert row["verdict_flip_rate_parsed"] == pytest.approx(0.5) + # All pairs: s1/snap flip + s2/snap (exact vs missing) -> 2/4. + assert row["verdict_flip_rate_all"] == pytest.approx(0.5) + + def test_consistently_wrong_divergent_amount_rows_only(self): + gt = pd.DataFrame( + { + "scenario_id": ["s1", "s1"], + "variable": ["snap", "person_wic_eligible"], + "value": [500.0, 1.0], + } + ) + preds = pd.DataFrame( + { + "run_id": ["run_000"] * 2 + ["run_001"] * 2, + "model": ["m"] * 4, + "scenario_id": ["s1"] * 4, + "variable": ["snap", "person_wic_eligible"] * 2, + # snap: wrong both runs with different wrong answers. + # wic: wrong both runs (same wrong flag, the only one possible). + "prediction": [100.0, 0.0, 200.0, 0.0], + } + ) + table = row_stability_by_model(preds, gt, n_boot=10, seed=1) + out = table.set_index("model").loc["m"] + # Binary row excluded by construction; 1/1 amount rows divergent. + assert out["consistently_wrong_divergent_rate"] == pytest.approx(1.0) + assert out["n_consistently_wrong_amount_rows"] == 1 + + def test_bootstrap_ci_columns_present_and_ordered(self): + table = row_stability_by_model( + _repeated_predictions(), _ground_truth(), n_boot=50, seed=7 + ) + row = table.iloc[0] + for metric in ["answer_flip_rate", "verdict_flip_rate_parsed"]: + assert row[f"{metric}_ci_low"] <= row[metric] <= row[f"{metric}_ci_high"] + + def test_bootstrap_deterministic_under_seed(self): + a = row_stability_by_model(_repeated_predictions(), _ground_truth(), n_boot=30, seed=3) + b = row_stability_by_model(_repeated_predictions(), _ground_truth(), n_boot=30, seed=3) + pd.testing.assert_frame_equal(a, b) + + def test_empty_inputs(self): + assert row_stability_by_model(pd.DataFrame(), _ground_truth()).empty + assert row_stability_by_model(_repeated_predictions(), pd.DataFrame()).empty + + +class TestVarianceDecomposition: + def _inputs(self): + gt = pd.DataFrame( + { + "scenario_id": ["s1", "s2", "s3", "s4"], + "variable": ["snap"] * 4, + "value": [100.0, 200.0, 300.0, 400.0], + } + ) + rows = [] + # Model with run-to-run movement: run_000 all exact, run_001 half exact, + # run_002 all exact. + for run_id, preds in [ + ("run_000", [100.0, 200.0, 300.0, 400.0]), + ("run_001", [100.0, 200.0, 999.0, 999.0]), + ("run_002", [100.0, 200.0, 300.0, 400.0]), + ]: + for sid, p in zip(["s1", "s2", "s3", "s4"], preds, strict=True): + rows.append( + {"run_id": run_id, "model": "m", "scenario_id": sid, "variable": "snap", "prediction": p} + ) + return gt, pd.DataFrame(rows), {"s1": 0.0, "s2": 0.0, "s3": 0.0, "s4": 0.0} + + def test_per_model_columns_and_decision_rule(self): + gt, preds, market = self._inputs() + per_model, pooled = stability_variance_decomposition(gt, preds, market) + row = per_model.set_index("model").loc["m"] + assert row["k"] == 3 + # Run scores: 1.0, 0.5, 1.0 -> std = 0.288675. + assert row["run_score_mean"] == pytest.approx(5 / 6, abs=1e-6) + assert row["run_score_std"] == pytest.approx(0.288675, abs=1e-5) + assert row["run_score_std_ci_low"] < row["run_score_std"] < row["run_score_std_ci_high"] + assert row["sampling_se"] > 0 + assert row["run_to_sampling_ratio"] == pytest.approx( + row["run_score_std"] / row["sampling_se"] + ) + # Decision rule uses the CI upper bound, not the point ratio. + expected = bool(row["run_score_std_ci_high"] < row["sampling_se"]) + assert bool(row["sampling_dominates"]) == expected + # K=3 detectable floor from the spec. + assert row["detectable_ratio_floor"] == pytest.approx(1 / 4.415, abs=1e-3) + + def test_pooled_roster_estimate(self): + gt, preds, market = self._inputs() + second = preds.assign(model="m2") + both = pd.concat([preds, second], ignore_index=True) + per_model, pooled = stability_variance_decomposition(gt, both, market) + assert len(per_model) == 2 + assert pooled["n_models"] == 2 + assert pooled["pooled_df"] == 2 * 2 + assert pooled["pooled_run_score_std"] == pytest.approx(0.288675, abs=1e-5) + homogeneity = pooled["homogeneity"] + assert set(homogeneity) == {"m", "m2"} + assert sum(homogeneity.values()) == pytest.approx(1.0) + + def test_requires_run_id(self): + gt, preds, market = self._inputs() + per_model, pooled = stability_variance_decomposition( + gt, preds.drop(columns=["run_id"]), market + ) + assert per_model.empty + + +class TestCacheGuard: + def _write_ledger(self, tmp_path, records): + runs = tmp_path / "runs" + runs.mkdir() + (runs / "run_000.csv").write_text("model,scenario_id\n") + ledger = runs / "run_000.csv.spend.jsonl" + ledger.write_text("\n".join(json.dumps(r) for r in records) + "\n") + return runs + + def test_clean_ledger_passes(self, tmp_path): + runs = self._write_ledger(tmp_path, [{"cache_hit": False}, {"cache_hit": False}]) + report = scan_runs_for_cache_hits([runs]) + assert report["cache_hits"] == 0 + assert report["records"] == 2 + assert_cache_free([runs]) + + def test_cache_hit_raises(self, tmp_path): + runs = self._write_ledger(tmp_path, [{"cache_hit": False}, {"cache_hit": True}]) + assert scan_runs_for_cache_hits([runs])["cache_hits"] == 1 + with pytest.raises(CacheContaminationError): + assert_cache_free([runs]) + + def test_missing_ledgers_reported(self, tmp_path): + runs = tmp_path / "runs" + runs.mkdir() + (runs / "run_000.csv").write_text("model\n") + report = scan_runs_for_cache_hits([runs]) + assert report["ledgers"] == 0 + # No ledger is not proof of no cache: surfaced, not silently passed. + assert report["runs_without_ledger"] == 1 + + def test_cross_run_response_id_duplicates(self): + preds = pd.DataFrame( + { + "run_id": ["run_000", "run_001", "run_000", "run_001"], + "model": ["m"] * 4, + "scenario_id": ["s1", "s1", "s2", "s2"], + "variable": ["snap"] * 4, + "prediction": [1.0] * 4, + "provider_response_id": ["r1", "r1", "r2", "r3"], + } + ) + assert cross_run_response_id_duplicates(preds) == 1 + assert cross_run_response_id_duplicates(preds.drop(columns=["provider_response_id"])) is None + + +class TestLoadRunsDirs: + def _write_runs(self, path, model, run_ids): + path.mkdir(parents=True) + for run_id in run_ids: + pd.DataFrame( + { + "run_id": [run_id], + "model": [model], + "scenario_id": ["s1"], + "variable": ["snap"], + "prediction": [1.0], + } + ).to_csv(path / f"{run_id}.csv", index=False) + + def test_loads_and_concats_disjoint_models(self, tmp_path): + self._write_runs(tmp_path / "a", "m1", ["run_000", "run_001"]) + self._write_runs(tmp_path / "b", "m2", ["run_000", "run_001"]) + frame = load_runs_dirs([tmp_path / "a", tmp_path / "b"]) + assert set(frame["model"]) == {"m1", "m2"} + assert len(frame) == 4 + + def test_rejects_overlapping_models(self, tmp_path): + self._write_runs(tmp_path / "a", "m1", ["run_000"]) + self._write_runs(tmp_path / "b", "m1", ["run_000"]) + with pytest.raises(ValueError, match="model"): + load_runs_dirs([tmp_path / "a", tmp_path / "b"]) + + def test_rejects_mismatched_run_id_sets(self, tmp_path): + self._write_runs(tmp_path / "a", "m1", ["run_000", "run_001"]) + self._write_runs(tmp_path / "b", "m2", ["run_000"]) + with pytest.raises(ValueError, match="run_id"): + load_runs_dirs([tmp_path / "a", tmp_path / "b"]) diff --git a/tests/test_stability_harness.py b/tests/test_stability_harness.py new file mode 100644 index 0000000..ec1a7df --- /dev/null +++ b/tests/test_stability_harness.py @@ -0,0 +1,165 @@ +"""Harness-side changes backing the stability spec: condition fingerprints, +repeat-set metadata, and cache-free repeated runs.""" + +import json +from unittest.mock import patch + +import pandas as pd + +from policybench.eval_no_tools import ( + _response_contract_metadata, + run_repeated_no_tools_eval, +) +from policybench.scenarios import scenario_manifest + + +class TestConditionFingerprint: + def test_default_condition_is_forced_no_override(self, monkeypatch): + monkeypatch.delenv("POLICYBENCH_TOOL_CHOICE", raising=False) + monkeypatch.delenv("POLICYBENCH_CHUNK_OVERRIDE", raising=False) + contract = _response_contract_metadata() + assert contract["tool_choice_condition"] == "forced" + assert contract["chunk_override"] is None + + def test_env_conditions_enter_the_fingerprint(self, monkeypatch): + monkeypatch.setenv("POLICYBENCH_TOOL_CHOICE", "auto") + monkeypatch.setenv("POLICYBENCH_CHUNK_OVERRIDE", "none") + contract = _response_contract_metadata() + assert contract["tool_choice_condition"] == "auto" + assert contract["chunk_override"] == "none" + + +class TestRunsMetadata: + def test_repeated_eval_writes_runs_metadata( + self, tmp_path, sample_scenarios, monkeypatch + ): + monkeypatch.delenv("POLICYBENCH_TOOL_CHOICE", raising=False) + + def fake_run(scenarios, models=None, programs=None, output_path=None, run_id=None, include_explanations=True): + frame = pd.DataFrame( + { + "run_id": [run_id], + "model": ["m"], + "scenario_id": [scenarios[0].id], + "variable": ["snap"], + "prediction": [1.0], + } + ) + frame.to_csv(output_path, index=False) + return frame + + with patch("policybench.eval_no_tools.run_no_tools_eval", side_effect=fake_run): + run_repeated_no_tools_eval( + sample_scenarios, + repeats=2, + output_dir=str(tmp_path), + models={"m": "m"}, + programs=["snap"], + ) + + metadata = json.loads((tmp_path / "runs_metadata.json").read_text()) + assert metadata["repeats"] == 2 + assert metadata["models"] == {"m": "m"} + assert metadata["programs"] == ["snap"] + assert metadata["cache_enabled"] is False + assert metadata["response_contract"]["tool_choice_condition"] == "forced" + assert metadata["scenario_count"] == len(sample_scenarios) + + def test_mixed_model_groups_in_one_directory_rejected( + self, tmp_path, sample_scenarios, monkeypatch + ): + monkeypatch.delenv("POLICYBENCH_TOOL_CHOICE", raising=False) + + def fake_run(scenarios, models=None, programs=None, output_path=None, run_id=None, include_explanations=True): + frame = pd.DataFrame( + { + "run_id": [run_id], + "model": [next(iter(models))], + "scenario_id": [scenarios[0].id], + "variable": ["snap"], + "prediction": [1.0], + } + ) + frame.to_csv(output_path, index=False) + return frame + + with patch("policybench.eval_no_tools.run_no_tools_eval", side_effect=fake_run): + run_repeated_no_tools_eval( + sample_scenarios, + repeats=1, + output_dir=str(tmp_path), + models={"m1": "m1"}, + programs=["snap"], + ) + try: + run_repeated_no_tools_eval( + sample_scenarios, + repeats=1, + output_dir=str(tmp_path), + models={"m2": "m2"}, + programs=["snap"], + ) + except ValueError as exc: + assert "models" in str(exc) + else: + raise AssertionError("expected ValueError for mixed model groups") + + +class TestRepeatedRunsCacheFree: + def _manifest(self, tmp_path, sample_scenarios): + path = tmp_path / "scenarios.csv" + scenario_manifest(sample_scenarios).to_csv(path, index=False) + return path + + def test_repeated_command_does_not_enable_cache(self, tmp_path, sample_scenarios): + from policybench import cli + + manifest = self._manifest(tmp_path, sample_scenarios) + argv = [ + "policybench", + "eval-no-tools-repeated", + "--scenario-manifest", + str(manifest), + "--num-scenarios", + str(len(sample_scenarios)), + "--output-dir", + str(tmp_path / "runs"), + "--repeats", + "1", + ] + with ( + patch("policybench.cache.enable_cache") as enable_cache, + patch( + "policybench.eval_no_tools.run_repeated_no_tools_eval", + return_value=pd.DataFrame(), + ) as run, + patch("sys.argv", argv), + ): + cli.main() + enable_cache.assert_not_called() + run.assert_called_once() + + def test_single_eval_still_enables_cache(self, tmp_path, sample_scenarios): + from policybench import cli + + manifest = self._manifest(tmp_path, sample_scenarios) + argv = [ + "policybench", + "eval-no-tools", + "--scenario-manifest", + str(manifest), + "--num-scenarios", + str(len(sample_scenarios)), + "--output", + str(tmp_path / "predictions.csv"), + ] + with ( + patch("policybench.cache.enable_cache") as enable_cache, + patch( + "policybench.eval_no_tools.run_no_tools_eval", + return_value=pd.DataFrame(), + ), + patch("sys.argv", argv), + ): + cli.main() + enable_cache.assert_called_once() From d87ffdd82b8ec7a8cfd9744f0423723d0c0ba96d Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Sun, 23 Aug 2026 12:35:36 +0200 Subject: [PATCH 3/7] Layer 2: reasoning-stability judge scaffolding (TDD) Mechanism-label extraction judge scoped to one explanation at a time (no pairs, no comparison, no arithmetic), committed few-shot anchors from the reference explanations, disk-deduplicated async extraction that never touches the LiteLLM cache, deterministic numeric-claim channel with half-up rounding and prediction exclusion, answer-stable pair frames with verbatim short-circuit, headline + joint/standardized/nonidentical/ attenuation companions and suppression rule, reference alignment, and the validation statistics the review required (Wilson CI, Gwet AC1, Krippendorff alpha over Jaccard, dominant-set share, gold-set eval). Co-Authored-By: Claude Fable 5 --- policybench/reasoning_stability.py | 802 +++++++++++++++++++++++++++++ tests/test_reasoning_stability.py | 451 ++++++++++++++++ 2 files changed, 1253 insertions(+) create mode 100644 policybench/reasoning_stability.py create mode 100644 tests/test_reasoning_stability.py diff --git a/policybench/reasoning_stability.py b/policybench/reasoning_stability.py new file mode 100644 index 0000000..74c7cb6 --- /dev/null +++ b/policybench/reasoning_stability.py @@ -0,0 +1,802 @@ +"""Reasoning stability (stability spec layer 2). + +When a model gives the same answer twice, does it cite the same mechanism +twice? The LLM judge here does one thing only: it maps a single explanation +text onto discrete mechanism labels drawn from the audit taxonomy. It never +sees a pair, never compares, never scores, and never does arithmetic. +Everything else — pairing, agreement, numeric-claim comparison, validation +statistics — is deterministic code (docs/stability_spec.md, layer 2). + +Per the adversarial review, 12-label extraction is the Stability Trap +paper's Tier-2 semantic-classification regime, not its >90% verbatim +extraction regime; the validation battery (gold set, cross-judge +agreement, AC1/Krippendorff) carries the reliability claim. +""" + +from __future__ import annotations + +import asyncio +import hashlib +import json +import math +import re +from decimal import ROUND_HALF_UP, Decimal +from itertools import combinations +from pathlib import Path + +import litellm +import numpy as np +import pandas as pd + +from policybench.analysis import row_hit_scores +from policybench.annotation_taxonomy import FAILURE_SUBTYPE_VALUES +from policybench.spec import output_group_id +from policybench.stability import run_pair_frame + +JUDGE_PROMPT_VERSION = "2026-08-18-v1" +DEFAULT_JUDGE_MODEL = "gemini/gemini-3.7-flash" +DEFAULT_CROSS_JUDGE_MODEL = "gpt-5.4-mini" +DEFAULT_CONCURRENCY = 8 +JUDGE_MAX_TOKENS = 200 +MIN_STABLE_EXACT_PAIRS = 200 +VALIDATION_SAMPLE_MODULUS = 10 # ~10% of unique texts, by key hash + +MECHANISM_LABELS: tuple[str, ...] = tuple( + label for label in FAILURE_SUBTYPE_VALUES if label != "missing_output" +) + +# Pre-registered fallback if the judge fails its reliability gate. +SUPER_DOMAINS: dict[str, tuple[str, ...]] = { + "income_and_deductions": ( + "taxable_income_or_deductions", + "asset_resource", + "period_annualization", + ), + "rates_and_phaseouts": ("thresholds_rates", "credit_phaseout"), + "eligibility": ( + "categorical_eligibility", + "age_disability", + "household_unit_or_filing_status", + ), + "program_specific": ("health_coverage", "payroll_tax_base", "state_local_rule"), + "other": ("other",), +} + +LABEL_GLOSSARY: dict[str, str] = { + "taxable_income_or_deductions": ( + "how taxable or countable income is built: deductions, exclusions, " + "itemization, allowances, AGI construction" + ), + "credit_phaseout": ( + "a credit's amount, phase-in, phase-out, or refundability (EITC, CTC, " + "state credits)" + ), + "thresholds_rates": ( + "a specific rate, bracket, threshold, poverty-line multiple, taper, " + "or exemption amount applied to a quantity" + ), + "categorical_eligibility": ( + "a program's eligibility test or pathway: who qualifies, income or " + "categorical tests, enrollment rules" + ), + "asset_resource": "asset, resource, savings, or capital limits and tests", + "health_coverage": ( + "health coverage mechanics: premiums, Marketplace plans, " + "employer-sponsored insurance, Medicaid/CHIP/Medicare coverage logic" + ), + "age_disability": ( + "age limits, disability or blindness status, Medicare age, " + "child-age windows" + ), + "period_annualization": ( + "converting between monthly, weekly, and annual amounts or " + "certification periods" + ), + "payroll_tax_base": ( + "payroll or self-employment tax: Social Security, Medicare, FICA, " + "wage bases" + ), + "state_local_rule": ( + "a state or local rule, parameter, supplement, conformity election, " + "or absence of a state tax" + ), + "household_unit_or_filing_status": ( + "filing status, dependents, household or unit composition, " + "head-of-household or joint rules, deeming across members" + ), + "other": ( + "a mechanism that fits none of the above, or an explanation that " + "names no mechanism" + ), +} + +# Few-shot anchors, verbatim from the committed reference-explanation CSV +# (annotations/us_full_run_20260612_policyengine_4_16_1_populace/ +# us_case_reference_explanations.csv), labeled by the developers. +FEW_SHOT_EXAMPLES: tuple[tuple[str, str, tuple[str, ...]], ...] = ( + ( + "payroll_tax", + "PolicyEngine calculated the annual payroll tax for this single Texas " + "resident at $229.50 by summing two components of the employee payroll " + "tax. The employee Social Security tax contributed $186.00, which is " + "calculated as 6.2% of the individual's covered wages up to the annual " + "Social Security wage base. The employee Medicare tax added $43.50, " + "computed at 1.45% of total wages with no wage cap. Since this " + "individual is classified as a tax unit head, these employee-side " + "payroll obligations were attributed directly to their tax liability " + "for the 2026 tax year.", + ("payroll_tax_base", "thresholds_rates"), + ), + ( + "snap", + "PolicyEngine calculated a SNAP (Supplemental Nutrition Assistance " + "Program) benefit of $0 for this Ohio household of two adults with no " + "children and approximately $94,925 in annual household income. The " + "household's income substantially exceeds the SNAP eligibility " + "threshold for a two-person household in 2026, which is set at 130% " + "of the federal poverty line. With this income level, the household " + "does not qualify for any SNAP benefits under federal program rules, " + "resulting in the zero benefit amount.", + ("categorical_eligibility", "thresholds_rates"), + ), + ( + "child1_early_head_start_eligible", + "PolicyEngine determined that neither child in this California " + "household is eligible for Early Head Start in 2026, resulting in a " + "value of False for both children. Early Head Start eligibility is " + "typically restricted to children under age 3 from families with " + "incomes at or below 100% of the federal poverty line or receiving " + "public assistance. With a household income of approximately " + "$159,676, this family's earnings substantially exceed the income " + "threshold required for the program. Consequently, PolicyEngine " + "assigned an eligibility status of False to each child, yielding the " + "reference value of 0.0 for the household.", + ("categorical_eligibility", "age_disability", "thresholds_rates"), + ), + ( + "ssi", + "For this Ohio household in 2026, PolicyEngine calculated Supplemental " + "Security Income (SSI) of $0.00 because neither household member meets " + "SSI's eligibility criteria. The head of household, age 61, and the " + "spouse, age 57, both have `is_ssi_aged_blind_disabled` set to False, " + "meaning neither qualifies under SSI's three eligibility pathways: " + "being age 65 or older, blind, or disabled. Since SSI eligibility is " + "determined on an individual basis with spousal deeming rules applied " + "per person rather than as a pooled household total, the absence of " + "any qualifying member results in zero SSI benefits for the " + "household.", + ("categorical_eligibility", "age_disability", "household_unit_or_filing_status"), + ), +) + +_USD_RE = re.compile(r"\$\s?(\d[\d,]*(?:\.\d+)?)") +_PCT_RE = re.compile(r"(\d+(?:\.\d+)?)\s?(?:%|percent\b)") +_WS_RE = re.compile(r"\s+") +_JSON_OBJECT_RE = re.compile(r"\{.*\}", re.DOTALL) + + +def _round_half_up(value: float, ndigits: int) -> float: + """Round like a reader would (1.45 -> 1.5), not like float repr does.""" + quantum = Decimal(1).scaleb(-ndigits) + return float(Decimal(str(value)).quantize(quantum, rounding=ROUND_HALF_UP)) + + +def normalize_explanation_text(text) -> str: + """Casefold and collapse whitespace so trivially different texts match.""" + if text is None: + return "" + try: + if pd.isna(text): + return "" + except (TypeError, ValueError): + pass + return _WS_RE.sub(" ", str(text)).strip().casefold() + + +def extract_numeric_claims(text, exclude_value: float | None = None) -> frozenset: + """Deterministic numeric-claim channel: dollars (to $1) and percents (0.1pp). + + Values within $1 of ``exclude_value`` (the row's own prediction) are + dropped — the restated answer would inflate overlap by construction. + """ + normalized = normalize_explanation_text(text) + if not normalized: + return frozenset() + claims = set() + for match in _USD_RE.finditer(normalized): + try: + value = float(match.group(1).replace(",", "")) + except ValueError: + continue + if exclude_value is not None and abs(value - float(exclude_value)) <= 1.0: + continue + claims.add(("usd", int(_round_half_up(value, 0)))) + for match in _PCT_RE.finditer(normalized): + try: + value = float(match.group(1)) + except ValueError: + continue + claims.add(("pct", _round_half_up(value, 1))) + return frozenset(claims) + + +def text_key(variable: str, text) -> str: + """Judge-independent identity of an explanation text for one variable.""" + payload = f"{JUDGE_PROMPT_VERSION}|{variable}|{normalize_explanation_text(text)}" + return hashlib.sha256(payload.encode("utf-8")).hexdigest() + + +def judge_cache_key(judge_model: str, variable: str, text, grade_pass: int = 0) -> str: + """Dedup key for one extraction call; grade_pass forces independent repeats.""" + payload = f"{judge_model}|{text_key(variable, text)}|{grade_pass}" + return hashlib.sha256(payload.encode("utf-8")).hexdigest() + + +def build_judge_prompt(text: str, variable: str, country: str = "us", year: int = 2026) -> str: + glossary = "\n".join( + f"- {label}: {definition}" for label, definition in LABEL_GLOSSARY.items() + ) + examples = "\n\n".join( + f"Example (output: {example_variable})\nExplanation: {example_text}\n" + f'Labels: {{"labels": {json.dumps(list(labels))}}}' + for example_variable, example_text, labels in FEW_SHOT_EXAMPLES + ) + return f"""You label the mechanisms that one tax/benefit explanation relies on. +Prompt version: {JUDGE_PROMPT_VERSION} + +Read the single explanation below and list every mechanism label whose +definition matches something the explanation's stated derivation actually +rests on. Use only labels from this list: +{glossary} + +Rules: label only what the text states, not what the correct derivation +would be; do not evaluate whether the explanation is right; do not +calculate anything; return strict JSON with one key "labels" holding a +list of label strings (an explanation that names no mechanism gets +["other"]). + +{examples} + +Now label this explanation. +Output: {variable} (country {country.upper()}, tax year {year}) +Explanation: {text} +Labels:""" + + +def parse_judge_labels(content) -> list[str]: + """Validate a judge response into a deduplicated list of taxonomy labels.""" + if content is None: + raise ValueError("empty judge response") + match = _JSON_OBJECT_RE.search(str(content)) + if not match: + raise ValueError("invalid judge response: no JSON object found") + try: + payload = json.loads(match.group(0)) + except json.JSONDecodeError as exc: + raise ValueError(f"invalid judge response: {exc}") from exc + labels = payload.get("labels") if isinstance(payload, dict) else None + if not isinstance(labels, list) or not all(isinstance(x, str) for x in labels): + raise ValueError("invalid judge response: 'labels' must be a list of strings") + out: list[str] = [] + for label in labels: + if label not in MECHANISM_LABELS: + raise ValueError(f"invalid judge response: unknown label {label!r}") + if label not in out: + out.append(label) + if not out: + raise ValueError("invalid judge response: empty label list") + return out + + +async def _extract_one( + semaphore: asyncio.Semaphore, + judge_model: str, + item: dict, +) -> dict: + prompt = build_judge_prompt( + item["text"], item["variable"], item.get("country", "us"), item.get("year", 2026) + ) + last_error = "" + async with semaphore: + for _attempt in range(2): + try: + response = await litellm.acompletion( + model=judge_model, + messages=[{"role": "user", "content": prompt}], + temperature=0, + max_tokens=JUDGE_MAX_TOKENS, + # Never the shared disk cache: validation re-grades must be + # independent calls, and identical texts are deduplicated + # by this module's own key instead. + caching=False, + ) + content = response.choices[0].message.content + labels = parse_judge_labels(content) + return {"labels": labels, "error": "", "raw": str(content)} + except ValueError as exc: + last_error = str(exc) + except Exception as exc: # provider/transport failures + last_error = f"{type(exc).__name__}: {exc}" + return {"labels": None, "error": last_error or "invalid judge response", "raw": ""} + + +async def _extract_batch(items: list[dict], judge_model: str, concurrency: int) -> list[dict]: + semaphore = asyncio.Semaphore(max(1, concurrency)) + return await asyncio.gather( + *(_extract_one(semaphore, judge_model, item) for item in items) + ) + + +def _read_cache(cache_path: Path) -> dict[str, dict]: + if not cache_path.exists(): + return {} + out: dict[str, dict] = {} + for line in cache_path.read_text(encoding="utf-8").splitlines(): + line = line.strip() + if not line: + continue + try: + record = json.loads(line) + except json.JSONDecodeError: + continue + key = record.get("key") + if key: + out[key] = record + return out + + +def run_label_extraction( + items: list[dict], + judge_model: str, + cache_path: str | Path, + concurrency: int = DEFAULT_CONCURRENCY, + grade_pass: int = 0, +) -> dict[str, dict]: + """Extract mechanism labels for explanation items with disk dedup. + + ``items`` carry ``variable``, ``text`` and optionally ``country``/``year``. + Returns ``{judge_cache_key: record}`` where a record has ``labels`` (list + or None), ``error``, ``text_key``, ``variable``, ``judge_model``, + ``prompt_version`` and ``grade_pass``. Identical normalized texts for one + variable share one call; failed extractions are recorded (and retried on + the next invocation) rather than cached as results. + """ + cache_path = Path(cache_path) + cache = _read_cache(cache_path) + unique: dict[str, dict] = {} + for item in items: + key = judge_cache_key(judge_model, item["variable"], item["text"], grade_pass) + unique.setdefault(key, item) + todo = {key: item for key, item in unique.items() if key not in cache} + if todo: + keys = list(todo) + results = asyncio.run( + _extract_batch([todo[key] for key in keys], judge_model, concurrency) + ) + cache_path.parent.mkdir(parents=True, exist_ok=True) + with cache_path.open("a", encoding="utf-8") as handle: + for key, result in zip(keys, results, strict=True): + item = todo[key] + record = { + "key": key, + "text_key": text_key(item["variable"], item["text"]), + "variable": item["variable"], + "judge_model": judge_model, + "prompt_version": JUDGE_PROMPT_VERSION, + "grade_pass": grade_pass, + "labels": result["labels"], + "error": result["error"], + } + cache[key] = record + if result["labels"] is not None: + handle.write(json.dumps(record, sort_keys=True) + "\n") + return {key: cache[key] for key in unique} + + +def labels_by_text_key_from_results(results: dict[str, dict]) -> dict[str, frozenset]: + """Collapse extraction records into {text_key: label set} (errors dropped).""" + return { + record["text_key"]: frozenset(record["labels"]) + for record in results.values() + if record.get("labels") + } + + +def explanation_pair_frame( + repeated_predictions: pd.DataFrame, + ground_truth: pd.DataFrame, +) -> pd.DataFrame: + """Run pairs with explanations, answer-stability strata, and text keys.""" + pairs = run_pair_frame(repeated_predictions, ground_truth) + if pairs.empty: + return pairs + has_explanations = "explanation" in repeated_predictions.columns + lookup = { + (row.model, row.run_id, row.scenario_id, row.variable): ( + row.prediction, + getattr(row, "explanation", None) if has_explanations else None, + ) + for row in repeated_predictions.itertuples() + } + records = [] + for row in pairs.itertuples(): + pred_a, text_a = lookup.get((row.model, row.run_a, row.scenario_id, row.variable), (None, None)) + pred_b, text_b = lookup.get((row.model, row.run_b, row.scenario_id, row.variable), (None, None)) + norm_a = normalize_explanation_text(text_a) + norm_b = normalize_explanation_text(text_b) + stable = bool(row.both_parsed and row.mutually_exact) + records.append( + { + "model": row.model, + "scenario_id": row.scenario_id, + "variable": row.variable, + "output_group": output_group_id(row.variable), + "run_a": row.run_a, + "run_b": row.run_b, + "stable": stable, + "stable_exact": bool(stable and row.exact_a and row.exact_b), + "prediction_a": pred_a, + "prediction_b": pred_b, + "explanation_a": text_a, + "explanation_b": text_b, + "has_explanations": bool(norm_a and norm_b), + "verbatim_identical": bool(norm_a and norm_a == norm_b), + "text_key_a": text_key(row.variable, text_a), + "text_key_b": text_key(row.variable, text_b), + } + ) + return pd.DataFrame(records) + + +def _jaccard(a: frozenset, b: frozenset) -> float: + if not a and not b: + return 1.0 + return len(a & b) / len(a | b) + + +def _rate(mask: pd.Series, values: pd.Series) -> float: + selected = values[mask] + return float(selected.mean()) if len(selected) else float("nan") + + +def reasoning_stability_by_model( + pairs: pd.DataFrame, + labels_by_text_key: dict[str, frozenset], + pair_noise_floor: float | None = None, + min_stable_exact_pairs: int = MIN_STABLE_EXACT_PAIRS, +) -> dict[str, pd.DataFrame]: + """Per-model reasoning-stability metrics from judge labels. + + Verbatim-identical explanation pairs count as agreeing without a judge + call; the share of stable pairs resolved that way is published because a + model that repeats canned text incurs zero judge noise. Pairs lacking a + label on either side (judge error, unjudged) are excluded from label + rates and counted in ``n_unjudged_stable_pairs``. + """ + if pairs.empty: + return {"summary": pd.DataFrame(), "composition": pd.DataFrame()} + frame = pairs.copy() + labels_a = frame["text_key_a"].map(labels_by_text_key) + labels_b = frame["text_key_b"].map(labels_by_text_key) + frame["judged"] = frame["verbatim_identical"] | (labels_a.notna() & labels_b.notna()) + same = [] + jaccard = [] + claim_jaccard = [] + claim_disjoint = [] + for row, la, lb in zip(frame.itertuples(), labels_a, labels_b, strict=True): + if row.verbatim_identical: + same.append(True) + jaccard.append(1.0) + elif isinstance(la, frozenset) and isinstance(lb, frozenset): + same.append(la == lb) + jaccard.append(_jaccard(la, lb)) + else: + same.append(False) + jaccard.append(float("nan")) + claims_a = extract_numeric_claims(row.explanation_a, row.prediction_a) + claims_b = extract_numeric_claims(row.explanation_b, row.prediction_b) + claim_jaccard.append(_jaccard(claims_a, claims_b)) + claim_disjoint.append(bool(claims_a and claims_b and not (claims_a & claims_b))) + frame["same_labels"] = same + frame["label_jaccard"] = jaccard + frame["claim_jaccard"] = claim_jaccard + frame["claim_disjoint"] = claim_disjoint + frame["divergent"] = frame["judged"] & ~frame["same_labels"] + + stable_exact_judged = frame[frame["stable_exact"] & frame["judged"]] + composition_rows = [] + for (model, group), sub in stable_exact_judged.groupby(["model", "output_group"]): + composition_rows.append( + { + "model": model, + "output_group": group, + "n_stable_exact_pairs": int(len(sub)), + "unstable_reasoning_rate": float((~sub["same_labels"]).mean()), + } + ) + composition = pd.DataFrame(composition_rows) + if not composition.empty: + pooled = composition.groupby("output_group")["n_stable_exact_pairs"].sum() + standard_weights = pooled / pooled.sum() + composition = composition.merge( + composition.groupby("model")["n_stable_exact_pairs"] + .sum() + .rename("model_total"), + on="model", + ) + composition["share_within_model"] = ( + composition["n_stable_exact_pairs"] / composition["model_total"] + ) + composition = composition.drop(columns=["model_total"]) + else: + standard_weights = pd.Series(dtype=float) + + rows = [] + for model, group in frame.groupby("model"): + stable = group["stable"] + stable_exact = group["stable_exact"] + judged = group["judged"] + verbatim = group["verbatim_identical"] + n_judged_se = int((stable_exact & judged).sum()) + agreement_se = _rate(stable_exact & judged, group["same_labels"]) + headline = 1.0 - agreement_se if not math.isnan(agreement_se) else float("nan") + nonidentical = _rate(stable_exact & judged & ~verbatim, ~group["same_labels"]) + adjusted = float("nan") + if pair_noise_floor is not None and not math.isnan(headline): + adjusted = max(0.0, (headline - pair_noise_floor) / (1.0 - pair_noise_floor)) + standardized = float("nan") + if not composition.empty: + model_comp = composition[composition["model"] == model].set_index("output_group") + if not model_comp.empty: + weights = standard_weights.reindex(model_comp.index).fillna(0.0) + if weights.sum() > 0: + standardized = float( + (model_comp["unstable_reasoning_rate"] * weights).sum() + / weights.sum() + ) + rows.append( + { + "model": model, + "n_pairs_total": int(len(group)), + "n_stable_pairs": int(stable.sum()), + "n_stable_exact_pairs": int(stable_exact.sum()), + "n_judged_stable_exact_pairs": n_judged_se, + "n_unjudged_stable_pairs": int((stable & ~judged).sum()), + "short_circuit_share_stable": _rate(stable, verbatim), + "mechanism_agreement_rate_stable": _rate(stable & judged, group["same_labels"]), + "mechanism_agreement_rate_stable_exact": agreement_se, + "right_answer_unstable_reasoning_rate": headline, + "right_answer_unstable_reasoning_rate_nonidentical": nonidentical, + "right_answer_unstable_reasoning_rate_adjusted": adjusted, + "joint_unstable_reasoning_rate_all_pairs": float( + (stable_exact & group["divergent"]).sum() / len(group) + ), + "standardized_unstable_reasoning_rate": standardized, + "mechanism_jaccard_mean_stable": _rate(stable & judged, group["label_jaccard"]), + "mechanism_jaccard_mean_stable_exact": _rate( + stable_exact & judged, group["label_jaccard"] + ), + "numeric_claim_jaccard_mean_stable_exact": _rate( + stable_exact, group["claim_jaccard"] + ), + "numeric_claim_disjoint_rate_stable_exact": _rate( + stable_exact, group["claim_disjoint"] + ), + "reasoning_unstable_strict_rate": _rate( + stable_exact & judged, ~group["same_labels"] | group["claim_disjoint"] + ), + "headline_suppressed": n_judged_se < min_stable_exact_pairs, + } + ) + return {"summary": pd.DataFrame(rows), "composition": composition} + + +def reference_alignment_by_model( + repeated_predictions: pd.DataFrame, + ground_truth: pd.DataFrame, + labels_by_text_key: dict[str, frozenset], + reference_labels: dict[tuple[str, str], frozenset], +) -> pd.DataFrame: + """Among exact-correct run-explanations, share whose labels match the anchor.""" + truth = ground_truth.set_index(["scenario_id", "variable"])["value"] + rows = [] + for model, group in repeated_predictions.groupby("model"): + n_exact = 0 + n_labeled = 0 + n_aligned = 0 + for row in group.itertuples(): + key = (row.scenario_id, row.variable) + if key not in truth.index: + continue + if not row_hit_scores(row.variable, truth[key], row.prediction)["exact"]: + continue + n_exact += 1 + labels = labels_by_text_key.get( + text_key(row.variable, getattr(row, "explanation", None)) + ) + anchor = reference_labels.get(key) + if labels is None or anchor is None: + continue + n_labeled += 1 + n_aligned += int(labels == anchor) + rows.append( + { + "model": model, + "n_exact_rows": n_exact, + "n_exact_labeled": n_labeled, + "reference_alignment_rate": ( + n_aligned / n_labeled if n_labeled else float("nan") + ), + } + ) + return pd.DataFrame(rows) + + +# --- validation statistics ------------------------------------------------- + + +def wilson_ci(successes: int, n: int, z: float = 1.96) -> tuple[float, float]: + if n <= 0: + return float("nan"), float("nan") + p = successes / n + denominator = 1.0 + z * z / n + center = (p + z * z / (2 * n)) / denominator + half = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / denominator + return max(0.0, center - half), min(1.0, center + half) + + +def cohen_kappa(a: list[bool], b: list[bool]) -> float: + if not a or len(a) != len(b): + return float("nan") + a_arr = np.asarray(a, dtype=bool) + b_arr = np.asarray(b, dtype=bool) + pa = float((a_arr == b_arr).mean()) + pa_pos, pb_pos = float(a_arr.mean()), float(b_arr.mean()) + pe = pa_pos * pb_pos + (1 - pa_pos) * (1 - pb_pos) + if pe >= 1.0: + return float("nan") + return (pa - pe) / (1 - pe) + + +def gwet_ac1(a: list[bool], b: list[bool]) -> float: + """Gwet's AC1 for two raters on a binary label (prevalence-robust).""" + if not a or len(a) != len(b): + return float("nan") + a_arr = np.asarray(a, dtype=bool) + b_arr = np.asarray(b, dtype=bool) + pa = float((a_arr == b_arr).mean()) + pi = (float(a_arr.mean()) + float(b_arr.mean())) / 2.0 + pe = 2.0 * pi * (1.0 - pi) + if pe >= 1.0: + return float("nan") + return (pa - pe) / (1 - pe) + + +def krippendorff_alpha_jaccard( + set_pairs: list[tuple[frozenset, frozenset]], max_pooled: int = 2000, seed: int = 0 +) -> float: + """Krippendorff's alpha for two coders over label sets, Jaccard distance.""" + if not set_pairs: + return float("nan") + observed = float(np.mean([1.0 - _jaccard(a, b) for a, b in set_pairs])) + pooled = [s for pair in set_pairs for s in pair] + if len(pooled) > max_pooled: + rng = np.random.default_rng(seed) + pooled = [pooled[i] for i in rng.choice(len(pooled), max_pooled, replace=False)] + distances = [1.0 - _jaccard(x, y) for x, y in combinations(pooled, 2)] + expected = float(np.mean(distances)) if distances else 0.0 + if expected == 0.0: + return 1.0 if observed == 0.0 else float("nan") + return 1.0 - observed / expected + + +def dominant_set_share(label_sets: list[frozenset]) -> float: + """Share of runs agreeing with the modal label set (the R_stab analogue).""" + if not label_sets: + return float("nan") + counts: dict[frozenset, int] = {} + for s in label_sets: + counts[s] = counts.get(s, 0) + 1 + return max(counts.values()) / len(label_sets) + + +def evaluate_against_gold( + gold: pd.DataFrame, + labels_by_key: dict[str, frozenset], + judge_model: str, + grade_pass: int = 0, +) -> dict: + """Judge-vs-gold exact-set accuracy (Wilson CI) and per-label agreement. + + ``gold`` has columns ``variable``, ``explanation``, ``labels`` (pipe- + separated taxonomy labels); ``labels_by_key`` is keyed by judge cache key. + """ + matches = [] + judge_sets: list[frozenset] = [] + gold_sets: list[frozenset] = [] + for row in gold.itertuples(): + key = judge_cache_key(judge_model, row.variable, row.explanation, grade_pass) + judged = labels_by_key.get(key) + if judged is None: + continue + gold_set = frozenset( + label.strip() for label in str(row.labels).split("|") if label.strip() + ) + judge_sets.append(frozenset(judged)) + gold_sets.append(gold_set) + matches.append(frozenset(judged) == gold_set) + n_graded = len(matches) + accuracy = float(np.mean(matches)) if matches else float("nan") + ci_low, ci_high = wilson_ci(int(sum(matches)), n_graded) + per_label_rows = [] + for label in MECHANISM_LABELS: + judge_flags = [label in s for s in judge_sets] + gold_flags = [label in s for s in gold_sets] + per_label_rows.append( + { + "label": label, + "gold_positives": int(sum(gold_flags)), + "judge_positives": int(sum(judge_flags)), + "agreement": ( + float(np.mean([g == j for g, j in zip(gold_flags, judge_flags, strict=True)])) + if n_graded + else float("nan") + ), + "gwet_ac1": gwet_ac1(gold_flags, judge_flags), + } + ) + return { + "judge_model": judge_model, + "n_gold": int(len(gold)), + "n_graded": n_graded, + "exact_set_accuracy": accuracy, + "exact_set_accuracy_ci_low": ci_low, + "exact_set_accuracy_ci_high": ci_high, + "krippendorff_alpha_jaccard": krippendorff_alpha_jaccard( + list(zip(gold_sets, judge_sets, strict=True)) + ), + "per_label": pd.DataFrame(per_label_rows), + } + + +def validation_sample_keys(text_keys: list[str], modulus: int = VALIDATION_SAMPLE_MODULUS) -> list[str]: + """Deterministic ~1/modulus sample of unique texts for re-grading.""" + return [key for key in sorted(set(text_keys)) if int(key[:8], 16) % modulus == 0] + + +def pair_agreement_summary( + primary: dict[str, frozenset], + secondary: dict[str, frozenset], +) -> dict: + """Set agreement, noise floor, AC1/alpha/kappa over texts graded by both.""" + keys = sorted(set(primary) & set(secondary)) + if not keys: + return {"n": 0, "set_agreement": float("nan"), "pair_noise_floor": float("nan")} + pairs = [(primary[k], secondary[k]) for k in keys] + agree = [a == b for a, b in pairs] + n = len(keys) + agreement = float(np.mean(agree)) + ci_low, ci_high = wilson_ci(int(sum(agree)), n) + per_label = [] + for label in MECHANISM_LABELS: + a_flags = [label in a for a, _ in pairs] + b_flags = [label in b for _, b in pairs] + occurrences = int(sum(a_flags) + sum(b_flags)) + per_label.append( + { + "label": label, + "prevalence": occurrences / (2 * n), + "raw_agreement": float(np.mean([x == y for x, y in zip(a_flags, b_flags, strict=True)])), + "gwet_ac1": gwet_ac1(a_flags, b_flags), + "cohen_kappa": cohen_kappa(a_flags, b_flags) if occurrences >= 20 else float("nan"), + } + ) + return { + "n": n, + "set_agreement": agreement, + "set_agreement_ci_low": ci_low, + "set_agreement_ci_high": ci_high, + "pair_noise_floor": 1.0 - agreement, + "krippendorff_alpha_jaccard": krippendorff_alpha_jaccard(pairs), + "per_label": pd.DataFrame(per_label), + } diff --git a/tests/test_reasoning_stability.py b/tests/test_reasoning_stability.py new file mode 100644 index 0000000..e8e6d2c --- /dev/null +++ b/tests/test_reasoning_stability.py @@ -0,0 +1,451 @@ +"""Tests for reasoning stability (stability spec layer 2). + +The LLM judge is mocked everywhere; these tests pin the deterministic +machinery around it: numeric-claim extraction, pair construction, label +agreement metrics with their composition companions, the validation +statistics, and the extraction runner's dedup/retry/cache behavior. +""" + +import math +from types import SimpleNamespace +from unittest.mock import AsyncMock, patch + +import pandas as pd +import pytest + +from policybench.annotation_taxonomy import FAILURE_SUBTYPE_VALUES +from policybench.reasoning_stability import ( + JUDGE_PROMPT_VERSION, + MECHANISM_LABELS, + SUPER_DOMAINS, + build_judge_prompt, + cohen_kappa, + dominant_set_share, + evaluate_against_gold, + explanation_pair_frame, + extract_numeric_claims, + gwet_ac1, + judge_cache_key, + krippendorff_alpha_jaccard, + normalize_explanation_text, + parse_judge_labels, + reasoning_stability_by_model, + reference_alignment_by_model, + run_label_extraction, + wilson_ci, +) + + +class TestLabels: + def test_label_set_is_taxonomy_minus_missing_output(self): + assert set(MECHANISM_LABELS) == set(FAILURE_SUBTYPE_VALUES) - {"missing_output"} + assert len(MECHANISM_LABELS) == 12 + + def test_super_domains_partition_the_labels(self): + covered = [label for labels in SUPER_DOMAINS.values() for label in labels] + assert sorted(covered) == sorted(MECHANISM_LABELS) + assert len(SUPER_DOMAINS) == 5 + + +class TestNumericClaims: + def test_extracts_dollars_and_percents_with_rounding(self): + claims = extract_numeric_claims( + "The 6.2% rate applied up to the $176,100 wage base gave $186.00, " + "plus 1.45% Medicare ($43.50)." + ) + assert ("pct", 6.2) in claims + assert ("pct", 1.5) in claims # 1.45 rounds to 0.1pp + assert ("usd", 176100) in claims + assert ("usd", 186) in claims + assert ("usd", 44) in claims # 43.50 -> 44 to the dollar + + def test_excludes_the_restated_prediction(self): + text = "SNAP comes to $288 after the $177 standard deduction." + assert ("usd", 288) not in extract_numeric_claims(text, exclude_value=287.68) + assert ("usd", 177) in extract_numeric_claims(text, exclude_value=287.68) + + def test_empty_text(self): + assert extract_numeric_claims("") == frozenset() + assert extract_numeric_claims(None) == frozenset() + + +class TestNormalizationAndPrompt: + def test_normalization_collapses_case_and_whitespace(self): + assert normalize_explanation_text(" Two spaces\nand CASE ") == ( + normalize_explanation_text("two spaces and case") + ) + + def test_prompt_carries_labels_and_version_but_not_comparison(self): + prompt = build_judge_prompt("Payroll tax is 7.65% of wages.", "payroll_tax") + for label in MECHANISM_LABELS: + assert label in prompt + assert JUDGE_PROMPT_VERSION in prompt + assert "payroll_tax" in prompt + # The judge labels one explanation; it never sees a pair or a score. + assert "compare" not in prompt.lower() + + def test_cache_key_depends_on_text_model_and_pass(self): + base = judge_cache_key("judge-a", "snap", "Text one.", grade_pass=0) + assert base == judge_cache_key("judge-a", "snap", "text ONE.", grade_pass=0) + assert base != judge_cache_key("judge-b", "snap", "Text one.", grade_pass=0) + assert base != judge_cache_key("judge-a", "snap", "Text one.", grade_pass=1) + assert base != judge_cache_key("judge-a", "ssi", "Text one.", grade_pass=0) + + +class TestParseJudgeLabels: + def test_parses_json_object(self): + assert parse_judge_labels('{"labels": ["payroll_tax_base", "thresholds_rates"]}') == [ + "payroll_tax_base", + "thresholds_rates", + ] + + def test_tolerates_fenced_json_and_dedups(self): + content = '```json\n{"labels": ["other", "other"]}\n```' + assert parse_judge_labels(content) == ["other"] + + def test_rejects_unknown_labels_and_missing_key(self): + with pytest.raises(ValueError): + parse_judge_labels('{"labels": ["not_a_label"]}') + with pytest.raises(ValueError): + parse_judge_labels('{"mechanisms": ["other"]}') + with pytest.raises(ValueError): + parse_judge_labels("no json here") + + +def _judge_response(content: str): + return SimpleNamespace( + choices=[SimpleNamespace(message=SimpleNamespace(content=content))] + ) + + +class TestExtractionRunner: + def test_dedups_identical_texts_and_caches_to_disk(self, tmp_path): + items = [ + {"variable": "snap", "country": "us", "year": 2026, "text": "Income exceeds 130% FPL."}, + {"variable": "snap", "country": "us", "year": 2026, "text": "income exceeds 130% fpl."}, + {"variable": "ssi", "country": "us", "year": 2026, "text": "Neither adult is aged or disabled."}, + ] + cache = tmp_path / "labels.jsonl" + acompletion = AsyncMock( + side_effect=[ + _judge_response('{"labels": ["categorical_eligibility", "thresholds_rates"]}'), + _judge_response('{"labels": ["categorical_eligibility", "age_disability"]}'), + ] + ) + with patch("policybench.reasoning_stability.litellm.acompletion", acompletion): + results = run_label_extraction(items, judge_model="judge", cache_path=cache) + # Two unique normalized texts -> two calls, three keyed results. + assert acompletion.call_count == 2 + assert len(results) == 2 + snap_key = judge_cache_key("judge", "snap", items[0]["text"], 0) + assert results[snap_key]["labels"] == ["categorical_eligibility", "thresholds_rates"] + # Second invocation is served entirely from the cache file. + acompletion.reset_mock() + with patch("policybench.reasoning_stability.litellm.acompletion", acompletion): + again = run_label_extraction(items, judge_model="judge", cache_path=cache) + assert acompletion.call_count == 0 + assert again == results + assert len(cache.read_text().strip().splitlines()) == 2 + + def test_retries_once_then_records_error(self, tmp_path): + items = [{"variable": "snap", "country": "us", "year": 2026, "text": "x"}] + acompletion = AsyncMock( + side_effect=[_judge_response("garbage"), _judge_response("still garbage")] + ) + with patch("policybench.reasoning_stability.litellm.acompletion", acompletion): + results = run_label_extraction( + items, judge_model="judge", cache_path=tmp_path / "c.jsonl" + ) + assert acompletion.call_count == 2 + (result,) = results.values() + assert result["labels"] is None + assert "invalid" in result["error"].lower() + + def test_requests_never_use_litellm_cache_and_pass_temperature_zero(self, tmp_path): + items = [{"variable": "snap", "country": "us", "year": 2026, "text": "x"}] + acompletion = AsyncMock(return_value=_judge_response('{"labels": ["other"]}')) + with patch("policybench.reasoning_stability.litellm.acompletion", acompletion): + run_label_extraction(items, judge_model="judge", cache_path=tmp_path / "c.jsonl") + kwargs = acompletion.call_args.kwargs + assert kwargs["caching"] is False + assert kwargs["temperature"] == 0 + assert kwargs["model"] == "judge" + + def test_grade_pass_forces_independent_calls(self, tmp_path): + items = [{"variable": "snap", "country": "us", "year": 2026, "text": "x"}] + acompletion = AsyncMock(return_value=_judge_response('{"labels": ["other"]}')) + with patch("policybench.reasoning_stability.litellm.acompletion", acompletion): + run_label_extraction(items, judge_model="judge", cache_path=tmp_path / "c.jsonl") + run_label_extraction( + items, judge_model="judge", cache_path=tmp_path / "c.jsonl", grade_pass=1 + ) + assert acompletion.call_count == 2 + + +def _ground_truth(): + return pd.DataFrame( + { + "scenario_id": ["s1", "s1", "s2", "s2"], + "variable": ["snap", "payroll_tax", "snap", "payroll_tax"], + "value": [100.0, 500.0, 0.0, 700.0], + } + ) + + +def _repeated_predictions(): + """Two runs, one model. + + s1/snap: stable + exact, explanations differ (mechanism A vs B). + s1/payroll: stable + exact, verbatim-identical explanations. + s2/snap: stable but wrong (both 50 vs ref 0), explanations differ. + s2/payroll: answer flips (700 vs 900) -> not a stable pair. + """ + rows = [] + for run_id, preds, texts in [ + ( + "run_000", + [100.0, 500.0, 50.0, 700.0], + ["Net income test applied.", "7.65% of wages.", "Asset test failed.", "A"], + ), + ( + "run_001", + [100.0, 500.0, 50.0, 900.0], + ["Gross income under 130% FPL.", "7.65% of wages.", "Gross test.", "B"], + ), + ]: + for (sid, var), pred, text in zip( + [("s1", "snap"), ("s1", "payroll_tax"), ("s2", "snap"), ("s2", "payroll_tax")], + preds, + texts, + strict=True, + ): + rows.append( + { + "run_id": run_id, + "model": "m", + "scenario_id": sid, + "variable": var, + "prediction": pred, + "explanation": text, + } + ) + return pd.DataFrame(rows) + + +class TestExplanationPairFrame: + def test_pairs_carry_strata_and_verbatim_flag(self): + pairs = explanation_pair_frame(_repeated_predictions(), _ground_truth()) + by_row = pairs.set_index(["scenario_id", "variable"]) + assert len(pairs) == 4 + s1_snap = by_row.loc[("s1", "snap")] + assert bool(s1_snap["stable"]) and bool(s1_snap["stable_exact"]) + assert not bool(s1_snap["verbatim_identical"]) + s1_pay = by_row.loc[("s1", "payroll_tax")] + assert bool(s1_pay["stable_exact"]) and bool(s1_pay["verbatim_identical"]) + s2_snap = by_row.loc[("s2", "snap")] + assert bool(s2_snap["stable"]) and not bool(s2_snap["stable_exact"]) + s2_pay = by_row.loc[("s2", "payroll_tax")] + assert not bool(s2_pay["stable"]) + assert "text_key_a" in pairs.columns and "output_group" in pairs.columns + + +def _labels_for(pairs, mapping): + """Build labels_by_text_key from explanation text -> label set.""" + out = {} + for _, row in pairs.iterrows(): + for side in ("a", "b"): + text = row[f"explanation_{side}"] + if text in mapping: + out[row[f"text_key_{side}"]] = frozenset(mapping[text]) + return out + + +class TestReasoningMetrics: + def test_headline_and_companions(self): + pairs = explanation_pair_frame(_repeated_predictions(), _ground_truth()) + labels = _labels_for( + pairs, + { + "Net income test applied.": {"thresholds_rates"}, + "Gross income under 130% FPL.": {"categorical_eligibility", "thresholds_rates"}, + "Asset test failed.": {"asset_resource"}, + "Gross test.": {"thresholds_rates"}, + }, + ) + result = reasoning_stability_by_model(pairs, labels, min_stable_exact_pairs=1) + row = result["summary"].set_index("model").loc["m"] + assert row["n_pairs_total"] == 4 + assert row["n_stable_pairs"] == 3 + assert row["n_stable_exact_pairs"] == 2 + # Stable-exact: s1/snap disagrees, s1/payroll verbatim-identical agrees. + assert row["mechanism_agreement_rate_stable_exact"] == pytest.approx(0.5) + assert row["right_answer_unstable_reasoning_rate"] == pytest.approx(0.5) + # Restricted to non-identical texts: only s1/snap remains -> 1.0. + assert row["right_answer_unstable_reasoning_rate_nonidentical"] == pytest.approx(1.0) + assert row["short_circuit_share_stable"] == pytest.approx(1 / 3) + # Joint rate over ALL pairs: one stable∧exact∧divergent pair of 4. + assert row["joint_unstable_reasoning_rate_all_pairs"] == pytest.approx(0.25) + # Stable stratum (3 pairs): s1/snap differ, s1/payroll same, s2/snap differ. + assert row["mechanism_agreement_rate_stable"] == pytest.approx(1 / 3) + assert row["mechanism_jaccard_mean_stable_exact"] == pytest.approx((0.5 + 1.0) / 2) + assert bool(row["headline_suppressed"]) is False + + def test_suppression_rule(self): + pairs = explanation_pair_frame(_repeated_predictions(), _ground_truth()) + labels = _labels_for(pairs, {}) + result = reasoning_stability_by_model(pairs, labels, min_stable_exact_pairs=200) + row = result["summary"].set_index("model").loc["m"] + assert bool(row["headline_suppressed"]) is True + + def test_judge_errors_excluded_and_counted(self): + pairs = explanation_pair_frame(_repeated_predictions(), _ground_truth()) + # No labels at all: every non-identical stable pair is unjudged. + result = reasoning_stability_by_model(pairs, {}, min_stable_exact_pairs=1) + row = result["summary"].set_index("model").loc["m"] + assert row["n_unjudged_stable_pairs"] == 2 + # Only the verbatim pair is gradeable -> agreement 1.0 on n=1. + assert row["mechanism_agreement_rate_stable_exact"] == pytest.approx(1.0) + + def test_attenuation_adjustment(self): + pairs = explanation_pair_frame(_repeated_predictions(), _ground_truth()) + labels = _labels_for( + pairs, + { + "Net income test applied.": {"thresholds_rates"}, + "Gross income under 130% FPL.": {"categorical_eligibility"}, + }, + ) + result = reasoning_stability_by_model( + pairs, labels, min_stable_exact_pairs=1, pair_noise_floor=0.2 + ) + row = result["summary"].set_index("model").loc["m"] + observed = row["right_answer_unstable_reasoning_rate"] + assert row["right_answer_unstable_reasoning_rate_adjusted"] == pytest.approx( + max(0.0, (observed - 0.2) / 0.8) + ) + + def test_numeric_claim_channel(self): + preds = _repeated_predictions() + preds.loc[ + (preds["scenario_id"] == "s1") & (preds["variable"] == "snap"), "explanation" + ] = ["Deduction of $177 and 30% rate.", "Deduction of $177 and 24% rate."] + pairs = explanation_pair_frame(preds, _ground_truth()) + result = reasoning_stability_by_model(pairs, {}, min_stable_exact_pairs=1) + row = result["summary"].set_index("model").loc["m"] + # Claims {177, 30%} vs {177, 24%}: Jaccard 1/3; shared $177 -> not disjoint. + assert row["numeric_claim_jaccard_mean_stable_exact"] == pytest.approx( + (1 / 3 + 1.0) / 2 # the verbatim payroll pair contributes Jaccard 1 + ) + + def test_composition_table_and_standardized_rate(self): + pairs = explanation_pair_frame(_repeated_predictions(), _ground_truth()) + labels = _labels_for( + pairs, + { + "Net income test applied.": {"thresholds_rates"}, + "Gross income under 130% FPL.": {"categorical_eligibility"}, + }, + ) + result = reasoning_stability_by_model(pairs, labels, min_stable_exact_pairs=1) + composition = result["composition"] + assert set(composition["output_group"]) == {"snap", "payroll_tax"} + row = result["summary"].set_index("model").loc["m"] + # Equal composition (one pair per group): standardized equals the mean + # of per-group rates (1.0 for snap, 0.0 for payroll). + assert row["standardized_unstable_reasoning_rate"] == pytest.approx(0.5) + + +class TestReferenceAlignment: + def test_alignment_among_exact_rows(self): + preds = _repeated_predictions() + pairs = explanation_pair_frame(preds, _ground_truth()) + labels = _labels_for( + pairs, + { + "Net income test applied.": {"thresholds_rates"}, + "Gross income under 130% FPL.": {"categorical_eligibility"}, + "7.65% of wages.": {"payroll_tax_base"}, + "7.65% of wages.": {"payroll_tax_base"}, + }, + ) + reference_labels = { + ("s1", "snap"): frozenset({"categorical_eligibility"}), + ("s1", "payroll_tax"): frozenset({"payroll_tax_base"}), + ("s2", "snap"): frozenset({"thresholds_rates"}), + ("s2", "payroll_tax"): frozenset({"payroll_tax_base"}), + } + table = reference_alignment_by_model(preds, _ground_truth(), labels, reference_labels) + row = table.set_index("model").loc["m"] + # Exact-correct run-explanations: s1/snap x2, s1/payroll x2, s2/payroll run_000. + # Labeled among them: s1/snap (1 of 2 aligned), s1/payroll (2 aligned). + assert row["n_exact_labeled"] == 4 + assert row["reference_alignment_rate"] == pytest.approx(3 / 4) + + +class TestAgreementStatistics: + def test_gwet_ac1_and_kappa_known_values(self): + # 40 items, one disagreement each way at 97.5% prevalence: raw + # agreement 0.95, kappa goes negative (the prevalence paradox), AC1 holds. + a = [True] * 39 + [False] + b = [True] * 38 + [False] + [True] + kappa = cohen_kappa(a, b) + ac1 = gwet_ac1(a, b) + assert kappa < 0.0 + assert ac1 > 0.9 + assert gwet_ac1([True, False], [True, False]) == pytest.approx(1.0) + + def test_kappa_perfect_and_chance(self): + assert cohen_kappa([True, False, True, False], [True, False, True, False]) == pytest.approx(1.0) + assert cohen_kappa([True, True, False, False], [True, False, True, False]) == pytest.approx(0.0) + + def test_krippendorff_alpha_jaccard(self): + perfect = [ + (frozenset({"a"}), frozenset({"a"})), + (frozenset({"b"}), frozenset({"b"})), + (frozenset({"a", "c"}), frozenset({"a", "c"})), + ] + assert krippendorff_alpha_jaccard(perfect) == pytest.approx(1.0) + noisy = [ + (frozenset({"a"}), frozenset({"b"})), + (frozenset({"b"}), frozenset({"a"})), + ] + assert krippendorff_alpha_jaccard(noisy) < 0.0 + + def test_dominant_set_share(self): + sets = [frozenset({"a"}), frozenset({"a"}), frozenset({"b"})] + assert dominant_set_share(sets) == pytest.approx(2 / 3) + assert math.isnan(dominant_set_share([])) + + def test_wilson_ci(self): + low, high = wilson_ci(90, 100) + assert 0.82 < low < 0.90 < high < 0.95 + low, high = wilson_ci(0, 0) + assert math.isnan(low) and math.isnan(high) + + +class TestGoldEvaluation: + def test_exact_set_accuracy_and_per_label(self): + gold = pd.DataFrame( + { + "variable": ["snap", "ssi"], + "explanation": ["Income exceeds 130% FPL.", "Not aged or disabled."], + "labels": ["categorical_eligibility|thresholds_rates", "age_disability"], + } + ) + labels_by_key = { + judge_cache_key("judge", "snap", "Income exceeds 130% FPL.", 0): frozenset( + {"categorical_eligibility", "thresholds_rates"} + ), + judge_cache_key("judge", "ssi", "Not aged or disabled.", 0): frozenset( + {"age_disability", "other"} + ), + } + summary = evaluate_against_gold(gold, labels_by_key, judge_model="judge") + assert summary["n_gold"] == 2 + assert summary["n_graded"] == 2 + assert summary["exact_set_accuracy"] == pytest.approx(0.5) + assert summary["exact_set_accuracy_ci_low"] < 0.5 + per_label = summary["per_label"].set_index("label") + assert per_label.loc["other", "judge_positives"] == 1 + assert per_label.loc["other", "gold_positives"] == 0 + assert per_label.loc["age_disability", "agreement"] == pytest.approx(1.0) From fa477bd10c731ed77e2a25f263c37a12fdbbc621 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Sun, 23 Aug 2026 12:43:53 +0200 Subject: [PATCH 4/7] Stability suite CLI: stability-report, reasoning-stability, counterfactual-manifest/report, stability-cost-plan Command layer in policybench/stability_report.py (testable without argparse): cache guard + runs-metadata fingerprint pooling check + serving-config diff against the board registry for layer 1; judge extraction with reference anchors, cross-judge and determinism validation, gold-set gate with CI-aware status and headline withholding for layer 2; twin manifest with both-arm references, truth-delta summary and frozen-reference drift, delta report with noise floor and provider-fingerprint mismatch counts for layer 3; cost ladder from logged usage with price-override fallback. Co-Authored-By: Claude Fable 5 --- policybench/cli.py | 257 +++++++++- policybench/counterfactual.py | 13 +- policybench/reasoning_stability.py | 77 ++- policybench/stability.py | 24 +- policybench/stability_report.py | 755 +++++++++++++++++++++++++++++ tests/test_counterfactual.py | 15 +- tests/test_reasoning_stability.py | 91 +++- tests/test_stability.py | 71 ++- tests/test_stability_harness.py | 18 +- tests/test_stability_report.py | 508 +++++++++++++++++++ 10 files changed, 1757 insertions(+), 72 deletions(-) create mode 100644 policybench/stability_report.py create mode 100644 tests/test_stability_report.py diff --git a/policybench/cli.py b/policybench/cli.py index 00ccc76..f69e33e 100644 --- a/policybench/cli.py +++ b/policybench/cli.py @@ -11,6 +11,7 @@ DEFAULT_PROGRAM_SET, MODELS, PROGRAMS, + TAX_YEAR, get_programs, ) @@ -774,6 +775,141 @@ def main(): help="Optional directory of repeated-run CSVs for stability analysis", ) + # Stability suite (docs/stability_spec.md). Layer 1: answer stability. + stability_parser = subparsers.add_parser( + "stability-report", + help="Answer-stability export over cache-free repeated runs (spec layer 1)", + ) + stability_parser.add_argument( + "--runs-dir", + action="append", + dest="runs_dirs", + required=True, + help=( + "Directory of run_NNN.csv repeats; repeat the flag for per-provider-" + "group directories (model sets must be disjoint, run ids identical)" + ), + ) + stability_parser.add_argument( + "-g", "--reference-outputs", required=True, help="PolicyEngine reference CSV" + ) + stability_parser.add_argument( + "-s", + "--scenario-manifest", + required=True, + help="Scenario manifest (supplies total_income for headline weights)", + ) + stability_parser.add_argument("-o", "--output-dir", required=True) + stability_parser.add_argument("--n-boot", type=int, default=1000) + stability_parser.add_argument("--seed", type=int, default=20260818) + stability_parser.add_argument( + "--serving-config", + default="paper/snapshot/20260501/model_serving_config.json", + help="Board serving registry to diff effective treatments against", + ) + + # Layer 2: reasoning stability. + reasoning_parser = subparsers.add_parser( + "reasoning-stability", + help="Mechanism-label judge over answer-stable repeat pairs (spec layer 2)", + ) + reasoning_parser.add_argument( + "--runs-dir", action="append", dest="runs_dirs", required=True + ) + reasoning_parser.add_argument("-g", "--reference-outputs", required=True) + reasoning_parser.add_argument( + "--reference-explanations", + default=( + "annotations/us_full_run_20260612_policyengine_4_16_1_populace/" + "us_case_reference_explanations.csv" + ), + help="Reference explanations used as per-row mechanism anchors", + ) + reasoning_parser.add_argument( + "--gold-set", + default="annotations/stability_reasoning_gold_set.csv", + help="Developer-labeled gold set for the judge validity gate", + ) + reasoning_parser.add_argument("--judge-model", default="gemini/gemini-3.7-flash") + reasoning_parser.add_argument("--cross-judge-model", default="gpt-5.4-mini") + reasoning_parser.add_argument( + "--no-cross-judge", action="store_true", help="Skip cross-judge validation" + ) + reasoning_parser.add_argument("--cache-dir", default=None) + reasoning_parser.add_argument( + "--deterministic-only", + action="store_true", + help="Run every deterministic channel and skip the LLM judge", + ) + reasoning_parser.add_argument("--concurrency", type=int, default=8) + reasoning_parser.add_argument("--validation-modulus", type=int, default=10) + reasoning_parser.add_argument("--min-stable-exact-pairs", type=int, default=200) + reasoning_parser.add_argument("--country", default="us") + reasoning_parser.add_argument("--year", type=int, default=TAX_YEAR) + reasoning_parser.add_argument("-o", "--output-dir", required=True) + + # Layer 3: counterfactual consistency. + cf_manifest_parser = subparsers.add_parser( + "counterfactual-manifest", + help="Build +$1,000 head-wage twins and both-arm PolicyEngine references", + ) + cf_manifest_parser.add_argument( + "-s", + "--scenario-manifest", + default="paper/snapshot/20260501/us_scenarios.csv", + ) + cf_manifest_parser.add_argument("-o", "--output-dir", required=True) + cf_manifest_parser.add_argument("--amount", type=float, default=1000.0) + cf_manifest_parser.add_argument( + "--country", choices=sorted(COUNTRY_PROGRAMS), default="us" + ) + cf_manifest_parser.add_argument("--program-set", default=DEFAULT_PROGRAM_SET) + cf_manifest_parser.add_argument("--year", type=int, default=TAX_YEAR) + cf_manifest_parser.add_argument( + "--skip-reference-outputs", + action="store_true", + help="Write the twin manifest only (no PolicyEngine computation)", + ) + cf_manifest_parser.add_argument( + "--frozen-reference-outputs", + default="paper/snapshot/20260501/us_reference_outputs.csv", + help="Frozen references to report base-arm drift against", + ) + + cf_report_parser = subparsers.add_parser( + "counterfactual-report", + help="Compare predicted deltas to PolicyEngine true deltas (spec layer 3)", + ) + cf_report_parser.add_argument("--perturbed-predictions", required=True) + cf_report_parser.add_argument("--truth-deltas", required=True) + cf_report_parser.add_argument( + "--base-predictions", default=None, help="Single base-arm predictions CSV" + ) + cf_report_parser.add_argument( + "--base-runs-dir", + action="append", + dest="base_runs_dirs", + default=None, + help="Layer-1 repeats as the base arm (enables the noise floor)", + ) + cf_report_parser.add_argument("-o", "--output-dir", required=True) + cf_report_parser.add_argument("--n-boot", type=int, default=1000) + cf_report_parser.add_argument("--seed", type=int, default=20260818) + + cost_parser = subparsers.add_parser( + "stability-cost-plan", + help="Price repeats + counterfactual arms + judge from logged usage", + ) + cost_parser.add_argument("-p", "--predictions", required=True) + cost_parser.add_argument("--repeats", type=int, default=3) + cost_parser.add_argument("--cf-arms", type=int, default=1) + cost_parser.add_argument("--model", action="append", dest="models", default=None) + cost_parser.add_argument("--judge-input-tokens", type=int, default=800) + cost_parser.add_argument("--judge-output-tokens", type=int, default=30) + cost_parser.add_argument("--judge-price-in", type=float, default=0.75) + cost_parser.add_argument("--judge-price-out", type=float, default=3.75) + cost_parser.add_argument("-o", "--output", default=None, help="Optional CSV path") + # Defined inline (not via policybench.runstore) so building the parser # never imports pandas; the dispatch below imports lazily like every # other subcommand. @@ -1289,7 +1425,6 @@ def main(): ) elif args.command == "population-weights": - from policybench.config import TAX_YEAR from policybench.population_weights import write_population_weight_payload year = TAX_YEAR if args.year is None else args.year @@ -1431,6 +1566,126 @@ def main(): for name, path in exported.items(): print(f"{name}: {path}") + elif args.command == "stability-report": + from policybench.stability_report import run_stability_report + + try: + result = run_stability_report( + runs_dirs=args.runs_dirs, + reference_outputs=args.reference_outputs, + scenario_manifest=args.scenario_manifest, + output_dir=args.output_dir, + n_boot=args.n_boot, + seed=args.seed, + serving_config=args.serving_config, + ) + except (ValueError, RuntimeError) as exc: + raise SystemExit(str(exc)) from exc + print(f"Stability report written to {args.output_dir}") + pooled = result["pooled"] + if pooled: + print( + "Pooled run-to-sampling ratio: " + f"{pooled['pooled_run_to_sampling_ratio']:.3f} " + f"(sampling dominates: {pooled['pooled_sampling_dominates']})" + ) + + elif args.command == "reasoning-stability": + from policybench.stability_report import run_reasoning_stability + + try: + result = run_reasoning_stability( + runs_dirs=args.runs_dirs, + reference_outputs=args.reference_outputs, + output_dir=args.output_dir, + reference_explanations=args.reference_explanations, + gold_set=args.gold_set, + judge_model=args.judge_model, + cross_judge_model=None + if args.no_cross_judge + else args.cross_judge_model, + cache_dir=args.cache_dir, + deterministic_only=args.deterministic_only, + concurrency=args.concurrency, + validation_modulus=args.validation_modulus, + min_stable_exact_pairs=args.min_stable_exact_pairs, + country=args.country, + year=args.year, + ) + except (ValueError, RuntimeError) as exc: + raise SystemExit(str(exc)) from exc + print(f"Reasoning-stability report written to {args.output_dir}") + validation = result["validation"] + if validation: + gates = { + key: value.get("status") + for key, value in validation.items() + if key.endswith("_gate") + } + print(f"Judge gates: {gates}") + + elif args.command == "counterfactual-manifest": + from policybench.stability_report import run_counterfactual_manifest + + try: + metadata = run_counterfactual_manifest( + scenario_manifest=args.scenario_manifest, + output_dir=args.output_dir, + programs=get_programs(args.country, args.program_set), + year=args.year, + amount=args.amount, + compute_references=not args.skip_reference_outputs, + frozen_reference_outputs=args.frozen_reference_outputs, + ) + except (ValueError, RuntimeError) as exc: + raise SystemExit(str(exc)) from exc + print(f"Counterfactual manifest written to {metadata['cf_scenarios']}") + summary = metadata.get("truth_delta_summary") + if summary: + print( + f"True deltas: {summary['n_nonzero_amount_rows']} nonzero amount rows, " + f"{summary['n_binary_flips']} binary flips, " + f"zero-delta share {summary['zero_delta_share']:.4f}" + ) + + elif args.command == "counterfactual-report": + from policybench.stability_report import run_counterfactual_report + + try: + run_counterfactual_report( + perturbed_predictions=args.perturbed_predictions, + truth_deltas=args.truth_deltas, + output_dir=args.output_dir, + base_predictions=args.base_predictions, + base_runs_dirs=args.base_runs_dirs, + n_boot=args.n_boot, + seed=args.seed, + ) + except (ValueError, RuntimeError) as exc: + raise SystemExit(str(exc)) from exc + print(f"Counterfactual report written to {args.output_dir}") + + elif args.command == "stability-cost-plan": + import pandas as pd + + from policybench.stability_report import stability_cost_plan + + plan = stability_cost_plan( + pd.read_csv(args.predictions, low_memory=False), + repeats=args.repeats, + cf_arms=args.cf_arms, + models=args.models, + judge_input_tokens=args.judge_input_tokens, + judge_output_tokens=args.judge_output_tokens, + judge_price_in_per_1m=args.judge_price_in, + judge_price_out_per_1m=args.judge_price_out, + ) + with pd.option_context("display.max_rows", 200, "display.width", 200): + print(plan.round(3).to_string(index=False)) + if args.output: + plan.to_csv(args.output, index=False) + print(f"Cost plan written to {args.output}") + elif args.command == "runstore": from policybench.runstore import run_runstore_command diff --git a/policybench/counterfactual.py b/policybench/counterfactual.py index f63a918..a14fd33 100644 --- a/policybench/counterfactual.py +++ b/policybench/counterfactual.py @@ -118,9 +118,7 @@ def compute_truth_deltas( "variable": merged["variable"], "base_value": merged["value_base"], "perturbed_value": merged["value_perturbed"], - "true_delta": ( - (merged["value_perturbed"] - merged["value_base"]).round(2) - ), + "true_delta": ((merged["value_perturbed"] - merged["value_base"]).round(2)), "is_binary": merged["variable"].map(_is_binary_variable), "output_group": merged["variable"].map(output_group_id), "first_dollar": merged["scenario_id"].map(first_dollar).fillna(False), @@ -303,8 +301,8 @@ def delta_metrics_by_model( amount = matched[~matched["is_binary"] & matched["both_parsed"]].copy() amount["nonzero_true"] = amount["true_delta"].abs() > DELTA_EXACT_TOLERANCE amount["exact"] = ( - (amount["pred_delta"] - amount["true_delta"]).abs() <= DELTA_EXACT_TOLERANCE - ) + amount["pred_delta"] - amount["true_delta"] + ).abs() <= DELTA_EXACT_TOLERANCE amount["within_band"] = [ _delta_within_band(p, t) for p, t in zip(amount["pred_delta"], amount["true_delta"], strict=True) @@ -545,10 +543,7 @@ def signal_vs_noise_test( [signal_by_scenario[scenarios[i]] for i in draw] ) floor_values = np.concatenate( - [ - floor_by_scenario.get(scenarios[i], np.array([])) - for i in draw - ] + [floor_by_scenario.get(scenarios[i], np.array([])) for i in draw] ) if not len(floor_values): continue diff --git a/policybench/reasoning_stability.py b/policybench/reasoning_stability.py index 74c7cb6..54b55e9 100644 --- a/policybench/reasoning_stability.py +++ b/policybench/reasoning_stability.py @@ -85,16 +85,14 @@ "employer-sponsored insurance, Medicaid/CHIP/Medicare coverage logic" ), "age_disability": ( - "age limits, disability or blindness status, Medicare age, " - "child-age windows" + "age limits, disability or blindness status, Medicare age, child-age windows" ), "period_annualization": ( "converting between monthly, weekly, and annual amounts or " "certification periods" ), "payroll_tax_base": ( - "payroll or self-employment tax: Social Security, Medicare, FICA, " - "wage bases" + "payroll or self-employment tax: Social Security, Medicare, FICA, wage bases" ), "state_local_rule": ( "a state or local rule, parameter, supplement, conformity election, " @@ -165,7 +163,11 @@ "per person rather than as a pooled household total, the absence of " "any qualifying member results in zero SSI benefits for the " "household.", - ("categorical_eligibility", "age_disability", "household_unit_or_filing_status"), + ( + "categorical_eligibility", + "age_disability", + "household_unit_or_filing_status", + ), ), ) @@ -232,7 +234,9 @@ def judge_cache_key(judge_model: str, variable: str, text, grade_pass: int = 0) return hashlib.sha256(payload.encode("utf-8")).hexdigest() -def build_judge_prompt(text: str, variable: str, country: str = "us", year: int = 2026) -> str: +def build_judge_prompt( + text: str, variable: str, country: str = "us", year: int = 2026 +) -> str: glossary = "\n".join( f"- {label}: {definition}" for label, definition in LABEL_GLOSSARY.items() ) @@ -294,7 +298,10 @@ async def _extract_one( item: dict, ) -> dict: prompt = build_judge_prompt( - item["text"], item["variable"], item.get("country", "us"), item.get("year", 2026) + item["text"], + item["variable"], + item.get("country", "us"), + item.get("year", 2026), ) last_error = "" async with semaphore: @@ -320,7 +327,9 @@ async def _extract_one( return {"labels": None, "error": last_error or "invalid judge response", "raw": ""} -async def _extract_batch(items: list[dict], judge_model: str, concurrency: int) -> list[dict]: +async def _extract_batch( + items: list[dict], judge_model: str, concurrency: int +) -> list[dict]: semaphore = asyncio.Semaphore(max(1, concurrency)) return await asyncio.gather( *(_extract_one(semaphore, judge_model, item) for item in items) @@ -420,8 +429,12 @@ def explanation_pair_frame( } records = [] for row in pairs.itertuples(): - pred_a, text_a = lookup.get((row.model, row.run_a, row.scenario_id, row.variable), (None, None)) - pred_b, text_b = lookup.get((row.model, row.run_b, row.scenario_id, row.variable), (None, None)) + pred_a, text_a = lookup.get( + (row.model, row.run_a, row.scenario_id, row.variable), (None, None) + ) + pred_b, text_b = lookup.get( + (row.model, row.run_b, row.scenario_id, row.variable), (None, None) + ) norm_a = normalize_explanation_text(text_a) norm_b = normalize_explanation_text(text_b) stable = bool(row.both_parsed and row.mutually_exact) @@ -478,7 +491,9 @@ def reasoning_stability_by_model( frame = pairs.copy() labels_a = frame["text_key_a"].map(labels_by_text_key) labels_b = frame["text_key_b"].map(labels_by_text_key) - frame["judged"] = frame["verbatim_identical"] | (labels_a.notna() & labels_b.notna()) + frame["judged"] = frame["verbatim_identical"] | ( + labels_a.notna() & labels_b.notna() + ) same = [] jaccard = [] claim_jaccard = [] @@ -543,10 +558,14 @@ def reasoning_stability_by_model( nonidentical = _rate(stable_exact & judged & ~verbatim, ~group["same_labels"]) adjusted = float("nan") if pair_noise_floor is not None and not math.isnan(headline): - adjusted = max(0.0, (headline - pair_noise_floor) / (1.0 - pair_noise_floor)) + adjusted = max( + 0.0, (headline - pair_noise_floor) / (1.0 - pair_noise_floor) + ) standardized = float("nan") if not composition.empty: - model_comp = composition[composition["model"] == model].set_index("output_group") + model_comp = composition[composition["model"] == model].set_index( + "output_group" + ) if not model_comp.empty: weights = standard_weights.reindex(model_comp.index).fillna(0.0) if weights.sum() > 0: @@ -563,7 +582,9 @@ def reasoning_stability_by_model( "n_judged_stable_exact_pairs": n_judged_se, "n_unjudged_stable_pairs": int((stable & ~judged).sum()), "short_circuit_share_stable": _rate(stable, verbatim), - "mechanism_agreement_rate_stable": _rate(stable & judged, group["same_labels"]), + "mechanism_agreement_rate_stable": _rate( + stable & judged, group["same_labels"] + ), "mechanism_agreement_rate_stable_exact": agreement_se, "right_answer_unstable_reasoning_rate": headline, "right_answer_unstable_reasoning_rate_nonidentical": nonidentical, @@ -572,7 +593,9 @@ def reasoning_stability_by_model( (stable_exact & group["divergent"]).sum() / len(group) ), "standardized_unstable_reasoning_rate": standardized, - "mechanism_jaccard_mean_stable": _rate(stable & judged, group["label_jaccard"]), + "mechanism_jaccard_mean_stable": _rate( + stable & judged, group["label_jaccard"] + ), "mechanism_jaccard_mean_stable_exact": _rate( stable_exact & judged, group["label_jaccard"] ), @@ -583,7 +606,8 @@ def reasoning_stability_by_model( stable_exact, group["claim_disjoint"] ), "reasoning_unstable_strict_rate": _rate( - stable_exact & judged, ~group["same_labels"] | group["claim_disjoint"] + stable_exact & judged, + ~group["same_labels"] | group["claim_disjoint"], ), "headline_suppressed": n_judged_se < min_stable_exact_pairs, } @@ -738,7 +762,14 @@ def evaluate_against_gold( "gold_positives": int(sum(gold_flags)), "judge_positives": int(sum(judge_flags)), "agreement": ( - float(np.mean([g == j for g, j in zip(gold_flags, judge_flags, strict=True)])) + float( + np.mean( + [ + g == j + for g, j in zip(gold_flags, judge_flags, strict=True) + ] + ) + ) if n_graded else float("nan") ), @@ -759,7 +790,9 @@ def evaluate_against_gold( } -def validation_sample_keys(text_keys: list[str], modulus: int = VALIDATION_SAMPLE_MODULUS) -> list[str]: +def validation_sample_keys( + text_keys: list[str], modulus: int = VALIDATION_SAMPLE_MODULUS +) -> list[str]: """Deterministic ~1/modulus sample of unique texts for re-grading.""" return [key for key in sorted(set(text_keys)) if int(key[:8], 16) % modulus == 0] @@ -786,9 +819,13 @@ def pair_agreement_summary( { "label": label, "prevalence": occurrences / (2 * n), - "raw_agreement": float(np.mean([x == y for x, y in zip(a_flags, b_flags, strict=True)])), + "raw_agreement": float( + np.mean([x == y for x, y in zip(a_flags, b_flags, strict=True)]) + ), "gwet_ac1": gwet_ac1(a_flags, b_flags), - "cohen_kappa": cohen_kappa(a_flags, b_flags) if occurrences >= 20 else float("nan"), + "cohen_kappa": cohen_kappa(a_flags, b_flags) + if occurrences >= 20 + else float("nan"), } ) return { diff --git a/policybench/stability.py b/policybench/stability.py index fe6eeef..8fbb7fc 100644 --- a/policybench/stability.py +++ b/policybench/stability.py @@ -289,8 +289,7 @@ def _row_level_frame( if binary_flag(p) is None: numeric_non_flag = True all_pairwise_exact = all( - mutually_exact(variable, a, b) - for a, b in combinations(predictions, 2) + mutually_exact(variable, a, b) for a, b in combinations(predictions, 2) ) exact_flags = [ bool(row_hit_scores(variable, gt_row.value, p)["exact"]) @@ -323,8 +322,9 @@ def _scenario_aggregates( pair = pair_frame.assign( flip_num=lambda df: df["both_parsed"] & ~df["mutually_exact"], flip_den=lambda df: df["both_parsed"], - verdict_parsed_num=lambda df: df["both_parsed"] - & (df["exact_a"] != df["exact_b"]), + verdict_parsed_num=lambda df: ( + df["both_parsed"] & (df["exact_a"] != df["exact_b"]) + ), verdict_parsed_den=lambda df: df["both_parsed"], verdict_all_num=lambda df: df["exact_a"] != df["exact_b"], verdict_all_den=1, @@ -440,7 +440,9 @@ def row_stability_by_model( num_sum = wide[num].values[draw].sum(axis=0) den_sum = wide[den].values[draw].sum(axis=0) with np.errstate(invalid="ignore", divide="ignore"): - boot_values[metric].append(np.where(den_sum > 0, num_sum / den_sum, np.nan)) + boot_values[metric].append( + np.where(den_sum > 0, num_sum / den_sum, np.nan) + ) run_ids = sorted(repeated_predictions["run_id"].dropna().unique()) result_rows = [] @@ -521,7 +523,13 @@ def stability_variance_decomposition( household_scores.groupby(["model", "run_id"])["score"] .agg(["mean", "std", "count"]) .reset_index() - .rename(columns={"mean": "run_score", "std": "household_std", "count": "n_households"}) + .rename( + columns={ + "mean": "run_score", + "std": "household_std", + "count": "n_households", + } + ) ) run_level["run_sampling_se"] = run_level["household_std"] / np.sqrt( run_level["n_households"] @@ -572,7 +580,9 @@ def stability_variance_decomposition( pooled: dict = {} if pooled_df > 0: pooled_std = math.sqrt(pooled_ss / pooled_df) - mean_sampling_se = float(np.mean(sampling_ses)) if sampling_ses else float("nan") + mean_sampling_se = ( + float(np.mean(sampling_ses)) if sampling_ses else float("nan") + ) pooled_ci_low = pooled_std * math.sqrt(pooled_df / chi2_ppf(0.95, pooled_df)) pooled_ci_high = pooled_std * math.sqrt(pooled_df / chi2_ppf(0.05, pooled_df)) total_ss = pooled_ss if pooled_ss > 0 else float("nan") diff --git a/policybench/stability_report.py b/policybench/stability_report.py new file mode 100644 index 0000000..13265c6 --- /dev/null +++ b/policybench/stability_report.py @@ -0,0 +1,755 @@ +"""Command implementations for the stability suite (docs/stability_spec.md). + +``policybench stability-report``, ``reasoning-stability``, +``counterfactual-manifest``, ``counterfactual-report`` and +``stability-cost-plan`` dispatch here. Each export directory carries a +``stability_metadata.json`` with the provenance the spec requires. +""" + +from __future__ import annotations + +import json +from datetime import datetime, timezone +from importlib import metadata as importlib_metadata +from pathlib import Path + +import pandas as pd + +from policybench.analysis import ( + run_stability_by_model, + summarize_runs_by_model, + usage_summary_by_model, +) +from policybench.config import MODELS, PRICE_OVERRIDES_PER_1M +from policybench.counterfactual import ( + CF_DEFAULT_AMOUNT, + build_counterfactual_manifest, + compute_truth_deltas, + delta_metrics_by_model, + matched_delta_frame, + noise_floor_frame, + signal_vs_noise_test, +) +from policybench.model_cards import ( + PROMPT_CONTRACT_VERSION, + answer_contract_for, + explanation_chunk_size_for, +) +from policybench.reasoning_stability import ( + DEFAULT_CROSS_JUDGE_MODEL, + DEFAULT_JUDGE_MODEL, + JUDGE_PROMPT_VERSION, + MIN_STABLE_EXACT_PAIRS, + dominant_set_share, + evaluate_against_gold, + explanation_pair_frame, + labels_by_text_key_from_results, + pair_agreement_summary, + reasoning_stability_by_model, + reference_alignment_by_model, + run_label_extraction, + text_key, + validation_sample_keys, +) +from policybench.stability import ( + STABILITY_SPEC_VERSION, + assert_cache_free, + cross_run_response_id_duplicates, + load_runs_dirs, + row_stability_by_model, + stability_variance_decomposition, +) + +DEFAULT_SERVING_CONFIG = Path("paper/snapshot/20260501/model_serving_config.json") +DEFAULT_GOLD_SET = Path("annotations/stability_reasoning_gold_set.csv") + +GOLD_ACCURACY_GATE = 0.80 +GOLD_ACCURACY_CI_FLOOR = 0.70 +CROSS_JUDGE_AGREEMENT_GATE = 0.90 + + +def _now() -> str: + return datetime.now(timezone.utc).isoformat() + + +def _installed_version(package: str) -> str | None: + try: + return importlib_metadata.version(package) + except importlib_metadata.PackageNotFoundError: + return None + + +def _json_safe(value): + if isinstance(value, dict): + return {str(k): _json_safe(v) for k, v in value.items()} + if isinstance(value, (list, tuple)): + return [_json_safe(v) for v in value] + if isinstance(value, pd.DataFrame): + return _json_safe(value.to_dict(orient="records")) + if isinstance(value, float) and value != value: # NaN + return None + if hasattr(value, "item"): + try: + return _json_safe(value.item()) + except (ValueError, TypeError): + return str(value) + return value + + +def write_metadata(output_dir: Path, payload: dict) -> Path: + output_dir.mkdir(parents=True, exist_ok=True) + path = output_dir / "stability_metadata.json" + base = { + "stability_spec_version": STABILITY_SPEC_VERSION, + "prompt_contract_version": PROMPT_CONTRACT_VERSION, + "generated_at": _now(), + "installed_policyengine_us": _installed_version("policyengine-us"), + "installed_policyengine": _installed_version("policyengine"), + } + base.update(payload) + path.write_text( + json.dumps(_json_safe(base), indent=2, sort_keys=True), encoding="utf-8" + ) + return path + + +def _write_tables(output_dir: Path, tables: dict[str, pd.DataFrame]) -> dict[str, str]: + output_dir.mkdir(parents=True, exist_ok=True) + written = {} + for name, frame in tables.items(): + if frame is None or (isinstance(frame, pd.DataFrame) and frame.empty): + continue + path = output_dir / f"{name}.csv" + frame.to_csv(path, index=False) + written[name] = str(path) + return written + + +def read_runs_metadata(runs_dirs: list[str | Path]) -> dict: + """Load runs_metadata.json sidecars and check they agree on the fingerprint.""" + sidecars = {} + for runs_dir in runs_dirs: + path = Path(runs_dir) / "runs_metadata.json" + sidecars[str(runs_dir)] = ( + json.loads(path.read_text(encoding="utf-8")) if path.exists() else None + ) + present = {k: v for k, v in sidecars.items() if v is not None} + missing = [k for k, v in sidecars.items() if v is None] + mismatches = [] + if len(present) > 1: + reference_key, reference = next(iter(present.items())) + for key, value in present.items(): + for field in ("scenario_hash", "programs", "response_contract", "repeats"): + if value.get(field) != reference.get(field): + mismatches.append({"field": field, "a": reference_key, "b": key}) + if mismatches: + raise ValueError( + "runs directories disagree on repeat-set fingerprints; refusing to " + f"pool: {mismatches}" + ) + condition = None + if present: + contract = next(iter(present.values())).get("response_contract") or {} + condition = { + "tool_choice_condition": contract.get("tool_choice_condition"), + "chunk_override": contract.get("chunk_override"), + } + return { + "sidecars_present": sorted(present), + "sidecars_missing": missing, + "condition": condition, + "cache_enabled_flags": {k: v.get("cache_enabled") for k, v in present.items()}, + } + + +def serving_config_diff( + models: list[str], + condition: dict | None, + serving_config_path: str | Path | None, +) -> dict: + """Diff each model's effective serving treatment against the board registry.""" + registry = {} + path = Path(serving_config_path) if serving_config_path else None + if path is not None and path.exists(): + registry = (json.loads(path.read_text(encoding="utf-8")) or {}).get( + "models", {} + ) + chunk_override = (condition or {}).get("chunk_override") + tool_choice_condition = (condition or {}).get("tool_choice_condition") or "forced" + out = {} + for model in models: + litellm_id = MODELS.get(model, model) + contract = answer_contract_for(litellm_id) + chunk = explanation_chunk_size_for(litellm_id, chunk_override=chunk_override) + effective = { + "answer_contract": contract, + "request_shape": ( + f"{chunk} output{'s' if chunk != 1 else ''}/request" + if chunk + else "whole scenario" + ), + "tool_choice": tool_choice_condition if contract == "tool" else None, + } + snapshot = registry.get(model) + diffs = {} + if snapshot: + for field, value in effective.items(): + if snapshot.get(field) != value: + diffs[field] = {"effective": value, "snapshot": snapshot.get(field)} + out[model] = { + "effective": effective, + "in_snapshot_registry": snapshot is not None, + "diffs": diffs, + } + return out + + +def _market_income(manifest: pd.DataFrame) -> dict[str, float]: + if manifest is None or manifest.empty or "total_income" not in manifest.columns: + return {} + return dict( + zip( + manifest["scenario_id"].astype(str), + pd.to_numeric(manifest["total_income"], errors="coerce").fillna(0.0), + strict=True, + ) + ) + + +def _country(manifest: pd.DataFrame) -> str | None: + if manifest is None or manifest.empty or "country" not in manifest.columns: + return None + values = manifest["country"].dropna().unique() + return str(values[0]).lower() if len(values) == 1 else None + + +def run_stability_report( + *, + runs_dirs: list[str | Path], + reference_outputs: str | Path, + scenario_manifest: str | Path, + output_dir: str | Path, + n_boot: int = 1000, + seed: int = 20260818, + serving_config: str | Path | None = DEFAULT_SERVING_CONFIG, +) -> dict: + """Layer 1: answer-stability export with the cache guard enforced.""" + output_path = Path(output_dir) + cache_report = assert_cache_free(runs_dirs) + runs_meta = read_runs_metadata(runs_dirs) + repeated = load_runs_dirs(runs_dirs) + ground_truth = pd.read_csv(reference_outputs) + manifest = pd.read_csv(scenario_manifest) + market_income = _market_income(manifest) + country = _country(manifest) + + run_model_summary = summarize_runs_by_model(ground_truth, repeated) + tables = { + "run_model_summary": run_model_summary, + "run_stability_by_model": run_stability_by_model(run_model_summary), + "row_stability_by_model": row_stability_by_model( + repeated, ground_truth, n_boot=n_boot, seed=seed + ), + } + per_model, pooled = stability_variance_decomposition( + ground_truth, repeated, market_income, country=country + ) + tables["variance_decomposition_by_model"] = per_model + models = sorted(repeated["model"].dropna().unique()) + written = _write_tables(output_path, tables) + metadata = { + "layer": "answer_stability", + "runs_dirs": [str(d) for d in runs_dirs], + "models": models, + "run_ids": sorted(repeated["run_id"].dropna().unique()), + "reference_outputs": str(reference_outputs), + "scenario_manifest": str(scenario_manifest), + "cache_guard": cache_report, + "cross_run_response_id_duplicates": cross_run_response_id_duplicates(repeated), + "runs_metadata": runs_meta, + "serving_config_diff": serving_config_diff( + models, runs_meta.get("condition"), serving_config + ), + "pooled_variance_decomposition": pooled, + "bootstrap": {"n_boot": n_boot, "seed": seed}, + "tables": written, + } + write_metadata(output_path, metadata) + return {"tables": tables, "pooled": pooled, "metadata": metadata} + + +def run_counterfactual_manifest( + *, + scenario_manifest: str | Path, + output_dir: str | Path, + programs: list[str], + year: int, + amount: float = CF_DEFAULT_AMOUNT, + compute_references: bool = True, + frozen_reference_outputs: str | Path | None = None, +) -> dict: + """Layer 3 setup: twin manifest, both-arm references, truth deltas.""" + from policybench.scenarios import load_scenarios_from_manifest + + output_path = Path(output_dir) + output_path.mkdir(parents=True, exist_ok=True) + base = load_scenarios_from_manifest(scenario_manifest) + twins, manifest = build_counterfactual_manifest(base, amount) + manifest_path = output_path / "cf_scenarios.csv" + manifest.to_csv(manifest_path, index=False) + metadata: dict = { + "layer": "counterfactual", + "scenario_manifest": str(scenario_manifest), + "perturbation": { + "field": "employment_income", + "person": "adults[0]", + "amount": float(amount), + "n_scenarios": len(base), + "n_first_dollar": int(manifest["first_dollar"].sum()), + }, + "cf_scenarios": str(manifest_path), + "programs": list(programs), + "year": year, + } + if compute_references: + deltas = compute_truth_deltas(base, twins, programs, year) + deltas_path = output_path / "truth_deltas.csv" + deltas.to_csv(deltas_path, index=False) + amount_rows = deltas[~deltas["is_binary"]] + nonzero = amount_rows[amount_rows["true_delta"].abs() > 1.0] + metadata["truth_deltas"] = str(deltas_path) + metadata["truth_delta_summary"] = { + "n_rows": int(len(deltas)), + "n_binary_rows": int(deltas["is_binary"].sum()), + "n_binary_flips": int( + (deltas["is_binary"] & (deltas["true_delta"].abs() > 0)).sum() + ), + "n_amount_rows": int(len(amount_rows)), + "n_nonzero_amount_rows": int(len(nonzero)), + "zero_delta_share": float((deltas["true_delta"].abs() <= 1.0).mean()), + "n_positive": int((nonzero["true_delta"] > 0).sum()), + "n_negative": int((nonzero["true_delta"] < 0).sum()), + "nonzero_by_group": nonzero.groupby("output_group").size().to_dict(), + } + if frozen_reference_outputs and Path(frozen_reference_outputs).exists(): + frozen = pd.read_csv(frozen_reference_outputs) + merged = deltas.merge(frozen, on=["scenario_id", "variable"], how="inner") + drift = (merged["base_value"] - merged["value"]).abs() + metadata["base_vs_frozen_reference_drift"] = { + "frozen_reference_outputs": str(frozen_reference_outputs), + "n_compared": int(len(merged)), + "n_within_1usd": int((drift <= 1.0).sum()), + "max_abs_drift": float(drift.max()) if len(drift) else None, + } + write_metadata(output_path, metadata) + return metadata + + +def _fingerprint_summary(frame: pd.DataFrame) -> dict: + out = {} + for column in ("provider_resolved_model", "provider_system_fingerprint"): + if column in frame.columns: + out[column] = ( + frame.dropna(subset=[column]) + .groupby("model")[column] + .nunique() + .to_dict() + ) + return out + + +def run_counterfactual_report( + *, + perturbed_predictions: str | Path, + truth_deltas: str | Path, + output_dir: str | Path, + base_predictions: str | Path | None = None, + base_runs_dirs: list[str | Path] | None = None, + n_boot: int = 1000, + seed: int = 20260818, +) -> dict: + """Layer 3 analysis: delta metrics, baseline, noise floor, fingerprints.""" + output_path = Path(output_dir) + if base_runs_dirs: + cache_report = assert_cache_free(base_runs_dirs) + base = load_runs_dirs(base_runs_dirs) + elif base_predictions: + cache_report = None + base = pd.read_csv(base_predictions) + else: + raise ValueError("provide base_predictions or base_runs_dirs") + perturbed = pd.read_csv(perturbed_predictions) + truth = pd.read_csv(truth_deltas) + matched = matched_delta_frame(base, perturbed, truth) + tables = delta_metrics_by_model(matched) + tables = {f"delta_{name}": frame for name, frame in tables.items()} + metadata: dict = { + "layer": "counterfactual", + "perturbed_predictions": str(perturbed_predictions), + "truth_deltas": str(truth_deltas), + "base_predictions": str(base_predictions) if base_predictions else None, + "base_runs_dirs": [str(d) for d in (base_runs_dirs or [])], + "cache_guard": cache_report, + "n_matched_rows": int(len(matched)), + "provider_fingerprints": { + "base": _fingerprint_summary(base), + "perturbed": _fingerprint_summary(perturbed), + }, + } + if "provider_system_fingerprint" in base.columns and ( + "provider_system_fingerprint" in perturbed.columns + ): + pert = perturbed.copy() + pert["scenario_id"] = pert["scenario_id"].str.replace("__cf1k", "", regex=False) + joined = base.merge( + pert[["model", "scenario_id", "variable", "provider_system_fingerprint"]], + on=["model", "scenario_id", "variable"], + suffixes=("_base", "_perturbed"), + ).dropna( + subset=[ + "provider_system_fingerprint_base", + "provider_system_fingerprint_perturbed", + ] + ) + metadata["fingerprint_mismatched_pairs"] = int( + ( + joined["provider_system_fingerprint_base"] + != joined["provider_system_fingerprint_perturbed"] + ).sum() + ) + if "run_id" in base.columns and base["run_id"].nunique() >= 2: + floor = noise_floor_frame(base, truth) + floor_tables = delta_metrics_by_model(floor) + tables["noise_floor_summary"] = floor_tables["summary"] + tables["signal_vs_noise"] = signal_vs_noise_test( + matched, floor, n_boot=n_boot, seed=seed + ) + written = _write_tables(output_path, tables) + metadata["tables"] = written + write_metadata(output_path, metadata) + return {"tables": tables, "metadata": metadata} + + +def _gate( + point: float, ci_low: float, threshold: float, ci_floor: float | None = None +) -> dict: + if point != point: # NaN + return {"status": "not_evaluated", "point": None, "threshold": threshold} + if point < threshold or (ci_floor is not None and ci_low < ci_floor): + status = "fail" + elif ci_low < threshold: + status = "pass_marginal" + else: + status = "pass" + return {"status": status, "point": point, "ci_low": ci_low, "threshold": threshold} + + +def run_reasoning_stability( + *, + runs_dirs: list[str | Path], + reference_outputs: str | Path, + output_dir: str | Path, + reference_explanations: str | Path | None = None, + gold_set: str | Path | None = DEFAULT_GOLD_SET, + judge_model: str = DEFAULT_JUDGE_MODEL, + cross_judge_model: str | None = DEFAULT_CROSS_JUDGE_MODEL, + cache_dir: str | Path | None = None, + deterministic_only: bool = False, + concurrency: int = 8, + validation_modulus: int = 10, + min_stable_exact_pairs: int = MIN_STABLE_EXACT_PAIRS, + country: str = "us", + year: int = 2026, +) -> dict: + """Layer 2: pair construction, judge extraction, validation, metrics.""" + output_path = Path(output_dir) + cache_path = Path(cache_dir) if cache_dir else output_path / "judge_cache" + cache_report = assert_cache_free(runs_dirs) + runs_meta = read_runs_metadata(runs_dirs) + repeated = load_runs_dirs(runs_dirs) + ground_truth = pd.read_csv(reference_outputs) + pairs = explanation_pair_frame(repeated, ground_truth) + + metadata: dict = { + "layer": "reasoning_stability", + "runs_dirs": [str(d) for d in runs_dirs], + "cache_guard": cache_report, + "runs_metadata": runs_meta, + "judge": { + "prompt_version": JUDGE_PROMPT_VERSION, + "judge_model": None if deterministic_only else judge_model, + "cross_judge_model": None if deterministic_only else cross_judge_model, + "deterministic_only": deterministic_only, + }, + "min_stable_exact_pairs": min_stable_exact_pairs, + } + labels: dict[str, frozenset] = {} + reference_labels: dict[tuple[str, str], frozenset] = {} + pair_noise_floor = None + validation: dict = {} + + if not deterministic_only: + explanation_items = [] + seen = set() + if "explanation" in repeated.columns: + for row in repeated.dropna(subset=["explanation"]).itertuples(): + key = text_key(row.variable, row.explanation) + if key in seen: + continue + seen.add(key) + explanation_items.append( + { + "variable": row.variable, + "country": country, + "year": year, + "text": row.explanation, + } + ) + primary = run_label_extraction( + explanation_items, judge_model, cache_path / "primary.jsonl", concurrency + ) + labels = labels_by_text_key_from_results(primary) + metadata["judge"]["n_unique_texts"] = len(explanation_items) + metadata["judge"]["n_judge_errors"] = sum( + 1 for r in primary.values() if r.get("labels") is None + ) + + if reference_explanations and Path(reference_explanations).exists(): + ref = pd.read_csv(reference_explanations) + ref = ref[ref["explanation"].fillna("").str.strip() != ""] + ref_items = [ + { + "variable": r.variable, + "country": country, + "year": year, + "text": r.explanation, + } + for r in ref.itertuples() + ] + ref_results = run_label_extraction( + ref_items, judge_model, cache_path / "reference.jsonl", concurrency + ) + ref_labels = labels_by_text_key_from_results(ref_results) + for r in ref.itertuples(): + value = ref_labels.get(text_key(r.variable, r.explanation)) + if value is not None: + reference_labels[(r.scenario_id, r.variable)] = value + metadata["judge"]["n_reference_anchors"] = len(reference_labels) + + # Validation battery. + sample_keys = set( + validation_sample_keys( + [text_key(i["variable"], i["text"]) for i in explanation_items], + modulus=validation_modulus, + ) + ) + sample_items = [ + i + for i in explanation_items + if text_key(i["variable"], i["text"]) in sample_keys + ] + primary_sample = { + r["text_key"]: frozenset(r["labels"]) + for r in primary.values() + if r.get("labels") and r["text_key"] in sample_keys + } + if cross_judge_model and sample_items: + cross = run_label_extraction( + sample_items, cross_judge_model, cache_path / "cross.jsonl", concurrency + ) + cross_summary = pair_agreement_summary( + primary_sample, labels_by_text_key_from_results(cross) + ) + pair_noise_floor = cross_summary.get("pair_noise_floor") + validation["cross_judge"] = cross_summary + validation["cross_judge_gate"] = _gate( + cross_summary.get("set_agreement", float("nan")), + cross_summary.get("set_agreement_ci_low", float("nan")), + CROSS_JUDGE_AGREEMENT_GATE, + ) + if sample_items: + repeat = run_label_extraction( + sample_items, + judge_model, + cache_path / "primary_pass1.jsonl", + concurrency, + grade_pass=1, + ) + validation["determinism_floor"] = pair_agreement_summary( + primary_sample, labels_by_text_key_from_results(repeat) + ) + if gold_set and Path(gold_set).exists(): + gold = pd.read_csv(gold_set) + gold_items = [ + { + "variable": r.variable, + "country": country, + "year": year, + "text": r.explanation, + } + for r in gold.itertuples() + ] + gold_results = run_label_extraction( + gold_items, judge_model, cache_path / "gold.jsonl", concurrency + ) + gold_labels = { + key: frozenset(r["labels"]) + for key, r in gold_results.items() + if r.get("labels") + } + gold_eval = evaluate_against_gold(gold, gold_labels, judge_model) + validation["gold"] = gold_eval + validation["gold_gate"] = _gate( + gold_eval["exact_set_accuracy"], + gold_eval["exact_set_accuracy_ci_low"], + GOLD_ACCURACY_GATE, + ci_floor=GOLD_ACCURACY_CI_FLOOR, + ) + failed = [ + name + for name in ("gold_gate", "cross_judge_gate") + if validation.get(name, {}).get("status") == "fail" + ] + validation["judge_below_reliability_bar"] = bool(failed) + validation["failed_gates"] = failed + + result = reasoning_stability_by_model( + pairs, + labels, + pair_noise_floor=pair_noise_floor, + min_stable_exact_pairs=min_stable_exact_pairs, + ) + tables = { + "reasoning_stability_by_model": result["summary"], + "reasoning_stability_composition": result["composition"], + } + if reference_labels: + tables["reference_alignment_by_model"] = reference_alignment_by_model( + repeated, ground_truth, labels, reference_labels + ) + if labels and "explanation" in repeated.columns: + rows = [] + for (model, scenario_id, variable), group in repeated.groupby( + ["model", "scenario_id", "variable"] + ): + sets = [ + labels.get(text_key(variable, e)) + for e in group["explanation"] + if isinstance(e, str) and e.strip() + ] + sets = [s for s in sets if s is not None] + if len(sets) >= 2: + rows.append( + { + "model": model, + "scenario_id": scenario_id, + "variable": variable, + "k_labeled": len(sets), + "dominant_set_share": dominant_set_share(sets), + } + ) + if rows: + dominant = pd.DataFrame(rows) + tables["dominant_label_set_share_rows"] = dominant + tables["dominant_label_set_share_by_model"] = ( + dominant.groupby("model")["dominant_set_share"] + .mean() + .rename("mean_dominant_set_share") + .reset_index() + ) + if not result["summary"].empty and validation.get("judge_below_reliability_bar"): + result["summary"]["judge_below_reliability_bar"] = True + result["summary"]["right_answer_unstable_reasoning_rate"] = float("nan") + written = _write_tables(output_path, tables) + for name, value in list(validation.items()): + if isinstance(value, dict) and isinstance(value.get("per_label"), pd.DataFrame): + per_label = value.pop("per_label") + per_label.to_csv( + output_path / f"validation_{name}_per_label.csv", index=False + ) + metadata["validation"] = validation + metadata["tables"] = written + write_metadata(output_path, metadata) + return {"tables": tables, "validation": validation, "metadata": metadata} + + +def stability_cost_plan( + predictions: pd.DataFrame, + *, + repeats: int = 3, + cf_arms: int = 1, + models: list[str] | None = None, + judge_input_tokens: int = 800, + judge_output_tokens: int = 30, + judge_price_in_per_1m: float = 0.75, + judge_price_out_per_1m: float = 3.75, + validation_fraction: float = 0.10, +) -> pd.DataFrame: + """Per-model cost ladder from a predictions file's logged usage. + + Model spend = per-run cost × (repeats + cf_arms); judge spend = rows × + repeats × per-call price × (1 + validation fraction, for the double-graded + sample). Missing reconstructed costs fall back to PRICE_OVERRIDES_PER_1M + on logged tokens; models with no usable cost are flagged, not priced. + """ + usage = usage_summary_by_model(predictions) + if models: + usage = usage[usage["model"].isin(models)] + households = int(predictions["scenario_id"].nunique()) or 1 + rows_per_model = predictions.groupby("model").size() + judge_per_call = ( + judge_input_tokens * judge_price_in_per_1m + + judge_output_tokens * judge_price_out_per_1m + ) / 1e6 + rows = [] + for _, row in usage.iterrows(): + model = str(row["model"]) + cost = row.get("total_cost_usd") + basis = "logged" + if cost is None or pd.isna(cost) or float(cost) == 0.0: + override = PRICE_OVERRIDES_PER_1M.get(model) + prompt = row.get("prompt_tokens") + completion = row.get("completion_tokens") + if override and prompt is not None and not pd.isna(prompt): + cost = ( + float(prompt) * override["input"] / 1e6 + + float(completion or 0.0) * override["output"] / 1e6 + ) + basis = "override_prices" + else: + cost = float("nan") + basis = "unpriced" + n_rows = int(rows_per_model.get(model, 0)) + judge_calls = n_rows * repeats * (1.0 + validation_fraction) + rows.append( + { + "model": model, + "cost_basis": basis, + "cost_per_run_usd": float(cost) if cost == cost else float("nan"), + "cost_per_household_usd": ( + float(cost) / households if cost == cost else float("nan") + ), + "arms": repeats + cf_arms, + "model_spend_usd": ( + float(cost) * (repeats + cf_arms) if cost == cost else float("nan") + ), + "rows_per_run": n_rows, + "judge_calls": int(round(judge_calls)), + "judge_spend_usd": judge_calls * judge_per_call, + } + ) + plan = pd.DataFrame(rows) + if plan.empty: + return plan + total = { + "model": "__total__", + "cost_basis": "", + "cost_per_run_usd": plan["cost_per_run_usd"].sum(skipna=True), + "cost_per_household_usd": float("nan"), + "arms": repeats + cf_arms, + "model_spend_usd": plan["model_spend_usd"].sum(skipna=True), + "rows_per_run": int(plan["rows_per_run"].sum()), + "judge_calls": int(plan["judge_calls"].sum()), + "judge_spend_usd": plan["judge_spend_usd"].sum(), + } + return pd.concat([plan, pd.DataFrame([total])], ignore_index=True) diff --git a/tests/test_counterfactual.py b/tests/test_counterfactual.py index 0975844..bfe9769 100644 --- a/tests/test_counterfactual.py +++ b/tests/test_counterfactual.py @@ -6,8 +6,8 @@ import pytest from policybench.counterfactual import ( - ZERO_DELTA_BASELINE_MODEL, CF_ID_SUFFIX, + ZERO_DELTA_BASELINE_MODEL, binomial_ci_exact, build_counterfactual_manifest, build_counterfactual_twin, @@ -205,7 +205,9 @@ def test_multiple_base_runs_produce_one_match_per_run(self): ], ignore_index=True, ) - matched = matched_delta_frame(base, _predictions(perturbed=True), _truth_deltas()) + matched = matched_delta_frame( + base, _predictions(perturbed=True), _truth_deltas() + ) assert set(matched["base_run_id"]) == {"run_000", "run_001"} assert len(matched) == 2 * len(_truth_deltas()) @@ -247,7 +249,14 @@ def test_small_delta_band_floor_makes_within_superset_of_exact(self): ] ) base = pd.DataFrame( - [{"model": "m", "scenario_id": "s0", "variable": "state_refundable_credits", "prediction": 100.0}] + [ + { + "model": "m", + "scenario_id": "s0", + "variable": "state_refundable_credits", + "prediction": 100.0, + } + ] ) pert = base.assign(prediction=104.0) # pred_delta = 4.0, error $0.17 matched = matched_delta_frame(base, pert, truth) diff --git a/tests/test_reasoning_stability.py b/tests/test_reasoning_stability.py index e8e6d2c..2a1412e 100644 --- a/tests/test_reasoning_stability.py +++ b/tests/test_reasoning_stability.py @@ -94,7 +94,9 @@ def test_cache_key_depends_on_text_model_and_pass(self): class TestParseJudgeLabels: def test_parses_json_object(self): - assert parse_judge_labels('{"labels": ["payroll_tax_base", "thresholds_rates"]}') == [ + assert parse_judge_labels( + '{"labels": ["payroll_tax_base", "thresholds_rates"]}' + ) == [ "payroll_tax_base", "thresholds_rates", ] @@ -121,15 +123,34 @@ def _judge_response(content: str): class TestExtractionRunner: def test_dedups_identical_texts_and_caches_to_disk(self, tmp_path): items = [ - {"variable": "snap", "country": "us", "year": 2026, "text": "Income exceeds 130% FPL."}, - {"variable": "snap", "country": "us", "year": 2026, "text": "income exceeds 130% fpl."}, - {"variable": "ssi", "country": "us", "year": 2026, "text": "Neither adult is aged or disabled."}, + { + "variable": "snap", + "country": "us", + "year": 2026, + "text": "Income exceeds 130% FPL.", + }, + { + "variable": "snap", + "country": "us", + "year": 2026, + "text": "income exceeds 130% fpl.", + }, + { + "variable": "ssi", + "country": "us", + "year": 2026, + "text": "Neither adult is aged or disabled.", + }, ] cache = tmp_path / "labels.jsonl" acompletion = AsyncMock( side_effect=[ - _judge_response('{"labels": ["categorical_eligibility", "thresholds_rates"]}'), - _judge_response('{"labels": ["categorical_eligibility", "age_disability"]}'), + _judge_response( + '{"labels": ["categorical_eligibility", "thresholds_rates"]}' + ), + _judge_response( + '{"labels": ["categorical_eligibility", "age_disability"]}' + ), ] ) with patch("policybench.reasoning_stability.litellm.acompletion", acompletion): @@ -138,7 +159,10 @@ def test_dedups_identical_texts_and_caches_to_disk(self, tmp_path): assert acompletion.call_count == 2 assert len(results) == 2 snap_key = judge_cache_key("judge", "snap", items[0]["text"], 0) - assert results[snap_key]["labels"] == ["categorical_eligibility", "thresholds_rates"] + assert results[snap_key]["labels"] == [ + "categorical_eligibility", + "thresholds_rates", + ] # Second invocation is served entirely from the cache file. acompletion.reset_mock() with patch("policybench.reasoning_stability.litellm.acompletion", acompletion): @@ -165,7 +189,9 @@ def test_requests_never_use_litellm_cache_and_pass_temperature_zero(self, tmp_pa items = [{"variable": "snap", "country": "us", "year": 2026, "text": "x"}] acompletion = AsyncMock(return_value=_judge_response('{"labels": ["other"]}')) with patch("policybench.reasoning_stability.litellm.acompletion", acompletion): - run_label_extraction(items, judge_model="judge", cache_path=tmp_path / "c.jsonl") + run_label_extraction( + items, judge_model="judge", cache_path=tmp_path / "c.jsonl" + ) kwargs = acompletion.call_args.kwargs assert kwargs["caching"] is False assert kwargs["temperature"] == 0 @@ -175,9 +201,14 @@ def test_grade_pass_forces_independent_calls(self, tmp_path): items = [{"variable": "snap", "country": "us", "year": 2026, "text": "x"}] acompletion = AsyncMock(return_value=_judge_response('{"labels": ["other"]}')) with patch("policybench.reasoning_stability.litellm.acompletion", acompletion): - run_label_extraction(items, judge_model="judge", cache_path=tmp_path / "c.jsonl") run_label_extraction( - items, judge_model="judge", cache_path=tmp_path / "c.jsonl", grade_pass=1 + items, judge_model="judge", cache_path=tmp_path / "c.jsonl" + ) + run_label_extraction( + items, + judge_model="judge", + cache_path=tmp_path / "c.jsonl", + grade_pass=1, ) assert acompletion.call_count == 2 @@ -214,7 +245,12 @@ def _repeated_predictions(): ), ]: for (sid, var), pred, text in zip( - [("s1", "snap"), ("s1", "payroll_tax"), ("s2", "snap"), ("s2", "payroll_tax")], + [ + ("s1", "snap"), + ("s1", "payroll_tax"), + ("s2", "snap"), + ("s2", "payroll_tax"), + ], preds, texts, strict=True, @@ -267,7 +303,10 @@ def test_headline_and_companions(self): pairs, { "Net income test applied.": {"thresholds_rates"}, - "Gross income under 130% FPL.": {"categorical_eligibility", "thresholds_rates"}, + "Gross income under 130% FPL.": { + "categorical_eligibility", + "thresholds_rates", + }, "Asset test failed.": {"asset_resource"}, "Gross test.": {"thresholds_rates"}, }, @@ -281,13 +320,17 @@ def test_headline_and_companions(self): assert row["mechanism_agreement_rate_stable_exact"] == pytest.approx(0.5) assert row["right_answer_unstable_reasoning_rate"] == pytest.approx(0.5) # Restricted to non-identical texts: only s1/snap remains -> 1.0. - assert row["right_answer_unstable_reasoning_rate_nonidentical"] == pytest.approx(1.0) + assert row[ + "right_answer_unstable_reasoning_rate_nonidentical" + ] == pytest.approx(1.0) assert row["short_circuit_share_stable"] == pytest.approx(1 / 3) # Joint rate over ALL pairs: one stable∧exact∧divergent pair of 4. assert row["joint_unstable_reasoning_rate_all_pairs"] == pytest.approx(0.25) # Stable stratum (3 pairs): s1/snap differ, s1/payroll same, s2/snap differ. assert row["mechanism_agreement_rate_stable"] == pytest.approx(1 / 3) - assert row["mechanism_jaccard_mean_stable_exact"] == pytest.approx((0.5 + 1.0) / 2) + assert row["mechanism_jaccard_mean_stable_exact"] == pytest.approx( + (0.5 + 1.0) / 2 + ) assert bool(row["headline_suppressed"]) is False def test_suppression_rule(self): @@ -327,7 +370,8 @@ def test_attenuation_adjustment(self): def test_numeric_claim_channel(self): preds = _repeated_predictions() preds.loc[ - (preds["scenario_id"] == "s1") & (preds["variable"] == "snap"), "explanation" + (preds["scenario_id"] == "s1") & (preds["variable"] == "snap"), + "explanation", ] = ["Deduction of $177 and 30% rate.", "Deduction of $177 and 24% rate."] pairs = explanation_pair_frame(preds, _ground_truth()) result = reasoning_stability_by_model(pairs, {}, min_stable_exact_pairs=1) @@ -374,7 +418,9 @@ def test_alignment_among_exact_rows(self): ("s2", "snap"): frozenset({"thresholds_rates"}), ("s2", "payroll_tax"): frozenset({"payroll_tax_base"}), } - table = reference_alignment_by_model(preds, _ground_truth(), labels, reference_labels) + table = reference_alignment_by_model( + preds, _ground_truth(), labels, reference_labels + ) row = table.set_index("model").loc["m"] # Exact-correct run-explanations: s1/snap x2, s1/payroll x2, s2/payroll run_000. # Labeled among them: s1/snap (1 of 2 aligned), s1/payroll (2 aligned). @@ -395,8 +441,12 @@ def test_gwet_ac1_and_kappa_known_values(self): assert gwet_ac1([True, False], [True, False]) == pytest.approx(1.0) def test_kappa_perfect_and_chance(self): - assert cohen_kappa([True, False, True, False], [True, False, True, False]) == pytest.approx(1.0) - assert cohen_kappa([True, True, False, False], [True, False, True, False]) == pytest.approx(0.0) + assert cohen_kappa( + [True, False, True, False], [True, False, True, False] + ) == pytest.approx(1.0) + assert cohen_kappa( + [True, True, False, False], [True, False, True, False] + ) == pytest.approx(0.0) def test_krippendorff_alpha_jaccard(self): perfect = [ @@ -429,7 +479,10 @@ def test_exact_set_accuracy_and_per_label(self): { "variable": ["snap", "ssi"], "explanation": ["Income exceeds 130% FPL.", "Not aged or disabled."], - "labels": ["categorical_eligibility|thresholds_rates", "age_disability"], + "labels": [ + "categorical_eligibility|thresholds_rates", + "age_disability", + ], } ) labels_by_key = { diff --git a/tests/test_stability.py b/tests/test_stability.py index ef67b1f..58d2cf4 100644 --- a/tests/test_stability.py +++ b/tests/test_stability.py @@ -103,10 +103,34 @@ def _repeated_predictions(): ("run_001", 250.0, 1.0, None, 0.5), ]: rows += [ - {"run_id": run_id, "model": "m", "scenario_id": "s1", "variable": "snap", "prediction": snap1}, - {"run_id": run_id, "model": "m", "scenario_id": "s1", "variable": "person_wic_eligible", "prediction": wic1}, - {"run_id": run_id, "model": "m", "scenario_id": "s2", "variable": "snap", "prediction": snap2}, - {"run_id": run_id, "model": "m", "scenario_id": "s2", "variable": "person_wic_eligible", "prediction": wic2}, + { + "run_id": run_id, + "model": "m", + "scenario_id": "s1", + "variable": "snap", + "prediction": snap1, + }, + { + "run_id": run_id, + "model": "m", + "scenario_id": "s1", + "variable": "person_wic_eligible", + "prediction": wic1, + }, + { + "run_id": run_id, + "model": "m", + "scenario_id": "s2", + "variable": "snap", + "prediction": snap2, + }, + { + "run_id": run_id, + "model": "m", + "scenario_id": "s2", + "variable": "person_wic_eligible", + "prediction": wic2, + }, ] return pd.DataFrame(rows) @@ -123,7 +147,11 @@ def test_pair_flags(self): assert bool(snap1["exact_a"]) and not bool(snap1["exact_b"]) wic1 = by_row.loc[("s1", "person_wic_eligible")] - assert bool(wic1["mutually_exact"]) and bool(wic1["exact_a"]) and bool(wic1["exact_b"]) + assert ( + bool(wic1["mutually_exact"]) + and bool(wic1["exact_a"]) + and bool(wic1["exact_b"]) + ) snap2 = by_row.loc[("s2", "snap")] assert not bool(snap2["both_parsed"]) @@ -196,8 +224,12 @@ def test_bootstrap_ci_columns_present_and_ordered(self): assert row[f"{metric}_ci_low"] <= row[metric] <= row[f"{metric}_ci_high"] def test_bootstrap_deterministic_under_seed(self): - a = row_stability_by_model(_repeated_predictions(), _ground_truth(), n_boot=30, seed=3) - b = row_stability_by_model(_repeated_predictions(), _ground_truth(), n_boot=30, seed=3) + a = row_stability_by_model( + _repeated_predictions(), _ground_truth(), n_boot=30, seed=3 + ) + b = row_stability_by_model( + _repeated_predictions(), _ground_truth(), n_boot=30, seed=3 + ) pd.testing.assert_frame_equal(a, b) def test_empty_inputs(self): @@ -224,7 +256,13 @@ def _inputs(self): ]: for sid, p in zip(["s1", "s2", "s3", "s4"], preds, strict=True): rows.append( - {"run_id": run_id, "model": "m", "scenario_id": sid, "variable": "snap", "prediction": p} + { + "run_id": run_id, + "model": "m", + "scenario_id": sid, + "variable": "snap", + "prediction": p, + } ) return gt, pd.DataFrame(rows), {"s1": 0.0, "s2": 0.0, "s3": 0.0, "s4": 0.0} @@ -236,7 +274,11 @@ def test_per_model_columns_and_decision_rule(self): # Run scores: 1.0, 0.5, 1.0 -> std = 0.288675. assert row["run_score_mean"] == pytest.approx(5 / 6, abs=1e-6) assert row["run_score_std"] == pytest.approx(0.288675, abs=1e-5) - assert row["run_score_std_ci_low"] < row["run_score_std"] < row["run_score_std_ci_high"] + assert ( + row["run_score_std_ci_low"] + < row["run_score_std"] + < row["run_score_std_ci_high"] + ) assert row["sampling_se"] > 0 assert row["run_to_sampling_ratio"] == pytest.approx( row["run_score_std"] / row["sampling_se"] @@ -278,7 +320,9 @@ def _write_ledger(self, tmp_path, records): return runs def test_clean_ledger_passes(self, tmp_path): - runs = self._write_ledger(tmp_path, [{"cache_hit": False}, {"cache_hit": False}]) + runs = self._write_ledger( + tmp_path, [{"cache_hit": False}, {"cache_hit": False}] + ) report = scan_runs_for_cache_hits([runs]) assert report["cache_hits"] == 0 assert report["records"] == 2 @@ -311,7 +355,12 @@ def test_cross_run_response_id_duplicates(self): } ) assert cross_run_response_id_duplicates(preds) == 1 - assert cross_run_response_id_duplicates(preds.drop(columns=["provider_response_id"])) is None + assert ( + cross_run_response_id_duplicates( + preds.drop(columns=["provider_response_id"]) + ) + is None + ) class TestLoadRunsDirs: diff --git a/tests/test_stability_harness.py b/tests/test_stability_harness.py index ec1a7df..330e238 100644 --- a/tests/test_stability_harness.py +++ b/tests/test_stability_harness.py @@ -35,7 +35,14 @@ def test_repeated_eval_writes_runs_metadata( ): monkeypatch.delenv("POLICYBENCH_TOOL_CHOICE", raising=False) - def fake_run(scenarios, models=None, programs=None, output_path=None, run_id=None, include_explanations=True): + def fake_run( + scenarios, + models=None, + programs=None, + output_path=None, + run_id=None, + include_explanations=True, + ): frame = pd.DataFrame( { "run_id": [run_id], @@ -70,7 +77,14 @@ def test_mixed_model_groups_in_one_directory_rejected( ): monkeypatch.delenv("POLICYBENCH_TOOL_CHOICE", raising=False) - def fake_run(scenarios, models=None, programs=None, output_path=None, run_id=None, include_explanations=True): + def fake_run( + scenarios, + models=None, + programs=None, + output_path=None, + run_id=None, + include_explanations=True, + ): frame = pd.DataFrame( { "run_id": [run_id], diff --git a/tests/test_stability_report.py b/tests/test_stability_report.py new file mode 100644 index 0000000..99c1e4a --- /dev/null +++ b/tests/test_stability_report.py @@ -0,0 +1,508 @@ +"""End-to-end tests for the stability suite commands (all external calls mocked).""" + +import json +from unittest.mock import patch + +import pandas as pd +import pytest + +from policybench.reasoning_stability import text_key +from policybench.stability import CacheContaminationError +from policybench.stability_report import ( + read_runs_metadata, + run_counterfactual_manifest, + run_counterfactual_report, + run_reasoning_stability, + run_stability_report, + serving_config_diff, + stability_cost_plan, +) + +VARIABLES = ["snap", "payroll_tax", "person_wic_eligible"] +SCENARIOS = ["scenario_000", "scenario_001"] + + +def _reference(tmp_path): + rows = [] + for sid, values in zip( + SCENARIOS, [[100.0, 500.0, 1.0], [0.0, 700.0, 0.0]], strict=True + ): + for var, value in zip(VARIABLES, values, strict=True): + rows.append({"scenario_id": sid, "variable": var, "value": value}) + path = tmp_path / "reference_outputs.csv" + pd.DataFrame(rows).to_csv(path, index=False) + return path + + +def _manifest(tmp_path): + path = tmp_path / "scenarios.csv" + pd.DataFrame( + { + "scenario_id": SCENARIOS, + "country": ["us", "us"], + "total_income": [30000.0, 45000.0], + } + ).to_csv(path, index=False) + return path + + +def _run_frame(model, run_id, jitter=0.0, explanation_suffix=""): + rows = [] + for sid, values in zip( + SCENARIOS, [[100.0, 500.0, 1.0], [0.0, 700.0, 0.0]], strict=True + ): + for var, value in zip(VARIABLES, values, strict=True): + pred = value + (jitter if var == "snap" else 0.0) + rows.append( + { + "run_id": run_id, + "model": model, + "scenario_id": sid, + "variable": var, + "prediction": pred, + "explanation": f"{var} derived for {sid}{explanation_suffix}", + "provider_response_id": f"{model}-{run_id}-{sid}", + } + ) + return pd.DataFrame(rows) + + +def _write_runs(path, model, k=2, cache_hit=False, with_metadata=True, jitter=0.0): + path.mkdir(parents=True, exist_ok=True) + for i in range(k): + run_id = f"run_{i:03d}" + frame = _run_frame( + model, + run_id, + jitter=jitter * i, + explanation_suffix=f" (run {i})" if jitter else "", + ) + frame.to_csv(path / f"{run_id}.csv", index=False) + ledger = path / f"{run_id}.csv.spend.jsonl" + ledger.write_text(json.dumps({"cache_hit": cache_hit and i > 0}) + "\n") + if with_metadata: + (path / "runs_metadata.json").write_text( + json.dumps( + { + "scenario_hash": "abc", + "programs": VARIABLES, + "repeats": k, + "response_contract": { + "tool_choice_condition": "forced", + "chunk_override": None, + }, + "cache_enabled": False, + } + ) + ) + return path + + +class TestStabilityReport: + def test_writes_tables_and_metadata(self, tmp_path): + runs_a = _write_runs(tmp_path / "runs" / "a", "gpt-5.6-sol", jitter=5.0) + runs_b = _write_runs(tmp_path / "runs" / "b", "claude-opus-5") + out = tmp_path / "stability" + result = run_stability_report( + runs_dirs=[runs_a, runs_b], + reference_outputs=_reference(tmp_path), + scenario_manifest=_manifest(tmp_path), + output_dir=out, + n_boot=20, + seed=1, + serving_config=None, + ) + assert (out / "row_stability_by_model.csv").exists() + assert (out / "variance_decomposition_by_model.csv").exists() + assert (out / "run_stability_by_model.csv").exists() + metadata = json.loads((out / "stability_metadata.json").read_text()) + assert metadata["stability_spec_version"] == "2026-08-18-v1" + assert metadata["cache_guard"]["cache_hits"] == 0 + assert ( + metadata["runs_metadata"]["condition"]["tool_choice_condition"] == "forced" + ) + assert metadata["cross_run_response_id_duplicates"] == 0 + assert set(metadata["models"]) == {"gpt-5.6-sol", "claude-opus-5"} + row_table = result["tables"]["row_stability_by_model"].set_index("model") + # The jittered model flips its snap answers; the other is unanimous. + assert row_table.loc["gpt-5.6-sol", "answer_flip_rate"] > 0 + assert row_table.loc["claude-opus-5", "answer_flip_rate"] == 0 + assert ( + "pooled_run_to_sampling_ratio" in metadata["pooled_variance_decomposition"] + ) + + def test_cache_contamination_hard_fails(self, tmp_path): + runs = _write_runs(tmp_path / "runs", "gpt-5.6-sol", cache_hit=True) + with pytest.raises(CacheContaminationError): + run_stability_report( + runs_dirs=[runs], + reference_outputs=_reference(tmp_path), + scenario_manifest=_manifest(tmp_path), + output_dir=tmp_path / "out", + n_boot=5, + serving_config=None, + ) + + def test_mismatched_condition_refuses_to_pool(self, tmp_path): + runs_a = _write_runs(tmp_path / "a", "m1") + runs_b = _write_runs(tmp_path / "b", "m2") + meta = json.loads((runs_b / "runs_metadata.json").read_text()) + meta["response_contract"]["tool_choice_condition"] = "auto" + (runs_b / "runs_metadata.json").write_text(json.dumps(meta)) + with pytest.raises(ValueError, match="fingerprint"): + read_runs_metadata([runs_a, runs_b]) + + def test_missing_sidecars_are_reported_not_fatal(self, tmp_path): + runs = _write_runs(tmp_path / "a", "m1", with_metadata=False) + meta = read_runs_metadata([runs]) + assert meta["sidecars_missing"] == [str(runs)] + assert meta["condition"] is None + + +class TestServingConfigDiff: + def test_diffs_against_registry(self, tmp_path): + registry = tmp_path / "serving.json" + registry.write_text( + json.dumps( + { + "models": { + "gpt-5.6-sol": { + "answer_contract": "tool", + "request_shape": "whole scenario", + "tool_choice": "forced", + } + } + } + ) + ) + diff = serving_config_diff( + ["gpt-5.6-sol", "claude-fable-5"], + {"tool_choice_condition": "auto", "chunk_override": None}, + registry, + ) + assert diff["gpt-5.6-sol"]["in_snapshot_registry"] is True + assert diff["gpt-5.6-sol"]["diffs"]["tool_choice"] == { + "effective": "auto", + "snapshot": "forced", + } + assert diff["claude-fable-5"]["in_snapshot_registry"] is False + assert ( + diff["claude-fable-5"]["effective"]["request_shape"] == "1 output/request" + ) + + +class TestCounterfactualManifestCommand: + def test_writes_manifest_deltas_and_drift(self, tmp_path, sample_scenarios): + from policybench.scenarios import scenario_manifest + + base_manifest = tmp_path / "base.csv" + scenario_manifest(sample_scenarios).to_csv(base_manifest, index=False) + frozen = tmp_path / "frozen.csv" + pd.DataFrame( + { + "scenario_id": [s.id for s in sample_scenarios], + "variable": ["snap"] * 3, + "value": [100.0, 100.0, 160.0], + } + ).to_csv(frozen, index=False) + + def fake_deltas(base, twins, programs, year): + return pd.DataFrame( + { + "scenario_id": [s.id for s in base], + "variable": ["snap"] * 3, + "base_value": [100.0, 100.0, 100.0], + "perturbed_value": [100.0, 50.0, 100.0], + "true_delta": [0.0, -50.0, 0.0], + "is_binary": [False] * 3, + "output_group": ["snap"] * 3, + "first_dollar": [False] * 3, + } + ) + + with patch( + "policybench.stability_report.compute_truth_deltas", side_effect=fake_deltas + ): + metadata = run_counterfactual_manifest( + scenario_manifest=base_manifest, + output_dir=tmp_path / "cf", + programs=["snap"], + year=2026, + frozen_reference_outputs=frozen, + ) + assert (tmp_path / "cf" / "cf_scenarios.csv").exists() + assert (tmp_path / "cf" / "truth_deltas.csv").exists() + summary = metadata["truth_delta_summary"] + assert summary["n_nonzero_amount_rows"] == 1 + assert summary["n_negative"] == 1 + assert summary["n_binary_flips"] == 0 + drift = metadata["base_vs_frozen_reference_drift"] + assert drift["n_compared"] == 3 + assert drift["n_within_1usd"] == 2 + assert drift["max_abs_drift"] == pytest.approx(60.0) + + def test_skip_reference_outputs(self, tmp_path, sample_scenarios): + from policybench.scenarios import scenario_manifest + + base_manifest = tmp_path / "base.csv" + scenario_manifest(sample_scenarios).to_csv(base_manifest, index=False) + metadata = run_counterfactual_manifest( + scenario_manifest=base_manifest, + output_dir=tmp_path / "cf", + programs=["snap"], + year=2026, + compute_references=False, + ) + assert "truth_deltas" not in metadata + assert metadata["perturbation"]["n_scenarios"] == 3 + + +class TestCounterfactualReportCommand: + def _truth(self, tmp_path): + rows = [] + for sid in SCENARIOS: + rows.append( + { + "scenario_id": sid, + "variable": "payroll_tax", + "base_value": 500.0, + "true_delta": 76.5, + "is_binary": False, + "output_group": "payroll_tax", + "first_dollar": False, + } + ) + rows.append( + { + "scenario_id": sid, + "variable": "person_wic_eligible", + "base_value": 1.0, + "true_delta": 0.0, + "is_binary": True, + "output_group": "person_wic_eligible", + "first_dollar": False, + } + ) + path = tmp_path / "truth_deltas.csv" + pd.DataFrame(rows).to_csv(path, index=False) + return path + + def test_with_repeats_as_base_arm(self, tmp_path): + runs = _write_runs(tmp_path / "runs", "m", k=2) + perturbed = _run_frame("m", "run_000").drop(columns=["run_id"]) + perturbed["scenario_id"] = perturbed["scenario_id"] + "__cf1k" + perturbed.loc[perturbed["variable"] == "payroll_tax", "prediction"] += 76.5 + perturbed_path = tmp_path / "perturbed.csv" + perturbed.to_csv(perturbed_path, index=False) + out = tmp_path / "cf_report" + result = run_counterfactual_report( + perturbed_predictions=perturbed_path, + truth_deltas=self._truth(tmp_path), + output_dir=out, + base_runs_dirs=[runs], + n_boot=20, + ) + tables = result["tables"] + summary = tables["delta_summary"].set_index("model") + assert summary.loc["m", "delta_exact_rate_nonzero"] == pytest.approx(1.0) + assert "noise_floor_summary" in tables + assert "signal_vs_noise" in tables + assert (out / "delta_summary.csv").exists() + metadata = json.loads((out / "stability_metadata.json").read_text()) + assert metadata["cache_guard"]["cache_hits"] == 0 + + def test_single_base_file(self, tmp_path): + base = _run_frame("m", "run_000").drop(columns=["run_id"]) + base_path = tmp_path / "base.csv" + base.to_csv(base_path, index=False) + perturbed = base.copy() + perturbed["scenario_id"] = perturbed["scenario_id"] + "__cf1k" + perturbed_path = tmp_path / "perturbed.csv" + perturbed.to_csv(perturbed_path, index=False) + result = run_counterfactual_report( + perturbed_predictions=perturbed_path, + truth_deltas=self._truth(tmp_path), + output_dir=tmp_path / "out", + base_predictions=base_path, + n_boot=5, + ) + assert "noise_floor_summary" not in result["tables"] + summary = result["tables"]["delta_summary"].set_index("model") + # No movement on payroll -> misses both nonzero rows. + assert summary.loc["m", "delta_exact_rate_nonzero"] == pytest.approx(0.0) + + +class TestReasoningStabilityCommand: + def test_deterministic_only_path(self, tmp_path): + runs = _write_runs(tmp_path / "runs", "m", k=2, jitter=0.0) + out = tmp_path / "reasoning" + result = run_reasoning_stability( + runs_dirs=[runs], + reference_outputs=_reference(tmp_path), + output_dir=out, + reference_explanations=None, + gold_set=None, + deterministic_only=True, + min_stable_exact_pairs=1, + ) + summary = result["tables"]["reasoning_stability_by_model"].set_index("model") + # Identical explanations across runs: every stable pair short-circuits. + assert summary.loc["m", "short_circuit_share_stable"] == pytest.approx(1.0) + assert summary.loc[ + "m", "right_answer_unstable_reasoning_rate" + ] == pytest.approx(0.0) + metadata = json.loads((out / "stability_metadata.json").read_text()) + assert metadata["judge"]["deterministic_only"] is True + assert metadata["validation"] == {} + + def test_judged_path_with_mocked_extraction(self, tmp_path): + runs = _write_runs(tmp_path / "runs", "m", k=2, jitter=0.0) + # Make explanations differ across runs so the judge is needed. + for i in range(2): + path = runs / f"run_{i:03d}.csv" + frame = pd.read_csv(path) + frame["explanation"] = frame["explanation"] + f" (run {i})" + frame.to_csv(path, index=False) + ref_expl = tmp_path / "reference_explanations.csv" + pd.DataFrame( + { + "scenario_id": SCENARIOS * 3, + "variable": [v for v in VARIABLES for _ in SCENARIOS], + "explanation": [ + f"reference {v} {s}" for v in VARIABLES for s in SCENARIOS + ], + } + ).to_csv(ref_expl, index=False) + gold = tmp_path / "gold.csv" + pd.DataFrame( + { + "variable": ["snap", "payroll_tax"], + "explanation": ["gold snap text", "gold payroll text"], + "labels": ["categorical_eligibility", "payroll_tax_base"], + } + ).to_csv(gold, index=False) + + def fake_extraction( + items, judge_model, cache_path, concurrency=8, grade_pass=0 + ): + out = {} + for item in items: + item_key = text_key(item["variable"], item["text"]) + key = f"{judge_model}|{item_key}|{grade_pass}" + if item["variable"] == "payroll_tax": + labels = ["payroll_tax_base"] + elif item["variable"] == "snap": + # Primary judge disagrees with itself across runs on snap. + labels = ( + ["categorical_eligibility"] + if "run 0" in item["text"] + else ["thresholds_rates"] + ) + if "gold" in item["text"]: + labels = ["categorical_eligibility"] + else: + labels = ["age_disability"] + out[key] = { + "key": key, + "text_key": text_key(item["variable"], item["text"]), + "variable": item["variable"], + "judge_model": judge_model, + "prompt_version": "test", + "grade_pass": grade_pass, + "labels": labels, + "error": "", + } + return out + + def gold_eval(accuracy, ci_low): + return { + "judge_model": "judge", + "n_gold": 100, + "n_graded": 100, + "exact_set_accuracy": accuracy, + "exact_set_accuracy_ci_low": ci_low, + "exact_set_accuracy_ci_high": 1.0, + "krippendorff_alpha_jaccard": 1.0, + "per_label": pd.DataFrame({"label": ["other"], "agreement": [1.0]}), + } + + def run(gold_result, output_dir): + # evaluate_against_gold keys by judge_cache_key; the fake extraction + # keys differ, so the gold evaluation is stubbed at the gate level. + with ( + patch( + "policybench.stability_report.run_label_extraction", + side_effect=fake_extraction, + ), + patch( + "policybench.stability_report.evaluate_against_gold", + return_value=gold_result, + ), + ): + return run_reasoning_stability( + runs_dirs=[runs], + reference_outputs=_reference(tmp_path), + output_dir=output_dir, + reference_explanations=ref_expl, + gold_set=gold, + judge_model="judge", + cross_judge_model="cross", + validation_modulus=1, + min_stable_exact_pairs=1, + ) + + result = run(gold_eval(0.95, 0.89), tmp_path / "reasoning") + summary = result["tables"]["reasoning_stability_by_model"].set_index("model") + # snap pairs disagree (2 scenarios), payroll agree (2), wic agree (2). + assert summary.loc[ + "m", "right_answer_unstable_reasoning_rate" + ] == pytest.approx(2 / 6) + validation = result["validation"] + assert validation["gold_gate"]["status"] == "pass" + assert validation["cross_judge"]["n"] > 0 + assert validation["cross_judge_gate"]["status"] in {"pass", "pass_marginal"} + assert "determinism_floor" in validation + assert validation["judge_below_reliability_bar"] is False + assert "reference_alignment_by_model" in result["tables"] + assert "dominant_label_set_share_by_model" in result["tables"] + assert ( + tmp_path / "reasoning" / "validation_cross_judge_per_label.csv" + ).exists() + + # A gold gate whose CI lower bound sits under the floor fails CI-aware, + # and the headline is withheld (exported as NaN) while companions stay. + failed = run(gold_eval(0.85, 0.60), tmp_path / "reasoning_failed") + failed_summary = failed["tables"]["reasoning_stability_by_model"].set_index( + "model" + ) + assert failed["validation"]["gold_gate"]["status"] == "fail" + assert failed["validation"]["judge_below_reliability_bar"] is True + assert pd.isna(failed_summary.loc["m", "right_answer_unstable_reasoning_rate"]) + assert failed_summary.loc[ + "m", "joint_unstable_reasoning_rate_all_pairs" + ] == pytest.approx(2 / 6) + + +class TestCostPlan: + def test_prices_from_logged_usage_with_override_fallback(self): + predictions = pd.DataFrame( + { + "model": ["gpt-5.4-mini"] * 4 + ["kimi-k3"] * 4, + "scenario_id": ["s1", "s1", "s2", "s2"] * 2, + "variable": ["snap", "ssi"] * 4, + "prediction": [1.0] * 8, + "prompt_tokens": [100] * 8, + "completion_tokens": [50] * 8, + "total_tokens": [150] * 8, + "total_cost_usd": [0.01] * 4 + [0.0] * 4, + } + ) + plan = stability_cost_plan(predictions, repeats=3, cf_arms=1).set_index("model") + assert plan.loc["gpt-5.4-mini", "cost_basis"] == "logged" + assert plan.loc["gpt-5.4-mini", "cost_per_run_usd"] == pytest.approx(0.04) + assert plan.loc["gpt-5.4-mini", "model_spend_usd"] == pytest.approx(0.16) + assert plan.loc["kimi-k3", "cost_basis"] == "override_prices" + # 400 prompt tokens at $3/M + 200 completion at $15/M. + assert plan.loc["kimi-k3", "cost_per_run_usd"] == pytest.approx(0.0012 + 0.003) + assert plan.loc["gpt-5.4-mini", "judge_calls"] == round(4 * 3 * 1.1) + assert "__total__" in plan.index From 9b8221bc2bf1ccadcfc9fe7c9750c470af56a9a6 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Tue, 25 Aug 2026 18:13:27 +0200 Subject: [PATCH 5/7] Gold set, layer-2/3 runbook, rung-0 dry-run record - annotations/stability_reasoning_gold_set.csv: 90 reference explanations (5 per output group, seed 20260818) developer-labeled against the mechanism taxonomy for the judge validity gate; prevalence from thresholds_rates 59/90 down to period_annualization 2/90. - Spec runbook now covers all three layers with the implemented CLI (reasoning-stability, counterfactual-manifest/report, stability-cost-plan) and records the free rung-0 dry run: the truth arm reproduces 247 nonzero amount rows / 0 binary flips / 87.55% zero share through the shipped commands, and the cost plan prices 30 models x 4 arms at $1,302 + ~$140 judge (Fable run-level usage flagged unpriced). - docs/runbook.md cross-references the stability spec and its cache-free repeat rule. Co-Authored-By: Claude Fable 5 --- annotations/stability_reasoning_gold_set.csv | 91 ++++++++++++++++++++ docs/runbook.md | 6 +- docs/stability_spec.md | 53 +++++++++++- 3 files changed, 146 insertions(+), 4 deletions(-) create mode 100644 annotations/stability_reasoning_gold_set.csv diff --git a/annotations/stability_reasoning_gold_set.csv b/annotations/stability_reasoning_gold_set.csv new file mode 100644 index 0000000..eb26d4b --- /dev/null +++ b/annotations/stability_reasoning_gold_set.csv @@ -0,0 +1,91 @@ +scenario_id,variable,output_group,reference_value,explanation,labels,labeler,source +scenario_002,federal_income_tax_before_refundable_credits,federal_income_tax_before_refundable_credits,0.0,"PolicyEngine calculated federal income tax before refundable credits as $0 for this Washington joint filer household with approximately $134,820 in income. The primary component, income_tax_before_credits, was $0, indicating that the household's tax liability before applying any credits was fully eliminated, likely due to the standard deduction and tax bracket structure for joint filers in 2026. The net investment income tax also contributed $0 to the total, despite the household having $400 in taxable interest income and $400 in net investment income, because the adjusted gross income of $23,130 fell below the threshold for this tax. Additional tax components—recapture of investment credit, unreported payroll tax, qualified retirement penalty, and income_tax_capped_non_refundable_credits—all equaled $0, resulting in a final federal income tax before refundable credits of $0.",taxable_income_or_deductions|thresholds_rates,"developer adjudication (Claude Fable 5 agent, 2026-08-23; pending maintainer review)","us_full_run_20260612_policyengine_4_16_1_populace/us_case_reference_explanations.csv, stratified 5/output group, seed 20260818" +scenario_007,federal_income_tax_before_refundable_credits,federal_income_tax_before_refundable_credits,3707.695068359375,"PolicyEngine calculated the federal income tax before refundable credits for this single Idaho resident at $3,707.70 by applying 2026 tax rates to their taxable income of $32,964.12. The taxable income was derived from an adjusted gross income of $49,064.12 (comprising $35,280 in taxable pension income and $13,784.12 in taxable Social Security income) reduced by the standard deduction of $16,100. The tax was computed using the income_tax_main_rates calculation, which applies the applicable 2026 federal tax brackets to the taxable income amount. No additional taxes were added from net investment income tax, investment credit recapture, or other supplemental tax provisions, and no non-refundable credits were applied to reduce the tax liability.",taxable_income_or_deductions|thresholds_rates,"developer adjudication (Claude Fable 5 agent, 2026-08-23; pending maintainer review)","us_full_run_20260612_policyengine_4_16_1_populace/us_case_reference_explanations.csv, stratified 5/output group, seed 20260818" +scenario_013,federal_income_tax_before_refundable_credits,federal_income_tax_before_refundable_credits,0.0,"PolicyEngine calculated federal income tax before refundable credits as $0 for this single Arizona resident with approximately $30,472 in household income. The computation shows that income_tax_before_credits was $0, meaning the household's taxable income fell below the standard deduction threshold for a single filer in 2026. Although the household reported $16 in dividend income, contributing to a net_investment_income of $16, this did not trigger the net investment income tax. The net investment income tax itself was $0, and no recapture of investment credits, unreported payroll taxes, qualified retirement penalties, or non-refundable credits applied, resulting in a final tax liability of $0 before any refundable credits.",taxable_income_or_deductions|thresholds_rates,"developer adjudication (Claude Fable 5 agent, 2026-08-23; pending maintainer review)","us_full_run_20260612_policyengine_4_16_1_populace/us_case_reference_explanations.csv, stratified 5/output group, seed 20260818" +scenario_070,federal_income_tax_before_refundable_credits,federal_income_tax_before_refundable_credits,1979.9605712890625,"PolicyEngine calculated federal income tax before refundable credits of $1,979.96 for this single Illinois resident by first determining taxable income of $20,169.67, which resulted from gross income of $36,615.06 reduced by pre-tax contributions of $1,484.94 (traditional 401k), above-the-line deductions of $345.39, and the standard deduction of $16,100. Applying 2026 tax rates to this taxable income produced income tax before credits of $2,172.36. The household then qualified for a Saver's Credit of $192.40 based on $1,924 in qualified retirement contributions (including $1,484.94 in 401k contributions and $177.01 in IRA contributions across traditional and Roth accounts) and an adjusted gross income of $36,269.67, which fell within the income limits for this non-refundable credit. Subtracting the Saver's Credit from the tax before credits yielded the final amount of $1,979.96, with no additional adjustments for net investment income tax, investment credit recapture, or other supplemental taxes.",taxable_income_or_deductions|thresholds_rates|credit_phaseout,"developer adjudication (Claude Fable 5 agent, 2026-08-23; pending maintainer review)","us_full_run_20260612_policyengine_4_16_1_populace/us_case_reference_explanations.csv, stratified 5/output group, seed 20260818" +scenario_076,federal_income_tax_before_refundable_credits,federal_income_tax_before_refundable_credits,19611.80078125,"PolicyEngine calculated federal income tax before refundable credits of $19,611.80 by starting with a tax unit's adjusted gross income of $164,220, which comprised $160,000 in employment income, $4,000 in rental income, and $220 in taxable interest income. The taxable income was reduced to $139,270 after applying a standard deduction of $24,150 and a qualified business income deduction of $800, resulting in income tax before credits of $24,211.80 under the 2026 head of household tax brackets. From this amount, PolicyEngine subtracted $4,600 in non-refundable credits—consisting of $4,400 in child tax credits (for two children at $2,200 each) and $200 in child and dependent care credits based on $1,000 in qualifying childcare expenses—to arrive at the final federal income tax before refundable credits of $19,611.80.",taxable_income_or_deductions|thresholds_rates|credit_phaseout|household_unit_or_filing_status,"developer adjudication (Claude Fable 5 agent, 2026-08-23; pending maintainer review)","us_full_run_20260612_policyengine_4_16_1_populace/us_case_reference_explanations.csv, stratified 5/output group, seed 20260818" +scenario_002,federal_refundable_credits,federal_refundable_credits,0.0,"PolicyEngine calculated federal refundable credits of $0 for this benchmark household of two adults in Washington State with approximately $134,820 in household income. The refundable credits aggregate comprises five component credits: the Earned Income Tax Credit (EITC), the refundable American Opportunity Credit, the refundable Child Tax Credit (CTC), the Recovery Rebate Credit, and the refundable payroll tax credit. Each of these components evaluated to $0 for the household. The EITC phases out at higher income levels and does not apply to this household; the American Opportunity and Child Tax Credits do not apply because there are no qualifying students or dependent children; the Recovery Rebate Credit does not apply in 2026 as it was specific to prior pandemic relief; and the refundable payroll tax credit similarly does not apply. Consequently, the household receives no refundable federal tax credits.",credit_phaseout|household_unit_or_filing_status,"developer adjudication (Claude Fable 5 agent, 2026-08-23; pending maintainer review)","us_full_run_20260612_policyengine_4_16_1_populace/us_case_reference_explanations.csv, stratified 5/output group, seed 20260818" +scenario_007,federal_refundable_credits,federal_refundable_credits,0.0,"PolicyEngine calculated federal refundable credits of $0 for this single adult household in Idaho with approximately $54,565 in annual income. The computation evaluated five potential refundable credit components: the Earned Income Tax Credit (EITC), the refundable American Opportunity Credit, the refundable Child Tax Credit (CTC), the Recovery Rebate Credit, and the refundable payroll tax credit. Each of these components returned $0, resulting in a total refundable credits value of $0. This outcome reflects that the household's income level and composition do not qualify for any of these refundable tax benefits in 2026.",credit_phaseout|household_unit_or_filing_status,"developer adjudication (Claude Fable 5 agent, 2026-08-23; pending maintainer review)","us_full_run_20260612_policyengine_4_16_1_populace/us_case_reference_explanations.csv, stratified 5/output group, seed 20260818" +scenario_013,federal_refundable_credits,federal_refundable_credits,0.0,"For this single adult in Arizona with approximately $30,472 in household income and no children, PolicyEngine calculated federal refundable credits of $0. The calculation examined five potential refundable credit components: the Earned Income Tax Credit (EITC), the refundable American Opportunity Credit, the refundable Child Tax Credit (CTC), the Recovery Rebate Credit, and the refundable payroll tax credit. Each of these components evaluated to $0 for this household. The EITC yielded no credit because the household's income and filing status did not qualify for this benefit, the refundable American Opportunity Credit was $0 as there were no qualifying education expenses or students, the refundable CTC was $0 due to the absence of dependent children, and the other refundable credits similarly did not apply, resulting in a total refundable credits amount of $0.",credit_phaseout|household_unit_or_filing_status,"developer adjudication (Claude Fable 5 agent, 2026-08-23; pending maintainer review)","us_full_run_20260612_policyengine_4_16_1_populace/us_case_reference_explanations.csv, stratified 5/output group, seed 20260818" +scenario_070,federal_refundable_credits,federal_refundable_credits,0.0,"PolicyEngine calculated federal refundable credits of $0 for this single adult in Illinois with approximately $38,100 in household income. The computation evaluated five potential refundable credit components: the Earned Income Tax Credit (EITC), the refundable American Opportunity Credit, the refundable Child Tax Credit (CTC), the Recovery Rebate Credit, and the refundable payroll tax credit. Each of these components returned $0, resulting in a total refundable credits value of $0. For a single filer without dependents and at this income level, the household does not qualify for the EITC (which phases out at higher income thresholds for single filers without qualifying children) or the refundable CTC (which requires qualifying children), and no other refundable credits apply.",credit_phaseout|household_unit_or_filing_status,"developer adjudication (Claude Fable 5 agent, 2026-08-23; pending maintainer review)","us_full_run_20260612_policyengine_4_16_1_populace/us_case_reference_explanations.csv, stratified 5/output group, seed 20260818" +scenario_076,federal_refundable_credits,federal_refundable_credits,0.0,"For this head-of-household filer in Idaho with approximately $173,820 in household income and two children, PolicyEngine calculated federal refundable credits of $0 in tax year 2026. The calculation evaluated five potential refundable credit components: the Earned Income Tax Credit (EITC), the refundable American Opportunity Credit, the refundable portion of the Child Tax Credit (refundable_ctc), the recovery rebate credit, and the refundable payroll tax credit. Each of these components evaluated to zero, resulting in a total refundable credits amount of $0. This outcome reflects that the household's income level and composition did not qualify them for any of these refundable credit provisions under 2026 tax law.",credit_phaseout|household_unit_or_filing_status,"developer adjudication (Claude Fable 5 agent, 2026-08-23; pending maintainer review)","us_full_run_20260612_policyengine_4_16_1_populace/us_case_reference_explanations.csv, stratified 5/output group, seed 20260818" +scenario_062,free_school_meals_eligible,free_school_meals_eligible,0.0,"PolicyEngine determined that this Florida household is not eligible for free school meals, resulting in a value of 0.0. The household's school_meal_fpg_ratio of 3.86 indicates that their countable income of $61,649—derived from interest income ($257), social security ($21,312), pension income ($11,280), and survivor benefits ($28,800)—exceeds 3.86 times the federal poverty guideline of $15,960 for their household size. Since the household's income far exceeds the 185% reduced-price eligibility threshold and they do not qualify for categorical eligibility through programs like SNAP or TANF, they are classified in school_meal_tier 2 (PAID). Florida does not have universal free school meals, so the household receives no subsidy and must pay the full price for school meals.",taxable_income_or_deductions|thresholds_rates|categorical_eligibility|state_local_rule,"developer adjudication (Claude Fable 5 agent, 2026-08-23; pending maintainer review)","us_full_run_20260612_policyengine_4_16_1_populace/us_case_reference_explanations.csv, stratified 5/output group, seed 20260818" +scenario_068,free_school_meals_eligible,free_school_meals_eligible,0.0,"PolicyEngine determined that this Maryland household is not eligible for free school meals, resulting in a value of 0.0. The household's school_meal_countable_income of $107,083.53—comprising employment income of $34,083.53 and educational assistance of $73,000—was compared against the federal poverty guideline threshold. With a school_meal_fpg_ratio of 6.71, the household's income substantially exceeds the 185% reduced-price eligibility limit, placing them in school_meal_tier 2 (PAID). Since Maryland does not offer universal free school meals and the household does not qualify for categorical eligibility through programs like SNAP or TANF, they are required to pay the full price for school meals. Consequently, the school_meal_net_subsidy is 0, reflecting no public assistance toward meal costs.",taxable_income_or_deductions|thresholds_rates|categorical_eligibility|state_local_rule,"developer adjudication (Claude Fable 5 agent, 2026-08-23; pending maintainer review)","us_full_run_20260612_policyengine_4_16_1_populace/us_case_reference_explanations.csv, stratified 5/output group, seed 20260818" +scenario_075,free_school_meals_eligible,free_school_meals_eligible,0.0,"PolicyEngine determined that this Florida household is not eligible for free school meals, resulting in a value of 0.0. The household's school_meal_countable_income of $105,732.37—derived from employment income of $12,656.37, dividend income of $816, and pension income of $92,260—was compared against the federal poverty guideline threshold. With a school_meal_fpg_ratio of 6.62, the household's income substantially exceeds the 185% reduced-price eligibility limit (which would correspond to a ratio of 1.85), and the household does not qualify for categorical eligibility through programs like SNAP or TANF. Since Florida does not have universal free school meals, the household must pay the full price for school meals and receives no subsidy.",taxable_income_or_deductions|thresholds_rates|categorical_eligibility|state_local_rule,"developer adjudication (Claude Fable 5 agent, 2026-08-23; pending maintainer review)","us_full_run_20260612_policyengine_4_16_1_populace/us_case_reference_explanations.csv, stratified 5/output group, seed 20260818" +scenario_078,free_school_meals_eligible,free_school_meals_eligible,0.0,"PolicyEngine determined that this Maryland household is not eligible for free school meals, resulting in a value of $0. The household's school_meal_fpg_ratio of 12.44 indicates that their countable income of $198,505 is approximately 12.4 times the federal poverty guideline of $15,960 for a single-person household. Since Maryland does not have universal free school meals and the household does not qualify for categorical eligibility through programs like SNAP or TANF, their income far exceeds the 185% reduced-price limit threshold for federal school meal assistance. Consequently, the household is classified in school_meal_tier 2 (PAID), meaning they pay the full price for school meals with no subsidy.",taxable_income_or_deductions|thresholds_rates|categorical_eligibility|state_local_rule,"developer adjudication (Claude Fable 5 agent, 2026-08-23; pending maintainer review)","us_full_run_20260612_policyengine_4_16_1_populace/us_case_reference_explanations.csv, stratified 5/output group, seed 20260818" +scenario_088,free_school_meals_eligible,free_school_meals_eligible,0.0,"PolicyEngine determined that this Texas household is not eligible for free school meals, resulting in a value of 0. The household's school_meal_countable_income of $23,597.76—derived from employment income, dividends, interest, pension, and unemployment compensation—was compared against the federal poverty guideline of $15,960 for a single-person household, yielding a school_meal_fpg_ratio of 1.48. This ratio places the household in the REDUCED tier for school meal pricing, meaning they qualify for reduced-price meals rather than free meals. Since Texas does not have universal free school meals and the household does not meet categorical eligibility through programs like SNAP or TANF, they are ineligible for the free school meals benefit despite qualifying for the reduced-price tier.",taxable_income_or_deductions|thresholds_rates|categorical_eligibility|state_local_rule,"developer adjudication (Claude Fable 5 agent, 2026-08-23; pending maintainer review)","us_full_run_20260612_policyengine_4_16_1_populace/us_case_reference_explanations.csv, stratified 5/output group, seed 20260818" +scenario_031,local_income_tax,local_income_tax,0.0,"PolicyEngine calculated a local income tax of $0 for this California resident with approximately $23,853 in household income during the 2026 tax year. California does not impose a local income tax at the municipal or county level; instead, the state relies on state income tax and sales taxes for revenue. Since there is no applicable local income tax jurisdiction in California, the local_income_tax variable correctly returns zero regardless of the household's income level. This result reflects the actual tax code structure rather than any exemption or deduction applied to the household's earnings.",state_local_rule,"developer adjudication (Claude Fable 5 agent, 2026-08-23; pending maintainer review)","us_full_run_20260612_policyengine_4_16_1_populace/us_case_reference_explanations.csv, stratified 5/output group, seed 20260818" +scenario_059,local_income_tax,local_income_tax,0.0,"PolicyEngine calculated a local income tax of $0 for this Florida household in 2026. Florida does not impose a state income tax, and the local_income_tax variable, which captures taxes levied by local jurisdictions within states, returned zero because Florida municipalities do not assess local income taxes on residents. With a household income of approximately $62,704 and joint filing status, the household has no local income tax liability under Florida's tax structure.",state_local_rule,"developer adjudication (Claude Fable 5 agent, 2026-08-23; pending maintainer review)","us_full_run_20260612_policyengine_4_16_1_populace/us_case_reference_explanations.csv, stratified 5/output group, seed 20260818" +scenario_081,local_income_tax,local_income_tax,0.0,"PolicyEngine calculated a local income tax of $0 for this Massachusetts household in 2026. Massachusetts does not impose a local income tax at the state level, and the household's location within Massachusetts does not have a municipal income tax that would apply to wage or investment income. Therefore, despite the household's substantial income of approximately $174,088, no local income tax liability was generated under the applicable tax rules for this jurisdiction and tax year.",state_local_rule,"developer adjudication (Claude Fable 5 agent, 2026-08-23; pending maintainer review)","us_full_run_20260612_policyengine_4_16_1_populace/us_case_reference_explanations.csv, stratified 5/output group, seed 20260818" +scenario_093,local_income_tax,local_income_tax,0.0,"PolicyEngine calculated a local income tax of $0 for this Missouri household in 2026. Missouri does not impose a local income tax at the state level, and while some municipalities in Missouri may have local earnings taxes, PolicyEngine's calculation for this benchmark household resulted in no local income tax liability. The $0 value reflects the absence of applicable local income tax obligations for this joint household with three adults and approximately $164,156 in household income under the 2026 tax rules.",state_local_rule,"developer adjudication (Claude Fable 5 agent, 2026-08-23; pending maintainer review)","us_full_run_20260612_policyengine_4_16_1_populace/us_case_reference_explanations.csv, stratified 5/output group, seed 20260818" +scenario_109,local_income_tax,local_income_tax,0.0,"PolicyEngine calculated a local income tax of $0 for this Florida household in 2026. Florida does not impose a state income tax, and the local_income_tax variable in PolicyEngine's US tax module reflects taxes levied by local jurisdictions within states that permit such taxation. Since Florida has no state income tax framework and does not authorize local income taxes on wages and salaries, there is no local income tax liability for this household regardless of their $30,000 household income or family composition. The $0 value represents the absence of any applicable local income tax obligation under Florida law for the 2026 tax year.",state_local_rule,"developer adjudication (Claude Fable 5 agent, 2026-08-23; pending maintainer review)","us_full_run_20260612_policyengine_4_16_1_populace/us_case_reference_explanations.csv, stratified 5/output group, seed 20260818" +scenario_012,payroll_tax,payroll_tax,1683.0,"PolicyEngine calculated a payroll tax of $1,683 for this household by summing the employee payroll tax contributions of the tax unit head. The employee payroll tax total of $1,683 comprises two components: employee Social Security tax of $1,364 and employee Medicare tax of $319. These amounts reflect the standard payroll tax withholdings applied to the primary earner's wages, calculated at the statutory rates of 6.2% for Social Security and 1.45% for Medicare on earned income subject to these taxes. The calculation is attributed to the first household member (the tax unit head), while the second adult and child contribute zero payroll tax, consistent with their likely status as non-earners or dependents in this ~$22,000 income household.",payroll_tax_base|thresholds_rates,"developer adjudication (Claude Fable 5 agent, 2026-08-23; pending maintainer review)","us_full_run_20260612_policyengine_4_16_1_populace/us_case_reference_explanations.csv, stratified 5/output group, seed 20260818" +scenario_023,payroll_tax,payroll_tax,1561.1168212890625,"PolicyEngine calculated a total payroll tax of $1,561.12 for this California resident by summing three components: employee Social Security tax of $1,081.44, employee Medicare tax of $252.92, and California employee state payroll tax of $226.75. The state payroll tax component consists entirely of California's employee state disability insurance contribution, which was computed as 1% of the ca_payroll_tax_gross_wages of $17,442.65. This gross wage figure was derived from the household's employment_income of $17,442.65, which represents the base for all payroll tax calculations. The resulting total of $1,561.12 reflects the standard federal and state payroll tax obligations for a single adult in California with approximately $25,443 in household income during tax year 2026.",payroll_tax_base|thresholds_rates|state_local_rule,"developer adjudication (Claude Fable 5 agent, 2026-08-23; pending maintainer review)","us_full_run_20260612_policyengine_4_16_1_populace/us_case_reference_explanations.csv, stratified 5/output group, seed 20260818" +scenario_037,payroll_tax,payroll_tax,3208.668701171875,"PolicyEngine calculated a payroll tax liability of $3,208.67 for this single-adult household with approximately $41,943 in income. The total payroll tax comprises two components: employee Social Security tax of $2,600.49 and employee Medicare tax of $608.18. The Social Security tax was computed by applying the 6.2% employee rate to earnings up to the 2026 Social Security wage base, while the Medicare tax was calculated at 2.9% on total wages without a wage cap. Since this household member is identified as a tax unit head, they are subject to the standard employee payroll tax obligations rather than any alternative calculation method.",payroll_tax_base|thresholds_rates,"developer adjudication (Claude Fable 5 agent, 2026-08-23; pending maintainer review)","us_full_run_20260612_policyengine_4_16_1_populace/us_case_reference_explanations.csv, stratified 5/output group, seed 20260818" +scenario_043,payroll_tax,payroll_tax,330.7310791015625,"PolicyEngine calculated a payroll tax of $330.73 for this single Colorado resident with approximately $4,642 in household income. The total payroll tax comprises three components: employee Social Security tax of $253.47, employee Medicare tax of $59.28, and Colorado state payroll tax of $17.99. The Colorado state payroll tax is specifically a Family and Medical Leave Insurance (FAMLI) contribution, calculated on $4,088.15 in taxable wages derived from the household's employment income. These three tax components sum to the final payroll tax obligation of $330.73.",payroll_tax_base|thresholds_rates|state_local_rule,"developer adjudication (Claude Fable 5 agent, 2026-08-23; pending maintainer review)","us_full_run_20260612_policyengine_4_16_1_populace/us_case_reference_explanations.csv, stratified 5/output group, seed 20260818" +scenario_051,payroll_tax,payroll_tax,3060.0,"PolicyEngine calculated a payroll tax of $3,060 for this Louisiana single adult with approximately $40,000 in household income. This total comprises two components: employee Social Security tax of $2,480 and employee Medicare tax of $580. The Social Security tax was computed at the standard 6.2% rate on eligible wages up to the annual cap, while the Medicare tax was calculated at 1.45% on total wages. Since this individual is classified as a tax unit head, they are subject to the full employee-side payroll tax obligations without any special exemptions or reductions.",payroll_tax_base|thresholds_rates,"developer adjudication (Claude Fable 5 agent, 2026-08-23; pending maintainer review)","us_full_run_20260612_policyengine_4_16_1_populace/us_case_reference_explanations.csv, stratified 5/output group, seed 20260818" +scenario_012,head_chip_eligible,person_chip_eligible,0.0,"PolicyEngine determined that the head of household is not eligible for CHIP (Children's Health Insurance Program) in Mississippi for the 2026 tax year. The calculation evaluated the household against CHIP's eligibility criteria, which require that applicants be children or pregnant women under age 19 (with limited exceptions for pregnant women up to age 55). Since the head of household is 40 years old, they fall outside CHIP's age requirements and therefore do not qualify. Additionally, the household's income of approximately $22,000 exceeds Mississippi's CHIP income threshold for adults, further disqualifying them from the program. The engine also confirmed that the household is not Medicaid-eligible, meaning CHIP represents the only potential pathway to coverage for this household member, which is now closed due to both age and income criteria.",health_coverage|categorical_eligibility|age_disability|thresholds_rates,"developer adjudication (Claude Fable 5 agent, 2026-08-23; pending maintainer review)","us_full_run_20260612_policyengine_4_16_1_populace/us_case_reference_explanations.csv, stratified 5/output group, seed 20260818" +scenario_028,head_chip_eligible,person_chip_eligible,0.0,"PolicyEngine determined that the head of household in this Pennsylvania family is not eligible for CHIP (Children's Health Insurance Program). The household's income of approximately $61,277 exceeds Pennsylvania's CHIP income threshold, which is the primary factor disqualifying this family from the program. Additionally, since the head of household is 45 years old, they would not qualify for CHIP even if the income limit were met, as CHIP is designed for children and young adults, not adults in their mid-40s. The engine's output of `is_chip_eligible = False` reflects that this household fails CHIP's eligibility criteria on both the income dimension and the age/program design dimension.",health_coverage|categorical_eligibility|age_disability|thresholds_rates,"developer adjudication (Claude Fable 5 agent, 2026-08-23; pending maintainer review)","us_full_run_20260612_policyengine_4_16_1_populace/us_case_reference_explanations.csv, stratified 5/output group, seed 20260818" +scenario_038,child2_chip_eligible,person_chip_eligible,0.0,"PolicyEngine determined that child2, a 6-year-old in the household, is not eligible for CHIP (is_chip_eligible = False). The child qualifies for Medicaid under the OLDER_CHILD category, which makes them ineligible for CHIP because CHIP is designed to cover children who do not qualify for Medicaid. Since the child's Medicaid eligibility (is_medicaid_eligible = True) takes precedence, PolicyEngine correctly excludes them from CHIP coverage based on the program's targeting rules rather than any income threshold consideration.",health_coverage|categorical_eligibility|age_disability,"developer adjudication (Claude Fable 5 agent, 2026-08-23; pending maintainer review)","us_full_run_20260612_policyengine_4_16_1_populace/us_case_reference_explanations.csv, stratified 5/output group, seed 20260818" +scenario_080,head_chip_eligible,person_chip_eligible,0.0,"PolicyEngine determined that the head of household is not eligible for CHIP (is_chip_eligible = False) based on the chip_category value of 3. The person, age 41, is an adult and qualifies for Medicaid under the adult category, which makes them ineligible for CHIP. Since CHIP is a program designed to cover children who do not qualify for Medicaid, adults who are already Medicaid-eligible fall outside CHIP's scope of coverage. Therefore, despite the household's low income of approximately $940, the head's Medicaid eligibility status directly precludes CHIP eligibility.",health_coverage|categorical_eligibility|age_disability,"developer adjudication (Claude Fable 5 agent, 2026-08-23; pending maintainer review)","us_full_run_20260612_policyengine_4_16_1_populace/us_case_reference_explanations.csv, stratified 5/output group, seed 20260818" +scenario_109,child3_chip_eligible,person_chip_eligible,0.0,"PolicyEngine determined that child3 is not eligible for CHIP (Children's Health Insurance Program) in Florida for tax year 2026. The child, age 0.0, qualifies for Medicaid under the INFANT category based on the household's income of approximately $30,000. Since CHIP is designed to cover children who do not qualify for Medicaid, and this child already meets Medicaid eligibility criteria, the child is categorically ineligible for CHIP coverage. Therefore, is_chip_eligible returns False for this household member.",health_coverage|categorical_eligibility|age_disability|thresholds_rates,"developer adjudication (Claude Fable 5 agent, 2026-08-23; pending maintainer review)","us_full_run_20260612_policyengine_4_16_1_populace/us_case_reference_explanations.csv, stratified 5/output group, seed 20260818" +scenario_026,child3_early_head_start_eligible,person_early_head_start_eligible,0.0,"PolicyEngine determined that the third child in this North Carolina household is not eligible for Early Head Start, resulting in a value of False (0.0). Early Head Start eligibility is primarily determined by income thresholds relative to the federal poverty line, and this household's income of approximately $85,209 exceeds the maximum income limit for a family of five in 2026. Since the household's income surpasses the eligibility threshold, PolicyEngine correctly classified all children, including the third child, as ineligible for the program. The resulting value of False reflects that this child does not meet the income-based requirements necessary to qualify for Early Head Start services.",categorical_eligibility|thresholds_rates,"developer adjudication (Claude Fable 5 agent, 2026-08-23; pending maintainer review)","us_full_run_20260612_policyengine_4_16_1_populace/us_case_reference_explanations.csv, stratified 5/output group, seed 20260818" +scenario_038,child1_early_head_start_eligible,person_early_head_start_eligible,0.0,"PolicyEngine determined that neither child in this Louisiana household is eligible for Early Head Start in 2026, resulting in a value of False for both children. Early Head Start eligibility is typically restricted to children under age 3, and this household's children do not meet the age requirement for the program. With a household income of approximately $22,992, the family's income level would generally fall within the range that qualifies for federally-funded early childhood programs, but age eligibility is the binding constraint in this case. Therefore, PolicyEngine correctly assigned an eligibility status of False to both child records in the household.",categorical_eligibility|age_disability|thresholds_rates,"developer adjudication (Claude Fable 5 agent, 2026-08-23; pending maintainer review)","us_full_run_20260612_policyengine_4_16_1_populace/us_case_reference_explanations.csv, stratified 5/output group, seed 20260818" +scenario_046,child1_early_head_start_eligible,person_early_head_start_eligible,0.0,"PolicyEngine determined that neither child in this household is eligible for Early Head Start, resulting in a value of False for both children. Early Head Start eligibility is primarily determined by income thresholds relative to the federal poverty line, and this household's income of approximately $92,003 exceeds the maximum income limit for a family of four in 2026. Since both children failed the income eligibility criterion, PolicyEngine assigned is_early_head_start_eligible a value of False for each household member, yielding the reference value of 0.0 when aggregated across the household.",categorical_eligibility|thresholds_rates,"developer adjudication (Claude Fable 5 agent, 2026-08-23; pending maintainer review)","us_full_run_20260612_policyengine_4_16_1_populace/us_case_reference_explanations.csv, stratified 5/output group, seed 20260818" +scenario_099,child1_early_head_start_eligible,person_early_head_start_eligible,0.0,"PolicyEngine determined that neither child in this California household is eligible for Early Head Start in 2026, resulting in a value of False for both children. Early Head Start eligibility is typically restricted to children under age 3 from families with incomes at or below 100% of the federal poverty line or receiving public assistance. With a household income of approximately $159,676, this family's earnings substantially exceed the income threshold required for the program. Consequently, PolicyEngine assigned an eligibility status of False to each child, yielding the reference value of 0.0 for the household.",categorical_eligibility|age_disability|thresholds_rates,"developer adjudication (Claude Fable 5 agent, 2026-08-23; pending maintainer review)","us_full_run_20260612_policyengine_4_16_1_populace/us_case_reference_explanations.csv, stratified 5/output group, seed 20260818" +scenario_100,child1_early_head_start_eligible,person_early_head_start_eligible,0.0,"PolicyEngine determined that neither of the two children in this Montana household are eligible for Early Head Start, resulting in an eligibility array of [False, False, False]. Early Head Start is a federal program that serves infants and toddlers from low-income families, typically those under age 3 and with household income at or below 100% of the federal poverty line. Although this household's income of approximately $5,915 is well below the poverty threshold, the children's ages do not meet the program's age requirements for Early Head Start participation. The computation therefore returned False for each child's eligibility status.",categorical_eligibility|age_disability|thresholds_rates,"developer adjudication (Claude Fable 5 agent, 2026-08-23; pending maintainer review)","us_full_run_20260612_policyengine_4_16_1_populace/us_case_reference_explanations.csv, stratified 5/output group, seed 20260818" +scenario_008,child1_head_start_eligible,person_head_start_eligible,0.0,"PolicyEngine determined that none of the eight household members are eligible for Head Start, resulting in a value of False for child1_head_start_eligible. Head Start eligibility is primarily determined by household income relative to the federal poverty line, and this household's income of approximately $30,915 exceeds the income threshold for a family of this size in New Jersey for the 2026 tax year. Since the household income surpasses the maximum allowable threshold, all eight members—including the six children who would otherwise be the target population for Head Start services—were marked as ineligible. The uniform False designation across all household members reflects that income-based disqualification applies to the entire family unit.",categorical_eligibility|thresholds_rates,"developer adjudication (Claude Fable 5 agent, 2026-08-23; pending maintainer review)","us_full_run_20260612_policyengine_4_16_1_populace/us_case_reference_explanations.csv, stratified 5/output group, seed 20260818" +scenario_028,child3_head_start_eligible,person_head_start_eligible,0.0,"PolicyEngine determined that none of the four household members are eligible for Head Start, resulting in a value of `[False, False, False, False]`. Head Start eligibility in PolicyEngine is typically determined by comparing household income against the federal poverty line threshold—generally set at 130% of the federal poverty level for the program year. With a household income of approximately $61,277 in Pennsylvania for 2026, this household's income exceeds the income threshold required for Head Start eligibility, which would be substantially lower for a household of this size. Consequently, all four individuals in the household (the head of household and three children) were marked as ineligible for the program.",categorical_eligibility|thresholds_rates,"developer adjudication (Claude Fable 5 agent, 2026-08-23; pending maintainer review)","us_full_run_20260612_policyengine_4_16_1_populace/us_case_reference_explanations.csv, stratified 5/output group, seed 20260818" +scenario_100,child1_head_start_eligible,person_head_start_eligible,0.0,"PolicyEngine determined that neither child in this Montana household is eligible for Head Start in 2026, resulting in a value of False for both children. Head Start eligibility is primarily determined by federal income thresholds, which are typically set at or below 100% of the federal poverty line, though some programs allow up to 130% of poverty. With a household income of approximately $5,915 and two children, this household's income-to-poverty ratio likely exceeded the program's eligibility threshold, disqualifying both children from Head Start enrollment. The resulting array [False, False, False] reflects the ineligibility determination for each household member evaluated against the program's income and other qualifying criteria.",categorical_eligibility|thresholds_rates,"developer adjudication (Claude Fable 5 agent, 2026-08-23; pending maintainer review)","us_full_run_20260612_policyengine_4_16_1_populace/us_case_reference_explanations.csv, stratified 5/output group, seed 20260818" +scenario_117,child3_head_start_eligible,person_head_start_eligible,0.0,"PolicyEngine determined that the third child in this Arkansas household is not eligible for Head Start (value: False). Head Start eligibility is primarily determined by income thresholds, and this household's income of approximately $250,733 substantially exceeds the federal poverty guidelines used to qualify families for the program. With a household income at this level, the family does not meet the income-based eligibility requirements that Head Start programs use to target services toward low-income families. Consequently, PolicyEngine assigned a value of False for the is_head_start_eligible variable for the third child, along with the other children in the household.",categorical_eligibility|thresholds_rates,"developer adjudication (Claude Fable 5 agent, 2026-08-23; pending maintainer review)","us_full_run_20260612_policyengine_4_16_1_populace/us_case_reference_explanations.csv, stratified 5/output group, seed 20260818" +scenario_119,child2_head_start_eligible,person_head_start_eligible,0.0,"PolicyEngine determined that neither child in this Virginia household is eligible for Head Start in 2026, resulting in a value of False for both children. Head Start eligibility is primarily determined by federal income thresholds, which are typically set at or below 100% of the federal poverty line, though some programs allow up to 130% of poverty. With a household income of approximately $55,800 for a family of three (one adult and two children), this household's income exceeds the federal poverty line threshold for a family of that size, placing them above the income cutoff for Head Start eligibility. Consequently, PolicyEngine marked both children as ineligible (False) for the program.",categorical_eligibility|thresholds_rates,"developer adjudication (Claude Fable 5 agent, 2026-08-23; pending maintainer review)","us_full_run_20260612_policyengine_4_16_1_populace/us_case_reference_explanations.csv, stratified 5/output group, seed 20260818" +scenario_008,spouse_medicaid_eligible,person_medicaid_eligible,1.0,"PolicyEngine determined that the spouse is eligible for Medicaid in New Jersey for tax year 2026 based on her classification under the ACA adult expansion category. As a 41-year-old adult under age 65, she qualifies for Modified Adjusted Gross Income (MAGI)-based Medicaid eligibility rules. With a household MAGI income level at 0.55 times the Federal Poverty Level, she falls well below New Jersey's Medicaid income threshold for adults in the expansion category, which extends coverage to individuals up to 138% of FPL. Consequently, PolicyEngine assigned an is_medicaid_eligible value of True, indicating full eligibility for the program.",health_coverage|categorical_eligibility|age_disability|thresholds_rates,"developer adjudication (Claude Fable 5 agent, 2026-08-23; pending maintainer review)","us_full_run_20260612_policyengine_4_16_1_populace/us_case_reference_explanations.csv, stratified 5/output group, seed 20260818" +scenario_049,spouse_medicaid_eligible,person_medicaid_eligible,0.0,"PolicyEngine determined that the spouse is not eligible for Medicaid in New Hampshire for the 2026 tax year. The spouse, age 52, does not qualify through any of the available Medicaid eligibility pathways. With a Modified Adjusted Gross Income (MAGI) of 11.40 times the Federal Poverty Level, the spouse's income exceeds the threshold for MAGI-based categories such as the Affordable Care Act expansion group. The engine's medicaid_category variable returned NONE, indicating no qualifying pathway exists for this household member, resulting in an is_medicaid_eligible value of False.",health_coverage|categorical_eligibility|thresholds_rates,"developer adjudication (Claude Fable 5 agent, 2026-08-23; pending maintainer review)","us_full_run_20260612_policyengine_4_16_1_populace/us_case_reference_explanations.csv, stratified 5/output group, seed 20260818" +scenario_054,child1_medicaid_eligible,person_medicaid_eligible,1.0,"PolicyEngine determined that child1 is eligible for Medicaid in North Carolina for tax year 2026. The child, age 10, falls into the older child category (OLDER_CHILD) under North Carolina's Medicaid program. With a Modified Adjusted Gross Income (MAGI) of 0.58 times the Federal Poverty Level, child1's income is well below the eligibility threshold for this category. Based on this income level and categorical eligibility as an older child, PolicyEngine calculated is_medicaid_eligible as True.",health_coverage|categorical_eligibility|age_disability|thresholds_rates,"developer adjudication (Claude Fable 5 agent, 2026-08-23; pending maintainer review)","us_full_run_20260612_policyengine_4_16_1_populace/us_case_reference_explanations.csv, stratified 5/output group, seed 20260818" +scenario_085,head_medicaid_eligible,person_medicaid_eligible,0.0,"PolicyEngine determined that the head of household is not eligible for Medicaid in Pennsylvania in 2026, despite meeting the immigration status requirement. The person qualifies as immigration-eligible (having been in the US for 5 years), but this alone is insufficient for Medicaid eligibility. The engine evaluated two state-specific pathways—California's FFYP (Family Planning) program and Illinois's HBI (Health Benefits for Immigrants) program—both of which returned false, indicating the household does not qualify through these mechanisms. With a MAGI income of 3.69 times the federal poverty level and no qualifying Medicaid category identified, the person falls outside Pennsylvania's Medicaid eligibility pathways for 2026, resulting in an eligibility determination of False.",health_coverage|categorical_eligibility|thresholds_rates|state_local_rule,"developer adjudication (Claude Fable 5 agent, 2026-08-23; pending maintainer review)","us_full_run_20260612_policyengine_4_16_1_populace/us_case_reference_explanations.csv, stratified 5/output group, seed 20260818" +scenario_117,spouse_medicaid_eligible,person_medicaid_eligible,0.0,"PolicyEngine determined that the spouse is not eligible for Medicaid in Arkansas for 2026. The spouse, age 34, does not qualify through any of the five standard Medicaid eligibility pathways evaluated by the engine. With a Modified Adjusted Gross Income (MAGI) of 6.42 times the Federal Poverty Level, the spouse's income exceeds the thresholds for MAGI-based categories such as the adult expansion group, which typically covers non-elderly, non-pregnant adults up to 138% of FPL in expansion states. Additionally, the spouse does not meet categorical requirements for other pathways including pregnancy, disability, or other special populations. As a result, is_medicaid_eligible returns False, and the household receives no Medicaid benefit value for this individual.",health_coverage|categorical_eligibility|thresholds_rates|age_disability,"developer adjudication (Claude Fable 5 agent, 2026-08-23; pending maintainer review)","us_full_run_20260612_policyengine_4_16_1_populace/us_case_reference_explanations.csv, stratified 5/output group, seed 20260818" +scenario_025,spouse_medicare_eligible,person_medicare_eligible,0.0,"PolicyEngine determined that neither spouse in this Ohio household is Medicare eligible in 2026, resulting in a value of [False, False] for the `is_medicare_eligible` variable. Medicare eligibility is primarily determined by age, with individuals generally becoming eligible at age 65. Since this benchmark household consists of two adults with a combined household income of approximately $94,925, PolicyEngine's calculation indicates that both spouses fall below the age threshold required for Medicare eligibility. The reference value of 0.0 reflects that neither household member qualifies for Medicare benefits in the 2026 tax year under current eligibility rules.",health_coverage|age_disability,"developer adjudication (Claude Fable 5 agent, 2026-08-23; pending maintainer review)","us_full_run_20260612_policyengine_4_16_1_populace/us_case_reference_explanations.csv, stratified 5/output group, seed 20260818" +scenario_038,head_medicare_eligible,person_medicare_eligible,0.0,"PolicyEngine determined that the head of household is not Medicare eligible (is_medicare_eligible = False) for the 2026 tax year. Medicare eligibility in the United States is primarily determined by age, with individuals becoming eligible at age 65, and the benchmark household composition indicates that both adults are below this threshold. The array result [False, False, False, False] reflects eligibility status for all four household members—the two adults and two children—none of whom meet the age requirement for Medicare enrollment. Consequently, the reference value for head_medicare_eligible is 0.0, indicating no Medicare eligibility for the household head.",health_coverage|age_disability,"developer adjudication (Claude Fable 5 agent, 2026-08-23; pending maintainer review)","us_full_run_20260612_policyengine_4_16_1_populace/us_case_reference_explanations.csv, stratified 5/output group, seed 20260818" +scenario_046,spouse_medicare_eligible,person_medicare_eligible,0.0,"PolicyEngine determined that the spouse is not Medicare eligible in 2026 (spouse_medicare_eligible = 0.0) because the household's adults do not meet the age threshold for Medicare eligibility. Medicare eligibility in the United States is primarily determined by age, requiring individuals to be at least 65 years old, and the computation trace shows that is_medicare_eligible returned False for all four household members. Since neither adult in this Oklahoma joint household reaches the qualifying age of 65 in the 2026 tax year, PolicyEngine correctly assigned a value of 0.0 (False) to the spouse_medicare_eligible variable.",health_coverage|age_disability,"developer adjudication (Claude Fable 5 agent, 2026-08-23; pending maintainer review)","us_full_run_20260612_policyengine_4_16_1_populace/us_case_reference_explanations.csv, stratified 5/output group, seed 20260818" +scenario_086,child1_medicare_eligible,person_medicare_eligible,0.0,"PolicyEngine determined that child1_medicare_eligible equals False (represented as 0.0) for this Georgia household. Medicare eligibility is primarily determined by age, and the child in this household does not meet the age threshold of 65 years required for standard Medicare coverage. Additionally, the household does not qualify for Medicare based on disability or end-stage renal disease status, which are the other primary pathways to Medicare eligibility for individuals under 65. Therefore, PolicyEngine correctly classified the child as not Medicare-eligible for the 2026 tax year.",health_coverage|age_disability,"developer adjudication (Claude Fable 5 agent, 2026-08-23; pending maintainer review)","us_full_run_20260612_policyengine_4_16_1_populace/us_case_reference_explanations.csv, stratified 5/output group, seed 20260818" +scenario_109,head_medicare_eligible,person_medicare_eligible,0.0,"PolicyEngine determined that the head of household is not Medicare eligible (False) for the 2026 tax year. Medicare eligibility in the United States is primarily determined by age, requiring individuals to be at least 65 years old, and the household composition indicates this is a working-age family with two adults and three children earning approximately $30,000 annually. Since neither adult in this Florida household meets the age threshold for Medicare eligibility, PolicyEngine correctly assigned a value of False to the is_medicare_eligible variable for the head of household. The array output [False, False, False, False, False] reflects the eligibility status for all five household members, confirming that none qualify for Medicare based on age requirements.",health_coverage|age_disability,"developer adjudication (Claude Fable 5 agent, 2026-08-23; pending maintainer review)","us_full_run_20260612_policyengine_4_16_1_populace/us_case_reference_explanations.csv, stratified 5/output group, seed 20260818" +scenario_005,head_wic_eligible,person_wic_eligible,0.0,"PolicyEngine determined that neither adult in this California household is eligible for WIC (Women, Infants, and Children benefits), resulting in an `is_wic_eligible` value of [False, False]. WIC eligibility requires that a household member be pregnant, postpartum, breastfeeding, or a child under age 5, and that household income fall below 185% of the federal poverty line. This household has no children and no pregnant or postpartum members, which disqualifies it from WIC eligibility regardless of the household's income level of approximately $585,777. Therefore, PolicyEngine correctly computed that both household members have zero WIC eligibility status.",categorical_eligibility|age_disability|thresholds_rates|household_unit_or_filing_status,"developer adjudication (Claude Fable 5 agent, 2026-08-23; pending maintainer review)","us_full_run_20260612_policyengine_4_16_1_populace/us_case_reference_explanations.csv, stratified 5/output group, seed 20260818" +scenario_031,head_wic_eligible,person_wic_eligible,0.0,"PolicyEngine determined that this 67-year-old California resident is not eligible for WIC despite meeting multiple eligibility criteria. The household's countable income of $23,853.47—derived from social security ($9,445.47), pension income ($13,608), and retirement distributions ($800)—falls below the WIC income limit of $29,526 for a single-person household. The individual also meets categorical eligibility through Medicaid enrollment (qualifying as an optional senior under category 8) and demonstrates nutritional risk. However, WIC eligibility is restricted to specific age categories, and at age 67, this individual falls into WIC category 5, which does not qualify for the program. The combination of age-based categorical ineligibility overrides the otherwise satisfied income and nutritional risk requirements, resulting in a final eligibility determination of False.",categorical_eligibility|age_disability|thresholds_rates|taxable_income_or_deductions,"developer adjudication (Claude Fable 5 agent, 2026-08-23; pending maintainer review)","us_full_run_20260612_policyengine_4_16_1_populace/us_case_reference_explanations.csv, stratified 5/output group, seed 20260818" +scenario_046,child2_wic_eligible,person_wic_eligible,0.0,"PolicyEngine determined that the second child in this household is not eligible for WIC (Women, Infants, and Children benefits), resulting in a value of False. The household's annual income of approximately $92,003 exceeds the WIC income eligibility threshold, which typically caps at 185% of the federal poverty line. For a household of four in 2026, this threshold would be substantially lower than the reported household income. Since the household income surpasses the maximum allowable limit for WIC eligibility, none of the four household members—including the second child—qualify for the program, regardless of other potential eligibility factors such as age or nutritional risk status.",categorical_eligibility|thresholds_rates,"developer adjudication (Claude Fable 5 agent, 2026-08-23; pending maintainer review)","us_full_run_20260612_policyengine_4_16_1_populace/us_case_reference_explanations.csv, stratified 5/output group, seed 20260818" +scenario_077,head_wic_eligible,person_wic_eligible,0.0,"PolicyEngine determined that this 37-year-old single adult in Louisiana is not eligible for WIC (is_wic_eligible = False) because they do not meet the program's categorical eligibility requirements. While the household passed the income test with countable income of $26,006.18 against the limit of $29,526, WIC eligibility requires meeting categorical criteria based on age and life stage. The household's wic_category value of 5 indicates they fall outside WIC's target populations, which primarily serve pregnant women, postpartum women, breastfeeding women, infants, and children up to age 5. Although the household was assessed as being at nutritional risk (is_wic_at_nutritional_risk = True), this alone is insufficient without categorical eligibility, resulting in the final ineligibility determination.",categorical_eligibility|age_disability|thresholds_rates,"developer adjudication (Claude Fable 5 agent, 2026-08-23; pending maintainer review)","us_full_run_20260612_policyengine_4_16_1_populace/us_case_reference_explanations.csv, stratified 5/output group, seed 20260818" +scenario_109,child2_wic_eligible,person_wic_eligible,0.0,"PolicyEngine determined that child 2 is not eligible for WIC (Women, Infants, and Children) benefits in this Florida household for tax year 2026. The eligibility determination returned a boolean array across household members `[False, False, False, False, True]`, where the fifth position (child 2) shows `False`, indicating ineligibility. With a household income of approximately $30,000 and three children, the household likely exceeded the WIC income threshold, which is typically set at 185% of the federal poverty line. WIC eligibility is restricted to infants, children under age 5, and pregnant or postpartum women, so the specific ages of the children in this household and their relationship to the income limit determined that child 2 did not meet the program's requirements.",categorical_eligibility|age_disability|thresholds_rates,"developer adjudication (Claude Fable 5 agent, 2026-08-23; pending maintainer review)","us_full_run_20260612_policyengine_4_16_1_populace/us_case_reference_explanations.csv, stratified 5/output group, seed 20260818" +scenario_012,reduced_price_school_meals_eligible,reduced_price_school_meals_eligible,0.0,"PolicyEngine determined that this Mississippi household is not eligible for reduced-price school meals (reduced_price_school_meals_eligible = 0.0) because the household qualifies for the superior free meals tier instead. With an income of approximately $22,000 and a school_meal_fpg_ratio of 0.81 (meaning household income is 81% of the federal poverty guideline), the household falls well below the 130% income threshold for free meals. Additionally, the household meets categorical eligibility requirements, likely through SNAP or TANF participation. Since the household qualifies for free meals via the school_meal_tier classification, reduced-price eligibility is not applicable. The household receives a school_meal_net_subsidy of $1,130.96 annually through the free meals program.",categorical_eligibility|thresholds_rates,"developer adjudication (Claude Fable 5 agent, 2026-08-23; pending maintainer review)","us_full_run_20260612_policyengine_4_16_1_populace/us_case_reference_explanations.csv, stratified 5/output group, seed 20260818" +scenario_062,reduced_price_school_meals_eligible,reduced_price_school_meals_eligible,0.0,"PolicyEngine determined that this household is not eligible for reduced-price school meals, resulting in a value of 0. The household's income of approximately $37,642 translates to a school_meal_fpg_ratio of 3.86, meaning the household income is 386% of the federal poverty guideline—well above the 185% threshold required for reduced-price meal eligibility. Since Florida does not have universal free school meals and the household does not meet categorical eligibility through programs like SNAP or TANF, the household is assigned a school_meal_tier of PAID and receives no subsidy (school_meal_net_subsidy = 0). Consequently, the household must pay the full price for school meals.",categorical_eligibility|thresholds_rates|state_local_rule,"developer adjudication (Claude Fable 5 agent, 2026-08-23; pending maintainer review)","us_full_run_20260612_policyengine_4_16_1_populace/us_case_reference_explanations.csv, stratified 5/output group, seed 20260818" +scenario_068,reduced_price_school_meals_eligible,reduced_price_school_meals_eligible,0.0,"PolicyEngine determined that this Maryland household is not eligible for reduced-price school meals, resulting in a value of 0. The household's income of approximately $34,084 places it at 6.71 times the federal poverty guideline, which exceeds the 185% income threshold required for reduced-price meal eligibility. Since Maryland does not have universal free school meals and the household does not qualify for categorical eligibility through programs like SNAP or TANF, the household is classified in the PAID school_meal_tier and receives no subsidy (school_meal_net_subsidy = 0). Consequently, the household must pay the full price for school meals.",categorical_eligibility|thresholds_rates|state_local_rule,"developer adjudication (Claude Fable 5 agent, 2026-08-23; pending maintainer review)","us_full_run_20260612_policyengine_4_16_1_populace/us_case_reference_explanations.csv, stratified 5/output group, seed 20260818" +scenario_075,reduced_price_school_meals_eligible,reduced_price_school_meals_eligible,0.0,"PolicyEngine determined that this household is not eligible for reduced-price school meals, resulting in a value of 0. The household's income of approximately $109,286 places it at 6.62 times the federal poverty guideline, far exceeding the 185% federal poverty level threshold required for reduced-price meal eligibility. Since Florida does not offer universal free school meals and the household does not qualify for categorical eligibility through programs like SNAP or TANF, the household is assigned to the PAID school_meal_tier and receives no subsidy (school_meal_net_subsidy = 0). Consequently, the household must pay the full price for school meals.",categorical_eligibility|thresholds_rates|state_local_rule,"developer adjudication (Claude Fable 5 agent, 2026-08-23; pending maintainer review)","us_full_run_20260612_policyengine_4_16_1_populace/us_case_reference_explanations.csv, stratified 5/output group, seed 20260818" +scenario_078,reduced_price_school_meals_eligible,reduced_price_school_meals_eligible,0.0,"PolicyEngine determined that this Maryland household is not eligible for reduced-price school meals, resulting in a value of 0. The household's income of approximately $200,137 places it at 12.44 times the federal poverty guideline, far exceeding the 185% federal poverty level threshold required for reduced-price meal eligibility. Since Maryland does not have universal free school meals and the household meets neither the income requirement nor categorical eligibility criteria (such as SNAP or TANF participation), the household is assigned to the PAID school_meal_tier. Consequently, the school_meal_net_subsidy is 0, and the household must pay the full price for school meals.",categorical_eligibility|thresholds_rates|state_local_rule,"developer adjudication (Claude Fable 5 agent, 2026-08-23; pending maintainer review)","us_full_run_20260612_policyengine_4_16_1_populace/us_case_reference_explanations.csv, stratified 5/output group, seed 20260818" +scenario_012,self_employment_tax,self_employment_tax,0.0,"PolicyEngine calculated a self-employment tax of $0 for this Mississippi household in 2026 because neither adult in the household reported self-employment income. Self-employment tax is only assessed on net earnings from self-employment activities, and since this joint-filing household with approximately $22,000 in total household income derived their earnings from sources other than self-employment (such as wages or other non-self-employment income), no self-employment tax liability was triggered. The calculation correctly reflects that self-employment tax applies only to individuals with qualifying self-employment income, which was absent in this case.",payroll_tax_base,"developer adjudication (Claude Fable 5 agent, 2026-08-23; pending maintainer review)","us_full_run_20260612_policyengine_4_16_1_populace/us_case_reference_explanations.csv, stratified 5/output group, seed 20260818" +scenario_023,self_employment_tax,self_employment_tax,0.0,"PolicyEngine calculated a self-employment tax of $0 for this California household because the household has no self-employment income to tax. Self-employment tax is only assessed on net earnings from self-employment activities, and since this benchmark household's approximately $25,443 in total income derives from other sources (such as wages or other non-self-employment income), there is no self-employment income base upon which to apply the 15.3% self-employment tax rate. Therefore, PolicyEngine correctly determined that no self-employment tax liability exists for this household in tax year 2026.",payroll_tax_base|thresholds_rates,"developer adjudication (Claude Fable 5 agent, 2026-08-23; pending maintainer review)","us_full_run_20260612_policyengine_4_16_1_populace/us_case_reference_explanations.csv, stratified 5/output group, seed 20260818" +scenario_037,self_employment_tax,self_employment_tax,0.0,"PolicyEngine calculated a self-employment tax of $0 for this North Carolina household in 2026 because the household has no self-employment income. Self-employment tax is only assessed on net earnings from self-employment activities, and since this benchmark household's approximately $41,943 in total income derives from sources other than self-employment (such as wages or other non-self-employment sources), there is no self-employment income base upon which to calculate the 15.3% self-employment tax rate. Therefore, PolicyEngine correctly assigned a self-employment tax liability of $0.",payroll_tax_base|thresholds_rates,"developer adjudication (Claude Fable 5 agent, 2026-08-23; pending maintainer review)","us_full_run_20260612_policyengine_4_16_1_populace/us_case_reference_explanations.csv, stratified 5/output group, seed 20260818" +scenario_043,self_employment_tax,self_employment_tax,0.0,"PolicyEngine calculated a self-employment tax of $0 for this Colorado household in 2026. With a household income of approximately $4,642, this income level falls below the self-employment tax threshold, which requires net self-employment income of at least $400 to trigger self-employment tax obligations. Since the household's income did not meet this minimum threshold, no self-employment tax was assessed.",payroll_tax_base|thresholds_rates,"developer adjudication (Claude Fable 5 agent, 2026-08-23; pending maintainer review)","us_full_run_20260612_policyengine_4_16_1_populace/us_case_reference_explanations.csv, stratified 5/output group, seed 20260818" +scenario_051,self_employment_tax,self_employment_tax,0.0,"PolicyEngine calculated a self-employment tax of $0 for this Louisiana household because the household has no self-employment income to tax. Self-employment tax is only assessed on net earnings from self-employment activities, and since this benchmark household's approximately $40,000 in household income does not derive from self-employment sources, there is no tax base upon which to apply the self-employment tax rate. Without self-employment income, the self-employment tax liability remains at zero.",payroll_tax_base,"developer adjudication (Claude Fable 5 agent, 2026-08-23; pending maintainer review)","us_full_run_20260612_policyengine_4_16_1_populace/us_case_reference_explanations.csv, stratified 5/output group, seed 20260818" +scenario_031,snap,snap,0.0,"PolicyEngine calculated a SNAP (Supplemental Nutrition Assistance Program) benefit of $0 for this California household in 2026. The household, consisting of a single adult with approximately $23,853 in annual income, was determined to be ineligible for SNAP benefits based on income limits or other eligibility criteria. Although the model indicates that the household would take up SNAP benefits if eligible (takes_up_snap_if_eligible = True), the household's income level places it above the threshold for SNAP eligibility, resulting in a benefit amount of zero. The repeated computation trace showing consistent zero values across multiple calculation nodes confirms that no SNAP assistance was available to this household under 2026 policy rules.",categorical_eligibility|thresholds_rates,"developer adjudication (Claude Fable 5 agent, 2026-08-23; pending maintainer review)","us_full_run_20260612_policyengine_4_16_1_populace/us_case_reference_explanations.csv, stratified 5/output group, seed 20260818" +scenario_059,snap,snap,0.0,"PolicyEngine calculated a SNAP (Supplemental Nutrition Assistance Program) benefit of $0 for this Florida household of two adults with no children and approximately $62,704 in annual household income. The household's income level exceeds the SNAP eligibility threshold for a two-person household without dependents, which is substantially lower than their reported income. Although the model indicates the household would take up SNAP benefits if eligible (takes_up_snap_if_eligible = True), the income disqualification prevents any benefit from being awarded. Therefore, despite the household's willingness to participate in the program, their earnings place them above the income limit, resulting in zero SNAP benefits for the 2026 tax year.",categorical_eligibility|thresholds_rates,"developer adjudication (Claude Fable 5 agent, 2026-08-23; pending maintainer review)","us_full_run_20260612_policyengine_4_16_1_populace/us_case_reference_explanations.csv, stratified 5/output group, seed 20260818" +scenario_081,snap,snap,0.0,"PolicyEngine calculated a SNAP (Supplemental Nutrition Assistance Program) benefit of $0 for this Massachusetts household in 2026. The household's annual income of approximately $174,088 substantially exceeds the income eligibility threshold for SNAP, which is set at 130% of the federal poverty line for most households. Since this single-adult household's income far surpasses the maximum allowable limit, the household does not qualify for any SNAP benefits in this tax year.",categorical_eligibility|thresholds_rates,"developer adjudication (Claude Fable 5 agent, 2026-08-23; pending maintainer review)","us_full_run_20260612_policyengine_4_16_1_populace/us_case_reference_explanations.csv, stratified 5/output group, seed 20260818" +scenario_093,snap,snap,0.0,"PolicyEngine calculated a SNAP (Supplemental Nutrition Assistance Program) benefit of $0 for this Missouri household of three adults with no children and approximately $164,156 in annual household income. Although the model determined that the household would take up SNAP if eligible (takes_up_snap_if_eligible = True), the household's income substantially exceeds the federal and state income limits for SNAP eligibility. With a gross monthly income of approximately $13,680, this household far surpasses Missouri's SNAP income threshold, which is typically set at 130% of the federal poverty line for most household compositions. As a result, despite the modeled willingness to participate in the program, the household receives no SNAP benefits because it does not meet the basic income eligibility requirement.",categorical_eligibility|thresholds_rates|period_annualization,"developer adjudication (Claude Fable 5 agent, 2026-08-23; pending maintainer review)","us_full_run_20260612_policyengine_4_16_1_populace/us_case_reference_explanations.csv, stratified 5/output group, seed 20260818" +scenario_109,snap,snap,8020.55419921875,"PolicyEngine calculates SNAP month by month and sums the twelve benefits. From January through September, the $261 standard deduction leaves $1,739 of net monthly income; subtracting the $521.70 expected contribution from the $1,183 maximum allotment produces about $661.30 per month. From October through December, updated fiscal-year parameters raise the standard deduction to $266.85 and the maximum allotment to $1,209.52, producing about $689.62 per month after the $519.90 expected contribution. The household passes the income, asset, and work-requirement tests throughout the year, and summing the unrounded monthly benefits yields $8,020.55.",period_annualization|taxable_income_or_deductions|thresholds_rates|categorical_eligibility|asset_resource,"developer adjudication (Claude Fable 5 agent, 2026-08-23; pending maintainer review)","us_full_run_20260612_policyengine_4_16_1_populace/us_case_reference_explanations.csv, stratified 5/output group, seed 20260818" +scenario_031,ssi,ssi,0.0,"PolicyEngine calculated an SSI benefit of $0.00 for this 67-year-old California resident in 2026. Although the household head meets SSI eligibility criteria (is_ssi_aged_blind_disabled=True) and demonstrates willingness to take up the benefit (takes_up_ssi_if_eligible=True), the resulting benefit amount is zero. This outcome reflects SSI's income-counting rules: with household income of approximately $23,853, the individual's countable income exceeds the federal benefit rate threshold, resulting in a complete phase-out of the SSI payment. The calculation properly treats SSI as an individual benefit, evaluating eligibility and income counting on a per-person basis rather than pooling resources across the household.",categorical_eligibility|age_disability|taxable_income_or_deductions|thresholds_rates|household_unit_or_filing_status,"developer adjudication (Claude Fable 5 agent, 2026-08-23; pending maintainer review)","us_full_run_20260612_policyengine_4_16_1_populace/us_case_reference_explanations.csv, stratified 5/output group, seed 20260818" +scenario_059,ssi,ssi,0.0,"PolicyEngine calculated SSI of $0.00 for this household because the couple's countable income far exceeds the benefit amount they would otherwise receive. Both the head (age 79) and spouse (age 78) meet SSI eligibility requirements as aged individuals and pass the resource test with only $40 in countable resources. Under the couple computation rules, PolicyEngine determined an uncapped SSI amount of $1,491 (split as $745.50 per person). However, the household's countable income of $62,464—derived from $62,704 in unearned income with standard SSI deductions applied—results in a negative uncapped benefit of -$3,714.33 per person, which is capped at zero. Since the countable income substantially exceeds the maximum SSI payment threshold, both individuals receive $0.00 in SSI benefits despite meeting all other eligibility criteria.",categorical_eligibility|age_disability|asset_resource|taxable_income_or_deductions|thresholds_rates|household_unit_or_filing_status,"developer adjudication (Claude Fable 5 agent, 2026-08-23; pending maintainer review)","us_full_run_20260612_policyengine_4_16_1_populace/us_case_reference_explanations.csv, stratified 5/output group, seed 20260818" +scenario_081,ssi,ssi,0.0,"PolicyEngine calculated SSI of $0.00 for this Massachusetts household in 2026 because the single adult head of household, age 30, does not meet any of the categorical eligibility requirements for Supplemental Security Income. SSI eligibility requires that a household member be aged 65 or older, blind, or disabled; since the head of household is none of these (is_ssi_aged_blind_disabled=False), the household is ineligible for SSI regardless of income level. Although the engine's computation trace shows that takes_up_ssi_if_eligible was set to True, indicating the household would claim SSI if eligible, this parameter does not affect the outcome when no household member satisfies the underlying categorical criteria. Therefore, the SSI benefit amount is $0.00.",categorical_eligibility|age_disability,"developer adjudication (Claude Fable 5 agent, 2026-08-23; pending maintainer review)","us_full_run_20260612_policyengine_4_16_1_populace/us_case_reference_explanations.csv, stratified 5/output group, seed 20260818" +scenario_093,ssi,ssi,0.0,"PolicyEngine calculated SSI (Supplemental Security Income) as $0.00 for this Missouri household in 2026 because none of the three household members meet SSI's categorical eligibility requirements. The head (age 51), spouse (age 49), and dependent (age 27) are all below the aged threshold of 65 and have no recorded status as blind or disabled under the engine's inputs, which are the only pathways to SSI eligibility. Although all three household members indicated willingness to take up SSI if eligible (takes_up_ssi_if_eligible = True for each), the absence of any qualifying aged, blind, or disabled status means no benefit amount is calculated for any individual. SSI is computed on a per-person basis with individual eligibility determinations, so the household's combined income of approximately $164,156 is not the limiting factor—rather, the categorical ineligibility of each member results in the $0.00 household total.",categorical_eligibility|age_disability|household_unit_or_filing_status,"developer adjudication (Claude Fable 5 agent, 2026-08-23; pending maintainer review)","us_full_run_20260612_policyengine_4_16_1_populace/us_case_reference_explanations.csv, stratified 5/output group, seed 20260818" +scenario_109,ssi,ssi,0.0,"PolicyEngine calculated total household SSI for 2026 as $0.00 because none of the five household members—the head (age 30), spouse (age 29), or the three children (ages 9, 6, and 0)—meet SSI's eligibility criteria. SSI is an individual benefit that requires recipients to be aged 65 or older, blind, or disabled; the engine's inputs indicate that all five household members have `is_ssi_aged_blind_disabled=False`, meaning none qualify on these grounds. Although the computation trace shows that all household members have `takes_up_ssi_if_eligible=True`, indicating they would claim the benefit if eligible, the absence of any qualifying condition results in $0.00 SSI for each person and therefore $0.00 for the household overall.",categorical_eligibility|age_disability|household_unit_or_filing_status,"developer adjudication (Claude Fable 5 agent, 2026-08-23; pending maintainer review)","us_full_run_20260612_policyengine_4_16_1_populace/us_case_reference_explanations.csv, stratified 5/output group, seed 20260818" +scenario_023,state_income_tax_before_refundable_credits,state_income_tax_before_refundable_credits,6.7984771728515625,"For this California resident with approximately $25,443 in household income, PolicyEngine calculated a state income tax before refundable credits of $6.80. The calculation began with an adjusted gross income of $22,534.34, from which California's standard deduction of $5,706 was subtracted to arrive at a taxable income of $16,828.34. This taxable income was then subject to California's tax brackets, resulting in a tax before credits of $223.27. Finally, PolicyEngine applied non-refundable credits totaling $216.47—comprising $156.47 in exemptions and $60.00 in renter credits—which reduced the tax liability to the final amount of $6.80.",state_local_rule|taxable_income_or_deductions|thresholds_rates|credit_phaseout,"developer adjudication (Claude Fable 5 agent, 2026-08-23; pending maintainer review)","us_full_run_20260612_policyengine_4_16_1_populace/us_case_reference_explanations.csv, stratified 5/output group, seed 20260818" +scenario_031,state_income_tax_before_refundable_credits,state_income_tax_before_refundable_credits,0.0,"For this California resident with approximately $23,853 in household income, PolicyEngine calculated a state income tax before refundable credits of $0. The calculation began with an adjusted gross income of $13,243, which was reduced by California deductions totaling $5,706 (comprised of the standard deduction of $5,706 and itemized deductions of $1,441.57 for medical expenses), resulting in a taxable income of $7,537. This taxable income generated a tax before credits of $75.37; however, this liability was fully eliminated by non-refundable credits of $312.93, which included California exemptions based on the household member's aged or blind status. Since the credits exceeded the tax liability, the final state income tax before refundable credits was reduced to $0.",state_local_rule|taxable_income_or_deductions|thresholds_rates|credit_phaseout|age_disability,"developer adjudication (Claude Fable 5 agent, 2026-08-23; pending maintainer review)","us_full_run_20260612_policyengine_4_16_1_populace/us_case_reference_explanations.csv, stratified 5/output group, seed 20260818" +scenario_081,state_income_tax_before_refundable_credits,state_income_tax_before_refundable_credits,8238.40625,"PolicyEngine calculated the Massachusetts state income tax before refundable credits at $8,238.41 for this single adult household in Massachusetts with approximately $174,088 in income. The calculation applied Massachusetts's two-part tax structure: Part A taxes on dividend income of $110.12, and Part B taxes on ordinary income. For Part B, PolicyEngine determined taxable income of $164,658 by starting with Part B taxable income before exemption of $169,058 and subtracting the Massachusetts personal exemption of $4,400. The resulting tax liability of $8,238.41 reflects the application of Massachusetts's progressive tax rates to these taxable income components. All other states returned zero tax liability, confirming that only Massachusetts income tax applied to this resident household.",state_local_rule|taxable_income_or_deductions|thresholds_rates,"developer adjudication (Claude Fable 5 agent, 2026-08-23; pending maintainer review)","us_full_run_20260612_policyengine_4_16_1_populace/us_case_reference_explanations.csv, stratified 5/output group, seed 20260818" +scenario_093,state_income_tax_before_refundable_credits,state_income_tax_before_refundable_credits,3389.65087890625,"PolicyEngine calculated Missouri state income tax before refundable credits of $3,389.65 for this joint-filing household with three adults and ~$164,156 in household income. The calculation began with IRS gross income of $115,097.88, which was reduced by above-the-line deductions of $2,689.62 to arrive at Missouri adjusted gross income of $112,408.25. From this amount, Missouri itemized deductions of $12,469.50 were subtracted—comprising the standard deduction of $32,200, Missouri's deduction for federal income taxes paid of $431.45, and allowable itemized deductions including state taxes withheld of $3,409.93—yielding Missouri taxable income of $79,776.80. The tax was then calculated on this taxable income using Missouri's tax brackets and rates, resulting in a total tax of $3,389.65 distributed across the two primary earners in the household ($1,453.04 and $1,936.62 respectively), with the third household member exempt from Missouri income tax.",state_local_rule|taxable_income_or_deductions|thresholds_rates|household_unit_or_filing_status,"developer adjudication (Claude Fable 5 agent, 2026-08-23; pending maintainer review)","us_full_run_20260612_policyengine_4_16_1_populace/us_case_reference_explanations.csv, stratified 5/output group, seed 20260818" +scenario_109,state_income_tax_before_refundable_credits,state_income_tax_before_refundable_credits,0.0,"PolicyEngine calculated state_income_tax_before_refundable_credits as $0.00 for this Florida household by aggregating the income tax liabilities across all 50 states plus Washington D.C. and New York City. Since the household is located in Florida, the fl_income_tax_before_refundable_credits component evaluated to $0.00, reflecting Florida's lack of a state income tax. All other state-level income tax variables (al_income_tax_before_refundable_credits, ar_income_tax_before_refundable_credits, and so forth through wv_income_tax_before_refundable_credits) also returned $0.00, as PolicyEngine only calculates tax liability for the household's resident state. The sum of these components—zero from Florida plus zero from all non-resident states—produced the final value of $0.00 for total state income tax before refundable credits.",state_local_rule,"developer adjudication (Claude Fable 5 agent, 2026-08-23; pending maintainer review)","us_full_run_20260612_policyengine_4_16_1_populace/us_case_reference_explanations.csv, stratified 5/output group, seed 20260818" +scenario_031,state_refundable_credits,state_refundable_credits,0.0,"PolicyEngine calculated state_refundable_credits as $0 for this California resident with approximately $23,853 in household income. The computation aggregates refundable tax credits across all 50 states and select localities, with ca_refundable_credits—the California-specific component—evaluating to $0 since the household resides in California. All other state-level refundable credit variables (al_refundable_credits, ar_refundable_credits, az_refundable_credits, and so forth across all remaining states and NYC) also returned $0, as the household is not subject to those jurisdictions' tax regimes. The sum of these zero values across all state and local components yields the final result of $0 in total state refundable credits for tax year 2026.",state_local_rule|credit_phaseout,"developer adjudication (Claude Fable 5 agent, 2026-08-23; pending maintainer review)","us_full_run_20260612_policyengine_4_16_1_populace/us_case_reference_explanations.csv, stratified 5/output group, seed 20260818" +scenario_059,state_refundable_credits,state_refundable_credits,0.0,"PolicyEngine calculated state_refundable_credits as $0 for this Florida household in 2026 by aggregating refundable tax credits across all 50 states and the District of Columbia. Since the household is located in Florida (fl_refundable_credits), which has no state refundable tax credits, and all other state-level refundable credit variables (including al_refundable_credits, ca_refundable_credits, ny_refundable_credits, and 38 others) evaluated to $0, the sum of all state refundable credits equals $0. The household's income level of approximately $62,704 and filing status as a joint return do not qualify them for any refundable credits in their state of residence or any other state. Therefore, the total state_refundable_credits value is $0.",state_local_rule|credit_phaseout,"developer adjudication (Claude Fable 5 agent, 2026-08-23; pending maintainer review)","us_full_run_20260612_policyengine_4_16_1_populace/us_case_reference_explanations.csv, stratified 5/output group, seed 20260818" +scenario_081,state_refundable_credits,state_refundable_credits,0.0,"PolicyEngine calculated state_refundable_credits as $0 for this Massachusetts resident with ~$174,088 household income in 2026. The computation aggregates refundable tax credits across all 50 states and the District of Columbia, with ma_refundable_credits specifically evaluated for this household since the residence state is Massachusetts (MA = True). Since ma_refundable_credits returned $0, and all other state-level refundable credit variables (al_refundable_credits, ar_refundable_credits, ca_refundable_credits, and so forth) also evaluated to $0, the sum of all state refundable credits equals $0. This result indicates that either Massachusetts offers no refundable credits applicable to this household's income level and filing status, or the household does not qualify for any available refundable credits in the 2026 tax year.",state_local_rule|credit_phaseout,"developer adjudication (Claude Fable 5 agent, 2026-08-23; pending maintainer review)","us_full_run_20260612_policyengine_4_16_1_populace/us_case_reference_explanations.csv, stratified 5/output group, seed 20260818" +scenario_093,state_refundable_credits,state_refundable_credits,0.0,"PolicyEngine calculated state_refundable_credits for this Missouri household by summing refundable tax credits across all 50 states and the District of Columbia. Since the household is located in Missouri (MO = True), PolicyEngine evaluated mo_refundable_credits, which returned 0, along with all other state-level refundable credit variables (al_refundable_credits, ar_refundable_credits, ca_refundable_credits, and so on through wv_refundable_credits), each of which also evaluated to 0. The sum of all these state and local refundable credit components—0 + 0 + 0 + ... + 0 across all jurisdictions—resulted in a total state_refundable_credits value of 0.0 for the tax year 2026.",state_local_rule|credit_phaseout,"developer adjudication (Claude Fable 5 agent, 2026-08-23; pending maintainer review)","us_full_run_20260612_policyengine_4_16_1_populace/us_case_reference_explanations.csv, stratified 5/output group, seed 20260818" +scenario_109,state_refundable_credits,state_refundable_credits,0.0,"PolicyEngine calculated state_refundable_credits as $0 for this Florida household in 2026 by aggregating refundable tax credits across all 50 states and the District of Columbia. Since the household is located in Florida (fl_refundable_credits), PolicyEngine evaluated whether any state-level refundable credits applied, finding that Florida offers no refundable tax credits for this household composition and income level. All other state refundable credit variables—including those for high-credit states like California, New York, and Colorado—returned $0 because the household does not reside in those jurisdictions. The final state_refundable_credits value of $0 reflects the sum of all individual state contributions, with no offsetting refundable credits available at the state level for this benchmark family.",state_local_rule|credit_phaseout,"developer adjudication (Claude Fable 5 agent, 2026-08-23; pending maintainer review)","us_full_run_20260612_policyengine_4_16_1_populace/us_case_reference_explanations.csv, stratified 5/output group, seed 20260818" +scenario_018,tanf,tanf,0.0,"PolicyEngine calculated a TANF (Temporary Assistance for Needy Families) benefit of $0 for this Arizona household in 2026. This result reflects that the household, consisting of a single adult with no children and an annual income of approximately $68,055, does not qualify for TANF assistance. TANF is primarily designed to provide cash assistance to families with dependent children, and this household's composition and income level place it outside the program's eligibility parameters. Consequently, PolicyEngine's computation trace shows no intermediate calculations, as the household fails to meet the basic eligibility requirements that would trigger benefit determination.",categorical_eligibility|household_unit_or_filing_status,"developer adjudication (Claude Fable 5 agent, 2026-08-23; pending maintainer review)","us_full_run_20260612_policyengine_4_16_1_populace/us_case_reference_explanations.csv, stratified 5/output group, seed 20260818" +scenario_031,tanf,tanf,0.0,"PolicyEngine calculated a TANF (Temporary Assistance for Needy Families) benefit of $0 for this California household in 2026. The household consists of a single adult with no children and an annual income of approximately $23,853. TANF eligibility and benefit amounts are primarily determined by household composition and income thresholds, with the program designed to serve families with dependent children. Since this household contains no children, it does not meet the basic eligibility requirements for TANF benefits in California, resulting in a benefit value of zero.",categorical_eligibility|household_unit_or_filing_status|thresholds_rates,"developer adjudication (Claude Fable 5 agent, 2026-08-23; pending maintainer review)","us_full_run_20260612_policyengine_4_16_1_populace/us_case_reference_explanations.csv, stratified 5/output group, seed 20260818" +scenario_059,tanf,tanf,0.0,"PolicyEngine calculated a TANF (Temporary Assistance for Needy Families) benefit of $0 for this Florida household in 2026. TANF is a needs-based program primarily designed for families with children, and this benchmark household consists of two adults with no children, making them ineligible for the program. Since the household does not meet the basic eligibility criteria requiring dependent children, PolicyEngine correctly determined that no TANF benefit applies regardless of their income level of approximately $62,704.",categorical_eligibility|household_unit_or_filing_status,"developer adjudication (Claude Fable 5 agent, 2026-08-23; pending maintainer review)","us_full_run_20260612_policyengine_4_16_1_populace/us_case_reference_explanations.csv, stratified 5/output group, seed 20260818" +scenario_081,tanf,tanf,0.0,"PolicyEngine calculated a TANF (Temporary Assistance for Needy Families) benefit of $0 for this Massachusetts household in 2026. The household, consisting of a single adult with no children and an annual income of approximately $174,088, does not qualify for TANF benefits because the program is designed to provide cash assistance to families with dependent children. Since this household has zero children, it falls outside the eligibility criteria for TANF, resulting in no benefit payment regardless of income level.",categorical_eligibility|household_unit_or_filing_status,"developer adjudication (Claude Fable 5 agent, 2026-08-23; pending maintainer review)","us_full_run_20260612_policyengine_4_16_1_populace/us_case_reference_explanations.csv, stratified 5/output group, seed 20260818" +scenario_109,tanf,tanf,0.0,"PolicyEngine calculated a Temporary Assistance for Needy Families (TANF) benefit of $0 for this Florida household in 2026. Despite the household's modest income of approximately $30,000 with three children, the household did not qualify for TANF benefits under Florida's program rules. This result reflects either that the household's income exceeded Florida's TANF eligibility threshold, or that other household characteristics (such as work requirements, citizenship status, or asset limits) rendered them ineligible for assistance in that tax year. The zero benefit indicates no TANF payment was due to this household under the applicable 2026 Florida TANF policy parameters.",categorical_eligibility|thresholds_rates|state_local_rule|asset_resource,"developer adjudication (Claude Fable 5 agent, 2026-08-23; pending maintainer review)","us_full_run_20260612_policyengine_4_16_1_populace/us_case_reference_explanations.csv, stratified 5/output group, seed 20260818" diff --git a/docs/runbook.md b/docs/runbook.md index 62c124f..680e206 100644 --- a/docs/runbook.md +++ b/docs/runbook.md @@ -1,6 +1,10 @@ # Benchmark Runbook -This is the canonical procedure for paid no-tools benchmark runs. Treat +This is the canonical procedure for paid no-tools benchmark runs. The +stability suite (repeated runs, reasoning-stability judging, counterfactual +twins) has its own spec and runbook in +[stability_spec.md](stability_spec.md) — note that repeated runs are +deliberately cache-free, unlike the single-run commands below. Treat `results/local/` as scratch space; git history and release snapshots are the archive, not superseded local files. diff --git a/docs/stability_spec.md b/docs/stability_spec.md index d67e96c..2acea94 100644 --- a/docs/stability_spec.md +++ b/docs/stability_spec.md @@ -235,6 +235,48 @@ cache guard, and emits `stability_metadata.json` including each run's effective serving config diffed against `paper/snapshot/20260501/model_serving_config.json`. +Layers 2 and 3 continue from the same repeats: + +```bash +# Layer 2 (judge spend only; --deterministic-only for the free channels). +uv run policybench reasoning-stability \ + --runs-dir "$RUN_DIR/us/runs/openai" \ + --runs-dir "$RUN_DIR/us/runs/gemini" \ + -g "$FROZEN_REFS" \ + -o "$RUN_DIR/us/reasoning" + +# Layer 3 truth arm (free, local PolicyEngine; ~3 minutes). +uv run policybench counterfactual-manifest -o "$RUN_DIR/us/cf" + +# Layer 3 model arm (paid): run the perturbed manifest once per model with +# the ordinary runner, then report against the repeats as the base arm. +uv run policybench eval-no-tools-chunked \ + --country us \ + --scenario-manifest "$RUN_DIR/us/cf/cf_scenarios.csv" \ + --output-dir "$RUN_DIR/us/cf_arm" \ + --model gpt-5.4-mini --model gemini-3.7-flash \ + --chunk-size 5 --parallel 1 --model-parallel 1 --chunk-attempts 1 + +uv run policybench counterfactual-report \ + --perturbed-predictions "$RUN_DIR/us/cf_arm/predictions.csv" \ + --truth-deltas "$RUN_DIR/us/cf/truth_deltas.csv" \ + --base-runs-dir "$RUN_DIR/us/runs/openai" \ + --base-runs-dir "$RUN_DIR/us/runs/gemini" \ + -o "$RUN_DIR/us/cf_report" + +# Price any rung from logged usage before spending. +uv run policybench stability-cost-plan \ + -p paper/snapshot/20260501/runs/us_full_run_20260612_policyengine_4_16_1_populace/predictions.csv.gz \ + --repeats 3 --cf-arms 1 +``` + +The rung-0 dry run of these commands (2026-08-23, free): the truth arm +reproduces the measured distribution exactly (247 nonzero amount rows, 0 +binary flips, zero-delta share 0.8755), and the cost plan prices the +30-model roster at $1,302 for 4 arms + ≈$140 judge — Claude Fable 5 +reported `unpriced` (its snapshot usage is run-level) and priced +separately at ≈$216/4 arms, consistent with the ladder's ≈$1,520. + ## Layer 2 — reasoning stability **Question**: when a model gives the same answer twice, does it cite the @@ -312,9 +354,14 @@ wage base maps to `{payroll_tax_base, thresholds_rates}`). ### Validation battery (the load-bearing control) -1. **Gold-set validity gate.** A committed, stratified gold set of ~100 - explanations (drawn from reference explanations across output groups), - labeled by the developers and versioned in-repo. The gate: judge +1. **Gold-set validity gate.** A committed, stratified gold set of 90 + explanations — 5 per output group from the reference-explanation CSV, + sampled with seed 20260818 — labeled against the taxonomy and versioned + at `annotations/stability_reasoning_gold_set.csv` (labeled 2026-08-23 by + the maintainer agent as developer adjudication, pending maintainer + review; label prevalence ranges from thresholds_rates 59/90 to + period_annualization 2/90, with `other` at zero as expected for + reference explanations). The gate: judge exact-set accuracy vs gold ≥ 0.80, flagged CI-aware (fail when the 95% lower bound sits below 0.70). Per-label agreement and prevalence tables publish alongside. Thresholds are provisional until the rung-1 From e372f344246d6f7ff2802794e647cf58fe8954e5 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Tue, 25 Aug 2026 18:15:46 +0200 Subject: [PATCH 6/7] Refresh spec cost basis for the 32-model refreeze (grok-4.6, ox-alpha) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Rebased onto main's 2026-08-22 board refreeze: top-8 re-derived (ox-alpha fourth, grok-4.6 eighth; grok-4.5 and fable drop out), rung totals recomputed via stability-cost-plan ($1,337 priced + unpriced ox-alpha + Fable ≈$216, judge ≈$149), ox-alpha reported unpriced rather than guessed. Co-Authored-By: Claude Fable 5 --- docs/stability_spec.md | 24 +++++++++++++++++------- 1 file changed, 17 insertions(+), 7 deletions(-) diff --git a/docs/stability_spec.md b/docs/stability_spec.md index 2acea94..d534a55 100644 --- a/docs/stability_spec.md +++ b/docs/stability_spec.md @@ -48,7 +48,8 @@ PolicyBench is exposed in three places: Everything ships under the frozen-board discipline of `sensitivity/claude-thinking-2026-08.md` and [policybench#139](https://github.com/PolicyEngine/policybench/issues/139): -the 30-model board is never edited; stability results publish as labeled +frozen board scores are never edited (32 models as of the 2026-08-22 +refreeze that added Grok 4.6 and Ox Alpha); stability results publish as labeled sensitivity artifacts (a `sensitivity/*.md` doc plus release-attached CSVs), or fold into the v2 board protocol as a versioned re-run. @@ -273,9 +274,10 @@ uv run policybench stability-cost-plan \ The rung-0 dry run of these commands (2026-08-23, free): the truth arm reproduces the measured distribution exactly (247 nonzero amount rows, 0 binary flips, zero-delta share 0.8755), and the cost plan prices the -30-model roster at $1,302 for 4 arms + ≈$140 judge — Claude Fable 5 -reported `unpriced` (its snapshot usage is run-level) and priced -separately at ≈$216/4 arms, consistent with the ladder's ≈$1,520. +32-model roster at $1,337 for 4 arms + ≈$149 judge — Claude Fable 5 +and Ox Alpha report `unpriced` (run-level and stealth-preview usage +respectively) rather than guessed; Fable prices separately at ≈$216/4 +arms, consistent with the ladder's ≈$1,550+. ## Layer 2 — reasoning stability @@ -558,7 +560,11 @@ reported): One full-roster board-condition run ≈ $316 recorded + Claude Fable 5 ≈$54.10 (run-level usage; $0.541/hh) + two unpriced models at override -prices (grok-build-0.1 ≈$4.50, gemini-3.6-flash ≈$5.50) ≈ **$380/run**. +prices (grok-build-0.1 ≈$4.50, gemini-3.6-flash ≈$5.50) ≈ **$380/run** +for the original 30 models; the 2026-08-22 refreeze adds grok-4.6 +($8.70/run logged) and ox-alpha (stealth preview, no logged or listed +price — reported unpriced, never guessed), bringing the priced roster +total to ≈$389/run + Fable ≈$54. Repeats run cache-free, so arms price at full run cost. Judge basis: ≈800 input + 30 output tokens/call at gemini-3.7-flash overrides ($0.75/$3.75 per 1M) = $0.0007125/call; 5,952 calls per model per K=3 @@ -570,8 +576,12 @@ pilot reports the realized rate); one-time reference-anchor extraction |---|---|---|---|---| | 0 (this PR) | mocks + fixtures; free local truth-deltas | $0 | $0 | metrics, tests, dry run | | 1 pilot | gpt-5.4-mini ($0.58/run) + gemini-3.7-flash ($1.44/run) × (3 repeats + 1 cf arm) | ≈ $8 | ≈ $17 double-graded (2 models × 3 runs × 1,984 × 2 passes), plus $1.41 anchors | end-to-end validation; gold-set + cross-judge gates | -| 2 subset | top-8 board models as ranked (sol, kimi-k3, luna, inkling, gpt-5.5, terra, grok-4.5, fable) = $158.5/run × 4 arms | ≈ $634 (≈$446 without kimi-k3; ≈$418 without fable) | ≈ $39 (incl. 10% validation) | reportable sensitivity doc | -| 3 full | 30-model v1 roster × 4 arms ≈ $1,520, or fold into the 20-model v2 protocol | — | ≈ $141 (incl. validation + anchors) | board-grade artifact | +| 2 subset | top-8 as ranked on the 32-model board (sol, kimi-k3, luna, ox-alpha†, inkling, gpt-5.5, terra, grok-4.6) = $105.8/run priced × 4 arms | ≈ $423 + ox-alpha† (≈$235 without kimi-k3) | ≈ $39 (incl. 10% validation) | reportable sensitivity doc | +| 3 full | 32-model v1 roster × 4 arms ≈ $1,553 + ox-alpha† + Fable ≈$216, or fold into the 20-model v2 protocol | — | ≈ $149 (incl. validation + anchors) | board-grade artifact | + +† ox-alpha is an unpriced stealth preview: its usage logged $0 and no +list price exists, so `stability-cost-plan` reports it `unpriced`; its +spend must be observed at rung time, not estimated here. Optional −$1,000 arm (55 households): ≈0.55 × one arm per model. Every paid rung is gated on explicit approval; boring before billed. From 9100cb6288906cf97f53b6c2905ff8d864da4636 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Tue, 25 Aug 2026 18:23:05 +0200 Subject: [PATCH 7/7] Use CF_ID_SUFFIX constant in fingerprint join Co-Authored-By: Claude Fable 5 --- policybench/stability_report.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/policybench/stability_report.py b/policybench/stability_report.py index 13265c6..14669af 100644 --- a/policybench/stability_report.py +++ b/policybench/stability_report.py @@ -23,6 +23,7 @@ from policybench.config import MODELS, PRICE_OVERRIDES_PER_1M from policybench.counterfactual import ( CF_DEFAULT_AMOUNT, + CF_ID_SUFFIX, build_counterfactual_manifest, compute_truth_deltas, delta_metrics_by_model, @@ -400,7 +401,9 @@ def run_counterfactual_report( "provider_system_fingerprint" in perturbed.columns ): pert = perturbed.copy() - pert["scenario_id"] = pert["scenario_id"].str.replace("__cf1k", "", regex=False) + pert["scenario_id"] = pert["scenario_id"].str.replace( + CF_ID_SUFFIX, "", regex=False + ) joined = base.merge( pert[["model", "scenario_id", "variable", "provider_system_fingerprint"]], on=["model", "scenario_id", "variable"],