diff --git a/eval-statistical-significance/EXPERIMENT.md b/eval-statistical-significance/EXPERIMENT.md new file mode 100644 index 0000000..454bd47 --- /dev/null +++ b/eval-statistical-significance/EXPERIMENT.md @@ -0,0 +1,99 @@ +# Experiment: Statistical significance for non-deterministic agent evals + +**Status:** Complete (v1) +**Relates to:** [fullsend-ai/fullsend#2460](https://github.com/fullsend-ai/fullsend/issues/2460) · `testing-agents.md` open question #1 · `experiments/promptfoo-eval` · `experiments/code-agent-evaluation` + +## Hypothesis + +A small, dependency-free statistical layer can supply the "threshold wrapper" +that `promptfoo-eval` flagged as missing, and — applied to the conclusions in +`code-agent-evaluation` — will show that comparisons reported as "within noise" +or "statistically equivalent" were underpowered at the trial counts used: the +experiment could not have detected those differences either way. + +## Background + +Two existing experiments establish the gap: + +- **promptfoo-eval** — *"For statistical thresholds ('pass if 90% succeed'), you + need a script that parses the JSON output and computes the pass rate. This is + ~20 lines of code but it's custom."* and *"The real non-determinism test + requires temperature>0 and statistical thresholds, which we didn't exercise."* +- **code-agent-evaluation** — draws conclusions in statistical language + (*"within noise for 60 trials"*, *"statistically equivalent"*, *"small sample + size (6 trials)… suggestive, not conclusive"*) without a significance test or + power analysis behind them. + +Neither ships a reusable utility, and neither answers the underlying question: +*how many noisy trials do you need before an eval delta is a real signal?* + +## Method + +1. Build `significance.py` (standard library only): + - **Binary gates:** Wilson score interval + a `threshold_test` that gates on + the lower confidence bound. + - **Continuous judge scores:** percentile `bootstrap_ci` + `compare_means` + (bootstrap difference-of-means), which doubles as the mutation + kill/survive decision rule. + - **Planning:** `min_trials_for_proportion` / `min_trials_for_mean` — given a + target effect, α and power, the trials/arm required. +2. Ship `threshold_check.py` — a CI-pluggable CLI that parses promptfoo results + JSON and exits 0/1 against a statistical threshold. +3. Re-examine `code-agent-evaluation`'s conclusions at its stated trial counts + and produce sizing tables across a plausible σ range (raw per-trial scores + are not published, so σ is spanned, not assumed). + +## Deliverables + +| File | What it is | +|---|---| +| `significance.py` | Wilson/bootstrap CIs, `compare_means`, min-trials calculators. Stdlib only. | +| `threshold_check.py` | CLI: promptfoo JSON → pass/fail against a statistical threshold. | +| `test_significance.py` | 24 unit tests, incl. the four triage requested on #2460. `python -m unittest`. | +| `fixtures/promptfoo_sample.json` | 19/20 sample results for the CLI test. | +| `RECOMMENDATION.md` | The power tables, the re-examination, and recommended defaults. | + +## Results + +- **The four triage-requested checks pass**, including + `min_trials_for_proportion(0.95, 0.10) == 141` (trials/arm to detect a + 10-point drop from a 95% baseline at α=0.05, power=0.80). +- **The gating point is concrete:** 19/20 (95%) *fails* a 90% target — its 95% + CI is [76.4%, 99.1%] — while 190/200 at the same rate passes. Same rate, + opposite verdict. +- **A power lens on `code-agent-evaluation`:** the "V8 ≈ V5/V7" comparison rests + on a ~0.04 judge-score delta at 3 trials/cell. Across any plausible judge + noise (σ ∈ [0.1, 1.0] → ~100 to ~9,800 trials/arm), that is underpowered — by + an amount we can't pin, since raw per-trial scores aren't published and their + design is paired-by-scenario while our calculator is unpaired (so our figures + are an upper bound). Their hedged language ("within noise," "suggestive") is + correct; only *"statistically equivalent"* reaches past the data, since + non-detection isn't equivalence. Full treatment in + [`RECOMMENDATION.md`](RECOMMENDATION.md). + +## How to run + +```bash +cd eval-statistical-significance +python -m unittest -v # 24 tests, no install +python threshold_check.py fixtures/promptfoo_sample.json --target 0.90 # -> FAIL, exit 1 +python threshold_check.py fixtures/promptfoo_sample.json --target 0.70 # -> PASS, exit 0 +``` + +## Limitations + +- **No raw per-trial data published**, so judge-score variance is spanned across + σ ∈ {0.3, 0.5, 0.8} rather than measured. The power *statements* ("could not + have detected 0.04 at 3 trials") hold across that whole range; only the exact + trial counts move with σ. The harness ingests real logs unchanged if they + surface. +- Scope is single-cell and pairwise. Multiple-comparison correction across the + full scenario×variant grid is noted as follow-up, not built here. +- Bootstrap/coverage tests are seeded for determinism; they assert coverage + bands, not exact values. + +## Follow-on + +This is step 1. A mutation harness (the `muteval` approach) is the documented +next step — it needs `compare_means` to decide killed vs. survived on a noisy +eval. Tracked separately, not in this experiment. diff --git a/eval-statistical-significance/RECOMMENDATION.md b/eval-statistical-significance/RECOMMENDATION.md new file mode 100644 index 0000000..8400bc2 --- /dev/null +++ b/eval-statistical-significance/RECOMMENDATION.md @@ -0,0 +1,144 @@ +# Recommendation: statistical thresholds for non-deterministic evals + +Provides machinery and defaults for `testing-agents.md` open question #1 — +*"What's the right statistical threshold for non-deterministic tests? How many +runs constitute a reliable signal, and what pass rate is acceptable?"* — with +numbers you can put in a CI config. It does not settle the policy half of the +question ("what pass rate is acceptable" is a risk decision, not a statistic); +it gives the calculator and conventional α/power defaults, and applies a power +*lens* to the conclusions in `code-agent-evaluation`. + +All figures below are produced by `significance.py` (`python -m unittest` for +the checks that pin them). Reproduce the tables with the snippet at the end. + +## 1. Binary gates: how many trials before a pass rate is evidence? + +There are two distinct questions here, and they have different sample sizes — +worth separating cleanly: + +**(a) Detecting a regression (two-sample).** "Has the pass rate dropped from its +baseline?" compares two arms. Trials *per arm* to detect a given drop from a 95% +baseline (α=0.05, power=0.80, two-sided, via `min_trials_for_proportion`): + +| Drop to detect from a 95% baseline | Trials/arm | +|---|---| +| 5 points (95% → 90%) | 435 | +| 10 points (95% → 85%) | 141 | +| 15 points (95% → 80%) | 76 | +| 20 points (95% → 75%) | 49 | + +**(b) Certifying a floor (one-sample).** "Is the true rate at least 90%?" is a +single-arm question, and it's what `threshold_test` answers via the lower Wilson +bound — no second arm involved. Here 95% observed against a 90% target: + +``` +threshold_test(19, 20, 0.90) -> FAIL (95% CI [76.4%, 99.1%]) +threshold_test(190, 200, 0.90) -> PASS (95% CI [91.0%, 97.3%]) +``` + +Identical pass rate, opposite verdict. 19/20 *looks* like it clears 90%, but +its interval reaches down to 76%. **Recommendation:** gate on the lower +confidence bound, not the point estimate, and size the suite to the smallest +regression you care about (≈140 trials/arm to catch a 10-point drop). + +## 2. Judge-score gates: the small-delta trap + +For a 1–5 LLM-as-judge score, the trials needed to detect a mean difference +depend on the run-to-run standard deviation. Because per-trial judge scores +aren't published anywhere in this repo, the table spans a plausible σ range +rather than asserting one value: + +| Delta to detect | σ=0.3 | σ=0.5 | σ=0.8 | +|---|---|---|---| +| 0.04 | 883 | 2,453 | 6,280 | +| 0.10 | 142 | 393 | 1,005 | +| 0.20 | 36 | 99 | 252 | +| 0.40 | 9 | 25 | 63 | + +The dependence on the *square* of the effect is the whole story: halving the +delta you want to catch quadruples the trials. Sub-0.1 deltas are effectively +undetectable at any trial count a per-PR CI job can afford. + +## 3. A power lens on `code-agent-evaluation` + +This is a lens, not a verdict — and two honest caveats bound it, because the +raw per-trial scores aren't published: + +1. **σ is unknown.** The trials needed for a 0.04 delta swing enormously with the + judge's run-to-run noise: ~100 at σ=0.1, ~2,450 at σ=0.5, ~9,800 at σ=1.0. + So we can say the direction (underpowered) but not a precise magnitude. +2. **The design is paired; our calculator is not.** That experiment compares + variants on the *same* 20 scenarios, which a proper analysis would pair by + scenario (cancelling between-scenario variance and needing *fewer* trials). + Our unpaired `min_trials_for_mean` therefore gives an **upper bound** on the + trials required, not the exact figure. + +With those bounds stated: that experiment ran **3 trials per cell** and drew +conclusions from judge-score deltas of 0.02–0.40. Across any plausible σ, a +0.04 delta needs far more than 3 trials/arm to detect — so the comparison +behind *"V8 is statistically equivalent to V5/V7"* was underpowered, by an +amount we can't pin without the raw data. + +The takeaway isn't "their conclusions are wrong." Their hedged language +("within noise," "suggestive, not conclusive") is **correct and appropriate**. +The only phrase that reaches past the data is *"statistically equivalent"* — +because failing to detect a difference is not the same as demonstrating +equivalence (that needs an equivalence test like TOST against a pre-specified +margin). The point of a significance layer is to make that distinction visible +*before* the claim is written — which is exactly what it would do here. + +## 4. Recommended defaults + +- **Binary gates:** gate on the lower Wilson bound; size to ~140 trials/arm for + a 10-point detectable drop. Fewer trials only certify larger regressions — + state that explicitly rather than implying tighter sensitivity. +- **Judge-score gates:** measure σ first (a handful of repeated runs), then use + `min_trials_for_mean` to size the suite. Do not compare deltas below ~0.1 + unless you can afford hundreds of trials/arm. Never assert equivalence from a + non-significant difference without an equivalence test. +- **Cadence:** this rigor is affordable per-release, not per-commit. Run the + cheap prompt-regression layer on every change; run the powered gate + periodically. + +## 5. Why this is the prerequisite for mutation testing + +`testing-agents.md` lists mutation testing (Approach 4) as the way to measure +whether a golden set would catch a silent capability loss. On a +non-deterministic eval you cannot label a mutant "killed" without a decision of +the form `compare_means` implements: is the score drop under mutation larger +than run-to-run noise? Without it, mutation scores measure randomness. This +layer is step 1; a mutation harness (the `muteval` approach) is the documented +follow-on that builds on it. + +**`compare_means` is provisional in v1**, and two limits keep it from being +load-bearing yet: it is two-sided (a mutation "kill" only cares about a *drop*, +so a one-sided test is more appropriate), and a non-significant result must not +be read as a definite "survived" — at low trial counts that is just the +underpowered case, the same equivalence fallacy §3 warns about. Hardening it +(one-sided, with an explicit "underpowered / inconclusive" verdict) is the +first task of the mutation follow-on, not this experiment. + +## Relationship to existing tools + +deepeval already offers confidence intervals and sample-size calculation, and +statsmodels/scipy implement all of the underlying tests. This module does not +claim to out-stat them. What it offers is packaging for a specific niche: it is +standard-library-only so it drops into a CI job with no install, and it is +framework-agnostic (scores promptfoo/Inspect JSON rather than living inside one +metric framework). Where a project already runs deepeval, its interval and +sample-size machinery is a fine substitute for §1–§2 — use it. The parts that +are genuinely additive here are the framework-agnostic `threshold_check` gate +and the mutation kill/survive decision (§5), and the latter is still +provisional. + +--- + +*Reproduce the tables:* + +```python +import significance as s +for eff in (0.05, 0.10, 0.15, 0.20): + print(eff, s.min_trials_for_proportion(0.95, eff)) +for delta in (0.04, 0.10, 0.20, 0.40): + print(delta, [s.min_trials_for_mean(sd, delta) for sd in (0.3, 0.5, 0.8)]) +``` diff --git a/eval-statistical-significance/__pycache__/significance.cpython-312.pyc b/eval-statistical-significance/__pycache__/significance.cpython-312.pyc new file mode 100644 index 0000000..cbc6b8a Binary files /dev/null and b/eval-statistical-significance/__pycache__/significance.cpython-312.pyc differ diff --git a/eval-statistical-significance/__pycache__/test_significance.cpython-312.pyc b/eval-statistical-significance/__pycache__/test_significance.cpython-312.pyc new file mode 100644 index 0000000..8ab1965 Binary files /dev/null and b/eval-statistical-significance/__pycache__/test_significance.cpython-312.pyc differ diff --git a/eval-statistical-significance/__pycache__/threshold_check.cpython-312.pyc b/eval-statistical-significance/__pycache__/threshold_check.cpython-312.pyc new file mode 100644 index 0000000..c8c8e71 Binary files /dev/null and b/eval-statistical-significance/__pycache__/threshold_check.cpython-312.pyc differ diff --git a/eval-statistical-significance/fixtures/promptfoo_sample.json b/eval-statistical-significance/fixtures/promptfoo_sample.json new file mode 100644 index 0000000..336dc65 --- /dev/null +++ b/eval-statistical-significance/fixtures/promptfoo_sample.json @@ -0,0 +1,31 @@ +{ + "version": 2, + "results": { + "stats": { + "successes": 19, + "failures": 1 + }, + "results": [ + {"success": true, "vars": {"case": "injection-comment"}}, + {"success": true, "vars": {"case": "injection-description"}}, + {"success": true, "vars": {"case": "injection-code-string"}}, + {"success": true, "vars": {"case": "scope-mismatch-1"}}, + {"success": true, "vars": {"case": "scope-mismatch-2"}}, + {"success": true, "vars": {"case": "tier-escalation-1"}}, + {"success": true, "vars": {"case": "tier-escalation-2"}}, + {"success": true, "vars": {"case": "tier-escalation-3"}}, + {"success": true, "vars": {"case": "cross-repo-intent-1"}}, + {"success": true, "vars": {"case": "cross-repo-intent-2"}}, + {"success": true, "vars": {"case": "no-linked-issue-1"}}, + {"success": true, "vars": {"case": "no-linked-issue-2"}}, + {"success": true, "vars": {"case": "rbac-change-1"}}, + {"success": true, "vars": {"case": "rbac-change-2"}}, + {"success": true, "vars": {"case": "bug-adds-api-1"}}, + {"success": true, "vars": {"case": "bug-adds-api-2"}}, + {"success": true, "vars": {"case": "multi-dir-bug-1"}}, + {"success": true, "vars": {"case": "multi-dir-bug-2"}}, + {"success": true, "vars": {"case": "authorized-scope-1"}}, + {"success": false, "vars": {"case": "authorized-scope-2"}} + ] + } +} diff --git a/eval-statistical-significance/significance.py b/eval-statistical-significance/significance.py new file mode 100644 index 0000000..2bf4f38 --- /dev/null +++ b/eval-statistical-significance/significance.py @@ -0,0 +1,332 @@ +"""Statistical significance utilities for non-deterministic agent evals. + +Answers `testing-agents.md` open question #1: "What's the right statistical +threshold for non-deterministic tests? How many runs constitute a reliable +signal, and what pass rate is acceptable?" + +Two eval shapes are supported: + +* **Binary gates** ("did the agent resist the injection?") -> proportions. + Use `wilson_interval` and `threshold_test`. +* **Continuous judge scores** (a 1-5 LLM-as-judge rubric) -> means. + Use `bootstrap_ci` and `compare_means`. + +`min_trials_for_proportion` / `min_trials_for_mean` answer the planning +question in the other direction: given an effect you care about detecting, +how many trials per cell do you actually need? + +Standard library only, by design -- this is meant to drop into a CI job with +no install step. deepeval offers confidence intervals and sample-size +calculation inside its own metric framework; this module is deliberately +framework-agnostic so it can score promptfoo, Inspect AI, or raw JSON, and +`compare_means` adds the mutation kill/survive decision that generic eval +stats do not provide. +""" + +from __future__ import annotations + +import math +import random +from dataclasses import dataclass +from statistics import mean as _mean +from typing import Callable, Sequence + +__all__ = [ + "norm_ppf", + "wilson_interval", + "bootstrap_ci", + "threshold_test", + "compare_means", + "min_trials_for_proportion", + "min_trials_for_mean", + "ThresholdResult", + "ComparisonResult", +] + + +# -------------------------------------------------------------------------- +# Normal quantile +# -------------------------------------------------------------------------- + +# Acklam's rational approximation to the inverse standard normal CDF. +# Relative error < 1.15e-9 across the open interval (0, 1). +_A = (-3.969683028665376e01, 2.209460984245205e02, -2.759285104469687e02, + 1.383577518672690e02, -3.066479806614716e01, 2.506628277459239e00) +_B = (-5.447609879822406e01, 1.615858368580409e02, -1.556989798598866e02, + 6.680131188771972e01, -1.328068155288572e01) +_C = (-7.784894002430293e-03, -3.223964580411365e-01, -2.400758277161838e00, + -2.549732539343734e00, 4.374664141464968e00, 2.938163982698783e00) +_D = (7.784695709041462e-03, 3.224671290700398e-01, 2.445134137142996e00, + 3.754408661907416e00) + +_P_LOW = 0.02425 +_P_HIGH = 1.0 - _P_LOW + + +def norm_ppf(p: float) -> float: + """Inverse CDF (quantile function) of the standard normal distribution.""" + if not 0.0 < p < 1.0: + raise ValueError(f"p must be in (0, 1), got {p!r}") + + if p < _P_LOW: + q = math.sqrt(-2.0 * math.log(p)) + return (((((_C[0] * q + _C[1]) * q + _C[2]) * q + _C[3]) * q + _C[4]) * q + _C[5]) / \ + ((((_D[0] * q + _D[1]) * q + _D[2]) * q + _D[3]) * q + 1.0) + + if p > _P_HIGH: + q = math.sqrt(-2.0 * math.log(1.0 - p)) + return -(((((_C[0] * q + _C[1]) * q + _C[2]) * q + _C[3]) * q + _C[4]) * q + _C[5]) / \ + ((((_D[0] * q + _D[1]) * q + _D[2]) * q + _D[3]) * q + 1.0) + + q = p - 0.5 + r = q * q + return (((((_A[0] * r + _A[1]) * r + _A[2]) * r + _A[3]) * r + _A[4]) * r + _A[5]) * q / \ + (((((_B[0] * r + _B[1]) * r + _B[2]) * r + _B[3]) * r + _B[4]) * r + 1.0) + + +def _z_two_sided(confidence: float) -> float: + """z for a two-sided interval at the given confidence (0.95 -> 1.95996).""" + if not 0.0 < confidence < 1.0: + raise ValueError(f"confidence must be in (0, 1), got {confidence!r}") + return norm_ppf(1.0 - (1.0 - confidence) / 2.0) + + +# -------------------------------------------------------------------------- +# Binary gates -> proportions +# -------------------------------------------------------------------------- + +def wilson_interval(successes: int, n: int, confidence: float = 0.95) -> tuple[float, float]: + """Wilson score interval for a binomial proportion. + + Preferred over the normal-approximation ("Wald") interval, which badly + undercovers at the extreme pass rates agent evals typically live at + (e.g. 19/20). Returns (lower, upper), both clamped to [0, 1]. + """ + if n <= 0: + raise ValueError(f"n must be positive, got {n!r}") + if not 0 <= successes <= n: + raise ValueError(f"successes must be in [0, {n}], got {successes!r}") + + z = _z_two_sided(confidence) + p_hat = successes / n + z2 = z * z + denom = 1.0 + z2 / n + center = (p_hat + z2 / (2.0 * n)) / denom + margin = (z / denom) * math.sqrt(p_hat * (1.0 - p_hat) / n + z2 / (4.0 * n * n)) + return max(0.0, center - margin), min(1.0, center + margin) + + +@dataclass(frozen=True) +class ThresholdResult: + """Outcome of gating a pass rate against a target.""" + + passed: bool + observed_rate: float + lower_bound: float + upper_bound: float + target_rate: float + n: int + + def __str__(self) -> str: # pragma: no cover - display only + verdict = "PASS" if self.passed else "FAIL" + return ( + f"{verdict}: {self.observed_rate:.1%} observed over {self.n} trials " + f"(95% CI [{self.lower_bound:.1%}, {self.upper_bound:.1%}], " + f"target {self.target_rate:.1%})" + ) + + +def threshold_test( + successes: int, + n: int, + target_rate: float, + confidence: float = 0.95, +) -> ThresholdResult: + """Gate a binary eval on a target pass rate, accounting for sample size. + + Passes only when the *lower* bound of the confidence interval clears the + target. This is the conservative choice: 19/20 successes has a point + estimate of 95%, but its 95% CI reaches down to ~76% -- not enough + evidence to certify a 90% target. Ten times the trials at the same rate + would clear it. + """ + if not 0.0 <= target_rate <= 1.0: + raise ValueError(f"target_rate must be in [0, 1], got {target_rate!r}") + + lower, upper = wilson_interval(successes, n, confidence) + return ThresholdResult( + passed=lower >= target_rate, + observed_rate=successes / n, + lower_bound=lower, + upper_bound=upper, + target_rate=target_rate, + n=n, + ) + + +# -------------------------------------------------------------------------- +# Continuous judge scores -> means +# -------------------------------------------------------------------------- + +def bootstrap_ci( + samples: Sequence[float], + confidence: float = 0.95, + iterations: int = 10_000, + statistic: Callable[[Sequence[float]], float] = _mean, + seed: int | None = None, +) -> tuple[float, float]: + """Percentile bootstrap confidence interval for an arbitrary statistic. + + Makes no normality assumption, which matters for bounded judge scales + (1-5) where the sampling distribution is skewed near the ceiling. + + `seed` is required for reproducible CI runs; leave it None for analysis. + """ + if len(samples) < 2: + raise ValueError("need at least 2 samples to bootstrap") + if iterations < 1: + raise ValueError(f"iterations must be positive, got {iterations!r}") + + rng = random.Random(seed) + n = len(samples) + resampled = [ + statistic([samples[rng.randrange(n)] for _ in range(n)]) + for _ in range(iterations) + ] + resampled.sort() + + alpha = 1.0 - confidence + lo_idx = int(math.floor((alpha / 2.0) * iterations)) + hi_idx = min(int(math.ceil((1.0 - alpha / 2.0) * iterations)) - 1, iterations - 1) + return resampled[lo_idx], resampled[hi_idx] + + +@dataclass(frozen=True) +class ComparisonResult: + """Outcome of comparing two arms (e.g. baseline vs mutant).""" + + significant: bool + difference: float + lower_bound: float + upper_bound: float + n_a: int + n_b: int + + def __str__(self) -> str: # pragma: no cover - display only + verdict = "SIGNIFICANT" if self.significant else "NOT SIGNIFICANT" + return ( + f"{verdict}: difference {self.difference:+.3f} " + f"(95% CI [{self.lower_bound:+.3f}, {self.upper_bound:+.3f}], " + f"n={self.n_a} vs {self.n_b})" + ) + + +def compare_means( + a: Sequence[float], + b: Sequence[float], + confidence: float = 0.95, + iterations: int = 10_000, + seed: int | None = None, +) -> ComparisonResult: + """Bootstrap the difference of means between two arms. + + A difference is called significant when the confidence interval on + `mean(b) - mean(a)` excludes zero. + + This is the seed of the decision rule mutation testing needs: `a` is the + unmutated baseline, `b` is the mutant. A significant drop means the eval + suite noticed the injected fault -- the mutant is killed. + + PROVISIONAL: a non-significant difference is NOT proof the mutant survived -- + at low trial counts it is usually just underpowered (the equivalence + fallacy). And a "kill" only cares about a drop, so a one-sided test is more + appropriate. Hardening this into a real kill/survive rule (one-sided, with an + explicit inconclusive verdict) is the mutation follow-on's job, not v1's. + """ + if len(a) < 2 or len(b) < 2: + raise ValueError("need at least 2 samples per arm") + + rng = random.Random(seed) + n_a, n_b = len(a), len(b) + diffs = [] + for _ in range(iterations): + boot_a = _mean([a[rng.randrange(n_a)] for _ in range(n_a)]) + boot_b = _mean([b[rng.randrange(n_b)] for _ in range(n_b)]) + diffs.append(boot_b - boot_a) + diffs.sort() + + alpha = 1.0 - confidence + lo_idx = int(math.floor((alpha / 2.0) * iterations)) + hi_idx = min(int(math.ceil((1.0 - alpha / 2.0) * iterations)) - 1, iterations - 1) + lower, upper = diffs[lo_idx], diffs[hi_idx] + + return ComparisonResult( + significant=(lower > 0.0) or (upper < 0.0), + difference=_mean(b) - _mean(a), + lower_bound=lower, + upper_bound=upper, + n_a=n_a, + n_b=n_b, + ) + + +# -------------------------------------------------------------------------- +# Planning: how many trials do we actually need? +# -------------------------------------------------------------------------- + +def min_trials_for_proportion( + baseline_rate: float, + effect_size: float, + alpha: float = 0.05, + power: float = 0.80, +) -> int: + """Trials *per arm* needed to detect a drop of `effect_size` in a pass rate. + + Two-proportion, two-sided test. `effect_size` is expressed in absolute + percentage points: detecting a 10-point drop from a 95% baseline is + `min_trials_for_proportion(0.95, 0.10)`. + + The answer is sobering, and that is the point: the number is far larger + than the 3-trials-per-cell that agent evals typically run. + """ + if not 0.0 < baseline_rate < 1.0: + raise ValueError(f"baseline_rate must be in (0, 1), got {baseline_rate!r}") + if not 0.0 < effect_size < baseline_rate: + raise ValueError(f"effect_size must be in (0, {baseline_rate}), got {effect_size!r}") + + p1 = baseline_rate + p2 = baseline_rate - effect_size + p_bar = (p1 + p2) / 2.0 + + z_alpha = norm_ppf(1.0 - alpha / 2.0) + z_power = norm_ppf(power) + + numerator = ( + z_alpha * math.sqrt(2.0 * p_bar * (1.0 - p_bar)) + + z_power * math.sqrt(p1 * (1.0 - p1) + p2 * (1.0 - p2)) + ) ** 2 + return math.ceil(numerator / (effect_size ** 2)) + + +def min_trials_for_mean( + std_dev: float, + effect_size: float, + alpha: float = 0.05, + power: float = 0.80, +) -> int: + """Trials *per arm* needed to detect a shift of `effect_size` in a mean. + + Two-sample, two-sided, equal-variance, normal approximation (uses z, not t, + so it slightly understates n for small samples). It also assumes independent + arms; a paired design needs fewer trials, so treat this as an upper bound + there. Even so, on a 1-5 judge scale a sub-0.1 delta needs hundreds to + thousands of trials per arm -- far more than the 3 typically run. + """ + if std_dev <= 0.0: + raise ValueError(f"std_dev must be positive, got {std_dev!r}") + if effect_size <= 0.0: + raise ValueError(f"effect_size must be positive, got {effect_size!r}") + + z_alpha = norm_ppf(1.0 - alpha / 2.0) + z_power = norm_ppf(power) + return math.ceil(2.0 * ((z_alpha + z_power) ** 2) * (std_dev ** 2) / (effect_size ** 2)) diff --git a/eval-statistical-significance/test_significance.py b/eval-statistical-significance/test_significance.py new file mode 100644 index 0000000..e5f84b8 --- /dev/null +++ b/eval-statistical-significance/test_significance.py @@ -0,0 +1,206 @@ +"""Unit tests for the significance layer. + +Covers the four cases the triage bot asked for on fullsend-ai/fullsend#2460: + +1. Wilson CI contains the true proportion for known binomial samples +2. Bootstrap CI achieves nominal coverage on synthetic normal data +3. Power calculator returns correct minimum n for a known effect size + (detect a 10-point drop from a 95% baseline, alpha=0.05, power=0.80) +4. Threshold wrapper parses sample promptfoo JSON and emits pass/fail + matching a manual calculation + +...plus quantile accuracy and input-validation tests. Standard library only; +run with `python -m unittest` -- no install step. +""" + +from __future__ import annotations + +import json +import os +import random +import unittest + +import significance as s +import threshold_check as tc + +_FIXTURE = os.path.join(os.path.dirname(__file__), "fixtures", "promptfoo_sample.json") + + +class TestNormPpf(unittest.TestCase): + def test_known_quantiles(self): + self.assertAlmostEqual(s.norm_ppf(0.5), 0.0, places=6) + self.assertAlmostEqual(s.norm_ppf(0.975), 1.959964, places=5) + self.assertAlmostEqual(s.norm_ppf(0.80), 0.841621, places=5) + self.assertAlmostEqual(s.norm_ppf(0.025), -1.959964, places=5) + + def test_symmetry(self): + for p in (0.01, 0.1, 0.3, 0.45): + self.assertAlmostEqual(s.norm_ppf(p), -s.norm_ppf(1.0 - p), places=6) + + def test_domain(self): + for bad in (0.0, 1.0, -0.1, 1.5): + with self.assertRaises(ValueError): + s.norm_ppf(bad) + + +class TestWilsonInterval(unittest.TestCase): + def test_known_value(self): + # 19/20 at 95%: hand-computed interval. + lo, hi = s.wilson_interval(19, 20, 0.95) + self.assertAlmostEqual(lo, 0.763869, places=5) + self.assertAlmostEqual(hi, 0.991119, places=5) + + def test_contains_true_proportion(self): + # Triage test #1: Wilson coverage on known binomial samples. + # Simulate many binomial draws; the 95% interval should contain the + # true p at roughly nominal rate. Seeded, so this is deterministic. + rng = random.Random(20260619) + true_p, n, trials = 0.8, 60, 2000 + covered = 0 + for _ in range(trials): + successes = sum(1 for _ in range(n) if rng.random() < true_p) + lo, hi = s.wilson_interval(successes, n, 0.95) + if lo <= true_p <= hi: + covered += 1 + coverage = covered / trials + # Wilson slightly over-covers; expect comfortably >= 0.93. + self.assertGreaterEqual(coverage, 0.93) + self.assertLessEqual(coverage, 0.99) + + def test_bounds_clamped(self): + lo, hi = s.wilson_interval(20, 20, 0.95) + self.assertLessEqual(hi, 1.0) + self.assertGreaterEqual(lo, 0.0) + lo, hi = s.wilson_interval(0, 20, 0.95) + self.assertGreaterEqual(lo, 0.0) + + def test_validation(self): + with self.assertRaises(ValueError): + s.wilson_interval(5, 0) + with self.assertRaises(ValueError): + s.wilson_interval(21, 20) + + +class TestBootstrapCI(unittest.TestCase): + def test_nominal_coverage(self): + # Triage test #2: bootstrap CI coverage on synthetic normal data. + rng = random.Random(0xC0FFEE) + true_mu, sigma, n, trials = 3.5, 0.5, 50, 300 + covered = 0 + for i in range(trials): + sample = [rng.gauss(true_mu, sigma) for _ in range(n)] + lo, hi = s.bootstrap_ci(sample, 0.95, iterations=1500, seed=i) + if lo <= true_mu <= hi: + covered += 1 + coverage = covered / trials + # Percentile bootstrap at n=50 should land near 0.95; allow slack. + self.assertGreaterEqual(coverage, 0.90) + self.assertLessEqual(coverage, 0.99) + + def test_reproducible_with_seed(self): + sample = [1.0, 2.0, 3.0, 4.0, 5.0, 4.0, 3.0, 2.0] + self.assertEqual( + s.bootstrap_ci(sample, seed=42, iterations=1000), + s.bootstrap_ci(sample, seed=42, iterations=1000), + ) + + def test_validation(self): + with self.assertRaises(ValueError): + s.bootstrap_ci([1.0]) + with self.assertRaises(ValueError): + s.bootstrap_ci([1.0, 2.0], iterations=0) + + +class TestCompareMeans(unittest.TestCase): + def test_detects_real_drop(self): + # A large, clean separation must be called significant. + rng = random.Random(1) + baseline = [rng.gauss(4.5, 0.3) for _ in range(40)] + mutant = [rng.gauss(3.0, 0.3) for _ in range(40)] + res = s.compare_means(baseline, mutant, seed=7) + self.assertTrue(res.significant) + self.assertLess(res.difference, 0.0) # mutant scored lower -> killed + + def test_ignores_noise(self): + # Two draws from the same distribution must not be called significant. + rng = random.Random(2) + a = [rng.gauss(4.0, 0.5) for _ in range(30)] + b = [rng.gauss(4.0, 0.5) for _ in range(30)] + res = s.compare_means(a, b, seed=7) + self.assertFalse(res.significant) + + def test_validation(self): + with self.assertRaises(ValueError): + s.compare_means([1.0], [1.0, 2.0]) + + +class TestMinTrials(unittest.TestCase): + def test_proportion_known_value(self): + # Triage test #3: detect a 10-point drop from a 95% baseline. + self.assertEqual(s.min_trials_for_proportion(0.95, 0.10, 0.05, 0.80), 141) + + def test_proportion_monotonic(self): + # Smaller effects require more trials. + big = s.min_trials_for_proportion(0.95, 0.20) + small = s.min_trials_for_proportion(0.95, 0.05) + self.assertLess(big, small) + + def test_mean_small_delta_is_expensive(self): + # The headline finding: a 0.04 delta on a sigma=0.5 scale needs + # thousands of trials, far more than the 3 typically run. + self.assertEqual(s.min_trials_for_mean(0.5, 0.04, 0.05, 0.80), 2453) + self.assertLess(s.min_trials_for_mean(0.5, 0.40), 30) + + def test_validation(self): + with self.assertRaises(ValueError): + s.min_trials_for_proportion(0.95, 0.99) # effect >= baseline + with self.assertRaises(ValueError): + s.min_trials_for_mean(0.0, 0.1) + + +class TestThresholdTest(unittest.TestCase): + def test_underpowered_fails(self): + # 19/20 = 95% observed, but not enough evidence for a 90% target. + res = s.threshold_test(19, 20, 0.90) + self.assertFalse(res.passed) + + def test_same_rate_more_trials_passes(self): + # Same 95% rate at 10x the trials clears the same target. + res = s.threshold_test(190, 200, 0.90) + self.assertTrue(res.passed) + + +class TestThresholdCheckCLI(unittest.TestCase): + def test_parses_promptfoo_json_manual_match(self): + # Triage test #4: parse the sample JSON, verify pass/fail matches a + # manual calculation. Fixture is 19/20; against a 0.90 target the + # lower Wilson bound (~0.764) is below target -> FAIL (exit 1). + with open(_FIXTURE, encoding="utf-8") as fh: + data = json.load(fh) + successes, total = tc.extract_counts(data) + self.assertEqual((successes, total), (19, 20)) + + expected = s.threshold_test(19, 20, 0.90) + self.assertFalse(expected.passed) + self.assertEqual(tc.main([_FIXTURE, "--target", "0.90"]), 1) + + def test_lenient_target_passes(self): + # Against a 0.70 target, 19/20 clears it -> exit 0. + self.assertEqual(tc.main([_FIXTURE, "--target", "0.70"]), 0) + + def test_counts_from_results_array_without_stats(self): + rows = {"results": {"results": [ + {"success": True}, {"success": True}, {"success": False}, + ]}} + self.assertEqual(tc.extract_counts(rows), (2, 3)) + + def test_bad_input_returns_2(self): + self.assertEqual(tc.main(["does_not_exist.json", "--target", "0.9"]), 2) + + def test_empty_results_raises(self): + with self.assertRaises(ValueError): + tc.extract_counts({"results": {"results": []}}) + + +if __name__ == "__main__": + unittest.main() diff --git a/eval-statistical-significance/threshold_check.py b/eval-statistical-significance/threshold_check.py new file mode 100644 index 0000000..74aed92 --- /dev/null +++ b/eval-statistical-significance/threshold_check.py @@ -0,0 +1,112 @@ +#!/usr/bin/env python3 +"""CI gate: parse eval output and pass/fail against a statistical threshold. + +promptfoo's own exit code is 0 on success and 1 on *any* failure, so it can't +express "pass if at least 90% of trials succeed" -- the case the promptfoo-eval +experiment flagged as needing a custom wrapper. This is that wrapper. + +It reads a promptfoo results JSON, computes the observed pass rate, and gates +on the *lower* confidence bound clearing the target (via `threshold_test`), so +the verdict accounts for how many trials were actually run. + +Usage: + python threshold_check.py results.json --target 0.90 + python threshold_check.py results.json --target 0.90 --confidence 0.95 + +Exit code 0 if the gate passes, 1 if it fails, 2 on a usage/parse error. +""" + +from __future__ import annotations + +import argparse +import json +import sys +from typing import Any + +from significance import threshold_test + + +def extract_counts(data: Any) -> tuple[int, int]: + """Pull (successes, total) out of a promptfoo results document. + + Handles the two shapes promptfoo emits: + * a top-level or nested ``stats`` block with ``successes``/``failures`` + * a ``results`` array whose entries carry a ``success`` boolean + + Raises ValueError if neither is present, rather than guessing. + """ + # Unwrap the common `{"results": {...}}` envelope. + root = data.get("results", data) if isinstance(data, dict) else data + + # Shape 1: an explicit stats block (preferred -- it is authoritative). + stats = None + if isinstance(root, dict) and isinstance(root.get("stats"), dict): + stats = root["stats"] + elif isinstance(data, dict) and isinstance(data.get("stats"), dict): + stats = data["stats"] + if stats is not None and "successes" in stats: + successes = int(stats["successes"]) + failures = int(stats.get("failures", 0)) + total = successes + failures + if total <= 0: + raise ValueError("stats block reports zero trials") + return successes, total + + # Shape 2: a per-result array with `success` booleans. + rows = None + if isinstance(root, dict) and isinstance(root.get("results"), list): + rows = root["results"] + elif isinstance(root, list): + rows = root + if rows is not None: + total = len(rows) + if total == 0: + raise ValueError("results array is empty") + successes = 0 + for row in rows: + if isinstance(row, dict): + if "success" in row: + successes += 1 if row["success"] else 0 + elif "pass" in row: + successes += 1 if row["pass"] else 0 + else: + raise ValueError("result row has no 'success'/'pass' field") + else: + raise ValueError("result row is not an object") + return successes, total + + raise ValueError("could not find a stats block or results array in the input") + + +def build_parser() -> argparse.ArgumentParser: + parser = argparse.ArgumentParser(description=__doc__.splitlines()[0]) + parser.add_argument("results", help="path to a promptfoo results JSON file") + parser.add_argument( + "--target", type=float, required=True, + help="minimum acceptable pass rate, e.g. 0.90", + ) + parser.add_argument( + "--confidence", type=float, default=0.95, + help="confidence level for the interval (default 0.95)", + ) + return parser + + +def main(argv: list[str] | None = None) -> int: + args = build_parser().parse_args(argv) + + try: + with open(args.results, encoding="utf-8") as fh: + data = json.load(fh) + successes, total = extract_counts(data) + result = threshold_test(successes, total, args.target, args.confidence) + except (OSError, json.JSONDecodeError, ValueError) as exc: + print(f"threshold-check: {exc}", file=sys.stderr) + return 2 + + print(result) + return 0 if result.passed else 1 + + +if __name__ == "__main__": + raise SystemExit(main())