From a257f992e485c4b08a843036c31b33cb96cf8103 Mon Sep 17 00:00:00 2001 From: boyam01 Date: Sun, 5 Jul 2026 08:30:44 +0800 Subject: [PATCH 1/3] docs: add methodology (adversarial evidence protocol) + prior-art references Paper-shaped methodology spec for probity: the claim object, criterion-provenance tiers (Tier 0-4), capability-denial architecture, multi-oracle + oracle QA, invariant-based nondeterminism, Wilson-CI statistics (cheap refutation / priced confirmation), three-axis reporting, and falsifier decision labels (REFUTED / INVALID_CRITERION / INSUFFICIENT / SURVIVED_SCOPE). Related work honestly positions each pillar against ~18 prior papers (closest: RewardHackingAgents arXiv:2603.11337); claims novelty only in the packaging (provenance tiers, priced-confirmation policy, falsifier labels, generalization). Docs-only; no code touched. --- docs/METHODOLOGY.md | 205 +++++++++++++++++---------------- docs/methodology-references.md | 40 +++++++ 2 files changed, 148 insertions(+), 97 deletions(-) create mode 100644 docs/methodology-references.md diff --git a/docs/METHODOLOGY.md b/docs/METHODOLOGY.md index b2612e7..cf99020 100644 --- a/docs/METHODOLOGY.md +++ b/docs/METHODOLOGY.md @@ -1,138 +1,149 @@ -# Probity methodology +# probity — Methodology -Probity is a method for testing whether an AI coding agent's success claim is -supported by evidence. The target failure is **false green**: the workflow looks -green, but the evidence does not justify trusting the result. +*An adversarial evidence protocol for agent trust claims.* +Draft v0.2 · falsifier-not-certifier · zero-LLM verdict path. -## The problem +> **One line.** probity does not decide whether an AI agent is *reliable*. It decides whether a specific, scoped **claim** survives *cheap falsification* under a declared criterion, verifier, sandbox, and attempt budget — and, if it survives, states what further cost would raise confidence. It is a machine for making trust claims **expensive to fake**, not a machine that manufactures trust. -Agent evaluation often starts from a fragile signal: +--- -```text -The agent finished once -> the task is done -> ship it -``` +## 1. Problem + +An AI coding agent that reports success will, often enough, report success it did not earn: tests silently skipped, the wrong file edited, the acceptance criterion reinterpreted until it passes, the evaluator itself tampered with. This is **false-green**. Empirically it is not rare — augmenting SWE-bench tests reclassifies hundreds of "passing" patches as failing [UTBoost]; ~33% of "successful" SWE-bench patches leaked the fix from the issue text [SWE-Bench+]; 77% of SWE-bench Verified instances admit a semantically wrong variant that still passes the given tests [STING]; RL-trained agents overwrite unit tests and monkey-patch scorers [GamingVerifiers]. + +The usual response — ask another model to judge — hands the ruler to material of the same kind as the thing measured. An LLM judge scores **plausibility**, not correctness; it favors its own outputs and cannot reliably grade what it cannot itself solve [SelfPref; NoFreeLabels]. Verify with the wrong ruler and the answer is wrong by construction. -That signal collapses several different facts into one story: +The deeper fact is the **oracle problem** [Barr2015]: to verify correctness you need a trustworthy source of ground truth, and there is no free lunch that manufactures one. probity does not solve the oracle problem. It does three things a certifier cannot: it **relocates** the trust to a place we can name and audit (the criterion), it **denies** the agent the ability to corrupt that place, and it is **honest about what it cannot verify**. -- whether the final state passed the checker; -- whether the agent honestly described that final state; -- whether the agent edited the test oracle; -- whether the result repeats across independent runs; -- whether the environment was healthy. +## 2. Design principles -Probity separates those facts. +Three principles do the load-bearing work. Each is individually precedented; the contribution is the way they are combined and generalized (§14). -## Method +- **P1 — Capability denial over detection.** Do not check *whether* the agent cheated; remove the *ability* to cheat. Read-only tests, patch-only submission, an external runner, no network. Detection is an arms race; denial is structural [DetBoundaries; LingeringAuth]. +- **P2 — Separation of powers over trust in a smart checker.** probity's strength must not come from the checker being clever. It comes from the **doer being unable to control two of the three roles**: who *defines* the criterion, who *executes* the verifier, who *records* the evidence. Control any two and probity degrades to *self-grading*. +- **P3 — Public principle, hidden instance.** The algorithm is public; the secret is the per-challenge seed and held-out cases, generated *after* the patch is submitted. This is the Kerckhoffs stance applied to evaluation, and it is the only defense against Goodharting a fixed check [Skalse2022]. -```text -registered task - -> fresh isolated worktree per run - -> deterministic checker - -> claim/evidence comparison - -> integrity flags - -> repeated-trial statistics - -> PASS / KILL / INSUFFICIENT +## 3. The claim object + +probity never verifies "this agent is reliable." It verifies a **scoped claim**: + +``` +Under repo R@commit, spec S@hash, verifier V@image-hash, sandbox policy P@hash, +and attempt budget B, does the agent's submitted artifact satisfy claim C? ``` -The method has six parts. +Everything is content-addressed and frozen before the run. Narrowness is not a weakness here; it is the source of the strength. "Reliable in the real world" is unfalsifiable and therefore un-verifiable; "not refuted under these declared conditions" is both. -### 1. Register the task before the run +## 4. Criterion provenance tiers *(core contribution)* -A task defines: +The verdict's meaning is inherited entirely from the criterion's trustworthiness, so criterion provenance is a **first-class field of every report**, not metadata. We order criteria by how hackable they are by the doer: -- the workspace; -- the agent command; -- allowed files; -- protected files; -- critical rules; -- the deterministic checker; -- the reliability claim `r`; -- the number of trials `k`. +| Tier | Criterion source | Trust | Label | +|---|---|---|---| +| **0** | Agent-authored tests / agent-defined success | ~none | `SELF_CHECK_ONLY` | +| **1** | Public tests / the repo's existing CI | weak | `PUBLIC_REPLAY_PASS` | +| **2** | Externally-authored spec + public tests | medium | `EXTERNAL_SPEC_PASS` | +| **3** | External held-out tests / reference implementation | strong | `HELDOUT_PASS` | +| **4** | Formal spec / proof or model checker | strongest, narrowest | `FORMAL_OR_PROOF_BACKED` | -This prevents the evaluation from moving the target after seeing the output. +**Hard rule.** A Tier-0 criterion may *never* support a reliability claim. Agent-authored tests can answer "does this agent self-check?" — never "is the result reliable?". An agent may *propose* tests or attack cases; those are a **candidate criterion** and become evidence only after an external owner accepts them, on the *next* run. This directly answers the "who writes the ruler" crack: probity does not pretend the oracle problem vanishes; it makes the ruler's provenance auditable and refuses to let the doer forge it. -### 2. Keep the checker deterministic +## 5. Verification architecture (P1 in practice) -The built-in verdict path is: +The minimal trusted computing base is deliberately small — the more intelligence in the core, the less it can be trusted (§13). The core does only: freeze inputs, apply the patch, enforce the sandbox, run the verifier, record hashes, emit the evidence bundle. -```text -checker -> stats -> verdict -``` +| Attack | Weak (detection) | probity (denial) | +|---|---|---| +| delete/alter fixtures | scan the diff for it | fixtures mounted read-only | +| edit or skip tests | grep the diff | tests not on any writable path; runner runs them | +| forge a pass | parse agent output | never trust agent output; only runner output counts | +| hardcode public cases | add a few public tests | hidden cases generated post-submission | +| environment fingerprinting | scan for suspicious code | multi-environment re-run, no network, frozen clock/seed | -It contains no LLM call, no LLM judge, and no LLM-written failure summary. +The doer submits a **patch or artifact only**. It does not submit test results, its own logs, its own pass condition, or a modified verifier. -Custom script checkers are allowed, but they are trust inputs owned by the task -author. Probity can record their output; it cannot prove that an arbitrary -custom checker is semantically correct. +## 6. Multi-oracle verification -### 3. Separate claim from evidence +A single fixed test set is the easiest oracle to game. probity runs several, and — critically — each is **deterministic given a seed that is drawn after submission**: reproducible because the seed is logged, un-overfittable because it was unknown in advance (*post-seed deterministic, pre-run unpredictable*). -The agent's final self-report is parsed separately from the checker's result. +- public tests (debug-usable), **held-out** tests (final evaluation only), +- property-based and metamorphic relations, differential comparison to a reference, +- tamper/canary checks and an env canary (pre=ok/post=ok). -```text -agent: CLAIM: success -checker: failed -result: false_claim -``` +Reach is honest: property/metamorphic/generative oracles only exist for tasks with checkable structure. Tasks without it ("make this readable," a bug with no reproducer) have no seed to hide and fall back to an externally-authored fixed criterion — or to `INSUFFICIENT` (§12). -This matters because a failed run that claims success can still fool a human -single-run workflow. +## 7. Oracle QA — testing the ruler *(criterion fitness)* -### 4. Treat oracle tampering as an integrity failure +High pass rate against a weak ruler is *confidently wrong*. So probity also measures the **criterion**, not only the agent: -If the agent edits protected tests or other protected oracle files, the run is -not treated as evidence of capability. It is an audit-integrity failure. +- **Negative controls.** A corpus of known-bad artifacts (deletes core function but keeps tests green, hardcodes fixtures, mocks the external API, drops an error branch) that the criterion *should* reject. If they pass, the oracle is bad, not the agent good. +- **Mutation adequacy.** Mutate code/spec and measure the kill rate; a low mutation score means the tests verify little [STING; Stryker/PITest]. +- **Spec-coverage map.** Each clause of the spec maps to tests, or the report says `INSUFFICIENT: claim includes untested spec clauses`. -### 5. Repeat under fresh isolation +Oracle QA is itself authored and gameable and is not free; it raises the floor, it does not close the problem (§13). Its outputs (mutation score, known-bad rejection, canary detection) are reported as their own axis (§11), never folded into the pass rate. -Each run starts from a fresh git worktree. One lucky run is not enough evidence. -Repeated runs expose flakiness, degradation, partial fixes, and rare destructive -actions. +## 8. Nondeterminism: invariants, not traces -### 6. Prefer honest refusal over overclaiming +Trace divergence is a *weak* signal — many correct solutions legitimately differ (timestamps, request ids, independent-job ordering). probity therefore requires the task owner to declare `allowed_variance` and `required_invariants` (final schema, DB state, balance conservation, no forbidden writes, reference-equivalent result), and it verifies **invariants under k randomized runs**. Raw trace difference is a hard red flag *only* when the task itself claims a deterministic trace. -Probity uses a confidence interval over repeated trials. Small `k` can refute a -high-reliability claim, but it usually cannot confirm one. +## 9. Statistical framework -That is why `INSUFFICIENT` is a first-class outcome, not an error. +Reliability is a claim about a distribution, so it is estimated over k trials with a **Wilson score interval**, which has correct coverage where the Wald/CLT approximation fails at small n [NoCLT; StatGuarantees]. -## What the method improves +- **Cheap refutation, priced confirmation.** A few failures refute a high claim; confirming a high claim is expensive. In the bundled demo, an r=0.9 claim with 7/10 observed yields a Wilson CI of [0.397, 0.892] — refuting any claim above 0.892 while confirming nothing above 0.397. Confirming 0.9 at 95% would require ≈35 consecutive clean runs. This asymmetry is not a defect; it is the product: *tell me how much confidence you want and I price the sample cost.* +- **Runs are not i.i.d.** Same prompt, model, bug, and environment give correlated, not independent, evidence. probity logs seed / environment / task-instance / model-version / prompt diversity and does not let a monoculture of runs masquerade as k independent witnesses. +- **Pass rate ≠ correctness rate.** `p̂` describes the pass rate *against verifier V*, never correctness against the real world. This sentence is carved into the philosophy. -| Evaluation weakness | Improvement | -|---|---| -| Single-run optimism | Repeated trials | -| Self-reported completion | Claim/evidence separation | -| Test oracle mutation | Protected-path audit | -| Hidden scope changes | Allowed-path audit | -| Overconfident small samples | Wilson interval + INSUFFICIENT | -| Judge hallucination | Zero-LLM built-in verdict path | -| Hard-to-review reports | Evidence bundle and repro command | +## 10. Hidden-instance leakage policy -## Limits +If an agent can resubmit and read detailed hidden-test failures, it learns the held-out set and the held-out becomes public. Policy: public tests are debug-usable; hidden failures return **coarse category only** (no stack trace, no exact input), with a capped attempt budget, and **final evaluation uses a fresh held-out seed**. Otherwise the ruler decays into training data. -Probity does not: +## 11. Reporting: three axes, never one number -- prove correctness; -- prove that an agent never makes false claims; -- detect all hallucinations; -- rank models; -- replace a human security review; -- validate open-ended work without a deterministic checker; -- make a bad checker good. +probity does **not** emit `Reliability: 93%` — that fuses two different things (pass rate against the verifier, and the verifier's fidelity to the real goal). It reports three separate axes: -## Best current framing +- **Pass stability** — the k-trial pass rate + Wilson CI. +- **Criterion validity** — provenance tier (§4) + mutation score + known-bad rejection + spec coverage. +- **Tamper resistance** — runner external? tests read-only? network off? writable paths restricted? -Use: +High pass stability + Tier-0 criterion = little value. Medium stability + Tier-3 criterion = meaningful. The reader sees *why* to (dis)trust a verdict, not a laundered scalar. -```text -Agent reliability methodology for false-green testing of AI coding agents. -``` +## 12. Decision semantics — a falsifier, not a certifier -Avoid: +Not PASS/FAIL. Four labels: -```text -The only agent evaluation tool. -Proves agents are correct. -Detects all hallucinations. -Guarantees safe code. -``` +- **`REFUTED`** — a hard failure occurred (test failed, forbidden path modified, fixture tampered, canary triggered, known-bad behavior reproduced). The claim is falsified. +- **`INVALID_CRITERION`** — the verification design does not stand (agent authored its own criterion, spec modified after the run, tests unhashed, hidden tests seen by the agent, verifier run by the agent). Not "agent passed"; "the test was not a test." +- **`INSUFFICIENT`** — not refuted, evidence too weak (k too small, only public tests, low spec coverage, low mutation score, undeclared invariants under nondeterminism). The honest answer, and often the correct one. +- **`SURVIVED_SCOPE`** — survived declared falsification: *not refuted under R@…, S@…, V@…, P@…, k runs*. **Never** "certified reliable." The precise phrase is **not refuted under declared conditions**. + +## 13. Threat model and limitations *(stated, not hidden)* + +- **The oracle only moves.** probity relocates trust to the criterion; in the common solo-developer case there is no independent third party, so the binding constraint is *social* (will anyone author a Tier-3 criterion?), not cryptographic. probity's answer: be loud and useful even at low tiers — stamp `SELF_CHECK_ONLY` and say plainly "this proves nothing about reliability." Its honesty must not depend on the user being rigorous. +- **Goodhart's reach.** Public-principle/hidden-instance raises the cost of gaming but only covers tasks with a property/generator; fixed-judgment tasks remain Goodhartable. probity reliably catches *crude* evasion, *partially* catches sophisticated evasion. It does not close the door; it makes the door expensive. +- **Distribution shift.** "Not refuted under benchmark distribution D" is necessary, not sufficient, for production reliability. The real oracle is production (shadow mode, incident feedback); probity is a pre-deployment gate, not a deployment license. +- **The verifier is attack surface.** runner, sandbox, hash recorder, and report generator can have bugs. Hence a minimal TCB and the rule that the fancy oracles (§7) are themselves subject to probity-grade scrutiny. +- **Success condition.** Not "unfakeable." **The cost of fooling probity exceeds the benefit of the lie.** Fooling it should require stealing the held-out suite, breaking the read-only mount, forging signed evidence, escaping the sandbox, or predicting a post-submission random seed — a different epistemic level from "I tested it, it passed." + +## 14. Related work and novelty + +Every individual pillar is precedented; we claim none of them as new. + +- Deterministic (no-LLM) evidence + capability-denial "trust regimes" + Wilson-score CIs are already fused for ML-engineering reward-hacking by **RewardHackingAgents** [RHA] — the closest prior art; our related-work must distinguish from its trust-regime ladder. +- Capped/randomized-test cheating detection [CapCode]; rigorous-benchmark best practices incl. filesystem isolation and ground-truth separation [RigorousBench]; weak/leaked/self-authored tests as a false-pass source [UTBoost; SWE-Bench+; STING]; deterministic architectural boundaries and revocable capabilities [DetBoundaries; LingeringAuth]; reward-hacking definitions and catalogs [Skalse2022; GamingVerifiers; RewardHackBench]; LLM-judge unreliability [SelfPref; NoFreeLabels]; Wilson CIs for small-n evals [NoCLT; StatGuarantees]; the oracle problem and metamorphic testing [Barr2015; Chen2018]. + +**What survives as probity's contribution** is packaging and generalization, stated narrowly: +1. **Criterion-provenance tiers (Tier 0–4)** as a formal, reusable hackability ordering over oracles — not found named in the literature. +2. **Cheap-refutation / priced-confirmation** turned into an operational verdict policy. +3. **Falsifier decision semantics** (`REFUTED / INVALID_CRITERION / INSUFFICIENT / SURVIVED_SCOPE`) replacing PASS/FAIL; RewardHackingAgents' "inconclusive" is the nearest neighbor. +4. **Generalization** from a fixed benchmark to a falsifier for *arbitrary* scoped agent trust claims. + +We explicitly do **not** claim zero-LLM verdicts, capability denial, or Wilson-CI honesty as individually novel. + +## 15. Scope of claims — what probity does not do + +It does not certify reliability; it does not pronounce on deep correctness (right design, unstated intent) — there is no deterministic ruler for that, so it returns `INSUFFICIENT` and hands it to a human; it does not verify anything under a Tier-0 criterion; it does not guarantee production behavior. Refusing to fake a verdict is the feature. + +--- + +*Citation keys refer to `docs/methodology-references.md`. arXiv ids in that file marked `unverified` are search-derived and must be confirmed against arXiv before formal citation.* diff --git a/docs/methodology-references.md b/docs/methodology-references.md new file mode 100644 index 0000000..52e685c --- /dev/null +++ b/docs/methodology-references.md @@ -0,0 +1,40 @@ +# References — methodology prior art + +Keys used in `docs/METHODOLOGY.md`. **`verified`** = the paper was opened and read directly. +**`unverified`** = arXiv id is search-snippet-derived and MUST be confirmed against arXiv before +any formal citation (snippet ids are error-prone). + +## Closest prior art / contrast +- **[RHA]** Atinafu & Cohen (2026). *RewardHackingAgents: Benchmarking Evaluation Integrity for LLM ML-Engineering Agents.* arXiv:2603.11337. **verified.** Fuses no-LLM evidence + capability-denial "trust regimes" + Wilson CIs — the single closest prior art; distinguish from its trust-regime ladder. +- **[CapCode]** Lodkaew et al. (2026). *Do Coding Agents Deceive Us? … Capped Evaluation with Randomized Tests.* arXiv:2606.07379. **verified.** +- **[RigorousBench]** Zhu et al. (2026). *Establishing Best Practices for Building Rigorous Agentic Benchmarks.* arXiv:2507.02825. **verified.** + +## Weak / leaked / self-authored tests → false pass (supports provenance tiers) +- **[UTBoost]** Aleithan et al. (2026). *UTBoost: Rigorous Evaluation of Coding Agents on SWE-Bench.* arXiv:2506.09289. unverified. +- **[STING]** (2026). *Are Benchmark Tests Strong Enough? Mutation-Guided Diagnosis and Augmentation of Regression Suites.* arXiv:2604.01518. **verified.** +- **[SWE-Bench+]** Aleithan et al. (2024). *SWE-Bench+: Enhanced Coding Benchmark for LLMs.* arXiv:2410.06992. unverified. + +## Capability denial / privilege separation (supports P1) +- **[DetBoundaries]** (2026). *Trustworthy Agentic AI Requires Deterministic Architectural Boundaries.* arXiv:2602.09947. unverified. +- **[LingeringAuth]** (2026). *Lingering Authority: Revocable Resource-and-Effect Capabilities for Coding Agents.* arXiv:2606.22504. unverified. +- **[AgentWarden]** Zhang et al. (2026). *Beyond Static Sandboxing: Learned Capability Governance for Autonomous AI Agents.* arXiv:2604.11839. unverified. (Note: re-introduces a fine-tuned LLM classifier in enforcement — contrast.) + +## Reward hacking / specification gaming (threat model) +- **[Skalse2022]** Skalse et al. (2022). *Defining and Characterizing Reward Hacking.* arXiv:2209.13085 (NeurIPS 2022). unverified. +- **[GamingVerifiers]** (2026). *LLMs Gaming Verifiers: RLVR can Lead to Reward Hacking.* arXiv:2604.15149. unverified. +- **[RewardHackBench]** (2026). *Reward Hacking Benchmark: Measuring Exploits in LLM Agents with Tool Use.* arXiv:2605.02964. unverified. + +## LLM-as-judge unreliability (supports zero-LLM verdict) +- **[SelfPref]** Panickssery et al. (2024). *Self-Preference Bias in LLM-as-a-Judge.* arXiv:2410.21819. unverified. +- **[NoFreeLabels]** (2025). *No Free Labels: Limitations of LLM-as-a-Judge Without Human Grounding.* arXiv:2503.05061. unverified. + +## Statistical honesty (supports Wilson CI / priced confirmation) +- **[NoCLT]** (2025). *Position: Don't Use the CLT in LLM Evals With Fewer Than a Few Hundred Samples.* arXiv:2503.01747. unverified. +- **[StatGuarantees]** Miller et al. (2026). *Efficient Evaluation of LLM Performance with Statistical Guarantees.* arXiv:2601.20251. unverified. + +## Oracle problem / metamorphic testing (foundations) +- **[Barr2015]** Barr, Harman, McMinn, Shahbaz & Yoo (2015). *The Oracle Problem in Software Testing: A Survey.* IEEE TSE. (peer-reviewed; verify DOI.) +- **[Chen2018]** Chen et al. (2018). *Metamorphic Testing: A Review of Challenges and Opportunities.* ACM CSUR. (peer-reviewed; verify DOI.) + +## Adjacent OSS (landscape, not citations) +Closest philosophy (LLM-based, no sandbox/stats): `penacristian/coding-agent-reliability`. Same-name static scanner: `vinicq/falsegreen`. Cross-domain twin (trading): `bettyguo/agent-backtest-lab`. Ledger analog: `sipyourdrink-ltd/bernstein` (HMAC audit chain). Composable oracles: Stryker/PITest (mutation), Hypothesis (property), SWE-bench (harness), cordum / e2b (policy gate / sandbox). From 1d6a08d47f4abe17fd2439dcb6cbad5e6e6b3c32 Mon Sep 17 00:00:00 2001 From: boyam01 Date: Sun, 5 Jul 2026 08:32:14 +0800 Subject: [PATCH 2/3] fix: preserve existing METHODOLOGY.md; add expanded spec as docs/PROTOCOL.md Previous commit on this branch overwrote the existing docs/METHODOLOGY.md. Restore it verbatim from main; the expanded, prior-art-positioned protocol now lives in the new docs/PROTOCOL.md, framed as evolving from (not replacing) the current method. Additive only. --- docs/METHODOLOGY.md | 205 +++++++++++++++++++++----------------------- docs/PROTOCOL.md | 151 ++++++++++++++++++++++++++++++++ 2 files changed, 248 insertions(+), 108 deletions(-) create mode 100644 docs/PROTOCOL.md diff --git a/docs/METHODOLOGY.md b/docs/METHODOLOGY.md index cf99020..b2612e7 100644 --- a/docs/METHODOLOGY.md +++ b/docs/METHODOLOGY.md @@ -1,149 +1,138 @@ -# probity — Methodology +# Probity methodology -*An adversarial evidence protocol for agent trust claims.* -Draft v0.2 · falsifier-not-certifier · zero-LLM verdict path. +Probity is a method for testing whether an AI coding agent's success claim is +supported by evidence. The target failure is **false green**: the workflow looks +green, but the evidence does not justify trusting the result. -> **One line.** probity does not decide whether an AI agent is *reliable*. It decides whether a specific, scoped **claim** survives *cheap falsification* under a declared criterion, verifier, sandbox, and attempt budget — and, if it survives, states what further cost would raise confidence. It is a machine for making trust claims **expensive to fake**, not a machine that manufactures trust. +## The problem ---- - -## 1. Problem - -An AI coding agent that reports success will, often enough, report success it did not earn: tests silently skipped, the wrong file edited, the acceptance criterion reinterpreted until it passes, the evaluator itself tampered with. This is **false-green**. Empirically it is not rare — augmenting SWE-bench tests reclassifies hundreds of "passing" patches as failing [UTBoost]; ~33% of "successful" SWE-bench patches leaked the fix from the issue text [SWE-Bench+]; 77% of SWE-bench Verified instances admit a semantically wrong variant that still passes the given tests [STING]; RL-trained agents overwrite unit tests and monkey-patch scorers [GamingVerifiers]. - -The usual response — ask another model to judge — hands the ruler to material of the same kind as the thing measured. An LLM judge scores **plausibility**, not correctness; it favors its own outputs and cannot reliably grade what it cannot itself solve [SelfPref; NoFreeLabels]. Verify with the wrong ruler and the answer is wrong by construction. - -The deeper fact is the **oracle problem** [Barr2015]: to verify correctness you need a trustworthy source of ground truth, and there is no free lunch that manufactures one. probity does not solve the oracle problem. It does three things a certifier cannot: it **relocates** the trust to a place we can name and audit (the criterion), it **denies** the agent the ability to corrupt that place, and it is **honest about what it cannot verify**. - -## 2. Design principles - -Three principles do the load-bearing work. Each is individually precedented; the contribution is the way they are combined and generalized (§14). - -- **P1 — Capability denial over detection.** Do not check *whether* the agent cheated; remove the *ability* to cheat. Read-only tests, patch-only submission, an external runner, no network. Detection is an arms race; denial is structural [DetBoundaries; LingeringAuth]. -- **P2 — Separation of powers over trust in a smart checker.** probity's strength must not come from the checker being clever. It comes from the **doer being unable to control two of the three roles**: who *defines* the criterion, who *executes* the verifier, who *records* the evidence. Control any two and probity degrades to *self-grading*. -- **P3 — Public principle, hidden instance.** The algorithm is public; the secret is the per-challenge seed and held-out cases, generated *after* the patch is submitted. This is the Kerckhoffs stance applied to evaluation, and it is the only defense against Goodharting a fixed check [Skalse2022]. - -## 3. The claim object - -probity never verifies "this agent is reliable." It verifies a **scoped claim**: +Agent evaluation often starts from a fragile signal: +```text +The agent finished once -> the task is done -> ship it ``` -Under repo R@commit, spec S@hash, verifier V@image-hash, sandbox policy P@hash, -and attempt budget B, does the agent's submitted artifact satisfy claim C? -``` - -Everything is content-addressed and frozen before the run. Narrowness is not a weakness here; it is the source of the strength. "Reliable in the real world" is unfalsifiable and therefore un-verifiable; "not refuted under these declared conditions" is both. -## 4. Criterion provenance tiers *(core contribution)* +That signal collapses several different facts into one story: -The verdict's meaning is inherited entirely from the criterion's trustworthiness, so criterion provenance is a **first-class field of every report**, not metadata. We order criteria by how hackable they are by the doer: +- whether the final state passed the checker; +- whether the agent honestly described that final state; +- whether the agent edited the test oracle; +- whether the result repeats across independent runs; +- whether the environment was healthy. -| Tier | Criterion source | Trust | Label | -|---|---|---|---| -| **0** | Agent-authored tests / agent-defined success | ~none | `SELF_CHECK_ONLY` | -| **1** | Public tests / the repo's existing CI | weak | `PUBLIC_REPLAY_PASS` | -| **2** | Externally-authored spec + public tests | medium | `EXTERNAL_SPEC_PASS` | -| **3** | External held-out tests / reference implementation | strong | `HELDOUT_PASS` | -| **4** | Formal spec / proof or model checker | strongest, narrowest | `FORMAL_OR_PROOF_BACKED` | +Probity separates those facts. -**Hard rule.** A Tier-0 criterion may *never* support a reliability claim. Agent-authored tests can answer "does this agent self-check?" — never "is the result reliable?". An agent may *propose* tests or attack cases; those are a **candidate criterion** and become evidence only after an external owner accepts them, on the *next* run. This directly answers the "who writes the ruler" crack: probity does not pretend the oracle problem vanishes; it makes the ruler's provenance auditable and refuses to let the doer forge it. +## Method -## 5. Verification architecture (P1 in practice) - -The minimal trusted computing base is deliberately small — the more intelligence in the core, the less it can be trusted (§13). The core does only: freeze inputs, apply the patch, enforce the sandbox, run the verifier, record hashes, emit the evidence bundle. - -| Attack | Weak (detection) | probity (denial) | -|---|---|---| -| delete/alter fixtures | scan the diff for it | fixtures mounted read-only | -| edit or skip tests | grep the diff | tests not on any writable path; runner runs them | -| forge a pass | parse agent output | never trust agent output; only runner output counts | -| hardcode public cases | add a few public tests | hidden cases generated post-submission | -| environment fingerprinting | scan for suspicious code | multi-environment re-run, no network, frozen clock/seed | - -The doer submits a **patch or artifact only**. It does not submit test results, its own logs, its own pass condition, or a modified verifier. - -## 6. Multi-oracle verification - -A single fixed test set is the easiest oracle to game. probity runs several, and — critically — each is **deterministic given a seed that is drawn after submission**: reproducible because the seed is logged, un-overfittable because it was unknown in advance (*post-seed deterministic, pre-run unpredictable*). +```text +registered task + -> fresh isolated worktree per run + -> deterministic checker + -> claim/evidence comparison + -> integrity flags + -> repeated-trial statistics + -> PASS / KILL / INSUFFICIENT +``` -- public tests (debug-usable), **held-out** tests (final evaluation only), -- property-based and metamorphic relations, differential comparison to a reference, -- tamper/canary checks and an env canary (pre=ok/post=ok). +The method has six parts. -Reach is honest: property/metamorphic/generative oracles only exist for tasks with checkable structure. Tasks without it ("make this readable," a bug with no reproducer) have no seed to hide and fall back to an externally-authored fixed criterion — or to `INSUFFICIENT` (§12). +### 1. Register the task before the run -## 7. Oracle QA — testing the ruler *(criterion fitness)* +A task defines: -High pass rate against a weak ruler is *confidently wrong*. So probity also measures the **criterion**, not only the agent: +- the workspace; +- the agent command; +- allowed files; +- protected files; +- critical rules; +- the deterministic checker; +- the reliability claim `r`; +- the number of trials `k`. -- **Negative controls.** A corpus of known-bad artifacts (deletes core function but keeps tests green, hardcodes fixtures, mocks the external API, drops an error branch) that the criterion *should* reject. If they pass, the oracle is bad, not the agent good. -- **Mutation adequacy.** Mutate code/spec and measure the kill rate; a low mutation score means the tests verify little [STING; Stryker/PITest]. -- **Spec-coverage map.** Each clause of the spec maps to tests, or the report says `INSUFFICIENT: claim includes untested spec clauses`. +This prevents the evaluation from moving the target after seeing the output. -Oracle QA is itself authored and gameable and is not free; it raises the floor, it does not close the problem (§13). Its outputs (mutation score, known-bad rejection, canary detection) are reported as their own axis (§11), never folded into the pass rate. +### 2. Keep the checker deterministic -## 8. Nondeterminism: invariants, not traces +The built-in verdict path is: -Trace divergence is a *weak* signal — many correct solutions legitimately differ (timestamps, request ids, independent-job ordering). probity therefore requires the task owner to declare `allowed_variance` and `required_invariants` (final schema, DB state, balance conservation, no forbidden writes, reference-equivalent result), and it verifies **invariants under k randomized runs**. Raw trace difference is a hard red flag *only* when the task itself claims a deterministic trace. +```text +checker -> stats -> verdict +``` -## 9. Statistical framework +It contains no LLM call, no LLM judge, and no LLM-written failure summary. -Reliability is a claim about a distribution, so it is estimated over k trials with a **Wilson score interval**, which has correct coverage where the Wald/CLT approximation fails at small n [NoCLT; StatGuarantees]. +Custom script checkers are allowed, but they are trust inputs owned by the task +author. Probity can record their output; it cannot prove that an arbitrary +custom checker is semantically correct. -- **Cheap refutation, priced confirmation.** A few failures refute a high claim; confirming a high claim is expensive. In the bundled demo, an r=0.9 claim with 7/10 observed yields a Wilson CI of [0.397, 0.892] — refuting any claim above 0.892 while confirming nothing above 0.397. Confirming 0.9 at 95% would require ≈35 consecutive clean runs. This asymmetry is not a defect; it is the product: *tell me how much confidence you want and I price the sample cost.* -- **Runs are not i.i.d.** Same prompt, model, bug, and environment give correlated, not independent, evidence. probity logs seed / environment / task-instance / model-version / prompt diversity and does not let a monoculture of runs masquerade as k independent witnesses. -- **Pass rate ≠ correctness rate.** `p̂` describes the pass rate *against verifier V*, never correctness against the real world. This sentence is carved into the philosophy. +### 3. Separate claim from evidence -## 10. Hidden-instance leakage policy +The agent's final self-report is parsed separately from the checker's result. -If an agent can resubmit and read detailed hidden-test failures, it learns the held-out set and the held-out becomes public. Policy: public tests are debug-usable; hidden failures return **coarse category only** (no stack trace, no exact input), with a capped attempt budget, and **final evaluation uses a fresh held-out seed**. Otherwise the ruler decays into training data. +```text +agent: CLAIM: success +checker: failed +result: false_claim +``` -## 11. Reporting: three axes, never one number +This matters because a failed run that claims success can still fool a human +single-run workflow. -probity does **not** emit `Reliability: 93%` — that fuses two different things (pass rate against the verifier, and the verifier's fidelity to the real goal). It reports three separate axes: +### 4. Treat oracle tampering as an integrity failure -- **Pass stability** — the k-trial pass rate + Wilson CI. -- **Criterion validity** — provenance tier (§4) + mutation score + known-bad rejection + spec coverage. -- **Tamper resistance** — runner external? tests read-only? network off? writable paths restricted? +If the agent edits protected tests or other protected oracle files, the run is +not treated as evidence of capability. It is an audit-integrity failure. -High pass stability + Tier-0 criterion = little value. Medium stability + Tier-3 criterion = meaningful. The reader sees *why* to (dis)trust a verdict, not a laundered scalar. +### 5. Repeat under fresh isolation -## 12. Decision semantics — a falsifier, not a certifier +Each run starts from a fresh git worktree. One lucky run is not enough evidence. +Repeated runs expose flakiness, degradation, partial fixes, and rare destructive +actions. -Not PASS/FAIL. Four labels: +### 6. Prefer honest refusal over overclaiming -- **`REFUTED`** — a hard failure occurred (test failed, forbidden path modified, fixture tampered, canary triggered, known-bad behavior reproduced). The claim is falsified. -- **`INVALID_CRITERION`** — the verification design does not stand (agent authored its own criterion, spec modified after the run, tests unhashed, hidden tests seen by the agent, verifier run by the agent). Not "agent passed"; "the test was not a test." -- **`INSUFFICIENT`** — not refuted, evidence too weak (k too small, only public tests, low spec coverage, low mutation score, undeclared invariants under nondeterminism). The honest answer, and often the correct one. -- **`SURVIVED_SCOPE`** — survived declared falsification: *not refuted under R@…, S@…, V@…, P@…, k runs*. **Never** "certified reliable." The precise phrase is **not refuted under declared conditions**. +Probity uses a confidence interval over repeated trials. Small `k` can refute a +high-reliability claim, but it usually cannot confirm one. -## 13. Threat model and limitations *(stated, not hidden)* +That is why `INSUFFICIENT` is a first-class outcome, not an error. -- **The oracle only moves.** probity relocates trust to the criterion; in the common solo-developer case there is no independent third party, so the binding constraint is *social* (will anyone author a Tier-3 criterion?), not cryptographic. probity's answer: be loud and useful even at low tiers — stamp `SELF_CHECK_ONLY` and say plainly "this proves nothing about reliability." Its honesty must not depend on the user being rigorous. -- **Goodhart's reach.** Public-principle/hidden-instance raises the cost of gaming but only covers tasks with a property/generator; fixed-judgment tasks remain Goodhartable. probity reliably catches *crude* evasion, *partially* catches sophisticated evasion. It does not close the door; it makes the door expensive. -- **Distribution shift.** "Not refuted under benchmark distribution D" is necessary, not sufficient, for production reliability. The real oracle is production (shadow mode, incident feedback); probity is a pre-deployment gate, not a deployment license. -- **The verifier is attack surface.** runner, sandbox, hash recorder, and report generator can have bugs. Hence a minimal TCB and the rule that the fancy oracles (§7) are themselves subject to probity-grade scrutiny. -- **Success condition.** Not "unfakeable." **The cost of fooling probity exceeds the benefit of the lie.** Fooling it should require stealing the held-out suite, breaking the read-only mount, forging signed evidence, escaping the sandbox, or predicting a post-submission random seed — a different epistemic level from "I tested it, it passed." +## What the method improves -## 14. Related work and novelty +| Evaluation weakness | Improvement | +|---|---| +| Single-run optimism | Repeated trials | +| Self-reported completion | Claim/evidence separation | +| Test oracle mutation | Protected-path audit | +| Hidden scope changes | Allowed-path audit | +| Overconfident small samples | Wilson interval + INSUFFICIENT | +| Judge hallucination | Zero-LLM built-in verdict path | +| Hard-to-review reports | Evidence bundle and repro command | -Every individual pillar is precedented; we claim none of them as new. +## Limits -- Deterministic (no-LLM) evidence + capability-denial "trust regimes" + Wilson-score CIs are already fused for ML-engineering reward-hacking by **RewardHackingAgents** [RHA] — the closest prior art; our related-work must distinguish from its trust-regime ladder. -- Capped/randomized-test cheating detection [CapCode]; rigorous-benchmark best practices incl. filesystem isolation and ground-truth separation [RigorousBench]; weak/leaked/self-authored tests as a false-pass source [UTBoost; SWE-Bench+; STING]; deterministic architectural boundaries and revocable capabilities [DetBoundaries; LingeringAuth]; reward-hacking definitions and catalogs [Skalse2022; GamingVerifiers; RewardHackBench]; LLM-judge unreliability [SelfPref; NoFreeLabels]; Wilson CIs for small-n evals [NoCLT; StatGuarantees]; the oracle problem and metamorphic testing [Barr2015; Chen2018]. +Probity does not: -**What survives as probity's contribution** is packaging and generalization, stated narrowly: -1. **Criterion-provenance tiers (Tier 0–4)** as a formal, reusable hackability ordering over oracles — not found named in the literature. -2. **Cheap-refutation / priced-confirmation** turned into an operational verdict policy. -3. **Falsifier decision semantics** (`REFUTED / INVALID_CRITERION / INSUFFICIENT / SURVIVED_SCOPE`) replacing PASS/FAIL; RewardHackingAgents' "inconclusive" is the nearest neighbor. -4. **Generalization** from a fixed benchmark to a falsifier for *arbitrary* scoped agent trust claims. +- prove correctness; +- prove that an agent never makes false claims; +- detect all hallucinations; +- rank models; +- replace a human security review; +- validate open-ended work without a deterministic checker; +- make a bad checker good. -We explicitly do **not** claim zero-LLM verdicts, capability denial, or Wilson-CI honesty as individually novel. +## Best current framing -## 15. Scope of claims — what probity does not do +Use: -It does not certify reliability; it does not pronounce on deep correctness (right design, unstated intent) — there is no deterministic ruler for that, so it returns `INSUFFICIENT` and hands it to a human; it does not verify anything under a Tier-0 criterion; it does not guarantee production behavior. Refusing to fake a verdict is the feature. +```text +Agent reliability methodology for false-green testing of AI coding agents. +``` ---- +Avoid: -*Citation keys refer to `docs/methodology-references.md`. arXiv ids in that file marked `unverified` are search-derived and must be confirmed against arXiv before formal citation.* +```text +The only agent evaluation tool. +Proves agents are correct. +Detects all hallucinations. +Guarantees safe code. +``` diff --git a/docs/PROTOCOL.md b/docs/PROTOCOL.md new file mode 100644 index 0000000..2929458 --- /dev/null +++ b/docs/PROTOCOL.md @@ -0,0 +1,151 @@ +# probity — Methodology + +*An adversarial evidence protocol for agent trust claims.* +Draft v0.2 · falsifier-not-certifier · zero-LLM verdict path. + +> **One line.** probity does not decide whether an AI agent is *reliable*. It decides whether a specific, scoped **claim** survives *cheap falsification* under a declared criterion, verifier, sandbox, and attempt budget — and, if it survives, states what further cost would raise confidence. It is a machine for making trust claims **expensive to fake**, not a machine that manufactures trust. + +> **Relationship to `METHODOLOGY.md`.** That document states probity's *current implemented* method. This one is the *formal, expanded protocol* (v0.2) it is evolving toward — same philosophy, more parts nailed down, positioned against prior art. Read `METHODOLOGY.md` first for what ships today. + +--- + +## 1. Problem + +An AI coding agent that reports success will, often enough, report success it did not earn: tests silently skipped, the wrong file edited, the acceptance criterion reinterpreted until it passes, the evaluator itself tampered with. This is **false-green**. Empirically it is not rare — augmenting SWE-bench tests reclassifies hundreds of "passing" patches as failing [UTBoost]; ~33% of "successful" SWE-bench patches leaked the fix from the issue text [SWE-Bench+]; 77% of SWE-bench Verified instances admit a semantically wrong variant that still passes the given tests [STING]; RL-trained agents overwrite unit tests and monkey-patch scorers [GamingVerifiers]. + +The usual response — ask another model to judge — hands the ruler to material of the same kind as the thing measured. An LLM judge scores **plausibility**, not correctness; it favors its own outputs and cannot reliably grade what it cannot itself solve [SelfPref; NoFreeLabels]. Verify with the wrong ruler and the answer is wrong by construction. + +The deeper fact is the **oracle problem** [Barr2015]: to verify correctness you need a trustworthy source of ground truth, and there is no free lunch that manufactures one. probity does not solve the oracle problem. It does three things a certifier cannot: it **relocates** the trust to a place we can name and audit (the criterion), it **denies** the agent the ability to corrupt that place, and it is **honest about what it cannot verify**. + +## 2. Design principles + +Three principles do the load-bearing work. Each is individually precedented; the contribution is the way they are combined and generalized (§14). + +- **P1 — Capability denial over detection.** Do not check *whether* the agent cheated; remove the *ability* to cheat. Read-only tests, patch-only submission, an external runner, no network. Detection is an arms race; denial is structural [DetBoundaries; LingeringAuth]. +- **P2 — Separation of powers over trust in a smart checker.** probity's strength must not come from the checker being clever. It comes from the **doer being unable to control two of the three roles**: who *defines* the criterion, who *executes* the verifier, who *records* the evidence. Control any two and probity degrades to *self-grading*. +- **P3 — Public principle, hidden instance.** The algorithm is public; the secret is the per-challenge seed and held-out cases, generated *after* the patch is submitted. This is the Kerckhoffs stance applied to evaluation, and it is the only defense against Goodharting a fixed check [Skalse2022]. + +## 3. The claim object + +probity never verifies "this agent is reliable." It verifies a **scoped claim**: + +``` +Under repo R@commit, spec S@hash, verifier V@image-hash, sandbox policy P@hash, +and attempt budget B, does the agent's submitted artifact satisfy claim C? +``` + +Everything is content-addressed and frozen before the run. Narrowness is not a weakness here; it is the source of the strength. "Reliable in the real world" is unfalsifiable and therefore un-verifiable; "not refuted under these declared conditions" is both. + +## 4. Criterion provenance tiers *(core contribution)* + +The verdict's meaning is inherited entirely from the criterion's trustworthiness, so criterion provenance is a **first-class field of every report**, not metadata. We order criteria by how hackable they are by the doer: + +| Tier | Criterion source | Trust | Label | +|---|---|---|---| +| **0** | Agent-authored tests / agent-defined success | ~none | `SELF_CHECK_ONLY` | +| **1** | Public tests / the repo's existing CI | weak | `PUBLIC_REPLAY_PASS` | +| **2** | Externally-authored spec + public tests | medium | `EXTERNAL_SPEC_PASS` | +| **3** | External held-out tests / reference implementation | strong | `HELDOUT_PASS` | +| **4** | Formal spec / proof or model checker | strongest, narrowest | `FORMAL_OR_PROOF_BACKED` | + +**Hard rule.** A Tier-0 criterion may *never* support a reliability claim. Agent-authored tests can answer "does this agent self-check?" — never "is the result reliable?". An agent may *propose* tests or attack cases; those are a **candidate criterion** and become evidence only after an external owner accepts them, on the *next* run. This directly answers the "who writes the ruler" crack: probity does not pretend the oracle problem vanishes; it makes the ruler's provenance auditable and refuses to let the doer forge it. + +## 5. Verification architecture (P1 in practice) + +The minimal trusted computing base is deliberately small — the more intelligence in the core, the less it can be trusted (§13). The core does only: freeze inputs, apply the patch, enforce the sandbox, run the verifier, record hashes, emit the evidence bundle. + +| Attack | Weak (detection) | probity (denial) | +|---|---|---| +| delete/alter fixtures | scan the diff for it | fixtures mounted read-only | +| edit or skip tests | grep the diff | tests not on any writable path; runner runs them | +| forge a pass | parse agent output | never trust agent output; only runner output counts | +| hardcode public cases | add a few public tests | hidden cases generated post-submission | +| environment fingerprinting | scan for suspicious code | multi-environment re-run, no network, frozen clock/seed | + +The doer submits a **patch or artifact only**. It does not submit test results, its own logs, its own pass condition, or a modified verifier. + +## 6. Multi-oracle verification + +A single fixed test set is the easiest oracle to game. probity runs several, and — critically — each is **deterministic given a seed that is drawn after submission**: reproducible because the seed is logged, un-overfittable because it was unknown in advance (*post-seed deterministic, pre-run unpredictable*). + +- public tests (debug-usable), **held-out** tests (final evaluation only), +- property-based and metamorphic relations, differential comparison to a reference, +- tamper/canary checks and an env canary (pre=ok/post=ok). + +Reach is honest: property/metamorphic/generative oracles only exist for tasks with checkable structure. Tasks without it ("make this readable," a bug with no reproducer) have no seed to hide and fall back to an externally-authored fixed criterion — or to `INSUFFICIENT` (§12). + +## 7. Oracle QA — testing the ruler *(criterion fitness)* + +High pass rate against a weak ruler is *confidently wrong*. So probity also measures the **criterion**, not only the agent: + +- **Negative controls.** A corpus of known-bad artifacts (deletes core function but keeps tests green, hardcodes fixtures, mocks the external API, drops an error branch) that the criterion *should* reject. If they pass, the oracle is bad, not the agent good. +- **Mutation adequacy.** Mutate code/spec and measure the kill rate; a low mutation score means the tests verify little [STING; Stryker/PITest]. +- **Spec-coverage map.** Each clause of the spec maps to tests, or the report says `INSUFFICIENT: claim includes untested spec clauses`. + +Oracle QA is itself authored and gameable and is not free; it raises the floor, it does not close the problem (§13). Its outputs (mutation score, known-bad rejection, canary detection) are reported as their own axis (§11), never folded into the pass rate. + +## 8. Nondeterminism: invariants, not traces + +Trace divergence is a *weak* signal — many correct solutions legitimately differ (timestamps, request ids, independent-job ordering). probity therefore requires the task owner to declare `allowed_variance` and `required_invariants` (final schema, DB state, balance conservation, no forbidden writes, reference-equivalent result), and it verifies **invariants under k randomized runs**. Raw trace difference is a hard red flag *only* when the task itself claims a deterministic trace. + +## 9. Statistical framework + +Reliability is a claim about a distribution, so it is estimated over k trials with a **Wilson score interval**, which has correct coverage where the Wald/CLT approximation fails at small n [NoCLT; StatGuarantees]. + +- **Cheap refutation, priced confirmation.** A few failures refute a high claim; confirming a high claim is expensive. In the bundled demo, an r=0.9 claim with 7/10 observed yields a Wilson CI of [0.397, 0.892] — refuting any claim above 0.892 while confirming nothing above 0.397. Confirming 0.9 at 95% would require ≈35 consecutive clean runs. This asymmetry is not a defect; it is the product: *tell me how much confidence you want and I price the sample cost.* +- **Runs are not i.i.d.** Same prompt, model, bug, and environment give correlated, not independent, evidence. probity logs seed / environment / task-instance / model-version / prompt diversity and does not let a monoculture of runs masquerade as k independent witnesses. +- **Pass rate ≠ correctness rate.** `p̂` describes the pass rate *against verifier V*, never correctness against the real world. This sentence is carved into the philosophy. + +## 10. Hidden-instance leakage policy + +If an agent can resubmit and read detailed hidden-test failures, it learns the held-out set and the held-out becomes public. Policy: public tests are debug-usable; hidden failures return **coarse category only** (no stack trace, no exact input), with a capped attempt budget, and **final evaluation uses a fresh held-out seed**. Otherwise the ruler decays into training data. + +## 11. Reporting: three axes, never one number + +probity does **not** emit `Reliability: 93%` — that fuses two different things (pass rate against the verifier, and the verifier's fidelity to the real goal). It reports three separate axes: + +- **Pass stability** — the k-trial pass rate + Wilson CI. +- **Criterion validity** — provenance tier (§4) + mutation score + known-bad rejection + spec coverage. +- **Tamper resistance** — runner external? tests read-only? network off? writable paths restricted? + +High pass stability + Tier-0 criterion = little value. Medium stability + Tier-3 criterion = meaningful. The reader sees *why* to (dis)trust a verdict, not a laundered scalar. + +## 12. Decision semantics — a falsifier, not a certifier + +Not PASS/FAIL. Four labels: + +- **`REFUTED`** — a hard failure occurred (test failed, forbidden path modified, fixture tampered, canary triggered, known-bad behavior reproduced). The claim is falsified. +- **`INVALID_CRITERION`** — the verification design does not stand (agent authored its own criterion, spec modified after the run, tests unhashed, hidden tests seen by the agent, verifier run by the agent). Not "agent passed"; "the test was not a test." +- **`INSUFFICIENT`** — not refuted, evidence too weak (k too small, only public tests, low spec coverage, low mutation score, undeclared invariants under nondeterminism). The honest answer, and often the correct one. +- **`SURVIVED_SCOPE`** — survived declared falsification: *not refuted under R@…, S@…, V@…, P@…, k runs*. **Never** "certified reliable." The precise phrase is **not refuted under declared conditions**. + +## 13. Threat model and limitations *(stated, not hidden)* + +- **The oracle only moves.** probity relocates trust to the criterion; in the common solo-developer case there is no independent third party, so the binding constraint is *social* (will anyone author a Tier-3 criterion?), not cryptographic. probity's answer: be loud and useful even at low tiers — stamp `SELF_CHECK_ONLY` and say plainly "this proves nothing about reliability." Its honesty must not depend on the user being rigorous. +- **Goodhart's reach.** Public-principle/hidden-instance raises the cost of gaming but only covers tasks with a property/generator; fixed-judgment tasks remain Goodhartable. probity reliably catches *crude* evasion, *partially* catches sophisticated evasion. It does not close the door; it makes the door expensive. +- **Distribution shift.** "Not refuted under benchmark distribution D" is necessary, not sufficient, for production reliability. The real oracle is production (shadow mode, incident feedback); probity is a pre-deployment gate, not a deployment license. +- **The verifier is attack surface.** runner, sandbox, hash recorder, and report generator can have bugs. Hence a minimal TCB and the rule that the fancy oracles (§7) are themselves subject to probity-grade scrutiny. +- **Success condition.** Not "unfakeable." **The cost of fooling probity exceeds the benefit of the lie.** Fooling it should require stealing the held-out suite, breaking the read-only mount, forging signed evidence, escaping the sandbox, or predicting a post-submission random seed — a different epistemic level from "I tested it, it passed." + +## 14. Related work and novelty + +Every individual pillar is precedented; we claim none of them as new. + +- Deterministic (no-LLM) evidence + capability-denial "trust regimes" + Wilson-score CIs are already fused for ML-engineering reward-hacking by **RewardHackingAgents** [RHA] — the closest prior art; our related-work must distinguish from its trust-regime ladder. +- Capped/randomized-test cheating detection [CapCode]; rigorous-benchmark best practices incl. filesystem isolation and ground-truth separation [RigorousBench]; weak/leaked/self-authored tests as a false-pass source [UTBoost; SWE-Bench+; STING]; deterministic architectural boundaries and revocable capabilities [DetBoundaries; LingeringAuth]; reward-hacking definitions and catalogs [Skalse2022; GamingVerifiers; RewardHackBench]; LLM-judge unreliability [SelfPref; NoFreeLabels]; Wilson CIs for small-n evals [NoCLT; StatGuarantees]; the oracle problem and metamorphic testing [Barr2015; Chen2018]. + +**What survives as probity's contribution** is packaging and generalization, stated narrowly: +1. **Criterion-provenance tiers (Tier 0–4)** as a formal, reusable hackability ordering over oracles — not found named in the literature. +2. **Cheap-refutation / priced-confirmation** turned into an operational verdict policy. +3. **Falsifier decision semantics** (`REFUTED / INVALID_CRITERION / INSUFFICIENT / SURVIVED_SCOPE`) replacing PASS/FAIL; RewardHackingAgents' "inconclusive" is the nearest neighbor. +4. **Generalization** from a fixed benchmark to a falsifier for *arbitrary* scoped agent trust claims. + +We explicitly do **not** claim zero-LLM verdicts, capability denial, or Wilson-CI honesty as individually novel. + +## 15. Scope of claims — what probity does not do + +It does not certify reliability; it does not pronounce on deep correctness (right design, unstated intent) — there is no deterministic ruler for that, so it returns `INSUFFICIENT` and hands it to a human; it does not verify anything under a Tier-0 criterion; it does not guarantee production behavior. Refusing to fake a verdict is the feature. + +--- + +*Citation keys refer to `docs/methodology-references.md`. arXiv ids in that file marked `unverified` are search-derived and must be confirmed against arXiv before formal citation.* From c8b8302786060dddf096e661c51cf533080ff488 Mon Sep 17 00:00:00 2001 From: boyam01 Date: Sun, 5 Jul 2026 09:05:13 +0800 Subject: [PATCH 3/3] docs: mark all 18 references verified (abs-page checked, 0 corrections) Each arXiv id opened + cross-checked by title/first-author; two non-arXiv entries confirmed at venue/DOI. Added first authors + years; fixed 4 title strings (subtitle/system-name). Updated PROTOCOL footer accordingly. --- docs/PROTOCOL.md | 2 +- docs/methodology-references.md | 42 +++++++++++++++++----------------- 2 files changed, 22 insertions(+), 22 deletions(-) diff --git a/docs/PROTOCOL.md b/docs/PROTOCOL.md index 2929458..262ff27 100644 --- a/docs/PROTOCOL.md +++ b/docs/PROTOCOL.md @@ -148,4 +148,4 @@ It does not certify reliability; it does not pronounce on deep correctness (righ --- -*Citation keys refer to `docs/methodology-references.md`. arXiv ids in that file marked `unverified` are search-derived and must be confirmed against arXiv before formal citation.* +*Citation keys refer to `docs/methodology-references.md`. all 18 arXiv/venue entries were verified against their abs pages on 2026-07-05 (0 corrections).* diff --git a/docs/methodology-references.md b/docs/methodology-references.md index 52e685c..f5ffac5 100644 --- a/docs/methodology-references.md +++ b/docs/methodology-references.md @@ -1,40 +1,40 @@ # References — methodology prior art -Keys used in `docs/METHODOLOGY.md`. **`verified`** = the paper was opened and read directly. -**`unverified`** = arXiv id is search-snippet-derived and MUST be confirmed against arXiv before -any formal citation (snippet ids are error-prone). +Keys used in `docs/PROTOCOL.md`. **All 18 entries verified 2026-07-05** — each arXiv id was +opened on its abs page and cross-checked by title + first author; the two non-arXiv entries were +confirmed at their venues/DOIs. First author and year shown for each. ## Closest prior art / contrast -- **[RHA]** Atinafu & Cohen (2026). *RewardHackingAgents: Benchmarking Evaluation Integrity for LLM ML-Engineering Agents.* arXiv:2603.11337. **verified.** Fuses no-LLM evidence + capability-denial "trust regimes" + Wilson CIs — the single closest prior art; distinguish from its trust-regime ladder. -- **[CapCode]** Lodkaew et al. (2026). *Do Coding Agents Deceive Us? … Capped Evaluation with Randomized Tests.* arXiv:2606.07379. **verified.** -- **[RigorousBench]** Zhu et al. (2026). *Establishing Best Practices for Building Rigorous Agentic Benchmarks.* arXiv:2507.02825. **verified.** +- **[RHA]** Atinafu, Y. et al. (2026). *RewardHackingAgents: Benchmarking Evaluation Integrity for LLM ML-Engineering Agents.* arXiv:2603.11337. **verified.** Fuses no-LLM evidence + capability-denial "trust regimes" + Wilson CIs — the closest prior art; distinguish from its trust-regime ladder. +- **[CapCode]** Lodkaew, T. et al. (2026). *Do Coding Agents Deceive Us? Detecting and Preventing Cheating via Capped Evaluation with Randomized Tests.* arXiv:2606.07379. **verified.** +- **[RigorousBench]** Zhu, Y. et al. (2025). *Establishing Best Practices for Building Rigorous Agentic Benchmarks.* arXiv:2507.02825. **verified.** ## Weak / leaked / self-authored tests → false pass (supports provenance tiers) -- **[UTBoost]** Aleithan et al. (2026). *UTBoost: Rigorous Evaluation of Coding Agents on SWE-Bench.* arXiv:2506.09289. unverified. -- **[STING]** (2026). *Are Benchmark Tests Strong Enough? Mutation-Guided Diagnosis and Augmentation of Regression Suites.* arXiv:2604.01518. **verified.** -- **[SWE-Bench+]** Aleithan et al. (2024). *SWE-Bench+: Enhanced Coding Benchmark for LLMs.* arXiv:2410.06992. unverified. +- **[UTBoost]** Yu, B. et al. (2025). *UTBoost: Rigorous Evaluation of Coding Agents on SWE-Bench.* arXiv:2506.09289. **verified.** +- **[STING]** Li, C. et al. (2026). *Are Benchmark Tests Strong Enough? Mutation-Guided Diagnosis and Augmentation of Regression Suites.* arXiv:2604.01518. **verified.** ("STING" is the framework name.) +- **[SWE-Bench+]** Aleithan, R. et al. (2024). *SWE-Bench+: Enhanced Coding Benchmark for LLMs.* arXiv:2410.06992. **verified.** ## Capability denial / privilege separation (supports P1) -- **[DetBoundaries]** (2026). *Trustworthy Agentic AI Requires Deterministic Architectural Boundaries.* arXiv:2602.09947. unverified. -- **[LingeringAuth]** (2026). *Lingering Authority: Revocable Resource-and-Effect Capabilities for Coding Agents.* arXiv:2606.22504. unverified. -- **[AgentWarden]** Zhang et al. (2026). *Beyond Static Sandboxing: Learned Capability Governance for Autonomous AI Agents.* arXiv:2604.11839. unverified. (Note: re-introduces a fine-tuned LLM classifier in enforcement — contrast.) +- **[DetBoundaries]** Bhattarai, M. et al. (2026). *Trustworthy Agentic AI Requires Deterministic Architectural Boundaries.* arXiv:2602.09947. **verified.** +- **[LingeringAuth]** Santos-Grueiro, I. et al. (2026). *Lingering Authority: Revocable Resource-and-Effect Capabilities for Coding Agents.* arXiv:2606.22504. **verified.** +- **[AgentWarden]** Sidik, B. et al. (2026). *Beyond Static Sandboxing: Learned Capability Governance for Autonomous AI Agents.* arXiv:2604.11839. **verified.** ("AgentWarden" is the system name; re-introduces a fine-tuned LLM classifier in enforcement — contrast.) ## Reward hacking / specification gaming (threat model) -- **[Skalse2022]** Skalse et al. (2022). *Defining and Characterizing Reward Hacking.* arXiv:2209.13085 (NeurIPS 2022). unverified. -- **[GamingVerifiers]** (2026). *LLMs Gaming Verifiers: RLVR can Lead to Reward Hacking.* arXiv:2604.15149. unverified. -- **[RewardHackBench]** (2026). *Reward Hacking Benchmark: Measuring Exploits in LLM Agents with Tool Use.* arXiv:2605.02964. unverified. +- **[Skalse2022]** Skalse, J. et al. (2022). *Defining and Characterizing Reward Hacking.* arXiv:2209.13085 (NeurIPS 2022). **verified.** +- **[GamingVerifiers]** Helff, L. et al. (2026). *LLMs Gaming Verifiers: RLVR can Lead to Reward Hacking.* arXiv:2604.15149. **verified.** +- **[RewardHackBench]** Thaman, K. et al. (2026). *Reward Hacking Benchmark: Measuring Exploits in LLM Agents with Tool Use.* arXiv:2605.02964. **verified.** ## LLM-as-judge unreliability (supports zero-LLM verdict) -- **[SelfPref]** Panickssery et al. (2024). *Self-Preference Bias in LLM-as-a-Judge.* arXiv:2410.21819. unverified. -- **[NoFreeLabels]** (2025). *No Free Labels: Limitations of LLM-as-a-Judge Without Human Grounding.* arXiv:2503.05061. unverified. +- **[SelfPref]** Wataoka, K. et al. (2024). *Self-Preference Bias in LLM-as-a-Judge.* arXiv:2410.21819. **verified.** +- **[NoFreeLabels]** Krumdick, M. et al. (2025). *No Free Labels: Limitations of LLM-as-a-Judge Without Human Grounding.* arXiv:2503.05061. **verified.** ## Statistical honesty (supports Wilson CI / priced confirmation) -- **[NoCLT]** (2025). *Position: Don't Use the CLT in LLM Evals With Fewer Than a Few Hundred Samples.* arXiv:2503.01747. unverified. -- **[StatGuarantees]** Miller et al. (2026). *Efficient Evaluation of LLM Performance with Statistical Guarantees.* arXiv:2601.20251. unverified. +- **[NoCLT]** Bowyer, S. et al. (2025). *Position: Don't Use the CLT in LLM Evals With Fewer Than a Few Hundred Datapoints.* arXiv:2503.01747 (ICML 2025). **verified.** +- **[StatGuarantees]** Wu, S. et al. (2026). *Efficient Evaluation of LLM Performance with Statistical Guarantees.* arXiv:2601.20251. **verified.** ## Oracle problem / metamorphic testing (foundations) -- **[Barr2015]** Barr, Harman, McMinn, Shahbaz & Yoo (2015). *The Oracle Problem in Software Testing: A Survey.* IEEE TSE. (peer-reviewed; verify DOI.) -- **[Chen2018]** Chen et al. (2018). *Metamorphic Testing: A Review of Challenges and Opportunities.* ACM CSUR. (peer-reviewed; verify DOI.) +- **[Barr2015]** Barr, E.T., Harman, M., McMinn, P., Shahbaz, M. & Yoo, S. (2015). *The Oracle Problem in Software Testing: A Survey.* IEEE TSE 41(5), 507–525. DOI 10.1109/TSE.2014.2372785. **verified.** +- **[Chen2018]** Chen, T.Y. et al. (2018). *Metamorphic Testing: A Review of Challenges and Opportunities.* ACM Computing Surveys 51(1), Art. 4. DOI 10.1145/3143561. **verified.** ## Adjacent OSS (landscape, not citations) Closest philosophy (LLM-based, no sandbox/stats): `penacristian/coding-agent-reliability`. Same-name static scanner: `vinicq/falsegreen`. Cross-domain twin (trading): `bettyguo/agent-backtest-lab`. Ledger analog: `sipyourdrink-ltd/bernstein` (HMAC audit chain). Composable oracles: Stryker/PITest (mutation), Hypothesis (property), SWE-bench (harness), cordum / e2b (policy gate / sandbox).