Add progressive_debate method: adversarial adjudication replaces consolidation

[r-uben]

Summary

• Adds progressive_debate review method that replaces monolithic consolidation with per-comment adversarial adjudication
• Discovery phase is identical to the progressive method (sequential passage-by-passage with running summary)
• For each raw comment, a Challenger (small model) argues it's not a real issue, then a Verdict (large model) decides keep/drop
• Surviving comments are merged to remove duplicates

Motivation

The current consolidation step in review_progressive loses 43% of real findings on the Refine benchmark (44% → 25% LLM recall). It operates on raw JSON without paper context, so it cannot distinguish true positives from noise. See #35 and #36.

Benchmark Results (Seeded-Perturbation)

Metric	Zero-Shot	Progressive	Debate
Recall	92% (11/12)	83% (10/12)	92% (11/12)
Total comments	10	32	34
False positives	5 (50%)	24 (75%)	28 (82%)

Key result: param_sigma and param_labor_share that consolidation pruned are correctly kept by adversarial adjudication. The debate dropped 16/54 comments (30%) through targeted challenge-verdict pairs, vs 72% by monolithic consolidation.

Usage

openaireview review paper.pdf --method debate
openaireview review paper.pdf --method debate_full  # pre-debate raw comments
openaireview review paper.pdf --method debate --small-model anthropic/claude-haiku-4-5

Files Changed

• src/reviewer/method_debate.py — new method with challenge/verdict/merge prompts
• src/reviewer/cli.py — wire up debate and debate_full methods + --small-model flag
• benchmarks/seed_errors.py — add debate support to benchmark runner
• benchmarks/REPORT.md — document seeded-perturbation benchmark with all three methods

Test plan

• Module imports correctly (from reviewer.method_debate import review_progressive_debate)
• Benchmark on seeded paper: 92% recall (11/12), matching zero-shot
• Run on Refine benchmark papers to compare against progressive consolidation
• Test with different provider APIs (OpenAI, Anthropic native, Gemini)

[r-uben]

Broader Vision: Multi-Agent Debate for Academic Review

This PR is a minimal implementation — one challenger, one judge, sequential processing. But the core idea (adversarial pressure improves calibration) has strong support in the literature and could be pushed much further.

What the literature says

Du et al. (2023), "Improving Factuality and Reasoning in Language Models through Multiagent Debate" showed that multiple LLM instances proposing, critiquing, and revising answers over several rounds significantly improves factual accuracy and reasoning. Their "society of minds" approach reduces hallucinations — exactly our false positive problem.

Chan et al. (2023), "ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate" applied multi-agent debate specifically to evaluation tasks. Key finding: diverse role prompts (different personas) are essential — using the same role description degrades performance. This suggests our Reviewer/Challenger/Judge should have genuinely different perspectives, not just different instructions.

Zhang, Kraska & Khattab (2025), "Recursive Language Models" tackle the context length problem through recursive decomposition — processing documents in chunks with structured state passing between levels. Their RLM architecture maintains quality at 1M+ tokens where flat models degrade. Our progressive running summary is a crude version of this; a proper recursive architecture could handle much longer papers.

Tillman (2025), "Literature Review of Multi-Agent Debate for Problem Solving" synthesizes the field and identifies key open questions: how to terminate discussions, how systems scale with agents and rounds, and how communication structure affects performance. Our fixed 2-round protocol (challenge → verdict) is the simplest possible; the literature suggests dynamic termination (debate until convergence) and scaling to 3-5 agents improves results.

Concrete improvements

1. Multi-round debate: Instead of challenge → verdict, allow rebuttal → counter-challenge → final verdict. The current miss (claim_persistent) might survive if the reviewer could defend its finding with additional evidence.

2. Diverse agent personas: A Methodologist agent that focuses on statistical validity, a Domain Expert that checks against field conventions, and a Notation Checker that verifies mathematical consistency. ChatEval shows this diversity matters.

3. Recursive context: The verdict currently gets a local window. A recursive architecture (RLM-style) could give it access to a compressed representation of the entire paper, fixing the context window limitation that caused the claim_persistent miss.

4. Dynamic termination: Instead of fixed rounds, debate until the challenger can't find new counterarguments or the judge reaches high confidence. This naturally spends more compute on ambiguous cases and less on obvious errors.

5. Confidence-calibrated output: The verdict already returns a confidence score (high/medium/low). This could drive which comments get additional debate rounds and how they're presented to the user.

The 92% recall on seeded errors is encouraging, but the real test is the Refine benchmark where progressive consolidation drops from 44% to 25%. If debate can close that gap, it would be a significant step toward LLMs that catch what human reviewers catch.

[chenhaot]

@johnjosephhorton shared a similar idea over email

[reycn]

looks fantastic

[dangng2004]

Thanks for this. The adversarial adjudication idea is well motivated and the writeup is clear. The consolidation recall loss is real and worth fixing, so I want to get this in. Three structural asks first, then specific bugs.

Structural

1. Rebase onto current main. The branch forked in March and main has moved a lot since. Two changes break the new code directly (details in finding 5 below), so the rebase needs code changes and a re-test, and a plain merge will not be enough.

2. Drop benchmarks/seed_errors.py and the REPORT.md section. Main now has benchmarks/perturbation/, which does the same seeded-error evaluation with better scoring (fuzzy quote matching plus an LLM judge on the explanation, instead of keyword co-occurrence). The keyword scoring here credits errors on generic word overlap (for example, "calibration" plus "from" marks the discount-factor error as caught) and counts duplicate true positives as false positives, so the recall and false-positive numbers in the table are not reliable. The default --input file is also not committed, so the script fails from a fresh clone. Slimming the PR to method_debate.py plus the CLI wiring also keeps it to one concern.

3. Re-run the eval through benchmarks/perturbation/ after the benchmark cleanup lands (#105). The benchmark is not runnable from a fresh clone until that merges. Once it does, adding debate to the methods list in the config should be enough. Happy to run the comparison on our side at that point if API budget is a blocker, a small subset (a few papers, one or two models) is enough to validate the claim for this PR.

Bugs in method_debate.py

1. Default challenger model crashes every non-Anthropic run. small_model = f"{provider}/claude-haiku-4-5" grafts a Claude model name onto whatever prefix the large model has, so --model openai/gpt-4o produces openai/claude-haiku-4-5, which no provider serves. The first challenge call 404s and the run crashes after the entire discovery phase has already been paid for. The default needs a per-provider table (or require --small-model for non-Anthropic models).

2. Any --reasoning-effort setting crashes the debate phase. The verdict call uses max_tokens=512 and the challenge call 1024, but client.py floors the thinking budget at 1024, so budget_tokens >= max_tokens and Anthropic (also OpenRouter and Gemini) rejects every call with a 400. Raise the caps or skip reasoning for debate calls.

3. One hard API failure loses the whole run. The debate loop makes roughly 2N serial LLM calls with no try/except, and results are only written to disk after the method returns. A sustained 429 at call 60 of 108 discards all discovery output. Suggest keeping the comment when its debate fails, and consider a small thread pool, since the serial loop also adds 10 to 30 minutes of wall clock per paper.

4. Reported cost is overstated. Challenger tokens go into the same counters as the large model's, and compute_cost prices everything at the large model's rate. evaluate.py already supports blended pricing for "model1+model2" strings, or track the small-model tokens separately.

5. The copied discovery loop is duplicated and already stale. Phase 1 is a near-verbatim copy of review_progressive's discovery loop (about 90 lines), plus merge_duplicates clones consolidate_comments with only the prompt differing. The copy has already diverged from main in ways that break it: main's deep-check prompt now requires an ocr_caveat format field (the copied .format(...) call will raise KeyError on the first passage after rebase), and main replaced the per-passage comment anchoring with window-wide locate_comments_in_window. Please extract the discovery loop into a shared helper in method_progressive.py and have both methods call it. That fixes the rebase breakage once and keeps the two methods comparable in benchmarks. Same for consolidate_comments: a prompt parameter covers the merge step.

Happy to help with the rebase if useful. The method itself is a good addition and I would like to see the perturbation numbers for it.