feat: multi-judge consensus scoring with agreement signal

[brainsparker]

What this does

PromptLens can now score with a panel of 2-5 LLM judges instead of a single judge. Each judge on the panel evaluates every response independently and in parallel (asyncio.gather). The reported score is the panel median (robust to one outlier judge), and every result carries:

• individual_scores: per-judge score, explanation, model, and provider
• agreement_gap: max panel score minus min panel score
• low_confidence: true when the gap exceeds a configurable agreement_threshold

Low-confidence results get an amber "low confidence" badge in the HTML report; agreeing panels get a quiet "N-judge consensus" note. All fields flow through the JSON export automatically.

How to use it

judge:
  criteria: [accuracy, helpfulness]
  judges:
    - {provider: anthropic, model: claude-3-5-sonnet-20241022}
    - {provider: openai, model: gpt-4o}
    - {provider: google, model: gemini-1.5-pro}
  agreement_threshold: 1

Existing single-judge configs are untouched: when judges is absent, the runner builds the same LLMJudge as before (a new create_judge factory picks the implementation). Old serialized runs still validate because all new JudgeScore fields default.

Why this, why now

This was roadmap item one in the README, and the market evidence for it is fresh:

• The Concord paper (ACL 2026 GEM) measures a 3-judge panel at Cohen's kappa 0.805 against human labels, versus roughly 0.48 for single judges: https://aclanthology.org/2026.gem-main.46.pdf
• A June 2026 study of 21 judge models across 541k judgments finds agreement deflation of 33 to 41 points is universal for single judges: https://arxiv.org/html/2606.19544v1
• An August 21, 2026 Langfuse vs promptfoo comparison calls judge calibration the missing layer in every open-source eval tool: https://dev.to/devtoaaron/langfuse-vs-promptfoo-llm-testing-observability-compared-23em
• A practitioner post from May 2026 documents a team shipping regressions because a single unvalidated judge passed failures, at a cost of $4,200 in one weekend: https://medium.com/@ethan-writes-AI/promptfoo-is-a-ci-gate-not-an-eval-framework-treating-it-like-one-cost-us-4-200-6efdb36c673e

promptfoo, DeepEval, and Langfuse all default to single-judge scoring today. Shipping panel consensus with a visible disagreement signal is a real differentiator for a lightweight local tool, and mixing providers on the panel also reduces self-preference bias.

Runner-up candidates considered today: an A2A v1.0 operation-name upgrade for youagent (solid, but mechanical and that repo has an external interop PR in flight) and merging you.md's pending export targets plus an npm publish (release plumbing that needs npm credentials, not a feature).

Implementation notes

• promptlens/models/config.py: new JudgeInstanceConfig; JudgeConfig gains judges (validated to 2-5 entries) and agreement_threshold (0-4). A panel of exactly one is rejected with a pointer to the single-judge fields.
• promptlens/judges/multi_judge.py: MultiJudge composes existing LLMJudge instances, so tool-calling evaluation, retry behavior, and the per-judge failure fallback all come along for free. Tool evaluations are taken once (the automatic stage is deterministic); usage and efficiency scores are averaged.
• promptlens/judges/factory.py: create_judge picks LLMJudge or MultiJudge from config; the runner now calls this instead of constructing LLMJudge directly.
• promptlens/models/result.py: IndividualJudgeScore model plus three optional fields on JudgeScore, all defaulted for backward compatibility.
• HTML exporter and report template: low-confidence badge with a hover tooltip showing the gap and panel size.
• README: feature bullet, a Multi-Judge Consensus config section, roadmap checkbox.

Test status

101 passed (83 existing plus 18 new in tests/test_multi_judge.py). New tests cover config validation (panel size caps, threshold range), factory selection, median consensus with an outlier, half-up rounding on even panels, custom and zero thresholds, panel identity fields, judge-failure fallback behavior, and backward compatibility of the result model. Also verified end to end: a YAML config with a judge panel parses through RunConfig, and the HTML report renders the low-confidence badge. All judges are mocked; no live API keys needed.