feat: judge spend guard with deterministic checks, result caching, and a per-run judge budget

[brainsparker]

What this does

Adds a judge spend guard: three layers that cut LLM-as-judge cost without giving up judge quality. All local, no new dependencies.

1. Deterministic checks (per test case, opt in). Cheap assertions (contains, not_contains, equals, regex, not_regex, json_valid, min_length, max_length) run against the response before the judge is called. If any check fails, the case scores 1 with the failure details as the explanation and no judge tokens are spent. Check results are recorded on every evaluation result.
2. Judge result cache (on by default, --no-judge-cache to disable). Verdicts are cached in <output_dir>/.promptlens/judge_cache.json, keyed by a SHA-256 of the judge model and settings, the test case, and the exact response text (including tool calls). Re-running over unchanged responses reuses the verdict at zero cost. Failed judge calls are never cached. Cached verdicts are marked cached: true in exports.
3. Judge budget (judge.budget_usd in config or --judge-budget 5.00 on the CLI). Once the budget is exhausted, remaining judge calls are skipped, each affected result records judge_skipped_reason, and the summary reports the skip count. Model-under-test calls are never blocked. Documented caveat: with concurrency the budget can overshoot by up to parallel_requests - 1 in-flight calls; it is a circuit breaker, not exact accounting.

Judge spend is now also tracked as a first-class number: JudgeScore.cost_usd, run-level judge_cost_usd in JSON exports, and a Judge Cost line plus a Judge Spend Guard section in the run summary. Previously judge calls were entirely absent from cost reporting.

Usage

test_cases:
  - id: "cs-001"
    query: "How do I reset my password?"
    expected_behavior: "Provide clear step-by-step instructions"
    checks:
      - type: contains
        value: "password"
        case_sensitive: false
      - type: max_length
        value: 4000

promptlens run config.yaml --judge-budget 5.00
promptlens run config.yaml --no-judge-cache

Why this feature, why now

Judge cost runaway is the loudest complaint in the eval tooling space right now, and no lightweight local-first tool ships a guard for it:

• A widely shared incident post documents a $3,900 overnight CI bill from an eval suite after a dependency bot retrigger storm (270 runs x 1,200 cases x $0.012 per judge call). The author's fix was exactly this pattern: deterministic pre-filters plus a result cache keyed on rubric, case, and response hash. https://jas-blogs.hashnode.dev/a-3-900-overnight-bill-from-our-llm-eval-suite-the-incident-and-the-spend-guard-i-shipped-after
• A companion post reports the judge costing 30 percent more than the system under evaluation, and argues for deterministic gates in front of every judge call. https://ethanwalkerwrites.hashnode.dev/our-ci-eval-gate-sent-us-a-token-bill-the-deterministic-one-sent-nothing
• LangSmith's answer (Tuned Evaluators, Aug 18, 2026) is a managed, Plus/Enterprise-only judge with per-evaluation pricing, which reinforces the opening for a local, free alternative. https://www.langchain.com/blog/introducing-langsmith-tuned-evaluators-starting-with-perceived-error
• Promptfoo has strong deterministic assertions but no judge result cache; Braintrust and LangSmith solve cost with cloud features. PromptLens can own the cost axis with zero cloud dependency, which is its core positioning.

Runner-up candidates considered today and passed on: an MCP 2026-07-28 conformance assertion pack (bigger surface, better as a follow-up once this deterministic-check foundation exists) and an OpenTelemetry gen_ai.evaluation.result emitter (the semconv is still Development status, and it would add a dependency).

Implementation notes

• New: promptlens/models/checks.py (CheckSpec validation plus the check engine), promptlens/judges/cache.py (JudgeCache), promptlens/judges/spend.py (JudgeSpendTracker).
• Runner._judge_with_guard orchestrates the order: checks gate, then cache, then budget, then a paid judge call whose verdict is recorded and cached.
• LLMJudge now reports cost_usd on verdicts and marks failed calls with error so they are never cached.
• All new model fields are optional with defaults, so old exported runs still parse and promptlens compare is unaffected.
• Backward compatible: no checks means no gating, cache changes nothing observable except fewer duplicate calls, and no budget means unlimited, matching current behavior.

Test status

• 144 tests pass (83 before, 61 new) via pytest tests.
• New suites: test_deterministic_checks.py (spec validation and engine outcomes for all 8 check types), test_judge_cache.py (key stability, roundtrip, corrupt file recovery, error verdicts never cached), test_judge_spend_guard.py (gating skips the judge, cache hits skip the judge, budget exhaustion skips and records reasons, gated cases consume no budget).
• Verified end to end against a local mock HTTP model: run 1 gated 1 case, judged 3, budget-skipped 1; run 2 served 3 verdicts from cache and paid for only the previously skipped case. JSON export carries judge_cost_usd, check results, cached flags, and skip reasons.
• promptlens validate passes on the updated example golden set; promptlens run --dry-run accepts the new flags.