feat: deterministic checks, zero-cost assertions that run alongside the LLM judge

[brainsparker]

What this does

Adds a checks field to golden-set test cases: deterministic assertions evaluated in plain Python against the model response. No judge tokens, no extra latency, no API calls. The LLM judge keeps grading nuanced quality; checks catch hard rule violations and budget overruns.

Ten check types ship in this PR:

Type	Passes when
contains / not_contains	response does / does not contain a string
regex / not_regex	pattern does / does not match
exact_match	trimmed response equals the value
is_valid_json	response (raw or one fenced code block) parses as JSON
max_latency_ms / max_cost_usd	response stays within budget
min_length / max_length	response length in bounds

String checks accept case_insensitive: true. Example:

test_cases:
  - id: "test-003"
    query: "Return the user profile as JSON"
    expected_behavior: "Valid JSON with a name field"
    checks:
      - type: is_valid_json
      - type: contains
        value: '"name"'
      - type: max_latency_ms
        value: 5000

Check outcomes surface in three places: the CLI run summary (per-model Checks: N/M passed), the HTML report (pass/fail pills per response, reason on hover), and the JUnit XML export, where any failing check marks the test case as a CI failure regardless of judge score. A case whose checks all pass counts as a pass even with judging disabled, so a team can gate a build purely on deterministic checks with zero judge spend.

Why this feature, why now

• The eval tooling field is actively splitting deterministic graders from LLM judges. promptfoo 0.122.1 (released 2026-08-26, two days ago) added more deterministic assertion types including word-count and trace-span-duration percentile assertions: https://newreleases.io/project/github/promptfoo/promptfoo/release/0.122.1
• multivon-eval (v0.16.1, August 2026) ships 44 evaluators and explicitly markets its deterministic tier (ExactMatch, Contains, JSONSchemaEval) as graders that run without API keys: https://github.laiyagushi.com/multivon-ai/multivon-eval
• The most common complaint about lightweight eval tools is judge cost in fast CI loops. PromptLens already has --fail-under and JUnit export, but every gate currently requires paying for judge calls on every push. Checks close that gap: run checks on every commit, save the judge for nightly or pre-release runs.
• Everything in this PR is additive. checks defaults to an empty list, existing golden sets and serialized results load unchanged, and no new dependencies are added (JSON validation uses the stdlib).

Runner-up candidates considered today and passed on: a GitHub Copilot export target for you.md (strong fit but a much smaller change), and A2A v1.0 signed agent cards for youagent (timely but riskier given RFC 8785 canonicalization precision and no existing test coverage in that module).

Implementation notes

• promptlens/models/checks.py: CheckDefinition (validated per type: string checks need strings, budget checks need non-negative numbers, regexes must compile at load time) and CheckResult.
• promptlens/checks/engine.py: pure-Python evaluate_checks(). is_valid_json also accepts a response that is entirely one fenced code block, a common LLM output shape. A check that raises unexpectedly is recorded as a failed check rather than killing the run.
• promptlens/runners/runner.py: checks run after generation and before judging, skipped when the response errored.
• promptlens/exporters/junit_exporter.py: precedence is error > failed checks > judge threshold. Failing checks emit one aggregated <failure type="DeterministicCheckFailed"> listing each failed check and reason; all check outcomes are written to system-out.
• promptlens/exporters/html_exporter.py + templates/report.html: pass/fail check pills per model response.
• EvaluationResult gains check_results, checks_passed, and failed_checks; RunResult gains get_check_stats().
• New example golden set examples/golden_sets/deterministic_checks.yaml (passes promptlens validate), plus README docs for the check table and CI behavior.
• JSON export includes check results automatically via model serialization. CSV columns were intentionally left unchanged to avoid breaking downstream consumers; can add columns in a follow-up if wanted.

Test status

• 54 new tests in tests/test_checks.py: definition validation, every check type (pass, fail, case folding, fenced JSON, None-cost handling), declaration-order preservation, result helpers, backward compatibility of old serialized results, and JUnit integration (failure element, precedence, system-out, checks-only pass semantics).
• Full suite: 139 passed, 0 failed, on Python 3.9.
• Manual verification: promptlens validate on the new example golden set, and an HTML render smoke test confirming check pills appear in the report.

Opened by Daily Feature Builder.

🤖 Generated with Claude Code