feat: deterministic assertions (is_json, json_schema, contains, regex) that skip the LLM judge on failure

[brainsparker]

What this does

Adds deterministic assertions to golden-set test cases. Each case can now declare an assert block of zero-token checks that run locally before the LLM judge:

Type	Passes when
is_json	Response parses as JSON
json_schema	Response parses and validates against a JSON Schema
contains / not_contains	Substring present / absent (case-sensitive)
regex	re.search finds a match
starts_with	Response starts with the prefix (leading whitespace ignored)

test_cases:
  - id: "extract-001"
    query: "Extract name and age as JSON from: 'Jane Doe, 34'"
    expected_behavior: "Return valid JSON with correct values"
    assert:
      - type: is_json
      - type: json_schema
        value:
          type: object
          required: ["name", "age"]

Behavior:

• When any assertion fails, the case is marked failed and the LLM judge call is skipped entirely, so judge tokens are only spent on responses that pass the cheap checks first.
• All assertions in a case are always evaluated, so one run reports every failure at once.
• JUnit XML reports failed assertions as <failure type="AssertionFailed">, so CI fails the build.
• The --fail-under gate now fails on any assertion failure regardless of judge scores, and an assertion-only golden set (no judge scores at all) passes the gate when every assertion passes.
• Cases without an assert block behave exactly as before. No breaking changes.

New module: promptlens/judges/assertions.py. New dependency: jsonschema (pure Python). Example golden set: examples/golden_sets/structured_output.yaml.

Why now (market rationale)

• Structured outputs went GA across the major providers in early 2026 (Anthropic on Feb 4, 2026, no beta header; OpenAI strict json_schema is the production default). With grammar-constrained decoding, "is it valid JSON" is guaranteed at the token level, and the eval question shifts to "are the values right", which needs schema and value assertions, not an LLM judge. Sources: https://janeer.com/guides/openai-vs-anthropic-structured-outputs/ , https://devtoollab.com/blog/llm-structured-outputs-guide-2026
• Deterministic assertions are the most-cited feature when users compare lightweight eval tools; promptfoo's is-json plus JSON Schema assertions are its headline CI feature, and its Jan 2026 release added per-test structured output config. Sources: https://www.promptfoo.dev/docs/guides/evaluate-json/ , https://www.promptfoo.dev/docs/releases/
• Judge cost is a documented complaint: burning judge tokens on trivially checkable outputs is the exact pain the llm-evalgate project was built around ("deterministic gates are the cheap CI-friendly backbone; reach for a judge only where semantic nuance demands it"). Source: https://github.com/LesterALeong/llm-evalgate
• PromptLens routes every check through the LLM judge today, so this closes the most visible gap against comparable tools while staying true to the lightweight, local-first positioning: stdlib json and re, plus the pure-Python jsonschema package.

Runner-up opportunities considered today and passed on: A2A message/stream SSE support in youagent (strong, likely a future pick) and a GitHub Copilot export target in you.md.

Implementation notes and test status

• Assertion model validates types and value shapes at load time, so a bad golden set fails fast with a clear message.
• Runner evaluates assertions after generation; on failure it records results and skips judge.evaluate.
• EvaluationResult.assertion_results serializes into JSON/CSV/HTML exports automatically; JUnit adds an explicit failure element and an assertions: N/M passed line in system-out.
• Tests: 20 new tests in tests/test_assertions.py (model validation, every check type, YAML assert key parsing, runner judge-skip integration with mocks, JUnit mapping, gate behavior). Full suite: 113 passed locally on Python 3.9.
• Lint: new files formatted with black (100 cols) and clean under ruff; pre-existing lint findings in untouched code were left alone.