feat: deterministic trajectory assertions for agent evals with --fail-on-assertions CI gate

[brainsparker]

What it does

Adds deterministic trajectory assertions: behavior checks over the tool calls in a response, evaluated locally with zero LLM calls. Judge scores tell you whether an answer was good. Trajectory assertions tell you whether the agent did the right thing: called the required tools, avoided forbidden ones, followed the expected workflow order, and stayed within a call budget. Because they are pure Python over the captured tool calls, they cost nothing to run, never flake on judge variance, and are safe to gate CI on.

New trajectory block on any test case:

trajectory:
  must_call:
    - name: check_availability        # optional args matching: partial (default), exact, ignore
      args: {day: "Tuesday"}          # plus min_times / max_times call-count bounds
  must_not_call: [cancel_booking]     # safety: never call this
  call_order: [check_availability, create_booking]   # subsequence, extras may interleave
  max_calls: 4                        # total tool call budget (loop guard)
  allow_other_calls: false            # whitelist mode for safety testing

Wired end to end:

• Runner evaluates assertions after generation, before (and independent of) the judge
• JUnit export reports failures as <failure type="TrajectoryAssertionFailure">; a case with passing assertions and no judge score now counts as a pass instead of skipped, since it was evaluated deterministically
• New --fail-on-assertions CI gate (exit code 2, matching the --fail-under and --fail-on-regression semantics)
• HTML report gets pass/fail badges plus failed-check details; Markdown gets a Trajectory column and failure notes; JSON includes the full structured result
• Run summary prints per-model assertion pass counts
• New example golden set: examples/golden_sets/trajectory_assertions.yaml (required calls, safety, workflow order, whitelist)

Why now

Deterministic trajectory checking became table stakes across the eval ecosystem in Q2-Q3 2026, and PromptLens had no deterministic layer at all: every tool evaluation went through the LLM judge, which costs money and can flake in CI.

• LangChain's agentevals ships trajectory match evaluators with strict, unordered, subset, and superset modes plus per-tool argument matching: https://github.com/langchain-ai/agentevals
• Braintrust launched Behavior Specs as an open standard for asserting agent behavior (July 2026): https://www.braintrust.dev/blog/behavior-specs
• Agentest documents the same primitive set (matchMode strict/unordered/contains/within, arg modes ignore/partial/exact) and makes the core argument: trajectory assertions catch logic bugs at zero LLM cost, judges catch quality issues: https://r-prem.github.io/agentest/guide/trajectory-assertions.html
• Agent-Harness (pytest-based, on PyPI) builds an entire product on trace assertions like assert_called_before, assert_no_loop, and arg schema checks: https://github.com/Suirotciv/Agent-Harness

PromptLens's positioning is lightweight and local-first. A deterministic assertion layer is the one trajectory-eval capability that fits that positioning perfectly: no server, no new dependencies, no API spend. This was also flagged as a runner-up candidate in frugal PR #87 three weeks ago; the market signal has only strengthened since.

Runner-up opportunities considered today and passed on: (1) frugal: per-tool cost attribution via OpenTelemetry span export, timely because MCP's 2026-07-28 revision deprecated protocol-level logging in favor of OTel, but frugal received the previous feature PR and rotation favored spreading coverage; (2) frugal: tools/list TTL caching per SEP-2549 to cut tool-catalog token bloat.

Implementation notes

• promptlens/models/trajectory.py: pydantic models (TrajectoryAssertions, ToolCallMatcher, TrajectoryCheck, TrajectoryResult) with validation that rejects empty assertion blocks, invalid arg-match modes, contradictory count bounds, and whitelist mode without an allowed set
• promptlens/assertions/trajectory.py: pure evaluator, no LLM, no network; call_order is a greedy subsequence scan; must_call failure messages distinguish "tool never called" from "called with non-matching arguments"
• Backward compatible: trajectory is optional, existing golden sets and stored results parse unchanged, and JUnit mapping for cases without assertions is untouched
• No new dependencies

Testing

• 35 new tests in tests/test_trajectory_assertions.py: model validation, all matcher modes (partial/exact/ignore, min/max times), must_not_call, call_order subsequence semantics, max_calls, whitelist mode, TestCase parsing, JUnit mapping (failure type, error precedence, pass-without-judge, skip semantics, system-out stats), and the CLI gate helper
• Full suite: 118 passed
• End-to-end smoke: HTML, Markdown, JSON, and CSV exporters render trajectory results correctly; promptlens validate accepts the new example golden set

Built by Daily Feature Builder.