feat: deterministic trajectory assertions for agent tool-call evaluation

[brainsparker]

What this adds

Deterministic trajectory assertions for agent tool-call evaluation. A new trajectory block on any test case checks the sequence of tool calls a model makes, with zero LLM cost:

trajectory:
  require:
    - verify_identity
    - name: issue_refund
      args:
        order_id: "ORD-123"
  forbid:
    - delete_account
  order:
    - [verify_identity, issue_refund]
  min_calls: 1
  max_calls: 4
  no_repeat_calls: true

Five check types: require (tool must be called, optionally with an argument subset), forbid (tool must never be called), order (tools must appear in relative order, subsequence match so other calls may interleave), min_calls / max_calls (step budgets, max_calls: 0 asserts no tools were used), and no_repeat_calls (same tool with identical arguments twice fails, catching agent loops).

Results are pure functions of the recorded calls: stable across runs, free to compute, and they run alongside or entirely without LLM-as-judge scoring.

How to use:

• Add a trajectory block to a test case (see examples/golden_sets/agent_trajectory.yaml)
• Results appear in the run summary, JSON export, and JUnit XML (failed checks map to TrajectoryAssertionFailed failures)
• Gate CI with the new --fail-on-trajectory flag: exits 2 on any trajectory failure, costs nothing, never flakes

Why this, why now

Three signals from this week's market scan:

1. Judge cost and flakiness are the loudest 2026 eval complaints. LangSmith led with an 82 percent judge-cost reduction headline (Tuned Evaluators, Aug 18: https://www.langchain.com/blog/introducing-langsmith-tuned-evaluators-starting-with-perceived-error), and practitioners document DeepEval scores swinging 0.85 to 0.72 on identical data at temperature 0 (Jul 16: https://communities.stackinsight.net/community/llm-eval-frameworks/why-is-deepeval-producing-inconsistent-scores-on-same-data/). Deterministic checks sidestep both problems for the class of assertions that never needed a judge.

2. Agent trajectory evaluation is the consensus unsolved gap. The dominant practitioner pattern is hand-stitched deterministic unit tests for tool calls plus rubrics for overall quality, with no lightweight declarative format for the first half ("agents fail by multiplication", Aug 5: https://www.kunalganglani.com/blog/agent-evaluation-harness-replay). This PR gives PromptLens that declarative format.

3. CI judge spend is a practical blocker: roughly 3 to 8 dollars per PR for a 20-test suite run 5 times (Apr 9: https://blog.appxlab.io/2026/04/09/ai-agent-testing-cicd-pipeline/). A zero-cost gate lets teams run trajectory checks on every PR and reserve the judge for merges.

Runner-up candidates considered today: OTel gen_ai.evaluation.result span emission for PromptLens (strong timing, promptfoo aligned Aug 26, but the convention is still marked Development and it adds a dependency), and a memory-import bridge for you.md (the repo already ships an MCP server, native exports, and a ChatGPT connector, so the researched gap was thinner than expected).

Implementation notes

• New: promptlens/models/trajectory.py (TrajectorySpec, TrajectoryEvaluation, validation that rejects empty specs and malformed order constraints), promptlens/evaluators/trajectory.py (pure evaluator, no network, no LLM)
• Wired into: TestCase (new optional trajectory field), EvaluationResult (new optional trajectory_evaluation), the runner (checks run before judging so results exist even when the judge fails), the CLI (--fail-on-trajectory, summary line), and the JUnit exporter
• JUnit mapping: trajectory failure beats a passing judge score; a case with passing trajectory checks but no judge score now counts as a pass instead of skipped, since deterministic checks did run (documented in the exporter docstring)
• Fully backward compatible: no behavior change for golden sets without a trajectory block, no new dependencies
• New example golden set: examples/golden_sets/agent_trajectory.yaml

Test status

• 128 tests pass locally (Python 3.9), including 41 new tests across test_trajectory_evaluator.py, test_trajectory_spec_validation.py, and test_junit_trajectory.py
• promptlens validate examples/golden_sets/agent_trajectory.yaml passes
• ruff clean on all new files; pre-existing lint warnings elsewhere left untouched

🤖 Generated with Claude Code