Skip to content

test: add representative coding-agent evidence fixture - #258

Open
harshitethic wants to merge 5 commits into
Siddhant-K-code:mainfrom
harshitethic:test/representative-evidence-fixture
Open

harshitethic wants to merge 5 commits into
Siddhant-K-code:mainfrom
harshitethic:test/representative-evidence-fixture

Conversation

@harshitethic

Copy link
Copy Markdown

Summary

Adds a versioned, synthetic representative session fixture as a reusable regression input for #250.

Scenario covered

The fixture contains:

  • explicit session start/end
  • a redacted user prompt
  • file read and write events
  • a shell test command that fails
  • an error linked to the failed call
  • a retry linked to the first attempt
  • a successful recovery result
  • an agent decision
  • a human correction annotation
  • one deliberately orphaned tool result representing a known capture gap
  • a final assistant response

Artifacts

  • tests/fixtures/representative_session/events.ndjson — canonical TraceEvent input
  • tests/fixtures/representative_session/annotations.jsonl — human review overlay
  • tests/fixtures/representative_session/expected.json — deterministic IDs/count/gap expectations
  • tests/test_representative_fixture.py — schema + scenario regression coverage
  • docs/representative-fixture.md — trust boundary and refresh procedure

Why the gap is intentional

The fixture includes an orphaned tool result on purpose. Review UI/evidence-health work must surface missing relationships instead of inventing a parent and presenting a falsely complete timeline.

Scope

This is a focused foundation for #250 rather than a claim to close every consumer listed there. Follow-up PRs can wire the same fixture into evidence health, share/export, OTLP mapping, and the local review UI without each subsystem inventing different sample data.

Validation

The test parses the fixture through the repository's own TraceEvent.from_json() and Annotation.from_json() models and asserts the required failure/retry/recovery, redaction, annotation, relationship, and intentional-gap invariants.

I could not execute the suite locally because the connected development machine is offline; upstream CI is the verification gate for this branch.

Refs #250.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant