Tri9Trace

The CT-200 manuals were authored for this build

The provided manual files (data/ct200_manual.md, data/ct200_manual_v2.md) were not in the repo or anywhere on disk when the build started — only the PRD / TRD / execution prompts were. I searched Downloads, Documents, OneDrive, and Desktop. Rather than stop, I authored a realistic regulatory-style manual and a v2 whose diff deliberately exercises every matcher branch and every staleness category, including the dangerous "safety threshold rewritten, heading still intact" case. The parser-irregularity tests therefore target real irregularities present in the actual file, and the v2 diff is the actual diff the system runs on. The honest downside: I didn't get to test against a third-party-supplied manual's specific quirks, so the validation pass also runs a real open-source standards doc (the OpenAPI Specification README) through the parser. Stated openly in approach.md.

What this is

Tri9Trace takes a regulatory-style markdown manual, turns it into a versioned, browsable node tree, lets a user pick sections, and generates QA test-case drafts with an LLM — then, the actual point: it detects when a generated test case still looks valid but no longer matches a requirement that changed underneath it. A test case drafted against "alarm threshold 160 bpm" doesn't break loudly when the manual is revised to "140 bpm"; the heading is the same, the link is intact, and a hash diff only says text-edited. This system flags it intent-stale so a reviewer sees it before a patient does.

Why this is harder than it looks

• Node identity across versions with no stable IDs. Markdown headings have no GUIDs. Re-ingesting a manual means deciding, per node, whether "the same section" still exists — by content hash, then heading+path, then similarity, then split/merge heuristics, then an honest ambiguous when nothing clears a threshold. Get this wrong and every downstream staleness signal is wrong.
• Text-edited is not intent-stale. A typo fix and a changed safety threshold both produce text-edited at the matcher layer. The difference is the whole game, and pure hash-diff can't see it. One extra cheap LLM call (old text, new text, the test case) judges whether the case still holds — that's what separates a cosmetic text-stale from a dangerous intent-stale.
• Recall-biased on purpose. A false "stale" costs a reviewer a glance; a missed stale flag costs patient safety in the real-world analog. Thresholds and the intent judge are tuned to over-flag borderline cases, and the known false-positive cost is stated explicitly, not hidden.
• Never silently fabricate. LLM output that fails Pydantic validation gets one retry with the error fed back; a second failure is stored, returned as 422, and marked failed — never papered over into a fake success.

Tech stack

• API: FastAPI + Pydantic v2
• Relational store: SQLAlchemy 2.x + SQLite (document tree, versions, nodes, selections)
• Document store: MongoDB Atlas via pymongo (LLM generations, raw prompt/response attempts, validation failures, classification cache)
• LLM: Groq, model llama-3.1-8b-instant, behind a swappable provider interface
• Parser: markdown-it-py token stream (not HTML re-parse)
• Tests: pytest, 38 tests, offline (fake LLM + in-memory stores via dependency injection)
• Env: Python 3.12, uv; no Docker, no auth, no frontend

Quickstart

uv sync
cp .env.example .env        # then fill in GROQ_API_KEY and MONGODB_URI
uv run pytest               # 38 tests, offline (no Groq/Mongo needed)
uv run uvicorn app.main:app --reload    # http://127.0.0.1:8000/docs

There is no ingest HTTP endpoint by design — ingestion is a library call (the API is for browsing an already-ingested corpus). Populate the DB:

uv run python -c "from app.database import init_db, SessionLocal; from app.ingest import ingest_markdown; from pathlib import Path; init_db(); db=SessionLocal(); ingest_markdown(db,'ct200_manual',Path('data/ct200_manual.md').read_text(encoding='utf-8')); ingest_markdown(db,'ct200_manual',Path('data/ct200_manual_v2.md').read_text(encoding='utf-8')); db.close()"

The v1 -> v2 re-ingestion + staleness flow, end to end:

uv run pytest tests/test_e2e.py -q        # deterministic, offline, through the real API
uv run python scripts/e2e_demo.py         # live: real Groq + real MongoDB, fresh -> intent-stale

Other runnable demos: scripts/generate_demo.py (real generation + the failure/retry -> 422 path), scripts/staleness_demo.py (real intent-stale verdict with per-test-case reasoning), scripts/compliance_demo.py (bonus compliance report). Browse curl examples: notes/section4_curl.md.

What is actually novel here

• Layered node identity, not a single strategy. Exact hash -> heading +path -> similarity -> split/merge -> new/deleted -> ambiguous, mirroring how Git infers rename identity. Split detection runs per previous node (a matched node whose content now spans >1 new node), which fixed a real false-split storm where any short new node overlapping a matched node on common words got called a split.
• Change taxonomy, not binary changed/unchanged. unchanged | text-edited | moved | renamed | split | merged | new | ambiguous, plus deleted derived at query time (a node with a v1 snapshot but none in v2 — not stored on a row that doesn't exist).
• Three-state staleness. fresh | text-stale | structure-stale | intent-stale, computed at retrieval time and inlined in the response, not a separate call.
• LLM-as-judge for intent-stale. One extra cheap call per test case instead of an embeddings pipeline; its accuracy is only as good as that call, which is named openly as the soft spot.
• Full generation provenance. Every stored generation records node+version, source hash, prompt template version, schema version, model id, and both raw attempts — so "why was this flagged" and "reproduce exactly what the LLM saw" are both answerable later.
• Bonus: compliance-mapping report. GET /documents/{id}/compliance-report classifies each test case into one of 8 regulatory categories (IEC 62304 lifecycle/maintenance, ISO 14971, FDA 21 CFR 820.30, IEC 62366, IEC 60601-1, 60601-1-8, SOUP) with a cached, retry-then-fallback classifier, then aggregates coverage %, stale coverage %, review backlog, and per-category counts — reusing the staleness engine, not duplicating it.

Where this could go (not built — just the shape of it)

Nothing below exists. This is the direction, sketched honestly so it is clear what is real and what is not.

• Agentic staleness, not queried staleness. Today staleness is something you ask for. The natural next shape is a pipeline that watches a document repo, auto-triggers re-ingestion on commit, and opens a review ticket (Jira / Azure DevOps) only for intent-stale items above a confidence threshold — turning detection from "something you query" into "something that finds you."
• Embeddings instead of difflib. The matcher's difflib.SequenceMatcher character ratio misreads heavy paraphrase (a rewritten section with the same meaning scores low). Token-overlap or embeddings would catch that without lowering the confident-match threshold. Listed in approach.md as a known gap.
• Ingest as an HTTP endpoint. Ingestion is currently a library call with no auth. An idempotent POST /documents/{id}/ingest with auth is the production-shaped version — also already noted in approach.md.

See PRD.md / TRD.md for scope and approach.md for the full decision log and validation pass.