TraceLens 🔍

An observability layer for multi-step AI pipelines. TraceLens traces every intermediate step of a document-processing pipeline, answers "where did this go wrong?" when the final output is bad, and feeds every confirmed failure back into a growing evaluation dataset — a mini LangSmith/Braintrust you can read in an afternoon.

I built an observability system that reduces mean time to root cause for AI pipeline failures from hours of manual debugging to seconds of automated diagnosis.

Why observability matters for AI systems

A traditional service fails loudly: an exception, a stack trace, a 500. A multi-step LLM pipeline fails silently — step 2 hallucinates an entity, steps 3 and 4 happily build on it, and the only visible symptom is a subtly wrong final summary. Without per-step traces you cannot distinguish "the summarizer is bad" from "the summarizer was fed garbage." TraceLens makes each step's input, output, prompt, raw response, confidence, and latency inspectable, so a bad output can be walked backward to the exact step that broke — and every diagnosed failure becomes a regression test, so the same bug can never come back unnoticed.

Quick start

pip install -r requirements.txt
python run_demo.py          # process 50 docs, auto-diagnose ~10 failures
streamlit run app.py        # open the trace explorer

Works fully offline out of the box via a deterministic mock LLM with realistic injected failure modes. Set OPENAI_API_KEY (and optionally TRACELENS_MODEL, default gpt-4o-mini) to run against the real OpenAI API.

Docker

docker build -t tracelens .
docker run -p 8501:8501 tracelens

What's inside

Piece	File	What it does
Pipeline	tracelens/pipeline.py	4 isolated, Pydantic-typed steps: intake → extraction → classification → summarization
Tracing	tracelens/tracing.py	@traced_step decorator (one line per step), spans with prompt/response/tokens/latency/confidence, JSON trace files + SQLite index
Analyzer	tracelens/analyzer.py	Backward root-cause analysis: LLM-as-judge scores each step's output given its input; earliest quality drop = root cause; categorized (hallucination, misclassification, propagation, prompt failure, context loss) with an evidence chain
Explorer	app.py	Streamlit UI: color-coded pipeline nodes, span drill-down, input/output diff, flag button, confirm/override diagnosis
Feedback loop	tracelens/evals.py	Confirmed flags become eval cases (JSONL); regression runner re-tests them; failure analytics dashboard
Failure injection	tracelens/documents.py	50 sample documents, 10 of which deliberately break different steps

Demo flow (what to record)

1. python run_demo.py, then open the explorer.
2. Select contract_nightfall_1.txt — final summary looks plausible but mentions "John Smith," who isn't in the document.
3. The pipeline view shows the extraction node in red; the diagnosis reads: extraction hallucination — "John Smith" does not appear in the input, propagated to summarization.
4. Click Confirm diagnosis — a new eval case appears in the Eval Dataset tab.
5. Re-run regression; the trend chart tracks known issues resolved over time.
6. The Analytics tab shows failure rate, failures by category and by step.

See docs/architecture.md for the full design.