feat: OpenTelemetry (OTLP) exporter following the GenAI semantic conventions

[brainsparker]

What this does

Adds an otel export format that writes each evaluation run as an OpenTelemetry trace (OTLP/JSON), following the OTel GenAI semantic conventions. Eval scores can now land in the same observability backend as production traces: Grafana Tempo, Jaeger, Datadog, Honeycomb, AWS, or any OTLP-compatible collector.

How to use it

output:
  formats: [html, json, otel]

# Writes traces.otlp.json next to the other reports
promptlens run config.yaml

# Also push to a local collector
promptlens run config.yaml --otel-endpoint http://localhost:4318

# Or via the standard env var
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
promptlens run config.yaml

# Re-export a stored run
promptlens export <run_id> --format otel

Mapping

• The run is a root span; each evaluated test case is a child chat <model> CLIENT span with gen_ai.* attributes (operation name, provider, request/response model, input/output tokens, finish reasons).
• Each judge verdict becomes a gen_ai.evaluation.result span event with gen_ai.evaluation.name, gen_ai.evaluation.score.value, a pass/fail gen_ai.evaluation.score.label derived from the --fail-under threshold, and gen_ai.evaluation.explanation. Per-criteria sub-scores and tool usage scores are additional events.
• Provider errors set span status ERROR plus error.type.
• PromptLens fields with no upstream convention (run id, test case id, cost) are namespaced promptlens.*.

Why now (market rationale)

• AWS launched AgentCore Evaluations, a framework-agnostic eval layer that uses OpenTelemetry as the contract: agents and evals interoperate through telemetry, not framework APIs. Coverage dated August 27, 2026: https://aws.amazon.com/blogs/machine-learning/evaluate-any-agent-framework-with-amazon-bedrock-agentcore-evaluations/ and https://dev.to/mech_app_ai/agentcore-evaluations-how-aws-built-a-framework-agnostic-eval-layer-using-opentelemetry-as-the-4lc1
• The GenAI semantic conventions moved to a dedicated repo in June 2026 and now define a first-class gen_ai.evaluation.result event, which is exactly the shape of PromptLens output: https://john-hodge.com/blog/opentelemetry-genai-semantic-conventions/ and https://github.com/open-telemetry/semantic-conventions
• The 2026 consensus across observability vendors is that evaluation is moving into traces rather than standalone reports (TrueFoundry, Aug 22, 2026: https://www.truefoundry.com/blog/opentelemetry-genai-semantic-conventions).

PromptLens already exports HTML, JSON, CSV, Markdown, and JUnit. This closes the observability gap with zero new dependencies, which keeps the lightweight, local-first positioning intact.

Runner-up candidates considered today: a nested AGENTS.md / Copilot instructions exporter for you.md (lower value since Copilot's coding agent already reads AGENTS.md), and further A2A interop work in youagent (an outbound interop PR is already in flight, so stacking more felt premature).

Implementation notes

• New promptlens/exporters/otel_exporter.py: emits the OTLP JSON protobuf encoding directly (proto3 JSON mapping, int64 as strings, hex trace/span ids, nanosecond timestamps). No OTel SDK dependency.
• Optional push to <endpoint>/v1/traces via stdlib urllib. A push failure logs a warning and never fails the run, so a down collector cannot break CI. The file on disk is the source of truth.
• Wired into promptlens run (new --otel-endpoint flag), promptlens export --format otel, config output.formats, and the exporter registry.
• Small adjacent fix: the config formats validator only allowed html/json/csv/md, so junit (already supported by the run command) was rejected in YAML configs. The validator now accepts junit and otel, with a regression test for each.
• Note: the GenAI conventions are marked Development upstream; the docstring and README flag that attribute names track the spec as of August 2026.

Test status

• 23 new tests in tests/test_otel_exporter.py covering document shape, span parenting, gen_ai attribute mapping, evaluation events (overall, criteria, tool scores, pass/fail labels), error status, None-attribute dropping, endpoint push (path handling, env var, failure tolerance), and config validation.
• Full suite: 106 passed locally on Python 3.9.
• New files are black and ruff clean. Verified end to end with promptlens export --format otel on a synthetic run.

🤖 Generated with Claude Code