Skip to content

feat(evals): link Datadog experiments to runtime traces - #3302

Draft
mehulsonowal wants to merge 2 commits into
vercel:mainfrom
mehulsonowal:feat/eve-runtime-trace-links
Draft

mehulsonowal wants to merge 2 commits into
vercel:mainfrom
mehulsonowal:feat/eve-runtime-trace-links

Conversation

@mehulsonowal

@mehulsonowal mehulsonowal commented Sep 11, 2026

Copy link
Copy Markdown
Contributor

Summary

Eve evals can already observe the runtime trace contexts for every participating session, but the Datadog reporter drops those references when it submits Experiment rows. This change records a content-free, namespaced eveRuntimeTraceLinks array in each row's metadata so Datadog can resolve the canonical runtime traces at query time without changing trace IDs or parentage. Repeated contexts are deduplicated and a primary-session context wins when the same trace/span pair appears more than once.

This is the producer PR for the cross-repository query-time trace-expansion work. It does not pre-create Experiment spans, copy prompts or outputs into link metadata, or change runtime instrumentation.

Related draft PRs:

Validation

  • pnpm --filter eve build:compiled
  • pnpm --filter eve exec vitest run --config vitest.unit.config.ts src/evals/runner/reporters/datadog.test.ts --reporter=verbose — 14 tests passed
  • pnpm exec oxlint packages/eve/src/evals/runner/reporters/datadog.ts packages/eve/src/evals/runner/reporters/datadog.test.ts — no warnings or errors
  • pnpm --filter eve typecheck
  • GitHub core CI: lint, typecheck, unit, integration (Ubuntu/Windows), and scenario checks pass. test-tui and secret-backed E2E/Vercel jobs fail on this fork PR because GitHub withholds AI_GATEWAY_API_KEY and Vercel credentials; the logs report MODEL_CALL_FAILED: AI Gateway received no credentials before exercising this change.

Checklist

  • This change was requested or approved by a maintainer
  • I ran the relevant checks from CONTRIBUTING.md
  • I added tests and documentation where relevant
  • I added a changeset if this touches the published eve package
  • DCO sign-off passes for every commit (git commit --signoff)

Signed-off-by: mehul.sonowal <mehul.sonowal@datadoghq.com>
Signed-off-by: mehul.sonowal <mehul.sonowal@datadoghq.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant