feat: upload benchmark results to Braintrust by Rodriguespn · Pull Request #101 · supabase/evals

Rodriguespn · 2026-07-21T14:16:54Z

Uploading benchmark results to Braintrust.

This is an additive dual-write agreed in Slack: eval-results.json stays exactly as it is (verified byte-identical) and remains the source of truth; Braintrust becomes a second destination for exploring and comparing results. Design doc: Braintrust as the storage layer of Supabase evals.

How our results map to Braintrust

A Braintrust experiment is a group of agent runs against a set of scenarios. We create one experiment per model on every refresh, named <experiment>@<git-sha> (e.g. claude-code-sonnet-5@fe0e167), so refreshes stay comparable over time — each new experiment is automatically diffed against the previous one for the same model.

Inside an experiment, each eval is one row. The row's input is the scenario prompt (what Braintrust uses to match the same scenario across models), the output is the agent's final report, and the scenario dimensions (stage, product, topic, interface…) are attached as metadata and tags for filtering.

Example of a trace: claude-code-sonnet-5@59fc9c7-raw-transcript

Opening a row shows the full trace of what the agent did:

a turn groups everything the agent did between user prompts (evals send a single prompt, so one turn is the norm);
inside it, each step is one model API call, carrying the model's verbatim text/thinking and that call's own token counts — you can watch the context grow step by step;
each tool call hangs under the step that requested it, with its args and result; subagent work (Claude Code's Task) nests under the tool call that spawned it.

The shape is identical for Claude Code, Codex and ai-sdk: each agent's parser knows its own transcript format, and a shared agent-agnostic model (AgentTrace in core) assembles the tree.

Scoring appears as a single passed scorer — matching how the web UI headlines pass/fail — with the full checks list inside its output instead of one scorer per check, so the experiment table keeps one clean score column. When a check used an LLM judge, the judge call is nested under the scorer with the rubric, the exact input the judge read, and its model/duration/tokens: you audit what the judge saw instead of trusting its notes.
Token metrics follow Braintrust's conventions (prompt_tokens, prompt_cached_tokens, …), taken from each agent's own accounting; Claude input tokens are normalized to include cache reads/writes so counts compare across agents. The row covers the real run wall-clock; span ordering inside is synthetic (CLI transcripts carry no per-event timestamps) and flagged stepTiming: synthetic-order-only.

Every row also carries two attachments for forensics: the raw result JSON and the agent CLI's verbatim transcript (<evalId>.transcript.jsonl).

Experiment metadata

Each experiment carries exactly the metadata fields from the design doc's table — the decomposed config identity (agent, modelProvider, modelId, reasoningEffort, skills) plus the run context (branch, prNumber, experimentSuite, trigger, runId, and latestMain for the current benchmark baseline) — so each is independently filterable/groupable in the Braintrust UI. skills is stored as a stable sorted comma-joined string, the same shape the baseline lookup queries on.

Baseline comparison

A full benchmark refresh on main stamps its experiments latestMain (old holders are demoted only after the whole batch uploads, so a crashed refresh never loses the baseline). PR runs look up the baseline for the same config identity and set it as the Braintrust base experiment, so every PR experiment opens pre-diffed against the latest main results.

In CI

After the export step, a continue-on-error upload runs for benchmark evals (missing credentials = a warning, never a failed job). The workflow upserts a single PR comment showing each experiment's pass rate, the Δ vs the latest-main baseline (linking to the pre-compared view), and pre-filtered links to this PR's experiments and the latest main results — see the comment on this PR for a live example produced by a simulated run. Ad hoc custom-suite dispatches upload under a branch-suffixed name so they're never mistaken for tracked results. Result files from before these fields existed still upload via a flat fallback.

How to review

Core logic, in dependency order:

packages/core/src/transcript/agent-trace.ts + packages/core/src/agents/* — parsing CLI transcripts into the shared AgentTrace model and per-call usage capture.
packages/core/src/braintrust-spans.ts — mapping a result + trace into the Braintrust span tree, row metadata, and token metrics (unit-tested).
apps/framework/scripts/upload-braintrust.ts — the uploader: experiment naming/metadata, baseline lookup/rotation, summary markdown for the PR comment.
apps/framework/lib/result-files.ts — result-file scan shared with the JSON export so the two destinations can't disagree.
.github/workflows/eval-refresh.yml — the upload + PR-comment steps (both continue-on-error).

Mechanical/generated (skim only): apps/web/src/data/eval-results.json and regression-eval-results.json are regenerated eval artifacts (the big line count is churn from a results refresh, verified byte-identical export behavior); pnpm-lock.yaml is the braintrust dependency.

Closes AI-907

One Braintrust experiment per repo experiment, one trace per eval: root eval row (scores from passed+checks, scenario metadata, token metrics) with llm/tool/task/score child spans rebuilt from the persisted transcript. Dual-write only — eval-results.json is untouched (export refactored onto a shared results scan, output byte-identical). Runners now also persist whole-run usage (Claude Code result line, Codex turn.completed, ai-sdk totalUsage) plus startedAt/durationMs; extra raw-result keys only, dropped by the web export. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

- Guard transcript↔toolCalls pairing by tool name (ai-sdk completion order can differ from call order under concurrent tools) - Normalize Claude Code inputTokens to include cache reads/writes so token counts compare across agents (OpenAI-style accounting) - Tolerate corrupt raw JSON files: skip + warn instead of aborting - Load experiment metadata once; hoist repeated lookups Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

This reverts commit 7953c07.

vercel · 2026-07-21T14:17:00Z

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated (UTC)
evals	Ignored	Preview	Jul 24, 2026 12:25pm

Trace-view counterpart of eval-results.json's headline passed flag, emitted ahead of the per-check score spans with a checksPassed summary. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Agent-agnostic turn model assembled from the canonical events: one turn per model API call with tool executions paired inside it, per-turn token usage and message ids (Claude Code stamps them from each assistant stream-json line), subagent sidechains nested under their Task call via parent_tool_use_id, codex grouped by closure rule, ai-sdk turns built straight from steps. Additive: the flat transcript/toolCalls scorer surface and eval-results.json are unchanged; raw files gain a trace field the Braintrust uploader prefers (llm turn spans owning their tool spans) with the flat mapping kept for older results. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

linear-code · 2026-07-21T16:38:44Z

AI-907 Upload results to Braintrust and validate experiments

Upload the results to Braintrust and try the workflow with Braintrust Experiments. This follows the idea that using Braintrust's experiments from the beginning may be more efficient than maintaining a separate internal view.

… spans A turn now means everything between user prompts (one per eval is normal); model API calls are steps inside it. Braintrust rows carry one scorer (passed) with the checks array in its output instead of a score per check. judge() calls are recorded via an AsyncLocalStorage collector (no per-eval wiring), persisted as judgeCalls in raw results, and rendered as llm children under the passed span with the rubric, the exact input the judge evaluated, model, duration, and tokens. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

The runner persists stream-json/exec-json stdout as a sibling <evalId>.transcript.jsonl (never inside the result JSON — it can be megabytes; .jsonl keeps it out of the results scan and the web export). The uploader attaches it to the root span via Braintrust Attachments, the documented mechanism for large documents (object storage, no span size limits), next to the existing raw-result attachment. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

- Anthropic usage accounting extracted to agents/claude-code/usage.ts, shared by the runner (run totals) and parser (per-message) so the OpenAI-style convention can't drift; finiteNumber joins json.ts and replaces the codex runner's inline coercer - Stale <evalId>.transcript.jsonl siblings removed on re-runs that produce no raw transcript, so an old transcript can't attach to a new result - Dead per-event startMs capture deleted from the trace assembler (nothing consumed it; span timing is synthetic by design) - turnKey renamed stepKey to match the turn/step model; stale module and test wording corrected - rawTranscriptPathFor and makeFilterPredicate centralized in lib/result-files.ts / lib/cli-args.ts, deleting the duplicated sibling-path convention and suite filters - judge() no longer records an all-undefined usage object; internal recordJudgeCall and BraintrustSpanNode dropped from the public core exports Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…7f477ae Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Lets workflow_dispatch pin exact eval/experiment pairs and publish them to Braintrust regardless of each eval's own suite: field, without touching the tracked eval-results.json/regression exports. Blocked on main since it's meant for in-progress branch work only. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Rodriguespn · 2026-07-22T13:30:14Z

-                experiments_json="$(pnpm --silent eval -- list --experiment-suite "$experiment_suite")"
-              fi

+          if jq -e 'index("custom") != null' <<< "$suite_json" > /dev/null 2>&1; then


To run one (or more) arbitrary pair evals/experiments (ex: investigate-security-001-public-table/codex-gpt5.4-mini) without being tied to a pre defined batch (benchmark or regression). This workflow only run in branhces. It fails if we try to run on main

…ploads init() now passes baseExperiment (most recent prior experiment for the same model, discovered via the REST API, best-effort) so summarize() and the UI diff against the previous refresh. --summary-md writes a markdown table (experiment links, passed %, Δ vs base); the workflow appends it to the step summary and upserts a single marker-matched PR comment — the braintrustdata/eval-action reviewer experience without adopting its execution model. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…ist links Experiments now carry trigger/branch/prNumber/ciRunId metadata and trigger/pr-N tags. The summary (and PR comment) links the experiments list pre-filtered by BTQL — all experiments from this CI run, this PR, or the scheduled main history — using the UI's search URL format. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Rodriguespn · 2026-07-22T17:31:23Z

Braintrust benchmark results

Experiment	passed	Δ vs main
claude-code-sonnet-5	3/3 · 100%	⚪ 0% vs main@aaaaaaa-f5d4a576

Braintrust results for PR #101 · Latest main results

Main refreshes (--set-baseline, passed by CI only for full benchmark refreshes on main) stamp metadata.latestMain and demote the previous holder after a successful upload, so exactly one experiment per model is the source-of-truth baseline — stale main history never matches. PR uploads compare against it via baseExperiment; the PR comment table becomes Experiment | passed (n/m · %) | Δ vs main, each delta linking the pre-compared view (?c=), with exactly two footer links: this PR's results and the latest main results. Experiment metadata trimmed to what the filters need (branch, prNumber). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…load Per the agreed sequence for multi-experiment refreshes: fetch every current latestMain holder per experiment up front, upload and mark the whole new batch, and only then clear the saved old holders. A batch that dies mid-way leaves the previous baselines in place, and clearing every holder (not just the newest) self-heals duplicates left by a past failed demotion. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Store exactly the doc's experiment-metadata fields: the decomposed config identity (agent/modelProvider/modelId/reasoningEffort/skills), branch, prNumber, experimentSuite, trigger, runId — plus latestMain stamped post-upload. Drops source/gitShortSha/ciRunUrl. Storing skills as the same joined string identityKey queries on also fixes the latestMain lookup: since e978e43 the stored array shape could never match the joined-string filter, so PR runs would silently lose their baseline. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Rodriguespn and others added 4 commits July 21, 2026 12:47

chore: refresh eval results

7953c07

Revert "chore: refresh eval results"

1094152

This reverts commit 7953c07.

Rodriguespn and others added 2 commits July 21, 2026 15:42

feat: overall passed verdict as a score span in Braintrust traces

f3e19bb

Trace-view counterpart of eval-results.json's headline passed flag, emitted ahead of the per-check score spans with a checksPassed summary. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Rodriguespn and others added 2 commits July 21, 2026 17:52

Rodriguespn changed the title ~~feat: upload benchmark results to Braintrust (dual-write with eval-results.json)~~ feat: upload benchmark results to Braintrust Jul 21, 2026

Rodriguespn and others added 3 commits July 22, 2026 12:55

fix: restore agent-skills submodule pointer accidentally downgraded in …

e83be44

…7f477ae Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Rodriguespn commented Jul 22, 2026

View reviewed changes

Rodriguespn and others added 2 commits July 22, 2026 14:40

Rodriguespn added the run-evals Add to a PR to refresh benchmark evals label Jul 22, 2026

chore: refresh eval results

631413e

vercel Bot deployed to Preview July 22, 2026 14:45 View deployment

Rodriguespn removed the run-evals Add to a PR to refresh benchmark evals label Jul 22, 2026

supabase deleted a comment from github-actions Bot Jul 22, 2026

Rodriguespn and others added 4 commits July 22, 2026 18:53

decouple experiment metadata field in four

e978e43

Rodriguespn mentioned this pull request Jul 29, 2026

feat: upload eval-results.json to Braintrust (AI-921) #89

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: upload benchmark results to Braintrust - #101

feat: upload benchmark results to Braintrust#101
Rodriguespn wants to merge 18 commits into
mainfrom
feat/braintrust-benchmark-upload

Rodriguespn commented Jul 21, 2026 •

edited

Loading

Uh oh!

vercel Bot commented Jul 21, 2026 •

edited

Loading

Uh oh!

linear-code Bot commented Jul 21, 2026 •

edited by Rodriguespn

Loading

Uh oh!

Rodriguespn Jul 22, 2026

Uh oh!

Rodriguespn commented Jul 22, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

Rodriguespn commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

How our results map to Braintrust

Experiment metadata

Baseline comparison

In CI

How to review

Uh oh!

vercel Bot commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

linear-code Bot commented Jul 21, 2026 • edited by Rodriguespn Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Rodriguespn Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

Rodriguespn commented Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Braintrust benchmark results

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Rodriguespn commented Jul 21, 2026 •

edited

Loading

vercel Bot commented Jul 21, 2026 •

edited

Loading

linear-code Bot commented Jul 21, 2026 •

edited by Rodriguespn

Loading

Rodriguespn commented Jul 22, 2026 •

edited

Loading