Skip to content

Preserve origin when mapping inline Playground rows - #213

Merged
David Elner (delner) merged 1 commit into
mainfrom
fix/preserve-origin-inline-playground-rows
Sep 16, 2026
Merged

David Elner (delner) merged 1 commit into
mainfrom
fix/preserve-origin-inline-playground-rows

Conversation

@delner

Copy link
Copy Markdown
Collaborator

What happened

Remote evals driven from a Playground never showed results. The dev server ran the eval to completion and streamed its output, but the Playground left every row in a loading state indefinitely.

Reproducing it depends on how the Playground sends data:

Playground action Wire format Behavior
Run an entire linked dataset data.dataset_id works
Run a selection of rows data.data (rows inline) rows never fill in

To reproduce: start a dev server with an evaluator, point a Playground at it, link a dataset, select a subset of rows, and run. The server completes successfully while the UI keeps spinning.

Why it happened

When rows are sent inline they arrive detached from their dataset, so each one carries an origin pointer identifying the row it came from. The Playground matches streamed results back to its grid rows using that pointer.

Eval#resolve_data_source rebuilt every inbound row from scratch and kept only two fields:

cases = data["data"].map do |d|
  {input: d["input"], expected: d["expected"]}
end

origin, tags and metadata were discarded. Everything downstream was already correct and waiting for them: Case declares an origin member, Runner#report_progress attaches origin to every progress event when present, and the SSE layer forwards it into the protocol. With origin dropped, progress events went out unidentified and the Playground had nothing to match them against.

A second problem sat behind the first. Origin was represented as a JSON string on the dataset path but arrives from the wire as a hash. OpenTelemetry rejects hash attribute values without raising, so simply passing the hash through would have silently dropped braintrust.origin from eval spans and broken the dataset link on experiment rows.

How this fixes it

The dev server stops rebuilding rows. It projects each inbound row onto the fields the SDK models, derived from the Case contract rather than restated alongside it:

CASE_FIELDS = Braintrust::Eval::Case.members.map(&:to_s).freeze

cases = data["data"].map { |row| row.slice(*CASE_FIELDS).transform_keys(&:to_sym) }

Deriving the list is what prevents a recurrence: adding a field to Case now carries it through the dev server automatically, instead of requiring a second edit that can be forgotten. Fields the SDK does not recognize are ignored rather than rejected, so a newer Playground cannot break an older SDK.

Origin is also now a hash throughout the SDK, serialized once at the span boundary. That removes a serialize and re-parse round trip, and makes pointers received from the wire identical in shape to pointers built from the dataset API.

@delner David Elner (delner) self-assigned this Sep 16, 2026
@delner
David Elner (delner) requested a review from a team as a code owner September 16, 2026 04:08
@delner David Elner (delner) added the bug Something isn't working label Sep 16, 2026
@delner
David Elner (delner) merged commit 17cbb85 into main Sep 16, 2026
10 checks passed
@delner
David Elner (delner) deleted the fix/preserve-origin-inline-playground-rows branch September 16, 2026 19:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants