[FEATURE] Evals: thin scorer / experiment-item result surface for external evidence consumers

[Rul1an]

We just shipped a small sample on the Assay side that consumes a frozen Mastra scorer / experiment-item result as external evidence:

https://github.com/Rul1an/assay/tree/main/examples/mastra-scorer-evidence

The reason for the sample is pretty narrow: we wanted to test the smallest honest Mastra reliability surface an external evidence consumer could ingest without collapsing back into traces, Studio metrics, or dashboard semantics.

So we kept the shape intentionally small:

• scorer name
• score
• outcome
• dataset version ref
• item ref
• target type
• timestamp

We are not treating that artifact as Mastra truth, and we are not assuming the checked-in fixture shape is a stable wire contract.

The question is mainly about seam choice:

If an external evidence consumer wants the smallest honest Mastra reliability surface, is a bounded scorer / experiment-item result roughly the right place to start, or is there a thinner scorer result surface you would rather point them at?

If we are aiming at the wrong layer, happy to adjust the sample.

[daneatmastra]

Thank you for reporting this issue! We have labeled it for trio-tracery and we will look into it as soon as possible.

If you're experiencing an error, please provide a minimal reproducible example whenever possible to help us resolve it quickly.

[epinzur]

Hey @Rul1an — to make sure I’m reading this right: you’re asking whether the scorer / experiment-item result is the right integration point for an external consumer that needs a minimal reliability signal.

A heads up that this may not be the right surface to build against. The scorer itself is just a score definition — it doesn’t carry result data. And experiment-item results aren’t likely to be the place where scored output lives going forward. Score results will live in the observability scores table.

If you’re looking for the thinnest honest surface for external consumption, that’s probably where you want to be looking.

Can you share a bit more about what Assay is doing with this data downstream? Would help us understand whether you’re pointed at the right thing.

[Rul1an]

Thanks, this is very helpful.

Yes, that’s the question we’re trying to answer: what is the smallest honest Mastra surface an external consumer could ingest as a bounded reliability signal.

What Assay does downstream is pretty narrow. We treat imported artifacts as external evidence, not as framework truth. In practice that means:

• carry a small upstream score signal into a reviewable evidence plane
• keep freshness and provenance attached
• avoid inheriting traces, dashboard semantics, or runtime correctness as truth

So if score results are moving to the observability scores table, that sounds like the better seam for us than scorer definitions or experiment-item results.

In that case we’d likely re-cut the sample around one bounded score record, probably something closer to:

• scorer identifier
• score value
• target reference
• timestamp
• optional short reason reference

and keep trace, log, and broader observability payloads out of v1.

To make this real on our side, we can switch from the current frozen scorer / experiment-item sample to a tiny Mastra app with scoring enabled, persist one actual score result to storage, and rebuild the sample from that row-level shape instead.

If that direction matches where Mastra is going, is a single score record from the observability scores store roughly the right place to start for an external consumer, or is there an even thinner score-result surface you’d rather point us at?

[epinzur]

Hey @Rul1an — yes, individual score records from observability are the right seam for an external evidence consumer with this use case.

To make this concrete: Mastra's observability layer has a pluggable exporter interface (ObservabilityExporter) that receives typed events — including dedicated ScoreEvent signals. You could write a minimal custom exporter that only handles score events and ignores everything else (traces, logs, metrics).

The ScoreEvent contains an ExportedScore with exactly the kind of bounded shape you described:

• scorerId / scorerName — scorer identifier
• score — numeric value
• reason — optional explanation
• timestamp — when the score was recorded
• traceId / spanId — optional target anchors
• targetEntityType — what was scored
• scorerVersion, scoreSource, metadata — optional extras

A minimal exporter would look something like:

import { BaseExporter } from '@mastra/observability';
import type { ScoreEvent, TracingEvent } from '@mastra/core/observability';

class AssayScoreExporter extends BaseExporter {
  name = 'assay-score-exporter';

  async onScoreEvent(event: ScoreEvent): Promise<void> {
    const { scorerId, scorerName, score, reason, timestamp, traceId, targetEntityType } = event.score;
    // Forward to Assay however you want
  }

  // Required by interface, but can be a no-op if you only care about scores
  async exportTracingEvent(_event: TracingEvent): Promise<void> {}
}

Then wire it up:

import { Observability } from '@mastra/observability';

new Mastra({
  observability: new Observability({
    configs: {
      default: {
        serviceName: 'my-app',
        exporters: [new AssayScoreExporter()],
      },
    },
  }),
});

The event bus routes ScoreEvents only to exporters that implement onScoreEvent, so you get exactly the signal type you want without inheriting traces, dashboard semantics, or runtime concerns. The ExportedScore shape is intentionally the thinnest score-result surface we expose — it's closer to your target than the scorer definitions or experiment-item results.

For your sample, rebuilding around a real score record from this exporter interface would give you a stable, minimal integration point.

[Rul1an]

Quick update from our side: we’ve now merged the Assay-side recut and rebuilt the sample around the bounded exporter / score-event seam you pointed us to.

https://github.com/Rul1an/assay/tree/main/examples/mastra-score-event-evidence

We kept it intentionally small and intentionally honest. In particular, we are not treating the richer frozen sample shape as proof that the live exporter callback already arrives that way today. It is still pre-proof on the live callback path, and the next step for us is the boring but important one: capture one real callback and check the sample against that.

That steer was really helpful, trouwens. If there is one field here you’d drop or rename to keep the sample even closer to the exporter payload as it exists today, happy to tighten it a bit more. Otherwise we’ll treat this as the right direction and do the live capture next.

[epinzur]

@Rul1an I see in your document mention of exporter.addScoreToTrace(...)

Note that this is the old pathway for sending score data and it will be deprecated soon.

The ScoreEvent is the new path. Additionally the ExportedScore object inside the event will soon be gaining a few extra fields. The most important for y'all might be the ScoreId; a uuid linking the specific score back to the mastra system.

[Rul1an]

Quick update from our side: we merged the follow-up and tightened the sample another notch around the seam you pointed us to.

So it is now explicitly ScoreEvent-first, addScoreToTrace(...) is only kept as legacy / migration context, and we added one bounded slot for the upcoming ScoreId anchor as well.

https://github.com/Rul1an/assay/tree/main/examples/mastra-score-event-evidence

We still are not treating that as live callback proof yet. The next boring but important step for us is still to capture one real onScoreEvent and check the frozen sample against it.

That extra nudge was exactly right, thanks.