Skip to content

feat(eval): supersede interrupted trajectory attempts by attemptId in aggregation - #15

Merged
hsskey merged 2 commits into
mainfrom
fm/tokenloom-trajectory-24run
Sep 18, 2026
Merged

hsskey merged 2 commits into
mainfrom
fm/tokenloom-trajectory-24run

Conversation

@hsskey

@hsskey hsskey commented Sep 18, 2026

Copy link
Copy Markdown
Owner

Why

The trajectory report could not absorb a re-run of an interrupted attempt. A fresh run's rows landed in the same partition as an earlier attempt's rows for the same task/condition/repeat and double-counted, and an earlier errored row with a null cost held Total real cost at n/a permanently, because run records are append-only and never deleted. This adds an explicit attempt boundary so an interrupted attempt is superseded cleanly and a later clean run yields a fair comparison.

What changed

  • Trajectory run records carry an optional attemptId (the runTrajectory invocation instant, Date.prototype.toISOString()), captured once per invocation. Legacy rows without it sort as the empty string, forming one earliest attempt.
  • Aggregation supersedes rows by experiment slot {promptHash, requestedModel, invocation, task, condition, repeat} (resolvedModel excluded) before partitioning. Per slot the survivors maximise (comparable ? 1 : 0, attemptId): a comparable row outranks an errored one regardless of age, then the latest attempt wins, and ties survive. Superseded rows are excluded from every table, the medians, and the total, while remaining untouched in the append-only file.
  • The report prints one line Superseded rows: N. Total real cost sums costUsd over the surviving rows; a null among survivors still reads n/a.
  • The producer (packages/eval/src/trajectory.ts, trajectory-report.ts) and the independent trajectoryEvidence gate implement the same seven-step aggregation, kept in lock-step with spec 9.7 (and a 9.6 cross-reference distinguishing this from the rescored marker) and the trajectory tests in both packages.
  • Partition identity, comparability, S1/S2, coverage/S3, and every threshold are unchanged.

Provider data included: the fresh promptHash 3d7bfc957c1a run resolved to claude-opus-4-8 and stopped early at 9 of 24 runs on a transient provider error. known-component is a complete, fair comparison across all three conditions (each 2/2; mcp-agent lowest cost and input); unknown-component, variant-only, and recovery are recorded as incomplete. The earlier interrupted attempt's four rows are superseded, not deleted (Superseded rows: 4).

This spends real provider budget (eval). Each invocation stayed within its --budget-usd 5 ceiling; the report's Total real cost sums all surviving committed rows across time.

How I checked

pnpm build && pnpm typecheck && pnpm lint
pnpm test                       # trajectory + gate suites, incl. supersede scenarios A/B, all-errored, tie, no-op
pnpm verify                     # all 13 gates PASS (893s)
pnpm verify --gate scoring      # report recompute matches the run records (errors: [])

Supersede is proven both ways: packages/eval/test/trajectory-report.test.ts asserts the survivor selection and the Superseded rows line; packages/verify/test/trajectory-evidence.test.ts runs the real gate and rejects a report that leaves a superseded row in the aggregation.

What I left alone

No weakened gate, no vitest -u, no rewritten committed evidence: the interrupted attempt's rows stay in the append-only file and simply stop counting. No unrelated refactor. Reference-locked inputs (samples/**, eval/thresholds.json) untouched. As noted above, this change records real eval spend.

A captain-authorized clean re-run must yield a complete, fair task x condition
comparison without the earlier aborted attempt's rows double-counting. Records
now carry an optional attemptId (the runTrajectory invocation instant), and
aggregation supersedes rows by experiment slot before partitioning: a comparable
row outranks an errored one, then the latest attempt wins, and ties survive.
Superseded rows are excluded from every table and the total while remaining in
the append-only file; the report prints "Superseded rows: N".

The producer and the independent trajectoryEvidence gate implement the same rule,
kept in lock-step with spec 9.7 and the trajectory tests. This commit also lands
the aborted attempt's run records and a regenerated report as a no-op baseline.
…ctory report

The fresh promptHash 3d7bfc957c1a run resolved to claude-opus-4-8 and stopped
early (9 of 24 runs) on a transient provider error. Its rows supersede the
earlier interrupted attempt's known-component rows, so the report presents a
complete, fair known-component comparison across all three conditions and marks
the remaining tasks incomplete. The report records Superseded rows: 4 and a
Total real cost summed over the surviving rows. No record was modified or deleted.
@hsskey
hsskey merged commit 7bff97b into main Sep 18, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant