feat(eval): supersede interrupted trajectory attempts by attemptId in aggregation - #15
Merged
Merged
Conversation
A captain-authorized clean re-run must yield a complete, fair task x condition comparison without the earlier aborted attempt's rows double-counting. Records now carry an optional attemptId (the runTrajectory invocation instant), and aggregation supersedes rows by experiment slot before partitioning: a comparable row outranks an errored one, then the latest attempt wins, and ties survive. Superseded rows are excluded from every table and the total while remaining in the append-only file; the report prints "Superseded rows: N". The producer and the independent trajectoryEvidence gate implement the same rule, kept in lock-step with spec 9.7 and the trajectory tests. This commit also lands the aborted attempt's run records and a regenerated report as a no-op baseline.
…ctory report The fresh promptHash 3d7bfc957c1a run resolved to claude-opus-4-8 and stopped early (9 of 24 runs) on a transient provider error. Its rows supersede the earlier interrupted attempt's known-component rows, so the report presents a complete, fair known-component comparison across all three conditions and marks the remaining tasks incomplete. The report records Superseded rows: 4 and a Total real cost summed over the surviving rows. No record was modified or deleted.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The trajectory report could not absorb a re-run of an interrupted attempt. A fresh run's rows landed in the same partition as an earlier attempt's rows for the same task/condition/repeat and double-counted, and an earlier errored row with a null cost held
Total real costatn/apermanently, because run records are append-only and never deleted. This adds an explicit attempt boundary so an interrupted attempt is superseded cleanly and a later clean run yields a fair comparison.What changed
attemptId(therunTrajectoryinvocation instant,Date.prototype.toISOString()), captured once per invocation. Legacy rows without it sort as the empty string, forming one earliest attempt.{promptHash, requestedModel, invocation, task, condition, repeat}(resolvedModel excluded) before partitioning. Per slot the survivors maximise(comparable ? 1 : 0, attemptId): a comparable row outranks an errored one regardless of age, then the latest attempt wins, and ties survive. Superseded rows are excluded from every table, the medians, and the total, while remaining untouched in the append-only file.Superseded rows: N.Total real costsumscostUsdover the surviving rows; a null among survivors still readsn/a.packages/eval/src/trajectory.ts,trajectory-report.ts) and the independenttrajectoryEvidencegate implement the same seven-step aggregation, kept in lock-step with spec 9.7 (and a 9.6 cross-reference distinguishing this from therescoredmarker) and the trajectory tests in both packages.Provider data included: the fresh promptHash
3d7bfc957c1arun resolved toclaude-opus-4-8and stopped early at 9 of 24 runs on a transient provider error.known-componentis a complete, fair comparison across all three conditions (each 2/2;mcp-agentlowest cost and input);unknown-component,variant-only, andrecoveryare recorded as incomplete. The earlier interrupted attempt's four rows are superseded, not deleted (Superseded rows: 4).This spends real provider budget (eval). Each invocation stayed within its
--budget-usd 5ceiling; the report'sTotal real costsums all surviving committed rows across time.How I checked
Supersede is proven both ways:
packages/eval/test/trajectory-report.test.tsasserts the survivor selection and theSuperseded rowsline;packages/verify/test/trajectory-evidence.test.tsruns the real gate and rejects a report that leaves a superseded row in the aggregation.What I left alone
No weakened gate, no
vitest -u, no rewritten committed evidence: the interrupted attempt's rows stay in the append-only file and simply stop counting. No unrelated refactor. Reference-locked inputs (samples/**,eval/thresholds.json) untouched. As noted above, this change records real eval spend.