feat(eval): complete the fresh 24-run trajectory set across all cells - #16
Merged
Merged
Conversation
One clean attempt (attemptId 2026-09-18T...59.148Z) ran all 24 combinations without the recurring provider stop. The supersede mechanism now yields a complete comparison for prompt 3d7bfc957c1a / claude-opus-4-8: every task x condition x repeat cell has a comparable row (Runs 8, Incomparable 0, Missing 0 per condition). Regenerate reports/trajectory.md from the accumulated records; runs/2026-09-18.jsonl is append-only.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The fresh 24-run trajectory comparison (prompt
3d7bfc957c1a, modelclaude-opus-4-8) was incomplete: prior paid attempts each stopped partway through on the same intermittent provider error, leavingunknown-component,variant-only, andrecoverywithout a comparable row in every cell. After this change the set is complete, so the per-condition comparison can be read for every task x repeat.What changed
Appended one clean attempt's 24 trajectory rows to
runs/2026-09-18.jsonland regeneratedreports/trajectory.mdfrom the accumulated records. The supersede-by-attemptIdmechanism (merged in #15) replaced the earlier errored and incomparable slots, giving Runs 8 / Incomparable 0 / Missing 0 for each ofcli-canonical,cli-agent,mcp-agent. No source changed.Success per task in this partition (
prompt 3d7bfc957c1a / claude-opus-4-8):On the hard tasks (
unknown-component,recovery) the agent views match or beat the canonical baseline's success while roughly halving input tokens.How I checked
The regenerated report satisfies the
trajectoryEvidencegate: report measurements match the survivor records and no unsupported adoption claim is present.What I left alone
No gate weakened, no
vitest -u, no committed evidence rewritten.runs/*.jsonlis append-only and its committed prefix is byte-identical to the base. No source or adapter change. The verify run's incidentalbench/results.jsonlappend and timestamped proof record were reverted / left uncommitted, matching prior PRs.This spends money: the attempt made real Anthropic calls through the
claudeCLI under--budget-usd 5, cost$4.1103, each invocation under the$5ceiling. Cumulative for this converge effort is about$5.94($1.83prior +$4.11here).Diagnostic note (no fix applied here, for maintainer awareness): the recurring stop that ended the prior attempts is a single
claude -psubprocess exiting 1 with only aninitline and no result/usage, which the harness maps toMISSING_AUTHORITATIVE_COSTand treats as a set-level stop per SPEC 9.6. The capturedinitline shows the subprocess loading the machine's user-global Claude Code plugins, hooks, and MCP servers;claude --restrictedremoves only project and local settings, not user-global configuration (its help notes--strict-mcp-configis needed to skip MCP). So each eval call boots a heavy external stack that intermittently fails, and one such failure halts the whole 24-run. This attempt completed 24 calls cleanly, but the fragility remains and may be worth isolating the eval subprocess configuration or not halting the entire set on a single missing-cost call.