Skip to content

feat(eval): complete the fresh 24-run trajectory set across all cells - #16

Merged
hsskey merged 1 commit into
mainfrom
fm/tokenloom-24run-converge
Sep 18, 2026
Merged

hsskey merged 1 commit into
mainfrom
fm/tokenloom-24run-converge

Conversation

@hsskey

@hsskey hsskey commented Sep 18, 2026

Copy link
Copy Markdown
Owner

Why

The fresh 24-run trajectory comparison (prompt 3d7bfc957c1a, model claude-opus-4-8) was incomplete: prior paid attempts each stopped partway through on the same intermittent provider error, leaving unknown-component, variant-only, and recovery without a comparable row in every cell. After this change the set is complete, so the per-condition comparison can be read for every task x repeat.

What changed

Appended one clean attempt's 24 trajectory rows to runs/2026-09-18.jsonl and regenerated reports/trajectory.md from the accumulated records. The supersede-by-attemptId mechanism (merged in #15) replaced the earlier errored and incomparable slots, giving Runs 8 / Incomparable 0 / Missing 0 for each of cli-canonical, cli-agent, mcp-agent. No source changed.

Success per task in this partition (prompt 3d7bfc957c1a / claude-opus-4-8):

Task cli-canonical cli-agent mcp-agent
known-component 2/2 (~28k in) 2/2 (~28k) 2/2 (~28k)
unknown-component 1/2 (~102k) 2/2 (~43k) 2/2 (~43k)
variant-only 2/2 2/2 1/2
recovery 2/2 (~79k) 1/2 2/2 (~66k)

On the hard tasks (unknown-component, recovery) the agent views match or beat the canonical baseline's success while roughly halving input tokens.

How I checked

pnpm tokenloom eval trajectory --matrix eval/trajectory.yaml --budget-usd 5
# 24/24 rows appended, $4.1103, exit 3 (three runs unsuccessful: MAX_TURNS / NO_SUCCESSFUL_TOOL_CALL / NO_TOOL_REQUEST; not a set stop)

pnpm verify
# PASS types patterns tests reference determinism properties mutations rules benchmarks scope scoring encoding selftest (890s)

The regenerated report satisfies the trajectoryEvidence gate: report measurements match the survivor records and no unsupported adoption claim is present.

What I left alone

No gate weakened, no vitest -u, no committed evidence rewritten. runs/*.jsonl is append-only and its committed prefix is byte-identical to the base. No source or adapter change. The verify run's incidental bench/results.jsonl append and timestamped proof record were reverted / left uncommitted, matching prior PRs.

This spends money: the attempt made real Anthropic calls through the claude CLI under --budget-usd 5, cost $4.1103, each invocation under the $5 ceiling. Cumulative for this converge effort is about $5.94 ($1.83 prior + $4.11 here).

Diagnostic note (no fix applied here, for maintainer awareness): the recurring stop that ended the prior attempts is a single claude -p subprocess exiting 1 with only an init line and no result/usage, which the harness maps to MISSING_AUTHORITATIVE_COST and treats as a set-level stop per SPEC 9.6. The captured init line shows the subprocess loading the machine's user-global Claude Code plugins, hooks, and MCP servers; claude --restricted removes only project and local settings, not user-global configuration (its help notes --strict-mcp-config is needed to skip MCP). So each eval call boots a heavy external stack that intermittently fails, and one such failure halts the whole 24-run. This attempt completed 24 calls cleanly, but the fragility remains and may be worth isolating the eval subprocess configuration or not halting the entire set on a single missing-cost call.

One clean attempt (attemptId 2026-09-18T...59.148Z) ran all 24 combinations without the recurring provider stop.
The supersede mechanism now yields a complete comparison for prompt 3d7bfc957c1a / claude-opus-4-8: every task x condition x repeat cell has a comparable row (Runs 8, Incomparable 0, Missing 0 per condition).
Regenerate reports/trajectory.md from the accumulated records; runs/2026-09-18.jsonl is append-only.
@hsskey
hsskey merged commit ec071d6 into main Sep 18, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant