Reset the eval suite to four graded cases - #577
Merged
Conversation
The 17-case suite measured nothing. The last full matrix scored 92/95 with all three failures on one case, and suite v1's 14 cases were 14/14 on every model. It cost ~500 turns per model per run to learn almost nothing. Delete all 17 cases, the 13 eval-owned fixtures, and the demo scripts that depended on one of them. Keep the seven tests/fixtures directories that back real unit tests. EvalTier becomes easy|med|hard|xhard. Four new cases, one per tier, each with a target pass band: - tier-easy (15t, ~100%) floor tripwire; saturation here is intentional - tier-med (25t, 70-90%) a doc says 150 bps, a config says 200, an unused module already holds the right 175; only fixing the imported source counts - tier-hard (30t, 30-60%) the crash is raised beside a decoy rounding TODO but caused one hop away by a grouping key that collapses us-east and us-west - tier-xhard (40t, 0-25%) the functional suite is already green; the rubric grades production shape (versioned migrations, multi-worker-safe claiming, dead-letter inspection, no in-process polling) Each grader was driven against the cheats, not just the fix: contract tests are sha256-pinned, tier-med rejects rewiring the import to the decoy module, tier-hard's held-out assertions catch `bucket?.total ?? 0` masking the crash while the visible suite goes green, and tier-xhard fails its rubric on the starting fixture despite three passing tests. A reference implementation was written for tier-xhard to confirm it is hard rather than impossible. Tool-discipline baits become requireBehaviors bounds on the tier cases rather than a case each. The positive baits that need fixture affordances -- must use web_fetch, must discover a tool via tool_search, must dispatch via task -- are not folded yet and that coverage is currently dropped. Standing rule: a case that saturates its band gets promoted or retired, never kept. Every number in the tracking doc is now against the old suite; the first run on this suite sets the baseline. Cost: ~500 turns per model per run to 110.
tier-med: the decoys announced themselves. legacy/pricing.ts said "Nothing imports this module" and pricing.config.json said "Not read by the runtime", which handed over the answer without tracing the import graph. Both now read as ordinary plausible sources. tier-xhard rubric 1 only inspected notify.ts, so moving declareTable into another src file passed. It now checks every file in src/ that calls declareTable and requires it to be the code applying the migrations, with store.ts excluded since it defines the helper. tier-xhard rubric 4 rejected any setTimeout in src/, which fails a retry backoff -- good engineering -- for the wrong reason. Narrowed to setInterval. tier-easy accepted any test file that merely contained the word "version". It now requires a test that calls handleRequest against /version. Each change re-validated against a correct fix and the corresponding cheat.
Merging this branch with main exposed two typecheck failures that the branch alone could not show, because it never touches tsconfig.json and PR #578 widened typecheck scope to tests/ and evals/ after this branch was cut. parseCaseJson narrowed raw.tier with a `tier as EvalTier` cast, which is not a type guard, so it returned string where the EvalCase field is EvalTier. Replaced with a real isEvalTier predicate. The four tier-* sandbox fixtures now fall inside typecheck scope. They have their own package.json and tier-hard is intentionally buggy, so they join the existing eval-sandbox exclusions. crash-run and plugins/implement-feature stay in scope deliberately — they import production modules and #578 carved them out for that reason. Also swept the exclusion entries for fixtures this branch deletes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
187 files, +628 / −3,929. 17 cases → 4. ~500 turns per model per run → 110.
Supersedes #565 — close that one, this replaces it entirely.
Why
The suite had no discriminating power. Last full matrix: 92/95, with all three failures on one case. Suite v1's 14 cases were 14/14 on every model. A suite nothing fails cannot tell us whether CL-6940 helped, which is the whole reason we need it.
What changed
Deleted: all 17 cases, 13 eval-owned fixtures,
scripts/demo.ts+scripts/demo-compare.sh(the only non-eval consumers of a deleted fixture). Kept the seventests/fixtures/dirs backing real unit tests — audited each forsrc//tests/references first, since that directory is shared.Harness:
EvalTieris noweasy|med|hard|xhardwith an exportedEVAL_TIERS. Also drops the now-dead eslint ignore for the deletedbroken-toolchainfixture.The four cases
tier-easytier-medtier-hardus-east/us-west.tier-xhardGraders were validated against the cheats, not just the fix
A grader that can't distinguish a real fix from a shortcut is what got us here, so each was driven manually:
bucket?.total ?? 0makes the visible suite green and is rejected:FAIL: totalCents is 0 expected 2500 -- crash was masked, not fixed.claimBatch,listDeadLetters(), nosetInterval) passes — so it's hard, not impossible.Standing rule
A case that saturates its band gets promoted in difficulty or retired — never kept as-is. Without it the suite decays back into a wall of passing tests.
Known gaps, stated rather than buried
editViaShellCount: 0,envAssignmentCommandCount: 0,repeatedSearchCountbounds) ride on the tier cases. "Must useweb_fetch", "must discover viatool_search", "must dispatch viatask" each need the fixture to offer that affordance — that coverage is dropped for now.Harness tests: 118 pass, 0 fail. Prettier clean; eslint 0 errors on all touched paths.
Refs CL-6963, CL-6886 (superseded), CL-6931 (built as the xhard tier), CL-6959.