Commit fba0fb7
committed
Cut the eval suite from 19 to 17 cases, one capability each
Four cases shared tests/fixtures/demo-comparison and tested the same thing —
implement or modify an endpoint in a toy API until tests pass — at 40 turns
apiece. Keep complex-stock-gate (richest: multi-file mutable state) and drop
complex-jwt, complex-pagination, and complex-idempotent-orders. Drop
complex-bugfix and its buggy-service fixture too; simple-health and
hidden-contract-inventory already cover implement-against-tests at both ends
of the difficulty range.
Of the four rescued cases, keep only the two that validate something nothing
else does:
- complex-fee-authority: authority resolution when docs and config actively
disagree with the import graph
- freename-list-dir: discovering an off-wire tool through tool_search
Drop the other two as duplicates of baselined cases: complex-rename-spread
duplicates complex-rename-user (rename across call sites), and
complex-pointer-chase duplicates misleading-symptom (trace indirection to the
real target) — which is also currently the only case that discriminates
between models.
Net: 19 to 17 cases, and 103 fewer turns of budget per model per matrix run.
Every recorded per-model total is now against a different suite, so the
tracking doc needs a fresh baseline before these numbers mean anything.1 parent 2ca0ae4 commit fba0fb7
39 files changed
Lines changed: 2 additions & 959 deletions
File tree
- evals/capability
- cases
- complex-bugfix
- complex-idempotent-orders
- complex-jwt
- complex-pagination
- complex-pointer-chase
- complex-rename-spread
- tests/fixtures
- buggy-service
- src
- middleware
- routes
- services
- types
- tests
- pointer-chase
- config
- keys
- src
- tests
- rename-spread
- src
- tests
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
15 | 15 | | |
16 | 16 | | |
17 | 17 | | |
18 | | - | |
19 | 18 | | |
20 | | - | |
21 | | - | |
22 | | - | |
| 19 | + | |
23 | 20 | | |
24 | 21 | | |
25 | 22 | | |
| |||
34 | 31 | | |
35 | 32 | | |
36 | 33 | | |
| 34 | + | |
37 | 35 | | |
38 | 36 | | |
39 | 37 | | |
| |||
This file was deleted.
This file was deleted.
Lines changed: 0 additions & 9 deletions
This file was deleted.
Lines changed: 0 additions & 121 deletions
This file was deleted.
This file was deleted.
0 commit comments