Add hidden-contract-inventory eval case (held-out tests) - #521
Merged
TheGreatAxios merged 3 commits intoAug 22, 2026
Merged
Conversation
Fixture ships an inventory service with a prose contract (API.md) for stock reservations that is unimplemented; visible tests only cover the unrelated products routes. verify.sh copies held-out tests from the case dir into the workdir at grade time so the agent never sees them, covering the contract's edge cases (available-vs-on-hand stock, lazy TTL expiry, boundary reservation size, cancel semantics). Verified against an oracle (solution/solve.sh) which passes all 16 tests, and against a lazy implementation that only checks on-hand stock and skips lazy expiry on cancel: bun test alone stays green on it, but verify.sh catches 3 held-out failures.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
New capability eval case
hidden-contract-inventory, following the SWE-bench FAIL_TO_PASS / PASS_TO_PASS pattern: the grader's tests are never visible to the agent.tests/fixtures/inventory-service): a small inventory service.API.mdis a prose contract for a stock-reservations feature (available-vs-on-hand stock distinction, lazy TTL expiry, cancel semantics) that is unimplemented —src/services/reservations.tsthrows, the route returns 501. Only the unrelated/productsroutes have visible tests (tests/products.test.ts), so a greenbun teston arrival proves nothing about the target feature./reservationsperAPI.md, without enumerating edge cases — reading the contract carefully is the task.evals/capability/cases/hidden-contract-inventory/verify.sh): copieshidden/reservations.test.ts(10 held-out tests) from the case directory into the workdir at grade time and runsbun test. The case directory is never copied into the agent's workdir by the runner, so the agent cannot read or hard-code against these tests. The held-out suite also re-runs the shipped product tests, so vandalizing unrelated code is caught too.evals/capability/cases/hidden-contract-inventory/solution/solve.sh+ reference implementation, proven to pass verify.sh end to end.Evidence
bash solve.sh && bash verify.sh→ exit 0, 16/16 tests pass (10 hidden + 6 visible).bash verify.sh→ exit 1, 9/16 fail (as expected — feature not built).bun testalone stays green (0 fail, since no reservation tests are visible), butbash verify.sh(held-out tests injected) → exit 1, 3/16 fail. This is the core exploit-pass check: making only the visible suite green is not sufficient, and the failures are genuine behavioral misses, not crashes.Turn budget: 40, matching the closest existing multi-file/mutable-state case (
complex-stock-gate, also 40) given comparable file count and edge-case surface, times a safety margin over the oracle's ~2-file change.No changes to
scripts/eval-capability.ts— held-out grading works with the existing runner (case dir is never copied into the workdir).Test plan
bun testgreen on arrival (visible-only, feature unimplemented)bun run typecheckclean on arrivalverify.sh(16/16)verify.shwhile passing plainbun testverify.shFixes CL-6881
https://linear.app/abklabs/issue/CL-6881