Skip to content

Add hidden-contract-inventory eval case (held-out tests) - #521

Merged
TheGreatAxios merged 3 commits into
mainfrom
cl-6881-eval-case-hidden-contract-inventory-held-out-tests-prose
Aug 22, 2026
Merged

Add hidden-contract-inventory eval case (held-out tests)#521
TheGreatAxios merged 3 commits into
mainfrom
cl-6881-eval-case-hidden-contract-inventory-held-out-tests-prose

Conversation

@TheGreatAxios

Copy link
Copy Markdown
Collaborator

Summary

New capability eval case hidden-contract-inventory, following the SWE-bench FAIL_TO_PASS / PASS_TO_PASS pattern: the grader's tests are never visible to the agent.

  • Fixture (tests/fixtures/inventory-service): a small inventory service. API.md is a prose contract for a stock-reservations feature (available-vs-on-hand stock distinction, lazy TTL expiry, cancel semantics) that is unimplemented — src/services/reservations.ts throws, the route returns 501. Only the unrelated /products routes have visible tests (tests/products.test.ts), so a green bun test on arrival proves nothing about the target feature.
  • Prompt: asks the agent to implement /reservations per API.md, without enumerating edge cases — reading the contract carefully is the task.
  • Grader (evals/capability/cases/hidden-contract-inventory/verify.sh): copies hidden/reservations.test.ts (10 held-out tests) from the case directory into the workdir at grade time and runs bun test. The case directory is never copied into the agent's workdir by the runner, so the agent cannot read or hard-code against these tests. The held-out suite also re-runs the shipped product tests, so vandalizing unrelated code is caught too.
  • Oracle: evals/capability/cases/hidden-contract-inventory/solution/solve.sh + reference implementation, proven to pass verify.sh end to end.

Evidence

  • Oracle: bash solve.sh && bash verify.sh → exit 0, 16/16 tests pass (10 hidden + 6 visible).
  • Unimplemented stub: bash verify.sh → exit 1, 9/16 fail (as expected — feature not built).
  • Lazy/naive implementation (checks on-hand stock instead of availability net of active reservations; skips lazy expiry before cancel — both plausible-but-wrong, not error-throwing): bun test alone stays green (0 fail, since no reservation tests are visible), but bash verify.sh (held-out tests injected) → exit 1, 3/16 fail. This is the core exploit-pass check: making only the visible suite green is not sufficient, and the failures are genuine behavioral misses, not crashes.

Turn budget: 40, matching the closest existing multi-file/mutable-state case (complex-stock-gate, also 40) given comparable file count and edge-case surface, times a safety margin over the oracle's ~2-file change.

No changes to scripts/eval-capability.ts — held-out grading works with the existing runner (case dir is never copied into the workdir).

Test plan

  • bun test green on arrival (visible-only, feature unimplemented)
  • bun run typecheck clean on arrival
  • Oracle passes verify.sh (16/16)
  • Lazy implementation fails verify.sh while passing plain bun test
  • Fully unimplemented stub fails verify.sh

Fixes CL-6881

https://linear.app/abklabs/issue/CL-6881

Fixture ships an inventory service with a prose contract (API.md) for
stock reservations that is unimplemented; visible tests only cover the
unrelated products routes. verify.sh copies held-out tests from the
case dir into the workdir at grade time so the agent never sees them,
covering the contract's edge cases (available-vs-on-hand stock, lazy
TTL expiry, boundary reservation size, cancel semantics).

Verified against an oracle (solution/solve.sh) which passes all 16
tests, and against a lazy implementation that only checks on-hand
stock and skips lazy expiry on cancel: bun test alone stays green on
it, but verify.sh catches 3 held-out failures.
@linear-code

linear-code Bot commented Aug 22, 2026

Copy link
Copy Markdown

CL-6881

@TheGreatAxios
TheGreatAxios merged commit dba0d3f into main Aug 22, 2026
3 of 4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant