Add misleading-symptom eval case - #523
Merged
TheGreatAxios merged 2 commits intoAug 22, 2026
Merged
Conversation
Crash visibly implicates src/routes/reports.ts (a decoy rounding TODO sits right next to the throwing line); the actual defect is one hop away in src/services/aggregate.ts, which seeds a reduce with the first line item instead of 0. Guarding only the crash site removes the error but yields a plausible-but-wrong total, which the held-out tests catch. Verified: unfixed exit 1 (crash in reports.ts), guard-only-A patch exit 1 (wrong value, no crash), oracle fix in aggregate.ts exit 0.
…relax decoy check
TheGreatAxios
force-pushed
the
cl-6884-eval-case-misleading-symptom-error-points-away-from-the-root
branch
from
August 22, 2026 22:05
8407ca6 to
9aa445b
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
New v2 capability eval case per CL-6825 design #5:
misleading-symptom.tests/fixtures/report-pipeline:GET /reports/:idcomputes a total from line items.src/routes/reports.ts— the crash throws here (rawTotal.toFixed is not a function), right next to a decoy// TODO: rounding here is probably wrongcomment on a line that is actually correct.src/services/aggregate.ts—computeReportTotalseeds its reduce with the first line item itself instead of0, so it returns the wrong runtime type; that's what makes A's.toFixed()call throw.verify.sh) copies in held-out tests (hidden/reports.heldout.ts) at grade time asserting the actual correct totals for 1/2/5-item reports, runs the shipped route tests alongside, and asserts the decoy rounding line inreports.tsis untouched (the fix belongs inaggregate.ts).solution/solve.shreplacesaggregate.tswith a correct reduce seeded at0.Exploit evidence (all three scenarios run against a fresh workdir)
verify.shexitTypeError, stack trace points at the decoy linereports.tsaround.toFixed(), falling back tototal = 0solve.shfixesaggregate.tsTurn budget: 30 (oracle applies in a single file copy; ×3 headroom over that, floor 25).
Fixes CL-6884
https://linear.app/abklabs/issue/CL-6884