fix(eval): correct report statistic populations and label them - #9
Merged
Merged
Conversation
Fixed cost by input source computed the session prefix and per-component p50 over every sent row, so read-only repeats dragged the per-component value down to the input-tokens floor (2 for the committed snapshot group). Both columns now use cache-write rows only, and a group without a write reports n/a rather than a fabricated zero. The What works cost and latency columns keep the whole sent population but now label it and print n sent and n cache-write per group, so each p50's population is explicit. Nearest-rank p50 is retained and named in the report, spec 9.6, and verification section 7 alongside the ordinary-median definition the benchmark report uses. The cost column is documented as the provider CLI estimate, not billed cost. A new dated report is regenerated from committed rows at no cost; frozen reports and run records are untouched.
The cache-write filter is self-evident from the code; per the latest comment policy, general anti-regression rationale is not kept in active source.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The evaluation report's per-source statistics mixed cache-write and read-only rows in a single population. Repeated cache reads dragged the fixed-cost per-component token p50 down to the input-token floor (2 for the committed snapshot group), which is a population defect rather than a measurement. The cost and latency p50 columns also did not state which rows they covered, so two reports could look comparable when their populations differed.
What changed
Fixed cost by input sourcenow derives bothSession schema tokensandTokens per component p50from cache-write rows only, and printsn cache-write. A group without a cache-write row reportsn/ainstead of a fabricated zero.What workskeepsCost p50andLatency p50over all sent rows but labels that population and addsn sentandn cache-writecolumns.docs/reference/spec.mdsection 9.6, anddocs/reference/verification.mdsection 7 now name the nearest-rank p50 and document how it differs from the ordinary median the benchmark report uses. The cost column is labelled the provider CLI's estimate, not billed cost.How I checked
pnpm exec vitest run packages/eval/test/report.test.ts pnpm verifyreport.test.tsadds: odd/even p50 against the benchmark median (percentile([1,2,3,4],50)=2next tobenchMedian=2.5); cache-write population selection for the fixed-cost columns;n/afor a group with no cache-write row; per-column recomputation from the population named in each header; and a committed-rows regression check that the snapshot group no longer prints2.pnpm verifypasses every gate: types, patterns, tests, reference, determinism, properties, mutations, rules, benchmarks, scope, scoring, encoding, selftest.What I left alone
2026-09-03.md,2026-09-03-set2.md,2026-09-06.md),runs/, andreports/trajectory.mdhave no diff.vitest -u, no rewritten committed evidence, no unrelated refactor.This changes the report's statistics presentation, a documented output contract in
spec.mdsection 9.6 andverification.mdsection 7. It spends no money: no eval or Figma calls.