Skip to content

fix(eval): correct report statistic populations and label them - #9

Merged
hsskey merged 2 commits into
mainfrom
fm/tokenloom-eval-statistics
Sep 17, 2026
Merged

hsskey merged 2 commits into
mainfrom
fm/tokenloom-eval-statistics

Conversation

@hsskey

@hsskey hsskey commented Sep 17, 2026

Copy link
Copy Markdown
Owner

Why

The evaluation report's per-source statistics mixed cache-write and read-only rows in a single population. Repeated cache reads dragged the fixed-cost per-component token p50 down to the input-token floor (2 for the committed snapshot group), which is a population defect rather than a measurement. The cost and latency p50 columns also did not state which rows they covered, so two reports could look comparable when their populations differed.

What changed

  • Fixed cost by input source now derives both Session schema tokens and Tokens per component p50 from cache-write rows only, and prints n cache-write. A group without a cache-write row reports n/a instead of a fabricated zero.
  • What works keeps Cost p50 and Latency p50 over all sent rows but labels that population and adds n sent and n cache-write columns.
  • The report note, docs/reference/spec.md section 9.6, and docs/reference/verification.md section 7 now name the nearest-rank p50 and document how it differs from the ordinary median the benchmark report uses. The cost column is labelled the provider CLI's estimate, not billed cost.
  • A new dated report is regenerated from the already-committed run records at no cost.

How I checked

pnpm exec vitest run packages/eval/test/report.test.ts
pnpm verify
  • report.test.ts adds: odd/even p50 against the benchmark median (percentile([1,2,3,4],50)=2 next to benchMedian=2.5); cache-write population selection for the fixed-cost columns; n/a for a group with no cache-write row; per-column recomputation from the population named in each header; and a committed-rows regression check that the snapshot group no longer prints 2.
  • Full pnpm verify passes every gate: types, patterns, tests, reference, determinism, properties, mutations, rules, benchmarks, scope, scoring, encoding, selftest.

What I left alone

  • The nearest-rank percentile function is unchanged, so the byte-level scoring and trajectory gates keep their meaning; the median difference is documented, not altered.
  • Frozen dated reports (2026-09-03.md, 2026-09-03-set2.md, 2026-09-06.md), runs/, and reports/trajectory.md have no diff.
  • No weakened gate, no vitest -u, no rewritten committed evidence, no unrelated refactor.

This changes the report's statistics presentation, a documented output contract in spec.md section 9.6 and verification.md section 7. It spends no money: no eval or Figma calls.

Fixed cost by input source computed the session prefix and per-component
p50 over every sent row, so read-only repeats dragged the per-component
value down to the input-tokens floor (2 for the committed snapshot group).
Both columns now use cache-write rows only, and a group without a write
reports n/a rather than a fabricated zero.

The What works cost and latency columns keep the whole sent population but
now label it and print n sent and n cache-write per group, so each p50's
population is explicit. Nearest-rank p50 is retained and named in the
report, spec 9.6, and verification section 7 alongside the ordinary-median
definition the benchmark report uses. The cost column is documented as the
provider CLI estimate, not billed cost.

A new dated report is regenerated from committed rows at no cost; frozen
reports and run records are untouched.
The cache-write filter is self-evident from the code; per the latest
comment policy, general anti-regression rationale is not kept in active
source.
@hsskey
hsskey merged commit ec7c1b3 into main Sep 17, 2026
2 checks passed
@hsskey
hsskey deleted the fm/tokenloom-eval-statistics branch September 17, 2026 23:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant