Skip to content

Fair task x condition trajectory report + README evaluation section - #14

Merged
hsskey merged 3 commits into
mainfrom
fm/tokenloom-eval-harness-finish
Sep 18, 2026
Merged

hsskey merged 3 commits into
mainfrom
fm/tokenloom-eval-harness-finish

Conversation

@hsskey

@hsskey hsskey commented Sep 18, 2026

Copy link
Copy Markdown
Owner

Why

Two things a reviewer could not do before. First, the README now has an Evaluation section, so an external reviewer can understand what the development evaluation harness measures and how it differs from the deterministic CLI. Second, the trajectory report is now a fair comparison: it previously compared interface conditions by a single success rate computed over unequal task sets, so a condition that skipped the hardest task could outrank one that ran it.

What changed

  • README.md and README.ko.md gain an "Evaluation harness" section (EN/KO parity, English UI labels): the implemented metrics (S1, S2, variant coverage, S3, usage metrics), how run records and reports relate, comparison-set separation by prompt hash and model provenance, and no-cost dry-run/fake reproduction commands. It states plainly that the tokenloom CLI never calls a model while the harness is the only part that does.
  • The trajectory report becomes a task x condition matrix:
    • a primary table of raw comparable-success counts (s/k), never a rate, with - for no runs, short for an under-repeat cell, and a (+n incomparable) suffix for dropped runs;
    • an All tasks row that prints a count only for a complete comparable set and otherwise reads incomplete or incomparable;
    • a separate per-condition diagnostics table of medians, including Coverage p50 and S3 p50 (n/a on rows predating those metrics).
    • Sections are keyed by prompt hash, requested model, resolved model, and invocation, so a resolved-model-null legacy row never merges with a resolved-value row. The concluded cli-agent-compact condition is excluded from the primary by rule and kept as a diagnostics-only record.
  • The trajectoryEvidence scoring gate recomputes and line-matches the new format in lock-step with the producer. reports/trajectory.md is regenerated from the same committed run records. Spec 4.12/9.7 and verification 12.8 describe the new presentation; the scoring self-test overlay is updated to the new format.

The adoption bar, adopted-claim validation, and score-provenance checks are unchanged; only the presentation changed. S1/S2 and coverage/S3 semantics, thresholds, and committed runs/*.jsonl are untouched.

How I checked

pnpm typecheck
pnpm lint
pnpm build
pnpm vitest run packages/eval/test/trajectory.test.ts packages/eval/test/trajectory.rules.test.ts packages/eval/test/trajectory-report.test.ts packages/verify/test/trajectory-evidence.test.ts packages/verify/test/selftest-drift.test.ts
pnpm verify

All trajectory tests pass (89), including the scoring self-test overlay that proves a hand-written number is still rejected, and the drift check that the overlay stays a byte copy of the producer. pnpm verify passes all 13 gates. No paid model call runs at any point; the trajectory dry-run (eval trajectory --matrix eval/trajectory.yaml --dry-run) and the fake/--dry-run paths are model-free.

What I left alone

No weakened gate, no vitest -u, no rewritten committed evidence: runs/*.jsonl is unchanged and reports/trajectory.md is recomputed from it. No S1/S2/coverage/S3 semantic change, no threshold change, no benchmark-ledger edit. The approved Figma Community plugin URL in the README is unchanged. No real evaluation was run; the fresh 24-run set remains to be executed by the maintainer with an explicit budget.

Describe the development evaluation harness and its implemented metrics
(S1, S2, variant coverage, S3, usage metrics), how run records and reports
relate, and no-cost dry-run/fake reproduction commands. Clarify that the
tokenloom CLI is deterministic and never calls a model while the harness is
the only part that makes real model calls.
The trajectory report compared conditions by a single success rate over
unequal task sets, so a condition that skipped the hardest task could
outrank one that ran it. Replace the per-condition success-rate table with:

- a primary task x condition matrix of raw comparable-success counts (s/k),
  never a rate, with '-' for no runs, 'short' for an under-repeat cell, and
  a '(+n incomparable)' suffix for dropped runs;
- an 'All tasks' row that prints a count only for a complete comparable set
  and otherwise reads 'incomplete' or 'incomparable';
- a separate per-condition diagnostics table of medians, including Coverage
  p50 and S3 p50 ('n/a' on rows predating those metrics).

Sections are keyed by prompt hash, requested model, resolved model, and
invocation, so a resolved-model-null legacy row never merges with a
resolved-value row. cli-agent-compact is excluded from the primary by rule
and kept as a diagnostics-only record.

Producer (renderTrajectoryReport) and the trajectoryEvidence gate change in
lock-step; the adoption bar, adopted-claim validation, and score-provenance
checks are unchanged. reports/trajectory.md is regenerated from the same
committed run records. Spec 4.12/9.7 and verification 12.8 describe the new
presentation; the self-test overlay is updated to the new format.
…nd sync self-test overlay

Split the TrajectoryCell literal under the 140-char scope limit, and
regenerate the trajectory-handwritten-number self-test overlay so it is a
byte-for-byte copy of the new report producer plus the injected hand-written
number, satisfying the selftest-drift check.
@hsskey
hsskey merged commit 7a485aa into main Sep 18, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant