Fair task x condition trajectory report + README evaluation section - #14
Merged
Merged
Conversation
Describe the development evaluation harness and its implemented metrics (S1, S2, variant coverage, S3, usage metrics), how run records and reports relate, and no-cost dry-run/fake reproduction commands. Clarify that the tokenloom CLI is deterministic and never calls a model while the harness is the only part that makes real model calls.
The trajectory report compared conditions by a single success rate over
unequal task sets, so a condition that skipped the hardest task could
outrank one that ran it. Replace the per-condition success-rate table with:
- a primary task x condition matrix of raw comparable-success counts (s/k),
never a rate, with '-' for no runs, 'short' for an under-repeat cell, and
a '(+n incomparable)' suffix for dropped runs;
- an 'All tasks' row that prints a count only for a complete comparable set
and otherwise reads 'incomplete' or 'incomparable';
- a separate per-condition diagnostics table of medians, including Coverage
p50 and S3 p50 ('n/a' on rows predating those metrics).
Sections are keyed by prompt hash, requested model, resolved model, and
invocation, so a resolved-model-null legacy row never merges with a
resolved-value row. cli-agent-compact is excluded from the primary by rule
and kept as a diagnostics-only record.
Producer (renderTrajectoryReport) and the trajectoryEvidence gate change in
lock-step; the adoption bar, adopted-claim validation, and score-provenance
checks are unchanged. reports/trajectory.md is regenerated from the same
committed run records. Spec 4.12/9.7 and verification 12.8 describe the new
presentation; the self-test overlay is updated to the new format.
…nd sync self-test overlay Split the TrajectoryCell literal under the 140-char scope limit, and regenerate the trajectory-handwritten-number self-test overlay so it is a byte-for-byte copy of the new report producer plus the injected hand-written number, satisfying the selftest-drift check.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Two things a reviewer could not do before. First, the README now has an Evaluation section, so an external reviewer can understand what the development evaluation harness measures and how it differs from the deterministic CLI. Second, the trajectory report is now a fair comparison: it previously compared interface conditions by a single success rate computed over unequal task sets, so a condition that skipped the hardest task could outrank one that ran it.
What changed
s/k), never a rate, with-for no runs,shortfor an under-repeat cell, and a(+n incomparable)suffix for dropped runs;All tasksrow that prints a count only for a complete comparable set and otherwise readsincompleteorincomparable;n/aon rows predating those metrics).cli-agent-compactcondition is excluded from the primary by rule and kept as a diagnostics-only record.trajectoryEvidencescoring gate recomputes and line-matches the new format in lock-step with the producer.reports/trajectory.mdis regenerated from the same committed run records. Spec 4.12/9.7 and verification 12.8 describe the new presentation; the scoring self-test overlay is updated to the new format.The adoption bar, adopted-claim validation, and score-provenance checks are unchanged; only the presentation changed. S1/S2 and coverage/S3 semantics, thresholds, and committed
runs/*.jsonlare untouched.How I checked
All trajectory tests pass (89), including the scoring self-test overlay that proves a hand-written number is still rejected, and the drift check that the overlay stays a byte copy of the producer.
pnpm verifypasses all 13 gates. No paid model call runs at any point; the trajectory dry-run (eval trajectory --matrix eval/trajectory.yaml --dry-run) and the fake/--dry-runpaths are model-free.What I left alone
No weakened gate, no
vitest -u, no rewritten committed evidence:runs/*.jsonlis unchanged andreports/trajectory.mdis recomputed from it. No S1/S2/coverage/S3 semantic change, no threshold change, no benchmark-ledger edit. The approved Figma Community plugin URL in the README is unchanged. No real evaluation was run; the fresh 24-run set remains to be executed by the maintainer with an explicit budget.