feat(eval): wire S3 visual scoring to the evaluation runner - #11
Merged
Merged
Conversation
Score variant coverage as a metric separate from S1 and S2. The expected set comes from committed design context (base.props plus variants[].props), narrowed to the requested selector for a variant-only request and identical across input representations. Actual variants come from HTML [data-variant] markers extracted by a start-tag tokenizer and normalized through the public selectVariant grammar. Report variantRecall, variantPrecision, duplicate counts, and missing or unexpected diagnosis, with explicit measured/error/not-recorded states. A coverage fail or error dominates unknown scores in the per-run verdict, so a row with missing variants can never receive a fully passing evaluation, and legacy run records without the metric stay readable and never claim a full pass. S3 fields remain type-compatible without asserting visual scoring. scoreS1, scoreS2, and countLiterals are unchanged; no parser or schema source is touched. Adds thresholds variantRecall, variantPrecision, and duplicateVariantsMax, locked once through the reference-lock tool.
Connect the existing visual helpers to the real runner so S3 is measured instead of hard-coded null, while keeping S1, S2, and variant coverage unchanged and independently gated. variantReferences maps each expected selector to its snapshot component and recorded renderPng by exact variantSelectorOf equality, cross-checking the importer's node-id filename rule so a hand-edited path is caught. scoreS3 is pure over PNG buffers and carries an explicit status: measured (worst-variant ratio, mean recorded), not-applicable only when no expected variant has a reference, or error for a mapping, decode, render, or DOM-disagreement fault. An error withholds the gated s3, so a half-measured run can never read as a pass, and a variant the output did not render is a coverage miss, not an S3 error. measureS3 renders only when a reference exists, so a no-reference sample yields not-applicable without a browser; a RenderPort is injected so tests use synthetic shots and CI never launches a browser. A DOM marker list that disagrees with the tokenizer fails both S3 and coverage. Trajectory rows record s3Status through the same helper. C2 (worst variant versus mean as the gated value) is open; the worst-variant default is provisional and both values are recorded, so the gated choice is a one-line change. No committed row, report, or reference lock is modified. spec.md 9.5 documents the S3 status semantics and the provisional aggregate.
# Conflicts: # docs/reference/spec.md # packages/eval/src/agent-session-port.ts # packages/eval/src/report.ts # packages/eval/src/runner.ts # packages/eval/src/runs.ts # packages/eval/src/stats.ts # packages/eval/src/trajectory.ts # packages/eval/src/variant-reference.ts # verify/selftest/scoring/numeric-literal/apply/packages/eval/src/report.ts
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Before this change the evaluation runner hard-coded
s3 = null, so a generatedcomponent was never checked against its reference render. After it, S3 is a real
visual-difference metric with an explicit status, so a mapping or render fault can
no longer read as a passing evaluation, while S1, S2, and variant coverage are
unchanged and gated independently.
Stacked on the variant-coverage work (T1) in #10; it consumes T1's expected-set,
marker, and verdict interfaces. Base this PR on
fm/tokenloom-eval-coverageandmerge #10 first.
What changed
variantReferencesmaps each expected selector to its snapshot component andrecorded
renderPngby exactvariantSelectorOfequality, cross-checking theimporter's node-id filename rule.
scoreS3is pure over PNG buffers and returnsmeasured(worst-variant ratio,mean recorded),
not-applicable(only when no expected variant has a reference),or
error(mapping, decode, render, or DOM-disagreement fault). Anerrorwithholds the gated
s3; a variant the output did not render is a coverage miss,not an S3 error.
measureS3renders only when a reference exists, so a no-reference sample isnot-applicablewith no browser. ARenderPortis injected so tests usesynthetic shots and CI never launches a browser. A DOM marker list that disagrees
with the tokenizer fails both S3 and coverage.
s3Statusthrough the same helper (not-applicableforthe three current samples).
runs.tsandagent-session-port.tsgain optionals3Statusands3Detail.spec.md9.5 documents the S3 status semantics.C2 (worst variant versus mean as the gated value) is open. The worst-variant
default is provisional; both values are recorded, so the choice is a one-line
change. No committed run row, report, or reference lock is modified.
How I checked
pnpm build pnpm lint pnpm typecheck pnpm verify --gate scope,types,patterns,determinism pnpm verify --gate tests pnpm --filter @tokenloom/eval exec vitest runAll pass: the
testsgate reports 1375 tests, 0 skipped, 0 failed (includes thescoring report recomputation); the eval package runs 353 tests including the new
packages/eval/test/s3.test.ts(22) that cover every T3 acceptance item(identical/different PNG, no-reference null, each error code, one failing variant
keeps partial ratios, selector matching, recall < 1 stays measured with overall
false, DOM disagreement on both, no browser, trajectory via the same helper).
Full integrated verification with benchmarks and selftest is deferred to the
serialized heavy-validation slot.
What I left alone
No weakened gate, no
vitest -u, no rewritten committed run rows, reports, orreference lock, no change to S1/S2 or coverage semantics, and no product package
outside
packages/eval. This does not spend money, make a Figma call, or run apaid model. The
evalinternal CLI contract gains optional row fields only.