Skip to content

feat(eval): wire S3 visual scoring to the evaluation runner - #11

Merged
hsskey merged 3 commits into
mainfrom
fm/tokenloom-eval-visual
Sep 17, 2026
Merged

hsskey merged 3 commits into
mainfrom
fm/tokenloom-eval-visual

Conversation

@hsskey

@hsskey hsskey commented Sep 17, 2026

Copy link
Copy Markdown
Owner

Why

Before this change the evaluation runner hard-coded s3 = null, so a generated
component was never checked against its reference render. After it, S3 is a real
visual-difference metric with an explicit status, so a mapping or render fault can
no longer read as a passing evaluation, while S1, S2, and variant coverage are
unchanged and gated independently.

Stacked on the variant-coverage work (T1) in #10; it consumes T1's expected-set,
marker, and verdict interfaces. Base this PR on fm/tokenloom-eval-coverage and
merge #10 first.

What changed

  • variantReferences maps each expected selector to its snapshot component and
    recorded renderPng by exact variantSelectorOf equality, cross-checking the
    importer's node-id filename rule.
  • scoreS3 is pure over PNG buffers and returns measured (worst-variant ratio,
    mean recorded), not-applicable (only when no expected variant has a reference),
    or error (mapping, decode, render, or DOM-disagreement fault). An error
    withholds the gated s3; a variant the output did not render is a coverage miss,
    not an S3 error.
  • measureS3 renders only when a reference exists, so a no-reference sample is
    not-applicable with no browser. A RenderPort is injected so tests use
    synthetic shots and CI never launches a browser. A DOM marker list that disagrees
    with the tokenizer fails both S3 and coverage.
  • Trajectory rows record s3Status through the same helper (not-applicable for
    the three current samples).
  • runs.ts and agent-session-port.ts gain optional s3Status and s3Detail.
  • spec.md 9.5 documents the S3 status semantics.

C2 (worst variant versus mean as the gated value) is open. The worst-variant
default is provisional; both values are recorded, so the choice is a one-line
change. No committed run row, report, or reference lock is modified.

How I checked

pnpm build
pnpm lint
pnpm typecheck
pnpm verify --gate scope,types,patterns,determinism
pnpm verify --gate tests
pnpm --filter @tokenloom/eval exec vitest run

All pass: the tests gate reports 1375 tests, 0 skipped, 0 failed (includes the
scoring report recomputation); the eval package runs 353 tests including the new
packages/eval/test/s3.test.ts (22) that cover every T3 acceptance item
(identical/different PNG, no-reference null, each error code, one failing variant
keeps partial ratios, selector matching, recall < 1 stays measured with overall
false, DOM disagreement on both, no browser, trajectory via the same helper).

Full integrated verification with benchmarks and selftest is deferred to the
serialized heavy-validation slot.

What I left alone

No weakened gate, no vitest -u, no rewritten committed run rows, reports, or
reference lock, no change to S1/S2 or coverage semantics, and no product package
outside packages/eval. This does not spend money, make a Figma call, or run a
paid model. The eval internal CLI contract gains optional row fields only.

Score variant coverage as a metric separate from S1 and S2. The expected
set comes from committed design context (base.props plus variants[].props),
narrowed to the requested selector for a variant-only request and identical
across input representations. Actual variants come from HTML [data-variant]
markers extracted by a start-tag tokenizer and normalized through the public
selectVariant grammar.

Report variantRecall, variantPrecision, duplicate counts, and missing or
unexpected diagnosis, with explicit measured/error/not-recorded states. A
coverage fail or error dominates unknown scores in the per-run verdict, so a
row with missing variants can never receive a fully passing evaluation, and
legacy run records without the metric stay readable and never claim a full
pass. S3 fields remain type-compatible without asserting visual scoring.

scoreS1, scoreS2, and countLiterals are unchanged; no parser or schema
source is touched. Adds thresholds variantRecall, variantPrecision, and
duplicateVariantsMax, locked once through the reference-lock tool.
Connect the existing visual helpers to the real runner so S3 is measured
instead of hard-coded null, while keeping S1, S2, and variant coverage
unchanged and independently gated.

variantReferences maps each expected selector to its snapshot component and
recorded renderPng by exact variantSelectorOf equality, cross-checking the
importer's node-id filename rule so a hand-edited path is caught. scoreS3 is
pure over PNG buffers and carries an explicit status: measured (worst-variant
ratio, mean recorded), not-applicable only when no expected variant has a
reference, or error for a mapping, decode, render, or DOM-disagreement fault.
An error withholds the gated s3, so a half-measured run can never read as a
pass, and a variant the output did not render is a coverage miss, not an S3
error. measureS3 renders only when a reference exists, so a no-reference
sample yields not-applicable without a browser; a RenderPort is injected so
tests use synthetic shots and CI never launches a browser. A DOM marker list
that disagrees with the tokenizer fails both S3 and coverage. Trajectory rows
record s3Status through the same helper.

C2 (worst variant versus mean as the gated value) is open; the worst-variant
default is provisional and both values are recorded, so the gated choice is a
one-line change. No committed row, report, or reference lock is modified.

spec.md 9.5 documents the S3 status semantics and the provisional aggregate.
@hsskey
hsskey changed the base branch from fm/tokenloom-eval-coverage to main September 17, 2026 13:14
# Conflicts:
#	docs/reference/spec.md
#	packages/eval/src/agent-session-port.ts
#	packages/eval/src/report.ts
#	packages/eval/src/runner.ts
#	packages/eval/src/runs.ts
#	packages/eval/src/stats.ts
#	packages/eval/src/trajectory.ts
#	packages/eval/src/variant-reference.ts
#	verify/selftest/scoring/numeric-literal/apply/packages/eval/src/report.ts
@hsskey
hsskey merged commit 638885f into main Sep 17, 2026
2 checks passed
@hsskey
hsskey deleted the fm/tokenloom-eval-visual branch September 17, 2026 23:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant