Skip to content

feat(eval): add deterministic variant coverage metric - #10

Merged
hsskey merged 2 commits into
mainfrom
fm/tokenloom-eval-coverage
Sep 17, 2026
Merged

hsskey merged 2 commits into
mainfrom
fm/tokenloom-eval-coverage

Conversation

@hsskey

@hsskey hsskey commented Sep 17, 2026

Copy link
Copy Markdown
Owner

Why

Before this change, a generated component that renders only one of many required variants still scored S1 = 1 and S2 = 1, and no metric could observe the missing variants. This adds a deterministic variant coverage metric, so a response that omits variants can no longer receive a fully passing evaluation.

What changed

Adds variant coverage as a metric separate from S1 and S2 (docs/reference/spec.md 9.5):

  • Expected variants come from committed design context (base.props plus variants[].props), narrowed to the requested selector for a variant-only request and identical across input representations.
  • Actual variants come from HTML [data-variant] markers, extracted by a start-tag tokenizer and normalized through the public selectVariant grammar.
  • Reports variantRecall, variantPrecision, duplicate counts, and missing/unexpected diagnosis, with explicit measured/error/not-recorded states.
  • A coverage fail or error dominates unknown scores in the per-run verdict, so a coverage-failing row is overall: false even before S3 is wired, and legacy run records without the metric stay readable and never claim a full pass.
  • Adds thresholds variantRecall, variantPrecision, and duplicateVariantsMax, locked once through the reference-lock tool.

scoreS1, scoreS2, and countLiterals are unchanged; no parser or schema source is touched.

How I checked

pnpm exec vitest run packages/eval packages/verify scripts   # 640 passed
pnpm tsx scripts/reference-lock.ts --check                   # clean
pnpm verify                                                  # 13/13 gates pass

Full pnpm verify passed all 13 gates, including scoring, benchmarks, and selftest. The selftest gate applied 27 intentional failure samples with 0 failures, among them two scoring samples that exercise the report generator. New adversarial scoring tests (A16-A24) cover a 1-of-20 response that scores S1 = 1 and S2 = 1 but fails coverage, duplicate and junk markers, reordered keys and whitespace normalization, and the variant-only subset. Marker extraction is checked against DOM-equivalent expectations in html-markers.test.ts.

What I left alone

No gate weakened, no vitest -u, no rewritten committed evidence, no extra refactor. Committed run records, reports, and the benchmark ledger are unchanged. No provider or Figma calls were made. The public CLI contract is unchanged; this adds optional report fields and three thresholds. S3 fields stay type-compatible and unpopulated pending later work.

Score variant coverage as a metric separate from S1 and S2. The expected
set comes from committed design context (base.props plus variants[].props),
narrowed to the requested selector for a variant-only request and identical
across input representations. Actual variants come from HTML [data-variant]
markers extracted by a start-tag tokenizer and normalized through the public
selectVariant grammar.

Report variantRecall, variantPrecision, duplicate counts, and missing or
unexpected diagnosis, with explicit measured/error/not-recorded states. A
coverage fail or error dominates unknown scores in the per-run verdict, so a
row with missing variants can never receive a fully passing evaluation, and
legacy run records without the metric stay readable and never claim a full
pass. S3 fields remain type-compatible without asserting visual scoring.

scoreS1, scoreS2, and countLiterals are unchanged; no parser or schema
source is touched. Adds thresholds variantRecall, variantPrecision, and
duplicateVariantsMax, locked once through the reference-lock tool.
…erage

# Conflicts:
#	docs/reference/spec.md
#	packages/eval/src/agent-session-port.ts
#	packages/eval/src/report.ts
#	packages/eval/src/runner.ts
#	packages/eval/src/stats.ts
#	packages/eval/src/trajectory.ts
#	verify/selftest/scoring/numeric-literal/apply/packages/eval/src/report.ts
@hsskey
hsskey merged commit 3a1de97 into main Sep 17, 2026
2 checks passed
@hsskey
hsskey deleted the fm/tokenloom-eval-coverage branch September 17, 2026 23:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant