feat(eval): add deterministic variant coverage metric - #10
Merged
Merged
Conversation
Score variant coverage as a metric separate from S1 and S2. The expected set comes from committed design context (base.props plus variants[].props), narrowed to the requested selector for a variant-only request and identical across input representations. Actual variants come from HTML [data-variant] markers extracted by a start-tag tokenizer and normalized through the public selectVariant grammar. Report variantRecall, variantPrecision, duplicate counts, and missing or unexpected diagnosis, with explicit measured/error/not-recorded states. A coverage fail or error dominates unknown scores in the per-run verdict, so a row with missing variants can never receive a fully passing evaluation, and legacy run records without the metric stay readable and never claim a full pass. S3 fields remain type-compatible without asserting visual scoring. scoreS1, scoreS2, and countLiterals are unchanged; no parser or schema source is touched. Adds thresholds variantRecall, variantPrecision, and duplicateVariantsMax, locked once through the reference-lock tool.
…erage # Conflicts: # docs/reference/spec.md # packages/eval/src/agent-session-port.ts # packages/eval/src/report.ts # packages/eval/src/runner.ts # packages/eval/src/stats.ts # packages/eval/src/trajectory.ts # verify/selftest/scoring/numeric-literal/apply/packages/eval/src/report.ts
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Before this change, a generated component that renders only one of many required variants still scored S1 = 1 and S2 = 1, and no metric could observe the missing variants. This adds a deterministic variant coverage metric, so a response that omits variants can no longer receive a fully passing evaluation.
What changed
Adds variant coverage as a metric separate from S1 and S2 (
docs/reference/spec.md9.5):base.propsplusvariants[].props), narrowed to the requested selector for a variant-only request and identical across input representations.[data-variant]markers, extracted by a start-tag tokenizer and normalized through the publicselectVariantgrammar.variantRecall,variantPrecision, duplicate counts, and missing/unexpected diagnosis, with explicitmeasured/error/not-recordedstates.failorerrordominates unknown scores in the per-run verdict, so a coverage-failing row isoverall: falseeven before S3 is wired, and legacy run records without the metric stay readable and never claim a full pass.variantRecall,variantPrecision, andduplicateVariantsMax, locked once through the reference-lock tool.scoreS1,scoreS2, andcountLiteralsare unchanged; no parser or schema source is touched.How I checked
Full
pnpm verifypassed all 13 gates, includingscoring,benchmarks, andselftest. The selftest gate applied 27 intentional failure samples with 0 failures, among them two scoring samples that exercise the report generator. New adversarial scoring tests (A16-A24) cover a 1-of-20 response that scores S1 = 1 and S2 = 1 but fails coverage, duplicate and junk markers, reordered keys and whitespace normalization, and the variant-only subset. Marker extraction is checked against DOM-equivalent expectations inhtml-markers.test.ts.What I left alone
No gate weakened, no
vitest -u, no rewritten committed evidence, no extra refactor. Committed run records, reports, and the benchmark ledger are unchanged. No provider or Figma calls were made. The public CLI contract is unchanged; this adds optional report fields and three thresholds. S3 fields stay type-compatible and unpopulated pending later work.