feat(eval): finalize C2 decisions - exact trajectory selector and max S3 gate - #12
Merged
Merged
Conversation
… S3 gate Decision 1: point the variant-only trajectory task at the exact selector `size=md,state=default` instead of the ambiguous `size=md`, which matched five state variants and let each condition build a different subset. Adding the task's `variant` field narrows the coverage and S3 expected set to one component, and the `data-variant` coverage rule now rides in the trajectory instructions as the coverage requirement's consequence. Decision 2: state in spec 9.5 that the gated S3 aggregate is the worst variant's ratio (max), with the mean kept only as a diagnostic. The scoring code already returns `max`; this removes the "provisional pending a maintainer decision" framing. Max matches the per-variant 0.05 tolerance, while the mean would loosen it by the variant count. Tests: pin the variant-only task at the exact selector and its single expected variant; assert the S3 max stays null over an empty comparison set so a Math.max() regression cannot pass the gate; update the verdict helper to replace the now-present variant line rather than inject a duplicate key.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The
variant-onlytrajectory task and the S3 per-run aggregate were both under-specified, so a mapping or render fault could read as a passing evaluation. This finalizes both eval-design decisions: input ambiguity is removed and output dilution is removed.What changed
variant-onlytask now targets the exact selectorsize=md,state=defaultinstead of the ambiguoussize=md, which matched five state variants ofChipand made each condition build a different subset. The task'svariantfield narrows the coverage and S3 expected set to one component, and thedata-variantcoverage rule is now stated in the trajectory instructions as the coverage requirement's consequence.spec.md9.5 now states that the gated S3 value is the worst variant's ratio (max), with the mean kept only as a diagnostic, replacing the "provisional pending a maintainer decision" framing. The scoring code already returnsmax, so no scoring change was needed.maxmatches the per-variant 0.05 tolerance; the mean would loosen it by the variant count.NO_COMPARABLE_VARIANT(never a pass), and a variant that did not render is already a coverage miss that failsvariantRecall, sooverallis false. Tests were added to pin both.How I checked
variant-onlytask resolves to exactly["size=md,state=default"].s3Detail.maxstaysnullover an empty set, so aMath.max()default of-Infinitycannot pass the 0.05 gate.What I left alone
No weakened gate, no
vitest -u, no rewritten committed evidence, no extra refactor. This changes the trajectory prompt text, so itspromptHashchanges and a new comparison set begins; committed run records keep their existing hash. No money spent (fake adapter only), no public CLI contract change.