Skip to content

feat(eval): finalize C2 decisions - exact trajectory selector and max S3 gate - #12

Merged
hsskey merged 1 commit into
mainfrom
fm/tokenloom-eval-c2-impl
Sep 17, 2026
Merged

hsskey merged 1 commit into
mainfrom
fm/tokenloom-eval-c2-impl

Conversation

@hsskey

@hsskey hsskey commented Sep 17, 2026

Copy link
Copy Markdown
Owner

Why

The variant-only trajectory task and the S3 per-run aggregate were both under-specified, so a mapping or render fault could read as a passing evaluation. This finalizes both eval-design decisions: input ambiguity is removed and output dilution is removed.

What changed

  • Trajectory selector (Decision 1): the variant-only task now targets the exact selector size=md,state=default instead of the ambiguous size=md, which matched five state variants of Chip and made each condition build a different subset. The task's variant field narrows the coverage and S3 expected set to one component, and the data-variant coverage rule is now stated in the trajectory instructions as the coverage requirement's consequence.
  • S3 aggregation (Decision 2): spec.md 9.5 now states that the gated S3 value is the worst variant's ratio (max), with the mean kept only as a diagnostic, replacing the "provisional pending a maintainer decision" framing. The scoring code already returns max, so no scoring change was needed. max matches the per-variant 0.05 tolerance; the mean would loosen it by the variant count.
  • Correctness confirmation (no code change): an empty comparison set already returns NO_COMPARABLE_VARIANT (never a pass), and a variant that did not render is already a coverage miss that fails variantRecall, so overall is false. Tests were added to pin both.

How I checked

pnpm --filter @tokenloom/eval test -- s3 trajectory coverage verdict   # 380 passed
pnpm build && pnpm typecheck && pnpm lint                              # all OK
pnpm verify                                                           # all gates PASS (types..selftest)
  • Added test: the variant-only task resolves to exactly ["size=md,state=default"].
  • Added assertion: s3Detail.max stays null over an empty set, so a Math.max() default of -Infinity cannot pass the 0.05 gate.

What I left alone

No weakened gate, no vitest -u, no rewritten committed evidence, no extra refactor. This changes the trajectory prompt text, so its promptHash changes and a new comparison set begins; committed run records keep their existing hash. No money spent (fake adapter only), no public CLI contract change.

… S3 gate

Decision 1: point the variant-only trajectory task at the exact selector
`size=md,state=default` instead of the ambiguous `size=md`, which matched
five state variants and let each condition build a different subset. Adding
the task's `variant` field narrows the coverage and S3 expected set to one
component, and the `data-variant` coverage rule now rides in the trajectory
instructions as the coverage requirement's consequence.

Decision 2: state in spec 9.5 that the gated S3 aggregate is the worst
variant's ratio (max), with the mean kept only as a diagnostic. The scoring
code already returns `max`; this removes the "provisional pending a maintainer
decision" framing. Max matches the per-variant 0.05 tolerance, while the mean
would loosen it by the variant count.

Tests: pin the variant-only task at the exact selector and its single expected
variant; assert the S3 max stays null over an empty comparison set so a
Math.max() regression cannot pass the gate; update the verdict helper to
replace the now-present variant line rather than inject a duplicate key.
@hsskey
hsskey merged commit 3232784 into main Sep 17, 2026
2 checks passed
@hsskey
hsskey deleted the fm/tokenloom-eval-c2-impl branch September 17, 2026 23:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant