Summary
GUIDANCE_RANKING_POLICY.closeScoreGap (introduced by #1615 / PR #1617) is the constant that decides
which guidance candidates are treated as statistically tied. No test pins it.
Measured by an independent evaluator on PR #1617's head 5a0e7303e: setting the constant to 5 —
ten times its shipped value of 0.5 — leaves all 8 guidance-evaluation fixture cases passing
and leaves the new close-score grouping test passing.
Why the existing test does not pin it
guidance close-score groups use stable documents without chaining or flattening same-page scores
includes an element named pages/gamma#outside-leader-band:
rankedSection('pages/beta', 'highest', 10.4),
rankedSection('pages/alpha', 'stronger', 10.2),
rankedSection('pages/alpha', 'weaker', 10.1),
rankedSection('pages/gamma', 'outside-leader-band', 9.8),gamma sorts last by slug whether or not it is inside the band, so the assertion produces the
same expected order under both. The element's name claims a boundary check the assertion does not
perform.
The fix
A case whose slug sorts early but whose score falls outside the band. Then band membership
changes the observable order, and widening or narrowing the constant fails the test:
rankedSection('pages/zulu', 'leader', 10.4),
rankedSection('pages/alpha', 'out-of-band', 9.8), // 0.6 below leader -> must NOT be regroupedWith closeScoreGap: 0.5, alpha stays after zulu (score order). With a widened gap, alpha
jumps ahead on slug. Assert the boundary from both sides — just inside (0.5) and just outside
(0.5 + epsilon) — so the constant cannot move undetected in either direction.
Why this is worth fixing even though nothing is broken
This is the defect class 0.0.6 has spent the milestone eliminating: a guard that cannot fire.
0.0.4 shipped two guards whose predicate could never be true and they looked correct while doing
nothing; #1554 shipped a codename pattern that was green on a tree containing the very strings it
was meant to catch; #1565 fixed a gate walking build output; #1531 fixed a corpus whose freshness
nothing checked. A tuned constant that no test constrains belongs on that list.
The behaviour today is correct. The risk is that a later change to closeScoreGap — widening it to
absorb a new flip, which is the obvious next move when this gate re-reds — passes CI green while
silently converting the scorer to near-alphabetical ordering across documents.
Related, and the reason a change here is likely
PR #1617's own body records that the fix converts an instability at gap ~= 0 into one at gap ~= 0.5,
and quantifies the remaining headroom on the real pair as 0.1980198018138779 — about 2.6x the
+0.0748587451731435 movement produced by one corpus regeneration. So a future regeneration of
similar magnitude can re-red the guidance gate with the fix in place, and widening the constant will
be the tempting response. That is exactly when a pinning test earns its keep.
Acceptance
Summary
GUIDANCE_RANKING_POLICY.closeScoreGap(introduced by #1615 / PR #1617) is the constant that decideswhich guidance candidates are treated as statistically tied. No test pins it.
Measured by an independent evaluator on PR #1617's head
5a0e7303e: setting the constant to5—ten times its shipped value of
0.5— leaves all 8 guidance-evaluation fixture cases passingand leaves the new close-score grouping test passing.
Why the existing test does not pin it
guidance close-score groups use stable documents without chaining or flattening same-page scoresincludes an element named
pages/gamma#outside-leader-band:gammasorts last by slug whether or not it is inside the band, so the assertion produces thesame expected order under both. The element's name claims a boundary check the assertion does not
perform.
The fix
A case whose slug sorts early but whose score falls outside the band. Then band membership
changes the observable order, and widening or narrowing the constant fails the test:
With
closeScoreGap: 0.5,alphastays afterzulu(score order). With a widened gap,alphajumps ahead on slug. Assert the boundary from both sides — just inside (
0.5) and just outside(
0.5 + epsilon) — so the constant cannot move undetected in either direction.Why this is worth fixing even though nothing is broken
This is the defect class 0.0.6 has spent the milestone eliminating: a guard that cannot fire.
0.0.4 shipped two guards whose predicate could never be true and they looked correct while doing
nothing; #1554 shipped a codename pattern that was green on a tree containing the very strings it
was meant to catch; #1565 fixed a gate walking build output; #1531 fixed a corpus whose freshness
nothing checked. A tuned constant that no test constrains belongs on that list.
The behaviour today is correct. The risk is that a later change to
closeScoreGap— widening it toabsorb a new flip, which is the obvious next move when this gate re-reds — passes CI green while
silently converting the scorer to near-alphabetical ordering across documents.
Related, and the reason a change here is likely
PR #1617's own body records that the fix converts an instability at gap ~= 0 into one at gap ~= 0.5,
and quantifies the remaining headroom on the real pair as
0.1980198018138779— about 2.6x the+0.0748587451731435movement produced by one corpus regeneration. So a future regeneration ofsimilar magnitude can re-red the guidance gate with the fix in place, and widening the constant will
be the tempting response. That is exactly when a pinning test earns its keep.
Acceptance
closeScoreGapis widened (e.g.0.5->5), with the raw non-zero exit shown.outside-leader-bandelement is either made to verify the boundary or renamed to state what it actually checks.