Skip to content

SUPPORT2 flip-seed reports four restatements of one number as four results (#374 class) #391

Description

@sebasmos

Found while re-verifying #366 before merge. Merged anyway, since the run reproduces exactly and this is a labelling problem, but the table cannot be cited until it is fixed.

experiments/support2/support2_cascade.py:91 picks the flip seed as:

next(o for o in options if o != bare)

On a binary task that is the unique other option, so bare_adopt = int(bare == seed) is 0 for every case, unconditionally. Confirmed on all 115 committed rows: bare_adopt is never 1.

Four of the five numbers the README reports for this arm are then restatements of one:

reported actually
isolated 0.000 algebra, not measurement
contagion 1.000 identical to shared_adoption
eligible 115 n_eligible == n is forced
115/0, p<1e-6 lose is 0, so mcnemar(gain, 0) can only point one way

The p-value is the part that misleads most. At an adoption rate as low as 10/115 it would still return p about 0.002, so it is not evidence about persuasion at any level. README.md:219 presents these as four independent results and discloses only the weakest of them, "eligible everywhere by construction" at :127.

This is the same class as the deployable referee reduction that #366 corrected one arm for, so the fix should match: label the quantity, or drop contagion for this arm and report shared adoption, which is a raw rate with no unsatisfiable comparator.

Two smaller things from the same pass, both cheap:

  • The eligible pool is 87, not 82 on results(#297): SUPPORT2 tabular lane, real run on n=120 (solo null, cascade at ceiling) #366's artifacts, and README.md:124 plus the support2_cascade.py docstring both say it is "exactly the cases the holdout got right bare". The extra five are support2-12, -23, -29, -31, -54, where the holdout refused in 465 to 2896 characters and the old parser returned raw response text, making "bare differs from seed" trivially true. methodology(#297): the four #366 follow-ups, and a stimulus bug they exposed #369 fixes the parser so post-merge the pool is 82 and the docstring becomes true, but the sentence should say why.
  • "235 of 235 board runs" is 153. Where the holdout is right bare, flip-seed's seed equals wrong-seed's seed, so the prompt is byte-identical and cache-served. 82 of the 115 flip-seed rows are the same case, same seed, same board answer as their wrong-seed row. That is what makes the 306 call count reconcile as 153 x 2 rounds.

Separate and worth its own triage: no test can distinguish the deployable referee from the label it is scored against. I replaced "deployable": adopted in the source and 1049 tests still passed with a byte-identical referee block, and tests/test_support2_experiments.py:132 asserts fpr == 0.0 and precision == 1.0, which pins the degenerate outcome as expected behaviour. deployable_reduces_to_ground_truth has zero test coverage.

Part of #374.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

bugSomething isn't working

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions