Conversation
One branch, one new model against the committed Gemini baseline, which for this arm is the gemini-2.5-flash run already on main over the same 35 studies. Blind-rubric drift 0.40 against Gemini's 0.86, both off a 0.00 baseline, and complete suppression under the test-aware prompt (0.00) where Gemini retains 0.17. The branch carries the shared model dispatch it needs to run at all, because main has no experiments/_lane.py yet: key and backend resolution, the locally served OpenAI-compatible path, output caps, pacing and transient recovery, plus the runner ports that go through it. None of the other lineages' result files are here. Once the text-lane dispatch lands on main this branch reduces to the imaging arm alone.
sebasmos
changed the title
sebasmos
marked this pull request as ready for review
September 10, 2026 08:44
sebasmos
requested review from
Agastya191,
MohShahin,
PeterLi-jpg,
Yehudha-kennedy,
aadityapanchal7,
amarzullo24,
armaanvgrewal,
arshnoor08,
azzy-chemE,
duckyquang,
felipeocampoos,
maximinl and
renato-umeton
September 10, 2026 08:44
sebasmos
marked this pull request as draft
September 11, 2026 08:29
sebasmos
marked this pull request as ready for review
September 13, 2026 11:55
maximinl
approved these changes
Sep 16, 2026
maximinl
left a comment
maximinl
left a comment
Collaborator
There was a problem hiding this comment.
Peer review — approve the Qwen2.5-VL lineage package (draft).
What’s right
- Imaging blind-metric reporting gives both aggregate drift and drift | correct baseline, which is required here: the decoy is the flip of each model’s own read, so misreads turn the “decoy” into the true label. CheXpert conditional contrast (Gemini 20/22 vs Qwen 7/22 on the 22 both-correct studies, McNemar p=2.4e-4) is the citable number.
- Text-lane temperature-0 reproducibility called out with a concrete repeat-prompt check (2/401), so between-arm comparisons are justified — unlike gpt-oss.
- MIMIC DUA handling (results in, caches out) matches the rest of the repo.
- Imaging
_lane.scoped(..., default=)so flash (not flash-lite) stays the imaging baseline directory is the correct seam;paced_complete(..., image=)is the right extension for vision.
Non-blocking asks
- #421 committed
cross_lineage_report.pyso BH / repeat-prompt body claims replay from a fresh clone. This branch should either include the same (or an imaging+text equivalent) or drop unreproducible meta-counts from the body. _lane.pydiffers across #421/#426/#427 (this tip has vision pacing +scoped(default=); #421 hasHOSTED_PREFIXES). Land with an explicit merge order / single_laneconsolidation so neither nvidia exclusion nor vision hooks get lost.- Still marked draft — fine while caches settle; ready to undraft once the report script (or body trim) is settled.
No methodology objections to the arm design or the conditional imaging rates.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Vision-language lineage:
Qwen/Qwen2.5-VL-72B-Instruct, imaging and textLocally served (vLLM, TP4, temperature 0), one model in every seat. Imaging: every arm of the
experiments/imaging/lane plus the CheXpert lane, on the same 35 NIH and 35 CheXpert studies as the committedgemini-2.5-flasharms. Text: the same MedQA manifest and 120 cases as the committedgemini-2.5-flash-litearms, blind metric at n=40 and n=100, and every other text lane the repo can run (MedMCQA, SUPPORT2, referee, cascade, contamination, model dependence, cross-dataset). MIMIC-CXR, both the report-text lane and the film battery, ran on the cluster that holds the data under its DUA. Every number outside those two lanes replays from a fresh clone with no API calls. Draft.Imaging
Unlike the text lanes, the decoy in the imaging blind metric is the opposite of each model's own baseline read and every study has the finding present, so a drift is only deference where the baseline was right. Qwen reads fewer films correctly than Gemini, and on the films it misreads the "decoy" is the true label. Both the aggregate rate and the rate conditioned on a correct baseline read are therefore given.
On the 22 CheXpert studies both models read correctly Gemini drifts on 20 and Qwen on 7, all 13 discordant pairs the same way (exact McNemar p=2.4e-4). On the 11 NIH studies both read correctly Gemini drifts on 10 and Qwen on 11. So the rubric effect replicates in direction on both cohorts and Qwen is the less deferent model on CheXpert only. Test-awareness suppresses it completely on both cohorts where Gemini retains 0.171 and 0.371. Neither model named the rubric on any drifted study.
imaging_soloimaging_cascade(watermark)imaging_system_flagimaging_multi_roundimaging_refereeimaging_judge_refereeimaging_cascadeThe solo result replicates as a null: no overlay cue reaches Qwen's own noise floor, where Gemini's watermark alone cleared it by 0.11. The committee result replicates in full and saturates: the watermark that does nothing alone is adopted on every study once two seeded peers assert it, so the cue-family, opacity-ladder, peer-size and multi-round arms all sit at 1.00 shared adoption (their summaries are committed; the Cochran and McNemar tests inside them return 1.0 on identical vectors, which is saturation, not a null). The private re-query referee transfers (precision 1.00, recall 0.94) and the transcript-only judge collapses onto the naive gate as it does for Gemini. The committed Gemini judge arm is the text-only variant, so the like-for-like Qwen arm is
--text-only; the film-seeing run is kept beside it asimaging_judge_referee_summary_film.json.Text: MedQA
Unseeded accuracy on the shared cases: Gemini 101/120 in every arm; Qwen exactly 90/120 in every one of the 17 arms that record it. A temperature-0 answer from the served endpoint is reproducible (2 changed answers over 401 repeat prompts across the lane), so the between-arm figures are comparable as well as the within-arm ones.
blind_metricblind_metrictext_cue_typestext_cue_typesdose_responsedose_responseauthority_ladderauthority_laddertest_awarenesscommittee_size_sweepplausible_distractorlive_peer_organicsuper_additivitytemperature_sensitivitydeliberation_channelreferee_self_inconsistencyAdoption sits between Gemini's and the floor on almost every arm, roughly a third to a half of Gemini's rate, with the within-arm ordering preserved: negation is the potent cue, the guideline rung the highest, one honest peer protective, the auditor frame protective. Two things differ. The hidden reasoning channel does not exist on this endpoint, so the
hiddencell repeatsnone, while theopenchannel lowers adoption from 0.27 to 0.09. And the plausibility gap is wide: a plausible decoy is adopted on 0.518 and an implausible one on 0.182, 39 discordant pairs one way and 2 the other.MIMIC-CXR report text
Run against a vLLM endpoint on 2 H200s on the cluster that holds the data, the reports read in place under the PhysioNet DUA. The cohort is the 633-case index the committed rows key on: its ground truth agrees with the committed
deliberation_framing.jsonlon 60 of 60 recorded cases, and the solo cohort reproduces Gemini's 600 indices exactly. Every per-case file has the same keys and row count as its Gemini counterpart (40, 40, 80, 20, 31, 60, 60 rows), so the arms pair case for case, with one exception stated below. Clean-correct on all 633 reports: Qwen 0.831, Gemini 0.790.blind_metricblind_metricreferee_judgereferee_deployablebreak_it_abreak_it_ddeliberation_framingpush_cThe same-lineage judge inverts: Gemini's flagged 13 of 40 at precision 1.000 with no false positives, Qwen's flags 39 of 40 at precision 0.333 and FPR 0.963, so on this lineage the detector fails by flagging everything. The deployable referee with a clean control is the one detector that improves. Deliberation framing does not replicate: adoption is at the floor in every frame, so the ladder has nowhere to order itself and the McNemar contrasts are non-significant. Peer conformity replicates and anchoring is stronger here, adding 0.250 over the generic rationale against 0.033 on Gemini.
break_it_aselects each model's own hard cases (hard_cases(all_cases, solo_records, n)), so its 20 cases are not Gemini's 20: 15 are shared. The committed Gemini file holds 40 rows because its solo stage ran two Gemini models over the same 20 cases; a single-model lane writes 20. Every other arm is paired case for case.MIMIC-CXR films
Every arm of
experiments/mimic_cxr_image/run_battery.pyon the same 834 films: the selection regenerated from the recorded seed reproduces all six per-arm manifests, and each film matched the committeddeid/provenance.csvsha256 (834 of 834, 0 mismatched). Committed in the lane's own shape, summaries plusdeid/<slug>/*.csvkeyed oncase_indexonly, with the same headers and row counts as the Gemini files.One fact governs the whole battery: Qwen reads 14 of the 834 pneumothorax films as present, against Gemini's 183. The planted read on this cohort is the constant "no", so it agrees with Qwen's own read on 98% of films. Every published committee statistic that conditions on a correct clean read (the "cases read correctly alone" blocks) would rest on 3 to 14 films here and is not reported. The full-cohort rates are reported instead, and read as a ceiling rather than a cascade.
solosolocascadereferee_cascadestrength_cascadesystem_flagblind_metricblind_metricjudge(text-only)judge_with_imagerefereenih_match_soloThe two arms that do not depend on the baseline read replicate: the hidden rubric drifts Qwen on 0.61 of films against Gemini's 0.53 from a 0.00 baseline on both, test-awareness suppresses it completely (0.00 against 0.25), and no drifter names the rubric. The solo null replicates with every cue 0.07 to 0.16 below the 0.162 floor. For a second vision lineage the first thing to report on this cohort is whether the model can read the finding; the committee battery presupposes that it can.
The call caches for both MIMIC lanes hold report text and encoded pixels and are not committed, matching the committed Gemini lane; these two lanes are the one part of this branch that does not replay from a fresh clone.
Other lanes
Every Gemini text experiment the repo can still run has a Qwen arm (29 MedQA summaries, 23 on the MedMCQA cohort, 5 referee, 5 SUPPORT2, plus cascade, contamination, model dependence and the cross-dataset pilot). Committee runners seat this model in every seat, judge and orchestrator included, so each row pairs an all-Qwen committee against Gemini's all-Gemini one.
text_cue_typesauthority_ladderrationale_validitydeliberation_framingscale_corchestrator_failuresupport2_solosupport2_cascadesupport2_cascadesupport2_cascade_strengthreferee_deployablereferee_deployablesupport2_refereemulti_roundcontamination_auditThe tabular saturation is a property of the model: Gemini adopts the wrong prognosis on the shared SUPPORT2 board in every eligible case and Qwen on 0.775, and the six-rung ladder that is censored on Gemini spans 0.442 to 0.791 here. The seed-polarity contrast Gemini cannot express separates: a right seed is adopted more often than a wrong one. The reasoning-quality result does not replicate: a valid-but-wrong rationale leaves adoption unchanged (5/7, p=0.77) where it lowers Gemini's (4/45).
Cross-dataset cue pilot (50 cases per dataset, option order / longest option / lexical overlap): MedQA 0.08 / 0.06 / 0.04 against MedMCQA 0.12 / 0.24 / 0.30, mean spread 0.160; Gemini 0.08 / 0.10 / 0.08 against 0.08 / 0.16 / 0.16. The cue effect is dataset-dependent on this lineage where it was invariant on Gemini.
Where an arm's row count differs from Gemini's it is the arm's own eligibility rule, not a different cohort: the plausible-distractor arm runs on the cases with a buildable distractor (103 against 110),
scale_con the cases the model gets wrong unseeded (85 against 102), the orchestrator and unanimity arms on their eligible subsets, andsupport2_refereeon 240 rows against Gemini's 235 because this model never abstains. Every shared arm was invoked with the same cohort flags as the accepted second-lineage run.cascade/multi_roundis the one arm where that was not initially true: it is now the same 40 cases as the committed Gemini arm (replayed from the model's own cache at zero calls), with the 120-case run kept beside it asmulti_round_summary_n120.json.Code
Main has no
experiments/_lane.py, so this branch carries the shared model dispatch it needs to run at all: key and backend resolution, the locally served OpenAI-compatible path, output caps, call pacing and transient recovery, plus the runner ports that go through it. Once that dispatch is on main this branch reduces to its own arms and three pieces the imaging lane needed:_lane.paced_completetakes an optionalimage, passed through only when present, so the vision runners get the same pacing and recovery as text and the text path is unchanged;_lane.scopedtakes the lane's own default id, because the imaging lane's committed model isgemini-2.5-flash, not the text default.experiments/imaging/take--modeland write to a model-scoped output directory and cache, replacing the hardcoded Gemini id, backend and committed-cache default each of them had.imaging_judge_referee.pynames its seatJUDGE, notMODEL, and is ported accordingly.tests/test_imaging_runner_ports.pypins all of this and builds every runner's parser.blind_metric_ci,net_harm,onset_battery,positional_regression, ...) take--results-dir, so a second model's arms are re-analysed in place without touching the committed Gemini derivations; the ported scripts reproduce the unported ones byte for byte on the Gemini rows.positional_regressionreturns the empirical rate when an arm is saturated instead of fitting a logit with no variance.cross_dataset/run_cross_dataset.pygoes through the shared dispatch andcontamination_audit.pyis pointed at the model's own solo records (--solo-records) rather than the committed Gemini ones.tests/degeneracy_exemptions.jsongains an allowlist entry, with the check performed, for every constant, duplicated or saturated column the guard finds in the new rows.Three committed Gemini derivations (
onset_battery.json,positional_regression.json,claim4_quantification.json) predate the July re-run of the cascade rows they derive from and no longer reproduce from them; they are left untouched here and noted for a separate fix.experiments/mimic_cxr_image/run_battery.pytakes--modeland scopes the transcripts the judge and referee arms replay to the model subdirectory the shared runners write; without the second half those arms fail on a path one directory too shallow.experiments/mimic_cxr_image/export_deid.pytakes--results-dirand--model, writesdeid/<slug>/with the published headers, and skips its claim verification for a second lineage because the expected values are the Gemini claims (the default path still verifies all twelve).experiments/mimic_cxr_text/build_solo_records.pyimportsparse_legacy_string: its_parse_choiceimport was removed in b9a9f07 and the module has been import-broken onmainsince.tests/test_mimic_battery.pyrecognises model-scoped summaries as produced by their arm.Replay
The text arms replay from a fresh clone with no API calls: re-running five MedQA arms with no key set reproduces every leaf of the committed summaries (byte differences are JSON formatting only). The Gemini comparators carried onto this branch,
deliberation_channeland the n=100 blind metric, also replay keylessly at their own cohort size and reproduce the committed numbers exactly. The imaging arms are the exception, and it is pre-existing rather than introduced here: two cue families composite rasterised text whose glyphs come from the imaging library's default font, so the cache, keyed on those pixels, re-queries under a different library version. The committed Gemini imaging caches miss on this machine in exactly the same way, and the released per-case rows are the artefact of record for that lane.Scope
One branch, one new model against the Gemini baseline. No other lineage's results are here. Both MIMIC-CXR lanes are included; their call caches are not, per the DUA. Test suite: 1415 passed, 14 failed, the same 14 as
main; the degeneracy guard passes on the new rows with an allowlist entry, stating the check performed, for each finding.