-
Notifications
You must be signed in to change notification settings - Fork 0
Majority-pressure (Asch) variant (MedMCQA) #275
Copy link
Copy link
Closed
Labels
dataset:medmcqadifficulty: intermediateTouches one subsystem; some context neededTouches one subsystem; some context neededexperimentExperiment runner / study designExperiment runner / study designpriority: highDo this soon; unblocks the paper or other workDo this soon; unblocks the paper or other work
Description
Activity
Metadata
Metadata
Assignees
Labels
dataset:medmcqadifficulty: intermediateTouches one subsystem; some context neededTouches one subsystem; some context neededexperimentExperiment runner / study designExperiment runner / study designpriority: highDo this soon; unblocks the paper or other workDo this soon; unblocks the paper or other work
Replicate the Majority-pressure (Asch) variant (MedMCQA) experiment on MedMCQA, mirroring the MedQA template so results are apples-to-apples across datasets.
experiments/medqa/majority_pressure.pybenchmaxxing/datasets/medmcqa.py(already implemented, Dataset adapter: MedMCQA (text/MCQ) #112)What it measures
Seed 2 of 3 agents with the wrong answer and measure conformity of the third.
What to do
Run the same experiment on MedMCQA through the existing pipeline and the Gemini API. Replication, not new features: do not change the experiment code, only point it at a MedMCQA manifest (build one with the
medmcqaadapter). Write results underexperiments/medmcqa/results/in the same JSON/JSONL convention asexperiments/medqa/results/(not CSV, so the analysis tooling reads them).Acceptance
experiments/medmcqa/in the MedQA template's format.Part of the MedMCQA standard-battery replication (mirrors the MedQA/NIH template).