-
Notifications
You must be signed in to change notification settings - Fork 0
Attributed-tier identity of the seed (MedMCQA) #272
Copy link
Copy link
Closed
Labels
dataset:medmcqadifficulty: beginnerSelf-contained, no deep context neededSelf-contained, no deep context neededexperimentExperiment runner / study designExperiment runner / study designpriority: mediumValuable, not on the critical pathValuable, not on the critical path
Description
Activity
Metadata
Metadata
Assignees
Labels
dataset:medmcqadifficulty: beginnerSelf-contained, no deep context neededSelf-contained, no deep context neededexperimentExperiment runner / study designExperiment runner / study designpriority: mediumValuable, not on the critical pathValuable, not on the critical path
Replicate the Attributed-tier identity of the seed (MedMCQA) experiment on MedMCQA, mirroring the MedQA template so results are apples-to-apples across datasets.
experiments/medqa/attributed_tier.pybenchmaxxing/datasets/medmcqa.py(already implemented, Dataset adapter: MedMCQA (text/MCQ) #112)What it measures
Reputation/status cue: does labeling the seed as a senior/expert peer raise adoption.
What to do
Run the same experiment on MedMCQA through the existing pipeline and the Gemini API. Replication, not new features: do not change the experiment code, only point it at a MedMCQA manifest (build one with the
medmcqaadapter). Write results underexperiments/medmcqa/results/in the same JSON/JSONL convention asexperiments/medqa/results/(not CSV, so the analysis tooling reads them).Acceptance
experiments/medmcqa/in the MedQA template's format.Part of the MedMCQA standard-battery replication (mirrors the MedQA/NIH template).