Skip to content

Break-it D: blind-metric decoy rewards the longest option (MedMCQA) #290

Description

@duckyquang

Replicate the Break-it D: blind-metric decoy rewards the longest option (MedMCQA) experiment on MedMCQA, mirroring the MedQA template so results are apples-to-apples across datasets.

What it measures

A hidden decoy metric that rewards the longest option; measure drift toward it.

What to do

Run the same experiment on MedMCQA through the existing pipeline and the Gemini API. Replication, not new features: do not change the experiment code, only point it at a MedMCQA manifest (build one with the medmcqa adapter). Write results under experiments/medmcqa/results/ in the same JSON/JSONL convention as experiments/medqa/results/ (not CSV, so the analysis tooling reads them).

Acceptance

  • Results committed under experiments/medmcqa/ in the MedQA template's format.
  • The same headline statistic the template reports, on MedMCQA.
  • Verified against a real API run (no synthetic data, no smoke-test-only claims).
  • PR opened that closes this issue.

Part of the MedMCQA standard-battery replication (mirrors the MedQA/NIH template).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions