Skip to content

add stage-separated AMFV-Bench v0 eval - #20

Open
VibeCodingScientist wants to merge 2 commits into
MedARC-AI:mainfrom
VibeCodingScientist:feat/amfv-bench-v0
Open

add stage-separated AMFV-Bench v0 eval#20
VibeCodingScientist wants to merge 2 commits into
MedARC-AI:mainfrom
VibeCodingScientist:feat/amfv-bench-v0

Conversation

@VibeCodingScientist

Copy link
Copy Markdown

Summary

  • Frozen AMFV-Bench v0: 48 NICE-grounded cases, stage-separated decomposer / retrieval / verifier metrics, and a deterministic amfv-eval scorer (no network, no LLM).
  • Scorer rejects polarity and identifier swaps, counts omitted verdicts and skipped population-mismatch cases as misses, and does not reward decoy retrieval on insufficient cases.
  • Gold Likert, scope, and evidence spans are clinician work. Rubric: datasets/amfv_datasets/eval/taxonomy.md.

v0 is a seed (seed-v0-clinician), not locked gold. If this lands I will (1) lock the 48 labels under a named annotator id after a full clinical pass, (2) independently re-grade or recruit a second clinician and adjudicate disagreements per taxonomy.md, and (3) stay on as an annotator for later splits (high-stakes traps, not silent growth of v0). Gold Likert / scope / evidence is clinician work; please do not mint verdicts from an LLM judge.

Replaces closed #19 so the red-team scorer fixes ship in one reviewable PR.

Test plan

  • uv sync --dev
  • uv run pytest datasets/test
  • uv run ruff check datasets && uv run ruff format --check datasets
  • uv run amfv-eval --predictions datasets/test/fixtures/eval/predictions_demo.jsonl (demo file supports a paediatric claim on purpose; scope_abstention should be 0)
  • Skim datasets/amfv_datasets/eval/taxonomy.md and 2–3 gold cases for clinical sense

Made with Cursor

VibeCodingScientist and others added 2 commits August 13, 2026 16:45
Co-authored-by: Cursor <cursoragent@cursor.com>
Close claim-matching and metric-gaming holes found in review, and make gold Likert labels explicitly clinician work rather than LLM-as-judge.

Co-authored-by: Cursor <cursoragent@cursor.com>
@CLAassistant

CLAassistant commented Aug 13, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants