Skip to content

add stage-separated AMFV-Bench v0 eval - #19

Closed
VibeCodingScientist wants to merge 1 commit into
MedARC-AI:mainfrom
VibeCodingScientist:feat/amfv-bench-v0
Closed

add stage-separated AMFV-Bench v0 eval#19
VibeCodingScientist wants to merge 1 commit into
MedARC-AI:mainfrom
VibeCodingScientist:feat/amfv-bench-v0

Conversation

@VibeCodingScientist

Copy link
Copy Markdown

Summary

  • Adds AMFV-Bench v0: a frozen, stage-separated eval protocol (decomposer / retrieval / verifier) so later PRs can say retrieval A beats B instead of optimizing against vibes.
  • Ships 48 NICE-grounded gold cases (8 per stratum: strongly supported, weakly supported, refuted, insufficient, population mismatch, temporal), with Med-V1 Likert verdicts, scope Z, as-of date T, pyramid tier, and short quoted spans — not chunk hashes.
  • Includes a deterministic offline scorer (amfv-eval) that never calls a network or LLM, plus a clinician annotation rubric in datasets/amfv_datasets/eval/taxonomy.md.

This is the evaluation-first artifact from the 12 Aug restart. Population-mismatch cases must abstain (verdict 0); supporting them because a disease name matched is a fail. Temporal cases point at the replacement guideline with replaces set to the withdrawn id.

Test plan

  • uv sync --dev
  • uv run pytest datasets/test
  • uv run ruff check datasets && uv run ruff format --check datasets
  • uv run amfv-eval --predictions datasets/test/fixtures/eval/predictions_demo.jsonl (demo file supports a paediatric claim on purpose; scope_abstention should be 0)
  • Skim datasets/amfv_datasets/eval/taxonomy.md and 2–3 gold cases for clinical sense

Made with Cursor

Co-authored-by: Cursor <cursoragent@cursor.com>
@CLAassistant

CLAassistant commented Aug 13, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@VibeCodingScientist

Copy link
Copy Markdown
Author

Closing so I can review the branch locally before opening a PR.

@VibeCodingScientist

Copy link
Copy Markdown
Author

Superseded by #20 so the red-team scorer fixes and clinician-seed note land in one open PR. This closed PR can be ignored.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants