Add MAGIC scraper

[MrKoga]

What this does

Adds a MAGICapp scraper to the datasets scraping pipeline. Same shape as the NICE scraper — everything outputs a normalized ScrapedDocument, every network-touching function takes client: httpx.Client first, and the CLI gains a magic source.

MAGICapp is the GRADE-methodology publishing platform: 45 organizations publish through it — WHO, Stroke Foundation of Australia, Cancer Council Australia, the European Stroke Organisation, BMJ Rapid Recommendations and others — each with its own formats and nuances. The full catalogue holds 461 guidelines; after dropping non-English entries and training accounts, 233 remain, of which 225 render into the corpus.

Meditron lists MAGIC as a source; its scraper drives the single-page app with Selenium and is marked UNTESTED in the source. MAGICapp has a public unauthenticated JSON API, so this replaces that approach rather than porting it.

How discovery works

Unlike the HTML sources there is no listing page to parse: one catalogue request returns every published guideline with its metadata, and each entry links the full guideline as structured JSON — a section tree with recommendation and PICO objects. Eight catalogue entries are excluded as the result of the publisher's own words: consultation drafts, a demonstration guideline whose text disclaims clinical use, archived modules, and sub-400-character placeholders.

How extraction works

The guideline JSON renders to one markdown document per guideline. Recommendation objects render as headings carrying the publisher's own label plus GRADE strength and certainty, with the panel's rationale and evidence-to-decision judgments as labelled blocks beneath; evidence questions render with their population/intervention/comparator and one line per outcome (effect measure, 95% CI, participants, studies, certainty) — numbers that exist only as data fields and would otherwise be lost. Section bodies are HTML fragments and go through the shared html_to_markdown helper.

Because the corpus feeds a claim verifier, the test for every span is whether a medical claim could rest on it.

• Kept spans include: recommendations with strength and certainty, effect estimates, clinical and case definitions, thresholds and doses, findings of absence ("no RCTs were identified" is a finding), and the epidemiology publishers state in introductions.
• Dropped spans include: authorship and rosters, funding and conflicts of interest, process narrative, navigation and link directories, citations and bibliographies, changelogs, abbreviation-expansion tables, and label legends. No personal researcher data ships: zero personal-domain email addresses, no contributor rosters, no personal postal addresses.

Section-level drops are decided by heading rules backed by body-reading guards — a section whose subtree holds a recommendation object, a GRADE evidence table, a scope statement, or a finding of absence is kept whatever its heading says. Where a general rule would be wrong for one specific guideline, that guideline gets an exception entry keyed to its ID in magic_rules.py — a data module holding all the per-guideline and per-publisher tables, kept apart from the scraper logic so each can be reviewed on its own terms.

The rendered corpus is 35.5M characters across the 225 documents (median 83k, largest 1.8M — WHO's task-shifting guideline).

Open question: Useful data threshold

Each organization that drafts guidelines in this corpus enters information into the MAGIC database in slightly different ways. Some use recommendation blocks while others use plain prose; some tables include useful clinical information, others mix clinical data with process information. The result is that individual guidelines end up needing many specific organization and document level fixes which are difficult to process at scale and brittle in the event of future changes. I've tried to remove much of the non-clinical information but in some cases I've erred on the side of keeping some non-clinical information in order to keep verification worthy information. Admittedly, some clinical data is also lost, but by and large the error is the other way around. If it's found that too much noise has been entered into the database or conversely if certain verification worthy text has been omitted I am happy to refine further.

CLI

uv run amfv-scrape --source magic --documents 1
uv run amfv-scrape --source magic --url https://app.magicapp.org/#/guideline/nyxpZL
uv run amfv-scrape --source magic --documents 3 -f markdown -o ./magic-out/

--documents is an upper bound on a catalogue run: unreadable or placeholder guidelines are logged and skipped. --documents all scrapes the whole catalogue and takes roughly 30-45 minutes — a politeness delay of 5 seconds sits between document requests. --include-drafts keeps guidelines whose own title marks them a draft for consultation; they are dropped by default. --url accepts any app.magicapp.org link containing /guideline/<code>; the code is resolved against the catalogue.

Files

• datasets/amfv_datasets/scraping/magic.py — the scraper, laid out as the pipeline runs: catalogue access, HTML repair, conversion and cleanup, keep/drop policy, recommendation and PICO rendering, assembly, API
• datasets/amfv_datasets/scraping/magic_rules.py — the per-guideline and per-publisher keep/drop tables (data, not logic; keys and entries sorted for lookup)
• datasets/amfv_datasets/scraping/cli.py — --source magic dispatch and --include-drafts
• datasets/amfv_datasets/scraping/__init__.py — public names
• datasets/test/test_scraping_magic.py — offline tests, MockTransport, no fixture files; sections mirror magic.py's layout
• datasets/test/test_scraping_cli.py — the new flag

Test plan

• uv run pytest datasets/test — 233 passed, no regressions
• uv run ruff check datasets/ && uv run ruff format --check datasets/
• Full catalogue render before and after every keep/drop rule change, diffed — each change touched exactly the documents it named
• The split into magic.py + magic_rules.py and the accompanying cleanup were verified the same way: all 225 documents byte-identical before and after, content and metadata
• Live API smoke test: default run, --url run, and a 3-document run, including a guideline published after the rules were written

To reproduce quickly: uv run amfv-scrape --source magic --documents 3 -f markdown -o /tmp/magic-out/

Known limitations

• The per-guideline rules match exact text from living documents; when publishers edit, a rule can silently stop matching.
• Scraper currently only works on documents written in English (languages= parameter exists; the 218 non-English guidelines are skipped).
• A small measured set of accepted losses, each read before accepting: process-narrative sentences that mention doses in passing, and duplicated summary-table rows whose recommendations survive in document bodies.
• Images are dropped (they were 54% of raw output by bytes); MAGICapp stores a few flowcharts and dosing tables only as images or linked PDFs, which no text scraper can carry. Scope-wise, image removal is executed inside this scraper, not in html.py.

[CLAassistant]

All committers have signed the CLA.