add who guideline scraper with pdf full text extraction

[conscioustahoe]

What this does

Adds a WHO scraper to the datasets scraping pipeline and extracts the actual guideline text out of the PDFs.

The first version of this PR only scraped the HTML Overview from each landing page. Review feedback was that the Overview is an abstract and the guideline itself lives in a linked PDF, so the corpus would have been a pile of one-paragraph summaries. That is fixed here. Each document now leads with the Overview and continues with the full guideline body converted from the PDF.

Shape matches the NICE scraper — discovery and extraction stay separate and everything comes out as a normalized ScrapedDocument.

How discovery works

WHO runs on Sitefinity not Next.js. Listing goes through their OData publications API filtered to the Guidelines publishing office UUID (c09761c0-ab8e-4cfa-9744-99509c4d306b). That gives ~356 publications with $count=true for the progress bar.

One API quirk worth knowing: $top has to stay at 25 or below or WHO drops DownloadUrl from the response entirely. Page size is set to 25 for that reason alone. Since DownloadUrl is how we reach the PDF this is now load-bearing rather than cosmetic.

How extraction works

Each publication page at /publications/i/item/{id} is fetched and the Overview block under section.dynamic-content__section is converted to markdown with the shared html_to_markdown helper. Targeting the content container instead of the whole page keeps nav and footer cruft out.

Then the PDF. The download URL comes from the API or from the landing page link as a fallback. The PDF is fetched and converted to markdown by the new scraping/pdf.py, and the result is appended to the Overview.

A few details that matter:

• Page furniture. PDF running headers repeat the guideline title on every page. These are stripped using the known document title. The match is deliberately narrow — a line is dropped only when it is a long prefix of the title or the title is a prefix of it, because publishers shorten the title in headers and sometimes merge the left and right header onto one line. A naive "drop repeated lines" filter would have deleted real content, since lines like (Strong recommendation, moderate-certainty evidence) legitimately repeat a dozen times.
• Failure is not fatal. A missing PDF link, a failed download or a failed conversion logs a warning and falls back to the Overview. One bad PDF cannot abort a 356 document run.
• The result is observable. metadata.content_scope is full or overview, and pdf_backend and pdf_bytes describe the conversion. The keys are always present so the HuggingFace schema stays stable. section_count is now the real section count instead of a hardcoded 1.
• It is optional. PDF conversion needs the pdf extra. Without it every document degrades to overview rather than erroring, which is what keeps CI free of a heavy PDF stack.

Choosing a PDF converter

Picking the converter is a correctness decision not a formatting one. A backend that silently drops rows from a GRADE evidence table produces text that reads perfectly and states the wrong thing, which is the worst possible failure for fact verification.

Docling, PyMuPDF4LLM and Marker were compared on excerpts of five WHO guidelines spanning 50 to 270 pages. All five turned out to be born-digital with a clean text layer, so OCR is pure overhead and is disabled.

Backend	Converted	Seconds per doc	Stray page numbers	Verdict
Docling	5 / 5	2.7-13.7 warm	0	Chosen
PyMuPDF4LLM	5 / 5	1.9-5.1	40 across 2 docs	Rejected
Marker	spot check	72.7	0	Rejected

Marker lost table rows. On the table-dense carbohydrate guideline it recovered 87 rows where Docling and PyMuPDF4LLM both found ~136, dropping outcome-category rows out of a GRADE evidence table. Nothing in the surrounding prose signals the table is now incomplete.

PyMuPDF4LLM is the fastest by a wide margin and its row counts are close, but it emitted 40 bare page-number lines across two documents and corrupted table header rows. Stray page numbers turn into sentence fragments once documents get chunked for retrieval.

Docling converted all five, recovered the most heading structure, left no page furniture behind and kept table rows intact. It runs on CPU so nobody needs a GPU to work on this, and it is MIT licensed. Its 126.5 s first run is one-off model loading, not per document cost.

One honest gap: on 9789240084278 PyMuPDF4LLM emitted 42 table rows against Docling's 13 and I did not spot-check which is closer to the source. It did not change the decision since the other two failures are disqualifying on their own, but it is the open question if anyone revisits this.

The benchmark script is committed so none of the above has to be taken on faith:

uv run --group pdf python datasets/benchmarks/pdf_backends.py

Seeing what comes out

datasets/test/fixtures/pdf/ holds two six-page WHO guideline excerpts next to the markdown the scraper produces from each. Open them side by side to judge the conversion without running anything:

• who_9789240121805_excerpt.pdf → .expected.md — a GRADE recommendation keeping its certainty rating
• who_9789240124233_excerpt.pdf → .expected.md — a recommendation table keeping its rows and columns

How this is tested

Three layers, arranged so a Docling release can never break the build.

Always runs, including CI. 47 tests covering listing parse, URL normalization, the Overview path, metadata, and the PDF path through an injected stub converter — full text, download URL taken from the page, conversion failure fallback, download failure fallback and full-text disabled. No PDF stack required.

Real conversion, opt-in. uv sync --group pdf then pytest runs actual Docling over the committed sample PDFs and asserts the recommendation text, its certainty rating, heading structure, table rows and the absence of running headers survive. It ends with an end-to-end scrape_publication that proves a content_scope: full document. These assert on meaning not bytes so a Docling upgrade that reflows whitespace will not fail them.

Byte-exact diff, opt-in. AMFV_CHECK_GOLDEN=1 compares conversion against the committed .expected.md files. Off by default because it is deliberately version sensitive. When Docling does change the output you review the diff and accept it with AMFV_UPDATE_GOLDEN=1, which rewrites the files so the change lands in review instead of passing silently.

CI installs --group dev only, so the PDF tests report as skipped there: 47 passed, 7 skipped in 0.77 s. With the extra installed it is 52 passed in ~35 s, or 54 with the golden check on.

CLI

uv run amfv-scrape --source who --documents 1
uv run amfv-scrape --source who --url https://www.who.int/publications/i/item/9789240121805
uv run amfv-scrape --source who --documents 3 -f markdown -o ./who-out/

--source all includes WHO alongside NICE.

Licensing

WHO publications since Nov 2016 are CC BY-NC-SA 3.0 IGO. Each document carries metadata.license and metadata.attribution.

Flagging this explicitly: the two fixture PDFs and their extracted markdown are WHO content under CC BY-NC-SA 3.0 IGO, which is more restrictive than this repo's Apache-2.0. Apache-2.0 covers the code and does not extend to those files. Source, page ranges and attribution are recorded in LICENSE_NOTES.md. Happy to drop them for download-on-demand fixtures if that mix is not wanted.

Docling itself is MIT.

Files

Scraper and conversion

• datasets/amfv_datasets/scraping/pdf.py — new. pdf_to_markdown, PdfBackend, PdfConversionError, running-header stripping and section counting, with the optional Docling import gated behind HAS_DOCLING
• datasets/amfv_datasets/scraping/who.py — PDF fetch and convert wired into build_publication_text, real section_count, content scope and PDF metadata, fallback on failure
• datasets/amfv_datasets/scraping/__init__.py — re-exports
• datasets/pyproject.toml / pyproject.toml — pdf extra pinned to docling>=2.119.0,<3.0.0 plus a root pdf dependency group

Evidence and docs

• datasets/benchmarks/pdf_backends.py — runnable backend comparison
• datasets/benchmarks/README.md — measurements, the reasoning, and how to verify any of it
• datasets/test/fixtures/pdf/ — two guideline excerpts and their extracted markdown
• datasets/amfv_datasets/scraping/LICENSE_NOTES.md — content scope semantics and fixture attribution
• datasets/README.md — the optional extra

Tests

• datasets/test/test_scraping_pdf.py — conversion helpers, header stripping, section counting
• datasets/test/test_scraping_pdf_fixtures.py — real Docling over the sample PDFs plus the golden comparison
• datasets/test/test_scraping_who.py — PDF path via an injected converter and both fallback routes

Test plan

• uv run pytest with dev deps only — 47 passed, 7 skipped, no PDF stack needed
• uv sync --group pdf && uv run --group pdf pytest — 52 passed, real Docling over the sample PDFs
• AMFV_CHECK_GOLDEN=1 uv run --group pdf pytest datasets/test/test_scraping_pdf_fixtures.py — 7 passed, output matches committed markdown
• uv run ruff check . and uv run ruff format --check datasets
• uv build --package amfv-datasets
• End to end scrape_publication over a real WHO PDF — content_scope: full, 94 sections, ~54k chars, 0 running-header lines

[warner-benjamin]

I believe this only scrapes the abstract and not the guildines?