wikrs

Fast, honest wikitext extraction and parsing in Rust.

Turn Wikipedia XML dumps directly into clean text, JSONL, RAG-ready sections, or GFM Markdown. When wikrs cannot faithfully handle a construct, it emits a diagnostic instead of silently returning plausible but wrong output.

Latest full-dump validation (simplewiki 2026-07, 282,900 articles): 7.65 seconds with parallel multistream decoding and 98.1% residual-markup-clean. 73,873 pages (26.1%) reported at least one unsupported construct instead of silently guessing.

Quick start

Install the CLI and run the committed sample dump:

cargo install wikrs
curl -fsSL https://raw.githubusercontent.com/SFARL/wikrs/main/examples/quickstart.xml \
  -o /tmp/wikrs-quickstart.xml
wikrs --input /tmp/wikrs-quickstart.xml --format jsonl

Output:

{"title":"Earth","text":"Earth is a planet.","diagnostics":[]}

For a real multistream dump, add its companion index to decode streams in parallel:

wikrs --input enwiki-latest-pages-articles-multistream.xml.bz2 \
  --index enwiki-latest-pages-articles-multistream-index.txt.bz2 \
  --format sections > sections.jsonl

Install the Rust library with cargo add wikrs. GitHub releases include prebuilt Linux x86_64 and aarch64 binaries. Intel and Apple Silicon Mac archives are also available but are not signed or notarized, so macOS users should prefer cargo install wikrs to avoid Gatekeeper quarantine.

Why wikrs?

• Fast dump-native pipeline. Streams .xml and .xml.bz2 without loading the dump into memory.
• Honest output. Structured diagnostics travel with JSONL and sections output; --fail-on unsupported turns them into a pipeline gate.
• Useful structure. Choose plain text, structured AST, flat sections for RAG, or round-trip-tested GFM Markdown.
• Deliberate scope. Templates are dropped and flagged rather than partially expanded at much lower throughput.

What is this?

The text inside a Wikipedia XML dump isn't HTML — it's wikitext, MediaWiki's markup language. Anyone training a model or building RAG over Wikipedia has to strip that wikitext into clean text first.

The de-facto tool for that, WikiExtractor (Python), is slow and silently drops or mangles complex templates and tables.

wikrs is a Rust take, delivered in two tiers:

• Floor — a faster WikiExtractor. wikitext → clean plain text, measured ~22× faster than WikiExtractor on an 8 MB dump (it's Rust; see Benchmarks). Drop-in for the "I just need the text" use case.
• Ceiling — a modern wikitext engine. A structured AST that preserves tables, link anchor text, and document structure — and that emits a diagnostic when it hits input it can't faithfully handle, instead of silently corrupting the output.

CLI reference

Build from source:

cargo build --release

# wikitext dump -> clean plain text (or JSON Lines)
./target/release/wikrs --input enwiki-latest-pages-articles-multistream.xml.bz2 --format jsonl > out.jsonl

# report the conversion rate instead of writing pages
./target/release/wikrs --input dump.xml.bz2 --stats

--format text (default) emits one article's plain text per record; --format jsonl emits {"title":…,"text":…,"diagnostics":[…]} per line; --format sections emits one JSON object per page with the article split into flat, level-tagged sections — {"title", "sections": [{"level", "heading", "text"}], "diagnostics"} — ready for RAG chunking (level = the heading's = count, lead section is level: 0); --format markdown emits GFM markdown per page (# title + structure-preserving body: headings, links, lists, tables; out-of-range constructs appear as visible ```wikitext fenced blocks). Markdown output is conformance-tested by construction: a round-trip harness asserts that an independent GFM parser (pulldown-cmark) reads every emitted document back to exactly the structure the AST declares — over all 1,071 MediaWiki parserTests inputs and a fuzzer. Both .xml and multistream .xml.bz2 are accepted. The default ast engine (Stage 2 parser: structured, honest diagnostics) handles real articles; pass --engine strip for the Stage 1 fast/lossy path.

The diagnostics travel with the output. Each diagnostics entry is {"code","severity","start","end","message"} (byte span into the page's wikitext); with --engine strip the key is absent entirely — Stage 1 can't diagnose, which is different from "checked and found nothing". --stats reports the parser's view next to the residual-markup heuristic (zero-diag= warned= unsupported=), plain --format text prints a one-line stderr summary when anything was flagged (stdout stays pure article text), and --fail-on unsupported (or the stricter warning) turns any flagged page into a non-zero exit code for pipelines.

The CLI streams the dump in bounded batches — memory is bounded regardless of dump size — and a dump read/decode error is a hard error with a real exit code, never a silently skipped page. For a multistream dump, pass the companion index via --index to decode the bz2 streams in parallel:

./target/release/wikrs --input enwiki-…-multistream.xml.bz2 \
    --index enwiki-…-multistream-index.txt.bz2 --stats

The full English Wikipedia (7.19M articles, 26.4 GB .bz2) runs end-to-end in ~7.4 minutes on an Apple-Silicon laptop with --index (5.1×, ~560 MB peak RAM), or ~38 minutes single-stream without it (~210 MB peak) — identical output either way.

Why is this hard? (and why that's the moat)

wikitext has no clean grammar. The only complete spec is MediaWiki's ~6,200-line PHP regex engine, and its template system is a text macro processor: template expansion isn't guaranteed to produce self-contained markup (a template can emit just an opening <table>, or a lone <tr>). So "parse then expand" and "expand then parse" both fail — they're entangled. Even MediaWiki's official Parsoid, with a full-time team over a decade, fell back to calling the PHP preprocessor.

So wikrs does not chase byte-level MediaWiki compatibility — that path is a tar pit where you reimplement two decades of bugs. Instead:

High correctness within an honestly-declared support range, and explicit diagnostics outside it.

That honesty — telling you exactly what it couldn't parse — is the core difference from tools that are silently wrong.

Non-goals

• ❌ Byte-level MediaWiki / Parsoid compatibility
• ❌ Full template / Lua (Scribunto) expansion
• ❌ Editing or emitting wikitext — wikrs is read-direction only: wikitext → text / AST / sections JSONL / Markdown

Known differences vs WikiExtractor

wikrs's Stage 1 extractor is deliberately lossy, like WikiExtractor — but the exact choices differ. Current behavior:

• Templates ({{…}}) and tables ({|…|}) are dropped (nesting-aware).
• Internal links keep their visible text: [[A|text]]→text, [[A]]→A.
• [[File:…]] / [[Image:…]] are dropped, caption included.
• External links keep their label: [url text]→text; a bare [url] is dropped.
• <ref>…</ref>, HTML comments, and <nowiki> are removed (nowiki keeps its inner text).
• Headings, list markers, and bold/italic are reduced to their text.

Anything beyond this is honestly out of scope for Stage 1 — structure-preserving extraction (tables, link graph) is Stage 2. Behavior is tracked in docs/stages/stage-1-extractor.md.

Roadmap

Stage	What	Status
1	Plain-text extractor — wikitext → clean text, benchmarked against WikiExtractor	✅ done (shipped in 0.1.0)
2	Structured AST + diagnostics — preserves structure, warns on pathological input	🛠 in progress (~49% coverage; now the CLI default)
3	LLM-facing output — sections JSONL for RAG chunking + GFM Markdown with a round-trip conformance harness	✅ shipped (--format sections in 0.2.0, --format markdown in 0.3.0)

The headline metric: a precision/coverage differential vs Parsoid on real pages (now landing — see Benchmarks), plus a clear-eyed account of the rest. See docs/TESTING.md.

Benchmarks & test status

Kept current on every change via the project's wikrs-dev-workflow skill. Methodology: docs/TESTING.md.

Last updated: 2026-08-02

• Tests: green — cargo test --all-features

• ⚡ vs WikiExtractor (end-to-end, identical bench-compare harness): on the full real simplewiki dump (1.67 GB, 281,799 articles, 2026-06 snapshot, 10-core Apple Silicon) wikrs is ~32× faster — 5.2 s / 322 MB/s vs WikiExtractor 164 s / 10.2 MB/s. Single-core, wikrs is ~150 MiB/s end-to-end (it parallelises across cores; WikiExtractor streaming to one stdout does not). On the original 8.3 MB synthetic dump the figure was ~22× — conservative, not inflated: tiny inputs are dominated by wikrs's process-start overhead (a 16 MB slice shows only ~5× for the same reason), so the real full-dump gap is wider. Fairness: WikiExtractor ran with its default parallelism (cpu_count() − 1 = 9 worker processes on the test machine; we verified its default beats --processes 1 by ~2.2×), so this is parallel-vs-parallel on identical hardware, not wikrs-parallel vs WE-single-core. Reproduce: cargo xtask bench-compare <dump.xml> (real) or cargo xtask make-sample-dump && cargo xtask bench-compare target/bench-dump.xml (synthetic); the harness prints each side's wall time, throughput, and peak RSS (via /usr/bin/time), so the memory numbers in this README are reproducible from the same command that produces the speed numbers.

• Sample-article throughput (criterion, benches/compare.rs):

  Implementation	Throughput	Notes
  wikrs AST path (parse→plain, default)	~131 MiB/s	Stage 2 engine — ≥ strip throughput while also emitting diagnostics; structured where it can, strip-fallback for Unsupported blocks
  wikrs::extract::strip	~115 MiB/s	Stage 1 extractor → clean text (five allocating passes)
  parse_wiki_text (reference)	~298 MiB/s	community Rust parser → AST (no text out), 2018

  The Stage 2 AST path (parse → plain text) runs at roughly the same throughput as the Stage 1 strip while producing both text and diagnostics — the af0c5f0 DoS-robustness fix traded ~10% AST throughput for linear-on-adversarial-input safety, which is why it lands on par with strip rather than ahead of it. It does not expand templates — it drops them with a W-TEMPLATE warning and keeps the surrounding prose. Expanding templates (à la Bliki) would mean a Lua/Scribunto engine and ~2 orders of magnitude slower (Bliki runs at ~0.4 MB/s) — surrendering the one advantage wikrs has. So: honest drop + flag, keep the speed.

  Run it yourself: cargo bench --bench compare (or scripts/bench.sh in the repo, which wraps it with a recordable summary).

• Current full-dump release check: on the 2026-07 simplewiki multistream dump, the v0.4 release line processed 282,900 articles in 7.65 s with the companion index, using 206 MiB peak RSS. It completed with exit code 0 and 277,418 pages (98.1%) residual-markup-clean. This is a leniency floor — it catches leaked {{ }}, [[ ]], or {| |} markers, not semantic correctness vs Parsoid.

• Diagnostics at dump scale: 3,740 pages (1.3%) had zero diagnostics, 277,070 (98.0%) had at least one warning, and 73,873 (26.1%) had at least one unsupported/error diagnostic. Warning and unsupported counts overlap because one page can contain both. This is the current evidence behind the honesty claim: unsupported input is reported at scale rather than folded into a clean-looking success number.

• Earlier scale check: the pre-v0.4 2026-06 enwiki run processed 7,189,653 articles from the 26.4 GB compressed dump in 7.4 min with --index (38 min single-stream), with 98.0% residual-markup-clean and zero crashes. It remains evidence that the streaming architecture scales to enwiki, but it predates v0.4's diagnostics and stricter dump-boundary changes; the current release gate is the simplewiki run above. On the 1,077 synthetic parserTests cases the residual metric is 98.1% (cargo test --test parser_tests stage1_conversion_rate).

• Stage 2 parser coverage (parserTests, 1077 cases): 49.1% parse with zero diagnostics — fully inside the engine's declared support range (paragraphs, headings, bold/italic, internal + external links, flat, nested & definition lists, preformatted blocks, simple tables, refs/nowiki/comments, inline HTML formatting tags, presentational HTML containers <div>/<center>/<blockquote>/<p>, shown transclusion tags <noinclude>/<onlyinclude>, and HTML lists <ul>/<ol>/<li> unwrapped to their text). Inline templates are dropped with a W-TEMPLATE warning (prose kept, honestly flagged → not counted as fully supported). Track: cargo test --test parser_tests stage2_coverage_rate.

• Stage 2 differential — the "three numbers" (layer 2 of docs/TESTING.md; the headline Stage-2 DoD): wikrs's extracted prose vs Parsoid's rendered HTML over a fixed, committed sample of real pages. Seed run (18 featured-class articles, fetched 2026-06-27):

  • 99.7% word-precision (93.2% strict phrase-precision) — of the words wikrs emits, 99.7% are corroborated by the article (18/18 pages fully faithful, zero silent). The phrase/word gap is table-cell reordering (same words, different adjacency than Parsoid's grid), which order-robust word-precision correctly treats as faithful — not garbling.
  • ~49% coverage — wikrs extracts ~half of each article's prose; the rest is template-expanded content dropped by design (the speed moat, made measurable).
  • 100% transparently reported — every real article trips ≥1 out-of-range construct ({| table, <math>, gallery) and wikrs flags each rather than silently skipping. This is the honest contrast with WikiExtractor's silent errors — not a failure.

  Reproduce: cargo xtask diff-fetch && cargo xtask diff-report (pages cached gitignored; only the names-only title lists are committed). Representative evidence — on 120 random ns0 pages (cargo xtask diff-sample): 99.3% word-precision (order-independent), 0% silent structural-diff, 96% of pages fully faithful (115/120). The strict 3-gram phrase-precision is 91.3% — the gap is table-cell reordering, which order-robust word-precision treats as faithful. Zero silent errors is the headline: across 120 real pages wikrs never emits content absent from the article; the rest is honestly flagged (Reported). A real sample now — not yet N-thousand, but evidence.

• Backward-compatibility ratchet: the 529 cleanly-passing cases are pinned by name in tests/coverage_baseline.txt (names only — derived facts about wikrs, not the GPL fixture). cargo test --test parser_tests coverage_ratchet fails if any pinned case regresses, so coverage can only ratchet up and every change to it is a deliberate, reviewed baseline diff. The single coverage percentage can rise while individual cases silently break; this catches that. Re-bless an intended change: BLESS_COVERAGE=1 cargo test --test parser_tests coverage_ratchet.

• Robustness: strip never panics and stays linear — 2 MB of adversarial input in ~150 ms (tests/robustness.rs, runs in CI). Deeper fuzzing: cargo +nightly fuzz run strip.

Documentation

Doc	Contents
docs/DESIGN.md	Architecture, module layout, I/O contracts, error philosophy, non-goals (English)
docs/TESTING.md	Four-layer test strategy + benchmarks (English)
docs/stages/	Per-stage checkpoints and tasks (internal dev history, Chinese)
docs/PROJECT-HANDOFF.md	Strategic context & decision log (internal dev history, Chinese)
WORKLOG.md	Per-change evidence log — every fix with its measured before/after (internal dev history, Chinese; repo only, not in the crate)

Status & contributing

Pre-1.0 and moving fast — the design docs above are the source of truth. No API stability guarantees yet. Real-world workloads, minimal failing inputs, and focused patches are welcome; see CONTRIBUTING.md, Issues, and Discussions.

License

Licensed under either of

• Apache License, Version 2.0 (LICENSE-APACHE)
• MIT license (LICENSE-MIT)

at your option.

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.