Skip to content

docs: rewrite the README for the crate split - #2385

Open
ArthurZucker wants to merge 1 commit into
feat/train_encode_splitfrom
docs/readme-size-badge
Open

docs: rewrite the README for the crate split#2385
ArthurZucker wants to merge 1 commit into
feat/train_encode_splitfrom
docs/readme-size-badge

Conversation

@ArthurZucker

Copy link
Copy Markdown
Collaborator

The README still described the pre-split library. This rewrites it.

What's in it

  • Headline on the global set. "Fastest on the world's text, and the only fast one that runs every model." The perf section opens with the 20-cell all-scripts head-to-head against gigatoken (881 vs 590 MB/s, 15/20 cells won) rather than a Latin subset.
  • A section that names where we lose. english 0.87×, code 0.82×, and gigatoken leading tokbench's 10-engine ranking at 238.6 vs our 170.1 MB/s — with the explanation that this subset is the intersection of what ten engines support, so engines declining non-Latin shrink it toward English. The coverage asymmetry is stated in gigatoken's favour too: they are byte-exact on all 145 cells they accept, we attempt 203 of 210 and are exact on 174.
  • Subcrate map, seven <details> blocks, each answering why is this its own crate rather than just what it contains.
  • Bindings matrix — Rust / Python / Node shipping, C & C++ planned, Ruby community.
  • Hardware section. The claim is that no SIMD is required: every kernel has an always-compiled portable path that doubles as its byte-exact test oracle. A <details> table lists the four hardware-adapted ops and their fallbacks.
  • Footprint, 332799 bytes gzipped, reproducible with make slim-size.

Two figures hotlink from tokbench (figs/03-headtohead.png, figs/01-overall.png).

Caveats, deliberately on the page

Perf figures are the pipeline #2279 + #2296 snapshot and are labelled as such in a <sub> under the table. bitsplit, the thread pool and the load-time fold have all landed since, so these numbers understate the branch — re-running the matrix is follow-up work, not a blocker for replacing a README that describes a library we no longer ship.

Three things found while writing, not fixed here

  1. bitsplit's Cargo.toml description advertises Whitespace and Bert grammars that have no modules in models/mod.rs. The README says six (gpt2, cl100k, o200k, tekken, deepseek, kimi); the crate metadata still says eight.
  2. tokbench's GitHub Pages site is not deployed, so its own README's huggingface.github.io/tokbench/dashboard.html link 404s. This README links the repo instead. Enabling Pages would be worth it — that dashboard is the best artifact there.
  3. The size badge is a static shields URL, so the number needs a manual bump. A CI job to auto-update it was built and then dropped as not worth the machinery for a number that moves a few KB a year.

The branch name mentions a size badge for that reason; the badge automation is not in this PR.

Replaces the pre-split README. Adds the subcrate map (each with why it is its
own crate), the tokbench numbers including the gigatoken head-to-head, the
bindings matrix, and the per-op hardware table.

Perf figures are the `pipeline #2279 + #2296` tokbench snapshot and are labelled
as such; bitsplit, the thread pool and the load-time fold have landed since
without a re-run.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant