Add a rusty-expressions regex backend (Oniguruma semantics, pure Rust, wasm-capable) - #2368
Open
Ttimmahlax wants to merge 5 commits into
Open
Ttimmahlax wants to merge 5 commits into
Ttimmahlax wants to merge 5 commits into
Conversation
`rusty_expressions` is Oniguruma reimplemented in pure Rust. It gives the same engine semantics as the `onig` backend -- it is gated differentially against live libonig -- while removing the C from the build entirely. Why this is worth a third backend rather than a swap: - `onig` links libonig through `onig_sys`, so it needs a C toolchain and cannot target wasm32 at all. That is why `unstable_wasm` exists. - `fancy-regex` solves the wasm problem but is a different engine with different semantics, which is presumably why it is not the default. - `rusty_expressions` is neither: same semantics as Oniguruma, no C. Verified on this branch: - 201 lib tests pass with `--no-default-features --features rusty-expressions`. - Pre-tokenizer output is **byte-identical to the `onig` backend** across 95 dumps (ByteLevel, Whitespace, Punctuation, a `\s+(?!\S)` negative-lookahead Split, and a `(\w)\1` backreference Split) over a corpus covering ASCII, CJK, Cyrillic, Greek, emoji with modifiers, NFD vs NFC, zero-width characters, URLs and long repeats. - `wasm32-unknown-unknown` compiles with `rusty-expressions` and fails with `onig`, as expected. The feature is additive and off by default; `onig` remains the default backend and nothing changes for existing users. When enabled it takes precedence over both existing backends. The dependency is taken with `default-features = false` because rusty_expressions installs a `#[global_allocator]` under its default feature, which a library must never impose on its consumers. Upstream: https://github.com/Remade-With-Rust/rusty_expressions
`unstable_wasm` currently means two things at once: "configure getrandom for
wasm32" and "use fancy-regex". A wasm build that wants a different pure-Rust
backend has no way to ask for the first without the second.
Adds a `wasm` feature carrying only the getrandom configuration, and redefines
`unstable_wasm = ["fancy-regex", "wasm"]`, so its meaning is unchanged for
existing users. A wasm build can now pick its engine:
--features unstable_wasm # fancy-regex, as before
--features rusty-expressions,wasm # Oniguruma semantics, still no C
Verified: all three of `unstable_wasm`, `rusty-expressions,wasm` and the
default native `onig` build compile.
Author
|
Pushed a second commit that I think makes the first more useful on its own.
So this adds a That half stands alone and is useful even if you'd rather not take the new |
The backend features resolve by precedence and `rusty-expressions` wins, so enabling it changes the engine for every crate in the graph -- including for someone who explicitly asked for `onig`. Cargo features are additive, so that can happen because a transitive dependency turned it on, with nothing said. The regex engine decides how text is split, so a silent swap is worth being able to detect. `compile_error!` would be the loud version, but it would break any build where a dependency legally enabled the feature. So this reports rather than enforces: `utils::REGEX_BACKEND` names the engine that was compiled in, and `utils::REGEX_BACKEND_OVERRODE_ONIG` flags the case where libonig was built and linked and then never called. A build that cares can assert on either. The Cargo.toml feature comment now says the precedence out loud too. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
It asserted the rusty_expressions backend specifically, so it failed under the default features, where onig is selected. The doctest runs under whatever features the build has; assert the invariant that holds in all of them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
0.x minor bumps are breaking in cargo's semver, so the "0.1" pin would not have picked up 0.2.0. That release swapped the engine's test oracle to live libonig and fixed what it found -- POSIX BRE groups, three dialect flag tables, CJK character-boundary and character-class handling, FIND_LONGEST, and a `find_at` that answered positions past the end of the haystack. 202 tests green on the rusty-expressions backend. (The four added_tokens tests need `make test` to download GPT-2 fixtures first; unrelated.) Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Collaborator
|
Hey! ty, we are working on tokenizers v1, see #2119 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds a third optional
SysRegexbackend behind arusty-expressionsfeature,alongside the existing
onigandfancy-regexones.rusty_expressionsis Onigurumareimplemented in pure Rust. It aims at the same semantics as
onig— it isdeveloped against live
libonigas a differential oracle — while removing theC from the build.
Nothing changes by default.
onigremains the default backend; the newfeature is additive and off unless asked for. When enabled it takes precedence
over the other two.
Why a third backend rather than a swap
The two existing options each give up something:
onigfancy-regexrusty-expressionsunstable_wasmexists precisely becauseonigcannot build for wasm32. Thatworkaround also changes the engine, which is presumably why it is not the
default. This backend is the combination that was missing.
Verification on this branch
--no-default-features --features rusty-expressions.onigbackend. I built thesame dumper under both backends and diffed: 95 dumps across
ByteLevel,Whitespace,Punctuation, a\s+(?!\S)negative-lookaheadSplit, and a(\w)\1backreferenceSplit, over a corpus covering ASCII, CJK, Cyrillic,Greek, emoji with skin-tone modifiers, NFD vs NFC, zero-width characters,
URLs, timestamps and long repeated strings. No differences.
wasm32-unknown-unknowncompiles withrusty-expressions(given the samegetrandomconfigurationunstable_wasmalready supplies) and fails withonig, as expected.Notes for review
default-features = falsedeliberately:rusty_expressionsinstallsrusty_allocas a#[global_allocator]underits default feature, and a library must never impose an allocator on its
consumers.
find_iteradvances by a whole character on an empty match, so it cannotland inside a UTF-8 sequence, and stops on an engine limit rather than
propagating — matching the
fancy-regexbackend's behaviour.unsafe_code = "deny"; the onlyunsafein that crate is anoptional C-ABI shim which is not enabled here.
Happy to adjust naming, feature precedence, or add the backend to CI if you'd
like it covered there. I can also run a wider differential over real
tokenizer.json files from the Hub if that would help review.