perf(normalizers): fast-path BertNormalizer for non-CJK input

[bsachart]

Problem

do_handle_chinese_chars unconditionally allocates a Vec<(char, isize)>, iterates every character, and calls transform() to rebuild alignments — even when the input contains no Chinese characters. This is wasted work for the common English/Latin case.

Change

fn do_handle_chinese_chars(&self, normalized: &mut NormalizedString) {
    if !normalized.get().chars().any(is_chinese_char) {
        return;
    }
    // ... existing logic unchanged
}

One early-return guard. No API changes, no new dependencies, no behavior change.

What it skips (for non-CJK input)

• Vec allocation
• for_each() closure dispatch
• transform() + alignment reconstruction

Trade-off

CJK input pays for one extra chars().any() scan that short-circuits on the first match. For inputs that are mostly CJK this is negligible relative to the transform() cost that follows.

[bsachart]

Benchmark results

big.txt (6.5 MB English text), bert-base-uncased vocab, criterion 20 samples:

Benchmark	Baseline (main)	This PR	Change
Single encode	1.351 s	1.276 s	−5.6% (p < 0.05)
Batch encode	278 ms	277 ms	−0.5% (p = 0.52, not significant)

Single encode shows a statistically significant improvement. Batch encode is dominated by parallelism overhead so the normalizer cost is amortized away — expected.

cargo bench --bench bert_benchmark -- "WordPiece BERT encode"

[ArthurZucker]

Ty! See #2119 as we are refactor the codebase! if you want to target this new branch!