perf(pre_tokenizers): eliminate per-token heap allocations in ByteLevel decoder

[bsachart]

ByteLevel::decode_chain previously allocated a temporary Vec<u8> per token via try_fold(vec![], ...), incurring $N$ dynamic heap allocations per decode call.

This PR replaces per-token allocations with a single pre-allocated buffer sized to tokens.iter().map(String::len).sum(). Tokens containing special/unknown characters trigger an $O(1)$ bytes.truncate(start) rollback before inserting raw bytes, eliminating intermediate vector clones. CHAR_BYTES is derived dynamically from BYTES_CHAR in one line.

Benchmark Results (cargo bench --bench decode_benchmark -- '^decode-llama3-(en|ja)/decode$')

• Llama 3 English decode: 157.82 ms → 95.42 ms (~39.5% faster)
• Llama 3 Japanese decode: 25.22 ms → 18.83 ms (~25.3% faster)

All 204 library unit tests pass.

AI-assisted change.

[ArthurZucker]

ty, tho we should be comparing againt this: #2119

[bsachart]

Thanks for the pointer! I checked out #2119 (feat/train_encode_split) and saw that ByteLevelDecoder in tk-encode/src/decoders/byte_level.rs still has the per-token try_fold(vec![], ...) allocation loop.

I tested the single-pass pre-allocated buffer fix directly on ByteLevelDecoder::decode_chain in #2119. On 5M tokens, execution time dropped from ~334ms to ~220ms (~34% faster latency / ~1.5x throughput).

Would you prefer I retarget this PR to feat/train_encode_split, or open a new PR against #2119?

[ArthurZucker]

Yes please target the new branch!