perf(parallel): a flat batch path that shares nothing between workers by ArthurZucker · Pull Request #2361 · huggingface/tokenizers

ArthurZucker · 2026-08-25T03:14:53Z

encode hands back a Vec<Encoding>, one per document, and the machinery for that — a
chunk table, a task list, a completion queue, an Encoding and its Vec per input — is
most of the cost when the documents are short. encode_batch_flat returns the ids as one
contiguous run with row offsets, and lets each worker fill an arena of its own: nothing
shared between workers but the immutable tokenizer.

A template the flat layout cannot model (a pair template, or one carrying type ids) falls
back to encode and copies, so the entry point is total.

Concatenation back to the pool. Joining the arenas was the remaining serial tail.
split_at_mut hands each job a disjoint run of the output and the pool fills them at once,
and the output is handed out as spare_capacity_mut rather than vec![PipelineToken(0); n]
so a zero-fill of three megabytes the workers are about to overwrite does not happen serially.

Now scoped to the flat path only

This PR bundled four changes. Three are gone:

was	now
sort → partition in `plan_work`	landed as #2378
physical-core pool sizing (+166)	split to #2380
`post_process` buffer reuse	split to #2381

What is left is the flat path alone: +378/−20 across 5 files, down from +593/−42. Each of
the others can be reviewed and measured on its own, and a problem with one no longer blocks
the other two.

Measured on top of #2365

deepseek-v4, 3.5 MiB of english as 20k documents, min of 7 passes, best of 3 interleaved
rounds, ids_hash identical across every config and thread count:

config	1	2	4	8	10
base `encode`	274.2	615.0	895.1	1197.7	1311.8
`encode` + #2365	275.7	612.5	815.3	990.7	1069.9
this + #2365, `encode`	274.2	644.1	901.1	1229.0	1225.8
this + #2365, `encode_batch_flat`	311.6	657.8	1158.6	1840.5	2033.3

1.50× at 8 threads, 1.66× at 10, +13% at one thread. #2365 is orthogonal, and for a
concrete reason: the batch path takes a scratch once per worker per batch, not once per
document, so the sharded pool has no contention there to remove. Its own 20× is in the
serving shape — N application threads each calling encode. Both are worth having.

Two caveats on the numbers:

the 3.4× in the original description does not reproduce on this machine — I measure
1.4–1.5× at 8 threads, because the encode baseline here (1198 MiB/s @8) is far above the
497 the table quotes. Different machine and corpus; the table above supersedes it.
the within-config spread at 8 threads is wide (base alone ranges 974–1247 across 6 samples),
so read the 1.5× — well outside it — and not the smaller gaps.

The concat-to-pool commit earns its keep: a version of this path without it measures
17% slower at 8 threads and 13% at 10 (1491 vs 1796 MiB/s), because the serial memcpy
join only becomes the bottleneck once the workers are fast enough. It is invisible below 4
threads, which is why it reads as a micro-optimisation and is not one.

One simplification still on the table

flat_specials and encode_into's reproduces_sequence ask the same question of the
template in two places. Folding them into one predicate deletes flat_via_encode entirely
— including the String it allocates per input — and would make existing encode_into
callers faster on BERT/RoBERTa-style templates, which currently assemble an Encoding and
copy even though extending by a prefix and a suffix would do. Not in this PR; happy to do it
as a follow-up.

`num_threads()` defaulted to `available_parallelism()`, which counts SMT siblings. Filling them costs throughput on the encode path: on medium documents (~8 KiB) throughput peaks at the physical core count and then falls off, -20% going 88 -> 176 threads on aarch64 and -47% going 128 -> 512 on a two-socket Granite Rapids. Two workers on one physical core share a front-end and an L1, and this path is branchy and L1i-hungry, so it feels that more than most workloads do. Default to physical cores instead, read from sysfs on Linux and sysctl on macOS, and clamp to `available_parallelism()` so a cgroup quota or an affinity mask still wins -- this only ever lowers the count. Platforms that will not report a physical count keep the old behaviour. `set_num_threads()` still overrides. The sibling-list parser is compiled and tested on every platform, not just Linux: a misparse there would divide the pool size by the wrong number everywhere, and it is the only real logic here. (cherry picked from commit e1cd411)

Encoding one document allocated twice: `encode_sequence` built the ids, then `post_process` allocated a second buffer of its own and copied them in, dropping the first. For a batch of short documents that pair is a large share of the work, and it is worse than it looks -- a worker thread allocates the buffer and the consumer thread frees it, so every document costs a cross-thread free. For a single-sequence template, A's buffer already holds almost the whole answer, so use it as the output and splice the specials around it: the suffix appends, the prefix costs one memmove. Multi-sequence templates and templates carrying type ids keep the old path. Measured on 26k short documents (gpt2, 5 MiB of english, one document per line): allocations per document 2.28 -> 1.28 single-threaded 3.28 -> 2.28 on the pool throughput 152 -> 171 MiB/s single-threaded Byte-exact: the pipeline oracle passes on all 9 models -- including llama2, llama3, gemma4, t5 and bert, whose templates add BOS/CLS/SEP and so exercise the splice -- over 15 corpora at two window sizes, for add_special_tokens both false and true. `examples/batch_alloc.rs` is how those allocation counts were taken; it also reports the mdoc/batch split, which is where the remaining work is. (cherry picked from commit db9d5cc)

Batches of short documents did not scale: the pool was there, the workers were free, and the throughput stopped climbing at four threads. The work the threads do is fine -- what caps them is everything the *calling* thread does before any of them starts. Measured on 20k-line batches of `big.txt` at 8 threads, that was about 35% of a batch encode, which by Amdahl caps the speedup near 2.9x however many cores are idle. Three pieces of it, none of which had to be serial: **The chunk sort.** `plan_work` sorted every chunk longest-first, 18k of them in a 20k-line batch, for scheduling. Tasks are already byte-balanced -- they are grown until they hold `PARALLEL_MIN_BYTES` -- so a total order bought almost nothing; what matters is only that a chunk big enough to be its own task is not picked last. Partitioning those to the front is O(n) against the sort's O(n log n), and was worth ~14% of the batch on its own. **A `Vec` per sequence.** The per-sequence results were a `Vec<Vec<ChunkResult>>`, so a batch of 20k short lines asked the allocator for 20k vectors before any encoding began. They are now one flat buffer with per-sequence offsets, which took planning from 2.30ms to 0.91ms. **The input copy.** `Input` owns its text, so `From<&[&str]>` clones every line, on the calling thread, ahead of every worker -- 19% of the batch. The copies are independent, so they go to the pool. threads 1 2 4 8 before 177 261 286 305 MiB/s after 171 303 442 509 2.98x at eight threads, from 1.72x. Not linear yet: what remains is the per-document `Encoding` allocation, which `perf/batch-shape-scaling` addresses with a flat `BatchEncoding` and worker-side reconstruction. That branch forked before #2338 and #2352 and rewrites the same lock-free code a different way, so it wants a real rebase rather than a merge -- see the PR description. `tk-encode/examples/batch_alloc.rs` came in with the cherry-picked `post_process` commit and uses `tk_encode::Tokenizer`, which #2352 removed; it is dropped rather than half-ported. 247/247 tests pass, fmt and clippy clean.

The general batch path exists to hand back a `Vec<Encoding>`, one per document, and almost all of its machinery is there to make that safe: a plan, a chunk table, a task list, a completion queue, per-sequence result slots. On a handful of long documents that costs nothing. On twenty thousand short ones it is most of the work, and none of it is work the threads can help with. `encode_batch_flat` takes the documents borrowed and returns one contiguous id buffer with row offsets. Each worker takes a run of documents, encodes them into an arena of its own with a scratch of its own, and records how long each came out; the arenas are concatenated in input order at the end. There is no plan, no completion queue, and nothing shared between workers -- which is the property the POC write-up puts at the centre of its scaling result, and the reason this scales where the general path plateaus. deepseek-v4, 6.5 MiB of `big.txt` as 20k-line batches, timed loop repeated so the fixed costs are amortised: threads 1 2 4 8 general 174 319 449 497 MiB/s flat 223 407 716 1111 5.0x at eight threads against 1.7x when this branch started, and the flat path is 1.3x faster even on one thread because it stops allocating per document. The serial tail is small: concatenating the arenas measured 0.05ms against 0.9ms of parallel work, about 5%. Chunk size is two times `PARALLEL_MIN_BYTES` worth of documents, which measured best over a sweep from 64 to 1250; below that rayon's per-task cost shows, above it a thread can end up with a single run. A template the flat layout cannot model -- a pair template, or one carrying type ids -- falls back to `encode` and copies, so the entry point is total. `flat_batch.rs` checks the two paths agree document for document on 10k documents with `add_special_tokens` both ways, including empty inputs, CJK and literal special tokens. It is a second implementation of the same contract, not a wrapper, so it needs a test that pins them together. Idea and shape taken from `perf/batch-shape-scaling`, ported rather than merged: that branch forked before #2338 and #2352 and rewrites the same lock-free code a different way. 249/249 tests pass, fmt and clippy clean.

Concatenating the per-worker arenas was the flat path's serial tail, and it was worth measuring rather than assuming: with the concatenation removed entirely the eight-thread number goes from 1652 to 1805 MiB/s, so it was costing 9%. Each part's destination is known once the lengths are, so the copy does not have to be serial. `split_at_mut` hands each job a disjoint run of the output and the pool fills them at once. The first version of this allocated with `vec![PipelineToken(0); n]` and recovered almost nothing -- a zero-fill of three megabytes the workers are about to overwrite is itself serial, and cost about what the parallel copy saved. Handing out `spare_capacity_mut` instead skips the memset: threads 1 2 4 8 10 zero-fill 308 581 1040 1646 1807 MiB/s uninit 317 586 1068 1712 1852 1712 against the 1805 ceiling: the tail is down from 9% to 5%. Eight threads give 5.4x here, not 8x, and the reason is not this code. Profiling the pool shows the eight workers taking 388-389 samples each -- perfectly balanced, ~100% busy, no idle and no stealing tail. Three candidate causes were measured and all came back flat: one scratch per worker instead of one per chunk (`map_init`) 0.996x, jemalloc in place of the system allocator 0.992x, and QoS_USER_INTERACTIVE on the workers to keep them off the E-cores 0.999x. Chunk granularity is already at its optimum -- a sweep from 4 to 2500 documents peaks flat across 64-128, which is where the computed default lands. What is left is the word cache, and it inflates the baseline rather than slowing the workers. A benchmark that encodes the same corpus repeatedly lets one thread's 2 MB cache go warm over all of it; eight workers each cover an eighth. Shrinking the cache so warmth cannot accumulate makes the apparent scaling jump: cache slots 1 thread 8 threads speedup 256 167.1 1145.5 6.85x 4096 186.1 1238.1 6.65x 65536 (default) 304.1 1676.2 5.51x Absolute throughput is still best at the default -- 1676 beats 1238 -- so this is the baseline being generous, not the pool being slow. The same effect explains why eight independent processes reach 7.82x: each one has its own fully warm cache over the whole corpus. 249/249 workspace tests and 37/37 with the parallel feature on; `flat_batch` still checks the flat path document for document against `encode`. fmt and clippy clean.

Replaces the counting sort from the previous commit with a single partition pass, on review against #2361 which reached the same conclusion independently and expressed it in eight lines. The task grouping never needed the ordering the counting sort provided. Tasks are byte-balanced -- each grown until it holds `PARALLEL_MIN_BYTES` -- so how the chunks below that threshold sit among themselves cannot change a task's size. Exactly one property matters: a chunk big enough to be a task on its own must not be picked last and become the straggler the batch waits on. Partitioning those to the front is O(n) in place, and drops the `Vec<u32>` permutation, the `order` field, the size-class table, and the extra load per chunk in `encode_task`. Same measurement as before -- batched `encode`, 64 B documents, corpus repeated to fill 250 ms per pass, 3 interleaved rounds -- shows the two are equivalent in throughput, so the precision the counting sort bought was not worth its code: counting sort partition gpt2/chinese +17.8% +16.9% llama-3/chinese +11.0% +12.9% llama-3/code +6.8% +6.3% llama-3/english +1.5% +5.4% gpt2/code +2.9% +3.0% gpt2/english +4.0% +2.6% Net effect on this PR: 152 insertions become 88, and the worker's inner loop goes back to indexing `chunks` directly.

@McPatate

…2378) * perf(pipeline): decide the single-sequence serial fallback before planning `parallel::encode` gated on `inputs.size_bytes() < PARALLEL_MIN_BYTES`, then planned, then bailed to `encode_serial` when planning produced fewer than two tasks. For one document between `PARALLEL_MIN_BYTES` and twice it that bail is guaranteed: `plan_sequence` short-circuits any sequence under `2 * PARALLEL_MIN_BYTES` into a single chunk, one chunk makes one task, and one task fails the check. So the allocations were provably discarded, and the condition is knowable from the length alone. No measurable throughput effect, and it is worth recording why rather than implying one. Because `plan_sequence` short-circuits *before* its added-token scan, the only waste in that window is about six small `Vec` allocations, against ~22 us of encoding for a 9 kB document. Measured with single-document encodes at 4 kB / 9 kB / 20 kB (the first and last as controls, since the guard cannot fire there): every cell within +-1.5%, including the 9 kB window itself. Kept because it removes work that cannot ever be used, not because it is faster. The expensive version of this waste is at 16 kB and above, where `plan_sequence` does run the full `SpecialSegmentIterator` pass over the document, finds no special token, emits one chunk and discards it -- a duplicated added-token scan. This guard does not cover that, and it cannot cheaply: whether a sequence splits is only knowable from that scan. The real fix there is the intra-sequence splitting #2331 left as follow-up, which would make the planning useful instead of merely cheap to skip. * perf(pipeline): order chunks by size class with a counting sort `plan_work` sorted the chunk list with `sort_unstable_by_key(Reverse(len))` and then folded it a second time for the per-sequence `chunk_count`. Two things are wrong with that shape: - `SequenceChunk` is 40 bytes, so a comparison sort does O(n log n) moves of 40 bytes each. The container was never the problem -- `chunks` is a `Vec`, access was already O(1) -- what it sorts is. - It runs in the serial prologue, on the calling thread, before a single worker is spawned. It is pure Amdahl fraction, and worst exactly in the many-small-inputs case tasks were introduced for in #2338: one chunk per sequence means n is the batch size, so an 80k-document batch sorts 80k structs before any encoding starts. The grouping it feeds never needs a total order -- it walks the chunks accumulating bytes and cuts a task every `PARALLEL_MIN_BYTES`. Ordering by power-of-two size class is sufficient, and that is a counting sort: histogram, exclusive prefix sums largest-class-first, scatter. O(n), moving 4-byte indices instead of 40-byte structs, with `chunk_count` coming off the same histogram pass rather than a second fold. `tasks` now index `order` rather than `chunks`, so the worker pays one extra `u32` load per chunk -- against at least `PARALLEL_MIN_BYTES` of tokenizing per task. Batched `encode` of a 5 MB corpus, min of 5 passes with the corpus repeated to fill 250 ms per pass, 3 interleaved rounds, separately built binaries: 64 B documents n_docs base MB/s this commit gpt2/chinese 80457 362.7 +17.8% llama-3/chinese 80457 344.4 +11.0% llama-3/code 32691 709.0 +6.8% gpt2/english 81983 668.4 +4.0% gpt2/code 32691 683.9 +2.9% llama-3/english 81983 663.8 +1.5% 1024 B documents n_docs base MB/s this commit llama-3/chinese 5118 600.9 +2.1% gpt2/chinese 5118 846.5 +1.8% others -1.0% .. +0.9% Positive in every cell where n is large and gone by 1 kB documents, which is the shape an O(n log n) -> O(n) change in a serial prefix should have. The noise floor for this setup is about +-4%, measured from a commit that cannot affect the batch path at all, so the two chinese cells are the solid ones and the sub-3% entries are marginal. Token ids are unaffected: chunks are encoded independently and reassembled by `(seq, idx)`, so visiting order cannot change the result. Verified identical across all four builds in every cell. Only the task boundaries move. * perf(pipeline): partition the chunks rather than counting-sort them Replaces the counting sort from the previous commit with a single partition pass, on review against #2361 which reached the same conclusion independently and expressed it in eight lines. The task grouping never needed the ordering the counting sort provided. Tasks are byte-balanced -- each grown until it holds `PARALLEL_MIN_BYTES` -- so how the chunks below that threshold sit among themselves cannot change a task's size. Exactly one property matters: a chunk big enough to be a task on its own must not be picked last and become the straggler the batch waits on. Partitioning those to the front is O(n) in place, and drops the `Vec<u32>` permutation, the `order` field, the size-class table, and the extra load per chunk in `encode_task`. Same measurement as before -- batched `encode`, 64 B documents, corpus repeated to fill 250 ms per pass, 3 interleaved rounds -- shows the two are equivalent in throughput, so the precision the counting sort bought was not worth its code: counting sort partition gpt2/chinese +17.8% +16.9% llama-3/chinese +11.0% +12.9% llama-3/code +6.8% +6.3% llama-3/english +1.5% +5.4% gpt2/code +2.9% +3.0% gpt2/english +4.0% +2.6% Net effect on this PR: 152 insertions become 88, and the worker's inner loop goes back to indexing `chunks` directly. * Apply suggestion from @McPatate Co-authored-by: Luc Georges <McPatate@users.noreply.github.com> * Apply suggestion from @McPatate Co-authored-by: Luc Georges <McPatate@users.noreply.github.com> * Apply suggestion from @ArthurZucker --------- Co-authored-by: Luc Georges <McPatate@users.noreply.github.com>

#2378 landed the sort -> partition swap that this branch's "take the batch planning off the critical path" commit also made, so `plan_work` conflicted. Both sides had the identical partition loop; the resolution keeps the comment that landed on the base branch and this branch's `seq_start.push(outputs.len())`, which is the other half of that commit and is not on the base. `encode_flat` conflicted only because both sides append to the end of `parallel.rs`; kept whole.

This branch carried three unrelated changes besides the flat batch path. One (the sort -> partition swap) landed as #2378. The other two are now their own PRs and are reverted here so this branch is the flat path alone: * `perf(parallelism): default the pool to physical cores, not SMT siblings` -- 166 lines of pool sizing and Linux `/sys` topology parsing, nothing to do with the batch path. Was #2335. * `perf(pipeline): reuse sequence A's buffer in post_process` -- a change to the general path. Was #2336. Both revert cleanly and the flat path's tests still pass, which is the point of splitting them: each can be reviewed and measured against the base on its own, and a problem with one no longer blocks the other two.

…framed-document helper

@ArthurZucker

* perf(pipeline): reuse sequence A's buffer in post_process Extracted from #2361, which bundled it with the flat batch path; it is an independent change to the general path and reviews better on its own. The `batch_alloc` example the original commit added is left out: it imports `tk_encode::Tokenizer`, which #2352 removed, so it no longer compiles. #2361 deleted it two commits later for the same reason. * lol * remove bloat * Apply suggestion from @ArthurZucker * move post processor code where it belongs * more cleanup * Apply batched suggestions from code review Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com> * remove stuff that belongs in convert * update * nit * cleanup again * fix * fromat * nit * update * remove bloat shit * nit * Apply batched suggestions from code review Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com> * weave is ai * simple nit * fix(deps): bump chacha20 to 0.10.2, the yanked 0.10.1 fails cargo audit chacha20 0.10.1 and 0.10.0 are both yanked from crates.io. The audit jobs run `cargo audit -D warnings`, and a yanked crate is a denied warning, so the node audit failed. The `--ignore RUSTSEC-*` flags cannot suppress it: "yanked" is an index state, not an advisory ID. Pulled in transitively by rand 0.10.2. tokenizers/Cargo.lock was already on 0.10.2; the node and python lockfiles were missed. Verified with the exact CI command against all three lockfiles: node exit 0 python exit 0 (was 1: "1 denied warning found") tokenizers exit 0

@McPatate

…2378) * perf(pipeline): decide the single-sequence serial fallback before planning `parallel::encode` gated on `inputs.size_bytes() < PARALLEL_MIN_BYTES`, then planned, then bailed to `encode_serial` when planning produced fewer than two tasks. For one document between `PARALLEL_MIN_BYTES` and twice it that bail is guaranteed: `plan_sequence` short-circuits any sequence under `2 * PARALLEL_MIN_BYTES` into a single chunk, one chunk makes one task, and one task fails the check. So the allocations were provably discarded, and the condition is knowable from the length alone. No measurable throughput effect, and it is worth recording why rather than implying one. Because `plan_sequence` short-circuits *before* its added-token scan, the only waste in that window is about six small `Vec` allocations, against ~22 us of encoding for a 9 kB document. Measured with single-document encodes at 4 kB / 9 kB / 20 kB (the first and last as controls, since the guard cannot fire there): every cell within +-1.5%, including the 9 kB window itself. Kept because it removes work that cannot ever be used, not because it is faster. The expensive version of this waste is at 16 kB and above, where `plan_sequence` does run the full `SpecialSegmentIterator` pass over the document, finds no special token, emits one chunk and discards it -- a duplicated added-token scan. This guard does not cover that, and it cannot cheaply: whether a sequence splits is only knowable from that scan. The real fix there is the intra-sequence splitting #2331 left as follow-up, which would make the planning useful instead of merely cheap to skip. * perf(pipeline): order chunks by size class with a counting sort `plan_work` sorted the chunk list with `sort_unstable_by_key(Reverse(len))` and then folded it a second time for the per-sequence `chunk_count`. Two things are wrong with that shape: - `SequenceChunk` is 40 bytes, so a comparison sort does O(n log n) moves of 40 bytes each. The container was never the problem -- `chunks` is a `Vec`, access was already O(1) -- what it sorts is. - It runs in the serial prologue, on the calling thread, before a single worker is spawned. It is pure Amdahl fraction, and worst exactly in the many-small-inputs case tasks were introduced for in #2338: one chunk per sequence means n is the batch size, so an 80k-document batch sorts 80k structs before any encoding starts. The grouping it feeds never needs a total order -- it walks the chunks accumulating bytes and cuts a task every `PARALLEL_MIN_BYTES`. Ordering by power-of-two size class is sufficient, and that is a counting sort: histogram, exclusive prefix sums largest-class-first, scatter. O(n), moving 4-byte indices instead of 40-byte structs, with `chunk_count` coming off the same histogram pass rather than a second fold. `tasks` now index `order` rather than `chunks`, so the worker pays one extra `u32` load per chunk -- against at least `PARALLEL_MIN_BYTES` of tokenizing per task. Batched `encode` of a 5 MB corpus, min of 5 passes with the corpus repeated to fill 250 ms per pass, 3 interleaved rounds, separately built binaries: 64 B documents n_docs base MB/s this commit gpt2/chinese 80457 362.7 +17.8% llama-3/chinese 80457 344.4 +11.0% llama-3/code 32691 709.0 +6.8% gpt2/english 81983 668.4 +4.0% gpt2/code 32691 683.9 +2.9% llama-3/english 81983 663.8 +1.5% 1024 B documents n_docs base MB/s this commit llama-3/chinese 5118 600.9 +2.1% gpt2/chinese 5118 846.5 +1.8% others -1.0% .. +0.9% Positive in every cell where n is large and gone by 1 kB documents, which is the shape an O(n log n) -> O(n) change in a serial prefix should have. The noise floor for this setup is about +-4%, measured from a commit that cannot affect the batch path at all, so the two chinese cells are the solid ones and the sub-3% entries are marginal. Token ids are unaffected: chunks are encoded independently and reassembled by `(seq, idx)`, so visiting order cannot change the result. Verified identical across all four builds in every cell. Only the task boundaries move. * perf(pipeline): partition the chunks rather than counting-sort them Replaces the counting sort from the previous commit with a single partition pass, on review against #2361 which reached the same conclusion independently and expressed it in eight lines. The task grouping never needed the ordering the counting sort provided. Tasks are byte-balanced -- each grown until it holds `PARALLEL_MIN_BYTES` -- so how the chunks below that threshold sit among themselves cannot change a task's size. Exactly one property matters: a chunk big enough to be a task on its own must not be picked last and become the straggler the batch waits on. Partitioning those to the front is O(n) in place, and drops the `Vec<u32>` permutation, the `order` field, the size-class table, and the extra load per chunk in `encode_task`. Same measurement as before -- batched `encode`, 64 B documents, corpus repeated to fill 250 ms per pass, 3 interleaved rounds -- shows the two are equivalent in throughput, so the precision the counting sort bought was not worth its code: counting sort partition gpt2/chinese +17.8% +16.9% llama-3/chinese +11.0% +12.9% llama-3/code +6.8% +6.3% llama-3/english +1.5% +5.4% gpt2/code +2.9% +3.0% gpt2/english +4.0% +2.6% Net effect on this PR: 152 insertions become 88, and the worker's inner loop goes back to indexing `chunks` directly. * Apply suggestion from @McPatate Co-authored-by: Luc Georges <McPatate@users.noreply.github.com> * Apply suggestion from @McPatate Co-authored-by: Luc Georges <McPatate@users.noreply.github.com> * Apply suggestion from @ArthurZucker --------- Co-authored-by: Luc Georges <McPatate@users.noreply.github.com> (cherry picked from commit cbd94b8)

@ArthurZucker

* perf(pipeline): reuse sequence A's buffer in post_process Extracted from #2361, which bundled it with the flat batch path; it is an independent change to the general path and reviews better on its own. The `batch_alloc` example the original commit added is left out: it imports `tk_encode::Tokenizer`, which #2352 removed, so it no longer compiles. #2361 deleted it two commits later for the same reason. * lol * remove bloat * Apply suggestion from @ArthurZucker * move post processor code where it belongs * more cleanup * Apply batched suggestions from code review Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com> * remove stuff that belongs in convert * update * nit * cleanup again * fix * fromat * nit * update * remove bloat shit * nit * Apply batched suggestions from code review Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com> * weave is ai * simple nit * fix(deps): bump chacha20 to 0.10.2, the yanked 0.10.1 fails cargo audit chacha20 0.10.1 and 0.10.0 are both yanked from crates.io. The audit jobs run `cargo audit -D warnings`, and a yanked crate is a denied warning, so the node audit failed. The `--ignore RUSTSEC-*` flags cannot suppress it: "yanked" is an index state, not an advisory ID. Pulled in transitively by rand 0.10.2. tokenizers/Cargo.lock was already on 0.10.2; the node and python lockfiles were missed. Verified with the exact CI command against all three lockfiles: node exit 0 python exit 0 (was 1: "1 denied warning found") tokenizers exit 0 (cherry picked from commit 0743ac0)

ArthurZucker and others added 5 commits August 25, 2026 12:13

ArthurZucker changed the title ~~Batch scaling: take the planning off the critical path~~ Batch scaling: a flat batch path that shares nothing between workers Aug 25, 2026

ArthurZucker added the v1.rc0 label Aug 25, 2026

ArthurZucker mentioned this pull request Sep 2, 2026

perf(pipeline): partition plan_work's chunks instead of sorting them #2378

Merged

ArthurZucker added 2 commits September 2, 2026 18:35

This was referenced Sep 2, 2026

perf(parallelism): default the pool to physical cores, not SMT siblings #2380

Open

refactor post processing #2381

Merged

ArthurZucker changed the title ~~Batch scaling: a flat batch path that shares nothing between workers~~ perf(parallel): a flat batch path that shares nothing between workers Sep 2, 2026

ArthurZucker added 3 commits September 2, 2026 22:16

Merge feat/train_encode_split into perf/parallel-scaling

b4af78d

unbloat the flat batch path: one Encoding with optional offsets, one …

62489e4

…framed-document helper

gate Encoding::batch on parallelism, and rustfmt

17b8df9

Navigation Menu

Uh oh!

perf(parallel): a flat batch path that shares nothing between workers - #2361

perf(parallel): a flat batch path that shares nothing between workers#2361
ArthurZucker wants to merge 10 commits into
feat/train_encode_splitfrom
perf/parallel-scaling

ArthurZucker commented Aug 25, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

ArthurZucker commented Aug 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Now scoped to the flat path only

Measured on top of #2365

One simplification still on the table

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

ArthurZucker commented Aug 25, 2026 •

edited

Loading