perf(pipeline): partition plan_work's chunks instead of sorting them

[ArthurZucker]

plan_work sorted the chunk table by descending length in the serial prologue —
O(n log n) moves of a 40-byte struct on the calling thread, where n is the batch size
when each sequence is one chunk. A single partition pass is O(n) and keeps the only
property the scheduler needs. +2.6% to +16.9% on 80k-document batches, ids identical.

Relationship to #2361

#2361 already contains this, reached independently, and its eight-line version is
better than the counting sort I first wrote — so this PR now uses that shape. Whichever
lands first, the other should drop the hunk.

The reason to keep this one separate: #2361 bundles three changes, and #2365 undercuts
two of them. encode_batch_flat (+222 in mod.rs, +167 in parallelism.rs) and the
concat-to-pool machinery exist so workers share nothing — but #2365 shows the shared
ScratchPool mutex was the contention, taking scaling efficiency from 3.3–12.3% to
69–82%. #2361's "3.4× over the general path" was measured against a baseline whose
contention #2365 removes, so that comparison needs redoing before the flat path earns
its keep.

This PR is the planning fix alone: one file, +88/−2.

The change

let mut boundary = 0;
for i in 0..chunks.len() {
    if chunks[i].range.len() >= PARALLEL_MIN_BYTES {
        chunks.swap(boundary, i);
        boundary += 1;
    }
}

Tasks are already byte-balanced — each grown until it holds PARALLEL_MIN_BYTES — so how
the sub-threshold chunks sit among themselves cannot change a task's size. Only one thing
matters: a chunk big enough to be a task on its own must not be picked last and become the
straggler the batch waits on.

Measured

Batched encode, 64 B documents, corpus repeated to fill 250 ms per pass (borrowed from
measure_scaling, for its reason — one call is a few ms and spawns up to
current_num_threads() rayon tasks), min over passes, 3 interleaved rounds, separately
built binaries.

cell	n_docs	base MB/s	partition
gpt2/chinese	80457	361.5	+16.9%
llama-3/chinese	80457	337.7	+12.9%
llama-3/code	32691	701.3	+6.3%
llama-3/english	81983	651.4	+5.4%
gpt2/code	32691	675.9	+3.0%
gpt2/english	81983	668.3	+2.6%

Gone by 1 kB documents, which is the shape an O(n log n) → O(n) change in a serial prefix
should have. ids_hash identical to base in every cell.

The middle commit is the counting-sort version and the last one removes it; the history is
kept so the comparison is on record, and it squashes to the eight-line form.

The other commit

The single-sequence early-out has no measurable effect and its message says so:
plan_sequence short-circuits before its added-token scan, so the only waste in the
8–16 KiB window is ~6 small allocations. Measured at 4 kB / 9 kB / 20 kB single documents
(first and last as controls, where the guard cannot fire): every cell within ±1.5%. Kept
only because it removes provably-discarded work. Drop it if you would rather have the
smaller diff.

Note on measuring this

Engine::encode in tokbench takes a single &str, so plan_work there only ever sees one
chunk, fails tasks.len() < 2, and falls back to encode_serial — the planner is
unreachable through that trait, and --scaling runs N independent single-document engines.
These numbers come from a batched harness. A batch method on Engine with a default impl
looping the single-document one would make this shape measurable in tokbench proper.

[McPatate]

either reword this or delete pls, it's not very clear