fix(bpe): drop heap entries with non-positive pair_counts before u64 cast

[okxint]

Fixes #2320.

Problem

When continuing_subword_prefix (or end_of_word_suffix) and max_token_length are both set, BpeTrainer can emit a merge for a pair that occurs zero times in the corpus — a phantom merge.

Mechanism:

pair_counts is AHashMap<Pair, i32> (signed). A neighbour-delta update subtracts 1 from the pair that is being displaced. When max_token_length rejects the matching +1 side (because the would-be merged token would exceed the limit), the pair count can go negative.

The staleness re-check in the training loop reads this value through as u64 with no sign guard:

// trainer.rs — before fix
if top.count != pair_counts[&top.pair] as u64 {  // -1 casts to u64::MAX
    top.count = pair_counts[&top.pair] as u64;    // u64::MAX stored
    queue.push(top);
    continue;
}

On the next pop, the staleness check passes (both sides are u64::MAX), top.count < 1 is false, and the trainer merges a pair that nothing in the corpus could have produced — ahead of every legitimate pair, because this is a max-heap. The phantom merge silently consumes a vocabulary slot and shifts every subsequent merge rank, changing the trained tokenizer.

The two queue.push call sites correctly guard against non-positive counts:

let count = pair_counts[&pair];
if count > 0 { queue.push(Merge { ... count: count as u64, ... }); }

The staleness branch is the only path that skips the guard.

This affects WordPieceTrainer (which sets continuing_subword_prefix("##") by default) whenever max_token_length is also specified. Reproducible with 8 words / 38 bytes on tokenizers==0.23.1.

Fix

Read the live count as i32 first, drop the entry immediately if it is ≤ 0, then cast to u64 for the comparison and re-push:

let live = pair_counts[&top.pair];
if live <= 0 {
    continue;  // spent or invalidated — do not cast to u64
}
if top.count != live as u64 {
    top.count = live as u64;
    queue.push(top);
    continue;
}

Widening pair_counts to i64 (as proposed in the related #2058) does not fix this on its own — any negative value sign-extends through as u64 identically.

Test

Added bpe_no_phantom_merge_with_prefix_and_max_token_length using the 8-word corpus from the issue. It trains with continuing_subword_prefix = "##" and max_token_length = 4, then verifies that every entry in the trained merge table refers to vocabulary IDs that actually exist — a condition that would fail if a phantom merge were emitted, since the merged token would be unreachable from the corpus.