Conversation
Batches of sequences and of pairs, short lines and 4 KiB documents, with and without special tokens, on llama-3 (prefix-only template, type ids in the pair) and roberta (prefix, suffix and a two-token infix, no type ids). Every row of a length encodes the same bytes, so the post-processor's share shows up as the gap between rows; encode.rs cannot give it a stage row. 1 s warm-up, 2 s measurement, 50 samples: 16 rows in about 70 s. Two runs of the same build drift by 5-7 % on a laptop with 5 s windows as well, so the longer window bought nothing; the module doc says to alternate builds and compare medians. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The micro-bench named a `post_process_no_specials` that only existed in a scratch variant, so `cargo bench -p tk-encode` did not compile at HEAD. Both rows now go through `post_process::<true>` and `::<false>`. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
ArthurZucker
approved these changes
Sep 15, 2026
Collaborator
There was a problem hiding this comment.
a bit weird to have that single bench in tk-serialize, can you explain a bit the motivation?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.