Conversation
ArthurZucker
approved these changes
Sep 2, 2026
Comment on lines
+1211
to
+1215
| #[test] | ||
| fn truncation_makes_room_for_the_specials_the_template_adds() { | ||
| let pipeline = pipeline_with( | ||
| bert_post_processor(), | ||
| truncation(5, TruncationStrategy::LongestFirst), |
Collaborator
There was a problem hiding this comment.
same to move out imo!
| pub fn slices(&self) -> &[Slice] { | ||
| &self.slices | ||
| } | ||
|
|
Comment on lines
+83
to
+87
| // XXX: maybe we should error out when instantiating the PipelineTokenizer to avoid this | ||
| warn!( | ||
| "Truncation max_length is too short to include the tokens: `max_length` is {}, the post-processor adds {num_added_special_tokens} special tokens. Returning an empty sequence", | ||
| truncation.max_length | ||
| ); |
Collaborator
There was a problem hiding this comment.
yeah, we can break in v1, I'd say let's make it make sense?
Comment on lines
+12
to
+16
| // A `stride` asks for what the released crate returns as `overflowing` encodings: windows of | ||
| // `max_length` tokens that consecutive windows share `stride` of. The pipeline keeps the first | ||
| // window and drops the rest, so the value is read and carried rather than refused. 395 of the | ||
| // 5,053 popular Hub configs surveyed carry one, all question-answering fine-tunes, and | ||
| // transformers rebuilds truncation from the per-call `stride` anyway, so the file's value never |
Base automatically changed from
feat/truncation-padding
to
feat/train_encode_split
September 2, 2026 11:13
SBrandeis
force-pushed
the
feat/truncation
branch
from
September 10, 2026 09:13
00c9793 to
89ee3dd
Compare
SBrandeis
changed the base branch from
feat/train_encode_split
to
feat/rework-python-bindings
September 10, 2026 09:16
SBrandeis
force-pushed
the
feat/truncation
branch
3 times, most recently
from
September 15, 2026 13:14
14700c0 to
f3d8d5f
Compare
Base automatically changed from
feat/rework-python-bindings
to
feat/train_encode_split
September 15, 2026 15:24
SBrandeis
force-pushed
the
feat/truncation
branch
from
September 15, 2026 15:59
541cb95 to
4e68e4b
Compare
SBrandeis
marked this pull request as ready for review
September 15, 2026 16:00
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implement truncation in the
PipelineTokenizer.encode()pathstride != 0(ie windowing) is not supported yet