Skip to content

fix(python): preserve async encode character offsets - #2369

Open
gry67673905 wants to merge 1 commit into
huggingface:mainfrom
gry67673905:codex/fix-async-encode-char-offsets
Open

fix(python): preserve async encode character offsets#2369
gry67673905 wants to merge 1 commit into
huggingface:mainfrom
gry67673905:codex/fix-async-encode-char-offsets

Conversation

@gry67673905

Copy link
Copy Markdown

Summary

  • Keep Tokenizer.async_encode consistent with Tokenizer.encode for Python string inputs by preserving character-based offsets.
  • Add a Unicode regression for "é a", where UTF-8 byte offsets differ from Python character offsets.

Validation

  • python -m pytest -q -p no:cacheprovider tests/bindings/test_tokenizer.py -k test_async_encode_char_offsets1 passed, 48 deselected
  • python -m pytest -q -p no:cacheprovider tests/bindings/test_tokenizer.py::TestTokenizer30 passed (two existing Python 3.14 fork() deprecation warnings)
  • cargo fmt --manifest-path bindings/python/Cargo.toml -- --check
  • ruff check tests/bindings/test_tokenizer.py
  • ruff format --check tests/bindings/test_tokenizer.py

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant