Skip to content

fix: decode str_tokens per token for transformers >= 5#177

Open
robbiebusinessacc wants to merge 1 commit into
EleutherAI:mainfrom
robbiebusinessacc:contrib/per-token-decode
Open

fix: decode str_tokens per token for transformers >= 5#177
robbiebusinessacc wants to merge 1 commit into
EleutherAI:mainfrom
robbiebusinessacc:contrib/per-token-decode

Commits

Commits on Jun 10, 2026