Centralize tokenizer defaults and preserve sampled inference history

[bradhilton]

Templates can trim sampled reasoning, remove old analysis, or change turn terminators when a conversation is rendered again. That breaks token-prefix continuity and forces training code to split histories even when the caller only appends a turn.

This change centralizes default tokenizer loading in art.get_tokenizer(base_model, revision=..., **kwargs) and moves Caladan’s prefix-cache implementation into ART. Shared inference helpers preserve template history and native sampled prefixes across vLLM, SGLang, and the Tinker adapters. Ordinary requests use the server-side cache without client IDs or extra headers. Explicit history-removal options remain supported; captured training IDs and log probabilities remain the actual served evidence.

The lightweight art_inference package has no import-time training dependencies. The standalone vLLM runtime bundles the same source files, including in its sdist. Caladan retains distributed cache transport and transaction handling; its integration is in OpenPipe/caladan#419.

Validation:

• Hosted CI is green at eac8d9acd: 782 training integration tests and 1,504 unit tests passed; 68 unit tests skipped.
• 690 focused ART tests passed, 9 skipped; 61 subsequent adapter/tokenizer regression tests passed.
• Real cached templates: text and tool turns for all ten pinned Caladan models; native Harmony and DeepSeek V3.2/V4 encoders checked directly.
• ART and standalone vLLM runtime sdist-to-wheel builds and isolated imports passed.
• Ruff and lock checks passed. The hosted type-check failure for the engine-only Harmony import is fixed. Hosted lint, formatting, type checks, and lock validation pass.
• Fresh GPT-6 Astra and Claude Fable 5.1 reviews completed. Subsequent GitHub reviews identified two correctness issues, now fixed: completed-history observations clear the already-spent generation budget and prompt truncation, and explicit DeepSeek chat_template=None selects the native default encoder. All 69 focused regressions pass; the reviewer’s native vLLM budget and Hugging Face default-template probes also pass.

No live GPU inference or deployment was performed. Cache misses, evictions, ambiguous tokenizations, explicit history removal, and incomplete turns can still result in multiple training histories; this change does not rewrite training evidence to conceal those cases.

[bradhilton]

Review text removed on 2026-09-17; the consolidated review record is in the summary comment on this PR.

[bradhilton]

Review text removed on 2026-09-17; the consolidated review record is in the summary comment on this PR.

[bradhilton]

Consolidated review record (automated agent review traffic removed 2026-09-17)

Change. Centralizes default tokenizer loading in art.get_tokenizer and moves Caladan's prefix-cache into ART as the art_inference package, so re-rendered conversations keep template history and sampled prefixes across vLLM, SGLang and Tinker.

Review. McCarthy and Minsky reviewed at fac2e17: McCarthy requested changes for two regressions; Minsky found source correctness clear (import isolation, moved-code parity, sdist bundling verified) with one CI blocker. All were fixed before merged head eac8d9a, where hosted CI was green (782 integration, 1,504 unit tests passed); no agent re-verdict posted there.

Findings addressed during review.

• Completed-history observation kept max_tokens, so vLLM re-reserved output budget, rejecting successful generations; now cleared with prompt truncation.
• DeepSeek V4 wrapper treated explicit chat_template=None as a custom override, rendering the literal marker; None now selects the native encoder.
• Hosted type check failed on the lazy openai_harmony import; fixed.

Deferred / follow-ups.

• Caladan-side integration (distributed cache transport, external observer): feat: Add integration to smolagents #419.
• No live GPU inference or adapter qualification performed.