server: do not abort a completion when the model emits invalid UTF-8 by danielhanchen · Pull Request #202 · unslothai/llama.cpp

danielhanchen · 2026-09-06T20:49:59Z

What happens

Three different things, and only the first is a hard failure. The enumeration below separates
them.

(a) The parser rejects the text and the task is cancelled. A request can end with no tokens
at all. The server logs

W common_chat_peg_parse: unparsed Content-only output: <byte>
W srv          stop: cancel task, id_task = N
I slot      release: id X | task N | stop processing: n_tokens = 128, truncated = 0

and the client sees a 200 with a truncated SSE stream and zero content deltas. On a
non-streaming request the same thing surfaces as

{"error":{"code":500,"message":"The model produced output that does not match the expected Content-only format","type":"server_error"}}

(b) The parser accepts the text and invalid bytes travel onward. An overlong form
(C0 80, C1 BF, E0 80 AF, E0 9F BF, F0 80 80 AF, F0 8F BF BF), a surrogate
(ED A0 80, ED BF BF, a CESU-8 pair) or a code point above U+10FFFF (F4 90 80 80,
F5 80 80 80, F7 BF BF BF) is complete and correctly shaped, so common_parse_utf8_codepoint()
lets it through. Those bytes reach the JSON serialiser, which replaces them, so the client sees
U+FFFD anyway. This class is where the "no regression" evidence comes from: the bytes on the
wire are identical before and after.

(c) The text is silently truncated. An incomplete sequence that is still unresolved when
generation stops loses its trailing bytes with no marker at all.

Why

The generated text is a raw byte stream and is not guaranteed to be valid UTF-8. A byte
fallback token, or a prompt that ends in the middle of a multi-byte character, in which case
the model correctly continues with the remaining continuation bytes, makes the generated text
start with or contain bytes that do not decode.

task_result_state::update_chat_msg() hands that text to common_chat_parse() on every
token. The PEG parsers reject malformed UTF-8 by design, tests/peg-parser/test-unicode.cpp
asserts exactly that, and the std::runtime_error thrown for it is raised inside
server_response_reader::next(), which runs on the HTTP thread inside the streaming
res->next closure. It escapes into the HTTP layer, the connection is torn down, and the
reader's destructor cancels the task. The endpoint does not matter: /completion builds the
chat message too, so a plain completion request dies the same way.

validate_utf8() in the token loop only holds back a multi-byte sequence that is cut off at
the end. A byte that can never start a codepoint passes straight through to the parser.

The fix

Normalise the text before parsing: hold back the trailing bytes of an incomplete sequence
until the next chunk completes it, and replace bytes that can never form a codepoint with
U+FFFD. That is what the JSON serialiser already substitutes on the way to the client, so the
parser now sees exactly the text the client receives. generated_text in task_result_state
is only read by update_chat_msg(), so nothing else changes.

The substitution follows the Unicode "maximal subpart of an ill-formed subsequence" rule
(Unicode 15 core spec section 3.9, D93b): the longest prefix that could still have become a
well-formed sequence is replaced by one U+FFFD, and a converter "must not consume the successor
bytes as part of the ill-formed subsequence whenever those successor bytes themselves constitute
part of a well-formed UTF-8 code unit subsequence". nlohmann::json's
error_handler_t::replace does the same thing, which is why the two agree. Well-formedness is
Table 3-7 and RFC 3629 section 4: lead C2..DF for two bytes, E0 requires a first continuation
of A0..BF, ED requires 80..9F, F0 requires 90..BF, F4 requires 80..8F, and C0,
C1, F5..FF never appear.

What was measured

Reproduced deterministically with stories15M-q4_0.gguf on CPU, forcing the byte fallback
token 164 (a lone 0xA1) with a logit bias, so no GPU and no large model is needed:

curl -s http://127.0.0.1:8099/completion -d '{"prompt":"Once upon a time","n_predict":8,
  "ignore_eos":true,"stream":true,"temperature":0,"top_k":1,"logit_bias":[[164,50.0]]}'

Before: {"error":{"code":500,"message":"The model produced output that does not match the expected Content-only format"}}, zero tokens, streaming and non-streaming alike.
After: all 8 tokens are delivered, tokens_predicted = 8, content rendered as U+FFFD.

In production traffic this was one request in 64 at 32 concurrent users on Qwen3 27B
UD-Q4_K_XL with 128 token prompts drawn as arbitrary token id windows of a corpus, so a
prompt ending mid-character is common. Across 52 server logs from unrelated benchmark
windows there are 65 of these aborted requests and every single one of them contains invalid
UTF-8 in the text the parser was given: 38 with an invalid lead byte, 27 with a bad
continuation byte, none with anything else.

test-peg-parser, test-chat-peg-parser and test-chat pass unchanged. Plain completions,
chat completions and streaming chat completions were checked for regressions on the same
tiny model, and a multi-byte character split across two tokens still arrives whole.

Enumerated rather than sampled

tools/server/tests/test-server-utf8.cpp in this branch drives update_chat_msg() directly and
compares it against an independently written implementation of the WHATWG Encoding Standard
UTF-8 decoder, so "correct" is not defined by the code under test. Every case is driven twice:
once non-streaming, and once one byte at a time, which is the worst case for the three byte
hold-back buffer. It exits 1 on the parent of this branch and 0 here.

suite	inputs	before: parser threw	before: differs from the oracle
named enumeration	51	18	19
all single bytes	256	72	56
all two byte sequences	65,536	33,792	13,440
three and four byte lead sweep	204,800	167,936	26,240
biased random byte fuzz	20,000	18,824	626
total	290,643	220,642	40,381

The named enumeration includes the four sequences Unicode gives worked answers for, and this
branch matches all four: ED A0 80 gives 3 replacements (Table 3-9), F0 80 80 41 gives 3 and
keeps the A (Table 3-8), E2 80 41 gives 1 and keeps the A (Table 3-11), and
E1 80 E2 F0 91 92 F1 BF 41 gives 4 and keeps the A (Table 3-11 verbatim).

Controls that must not move, and do not

control	inputs	before	after
every valid Unicode scalar value, U+0001..U+10FFFF minus surrogates, 1,112,063 of them	1,070 batches	0 threw, 0 corrupted	0 threw, 0 corrupted
valid characters split at every byte offset, 2 and 3 chunk splits and one byte at a time	487 drives	0 threw, 0 corrupted	0 threw, 0 corrupted

Zero cases where the one-byte-at-a-time stream disagreed with the single-shot parse.

Comparing the bytes the client actually receives, by putting each result through
dump(..., error_handler_t::replace) the way safe_json_to_str() does: 26 of the 51 named
cases are byte identical before and after. That is every valid case plus every class (b) case
above, because the serialiser was already replacing those bytes. Class (c) is the only
client-visible change outside the failure class: a truncated tail now yields U+FFFD instead of
disappearing, which is what the standard recommends.

End to end

40 fault requests on stories15M-q4_0.gguf: 8 forced byte fallback tokens (164 = 0xA1,
131 = 0x80, 198 = 0xC3, 229 = 0xE2, 240 = 0xED, 247 = 0xF4, 251 = 0xF8, 258 = 0xFF;
token id is byte + 3 in this SPM vocabulary) across chat non-streaming, chat streaming and plain
/completion, plus n_predict: 1 so the fault lands on the final token with nothing after it to
complete it.

	before	after
failed, 500 or a 500 inside the stream	24 of 40	0 of 40
well formed 200	16	40
`cancel task` log lines	28	0
`unparsed Content-only` log lines	8	0

Regression controls on the same pair of servers: chat non-streaming, chat streaming with the
deltas concatenated, /completion, and /completion with n_probs: 5 are all identical, and
the prefix cache reports the same cache_n and tokens_cached on both.

Valid multi-byte generation, qwen25c15b-q8.gguf on CPU, greedy, seed 42, prompts in seven
scripts so the model generates rather than echoes: Chinese 288 non-ASCII bytes, Japanese 369,
Korean 333, Russian 408, Arabic 440, emoji 384, plus a mixed CJK/Greek/Hebrew/emoji case.
2,253 non-ASCII bytes, byte identical before and after, streamed and non streamed alike.

The full tools/server/tests pytest suite gives 361 passed on both arms, with the same six
failures on both, all of them -DLLAMA_OPENSSL=OFF in that build config.

Follow up this does not fix

A trailing incomplete sequence is still dropped in stream mode, where non-streaming now
substitutes for it. validate_utf8() at server-context.cpp:1745 reports the cut-off tail, so
process_token() skips send_partial_response() for that token entirely, and
send_final_response() supplies content = "" in stream mode, which means those bytes never
reach update_chat_msg() at all.

Streaming dropped them before this change too, so nothing is lost that was not lost already:
end to end with a forced lead byte at the end of generation, 12 streaming configurations
(n_predict 1 and 4, lead bytes 0xC3 0xE2 0xF4, /completion and
/v1/chat/completions) give empty content on both arms. What is new is the divergence, because
only the non-streaming side is fixed here. Closing it means sending bytes that
process_token() deliberately withholds, which changes what every streaming client receives,
so it belongs in its own change rather than behind a "do not abort" fix.

test-server-utf8 pipeline pins the current behaviour: it models the validate_utf8() hold-back
rather than calling update_chat_msg() directly, and asserts that no decodable content is
lost in either mode (0 before, 0 after). A complete but ill formed sequence at the end
(surrogate, overlong, above U+10FFFF, stray continuation byte) is not held back, so both modes
see it and both substitute identically; the divergence is confined to the incomplete-suffix case.

Platforms not executed

Windows, macOS and x86-64 were not run. append_utf8_sanitized() and its two helpers are byte at
a time on unsigned char, with an explicit static_cast<unsigned char> on every char, so
signed-char platforms are handled. There is no locale, no wchar_t and no
MultiByteToWideChar, and the one Windows-specific UTF-8 concern in this area, console code page
translation, is not on this path: the text goes to a socket through nlohmann::json::dump, not
to a console.

The generated text is a raw byte stream and is not guaranteed to be valid UTF-8. A byte fallback token, or a prompt that ends in the middle of a multi-byte character (the model then continues with the remaining continuation bytes), makes the generated text start with, or contain, bytes that do not decode. task_result_state::update_chat_msg() hands that text to common_chat_parse() on every token. The PEG parsers reject malformed UTF-8 by design, and the std::runtime_error thrown for it propagates out of the streaming loop, closes the connection and cancels the task, so the request ends with no tokens at all. On a non-streaming request it turns into a 500. The endpoint does not matter: /completion parses the text for the chat message as well. Normalise the text before parsing: hold back the trailing bytes of an incomplete sequence until the next chunk completes it, and replace bytes that can never form a codepoint with U+FFFD. That is what the JSON serialiser already substitutes on the way to the client, so the parser now sees exactly the text the client receives.

danielhanchen · 2026-09-06T22:31:50Z

Hardware evidence, since the report above was proved on CPU with a 19 MB model and a forced logit bias.

Reproduced and cured on a DGX Spark GPU with a real quantised model, using the harness and prompt shape that produced the original observation: Qwen3.5-4B UD-Q4_K_XL, -ngl 99 -fa on -c 16384 --parallel 32 --cache-ram 0 --kv-unified, 64 requests at 32 concurrent, prompts drawn as 128 token windows of a text corpus, n_predict 256. Two CUDA builds from one base, fbf9abcc7 and the same tree plus this commit.

Before: 63 of 64 requests return tokens, in six independent arms across two cells. The server log carries the signature three times, once per arm:

0.29.450.277 W common_chat_peg_parse: unparsed Content-only output: \xa1
0.29.450.738 W srv          stop: cancel task, id_task = 310
0.29.455.385 I slot      release: id 10 | task 310 | stop processing: n_tokens = 128, truncated = 0

n_tokens = 128 is the prompt alone, against 383 for every healthy request in the same cell (189 of them in that cell, plus 67 at 135 from the serial probe).

After: 64 of 64 in every arm, and zero occurrences of the warning.

The failing byte is 0xA1 and it is now pinned down exactly. Replaying the same prompt draws serially isolates it to one request, at corpus token offset 48080. The corpus there is a markdown table of status emoji, and 🟡 is three tokens in this vocabulary: 10838 = 20 F0 9F, 253 = 9F, 94 = A1. The 128 token window ends on 253, so the prompt stops after F0 9F 9F and the model correctly emits 94, a lone 0xA1, as its first generated token. Served serially that same request is a clean 500 on the base build and a 200 on this one, so the defect needs no concurrency at all; concurrency only decides which draw happens to straddle a character.

With the fix the request returns n_predict 8 tokens and the content "� | ✅ | ✅ | ✅ |", first bytes ef bf bd 20 7c 20 e2 9c 85. The U+FFFD is in the first position and nowhere else, which is exactly right: the orphaned continuation byte can never be completed because its lead bytes are in the prompt and were never in generated_text, while the multi-byte ✅ later in the same chunk arrives intact and the model goes on producing the table row it was in the middle of.

Cost. Bracketed base / fixed / base at 32 concurrent, three arms of 64 requests per cell, one lock hold, GPU clocks pinned at 1690 MHz for the whole block, every arm 1677 to 1683 MHz, cell maxima 54 C, no clock transition:

cell	build	ok	agg tok/s	median
base	`fbf9abc`	63, 63, 63	336.27, 333.35, 332.86	333.35
fixed	+ this PR	64, 64, 64	336.35, 333.78, 332.36	333.78
base	`fbf9abc`	63, 63, 63	336.91, 333.47, 335.28	335.28

The fixed cell lands between the two base cells, which themselves differ by 0.58 percent.

The per-token figure in that bracket is confounded, because the base arm loses a slot in its first decode step and then runs 31 requests where the fixed arm runs 32. So the cost was measured again on a shape where both builds complete all 64, concurrency 8 with 8 requests per client, same pin, same block:

cell	build	ok	agg tok/s	median TPOT ms
base	`fbf9abc`	64, 64	204.59, 201.52	36.99, 37.38
fixed	+ this PR	64, 64	207.03, 201.64	36.47, 37.34
base	`fbf9abc`	64, 64	206.20, 201.81	36.85, 37.16

With identical work on both sides the fixed build is inside the 0.46 percent drift between the two base cells and the TPOT distributions are indistinguishable. append_utf8_sanitized() scans only the bytes added since the last token, so the added work is linear in the generated bytes and is dwarfed by the full re-parse of the accumulated text that update_chat_msg() already did per token before this change.

danielhanchen · 2026-09-09T01:36:58Z

@codex security review

chatgpt-codex-connector · 2026-09-09T01:37:16Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-09T09:02:59.986131Z	`ff1d590`	Manual request

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

append_utf8_sanitized holds the tail of an incomplete sequence back so the next chunk can complete it, but on the last update nothing more is coming. A generation that stopped mid character therefore left those bytes in generated_text_pending forever, and because the parsed message is preferred over the raw content whenever it is nonempty, the response silently dropped them: 'hello\xC3' was returned as 'hello' rather than the 'hello' plus U+FFFD that dump_safe produces. One U+FFFD, not one per byte. nlohmann's replace handler emits a single replacement character per incomplete sequence, checked against the vendored header: '\xC3', '\xE2' and '\xE2\x82' each render as one, while '\xC3\xC3' renders as two because the first is terminated by the second lead byte. What stays pending is by construction exactly one incomplete sequence, since the loop consumes everything decidable before it breaks. Gated on the final update, so the streaming path is untouched.

Two defects in the previous commit's own flush, both raised in review and both real. A trailing lead byte followed by too few bytes, such as "\xE2A", is reported INCOMPLETE by common_parse_utf8_codepoint, which returns before checking that what followed the lead was even a continuation. The flush emitted one replacement character and cleared the buffer, so the valid A was discarded. A malformed multibyte prefix was replaced a byte at a time: "\xE2\x80A" gave two replacement characters where the serialiser gives one, because consuming only the lead byte left the continuation to be re-read as a second invalid sequence. Both now go through utf8_malformed_prefix, which consumes one failed sequence the way nlohmann's replace handler does: the lead byte plus the continuation bytes that actually followed it, never past the length the lead announced, with a bare continuation counting as a failed sequence on its own. Checked against the vendored header by dumping with error_handler_t::replace rather than from memory: hello \xE2 A -> hello U+FFFD A hello \xE2\x80 A -> hello U+FFFD A hello \xC3\xC3 -> hello U+FFFD U+FFFD a \xF0\x9F\x98 -> a U+FFFD \x80\x80 -> U+FFFD U+FFFD This implementation reproduces all five. The finality now reaches the helper as a flag rather than being handled by a separate block after it, so INCOMPLETE on the last call resolves through the same path.

E0, ED, F0 and F4 restrict the first continuation byte. The serialiser's decoder rejects the sequence at that byte and reprocesses it, so E0 80, ED A0, F0 80 and F4 90 each render as two replacement characters, not one. Treating every 10xxxxxx byte as part of the malformed prefix emitted one and disagreed with the client.

common_parse_utf8_codepoint checks continuation SHAPE only, never the lead's range, so it reports SUCCESS for C0 80, ED A0 80 and F4 90 80 80. Copying those through was not harmless. The JSON serialiser replaces them, so a client was shown something different from what a parser received, and the AST dump under params.debug is a plain dump() with no error handler, which throws on exactly these bytes and aborts the completion this change exists to keep alive. The check goes where the sanitising happens, not in the shared parser, which has other callers in common/trie.cpp and common/common.cpp. The lead table that utf8_malformed_prefix already carried is now shared with it, so the accept and the replace boundaries cannot drift apart. Differential tested against the serialiser over every 1- and 2-byte string plus 3-byte strings across 16 leads, 788766 inputs after excluding the ones JSON escapes rather than replaces: 0 mismatches, using the repo parser. The same test reported 34688 mismatches before this commit.

chatgpt-codex-connector · 2026-09-09T04:10:19Z

Codex Review: Didn't find any major issues. 👍

Reviewed commit: 9b2955e3e6

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Post-convergence pass, 42 comment lines to 12. Two sweeps: the first removed what the code already says, the second kept only what a reader cannot recover from the code, which is why the serialiser's replacement boundaries and the debug dump are still spelled out. Also puts the append_utf8_sanitized description back above that function. An earlier edit here left it stranded above utf8_lead_bounds with a duplicate of the prefix helper's description below it. Comments only, verified with comment_tools check; llama-server rebuilt.

chatgpt-codex-connector · 2026-09-09T04:43:44Z

Codex Review: Didn't find any major issues. Chef's kiss.

Reviewed commit: 6dc4a969d0

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

No code change: every remaining line carries a fact the code does not state.

danielhanchen · 2026-09-09T06:18:10Z

@codex review

chatgpt-codex-connector · 2026-09-09T06:24:33Z

Codex Review: Didn't find any major issues. Nice work!

Reviewed commit: b27c0b4427

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

…ATWG decoder

danielhanchen · 2026-09-09T08:42:46Z

Simulation to destruction

Put through a deliberate fault-injection bar rather than a repetition bar: the fault is injected, never waited for. Base fbf9abcc7 against head b27c0b442, CPU only, no GPU at any point.

New in this branch: tools/server/tests/test-server-utf8.cpp, which drives task_result_state::update_chat_msg() directly and compares it against an independently written WHATWG Encoding Standard UTF-8 decoder (https://encoding.spec.whatwg.org/#utf-8-decoder), so "correct" is not defined by the code under test. It exits 0 on this branch and 1 on the base commit.

Spec basis

Well-formedness is Unicode 15 core spec Table 3-7 and RFC 3629 §3/§4: lead C2..DF for two bytes, E0 requires a first continuation of A0..BF, ED requires 80..9F, F0 requires 90..BF, F4 requires 80..8F, and C0, C1, F5..FF never appear. The replacement policy is Unicode §3.9 D93b, "maximal subpart of an ill-formed subsequence": the maximal subpart is replaced by a single U+FFFD and a converter "must not consume the successor bytes as part of the ill-formed subsequence whenever those successor bytes themselves constitute part of a well-formed UTF-8 code unit subsequence". nlohmann::json's error_handler_t::replace does the same thing (dump_escaped() rolls back to bytes_after_last_accept and re-examines the offending byte), which is why this change and the serialiser agree.

This branch matches the maximal-subpart rule exactly, including the four sequences that Unicode gives worked answers for: ED A0 80 -> 3 replacements (Table 3-9), F0 80 80 41 -> 3 + A (Table 3-8), E2 80 41 -> 1 + A (Table 3-11), E1 80 E2 F0 91 92 F1 BF 41 -> 4 + A (Table 3-11 verbatim).

The flip pairs

Same harness source on both arms. threw is the PEG parser rejecting the text, which is what cancels the request.

suite	inputs	base threw	base differs from oracle
named enumeration	51	18	19
all single bytes	256	72	56
all two byte sequences	65,536	33,792	13,440
three and four byte lead sweep	204,800	167,936	26,240
biased random byte fuzz	20,000	18,824	626
total	290,643	220,642	40,381

Controls that must not move:

control	base	head
every valid Unicode scalar U+0001..U+10FFFF minus surrogates, 1,112,063 of them	0 threw, 0 corrupted	0 threw, 0 corrupted
valid characters split at every byte offset, 2 and 3 chunk splits and one byte at a time, 487 drives	0 threw, 0 corrupted	0 threw, 0 corrupted

Every named case was also driven one byte at a time, which is the worst case for the three byte hold-back buffer. Zero cases where the streamed result differed from the single-shot result.

Base does three different things, not one

The description covers one of them. The enumeration shows three, and only the first is a failure:

class	inputs	base	head
(a) parser throws, task cancelled	stray continuation bytes, leads `F8..FF`, maximal-subpart mixes	HTTP 500, or a truncated SSE stream whose last event is a 500	well-formed 200 with U+FFFD
(b) invalid bytes passed through	overlong (`C0 80`, `C1 BF`, `E0 80 AF`, `E0 9F BF`, `F0 80 80 AF`, `F0 8F BF BF`), surrogates (`ED A0 80`, `ED BF BF`, CESU-8 pairs), above U+10FFFF (`F4 90 80 80`, `F5 80 80 80`, `F7 BF BF BF`)	parser accepts them, the serialiser replaces them later	replaced earlier, identical bytes on the wire
(c) silently truncated	an incomplete sequence still unresolved when generation stops	trailing bytes vanish with no marker	one U+FFFD

Class (b) is the strongest no-regression evidence: I compared the bytes the client actually receives, obtained by putting each result through dump(..., error_handler_t::replace) the way safe_json_to_str() does. 26 of the 51 named cases produce byte-identical wire output on base and head, which is every valid case plus every class (b) case. Class (c) is the only client-visible behaviour change outside the failure class, and emitting U+FFFD there is what the standard recommends.

End to end, real server, real model

stories15M-q4_0.gguf on CPU, byte fallback forced with logit_bias (token id = byte + 3 in this SPM vocabulary, so 164 = 0xA1 as in the description, plus 131 = 0x80, 198 = 0xC3, 229 = 0xE2, 240 = 0xED, 247 = 0xF4, 251 = 0xF8, 258 = 0xFF). 40 fault requests: 8 forced tokens across chat non-streaming, chat streaming and plain /completion, plus n_predict: 1 so the fault lands on the final token with nothing after it to complete it.

arm	failed (500, or 500 inside the stream)	well-formed 200	`cancel task` log lines	`unparsed Content-only` lines
base `fbf9abcc7`	24 of 40	16	28	8
head `b27c0b442`	0 of 40	40	0	0

One correction the description needs: plain /completion is affected too, not only the chat endpoints. server_task_result_cmpl_final::update() runs for every inference task regardless of endpoint, so on base all 8 forced byte tokens returned

{"error":{"code":500,"message":"The model produced output that does not match the expected Content-only format","type":"server_error"}}

from /completion. All 8 return 200 on this branch.

No-regression controls

control	base vs head
`/v1/chat/completions` non streaming, no bias	content identical
`/v1/chat/completions` streaming, deltas concatenated	identical
`/completion` non streaming	identical
`/completion` with `n_probs: 5`	`completion_probabilities` identical
prefix cache, same prompt twice with `cache_prompt: true`	`cache_n` 5 then 9 and `tokens_cached` 17 on both arms

Valid multi-byte generation, qwen25c15b-q8.gguf on CPU, greedy, seed 42, prompts in seven scripts so the model genuinely generates rather than echoes: Chinese 288, Japanese 369, Korean 333, Russian 408, Arabic 440, emoji 384 and a mixed CJK/Greek/Hebrew/emoji case, 2,253 non-ASCII bytes, byte identical between the arms, streamed and non streamed alike.

tools/server/tests pytest, -k "not slow", curl-enabled builds: 6 failed, 361 passed, 6 skipped on base and identically on this branch. The same 6 failures on both, all of them -DLLAMA_OPENSSL=OFF in my config (two test_router HTTPS downloads, four test_vision_api image URLs). test-chat, test-peg-parser and test-chat-peg-parser pass on both.

Answers to the four questions

What happened before. Three things. A completion containing an ill-formed sequence the PEG parser rejects is cancelled: 500 non-streaming, truncated SSE with a 500 as the last event streaming, on chat and on /completion. A completion containing an overlong form, a surrogate or an out-of-range code point is not rejected; those bytes reach the JSON serialiser, which replaces them. A completion ending mid-character loses the trailing bytes silently.
What happens after. All three normalised at one point, to exactly what the serialiser would have produced, following the Unicode maximal-subpart rule.
Real or fake. Real. 220,642 of 290,643 enumerated inputs reproduce it at the unit level and 24 of 40 requests reproduce it end to end. Byte fallback tokens are how every SPM vocabulary represents bytes it cannot tokenise, so this is not a synthetic-only path.
Does merging break anything. Nothing found. 1,112,063 valid scalar values round trip byte identically, characters split at every byte offset survive, 26 of 51 named cases give identical wire bytes, all endpoint controls are identical, prefix caching is unchanged and the pytest suite is identical.

What I could not run

Windows and macOS: no machine for either. Reasoning from source, append_utf8_sanitized() and its two helpers are byte at a time on unsigned char with explicit static_cast<unsigned char> on every char, so signed-char platforms are handled; there is no locale, no wchar_t, no MultiByteToWideChar, nothing that differs on Windows. The one Windows-specific UTF-8 concern in this area, console code page translation on stdout, is not on this path: the text goes to a socket through nlohmann::json::dump, not to a console. x86-64 was not run either; nothing in the diff is architecture dependent.

danielhanchen · 2026-09-09T08:49:24Z

Retraction of one claim in my comment above. I wrote that the description needed correcting because it only mentioned the chat endpoints. That is wrong: the "Why" section already says "The endpoint does not matter: /completion builds the chat message too, so a plain completion request dies the same way." I read the failure symptom in "What happens" and did not read on. The measurement stands, all 8 forced byte tokens returned a 500 from /completion on the parent and 200 here, but it confirms what the description already said rather than correcting it.

The description has now been updated with the parts that were genuinely missing: the three distinct behaviours before this change rather than one, the 290,643 input enumeration against an independent WHATWG decoder, the 1,112,063 valid scalar round trip control, the 26 of 51 named cases that are byte identical on the wire, and the platforms I could not execute.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a5ac51d890

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-09-09T08:49:48Z

        bool filter_tool_calls) {
-    generated_text += text_added;
+    // Nothing can complete a held-back sequence after the last update; unresolved, it was dropped.
+    append_utf8_sanitized(generated_text, generated_text_pending, text_added, !is_partial);


Preserve trailing malformed bytes in streaming output

When a streamed completion ends with an incomplete UTF-8 sequence, this final flush cannot replace it because server-context.cpp detects the incomplete suffix at lines 1744–1749 and never sends those bytes in a partial result; the streaming final result then supplies an empty content at lines 2002–2005. Consequently generated_text_pending is empty here, so the malformed suffix is silently omitted rather than emitted as U+FFFD, while the equivalent non-streaming request is sanitized. This occurs when EOS or the token limit follows a token ending in a UTF-8 lead/prefix, and the new test misses it because it calls update_chat_msg() directly instead of exercising the server token pipeline.

Useful? React with 👍 / 👎.

Correct on the mechanism, and it is now pinned by a test. Measured rather than reasoned, on both arms.

What the streaming path actually does. validate_utf8() at server-context.cpp:1745 scans the last up to four bytes of slot.generated_text for a lead byte and returns len - i if the sequence is cut off, so incomplete is true and process_token() skips send_partial_response() for that token entirely. send_final_response() then sets content = "" in stream mode. So generated_text_pending is indeed empty at the final flush and the tail is dropped.

But it was dropped before this change too, so nothing is lost that was not lost already. End to end on stories15M-q4_0.gguf, logit_bias forcing a lead byte so generation ends on one, 12 streaming configurations (n_predict 1 and 4, three lead bytes 0xC3 0xE2 0xF4, /completion and /v1/chat/completions):

before after

streaming, trailing incomplete sequence empty content, 12 of 12 empty content, 12 of 12

non streaming, same inputs HTTP 500 or bytes dropped U+FFFD

With four forced 0xC3 tokens the stream carries exactly one SSE event, the final one with "content":"", on both arms: no partial is ever emitted for a run of lead bytes, because validate_utf8() holds the last one back every time.

So what this change creates is a divergence, not a loss: the non-streaming side is now substituted and the streaming side still is not, where before both were lossy. That is worth fixing, and the fix is in process_token() and send_final_response() rather than here: it means sending bytes that are currently deliberately withheld, which changes what every streaming client receives. Widening this PR to do that would put a streaming-content change behind a "do not abort" fix, so I have left it out and noted it as a follow up instead.

On the test missing it. Fair, and fixed in ff1d5901e. test-server-utf8 pipeline now models process_token()'s hold-back rather than calling update_chat_msg() directly: it accumulates a slot text, applies the real validate_utf8() gate, sends only what the slot would send, then does the empty final flush, and compares that against the same text driven non-streaming. 13 cases.

after PIPE tail-c3 text=[61 62 63 C3] held_back=[C3] stream=[61 62 63] nostream=[61 62 63 EF BF BD] stream-ok nostream-ok ASYMMETRIC PIPE run-c3 text=[C3 C3 C3 C3] held_back=[C3 C3 C3 C3] stream=[] nostream=[EF BF BD x4] stream-ok nostream-ok ASYMMETRIC PIPE surrogate text=[61 62 63 ED A0 80] held_back=[] stream=[61 62 63 EF BF BD x3] nostream=[same] stream-ok nostream-ok agree PIPE stray-80 text=[61 62 63 80] held_back=[] stream=[61 62 63 EF BF BD] nostream=[same] stream-ok nostream-ok agree RESULT pipeline stream_lost_content=0 nostream_bad=0 asymmetric=6 before PIPE tail-c3 stream=[61 62 63] nostream=[61 62 63] stream-ok NOSTREAM-BAD agree PIPE run-c3 stream=[] nostream=[THROW] stream-ok NOSTREAM-BAD ASYMMETRIC PIPE surrogate stream=[61 62 63 ED A0 80] nostream=[61 62 63 ED A0 80] STREAM-LOST-CONTENT NOSTREAM-BAD agree PIPE stray-80 stream=[THROW] nostream=[THROW] STREAM-LOST-CONTENT NOSTREAM-BAD agree RESULT pipeline stream_lost_content=4 nostream_bad=10 asymmetric=2

The hard assertion is that no decodable content may be lost in either mode: 0 before, 0 after, on both sides. The asymmetric rows are reported and deliberately not counted as failures, with the reason stated in the output. Note the tail-* rows are agree before the change because both modes dropped the tail, and ASYMMETRIC after because only one of them was fixed, which is exactly the point you are making.

Also worth noting from the same table: a complete but ill formed sequence at the end (surrogate, overlong, above U+10FFFF, stray continuation byte) is not held back by validate_utf8(), so both modes see it and both substitute identically. The divergence is confined to the incomplete-suffix case.

danielhanchen · 2026-09-09T08:55:12Z

@codex review

chatgpt-codex-connector · 2026-09-09T09:02:58Z

Codex Review: Didn't find any major issues. Keep them coming!

Reviewed commit: ff1d5901e7

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

danielhanchen · 2026-09-09T13:38:53Z

Independent confirmation of the production claim, from a run that was not testing for it.

While measuring PRs 200 and 201 on this pair, the load generator was building each prompt as a window of random token ids. That is exactly the condition this PR describes: a window frequently lands on a byte-fallback token or splits a multi-byte character. Three separate servers, none of them carrying this fix, aborted in the warmup pass:

base at fbf9abcc7
the PR 200 branch
the PR 201 branch

All three returned the same thing on Qwen3.8-27B UD-Q4_K_XL at 32 slots:

{'code': 500, 'message': 'The model produced output that does not match the expected Content-only format', 'type': 'server_error'}

Six occurrences across cells p200_base, p200_new, p200_base2, p201_new and t201_base, between 05:26 and 05:39. The only way to get the other two PRs measured on unpatched servers at all was to switch the generator to text prompts.

Worth recording because nothing about that run was constructed to make this point. The failure was in the way of measuring something else, which is a stronger signal than a purpose-built reproduction: it says the condition is easy to reach by accident on an ordinary model at ordinary settings, not just under a crafted input.

unslothai deleted a comment from chatgpt-codex-connector Bot Sep 9, 2026

This comment was marked as resolved.

Sign in to view

unslothai deleted a comment from chatgpt-codex-connector Bot Sep 9, 2026

This comment was marked as resolved.

Sign in to view

unslothai deleted a comment from chatgpt-codex-connector Bot Sep 9, 2026

tighten the comments added by this change

b27c0b4

No code change: every remaining line carries a fact the code does not state.

server: unit test for the UTF-8 sanitising, against an independent WH…

a5ac51d

…ATWG decoder

chatgpt-codex-connector Bot reviewed Sep 9, 2026

View reviewed changes

server: pin the token pipeline case, streaming against non streaming

ff1d590

	before	after
streaming, trailing incomplete sequence	empty content, 12 of 12	empty content, 12 of 12
non streaming, same inputs	HTTP 500 or bytes dropped	U+FFFD

Uh oh!

Conversation

danielhanchen commented Sep 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What happens

Why

The fix

What was measured

Enumerated rather than sampled

Controls that must not move, and do not

End to end

Follow up this does not fix

Platforms not executed

Uh oh!

danielhanchen commented Sep 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

danielhanchen commented Sep 9, 2026

Uh oh!

chatgpt-codex-connector Bot commented Sep 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

chatgpt-codex-connector Bot commented Sep 9, 2026

Uh oh!

chatgpt-codex-connector Bot commented Sep 9, 2026

Uh oh!

danielhanchen commented Sep 9, 2026

Uh oh!

chatgpt-codex-connector Bot commented Sep 9, 2026

Uh oh!

danielhanchen commented Sep 9, 2026

Simulation to destruction

Spec basis

The flip pairs

Base does three different things, not one

End to end, real server, real model

No-regression controls

Answers to the four questions

What I could not run

Uh oh!

danielhanchen commented Sep 9, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Sep 9, 2026

Choose a reason for hiding this comment

Uh oh!

danielhanchen Sep 9, 2026

Choose a reason for hiding this comment

Uh oh!

danielhanchen commented Sep 9, 2026

Uh oh!

chatgpt-codex-connector Bot commented Sep 9, 2026

Uh oh!

danielhanchen commented Sep 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

danielhanchen commented Sep 6, 2026 •

edited

Loading

danielhanchen commented Sep 6, 2026 •

edited

Loading

chatgpt-codex-connector Bot commented Sep 9, 2026 •

edited

Loading