streaming: publish partial text as increments, through one shared publisher

[christopherthompson81]

A streaming session's partial_text is the text decoded since the last one. app/cli/partial_render.h says so and the CLI appends them into a scrolling transcript; the reference server forwards each as an OpenAI-shaped transcript.text.delta, which is incremental by specification.

Two families published the whole running transcript instead. A consumer that appends — the CLI, or any client written against the OpenAI shape — built the transcript up quadratically:

partial_text=Some call me nat
partial_text=Some call me nature. Others call me
partial_text=Some call me nature. Others call me Mother Nature. I'
...

giving Some call me natSome call me nature. Others call me….

parakeet_tdt published merged_decode(). kroko_asr published result.text_output from combined_decoded(). Both now publish the increment, and the CLI output concatenates byte for byte to the run's own text_output.

Scope

Four families, per review:

family	why
parakeet_tdt	restated the whole transcript as a partial
kroko_asr	same, found by sweeping for the pattern rather than by hitting it
higgs_audio_stt	carried its own copy of the prefix diff…
vibevoice_asr	…duplicated verbatim, helper and caller both

voxtral_realtime, qwen3_asr and sense_asr keep their own byte-offset implementations. They were already publishing partials correctly and were optimised by the teams that own them, so migrating them would have been consistency rather than a fix. The net diff does not touch them.

engine::runtime::PartialTextPublisher is the single implementation for the four. It holds what has actually been published and returns the increment, with two properties the duplicated copies lacked:

• A delta never ends part way through a UTF-8 sequence. higgs_audio_stt and vibevoice_asr are ByteLevel, with all 256 single-byte tokens in vocabulary. Taking Higgs's real 151,643-entry vocabulary through the engine's own decode path:

  'ᚠ' U+16A0  ->  3 byte tokens [157, 248, 254]
    after token 1:  e1        INVALID UTF-8
    after token 2:  e19a      INVALID UTF-8
    after token 3:  e19aa0    valid: 'ᚠ'
  

  Five of fourteen characters tested (Runic, Devanagari, Hangul Jamo Extended, Gothic, CJK Compatibility) have no whole-character token, so the tokenizer must emit them as byte sequences — and text_decoder.cpp calls token_callback after every generated token, so steps 1 and 2 go on the wire. parakeet_tdt (Metaspace, no <0xNN> pieces) and kroko_asr (plain pieces) cannot produce this, and for them the check is a no-op that costs one bounded memcmp.

• A delta never starts inside one either, when a decode revises published text rather than only extending it. On a genuine revision nothing can retract a delta already sent, so the consumer is wrong either way; this stops it also being spliced into the middle of a character.

publish() is O(1) in transcript length — flat at ~0.025 µs per update whether the transcript is 5 KB or 200 KB, against ~0.007 µs for a byte offset allocating the same delta string.

word_timestamps deliberately stays cumulative alongside the now-incremental partial_text. It is not a delta field — it is the finalized set so far, which is why the provisional last word is dropped. Each field matches its own contract rather than each other; commented in place.

No server change. run_transcription_stream already skips an event with no partial text, and emit_if_nonempty drops an event only when every field is empty, so a window that decodes no new text still delivers its word timestamps.

kroko_asr's final result depended on the bug

The one change here that is not mechanical. Its finalize() was:

auto event = process_streaming_audio(true);
result.text_output = event.partial_text;

reading the whole transcript out of the partial — which worked only while the partial restated it. Making partials incremental therefore broke the final result: eleven correct deltas, then text_output = " times longer than you". Both sides are std::optional<Transcript>, so it compiles equally well either way and nothing in the tree covered it. finalize() now builds through make_result(), the same path the offline route uses.

The only other place that reads a transcript out of a partial is src/capi/audiocpp.cpp:226, which is correct and documented — it presents a stream event's partial through that event's text accessor, which is what a delta is.

Verification

Every family driven through the CLI on assets/resources/sample_16k.wav, including the three left on their own implementations:

family	partials	concatenate to text_output
parakeet_tdt	7	yes
kroko_asr	11	yes
higgs_audio_stt	4	yes
vibevoice_asr_streaming	5	yes
voxtral_realtime (own impl)	33	yes
qwen3_asr (own impl)	1	yes
sense_asr (own impl)	1	yes

Plus tests/unittests/test_partial_text.cpp: growth, 2/3/4-byte characters arriving one byte at a time, a revision inside the agreement window and one behind it, a truncated decode that regrows, a shrinking transcript, reset(), empty edges.

What is not covered

• The byte-fallback path is covered by unit test and by vocabulary analysis, not by a live multilingual decode. I have no sample that drives a tokenizer into fallback, so the held-back-character path has not run against a real model — only its inputs have been shown to be reachable.
• nemotron_asr and the two vibevoice_asr chunk-append sites are deliberately untouched: they receive deltas produced upstream rather than diffing a running transcript, so the publisher does not apply.

Reported downstream as christopherthompson81/AudioCpp-Bindings#68.

[0xShug0]

@christopherthompson81 I think it’s safe to migrate higgs_audio_stt, vibevoice_asr, parakeet_tdt, and kroko_asr to the shared publisher, since they either already have the same computational complexity or currently emit partials incorrectly.

However, Voxtral Realtime and Sense ASR have already been optimized directly by their respective teams, as have Granite ASR, Fun ASR, and VibeASR. Qwen3 ASR is also used in production pipelines. Since the shared publisher scans the entire transcript on every update, replacing the existing byte-offset approach could introduce a performance regression. We shouldn’t change already-correct, optimized implementations just for consistency, especially if doing so introduces a performance regression.

The framework streaming helpers are better suited to models without native streaming support, where we need to add pseudo-streaming.

Another issue is UTF-8. The UTF-8 unit test manually constructs incomplete byte sequences. The real question is: do the affected models actually produce such incomplete byte sequences through their tokenizer and streaming paths? This feel like a synthetic case, not a real issue.

For example, Qwen3-ASR outputs complete characters:

Update 1: "你"     // E4 BD A0
Update 2: "你好"   // E4 BD A0 E5 A5 BD

rather than incomplete UTF-8 fragments:

Update 1: "\xE4"
Update 2: "\xE4\xBD"
Update 3: "\xE4\xBD\xA0"  // 你

I’d suggest investigating which models actually exhibit/reproduce this issue under realistic workloads, rather than relying solely on synthetic unit tests, and making UTF-8 handling opt-in for the affected models rather than applying it broadly.

[0xShug0]

Another example: parakeet-tdt uses BPE with Metaspace decoding. Its 8,192 base vocabulary pieces are valid Unicode strings, with no byte-fallback pieces or byte-level decoding, so its token boundaries do not split UTF-8 characters.

[christopherthompson81]

Thanks — both objections were worth taking seriously, and the first one was simply right. I've pushed a commit rather than argued.

Performance: you were right, and it was incidental rather than inherent

Measured before defending. Over a 10,000-update session the publisher cost 7.3 µs per update against 0.004 µs for the byte offset, growing with transcript length. Three causes, all incidental:

• the prefix check walked byte by byte — memcmp over the shared span instead: 7.3 µs → 0.49 µs
• published_ was reassigned from the whole transcript each update, copying it every time — appended in the common case instead
• the remaining O(n) was the prefix scan, and it was buying nothing. A delta that has gone out cannot be retracted, so a divergence far behind the publish point is not something the publisher can act on even when it finds it. Checking agreement over a bounded window makes publish() O(1).

Now, with both paths allocating the delta string as the real code does:

transcript	publisher	byte-offset + substr
5 KB	0.027 µs/update	0.007 µs/update
50 KB	0.027 µs/update	0.007 µs/update
200 KB	0.024 µs/update	0.008 µs/update

Flat — same complexity class. The difference is one bounded memcmp, around 16 ns.

Behaviour on the optimised families: byte-identical

Rather than ask you to take that on trust, I drove each family that has a byte-offset implementation through the CLI on the same clip, once with its own implementation and once with the publisher, and diffed the partial sequences:

voxtral_realtime   byte-offset 33 partials | publisher 33 partials | IDENTICAL
qwen3_asr          byte-offset  1 partial  | publisher  1 partial  | IDENTICAL
sense_asr          byte-offset  1 partial  | publisher  1 partial  | IDENTICAL

So nothing optimised is degraded — same output, same complexity class. If you would still rather leave those three alone on principle, say so and I will drop them from the PR; the four you named are the ones that were actually broken. But the regression argument for excluding them no longer holds.

UTF-8: you are right about Parakeet, and it is not synthetic for two of the four you would migrate

Checked your Parakeet claim and it holds exactly — Metaspace decoder, and zero <0xNN> pieces in the vocabulary, so byte_fallback: true in its config can never actually fire. Token boundaries there cannot split a character. Same for kroko_asr, which uses plain pieces.

But higgs_audio_stt and vibevoice_asr are ByteLevel, with all 256 single-byte tokens present. Taking Higgs's real 151,643-entry vocabulary and running the engine's own decode path (decode_byte_level over the concatenated pieces):

'ᚠ' U+16A0  ->  3 byte tokens [157, 248, 254]
  after token 1:  e1        INVALID UTF-8
  after token 2:  e19a      INVALID UTF-8
  after token 3:  e19aa0    valid: 'ᚠ'

Five of fourteen characters I tested (Runic, Devanagari, Hangul Jamo Extended, Gothic, CJK Compatibility) have no whole-character token in that vocabulary, so the tokenizer must emit them as byte sequences. And text_decoder.cpp calls token_callback after every generated token, so steps 1 and 2 are partials that go on the wire.

I take the point that my unit test constructed the bytes by hand — the evidence above is from the model's actual vocabulary, and I should have gone and got it before claiming the risk.

On making it opt-in

I think it already is, by behaviour rather than by flag. For a model that emits whole characters per update — your Qwen3 example — the publisher's output is byte-identical to a byte offset:

update 1 "你"    -> emits 你
update 2 "你好"  -> emits 好

Nothing is held back, because nothing is incomplete. The hold-back only engages when a sequence is genuinely mid-character, and it releases on the very byte that completes it:

after byte 1 of 你   (nothing emitted)
after byte 2 of 你   (nothing emitted)
after byte 3 of 你   e4 bd a0  -> 你

So a family that cannot produce split sequences pays one bounded memcmp and gets identical output. A flag would let it skip 16 ns and add a way to get the setting wrong.

Happy to go either way on scope — the four you named, or all seven — but I wanted the decision to rest on measurements rather than on my say-so.

[0xShug0]

@christopherthompson81 Thanks for the quick update! Let’s keep the PR scoped to the four models (higgs_audio_stt, etc.), since they don’t require extensive testing.

[christopherthompson81]

Scoped to the four, and the description is updated to match.

voxtral_realtime, qwen3_asr and sense_asr are reverted to origin/main exactly — the net diff no longer touches those six files. Re-verified all seven families from the CLI after narrowing, including the three back on their own implementations:

parakeet_tdt      7 partials   concatenate to text_output
kroko_asr        11 partials   concatenate to text_output
higgs_audio_stt   4 partials   concatenate to text_output
voxtral_realtime 33 partials   concatenate to text_output  (own implementation)
qwen3_asr         1 partial    concatenate to text_output  (own implementation)
sense_asr         1 partial    concatenate to text_output  (own implementation)

The O(1) work stays — it still matters for the four that keep the publisher, and it means the UTF-8 check costs them one bounded memcmp rather than a rescan.

[christopherthompson81]

One thing worth passing on, and not a request to re-widen this PR — the scoping stands.

While checking whether the UTF-8 case was real for the families I was migrating, I looked at the three you asked me to leave alone. Two are fine; one has the bug already, independent of anything here.

sense_asr is not exposed. It concatenates pieces and replaces the metaspace marker, no byte-level decoding — same shape as parakeet_tdt. Nothing to do.

voxtral_realtime can emit a split character today. Its Tekken vocabulary is base64-decoded raw byte strings, and decode() concatenates them directly. All 256 single-byte tokens are present, and of twelve characters I tested, ten have no whole-character token — including a plain emoji:

'😀'  ->  4 byte tokens
  after token 1:  f0        INVALID UTF-8
  after token 2:  f09f      INVALID UTF-8
  after token 3:  f09f98    INVALID UTF-8
  after token 4:  f09f9880  valid: '😀'

take_stream_delta publishes streaming_text_.substr(streaming_published_bytes_) with no boundary check, and the family emits per token — 33 partials for the 14-second sample clip — so those intermediate states are individual transcript.text.delta payloads. An emoji in ASR output is not exotic.

The irony is that voxtral's delta emission came from #127, "Emit streaming transcript deltas instead of restating the transcript" — the same fix this PR makes for parakeet_tdt and kroko_asr. That PR solved the restating half; the UTF-8 half is still open there. You'll know better than me whether patrickvonplaten wants to pick it up — I'd rather you route it than have me tag someone into a PR they aren't part of.

qwen3_asr is byte-level too (decode_bpe maps each token codepoint back through unicode_utf8_to_byte), but it emits per window rather than per token, so it would need a window's generation to stop mid-character. Narrower, and I have not established whether that can actually happen — flagging it as unchecked rather than claiming it.

To be straight about the standard of evidence: this is vocabulary analysis plus the code path, not a live capture of the model emitting an emoji mid-stream. I have no sample that makes it do so. It is the same standard as the Higgs evidence above, and I would not call either one reproduced until someone has the audio to trigger it.

[christopherthompson81]

Followed up on the qwen3_asr caveat I left open above — it is not exposed, and I should retract the implication that it might be.

The structure rules it out rather than making it unlikely. process_one_stream_chunk calls run_single, which runs thinker_.generate() to completion and decodes the entire token list in a single postprocessor_.decode(). The delta it publishes is that whole window transcript:

auto item = run_single(make_request(item_request));
const std::string delta = streaming_text_.empty() ? item.text_output->text
                                                  : " " + item.text_output->text;
streaming_text_ += delta;

So streaming_published_bytes_ only ever advances to a point where a complete decode ended — the streaming boundary and the decode boundary are the same place. There is no mid-token state for the offset to cut through. That is the difference from voxtral_realtime, which publishes after every generated token and so has boundaries inside a character's byte sequence.

The one residual is if max_new_tokens truncated generation mid-character, but that would put the same bytes in the window's own text_output and in the final transcript, and the offline path would have it identically — a truncation concern, not a streaming-delta one.

So of the three you scoped out: sense_asr not exposed, qwen3_asr not exposed, and voxtral_realtime the only one worth passing on.

[0xShug0]

@christopherthompson81 Thanks! PR merged.