Keep a realtime session alive while queued audio is still playing - #193
Merged
MikeAlhayek merged 3 commits intoSep 16, 2026
Merged
MikeAlhayek merged 3 commits into
MikeAlhayek merged 3 commits into
Conversation
The idle watchdog measured silence from the last audio the provider sent. That is the same moment the listener stopped hearing only when a deployment produces speech at roughly real time. A cascaded deployment does not: it synthesizes a whole reply in moments and leaves most of it queued on a paced transport, so the last audio event can be a full idle window in the past while the assistant is still mid-sentence. A long answer was therefore cut off one window after the provider fell quiet, with the user still listening. WatchForIdleAsync now asks the sink what the transport still has to play once the window expires. Nothing queued ends the session exactly as before; anything queued is waited out, and the window then starts again from there so the silence is measured from when the reply actually finished rather than from when generation did. Transports that hand audio straight to the client report nothing queued (IRealtimeConversationSink.PendingPlaybackMs defaults to zero) and are unaffected. Adds a regression test that fails on the previous code: the audio arrives without the clock moving at all, the sink reports a minute of it still waiting to be heard, and the session has to survive past the idle window and then end only once playback has drained and a full window of real silence has passed. Documents the queued-playback rule in the realtime voice guide, whose existing claim that a spoken answer longer than the window never trips the timeout was not true for faster-than-real-time deployments. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
Author
|
fixes #194 |
…the browser is still playing Waiting queued playback out only reached the WebRTC transport: the WebSocket sink reports what it has handed over, and handing a chunk to the browser is not the browser having played it. The queue simply moves into the Web Audio graph, which plays it at real time, so a reply sent in two seconds still looked over two seconds in and the window ran while the user was listening. Both pass-through sinks now keep a clock-based estimate of what they have handed over but the client cannot have finished playing yet, flushed when the client drops its queue. The user's own side had the same hole. Speech-started and the commit at the end of an utterance are the only events an utterance raises, and nothing arrives in between, so an answer longer than the window was cut off while they were still speaking. An utterance the provider has not committed yet now holds the watchdog off, bounded at two minutes so a dropped turn cannot keep a billed session open. Waiting a queue out also assumes it is moving. One that has not moved at all by the time it has been waited out is nobody listening, so the session ends there rather than looping on it — the maximum-duration backstop that would eventually catch that can be turned off entirely. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The idle watchdog measured silence from the last audio the provider sent. That is the same moment the listener stopped hearing only when a deployment produces speech at roughly real time. A cascaded deployment does not: it synthesizes a whole reply in moments and leaves most of it queued on a paced transport, so the last audio event can be a full idle window in the past while the assistant is still mid-sentence. A long answer was therefore cut off one window after the provider fell quiet, with the user still listening.
WatchForIdleAsync now asks the sink what the transport still has to play once the window expires. Nothing queued ends the session exactly as before; anything queued is waited out, and the window then starts again from there so the silence is measured from when the reply actually finished rather than from when generation did.
Transports that hand audio straight to the client report nothing queued (IRealtimeConversationSink.PendingPlaybackMs defaults to zero) and are unaffected.
Adds a regression test that fails on the previous code: the audio arrives without the clock moving at all, the sink reports a minute of it still waiting to be heard, and the session has to survive past the idle window and then end only once playback has drained and a full window of real silence has passed.
Documents the queued-playback rule in the realtime voice guide, whose existing claim that a spoken answer longer than the window never trips the timeout was not true for faster-than-real-time deployments.