feat(chat): 流式转发带工具请求的正文,tool_call 围栏缓冲解析 - #93
Open
Sunsh1neY wants to merge 1 commit into
Open
Conversation
此前带工具的流式请求被整体降级为非流式:generate() 整段生成后再包装成 SSE 一次性发出,agent 类客户端(ZCode/dsh 等)每轮都带工具定义,因此 永远看不到流式效果。 新增工具流式路径:正文 delta 实时转发;检测到 ```tool_call 围栏开始 标记(含跨 delta 边界的情况)后停止转发、转入缓冲,流结束后解析为 OpenAI 流式 tool_calls delta 事件(id/name/arguments),finish_reason 正确置为 tool_calls。 附带修复:围栏解析失败(生成中断导致围栏未闭合、JSON 不完整等)时, 原实现会在 parse_tool_calls 中静默丢弃整个工具调用——模型的调用意图 凭空消失,agent 反复重试形成死循环。现在缓冲的原始文本会作为正文补发 给客户端,不再有内容凭空丢失。 - 纯文本回复:完全实时流式(仅末尾 11 字符延迟到流结束防围栏标记跨界) - 工具调用回合:正文部分流式,仅 tool_call JSON 部分等流结束后解析 - 现有 18 个测试全部通过,新增 3 个测试覆盖工具解析/残缺回退/纯文本 - 与 Sophomoresty#92(流中断优雅收尾)配合体验最佳;无硬依赖——重试耗尽时本路径 的异常分支会记录 Stream error 并结束,与既有降级路径行为一致
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
问题
带工具的流式请求此前被整体降级为非流式(
server.py中if stream and (not tools or tool_choice == "none")才走真流式):generate()整段生成、parse_tool_calls()解析完再包装成 SSE 一次性发出。agent 类客户端(ZCode、dsh、Codex CLI 等)每轮请求都携带工具定义,因此永远看不到流式效果——表现为「卡几秒后一大段同时出现」。更糟的是,生成中断导致围栏未闭合或 JSON 不完整时,
parse_tool_calls会except: pass静默丢弃整个工具调用:模型的调用意图凭空消失,客户端收到一段不完整正文,agent 下一轮重试、再失败——形成死循环(实测会话中观察到连续两轮输出截断的str_replace_editor调用)。方案:工具护栏流式(tool-call fencing)
新增第三个分支处理「tools 非空 + stream」:
围栏开始标记后停止转发、转入缓冲(标记可能跨 delta 边界,故常规转发时保留末尾len(marker)-1` 字符);parse_tool_calls统一解析,成功则以 OpenAI 流式tool_callsdelta 事件发出(id/type/function.name/function.arguments),finish_reason置为tool_calls;finish_reason: stop——内容零丢失,agent 可以看到模型的尝试并自行恢复。纯文本回复(含工具定义但模型直接回答)也走此路径,完全实时流式。
测试
get_weather工具调用端到端产出标准tool_callsdelta 事件(id=call_xxx, name=get_weather, args={"city":"Shanghai"})+finish_reason=tool_calls+[DONE]。与 #92 的关系
配合 #92(流中断优雅收尾)体验最佳;无硬依赖——若上游中断且重试耗尽,本路径的异常分支记录 Stream error 后结束,与既有降级路径行为一致。