docs(tokenless): add llm provider proxy design - #2452
Conversation
github-actions
Bot
added
component:tokenless
![qoderai[bot]](https://avatars.githubusercontent.com/in/1396158?s=60&v=4){kind=small}
There was a problem hiding this comment.
[P2] 文档中鉴权与日志策略未完全约束实现，存在敏感凭据被持久化的风险。
[P2] 多轮压缩标记的幂等性约束较弱，容易导致重复压缩或混合内容。
[P2] 中英文文档在流式压缩的边界描述上略有差异，建议统一成“v1 完全透传”。
🤖 Generated by Qoder • View workflow run
| conversation history. The proxy should avoid double-compressing content | ||
| that already contains `<<tokenless:HASH>>` markers. | ||
|
|
||
| 4. **Authentication** — The proxy must forward `Authorization` headers to the |
There was a problem hiding this comment.
[P2] 文档中代理对鉴权数据的处理缺少持久化约束
在 Authentication/鉴权 部分仅说明代理不查看或保存 Authorization 头，但未明确禁止任何形式的持久化或日志记录，容易被实现为写入访问日志。
建议在设计中补充“代理不得将 Authorization 或其他敏感鉴权字段写入日志、缓存或持久存储”，并约定只有经脱敏后的统计数据可持久化。
🤖 Generated by Qoder • Fix in Qoder
There was a problem hiding this comment.
Accepted. Added an explicit prohibition: the proxy must not write Authorization or other sensitive credential fields to logs, caches, or any persistent storage, and only desensitized aggregate statistics may be persisted. Updated in both EN and ZH docs.
| and decide whether it is a candidate for `compress-response` without | ||
| breaking the provider contract. | ||
|
|
||
| 3. **Multi-turn conversation** — Compressed tool results accumulate in the |
There was a problem hiding this comment.
[P2] 代理对压缩标记的处理缺少明确幂等性约束
英文/中文文档在多轮对话部分提到“避免二次压缩包含 <<tokenless:HASH>> 的内容”，但未明确代理必须以该标记为幂等保护，防止误判导致重复压缩或解压混乱。
建议补充：代理在检测到该标记时必须跳过压缩，并定义冲突策略（例如遇到无法解析或伪造标记时直接透传并记录告警）。
🤖 Generated by Qoder • Fix in Qoder
There was a problem hiding this comment.
Accepted. Strengthened the constraint: the proxy must skip compression when a <<tokenless:HASH>> marker is present (the marker is the idempotency guard). Added a conflict strategy — if the marker is malformed or cannot be parsed, the proxy passes content through unchanged and logs a warning rather than attempting compression. Updated in both EN and ZH docs.
|
|
||
| ## 待决设计问题 | ||
|
|
||
| 1. **流式压缩** — 流式 delta 中可能包含跨多个 chunk 的 tool 结果，压缩较复杂。第一版仅压缩非流式请求，流式响应保持原样。 |
There was a problem hiding this comment.
[P2] 双语文档对流式压缩策略的约束略有差异
英文文档在 Streaming compression 部分强调“只压缩非流式请求并保持流式响应不变”，中文版本的描述略短，容易被理解为后续可以在代理中随意摘要流式输出。
建议在中文文档中明确：v1 代理不对流式 delta 做任何内容修改，仅做透传，并在未来引入流式压缩时需要额外安全/兼容性评审。
🤖 Generated by Qoder • Fix in Qoder
There was a problem hiding this comment.
Accepted. Expanded the Chinese streaming section to explicitly state: v1 applies request-body compression to all requests regardless of the stream flag, streaming response chunks are passed through with no content modification, and future streaming compression requires additional security/compatibility review. The EN and ZH docs are now consistent.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d5c9d26843
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| |---|---|---| | ||
| | `tools[].function` definitions | `tokenless compress-schema` | High when many tools are registered | | ||
| | `messages[*].tool` results | `tokenless compress-response` + TOON | High for API/shell-heavy agents | | ||
| | `messages[*].content` (tool result text) | RTK rewrite for shell output | High for shell-heavy agents | |
There was a problem hiding this comment.
For hookless agents, the provider proxy only sees shell output after the shell tool has already run and been appended to the chat history. The existing RTK path rewrites or wraps the command before execution, so applying rtk rewrite to messages[*].content would either do nothing useful or replace tool-result text with a command-shaped artifact, corrupting the conversation while promising shell-heavy savings the proxy cannot deliver.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Accepted. Removed RTK rewrite from the proxy compression table and the request interceptor. RTK is a pre-execution command transformation — the proxy only sees tool results after execution, so applying rtk rewrite to messages[*].content would either be a no-op or corrupt the conversation with command-shaped artifacts. Added a note directing shell-heavy agents that need RTK to the adapter path. The Relationship to Existing Adapters section already documents this distinction. Updated in both EN and ZH docs.
| chunks. The first version will only compress non-streaming requests and | ||
| leave streaming responses untouched. |
There was a problem hiding this comment.
When an agent uses Chat Completions with stream: true, the proxy still receives the complete JSON request, including tools and prior role: tool messages, before it opens the upstream response stream. Tying request compression to non-streaming mode would silently disable the main schema/tool-result savings path for common streaming agents; the design can still pass streaming responses through without skipping request-body interception.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Accepted. Clarified that request-body compression (schema and tool-result) applies to all requests regardless of the stream flag, since the full JSON request body is available before the upstream response stream opens. Only streaming response compression is deferred to a future version. Updated in both EN and ZH docs.
d5c9d26 to
c052eb1
Compare
1 participant
Summary
Add a design document for an LLM Provider request interception proxy. The proxy
provides a fallback compression path for agents whose frameworks do not expose
the native hooks used by existing Tokenless adapters.
Design highlights
compress-schemato tool definitions andcompress-response+ TOON to tool results.Scope
This PR contains only the design document (English + Chinese). Implementation
will follow in subsequent PRs once the design is reviewed.
Risk and compatibility
No code changes; no runtime or compatibility impact.
Validation
specs/documentation-standard.md.Related Issue
no-issue: research task