From 13dffcb004ca5d0ab60699f000fdbafd78116225 Mon Sep 17 00:00:00 2001 From: "Shangxiang Fan (from Dev Box)" Date: Wed, 29 Jul 2026 10:53:26 +0800 Subject: [PATCH 1/6] feat: live quota from response headers, upstream read timeout, SSE buffer fixes Follows the observability work in #30 with four independent changes, plus the correction that PR's measurements force on one of them. Live quota with no extra API call - Copilot attaches per-SKU quota to every response (x-quota-snapshot-chat, -completions, -premium_interactions): entitlement, overage, overage-permitted, percent remaining, reset date. Parsed on every proxied response and surfaced on /health, /metrics (labelled by sku) and /usage. Quota previously required /usage, which costs a separate /copilot_internal/user request. - The headers are undocumented, so a value that stops parsing is ignored rather than reported as zero, and the previous reading is left in place. Reporting a confident 0 would read as "quota exhausted". Upstream read timeout - Without one, a half-open connection yields no data, no error and no end-of-stream: the request hangs forever and the stream-interrupted handling added in 1.3.0 never runs. - upstream_read_timeout_seconds bounds the silence *between* reads, not total duration, so long streaming answers are unaffected. - The default is 900s, not the 120s originally written. That figure was reasoned from the ~60s idle window the load balancer enforces; #30 measured the real upstream buffering input_json_delta until a tool call's argument JSON is complete, going 329.5s without a byte and scaling at ~9.5ms/token. 120s would have aborted healthy large tool calls. SSE line buffer - Reassembling one large event was quadratic: the buffer rescanned everything retained on every chunk. A 4 MB event in 4 KB chunks took 7.9s of CPU; the search now resumes where the previous one stopped. Correctness was never affected, only the cost. - A single line is capped at 64 MB so an upstream that never emits a newline cannot grow the buffer until the process dies. auto_upgrade defaults to true - Including for config files written before the key existed, which plain #[serde(default)] would have got wrong. - config_version bumped to 3 so existing files gain both new properties on the next start, via the mechanism added in #29. Also - Stopped scraping the AUR for the VS Code version; the maintainers asked projects to stop. Uses update.code.visualstudio.com instead. - Removed scripts/__pycache__ from version control and ignored it. - Backfilled the CHANGELOG entry for #30. --- .gitignore | 4 + CHANGELOG.md | 83 ++++++ Cargo.toml | 3 + README.md | 6 +- docs/api.md | 15 +- docs/configuration.md | 38 ++- scripts/__pycache__/replay.cpython-313.pyc | Bin 16344 -> 0 bytes src/config.rs | 115 ++++++- src/main.rs | 3 +- src/server.rs | 60 +++- src/state.rs | 330 ++++++++++++++++++++- src/util.rs | 103 ++++++- 12 files changed, 735 insertions(+), 25 deletions(-) delete mode 100644 scripts/__pycache__/replay.cpython-313.pyc diff --git a/.gitignore b/.gitignore index 1b59c51..9069a41 100644 --- a/.gitignore +++ b/.gitignore @@ -1,3 +1,7 @@ /target **/*.rs.bk Cargo.lock.orig + +# Python bytecode from scripts/ +__pycache__/ +*.py[cod] diff --git a/CHANGELOG.md b/CHANGELOG.md index aad37cc..1039efb 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,7 +4,49 @@ All notable changes to this project will be documented in this file. ## [Unreleased] +### Added +- **Live quota with no extra API call.** Copilot attaches per-SKU quota to every + response (`x-quota-snapshot-chat`, `-completions`, `-premium_interactions`), + carrying entitlement, overage, overage-permitted, percent remaining and reset + date. Those are now parsed on every proxied response and surfaced on: + - `GET /health` under `quota` + - `GET /metrics` as `ghc_proxy_quota_percent_remaining`, + `ghc_proxy_quota_entitlement` and `ghc_proxy_quota_overage`, labelled by `sku` + - `GET /usage` under `live`, alongside the existing authoritative response + + Previously quota was only available from `/usage`, which costs a separate + `/copilot_internal/user` request. The headers are undocumented, so a value + that stops parsing is ignored rather than reported as zero, and the previous + reading is left in place. +- **Observability for streams and failures.** Records now carry + `upstream_idle_max_ms` (longest silence between upstream chunks, which is what + assigns blame for a stall), `keepalive_probes`, `failure_kind`, `session_id` + parsed from `metadata.user_id`, and `output_tokens_final` so a count that is + still the `message_start` placeholder renders as `—` instead of being + presented as fact. The dashboard gained the matching columns, a failures-only + filter, and premium-request/cache-hit stats. +- `scripts/replay.py` replays a captured request at the upstream or back through + the proxy and times the SSE stream event by event, with no stall watchdog. +- `upstream_read_timeout_seconds` (default `900`, `0` disables, env + `GHC_PROXY_UPSTREAM_READ_TIMEOUT`). Bounds the silence *between* reads from an + upstream response rather than the total duration, so long streaming answers + are unaffected. Without it a half-open connection yields no data, no error and + no end-of-stream: the request hangs forever and the stream-interrupted + handling never runs. The default clears the longest silence measured against + the real upstream (329.5s, while a tool call's argument JSON was buffered) with + room to spare. + ### Changed +- **`auto_upgrade` now defaults to `true`**, including for config files written + before the setting existed. The proxy checks GitHub releases on startup and + replaces its own binary when a newer version is published; the replacement + takes effect on the next start. Disable with `auto_upgrade: false`, + `--no-auto-upgrade`, or `GHC_PROXY_AUTO_UPGRADE=0` — worth doing when the + binary is managed by a package manager, or lives in a build output directory + that `cargo build`/`cargo clean` also writes to. +- `config_version` bumped to `3`, so existing `config.yaml` files gain + `upstream_read_timeout_seconds` and the new `auto_upgrade` default on the next + start. - **Config schema upgrades apply automatically.** When a release introduces new `config.yaml` properties (signalled by a `config_version` bump), the missing keys are now filled with their defaults and written back to `config.yaml` on @@ -13,6 +55,47 @@ All notable changes to this project will be documented in this file. Opus 4.8 alias backfill is now scoped to pre-v2 files so it cannot overwrite customized mappings in current ones +### Fixed +- **Non-2xx upstreams on `/v1/messages` reached the client as an empty `200` + stream.** `messages_direct` only intercepted `400`; a `401`, `403`, `429` or + `5xx` fell through and was wrapped in a `200 text/event-stream` whose body was + a JSON error object, so the client waited on a stream that never produced an + event and reported a stall instead of the auth or rate-limit failure that + actually happened. This was the one path Claude Code takes. All five streaming + paths now gate on a single `is_streamable_status()` predicate +- **A client that disconnected mid-stream left no record.** axum drops the + response body on disconnect, which drops the generator, so the `store.add` + after the loop never ran. Recording now happens from `Drop` +- Pre-flight failures (token refresh, rate gate, connect errors) returned early + without recording; they are captured with a `failure_kind` +- **Keepalive probes could be silenced exactly during a stall.** The boundary + flag was set from the last chunk, so a TCP split mid-event left it stuck until + a new chunk arrived — and an upstream that went quiet right then produced no + probes at all. Partial events are now held back so the downstream always sits + on an event boundary, and the Anthropic path sends `event: ping` rather than + an SSE comment, since comments are discarded by the parser and never reset a + client's idle watchdog +- **Token counts were read from one bucket.** Anthropic's `input_tokens`, + `cache_read_input_tokens` and `cache_creation_input_tokens` are disjoint, so a + fully-cached Claude Code turn reported single digits for a 348,483-token + prompt. The three protocols slice the total differently and now have separate + extractors; cost reprices the cached buckets instead of charging every input + token at the full rate +- **Reassembling one large SSE event was quadratic.** The line buffer rescanned + the whole retained buffer on every chunk, so a single event that arrived in + many pieces was re-read from the start each time. A 4 MB event delivered in + 4 KB chunks took **7.9 seconds** of pure CPU; the search now resumes where the + previous one stopped, bringing it to milliseconds. Correctness was never + affected — only the cost of getting there. +- The line buffer could grow without bound if an upstream never emitted a + newline. A single line is now capped at 64 MB. +- Stopped scraping the Arch User Repository for the latest VS Code version. The + AUR maintainers asked projects to stop; `dynamic_vscode_version` now uses + Microsoft's own `update.code.visualstudio.com` release API and ignores + non-`major.minor.patch` builds. +- Removed `scripts/__pycache__` from version control and added the matching + `.gitignore` entries. + ## [1.3.0] - 2026-07-27 ### Added diff --git a/Cargo.toml b/Cargo.toml index a3ece90..39b97b4 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -38,3 +38,6 @@ self_update = { version = "0.44", default-features = false, features = ["reqwest toml = "1.1" [dev-dependencies] +# `test-util` (not part of tokio's `full`) enables the paused clock used to +# exercise the SSE keepalive timing deterministically. +tokio = { version = "1", features = ["full", "test-util"] } diff --git a/README.md b/README.md index 6ba6341..6655631 100644 --- a/README.md +++ b/README.md @@ -91,6 +91,7 @@ ghc-proxy [options] --fetch-version Fetch the latest VS Code version at startup --no-fetch-version Disable dynamic VS Code version fetching --auto-upgrade Auto-upgrade app when a newer release is available + (default: on) --no-auto-upgrade Disable app auto-upgrade --update-config Persist non-schema config write-backs (schema upgrades apply automatically) -v, --version Show version @@ -162,10 +163,10 @@ address: 127.0.0.1 port: 8314 debug: false account_type: individual # individual | business | enterprise -vscode_version: "1.123.0" +vscode_version: "1.130.0" api_version: "2025-05-01" copilot_version: "0.48.1" -auto_upgrade: false +auto_upgrade: true # self-update on startup; false to disable model_mappings: exact: opus: claude-opus-4.8 @@ -181,6 +182,7 @@ system_prompt_remove: [] system_prompt_add: [] tool_result_suffix_remove: [] max_connection_retries: 3 +upstream_read_timeout_seconds: 900 # max silence from upstream; 0 disables # Optional: require this key on all LLM endpoints (Bearer / x-api-key / # x-goog-api-key). Omit or leave empty to disable authentication. diff --git a/docs/api.md b/docs/api.md index b89baea..eac58a5 100644 --- a/docs/api.md +++ b/docs/api.md @@ -75,6 +75,12 @@ has loaded. A degraded proxy still answers `200` with `ready: false` so probes can distinguish "process alive" from "able to serve traffic". Add `?strict=true` to get `503 Service Unavailable` instead when the proxy is not ready. +The `quota` object holds the most recent per-SKU allowance reported by the +upstream. Copilot attaches it to every response, so it is current without any +extra API call — but it stays empty until the first request has been proxied. +The same figures are exported on `/metrics` as `ghc_proxy_quota_*` gauges +labelled by `sku`. + ## Retrieve a model `GET /v1/models/{model}` returns a single catalog entry in the OpenAI shape, @@ -239,7 +245,14 @@ counts, estimated cost, and prompt-cache hit rate. - **SSE keepalive** — a `: keepalive` comment is emitted after 15 seconds of silence so extended thinking does not trip the ~60 second idle timeout enforced by the upstream load balancer. Comments are ignored by every - spec-compliant SSE client. + spec-compliant SSE client. The comment is only injected at an event boundary: + if the upstream goes quiet *part way through* writing an event, splicing one + in would corrupt it, so that case is left to the read timeout below. +- **Read timeout** — `upstream_read_timeout_seconds` (default 120) bounds how + long an upstream response may stay silent between reads. It does not cap the + total duration, so long answers stream normally, but a half-open connection + is turned into a reported stream error instead of hanging the request + forever. Set it to `0` to disable. - **`anthropic-beta` passthrough** — the client's beta flags are forwarded and merged with the ones the proxy derives (`context-1m-2025-08-07` for extended context models, `context-management-2025-06-27` when the request uses diff --git a/docs/configuration.md b/docs/configuration.md index b92fa38..d05ee08 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -32,12 +32,12 @@ debug: false account_type: individual # Header version strings (mimic the VS Code Copilot Chat client) -vscode_version: "1.123.0" +vscode_version: "1.130.0" api_version: "2025-05-01" copilot_version: "0.48.1" -# Self-update behavior -auto_upgrade: false +# Self-update behavior (enabled by default) +auto_upgrade: true # Model name mappings: exact (full match) and prefix (starts-with) model_mappings: @@ -64,6 +64,11 @@ tool_result_suffix_remove: [] # Retry: max retries for upstream connection errors (0 = none) max_connection_retries: 3 +# Max seconds of silence from an upstream response before it is treated as +# dead. Bounds silence, not total duration, so long streams are fine. +# 0 disables the timeout. +upstream_read_timeout_seconds: 900 + # Optional: require this key on all LLM endpoints (Bearer / x-api-key / # x-goog-api-key). Omit or leave empty to disable authentication. # api_key: my-secret-key @@ -149,6 +154,7 @@ ghc-proxy [options] --fetch-version Fetch the latest VS Code version at startup --no-fetch-version Disable dynamic VS Code version fetching --auto-upgrade Auto-upgrade app when a newer release is available + (default: on) --no-auto-upgrade Disable app auto-upgrade --update-config Persist non-schema config write-backs (schema upgrades apply automatically) -v, --version Show version @@ -169,10 +175,11 @@ Every config field has a `GHC_PROXY_*` override: | `GHC_PROXY_API_VERSION` | `X-GitHub-Api-Version` string | | `GHC_PROXY_COPILOT_VERSION` | Copilot Chat plugin version string | | `GHC_PROXY_MAX_CONNECTION_RETRIES` | Max connection retries | +| `GHC_PROXY_UPSTREAM_READ_TIMEOUT` | Max seconds of upstream silence (`0` disables) | | `GHC_PROXY_REDIRECT_ANTHROPIC` | Always translate Anthropic via chat completions | | `GHC_PROXY_SHOW_TOKEN` | Log tokens on refresh (`true`/`1`) | | `GHC_PROXY_DYNAMIC_VSCODE_VERSION` | Fetch latest VS Code version (`true`/`1`) | -| `GHC_PROXY_AUTO_UPGRADE` | Auto-upgrade app on startup (`true`/`1`) | +| `GHC_PROXY_AUTO_UPGRADE` | Auto-upgrade app on startup (`true`/`1`); set `0` to disable | | `GHC_PROXY_RATE_LIMIT_SECONDS` | Minimum seconds between requests | | `GHC_PROXY_RATE_LIMIT_WAIT` | Wait instead of rejecting when limited (`true`/`1`) | | `GHC_PROXY_MANUAL_APPROVE` | Require manual approval per request (`true`/`1`) | @@ -215,3 +222,26 @@ stale clients: Enable `dynamic_vscode_version` (or `--fetch-version`) to refresh the VS Code version automatically at startup. + +## Self-update + +`auto_upgrade` is **enabled by default**, including for config files written +before the setting existed. On startup the proxy checks GitHub releases and, if +a newer version is published, downloads and replaces its own binary. The +replacement takes effect on the **next** start — the running process keeps +serving the old code until it restarts. + +Disable it with any of: + +```yaml +auto_upgrade: false +``` + +```bash +ghc-proxy --no-auto-upgrade +GHC_PROXY_AUTO_UPGRADE=0 ghc-proxy +``` + +Turn it off if the binary is managed by a package manager, or if it lives +somewhere the process should not rewrite — for example a build output directory +that `cargo build` or `cargo clean` also writes to. diff --git a/scripts/__pycache__/replay.cpython-313.pyc b/scripts/__pycache__/replay.cpython-313.pyc deleted file mode 100644 index 27145d02de7b6e3f986fa5e955efb6bba60288bb..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 16344 zcmbt*dvF`andbl)yao@x->>1zBqRcSlOn~3De6T%D49dZa;O*v0g!|Z4wxB04>*?~ zdz+Y)a!W8-hmhkdLAl+Ct|S|JQ@5dG*F|?%Tcy3b-5MHn2#-}cSFP-`{@5zo^5vm_ z?!IpZ15z}lB)cuK(bN5P_t!n$zy7|jzjPy~dkGXINz-#`%mj1NlCr4tWG6+yg8 zFa$?1Dn@-$#i^uUHK&$-HJnEJB{@?1)pA`m`Dc9+L3oZ2aX$Af z%`iu3Iq9G%CmZm;=+pE1*T}beY*#o1F3b7k9K=PVPN|T9dPp@ zz6s}nx-{bsJ<2dN%BJafAM52|HfVb8N@ym?(!7WB&xQDAPMUVd+!g36;O9e6rto{| zKF;R}a$dXNI{;NeZf@EaqLr!t*VUHas@*>8zHAQ#U-H5HbhNazQ1@}2j7AD(LZP|d z=H}M6Zbu9JYwbPI-r6DRht8foJTyKe5)lg5Mkvg(zCff9sd*rG&-oqG{?JT#(%}it zdeUo7`oTS@fxiduNPDn_V2O!L`B-}8XEGo$rrr4@rze!14EtBvBdlt|s8Bj#S2LO( zwVGhnPIVq}{3=1P8mDH=q)=@CDmwzp8`IhDa)JPA%QlsgV#rS58-qgY1zs4fQ*#jV z-N66p15({AnQCVYJ3C!B3^Ooygw;89PvU5=GQfm%8+(@OmD2O%UggSI0T_wX-zEqX zK@*Gt|0%Ob6HcOzs3w}W#|QmQscE7LXywv=O%*YvvKb?9jrrVOX+zOdzDu+x$WHmE z9ar450s5jl07wFR0yc(wZqDb1eZ>L<%=rVsPzGmapuUg8&2iSv@iN}bAh2+5*m8&g zu!%3hLZZ)Jd2Zyf~Ul9zOP7kdF1*aqMs0Py6!Li=au z0#_VV#I%FihuUyx%6{Mtm1uxQf?hv6EvjloRj;V35mgPNHiM`R@*+70Oo`-+K|d?% zhmRg}9Ud7I$sqJCY60^6b0P(R#)WufSTxQ-(jNlEvFSyPXKI?W!laApDQF_#XMLjT zB~j%PRTp_Z!WU@)t5td?5`Zm=EX; zFd~`)LATc>kBoyQOq9VN{}YI!#Fm*bS)#*tO_s&(h3>_^g}&GIiHfADIePf6(K7Fi zKl`q+ekJ#V{KEOesl1})-lg8}_RSyubzV`t z0C^e`uflR2+gX|>!(&=m(yR*MKsp40a$3WXPL&n5Y`dn8u<1rcv)khdhS`uSbY;#b zQhwI!zv%Y@#z!i?h-V&o1tY9Q%sY!CxoO1PL19yg8h~h_G}B4x7TYl;md`jV!W=;4&!H2>A4zATmHNLfnab@4)>s#P$x z31r&?s6ae2LkNS?41Ep8`zkS!nV*RaypG{4?h0eL5O;+M5F6yM4YV0(g?W7v$m|85 z3IMTc5Om<5@{VX?0=I`0#v_HPX_krB0|cc*4_&#J9NyPDC%3dq@}R8jzHMx}L$ zj$%v>4P$03JsRmKz*n*>DBJXJ$6aS&Bx1e^&50Gv^r3SMFXoI%d;v(&6_Hpqdj2Ouj;KEBHUE5Mvz zmYfh?eQAzq5Y+%(eC`mi1bFNN;LMiK^>lZk%OMWWVh-2NXr#1R;l|H`h{n1D%@^C6 zCC(x`d8`lh7kR)~U$bbH6D|iHW>eDBTs7FyX7!HYAi4B{X!!^U< z2`>RNhvzkiD^Dcxq~LJ5iH5X5%yTq8QpZNdk2?>!hR%+N*__)Cdlv*o7odJ94Clj( zJb*nZ?R5^teNjC-6iJM5T~OEofBY?oqQvKBg0g;la&dZLdNHsN zc)eSw>q=U?Z#TSaJaK0-WyxB6cH!A;&n>zZTuDpiyc%)zr0Sj_d(nK=eAiSk-y{^a z2!_@VtLhVziShWCQ*^_M=BN5M^g_$iw{6MBQ$oXOp>i};+WM)U%%Qd@A}8* zsLJG_T<7-NgWR;6O#@EEDr28L-C-mTN-%&&`>B8 z@7vNsa!W@Ts}uGuJ*EwWu_S&hF?{RT&0|7S-{*yUbZr>`v>_Wm=mC)SM z-_);IR==904{sG=b}?bB5GvcYO7NkSFc!r%TV_N@v`R}w~t&^)+Rg%31g zERSEt}O$X3l(z$@?dsApK z6gsDtA$vh&K{KCImrg@Jr(t+XnHQuR(0z2gx?kPf3$(zgQ!CYibM%awCeA=9qn!(% z0qL66gVv;XYDwaJfm4q!fzgx1Trp#i_uuJx;14^TugdOQ%7`MFWmg4QG3fI~yjHfs+^HeK(EfL^?P{ zH-b<MojW&#c53Mmn^b2!f1$hA#05qPW z!GcGN=nGPJ?)-@Hh>A#T0m?M z$T`qSM`tGtMB1Ga3{aFLhFd5L#lagT?jyF{$JD4zJv#aTS0g3G7U{H-$9W$&v!vjN zdPx@Y_;^GvX{h%x1;!l`EYol(@;LXR4u$~J$iaIA%REG)&I?9cuTRvW8I1MX%p%3P zFS(=+cu_MMo)R^vn5aNEk=~K0@p~_e28di9E-)o(FZsP-P|{8Yf}WQ|65p6eA_p9* zsFJZnK3nBj)-uri1nLP78j%;M{7BgZg9Jm6l`vVO!zs!P)}Z{#1;e~5WiTz6=QSy7 zZYsB6xpAp6-kr>?ogYpWl`daix|}d2i<;(-r?PTm&s@6{Kag;LzwiC*0_C|iVR*01 zvC{slGDoVSKJk?$=ZD25F>R`(JU$-x#7h#xE50{OD`ugtGuECesa-e$7N+8g8>Z`~ z#F1YW?~9SCqO#?emtIbECW~zF1lFg)rNKncW}baL&z>r)N~jX*_<{@aq&Tketk_C=GKn=F1lKsCi6WRIS3#GF2mZ>EwJ_Ksg z;70cT&+3V+yic`6LDkki$u1>hJHBNlpx%S1_ke;+rspen+E(Cz)W9*RZr`y-x3vLA zBWp1Vx`4d{*Payivc=SZT}z?e6Q?p}XjZ$U7wrxcc*fWaTp%*kV0xBS7@t^bDBK|5q} zdekrzXrh$DSjKLp?63-1 zV+e&0;OAnYB9gQj1nzlWq?r4M}$U zCy5%wbCe#~Y>BqA11tz|iELnVoH>9#894}Ru(@oWl$))*Hz_xtwSt9)hM6Zi$S@qJ zYyn%y7O}-R7LavebxQpI3tQr}C?jygsGJ3Rp$?3R?o6X1Q;ia*v%qO-z~2(3wlv;m zf~eeypR_Wc+rKb^%qnF|ouzyA$J8jJ*^M)BDOu($+p|usQirLNZWKVCcNXrzoV{z+ zE46m_#5|>xNZLT*V|r~+sFZnd!>QyfV;b;qVv8O#zQ#Rkl`@S|t>VYjvMIH8^V+gE zT7RJzFPxAdox+=RmNGW%OX(4ELFd6$w*#~GDg8Z?yMMe=!mFmePcCP6W+q^jF?MG* zLrXm&#zCDkD#o#AyUosAsofT5Ub_9g@vzmI|D^tzw!POdQ@Oo{A4SInw+!4yNbhk#oUGW@h&V zz{q-KUpYyp4{mMxdo*ydUV*#(IAEZRQLx7=#&EOl3QjXK0PkebY1zG=n5UT`C^_UT z`9jH4fGNWyNFU7M_RNJhO`PtLR}Uf%Lnv7B$htY=Bp;oHX*e->II($>wIVJ~$xj-1 zf^;=8QKK-dCvaEcMfWlFX*XPY6Y?F-<1fQU`_s2PM@J)Cs=vRVj*v9`M6_qmFypkO zI7G7M+*j~kR+i3t>4;9&$Rb&@pcVua(Y==zwNeJ;yD!VvD?AiJKG@Zi3@8N4BA7?8 zNGdca8Ii0|Fc^^YcbEK;%dMe zq}j_VGEO=PMx)eJ(mQ3237B&^I5T=ioxGyeta=LG$LO>|aUVx>|NlV%^R1S{D@aiT z9#Y(S$QL!RsBBtM9ReK`oRLJ8UnD`D=JA3I+_EG!7nfL6)+hD22BBQc2mRd92dXI0 z3^>xcKY|wduS4{^DDms8oK#+Myf)S=m@7a>ocF~h7has#-0kjrhq;~i_W3)iw=QgS zpGan%e7F0=YIW?Hc+Qe@sWk4nKDC}xm+;)2N@m;FyHCuMLe|NY!Mtc%FvXnlv4r~i zxuxe5V=L;L=Y;&G4TJqIXrvw2t_X(W-+!hhvQMcbu76xQ)J#fBJ{QE1YGIdwEt-_B z_p|-%TsY*ydkL4EZa-}}X1 z(4qb>R=M}~+aLe=Vy3?^%J;ZUXt|XgeTGr+mUBP(o0}iMy1Ju`mI@I{=#Liv49b9~ zkACsPG({K}Bn^H1pCA7)lD}i#GA%^%r3uVD%ZV^5Q1{+>?UNt;h;~ncWk;IVdu#vU zkvzP`r3MBDcDfMK?JX@leLp_`QS_Z&It-hNpd00>lIKXCFj`by;*hxr+2d#jyrCh- zE^693V6D^s7;FwvMTJ3TCaC+-m^^rRdmsH`@uQ#r?ML6b{?Rx8nzrSO+BptPW|DOo zR#JEtc8E^$5&}$ux*QD=tSvB7iN@I=8w|m%k4G|7111Ie5H2RX9)^2*aCJhk1Zv;q zkZ1;M< zz5rN-fIql;0i1-5-2&Gy?n`)`&CiL(Q0A)T%gBRHwm)$f;DMXO2%S*@Qe}QzfCxJ9 za)e**3Kp1P?<>rTI(%@4L?ix6=0#B_xzO;?4=faUDFP}V9?Z5w3!&H}i~=@iPk1gd+3JfJkAzh$~l3K{4RQcT1Zqt9((~_$;@zC%@0V9?Sm@V= zmk!5JRVv?ZTr6J<4teaTY$I$0Dw87q80uQcBE-NA31ijK^G<-;b& zs%o`kEnjHrUngzRqw^!NmlH?smQ*cR=2fx$Uz@U0Wz{!&ulFXrD?`aL$7&AP7-3y? zES&mvR^I#O{4KS{Sb%d*eTRy7z^u;?!JVVUx_D&aNUY)7=%$&DPl6>CoF+OKI%9>` z`h{`_EVPul;Pt%tGdG;qor!^DMcZb@fz`=mNzb~uXMRMXuGlnHt(&S+rDgHc>!thV zPbvjPivtS-sr-W2^wNp>BPp{*>dppJw`s0Ps5eaw>!yYk{idm9!_JsM?$1^)kFxMrXOPpM}l-WswnNE}?ECS`&Ae*Hc zbfO@k7pTS!(w3t4uXvJl>jEouACDfBiubP^Tj>!xM(*qv&OIlLd`bAy1%dkV2KoHQ zdedD?POL7jNld?Maje#SP~EU(iK*iGsqEZk!;&FB6!(H9yDj0p_0o!GwN|L_T6M4G z|FZOU?(Om09^t?Vq5I^WHlbwnSJ`Ji)etp};AvxRhSh7d3fcQt4zF|x*0%Qy?OR5; zc>Vq7E)|hGqWb*@7W#Le?<2BH9}p^I!3VHzH1PZIw;fRTJ&WTZ-v!Wh$8KQ?40i@d zi2p&S!??i8lcbRB zzlBVZ^b4*z(r;<{w~Q%==Zf}>2^(e$jgl8(=G8metAtBFR8DpU=%-3iYeAimO&Zlq z4GEWgOf9}P_^p$F_pJW}Lle_rhk1HTzZoOWW6B?E6k_&u=2Ro{iD#Oa zK-o+RPBWMn_oI0cXG3{Q+bTRdWP0Fge*&cfDePqmt^w1|-fs;I1@?)IQFvQrgM7zM zgEy#Md(C>Y!hPoL10R2omlmZK7$8twm-$AKbyommq=sq6x2Du!Nf-yciwqu>{@tKt z>`ehCnAQfM1OpVwv}0S60h(Ja(Qsp`=c!vHc516Oz^KUB*z7sS|jb0CT4t zU}n48)vTWBW)3ht4mE2y6Pd6nr4##<)ELUAyTVxWt_*mbK*tU#FulqSU=A{U(l;DN zr*W)ZDc}B8#yh4Ywu>@rjHPxPizzl%=x6cfoAUtlgmf_20j$&-u0V-FmR(SsMh4**Y^9|KDfgYw~! zXi%K6P{)C9yU<yJAe2t~|0KMew}I1t~#r^|;o@M#+rn1j9w4a=Lq1`hTD6)q-aQAilG+)G9+E?Pr=7pq8i-8H0=j??pxTxHHbuY4-eUFL^A-{ z6Ueb?hECG@iE-zZ(3LXsu>BoF!Zoro<;B(<TtR{(TG0}ce`cU-;K zRWa^O=~@JAPfpcE;#DnRF!U>7u&P@FU+Sp2H=&tPn});H$x##rkr{-a0Gg*;@|STe@wuXn=W_d^JSt5H}r0UzfA! zy1~JBDPFG&N~Jt-3SmPcC3|(?tCBp}@KYi%y5hU!<}son0v8dqFKGBHux)&o!5L06 zRNzwNmN3GBOP>dcI{6zyZUl33FhaA^E{Avy;ac)#k-Rh|k9FyL6-gK8mZ8*U5!GND zOutLqz|cRu9uBS_;8RqLzK7AP7~zQ^tN({EHH^`Jg^0(#cWVA=wV%Yb$0ecg*GTmp zwXF9Wb2g3iyGA-`tN}%t)V*@%8)st26BmR7r=w@q$x$Wis8HR78Qn_8iIvjTXNAW8 z+Y!tjQ?f^YoF}v&$E*=03%o{8UO02;=lMGW!gJ3H zR@a8%g%5H|<`3O9=LorNsF~=Nbg@vJNoH4rYVy*}!PU0aNujo9t!<56KX@7_VuS11 zwTa1FQ#Yps`*8B9!<$cy+)>>bU$1{gssHbi+4TwT*2S9_SJ-4j|7OGR?c6QBrt666 zQ$mw>MD-g&V>zkPPEsS_8>FYc@$y}RY0+@i5bK8T zzq+q?$8ADw?>p^lm)`DMbKEfqInQhu#xkENZJKJs-=blt72n_2Vvn0eh$kt!~a9-BYDP8Qsw3a;6qcWXh+yF9%#9eXbBO?2Ek zc=KSwE|^=QM{v?#{>IB-%+Xie%_)tU;(da(5geClJMWxY&wXw|f6tQhmo>``OAX6S zOHHp|5*iOAb9-(dd)IPGI6IaqDZ5d0z3BTTH>$2zB}?|rpFo}NwCcmGgY)Fw+?quD zk1ea7WMl8zi92L6_w>B}Zca(O_PZ6atoT<2YZKV8Nj-2q4N9Z`rvE*%<+moR|DNTP z{LR`wv{aoe)cnsv>SUc>G`Qf{@Ut$LO?4j;!&WTnT`q6X<8pC-56_ZGQQmL=3m@_H z;|4K8QApmBsGvw+rigkUd(jW>+cJ2gKMB{0Q8`967`hYNgKu}V`^v^`r zhvwR-5sXud4GRs6O$$viZaK0PS^nD6*Ao8K++l F{|~bpS9JgY diff --git a/src/config.rs b/src/config.rs index dce1c04..8ed8c5b 100644 --- a/src/config.rs +++ b/src/config.rs @@ -20,7 +20,10 @@ pub const API_VERSION: &str = "2025-05-01"; pub const COPILOT_VERSION: &str = "0.48.1"; /// Config schema version used to detect when defaults/options changed and a /// persisted config should be rewritten with migrated values. -pub const CONFIG_VERSION: u32 = 2; +/// +/// Bumped to 3 for `upstream_read_timeout_seconds` and the `auto_upgrade` +/// default flip, so existing files gain both on the next start. +pub const CONFIG_VERSION: u32 = 3; /// Default model name that Claude "opus"/"sonnet" requests are mapped to. pub const DEFAULT_OPUS: &str = "claude-opus-4.8"; @@ -135,6 +138,16 @@ pub struct Config { pub tool_result_suffix_remove: Vec, #[serde(default = "default_max_retries")] pub max_connection_retries: u32, + /// Maximum seconds of silence allowed between two reads from an upstream + /// response before the request is treated as dead. This bounds *silence*, + /// not total duration, so long streaming answers are unaffected. `0` + /// disables the timeout. + /// + /// Without it a half-open connection never errors and never ends: the + /// request hangs forever, the client hangs with it, and the stream- + /// interrupted handling never runs because no error is ever observed. + #[serde(default = "default_read_timeout")] + pub upstream_read_timeout_seconds: u64, /// When true, never route to the upstream `/v1/messages` endpoint; always /// translate Anthropic requests through the OpenAI chat completions API. #[serde(default)] @@ -147,9 +160,14 @@ pub struct Config { /// the `Editor-Version` header (falling back to `vscode_version`). #[serde(default)] pub dynamic_vscode_version: bool, - /// When true, check GitHub releases and auto-upgrade this binary when a - /// newer version is available. - #[serde(default)] + /// Check GitHub releases on startup and replace this binary when a newer + /// version is available. Enabled by default, including for config files + /// written before the key existed; disable with `auto_upgrade: false`, + /// `--no-auto-upgrade`, or `GHC_PROXY_AUTO_UPGRADE=0`. + /// + /// The replacement takes effect on the next start; the running process + /// keeps serving the old code until then. + #[serde(default = "default_true")] pub auto_upgrade: bool, /// Minimum number of seconds between successive proxied requests. `None` /// disables rate limiting. @@ -197,6 +215,19 @@ fn default_copilot_version() -> String { fn default_max_retries() -> u32 { 3 } +/// Deliberately far above the longest silence a healthy upstream produces. +/// +/// An earlier 120s default was wrong: it was reasoned from the ~60s idle window +/// the upstream load balancer enforces, but measurement against the real API +/// showed Copilot buffers `input_json_delta` until a tool call's argument JSON +/// is complete and then flushes it in one burst. A 35,899-token answer went +/// **329.5 seconds** without emitting a byte, and the silence scales with +/// output size at roughly 9.5ms/token — so 120s would have aborted perfectly +/// healthy large tool calls. 15 minutes clears the worst measured case with +/// room to spare while still bounding a genuinely dead connection. +fn default_read_timeout() -> u64 { + 900 +} impl Default for Config { fn default() -> Self { @@ -215,10 +246,11 @@ impl Default for Config { system_prompt_add: Vec::new(), tool_result_suffix_remove: Vec::new(), max_connection_retries: default_max_retries(), + upstream_read_timeout_seconds: default_read_timeout(), redirect_anthropic: false, show_token: false, dynamic_vscode_version: false, - auto_upgrade: false, + auto_upgrade: true, rate_limit_seconds: None, rate_limit_wait: false, manual_approve: false, @@ -385,6 +417,8 @@ pub fn render_config_yaml(cfg: &Config) -> String { let _ = writeln!(s, "vscode_version: \"{}\"", cfg.vscode_version); let _ = writeln!(s, "api_version: \"{}\"", cfg.api_version); let _ = writeln!(s, "copilot_version: \"{}\"", cfg.copilot_version); + s.push_str("# Check GitHub releases on startup and replace this binary when a newer\n"); + s.push_str("# version is available. Takes effect on the next start.\n"); let _ = writeln!(s, "auto_upgrade: {}", cfg.auto_upgrade); s.push('\n'); s.push_str("# Model Name Mappings\n"); @@ -447,6 +481,14 @@ pub fn render_config_yaml(cfg: &Config) -> String { s.push_str("# Retry Settings\n"); s.push_str("# Max retries for upstream connection errors (0 = no retries)\n"); let _ = writeln!(s, "max_connection_retries: {}", cfg.max_connection_retries); + s.push_str("# Max seconds of silence from an upstream response before it is treated as\n"); + s.push_str("# dead. Bounds silence, not total duration, so long streams are fine.\n"); + s.push_str("# 0 disables the timeout.\n"); + let _ = writeln!( + s, + "upstream_read_timeout_seconds: {}", + cfg.upstream_read_timeout_seconds + ); if cfg.redirect_anthropic { s.push('\n'); s.push_str( @@ -646,6 +688,21 @@ pub fn load_config_with_options(write_back_on_migration: bool) -> Config { ); } } + if let Ok(val) = std::env::var("GHC_PROXY_UPSTREAM_READ_TIMEOUT") { + match val.parse::() { + Ok(secs) => { + cfg.upstream_read_timeout_seconds = secs; + tracing::info!( + "✓ Overriding upstream_read_timeout_seconds from GHC_PROXY_UPSTREAM_READ_TIMEOUT: {}", + secs + ); + } + Err(_) => tracing::warn!( + "Invalid GHC_PROXY_UPSTREAM_READ_TIMEOUT value '{}': expected a number", + val + ), + } + } if let Ok(val) = std::env::var("GHC_PROXY_REDIRECT_ANTHROPIC") { cfg.redirect_anthropic = val.eq_ignore_ascii_case("true") || val == "1"; tracing::info!( @@ -792,6 +849,54 @@ fn migrate_config(cfg: &mut Config) -> bool { mod tests { use super::*; + #[test] + fn auto_upgrade_is_on_by_default_including_for_legacy_configs() { + assert!(Config::default().auto_upgrade); + + // Config files written before the key existed must also opt in, which + // is what `#[serde(default)]` would have got wrong. + let legacy = "address: 127.0.0.1\nport: 8314\nmax_connection_retries: 3\n"; + let cfg: Config = serde_norway::from_str(legacy).unwrap(); + assert!(cfg.auto_upgrade); + + // An explicit opt-out is still honoured, and survives a round trip + // through the generated config file. + let off: Config = serde_norway::from_str("auto_upgrade: false\n").unwrap(); + assert!(!off.auto_upgrade); + let rendered: Config = serde_norway::from_str(&render_config_yaml(&off)).unwrap(); + assert!(!rendered.auto_upgrade); + } + + #[test] + fn read_timeout_defaults_are_sane() { + // Silence, not total duration — a long stream must never be cut off by + // this, but a genuinely dead connection must be. + let cfg = Config::default(); + assert_eq!(cfg.upstream_read_timeout_seconds, 900); + // The longest silence measured against the real upstream was 329.5s, + // while a tool call's argument JSON was buffered. The default must + // clear that with margin or it aborts healthy requests. + assert!(cfg.upstream_read_timeout_seconds > 330); + // The rendered config round-trips the value, including the 0 (disabled) + // case, so operators can turn it off. + let off = Config { + upstream_read_timeout_seconds: 0, + ..Default::default() + }; + let parsed: Config = serde_norway::from_str(&render_config_yaml(&off)).unwrap(); + assert_eq!(parsed.upstream_read_timeout_seconds, 0); + let parsed: Config = serde_norway::from_str(&render_config_yaml(&cfg)).unwrap(); + assert_eq!(parsed.upstream_read_timeout_seconds, 900); + } + + #[test] + fn read_timeout_missing_from_legacy_config_uses_default() { + // Existing config files predate the key and must keep working. + let legacy = "address: 127.0.0.1\nport: 8314\nmax_connection_retries: 3\n"; + let cfg: Config = serde_norway::from_str(legacy).unwrap(); + assert_eq!(cfg.upstream_read_timeout_seconds, 900); + } + #[test] fn github_models_defaults_enabled() { let gm = GithubModels::default(); diff --git a/src/main.rs b/src/main.rs index 259a1e5..965d537 100644 --- a/src/main.rs +++ b/src/main.rs @@ -124,6 +124,7 @@ Options: --fetch-version Fetch the latest VS Code version at startup --no-fetch-version Disable dynamic VS Code version fetching --auto-upgrade Auto-upgrade app when a newer release is available + (default: on) --no-auto-upgrade Disable app auto-upgrade --update-config Persist non-schema config write-backs (schema upgrades apply automatically) -v, --version Show version @@ -141,7 +142,7 @@ Environment Variables: GHC_PROXY_REDIRECT_ANTHROPIC Redirect Anthropic requests (true/1) GHC_PROXY_SHOW_TOKEN Log tokens on refresh (true/1) GHC_PROXY_DYNAMIC_VSCODE_VERSION Fetch latest VS Code version (true/1) - GHC_PROXY_AUTO_UPGRADE Auto-upgrade app on startup (true/1) + GHC_PROXY_AUTO_UPGRADE Auto-upgrade app on startup (true/1); 0 disables GHC_PROXY_RATE_LIMIT_SECONDS Minimum seconds between requests GHC_PROXY_RATE_LIMIT_WAIT Wait instead of rejecting when limited (true/1) GHC_PROXY_MANUAL_APPROVE Require manual approval per request (true/1) diff --git a/src/server.rs b/src/server.rs index b1752d2..bb47773 100644 --- a/src/server.rs +++ b/src/server.rs @@ -1943,6 +1943,7 @@ async fn stream_gemini( Ok(r) => r, Err(e) => return gemini_error(StatusCode::GATEWAY_TIMEOUT, e.to_string()), }; + state.record_quota_headers(upstream.headers()); let status = upstream.status().as_u16(); // Surface a non-2xx upstream (JSON error, not SSE) as a normal error. if !is_streamable_status(status) { @@ -2146,8 +2147,18 @@ async fn usage(State(state): State) -> Response { if let Err(e) = state.ensure_copilot_token().await { return error_response(StatusCode::INTERNAL_SERVER_ERROR, e); } + // The live per-SKU snapshot rides along on every proxied response, so it is + // already current; the upstream call below adds the plan name and the + // detailed per-category breakdown it does not carry. + let live = state.quota_snapshot(); match state.fetch_usage().await { - Ok(v) => Json(crate::state::summarize_usage(&v)).into_response(), + Ok(v) => { + let mut summary = crate::state::summarize_usage(&v); + if !live.is_empty() { + summary["live"] = serde_json::to_value(&live).unwrap_or(Value::Null); + } + Json(summary).into_response() + } Err(e) => error_response(StatusCode::BAD_GATEWAY, e), } } @@ -2183,6 +2194,9 @@ async fn health( "models_loaded": model_count, "requests_served": stats.request_count, "auth_required": state.api_key().is_some(), + // Reported by the upstream on every response, so this costs no extra + // API call. Empty until the first request has been proxied. + "quota": state.quota_snapshot(), }); let strict = params .get("strict") @@ -2243,6 +2257,7 @@ async fn stream_openai( Ok(r) => r, Err(e) => return error_response(StatusCode::GATEWAY_TIMEOUT, e.to_string()), }; + state.record_quota_headers(upstream.headers()); let status = upstream.status().as_u16(); // A non-2xx upstream (e.g. GitHub Models returning 401/403 as JSON when // the token lacks the `models: read` permission) is not an SSE stream — @@ -2269,6 +2284,7 @@ async fn stream_openai( } break upstream; }; + state.record_quota_headers(upstream.headers()); let status = upstream.status().as_u16(); let model = translated.clone(); // Shared with the keepalive wrapper so the record can report how @@ -2419,6 +2435,7 @@ async fn stream_responses( Ok(r) => r, Err(e) => return error_response(StatusCode::GATEWAY_TIMEOUT, e.to_string()), }; + state.record_quota_headers(upstream.headers()); let status = upstream.status().as_u16(); // A non-2xx upstream returns a JSON error body, not an SSE stream. Forward // it as a normal error response instead of wrapping it in a 200 "stream", @@ -2550,6 +2567,7 @@ async fn stream_anthropic_direct( session_id, start, } = meta; + state.record_quota_headers(upstream.headers()); let status = upstream.status().as_u16(); // Without this the error body is wrapped in a 200 `text/event-stream`, so // the client waits on a stream that never produces an event and reports a @@ -2692,6 +2710,7 @@ async fn stream_anthropic_translated( Ok(r) => r, Err(e) => return anthropic_error(StatusCode::GATEWAY_TIMEOUT, e.to_string()), }; + state.record_quota_headers(upstream.headers()); let status = upstream.status().as_u16(); // Surface a non-2xx upstream (JSON error, not SSE) as a normal error. if !is_streamable_status(status) { @@ -3255,6 +3274,45 @@ async fn metrics_openmetrics(State(state): State) -> Response { state.uptime_secs() )); + // Quota comes from headers the upstream attaches to every response, so + // scraping this endpoint never costs an extra API call. + let quotas = state.quota_snapshot(); + if !quotas.is_empty() { + out.push_str( + "# HELP ghc_proxy_quota_percent_remaining Percent of the entitlement still available.\n", + ); + out.push_str("# TYPE ghc_proxy_quota_percent_remaining gauge\n"); + for (sku, q) in "as { + out.push_str(&format!( + "ghc_proxy_quota_percent_remaining{{sku=\"{}\"}} {}\n", + metrics_label_escape(sku), + q.percent_remaining + )); + } + + out.push_str( + "# HELP ghc_proxy_quota_entitlement Allowance for the period; negative means unlimited.\n", + ); + out.push_str("# TYPE ghc_proxy_quota_entitlement gauge\n"); + for (sku, q) in "as { + out.push_str(&format!( + "ghc_proxy_quota_entitlement{{sku=\"{}\"}} {}\n", + metrics_label_escape(sku), + q.entitlement + )); + } + + out.push_str("# HELP ghc_proxy_quota_overage Amount consumed beyond the entitlement.\n"); + out.push_str("# TYPE ghc_proxy_quota_overage gauge\n"); + for (sku, q) in "as { + out.push_str(&format!( + "ghc_proxy_quota_overage{{sku=\"{}\"}} {}\n", + metrics_label_escape(sku), + q.overage + )); + } + } + out.push_str( "# HELP ghc_proxy_estimated_cost_usd_total Total estimated request cost in USD.\n", ); diff --git a/src/state.rs b/src/state.rs index c5b0588..b451ef7 100644 --- a/src/state.rs +++ b/src/state.rs @@ -6,13 +6,13 @@ use crate::auth; use crate::config::{self, Config, ModelMappings}; use crate::store::RequestStore; use reqwest::header::{HeaderMap, HeaderName, HeaderValue}; +use std::collections::BTreeMap; use std::sync::Arc; use std::sync::RwLock as StdRwLock; use std::time::{Duration, Instant, SystemTime, UNIX_EPOCH}; use tokio::sync::{Mutex, RwLock}; -/// Mutable token state guarded by a mutex. -#[derive(Default)] +/// Mutable token state guarded by a mutex.#[derive(Default)] pub struct TokenState { pub github_token: String, pub copilot_token: Option, @@ -37,10 +37,108 @@ pub struct AppState { pub session_id: String, /// Instant the process started serving, used to report uptime on `/health`. pub started_at: Instant, + /// Latest per-SKU quota reported by the upstream, keyed by SKU name. + pub quotas: StdRwLock>, } pub type SharedState = Arc; +/// Prefix of the per-SKU quota headers Copilot attaches to every response. +const QUOTA_HEADER_PREFIX: &str = "x-quota-snapshot-"; + +/// A quota snapshot for one billing SKU, as reported by the upstream on every +/// response. +/// +/// Copilot returns these on each request (`x-quota-snapshot-chat`, +/// `-completions`, `-premium_interactions`), so live quota costs nothing — +/// unlike `/copilot_internal/user`, which is a separate API call. +#[derive(Debug, Clone, Default, serde::Serialize)] +pub struct QuotaSnapshot { + /// Allowance for the period. Negative means unlimited. + pub entitlement: f64, + /// Amount consumed beyond the entitlement. + pub overage: f64, + /// Whether going past the entitlement is allowed at all. + pub overage_permitted: bool, + /// Percentage of the entitlement still available. + pub percent_remaining: f64, + /// True when the SKU has no cap. + pub unlimited: bool, + /// When the allowance resets, if the upstream reported it. + #[serde(skip_serializing_if = "Option::is_none")] + pub reset_date: Option, +} + +/// Decodes the `%XX` escapes used in the header's date field. +fn percent_decode(s: &str) -> String { + let bytes = s.as_bytes(); + let mut out = Vec::with_capacity(bytes.len()); + let mut i = 0; + while i < bytes.len() { + if bytes[i] == b'%' && i + 2 < bytes.len() { + let hex = std::str::from_utf8(&bytes[i + 1..i + 3]).ok(); + if let Some(byte) = hex.and_then(|h| u8::from_str_radix(h, 16).ok()) { + out.push(byte); + i += 3; + continue; + } + } + out.push(bytes[i]); + i += 1; + } + String::from_utf8_lossy(&out).into_owned() +} + +/// Parses a quota header value such as +/// `ent=-1&ov=0.0&ovPerm=true&rem=100.0&rst=2026-08-01T00%3A00%3A00Z`. +/// +/// Returns `None` when nothing recognizable is present, so an upstream that +/// stops sending these (they are undocumented) simply leaves the cache empty +/// rather than reporting zeroed quota. +pub fn parse_quota_snapshot(value: &str) -> Option { + let mut snapshot = QuotaSnapshot::default(); + let mut saw_field = false; + for pair in value.split('&') { + let Some((key, raw)) = pair.split_once('=') else { + continue; + }; + match key.trim() { + "ent" => { + if let Ok(v) = raw.parse::() { + snapshot.entitlement = v; + snapshot.unlimited = v < 0.0; + saw_field = true; + } + } + "ov" => { + if let Ok(v) = raw.parse::() { + snapshot.overage = v; + saw_field = true; + } + } + "ovPerm" => { + snapshot.overage_permitted = raw.eq_ignore_ascii_case("true"); + saw_field = true; + } + "rem" => { + if let Ok(v) = raw.parse::() { + snapshot.percent_remaining = v; + saw_field = true; + } + } + "rst" => { + let decoded = percent_decode(raw); + if !decoded.is_empty() { + snapshot.reset_date = Some(decoded); + saw_field = true; + } + } + _ => {} + } + } + saw_field.then_some(snapshot) +} + fn now_secs() -> u64 { SystemTime::now() .duration_since(UNIX_EPOCH) @@ -58,14 +156,20 @@ fn now_millis() -> u128 { impl AppState { pub fn new(config: Config, github_token: String) -> Self { - // No global request timeout: SSE responses are long-lived streams. Only - // the connect phase and idle pooled connections are bounded so a dead - // upstream cannot wedge a request forever. - let http = reqwest::Client::builder() + // No global request timeout: SSE responses are long-lived streams, and + // capping their total duration would cut off legitimate long answers. + // Instead the connect phase, idle pooled connections, and the gap + // *between* reads are bounded. The read timeout is what turns a + // half-open upstream into an error the streaming paths can report, + // rather than a request that hangs forever. + let mut builder = reqwest::Client::builder() .connect_timeout(Duration::from_secs(30)) - .pool_idle_timeout(Duration::from_secs(90)) - .build() - .expect("failed to build HTTP client"); + .pool_idle_timeout(Duration::from_secs(90)); + if config.upstream_read_timeout_seconds > 0 { + builder = + builder.read_timeout(Duration::from_secs(config.upstream_read_timeout_seconds)); + } + let http = builder.build().expect("failed to build HTTP client"); AppState { http, config: StdRwLock::new(config), @@ -81,9 +185,41 @@ impl AppState { machine_id: auth::load_or_create_machine_id(), session_id: format!("{}{}", uuid::Uuid::new_v4(), now_millis()), started_at: Instant::now(), + quotas: StdRwLock::new(BTreeMap::new()), + } + } + + /// Records the per-SKU quota headers attached to an upstream response. + /// + /// Called for every Copilot response, so quota stays current without the + /// extra `/copilot_internal/user` round trip. Headers that do not parse are + /// ignored, leaving the previous value in place. + pub fn record_quota_headers(&self, headers: &HeaderMap) { + let mut parsed: Vec<(String, QuotaSnapshot)> = Vec::new(); + for (name, value) in headers { + let Some(sku) = name.as_str().strip_prefix(QUOTA_HEADER_PREFIX) else { + continue; + }; + let Ok(value) = value.to_str() else { continue }; + if let Some(snapshot) = parse_quota_snapshot(value) { + parsed.push((sku.to_string(), snapshot)); + } + } + if parsed.is_empty() { + return; + } + if let Ok(mut quotas) = self.quotas.write() { + for (sku, snapshot) in parsed { + quotas.insert(sku, snapshot); + } } } + /// The most recent per-SKU quota reported by the upstream. + pub fn quota_snapshot(&self) -> BTreeMap { + self.quotas.read().map(|q| q.clone()).unwrap_or_default() + } + /// Seconds this process has been running. pub fn uptime_secs(&self) -> u64 { self.started_at.elapsed().as_secs() @@ -700,7 +836,183 @@ fn premium_multiplier_from_catalog(catalog: &serde_json::Value, model: &str) -> #[cfg(test)] mod tests { use super::*; + use futures_util::StreamExt; use serde_json::json; + use tokio::io::AsyncWriteExt; + + #[test] + fn parses_a_real_quota_header() { + // Captured verbatim from a live Copilot response. + let q = parse_quota_snapshot( + "ent=-1&ov=0.0&ovPerm=true&rem=100.0&rst=2026-08-01T00%3A00%3A00Z", + ) + .expect("header should parse"); + assert_eq!(q.entitlement, -1.0); + assert!(q.unlimited, "a negative entitlement means unlimited"); + assert_eq!(q.overage, 0.0); + assert!(q.overage_permitted); + assert_eq!(q.percent_remaining, 100.0); + // The date arrives percent-encoded and must be decoded. + assert_eq!(q.reset_date.as_deref(), Some("2026-08-01T00:00:00Z")); + } + + #[test] + fn parses_a_capped_sku() { + let q = parse_quota_snapshot("ent=300&ov=12.5&ovPerm=false&rem=42.5").unwrap(); + assert_eq!(q.entitlement, 300.0); + assert!(!q.unlimited); + assert_eq!(q.overage, 12.5); + assert!(!q.overage_permitted); + assert_eq!(q.percent_remaining, 42.5); + assert!(q.reset_date.is_none()); + } + + #[test] + fn unrecognized_quota_headers_are_ignored() { + // These headers are undocumented; if the shape changes we must report + // nothing rather than a confident zero. + assert!(parse_quota_snapshot("").is_none()); + assert!(parse_quota_snapshot("garbage").is_none()); + assert!(parse_quota_snapshot("unknown=1&other=2").is_none()); + // A partially recognizable value still yields what it does contain. + assert_eq!( + parse_quota_snapshot("rem=7.5&junk=x") + .unwrap() + .percent_remaining, + 7.5 + ); + } + + #[test] + fn quota_headers_are_recorded_per_sku() { + let state = AppState::new(Config::default(), "t".into()); + assert!(state.quota_snapshot().is_empty()); + + let mut h = HeaderMap::new(); + h.insert( + "x-quota-snapshot-chat", + HeaderValue::from_static("ent=-1&ov=0.0&ovPerm=false&rem=100.0"), + ); + h.insert( + "x-quota-snapshot-premium_interactions", + HeaderValue::from_static("ent=300&ov=0.0&ovPerm=true&rem=88.0"), + ); + h.insert( + "content-type", + HeaderValue::from_static("text/event-stream"), + ); + state.record_quota_headers(&h); + + let q = state.quota_snapshot(); + assert_eq!(q.len(), 2, "only the quota headers are captured: {q:?}"); + assert_eq!(q["chat"].percent_remaining, 100.0); + assert_eq!(q["premium_interactions"].percent_remaining, 88.0); + assert_eq!(q["premium_interactions"].entitlement, 300.0); + + // A later response updates in place. + let mut h2 = HeaderMap::new(); + h2.insert( + "x-quota-snapshot-premium_interactions", + HeaderValue::from_static("ent=300&ov=0.0&ovPerm=true&rem=87.0"), + ); + state.record_quota_headers(&h2); + let q = state.quota_snapshot(); + assert_eq!(q["premium_interactions"].percent_remaining, 87.0); + // Untouched SKUs keep their previous value. + assert_eq!(q["chat"].percent_remaining, 100.0); + } + + #[test] + fn responses_without_quota_headers_leave_the_cache_alone() { + let state = AppState::new(Config::default(), "t".into()); + let mut h = HeaderMap::new(); + h.insert( + "x-quota-snapshot-chat", + HeaderValue::from_static("ent=-1&rem=55.0"), + ); + state.record_quota_headers(&h); + + // An upstream that stops sending them must not wipe what we know. + state.record_quota_headers(&HeaderMap::new()); + assert_eq!(state.quota_snapshot()["chat"].percent_remaining, 55.0); + } + + /// Serves one request: SSE headers plus half an event, then stalls forever + /// without ever closing the socket. Returns the bound address. + async fn stalling_upstream() -> String { + let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap(); + let addr = listener.local_addr().unwrap(); + tokio::spawn(async move { + if let Ok((mut sock, _)) = listener.accept().await { + let mut buf = [0u8; 4096]; + let _ = tokio::io::AsyncReadExt::read(&mut sock, &mut buf).await; + let _ = sock + .write_all( + b"HTTP/1.1 200 OK\r\n\ + Content-Type: text/event-stream\r\n\ + Transfer-Encoding: chunked\r\n\r\n", + ) + .await; + // A chunk that stops in the middle of a `data:` line. + let partial: &[u8] = b"data: {\"partial\":"; + let _ = sock + .write_all(format!("{:X}\r\n", partial.len()).as_bytes()) + .await; + let _ = sock.write_all(partial).await; + let _ = sock.write_all(b"\r\n").await; + let _ = sock.flush().await; + std::future::pending::<()>().await; + } + }); + format!("http://{addr}/") + } + + /// The streaming paths only notice a dead upstream when the body stream + /// yields an error. Without a read timeout a half-open connection yields + /// neither data nor error nor end-of-stream, so the request hangs forever + /// and none of the stream-interrupted handling ever runs. + #[tokio::test] + async fn read_timeout_surfaces_a_stalled_upstream_as_an_error() { + let url = stalling_upstream().await; + let cfg = Config { + upstream_read_timeout_seconds: 1, + ..Default::default() + }; + let state = AppState::new(cfg, "token".into()); + + let resp = state.http.get(&url).send().await.expect("headers arrive"); + assert!(resp.status().is_success()); + + let mut stream = resp.bytes_stream(); + let first = stream.next().await.expect("first chunk").expect("is data"); + assert_eq!(&first[..], b"data: {\"partial\":"); + + let outcome = tokio::time::timeout(Duration::from_secs(15), stream.next()).await; + let item = outcome.expect("must not hang past the read timeout"); + let err = item.expect("stream must yield an item").unwrap_err(); + assert!( + err.is_timeout(), + "expected a timeout error, got: {err:?} ({err})" + ); + } + + /// Setting the timeout to zero disables it, for operators who would rather + /// let a stream run indefinitely. + #[tokio::test] + async fn read_timeout_can_be_disabled() { + let url = stalling_upstream().await; + let cfg = Config { + upstream_read_timeout_seconds: 0, + ..Default::default() + }; + let state = AppState::new(cfg, "token".into()); + let resp = state.http.get(&url).send().await.expect("headers arrive"); + let mut stream = resp.bytes_stream(); + let _ = stream.next().await; + let outcome = tokio::time::timeout(Duration::from_secs(3), stream.next()).await; + assert!(outcome.is_err(), "expected the stream to still be hanging"); + } + #[test] fn premium_multiplier_reads_billing_from_the_catalog() { // Shape taken from a real `GET /models` response. diff --git a/src/util.rs b/src/util.rs index dad7ae8..5cbdef5 100644 --- a/src/util.rs +++ b/src/util.rs @@ -19,9 +19,23 @@ use std::time::Duration; /// partial character simply waits for the rest of its bytes. Anything left over /// when the stream ends is returned by [`SseLineBuffer::flush`], so a final /// event that arrives without a trailing newline is not lost either. +/// Largest single SSE line the buffer will accumulate before giving up. +/// +/// Real events are kilobytes; this is far above anything legitimate. It exists +/// so a broken or hostile upstream that never emits a newline cannot grow the +/// buffer until the process runs out of memory. +pub const MAX_SSE_LINE_BYTES: usize = 64 * 1024 * 1024; + #[derive(Default)] pub struct SseLineBuffer { + /// Bytes of the line currently being assembled. Never contains a newline: + /// `push` always drains past the last one it finds, which is what makes + /// "only the newly arrived bytes can complete a line" true. buf: Vec, + /// Set when a single line exceeded [`MAX_SSE_LINE_BYTES`]. The stream can + /// no longer be parsed correctly from this point, so callers treat it the + /// same as a dropped connection rather than emitting a truncated event. + poisoned: bool, } impl SseLineBuffer { @@ -29,20 +43,51 @@ impl SseLineBuffer { Self::default() } + /// Whether an oversized line forced the buffer to give up. Once true the + /// remaining stream is unparseable and the caller should report the + /// response as incomplete. + pub fn is_poisoned(&self) -> bool { + self.poisoned + } + /// Appends a chunk and returns every complete line it completed, with the /// trailing `\r` (CRLF streams) already stripped. pub fn push(&mut self, chunk: &[u8]) -> Vec { + if self.poisoned { + return Vec::new(); + } + // Whatever is already buffered was searched on an earlier call and, by + // the invariant above, holds no newline. So only the bytes that just + // arrived can terminate a line, and the search starts where they do — + // every byte is examined exactly once across the whole stream. + let search_from = self.buf.len(); self.buf.extend_from_slice(chunk); + let mut lines = Vec::new(); let mut start = 0usize; - while let Some(pos) = self.buf[start..].iter().position(|&b| b == b'\n') { - let end = start + pos; + let mut cursor = search_from; + while let Some(pos) = self.buf[cursor..].iter().position(|&b| b == b'\n') { + let end = cursor + pos; + // A line spans from the end of the previous one, which may reach + // back into bytes buffered by earlier calls. lines.push(decode_line(&self.buf[start..end])); start = end + 1; + cursor = start; } if start > 0 { self.buf.drain(..start); } + if self.buf.len() > MAX_SSE_LINE_BYTES { + tracing::error!( + "[sse] discarding a {} byte line with no newline (limit {} bytes); \ + treating the stream as broken", + self.buf.len(), + MAX_SSE_LINE_BYTES + ); + self.buf.clear(); + self.buf.shrink_to_fit(); + self.poisoned = true; + } lines } @@ -245,6 +290,9 @@ pub async fn post_with_retry( .await; match result { Ok(resp) => { + // Every Copilot response carries the current quota, so recording + // it here keeps it fresh for free on every request. + state.record_quota_headers(resp.headers()); let status = resp.status().as_u16(); // If response is successful or a non-retryable error, return it if status < 400 || !is_retryable_error(status) { @@ -511,6 +559,57 @@ mod tests { assert!(b.flush().is_none()); } + #[test] + fn sse_buffer_handles_a_huge_single_event_split_into_many_chunks() { + // One 4 MB `data:` line delivered in 4 KB pieces, the shape a very + // large tool-call payload or final Responses event takes. + let payload = "x".repeat(4 * 1024 * 1024); + let line = format!("data: {{\"t\":\"{payload}\"}}\n"); + let bytes = line.as_bytes(); + + let start = std::time::Instant::now(); + let mut b = SseLineBuffer::new(); + let mut lines = Vec::new(); + for chunk in bytes.chunks(4096) { + lines.extend(b.push(chunk)); + } + let elapsed = start.elapsed(); + + assert_eq!(lines.len(), 1); + let v: Value = serde_json::from_str(sse_data(&lines[0]).unwrap()).unwrap(); + assert_eq!(v["t"].as_str().unwrap().len(), payload.len()); + assert!(b.flush().is_none()); + + // Each byte must be examined a bounded number of times. Rescanning the + // whole retained buffer on every chunk makes this quadratic and takes + // orders of magnitude longer. + assert!( + elapsed < std::time::Duration::from_secs(5), + "reassembling one large event took {elapsed:?}; the buffer is rescanning" + ); + } + + #[test] + fn sse_buffer_gives_up_on_an_unbounded_line_instead_of_growing_forever() { + // An upstream that never emits a newline must not be able to grow the + // buffer until the process dies. + let mut b = SseLineBuffer::new(); + let mib = vec![b'x'; 1024 * 1024]; + let mut pushed = 0usize; + while !b.is_poisoned() { + b.push(&mib); + pushed += mib.len(); + assert!( + pushed <= MAX_SSE_LINE_BYTES + 2 * mib.len(), + "buffer grew past the cap without giving up" + ); + } + assert!(b.is_poisoned()); + // Once poisoned it stops accumulating entirely. + assert!(b.push(b"data: anything\n").is_empty()); + assert!(b.is_poisoned()); + } + #[test] fn sse_data_extracts_payload_only() { assert_eq!(sse_data("data: {}"), Some("{}")); From a902784cc104e8ad71f63fe294256598a9b060a0 Mon Sep 17 00:00:00 2001 From: "Shangxiang Fan (from Dev Box)" Date: Wed, 29 Jul 2026 19:40:10 +0800 Subject: [PATCH 2/6] fix(gemini): map the generation parameters that have counterparts `gemini_to_openai` forwarded four fields out of `generationConfig` and dropped the rest silently. A client asking for a fixed `seed` got a non-deterministic answer, and one asking for JSON output got prose -- neither with any sign the request had been altered on the way through. Adds the mappings that exist one-for-one: `candidateCount` -> `n`, `seed`, `presencePenalty`, `frequencyPenalty`, and `responseMimeType: "application/json"` -> `response_format`, carrying `responseSchema` across as a JSON schema when one is supplied. `topK` and `safetySettings` stay dropped on purpose. Neither has an OpenAI counterpart, and approximating one would be worse than not honouring them; a test asserts they are not invented. --- src/gemini.rs | 134 ++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 134 insertions(+) diff --git a/src/gemini.rs b/src/gemini.rs index 08e586a..2d1953c 100644 --- a/src/gemini.rs +++ b/src/gemini.rs @@ -162,6 +162,11 @@ pub fn gemini_to_openai(req: &Value, model: &str, stream: bool) -> Value { out.insert("stream".into(), Value::Bool(stream)); // generationConfig -> top-level OpenAI sampling params. + // + // Only `topK` and `safetySettings` have no counterpart in the chat + // completions schema; everything else here maps exactly, and dropping it + // silently meant a client's `seed` or `candidateCount` had no effect and no + // explanation. if let Some(gc) = req .get("generationConfig") .or_else(|| req.get("generation_config")) @@ -181,6 +186,49 @@ pub fn gemini_to_openai(req: &Value, model: &str, stream: bool) -> Value { if let Some(v) = gc.get("stopSequences").or_else(|| gc.get("stop_sequences")) { out.insert("stop".into(), v.clone()); } + if let Some(v) = gc + .get("candidateCount") + .or_else(|| gc.get("candidate_count")) + { + out.insert("n".into(), v.clone()); + } + if let Some(v) = gc.get("seed") { + out.insert("seed".into(), v.clone()); + } + if let Some(v) = gc + .get("presencePenalty") + .or_else(|| gc.get("presence_penalty")) + { + out.insert("presence_penalty".into(), v.clone()); + } + if let Some(v) = gc + .get("frequencyPenalty") + .or_else(|| gc.get("frequency_penalty")) + { + out.insert("frequency_penalty".into(), v.clone()); + } + // Gemini asks for JSON with a mime type; chat completions with a + // response_format object. A schema, when supplied, carries over as a + // json_schema format. + let mime = gc + .get("responseMimeType") + .or_else(|| gc.get("response_mime_type")) + .and_then(Value::as_str); + if mime == Some("application/json") { + let schema = gc + .get("responseSchema") + .or_else(|| gc.get("response_schema")); + out.insert( + "response_format".into(), + match schema { + Some(s) => json!({ + "type": "json_schema", + "json_schema": {"name": "response", "schema": s} + }), + None => json!({"type": "json_object"}), + }, + ); + } } // tools.functionDeclarations -> OpenAI function tools. @@ -349,6 +397,92 @@ mod tests { assert_eq!(out["top_p"], 0.9); } + /// Everything in `generationConfig` with an exact chat-completions + /// counterpart has to carry over. These used to be dropped silently, so a + /// client's `seed` had no effect and nothing said why. + #[test] + fn generation_config_maps_the_rest() { + let req = json!({ + "contents": [{"role": "user", "parts": [{"text": "hi"}]}], + "generationConfig": { + "candidateCount": 2, + "seed": 42, + "presencePenalty": 0.3, + "frequencyPenalty": 0.4, + "stopSequences": ["END"] + } + }); + let out = gemini_to_openai(&req, "m", false); + assert_eq!(out["n"], 2); + assert_eq!(out["seed"], 42); + assert_eq!(out["presence_penalty"], 0.3); + assert_eq!(out["frequency_penalty"], 0.4); + assert_eq!(out["stop"], json!(["END"])); + } + + #[test] + fn response_mime_type_becomes_response_format() { + let plain = gemini_to_openai( + &json!({ + "contents": [{"role": "user", "parts": [{"text": "hi"}]}], + "generationConfig": {"responseMimeType": "application/json"} + }), + "m", + false, + ); + assert_eq!(plain["response_format"], json!({"type": "json_object"})); + + let schema = json!({"type": "object", "properties": {"a": {"type": "string"}}}); + let with_schema = gemini_to_openai( + &json!({ + "contents": [{"role": "user", "parts": [{"text": "hi"}]}], + "generationConfig": { + "responseMimeType": "application/json", + "responseSchema": schema + } + }), + "m", + false, + ); + assert_eq!(with_schema["response_format"]["type"], "json_schema"); + assert_eq!( + with_schema["response_format"]["json_schema"]["schema"], + schema + ); + + // A plain-text turn must not acquire a response_format it never asked for. + let text = gemini_to_openai( + &json!({ + "contents": [{"role": "user", "parts": [{"text": "hi"}]}], + "generationConfig": {"responseMimeType": "text/plain"} + }), + "m", + false, + ); + assert!(text.get("response_format").is_none()); + } + + /// `topK` and `safetySettings` have no counterpart in the chat completions + /// schema. Asserting they stay out keeps the dashboard's "dropped in + /// translation" note honest. + #[test] + fn params_without_counterparts_are_not_invented() { + let out = gemini_to_openai( + &json!({ + "contents": [{"role": "user", "parts": [{"text": "hi"}]}], + "generationConfig": {"topK": 32}, + "safetySettings": [{"category": "HARM_CATEGORY_HARASSMENT", + "threshold": "BLOCK_NONE"}] + }), + "m", + false, + ); + assert!(out.get("top_k").is_none()); + assert!(out.get("topK").is_none()); + assert!(out.get("safetySettings").is_none()); + assert!(out.get("safety_settings").is_none()); + } + #[test] fn function_declarations_become_tools() { let req = json!({ From 087b51cb2d2bcba96b12a82c4aba5d7f337ec107 Mon Sep 17 00:00:00 2001 From: "Shangxiang Fan (from Dev Box)" Date: Wed, 29 Jul 2026 19:41:14 +0800 Subject: [PATCH 3/6] feat: responses over websocket, real billing figures, and a dashboard built around them MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three strands that turned out to be one: the dashboard could not answer "what is this costing me", and fixing that exposed both a transport the proxy did not speak and several figures it was reporting wrongly. ## Responses over WebSocket The model catalogue advertises `ws:/responses` on ten models, and the proxy did not speak it. GitHub publishes no inference API documentation -- confirmed by search; `docs.github.com` covers Copilot seats and billing only -- so the protocol was established by probing and is recorded in `docs/api.md`: - Same host as `endpoints.api` from the token response, path `/responses`. `/v1/responses` also upgrades; `/ws/responses` and `/realtime` are 404. - Auth and headers identical to the HTTP path. - The client sends one **flat** JSON text frame, `{"type":"response.create","model":"...", ...}`. `model` must be top level: nesting the request under `"response"` is rejected with `[ObjectParam] [input] [invalid_type] ... expected a string, but got an object`, and omitting `type` with `unsupported message type:`. - The server replies with the same `response.*` events as SSE, one per frame, each carrying a `sequence_number`. `GET /v1/responses` with an `Upgrade` header now relays to it, recording the exchange like any other request. A model that does not advertise the transport is refused with an error frame naming the endpoint that would work, rather than being proxied into an upstream rejection. `scripts/ws_check.py` discovers the capable models from the catalogue and checks both directions: 85 assertions across 8 models pass, and `scripts/protocol_check.py` covers the four HTTP surfaces with 72 more. Body capture on a WebSocket reads the debug flag once, when the socket opens, and uses that reading for both halves. Re-reading it at record time -- as the HTTP paths can safely do -- would let a long-lived socket produce a record with a captured request and no response, or the reverse. ## Figures that were wrong - **Output tokens were not comparable across surfaces.** The Responses API counts reasoning inside `output_tokens`; a translated Gemini response reports it as a disjoint `reasoning_tokens`. Summing them blindly produced a reasoning share of **262%** on the dashboard. `openai_usage` now normalises on field position -- nested under `completion_tokens_details` means already included, flat means additive -- so one output total means one thing everywhere. - **Cost was guessed from a price list.** Copilot states its own per-token rate for every token type it charges, in `copilot_usage.token_details`, and reports the total as `total_nano_aiu` (verified against the rates, 5 of 5). That is now the headline figure, replacing an estimate that could not know a model is included at no charge. - **Cache savings used one hard-coded discount for every model.** The entries differ: `claude-haiku-4.5` prices cache writes above its input rate, `gemini-3.5-flash` at zero, `gpt-5.5` not at all, and `gpt-4o-mini` prices nothing because Copilot includes it. The old maths claimed a saving on models that are free. Savings are now computed per request from the rates that request reported. - **Failed attempts counted as consumption.** A burst of rejected calls inflated `request_count`, added an empty-named row to the cache table, and dragged every cache disposition toward "uncached" -- a rate computed over requests that consumed nothing. Statistics now cover requests that produced an answer, with `failed_requests` reported separately. Token and billing totals still count either way: a stream cut off partway consumed what it consumed. - **SSE reassembly covered one and a half protocols.** OpenAI `delta.tool_calls[]` fragments and the entire Responses `response.*` event family rendered as "(stream produced no content)", and `reasoning_text` was discarded outright. ## Dashboard Split into Overview, Requests and Metrics over a shared stylesheet, with the landing page ordered by what it costs rather than by what is easy to count. The recent-requests table moved to its own tab, which is where it can afford the width to be useful. Each request now renders as a conversation, with the raw frames and the protocol-specific parameters behind tabs, and a summary of what the response actually reported -- different per protocol, because the protocols report different things. Two columns were removed rather than fixed. `Probes` was measured on one of five code paths, so its `—` read as "zero probes sent" on the other four; it is now in the `Idle` tooltip. `Session` and `New` are protocol-specific and moved to the detail panel. The prompt-cache panel states what it does not know. A hit rate of zero is shown as "nothing was cacheable yet" rather than as a measurement, and the per-model table distinguishes three states that all used to render as `0`: a metric the model never reported, one it reported as zero, and one the surface has no concept of. `Written` is hidden entirely on a workload that never writes, because only the Anthropic surface with an explicit `cache_control` marker ever bills a cache write -- measured across twelve calls with a 27k-token prefix, exactly one reported one. Body capture is togglable from the dashboard (`POST /api/config/debug`), deliberately not persisted: it is a debugging aid, and a config file that quietly starts recording request bodies after a restart is a liability. ## Verification - 134 unit tests, clippy clean under `-D warnings`. - `scripts/protocol_check.py` 72/72, `scripts/ws_check.py` 85/85. - Billing and reasoning figures confirmed on all six recording paths (four protocols x streamed/not, plus WebSocket). - Cache savings hand-checked against `token_details`: `gpt-5.5` 3840 x (500000-50000) = 1728000000, `gemini-3.5-flash` 4058 x (150000-15000) = 547830000. --- CHANGELOG.md | 212 +++++- Cargo.lock | 90 +++ Cargo.toml | 5 +- docs/api.md | 120 +++ public/app.css | 372 ++++++++++ public/dashboard.html | 491 +++++++++++-- public/metrics.html | 95 ++- public/requests.html | 1183 ++++++++++++++++++++++++++++-- scripts/anthropic_cache_probe.py | 79 ++ scripts/cache_demo.py | 96 +++ scripts/cache_sweep.py | 168 +++++ scripts/cache_write_probe.py | 111 +++ scripts/protocol_check.py | 475 ++++++++++++ scripts/stats_inventory.py | 123 ++++ scripts/ws_check.py | 221 ++++++ scripts/ws_explore.py | 154 ++++ scripts/ws_probe.py | 127 ++++ src/server.rs | 756 ++++++++++++++++++- src/state.rs | 28 + src/store.rs | 58 +- src/util.rs | 227 +++++- 21 files changed, 5003 insertions(+), 188 deletions(-) create mode 100644 public/app.css create mode 100644 scripts/anthropic_cache_probe.py create mode 100644 scripts/cache_demo.py create mode 100644 scripts/cache_sweep.py create mode 100644 scripts/cache_write_probe.py create mode 100644 scripts/protocol_check.py create mode 100644 scripts/stats_inventory.py create mode 100644 scripts/ws_check.py create mode 100644 scripts/ws_explore.py create mode 100644 scripts/ws_probe.py diff --git a/CHANGELOG.md b/CHANGELOG.md index 1039efb..b52df8b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -25,9 +25,56 @@ All notable changes to this project will be documented in this file. still the `message_start` placeholder renders as `—` instead of being presented as fact. The dashboard gained the matching columns, a failures-only filter, and premium-request/cache-hit stats. +- **The Responses API over WebSocket.** Eight models advertise `ws:/responses` + in the catalog — six of them alongside `/responses` and nothing else — and the + proxy had no way to reach it. `GET /v1/responses` with an upgrade now exposes + that transport, relayed to the matching upstream socket. + + The surface is undocumented — GitHub publishes no reference for the inference + API at all, and the token response carries no websocket URL — so the protocol + was established by probing: it lives on the same host as the HTTP API, speaks + the same `response.*` events one per text frame, and takes a *flat* request + frame with `model` at the top level. Nesting the body under `response` is + rejected, as is omitting `type`. `scripts/ws_probe.py` and + `scripts/ws_explore.py` are the tools that established this, kept for when it + changes. + + A model that does not advertise the transport is refused with an error frame + naming the alternative, rather than left waiting on the socket. Refusals and + malformed frames are recorded with a `failure_kind` like every other path, so + a failed attempt leaves something to diagnose from. WebSocket turns appear in + the dashboard under `ws:/responses`, and their transcript renders through the + same reassembly as SSE — only the transport differs. +- `scripts/ws_check.py` sweeps every model the catalog says supports the + transport, discovering the list rather than hard-coding it, and asserts the + event vocabulary, monotonic sequence numbers, termination, text and usage. - `scripts/replay.py` replays a captured request at the upstream or back through the proxy and times the SSE stream event by event, with no stall watchdog. -- `upstream_read_timeout_seconds` (default `900`, `0` disables, env +- `scripts/protocol_check.py` asserts protocol conformance against a running + proxy. The property that matters for a translating gateway is not "did it + return 200" but "did it return the shape the *client* asked for", so every + check is an assertion about the client-facing response: a Gemini caller must + receive `candidates`, never the `choices` the upstream actually sent. Covers + all four surfaces in both streaming and non-streaming modes, tool calls and + tool-result round trips, stream terminators (`message_stop`, `[DONE]`, + `response.completed`), token counting, and the error paths — an unknown model + must not arrive as a `200` SSE stream that never produces an event. +- **Prompt-cache statistics** at `GET /api/cache` and on the dashboard: where + input tokens came from (served from cache / written to cache / fresh), the + disposition of recent requests, and a per-model hit rate. A single global + number cannot tell you *which* conversation stopped matching its prefix, which + is the failure that quietly multiplies the bill. Savings are reported in both + directions — the read discount, the write premium, and the net — because + caching is not free. +- **Body capture can be toggled at runtime**, from the request browser or via + `POST /api/config/debug`. It previously required restarting the proxy with + `--debug`, by which point the request you wanted to inspect was gone; the flag + is read live, so it applies from the next call. It is not written back to + `config.yaml`, since capture puts prompts and any credentials they carry into + memory and the log and should lapse on restart. `GET /health` now reports it. + The `/api/config/` routes are guarded by the API key when one is configured — + read-only dashboard endpoints stay open, but turning on body capture must not + be something an unauthenticated caller can do.- `upstream_read_timeout_seconds` (default `900`, `0` disables, env `GHC_PROXY_UPSTREAM_READ_TIMEOUT`). Bounds the silence *between* reads from an upstream response rather than the total duration, so long streaming answers are unaffected. Without it a half-open connection yields no data, no error and @@ -37,6 +84,151 @@ All notable changes to this project will be documented in this file. room to spare. ### Changed +- **The cache panel is per model, and says why a cell is empty.** The panel + opened with one aggregate stacked bar, which cannot say *which* model stopped + matching — the question a collapsing hit rate raises. The bar is now a + Distribution column, one per model, and what remains at the top is the colour + key those bars need. Below the table, a footnote separates the two reasons a + row comes back empty: a prompt too short to be eligible at all, and one long + enough but never sent before, since nothing can be read back on first sight. + + **`Written` only appears when something was written.** Only the Anthropic + surface with an explicit `cache_control` marker ever bills a cache write. + Every other surface caches implicitly: the first call reads nothing, the next + reads the whole prefix back, and no write is billed. Measured across twelve + calls with a 27k-token prefix, exactly one reported a write. The column and + its swatch are hidden outright on a workload that never writes, rather than + showing a wall of zeros for an event that cannot happen. + + Measured while investigating: `claude-haiku-4.5` cached a 6902-token prefix + but not a 4082-token one, so a prompt in the low thousands legitimately + produces an empty row. Every surface — chat, messages, responses, Gemini and + WebSocket, streamed and not — was confirmed to carry + `copilot_usage.token_details`, so an empty row is never the proxy failing to + read what upstream reported. +- **Cache savings come from the model's own rates, not a price list.** The + per-model saving was `list price × (1 − 0.1)` for reads and + `× (1.25 − 1)` for writes — one hard-coded discount applied to every model. + Copilot states its actual per-token rate for every token type it charges, in + `copilot_usage.token_details`, and the entries are not the same on every + model: `claude-haiku-4.5` prices cache writes above its input rate, + `gemini-3.5-flash` prices them at zero, `gpt-5.5` does not price them at all, + and `gpt-4o-mini` prices nothing because Copilot includes it. The old maths + claimed a saving on models that are free. + + `/api/cache` now reports `saved_nano_aiu` in the same AI units the rest of + the dashboard bills in, computed per request from the rates that request + reported, and drops `saved_usd` / `write_premium_usd` / `net_saved_usd`. + + A `null` `saved_nano_aiu` now means no response reported rates to compute one + from, distinct from `0`, which is a real figure: nothing was cached, so + nothing was saved. +- **Failed attempts no longer count as consumption.** `request_count` counted + every record, so a burst of rejected calls inflated the request total, added + an empty-named row to the per-model cache table, and dragged every cache + disposition toward "uncached" — a rate computed over requests that consumed + nothing. Statistics now cover requests that produced an answer, `/api/stats` + reports `failed_requests` separately, and the overview links to them. Token + and billing totals still count either way: a stream cut off partway consumed + what it consumed, and hiding that would understate the bill. + + The predicate — non-2xx, or a `failure_kind` on an otherwise successful + status — now has one definition shared by the statistics, the failures-only + filter and the dashboard, instead of three. +- **Dashboard restructured around consumption.** The landing page opened with + eight identically-weighted stat cards followed by the full 78-row model + catalogue, so the number that tracks spend had no more prominence than + `bytes_received`, and reference data dominated a page you open to check usage. + It now leads with premium requests, input tokens (with the cache-hit share) + and output tokens; quota bars per SKU sit directly beneath, since that is the + same quantity seen from the other end. Traffic and proxy health follow, then a + recent-requests preview, with the model catalogue folded away. + - The three pages share one stylesheet at `/app.css` instead of three drifting + inline copies, and carry the same persistent nav with a live readiness and + version indicator — previously each page offered only a lone + `← Dashboard` link. + - **Wide screens are used.** The old 1180px cap left a 2560px monitor half + empty while the twelve-column request table still scrolled sideways. Width + is now per-page: the request table gets 2000px and the metric list 1600px, + since both are data a wide screen genuinely helps you read, while the + overview keeps a 1280px measure because a handful of large numbers only + drift apart when stretched. Message text keeps a 100ch measure so prose + stays readable at full width; tool payloads are code and stay unconstrained. + - `/health` data (per-SKU quota, Copilot token expiry, uptime, readiness, + models loaded) was reachable but shown nowhere; it is now on the overview. + The request list stays on its own tab — the overview only reports how many + requests failed and links across. + - **Debug bodies render as a conversation.** `--debug` stores request and + response bodies verbatim, which is the right thing to store and the wrong + thing to read: finding what the model actually said meant scrolling past + tool schemas, content-filter blocks and, for streams, every SSE frame. The + detail view now reconstructs the exchange — system/user/assistant turns, + tool calls and results as collapsible cards, token and stop-reason chips — + and reassembles streamed fragments into the message they describe. All + three wire formats are handled, and none of them agree: Anthropic numbers + content blocks and tags every delta, chat completions hides tool arguments + inside `choices[].delta.tool_calls[]`, and the Responses API uses a flat + `response.*` vocabulary where the delta is a bare string. Reasoning is + picked up under every name the families use (`reasoning_text`, + `reasoning_content`, `response.reasoning_summary_text.delta`) rather than + dropped, and a turn that produced no visible output says whether it ran out + of token budget instead of reporting an empty stream. The raw bodies remain + one click away on a `Raw` tab. + - A third tab shows what the wire actually carried, which is the question when + a stream stalls, repeats, or ends somewhere unexpected. For a stream it + lists **every SSE frame** — sequence number, event name and a one-line gist, + each expandable to the full payload — so the `[DONE]` sentinel, a missing + `message_stop` or a duplicated index is visible instead of buried in a + scroll of raw text. For a non-streaming completion it lists **every + top-level field**, including the ones the conversation view has no place for + (`content_filter_results`, `prompt_filter_results`, `system_fingerprint`, + `service_tier`, `copilot_usage`). + - The request table's full nanosecond ISO timestamp consumed a third of the + row width; it renders as local wall-clock time with the exact value in the + tooltip, and the table scrolls horizontally rather than letting the panel + clip columns off the right edge, with the expanded detail pinned to the left + so it stays readable while the table scrolls. Empty results say so instead + of showing a header row with nothing under it. + - **Each request now says what happened, not just what was said.** The + response section opened with a row of bare chips: the numbers were there, + but not their meaning. It now leads with the outcome, normalized across + surfaces and explained — `length`, `max_tokens`, `max_output_tokens` and + `MAX_TOKENS` are the same event on four different APIs, and all four mean + the answer on screen is not the whole answer. Endings are colour-coded by + whether they are normal (`end_turn`, `stop`, `completed`), truncated, or a + refusal or filter block. + + Beneath it sit the facts each surface actually reports, omitted when + absent: reasoning effort, service tier, prompt-cache retention, response + verbosity and output item types on the Responses API; the backend build + fingerprint on chat completions, which is worth noticing because a change + there can shift results for identical input; cache-write TTL split and + inference region on Anthropic; rejected speculative tokens, which are + billed. Content-filter verdicts appear only when something was actually + flagged — an all-clear on four categories on every request is noise. + - The `Probes` column is gone. Keepalive only runs on the Anthropic streaming + path, so four of the five request paths rendered a dash that read as "zero + probes sent" when it meant "not measured here". The count now rides in the + `Idle` tooltip, which is where it was useful anyway — the pair is what + assigns blame for a stall. + - **Protocol-specific parameters moved out of the list and into the detail.** + The three surfaces agree on very little, so a table column for any of them + is structurally empty for the other two: `New` (cache writes) only exists on + Anthropic, and `Session` only when the client encodes one into + `metadata.user_id` the way Claude Code does. Both columns are gone; the + detail now carries a panel naming the surface and listing what that request + actually set — `top_k`, `stop_sequences`, `cache_control` marks and + `thinking` for Anthropic; `seed`, penalties, `response_format` and + `logprobs` for chat completions; `reasoning.effort`, `max_output_tokens`, + `store` and `truncation` for the Responses API; `generationConfig` and + `safetySettings` for Gemini. Absent parameters are omitted rather than shown + as blanks. A Gemini request is labelled `Gemini → OpenAI Chat Completions + (translated)`, because the proxy rewrites it before forwarding and the + recorded body is the rewritten one — `topK` and `safetySettings` have no + counterpart and are dropped, which the panel says instead of leaving the + reader to assume they took effect. Gemini's `contents`/`parts`, + `functionCall` and `functionResponse` shapes render like every other + surface's. - **`auto_upgrade` now defaults to `true`**, including for config files written before the setting existed. The proxy checks GitHub releases on startup and replaces its own binary when a newer version is published; the replacement @@ -56,6 +248,24 @@ All notable changes to this project will be documented in this file. customized mappings in current ones ### Fixed +- **Output token totals were not comparable across surfaces.** The two + conventions for reasoning tokens disagree about whether they are already + counted: a Responses turn reports `input 11 + output 17 == total 28` with 10 + of that output being reasoning, while the translated Gemini surface reports + `prompt 5 + completion 1 + reasoning 97 == total 103`. Taken at face value the + dashboard showed a reasoning share of 262%. `output_tokens` is now normalized + to the true total on both, the way `input_tokens` already was. +- **Gemini requests lost parameters that had exact counterparts.** The + translation to chat completions carried `temperature`, `topP`, + `maxOutputTokens` and `stopSequences` and dropped the rest of + `generationConfig` in silence, so a client's `seed`, `candidateCount`, + `presencePenalty`, `frequencyPenalty` or `responseMimeType` had no effect and + nothing said why. They now map to `seed`, `n`, `presence_penalty`, + `frequency_penalty` and `response_format` (with `responseSchema` carried over + as a `json_schema` format). `topK` and `safetySettings` genuinely have no + counterpart in the chat completions schema and are still dropped — the + dashboard names them rather than describing the loss in general terms, and a + test asserts they are not invented. - **Non-2xx upstreams on `/v1/messages` reached the client as an empty `200` stream.** `messages_direct` only intercepted `400`; a `401`, `403`, `429` or `5xx` fell through and was wrapped in a `200 text/event-stream` whose body was diff --git a/Cargo.lock b/Cargo.lock index 5b59bdb..9775012 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -104,6 +104,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "31b698c5f9a010f6573133b09e0de5408834d0c82f8d7475a89fc1867a71cd90" dependencies = [ "axum-core", + "base64", "bytes", "form_urlencoded", "futures-util", @@ -122,8 +123,10 @@ dependencies = [ "serde_json", "serde_path_to_error", "serde_urlencoded", + "sha1", "sync_wrapper", "tokio", + "tokio-tungstenite 0.29.0", "tower", "tower-layer", "tower-service", @@ -394,6 +397,12 @@ dependencies = [ "syn 2.0.118", ] +[[package]] +name = "data-encoding" +version = "2.11.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a4ae5f15dda3c708c0ade84bfee31ccab44a3da4f88015ed22f63732abe300c8" + [[package]] name = "der" version = "0.7.10" @@ -759,6 +768,7 @@ dependencies = [ "serde_norway", "tiktoken-rs", "tokio", + "tokio-tungstenite 0.28.0", "toml", "tower", "tower-http 0.7.0", @@ -1905,6 +1915,17 @@ dependencies = [ "serde", ] +[[package]] +name = "sha1" +version = "0.10.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a978451301f4db1d02937a4ab3ccce137717b81826e79b7d49ffe3244a13c3b8" +dependencies = [ + "cfg-if", + "cpufeatures", + "digest", +] + [[package]] name = "sha2" version = "0.10.9" @@ -2237,6 +2258,34 @@ dependencies = [ "tokio", ] +[[package]] +name = "tokio-tungstenite" +version = "0.28.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d25a406cddcc431a75d3d9afc6a7c0f7428d4891dd973e4d54c56b46127bf857" +dependencies = [ + "futures-util", + "log", + "rustls", + "rustls-native-certs", + "rustls-pki-types", + "tokio", + "tokio-rustls", + "tungstenite 0.28.0", +] + +[[package]] +name = "tokio-tungstenite" +version = "0.29.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8f72a05e828585856dacd553fba484c242c46e391fb0e58917c942ee9202915c" +dependencies = [ + "futures-util", + "log", + "tokio", + "tungstenite 0.29.0", +] + [[package]] name = "tokio-util" version = "0.7.18" @@ -2428,6 +2477,41 @@ version = "0.2.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e421abadd41a4225275504ea4d6566923418b7f05506fbc9c0fe86ba7396114b" +[[package]] +name = "tungstenite" +version = "0.28.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8628dcc84e5a09eb3d8423d6cb682965dea9133204e8fb3efee74c2a0c259442" +dependencies = [ + "bytes", + "data-encoding", + "http", + "httparse", + "log", + "rand", + "rustls", + "rustls-pki-types", + "sha1", + "thiserror", + "utf-8", +] + +[[package]] +name = "tungstenite" +version = "0.29.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6c01152af293afb9c7c2a57e4b559c5620b421f6d133261c60dd2d0cdb38e6b8" +dependencies = [ + "bytes", + "data-encoding", + "http", + "httparse", + "log", + "rand", + "sha1", + "thiserror", +] + [[package]] name = "typenum" version = "1.20.1" @@ -2522,6 +2606,12 @@ version = "2.1.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "daf8dba3b7eb870caf1ddeed7bc9d2a049f3cfdfae7cb521b087cc33ae4c49da" +[[package]] +name = "utf-8" +version = "0.7.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "09cc8ee72d2a9becf2f2febe0205bbed8fc6615b7cb429ad062dc7b7ddd036a9" + [[package]] name = "utf8-zero" version = "0.8.1" diff --git a/Cargo.toml b/Cargo.toml index 39b97b4..530e6d6 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -16,8 +16,11 @@ name = "ghc-proxy" path = "src/main.rs" [dependencies] -axum = { version = "0.8", features = ["json"] } +axum = { version = "0.8", features = ["json", "ws"] } tokio = { version = "1", features = ["full"] } +# Upstream transport for the catalog's `ws:/responses` surface. rustls with the +# platform trust store, to match reqwest and keep enterprise CAs working. +tokio-tungstenite = { version = "0.28", default-features = false, features = ["connect", "rustls-tls-native-roots"] } tower = { version = "0.5", features = ["util"] } tower-http = { version = "0.7", features = ["fs"] } reqwest = { version = "0.13", default-features = false, features = ["json", "rustls", "stream", "form"] } diff --git a/docs/api.md b/docs/api.md index eac58a5..a978f35 100644 --- a/docs/api.md +++ b/docs/api.md @@ -37,11 +37,131 @@ require a key on the LLM endpoints; see [Authentication](#authentication) below. | `GET /metrics` | OpenMetrics exposition endpoint | | `GET /requests` | Request browser | | `GET /api/stats` | Dashboard statistics (JSON) | +| `GET /api/cache` | Prompt-cache statistics, overall and per model | | `GET /api/requests` | Recent requests (JSON) | | `GET /api/audit` | Filtered audit records | | `GET /api/audit/summary` | Aggregated audit summary | | `POST /api/config/reload` | Reload `config.yaml` without restart | +| `POST /api/config/debug` | Turn request/response body capture on or off | | `GET /openapi.json` | OpenAPI v3 specification of the LLM endpoints | +| `GET /v1/responses` (Upgrade) | Responses API over WebSocket | + +## Responses over WebSocket + +Several models advertise `ws:/responses` in `supported_endpoints` alongside +`/responses`. `GET /v1/responses` (or `/responses`) with a WebSocket upgrade +exposes that transport. + +The protocol is the streaming Responses API with a different carrier: the same +`response.*` event vocabulary, one event per text frame, so a client already +written against the SSE stream needs no new parsing. Send one frame to start a +turn: + +```json +{"type": "response.create", "model": "gpt-5.5", "input": "...", "stream": true} +``` + +The frame is flat — `model` sits at the top level beside the request fields, not +nested under a `response` object. Omitting `type` or nesting the body is +rejected. + +A model that does not advertise the transport is refused with an error frame +naming the alternative rather than left waiting on the socket: + +```json +{"type": "error", "error": {"code": "unsupported_api_for_model", + "message": "Model 'claude-opus-4.6' does not support ws:/responses. Use POST /v1/responses instead."}} +``` + +WebSocket turns are recorded like any other request, under the endpoint +`ws:/responses`. + +Read-only dashboard endpoints are reachable without an API key so local +monitoring keeps working. The `/api/config/` routes are not: they mutate the +running process, and one of them turns on body capture, which writes whatever +the client sent — credentials included — into the request log. + +## Statistics and failed attempts + +`GET /api/stats` counts requests that produced an answer. Attempts that did not +— a non-2xx status, or a `failure_kind` on an otherwise successful one — are +reported separately as `failed_requests` and excluded from the rest, so a burst +of rejected calls cannot dilute a rate computed over requests that consumed +nothing. Token and billing totals count either way: a stream cut off partway +consumed what it consumed. + +## Prompt cache statistics + +`GET /api/cache` reports where input tokens came from. The hit rate is the early +warning for a broken prompt prefix: on an agent workload it should sit high and +stable, and a sudden drop means the prompt stopped matching and every turn is +paying full input price again. + +```json +{ + "totals": { + "input_tokens": 9397, + "cache_read_tokens": 4682, + "cache_creation_tokens": 4682, + "fresh_tokens": 33, + "hit_rate": 0.498, + "request_count": 4 + }, + "dispositions": { "served_from_cache": 1, "wrote_to_cache": 1, "no_cache": 2 }, + "sampled_requests": 4, + "by_model": [ + { + "model": "gemini-3.5-flash", + "requests": 4, + "input_tokens": 9397, + "cache_read_tokens": 4682, + "cache_creation_tokens": 0, + "fresh_tokens": 33, + "hit_rate": 0.498, + "saved_nano_aiu": 547830000 + } + ] +} +``` + +`totals` are the all-time running counters. `by_model` is derived from the +retained ring buffer, so it describes the most recent `sampled_requests` calls +rather than every one ever served. + +### What a zero means + +Copilot states its own per-token rates on each response, in +`copilot_usage.token_details`. `saved_nano_aiu` is computed from those rates +rather than from a price list, so a model Copilot includes at no charge +contributes nothing instead of an imagined discount. Positive means the cache +paid for itself; negative is the normal shape of a turn that populated a cache +it has not read back yet. `null` means no response for this model reported the +rates to compute one from — distinct from `0`, which is a real figure: nothing +was cached, so nothing was saved. + +`cache_creation_tokens` is only ever non-zero on the Anthropic surface with an +explicit `cache_control` marker. Every other surface caches implicitly: the +first call reads nothing, the next reads the whole prefix back, and no write is +billed. Measured across twelve calls with a 27k-token prefix, exactly one +reported a write, on `/v1/messages`. The dashboard hides the column outright +when nothing wrote. + +A model with no cache activity at all is usually not a fault either: Copilot +needs a minimum cacheable prefix before any of the prompt is eligible. +`claude-haiku-4.5` was observed caching a 6902-token prefix but not a +4082-token one. + +## Body capture + +`POST /api/config/debug` with `{"debug": true}` or `{"debug": false}` turns +request/response body capture on or off for the running process. The flag is +read live on every request, so it applies from the next call — no restart, and +no need to have predicted in advance that you would want the bodies. + +It is deliberately not written back to `config.yaml`. Capture puts prompts, tool +output and any credentials they carry into memory and the log, so it lapses on +restart rather than staying on because someone forgot. `GET /health` reports the +current value as `debug`. Streaming (SSE) is supported on the chat, responses, and messages endpoints by setting `"stream": true` in the request body. The Gemini surface streams via the diff --git a/public/app.css b/public/app.css new file mode 100644 index 0000000..42f2ebe --- /dev/null +++ b/public/app.css @@ -0,0 +1,372 @@ +/* ghc-proxy dashboard — shared design system. + * + * Served from /app.css so the three pages share one visual language instead of + * carrying three drifting copies of the same rules. */ + +:root { + color-scheme: dark; + + --bg: #0d1117; + --surface: #161b22; + --surface-2: #21262d; + --border: #30363d; + --border-soft: #21262d; + --text: #e6edf3; + --muted: #8b949e; + --accent: #58a6ff; + --ok: #3fb950; + --warn: #d29922; + --danger: #f85149; + + --radius: 8px; + --radius-sm: 6px; + --mono: ui-monospace, SFMono-Regular, Menlo, Consolas, monospace; + + /* Page width is per-page, not global: the overview is a handful of large + * numbers and short labels, which just drift apart when stretched, while the + * request table and the metric list are data that a wide screen genuinely + * helps you read. Capped rather than fluid so an ultrawide does not turn a + * table row into a journey for the eye. */ + --page-max: 1280px; +} + +body[data-page="requests"] { --page-max: 2000px; } +body[data-page="metrics"] { --page-max: 1600px; } + +* { box-sizing: border-box; } + +body { + font-family: system-ui, -apple-system, "Segoe UI", Roboto, sans-serif; + margin: 0; + background: var(--bg); + color: var(--text); + font-size: 14px; + line-height: 1.5; +} + +a { color: var(--accent); text-decoration: none; } +a:hover { text-decoration: underline; } + +.wrap { + max-width: var(--page-max); + margin: 0 auto; + padding: 0 1.5rem 4rem; +} + +/* ---------------------------------------------------------------- nav ---- */ +/* Persistent across all three pages: the "trunk test" wants every page to + * answer what this is, where you are, and what the other sections are. */ + +.nav { + position: sticky; + top: 0; + z-index: 10; + background: rgba(13, 17, 23, 0.85); + backdrop-filter: blur(8px); + border-bottom: 1px solid var(--border); + margin-bottom: 1.75rem; +} + +.nav-inner { + max-width: var(--page-max); + margin: 0 auto; + padding: 0 1.5rem; + display: flex; + align-items: center; + gap: 1.5rem; + height: 52px; +} + +.brand { + font-weight: 600; + color: var(--text); + letter-spacing: -0.01em; + white-space: nowrap; +} +.brand:hover { text-decoration: none; } + +.nav-links { display: flex; gap: 0.25rem; flex: 1; } + +.nav-links a { + color: var(--muted); + padding: 0.3rem 0.7rem; + border-radius: var(--radius-sm); + font-size: 0.875rem; +} +.nav-links a:hover { color: var(--text); background: var(--surface-2); text-decoration: none; } + +/* Current section, set via */ +body[data-page="overview"] .nav-links a[href="/"], +body[data-page="requests"] .nav-links a[href="/requests"], +body[data-page="metrics"] .nav-links a[href="/metrics/dashboard"] { + color: var(--text); + background: var(--surface); + box-shadow: inset 0 0 0 1px var(--border); +} + +.nav-status { + display: flex; + align-items: center; + gap: 0.5rem; + font-size: 0.8rem; + color: var(--muted); + white-space: nowrap; +} + +.dot { width: 8px; height: 8px; border-radius: 50%; background: var(--muted); flex: none; } +.dot.ok { background: var(--ok); } +.dot.warn { background: var(--warn); } +.dot.err { background: var(--danger); } + +/* -------------------------------------------------------------- panels --- */ + +.panel { + background: var(--surface); + border: 1px solid var(--border); + border-radius: var(--radius); + margin-bottom: 1.25rem; + overflow: hidden; +} + +.panel-head { + display: flex; + align-items: baseline; + justify-content: space-between; + gap: 1rem; + padding: 0.7rem 1rem; + border-bottom: 1px solid var(--border); +} + +.panel-head h2 { + margin: 0; + font-size: 0.72rem; + font-weight: 600; + text-transform: uppercase; + letter-spacing: 0.08em; + color: var(--muted); +} + +.panel-head .aside { font-size: 0.8rem; color: var(--muted); } +.panel-body { padding: 1rem; } +.panel-body.flush { padding: 0; } + +/* -------------------------------------------------------------- metrics -- */ +/* Deliberately unequal: the hero row is what the page is *for*, so it gets + * size, and everything else is demoted rather than competing with it. */ + +.hero { + display: grid; + grid-template-columns: repeat(auto-fit, minmax(200px, 1fr)); + gap: 1px; + background: var(--border); +} + +.hero .metric { background: var(--surface); padding: 1.1rem 1.25rem; } + +.metric .label { + color: var(--muted); + font-size: 0.72rem; + text-transform: uppercase; + letter-spacing: 0.07em; +} + +.metric .value { + font-size: 2rem; + font-weight: 600; + line-height: 1.15; + margin-top: 0.35rem; + letter-spacing: -0.02em; + font-variant-numeric: tabular-nums; +} + +.metric .sub { color: var(--muted); font-size: 0.8rem; margin-top: 0.2rem; } +.metric .sub b { color: var(--text); font-weight: 600; } + +/* Secondary stats: same information architecture, a third of the weight. */ +.stat-grid { + display: grid; + grid-template-columns: repeat(auto-fit, minmax(150px, 1fr)); + gap: 0.9rem 1.5rem; +} + +.stat .label { + color: var(--muted); + font-size: 0.72rem; + text-transform: uppercase; + letter-spacing: 0.07em; +} +.stat .value { + font-size: 1.15rem; + font-weight: 600; + font-variant-numeric: tabular-nums; + margin-top: 0.15rem; +} +.stat .value.ok { color: var(--ok); } +.stat .value.warn { color: var(--warn); } +.stat .value.err { color: var(--danger); } + +.cols-2 { display: grid; grid-template-columns: repeat(auto-fit, minmax(320px, 1fr)); gap: 1.25rem; } +.cols-2 > .panel { margin-bottom: 0; } + +/* ---------------------------------------------------------------- quota -- */ + +.quota-row { display: grid; gap: 0.3rem; padding: 0.7rem 0; border-bottom: 1px solid var(--border-soft); } +.quota-row:last-child { border-bottom: none; padding-bottom: 0; } +.quota-row:first-child { padding-top: 0; } + +.quota-top { display: flex; justify-content: space-between; align-items: baseline; gap: 1rem; } +.quota-name { font-family: var(--mono); font-size: 0.82rem; } +.quota-val { font-size: 0.82rem; color: var(--muted); font-variant-numeric: tabular-nums; } +.quota-val b { color: var(--text); font-weight: 600; } + +.bar { height: 6px; border-radius: 3px; background: var(--surface-2); overflow: hidden; } +.bar > span { display: block; height: 100%; background: var(--ok); border-radius: 3px; } +.bar > span.warn { background: var(--warn); } +.bar > span.err { background: var(--danger); } + +/* --------------------------------------------------------------- tables -- */ + +table { width: 100%; border-collapse: collapse; font-size: 0.85rem; } + +th, td { + text-align: left; + padding: 0.5rem 0.75rem; + border-bottom: 1px solid var(--border); + white-space: nowrap; +} + +th { + color: var(--muted); + text-transform: uppercase; + font-size: 0.7rem; + letter-spacing: 0.05em; + font-weight: 600; +} + +tbody tr:last-child td { border-bottom: none; } +td.num, th.num { text-align: right; font-variant-numeric: tabular-nums; } +.mono { font-family: var(--mono); } +.muted { color: var(--muted); } +.status-ok { color: var(--ok); } +.status-err { color: var(--danger); } + +/* -------------------------------------------------------------- controls -- */ + +button, .btn { + background: var(--surface-2); + color: var(--text); + border: 1px solid var(--border); + border-radius: var(--radius-sm); + padding: 0.35rem 0.75rem; + cursor: pointer; + font: inherit; + font-size: 0.85rem; +} +button:hover:not(:disabled) { border-color: var(--muted); } +button:disabled { opacity: 0.45; cursor: default; } + +input[type="text"], input:not([type]) { + background: var(--bg); + color: var(--text); + border: 1px solid var(--border); + border-radius: var(--radius-sm); + padding: 0.45rem 0.7rem; + font: inherit; + font-size: 0.85rem; +} +input:focus { outline: none; border-color: var(--accent); } + +.toolbar { + display: flex; + gap: 0.75rem; + align-items: center; + flex-wrap: wrap; + font-size: 0.85rem; + color: var(--muted); +} +.toolbar label { display: flex; gap: 0.35rem; align-items: center; cursor: pointer; } + +.pager { display: flex; gap: 0.5rem; align-items: center; padding: 0.75rem 1rem; color: var(--muted); font-size: 0.85rem; } + +/* Collapsible reference material — present but never competing with the + * live numbers above it. */ +details.fold > summary { + cursor: pointer; + padding: 0.7rem 1rem; + color: var(--muted); + font-size: 0.72rem; + text-transform: uppercase; + letter-spacing: 0.08em; + font-weight: 600; + list-style: none; + user-select: none; +} +details.fold > summary::-webkit-details-marker { display: none; } +details.fold > summary::before { content: "▸ "; } +details.fold[open] > summary::before { content: "▾ "; } +details.fold > summary:hover { color: var(--text); } +details.fold[open] > summary { border-bottom: 1px solid var(--border); } + +/* Where the input tokens came from, at a glance. Three shares of one bar read + * faster than three numbers you have to divide in your head. */ +.split { display: flex; height: 8px; border-radius: 4px; overflow: hidden; background: var(--surface-2); } +.split > span { display: block; height: 100%; } +.split .s-read { background: var(--ok); } +.split .s-write { background: var(--warn); } +.split .s-fresh { background: #6e7681; } +/* Inline in a table row, where 8px would crowd the text beside it. */ +.split.mini { height: 6px; min-width: 120px; } + +.legend { display: flex; flex-wrap: wrap; gap: 0.9rem; margin-top: 0.55rem; font-size: 0.78rem; color: var(--muted); } +/* Standing on its own as a key rather than captioning a bar above it. */ +.legend:first-child { margin-top: 0; } +.legend span { display: flex; align-items: center; gap: 0.35rem; } +.legend i { width: 8px; height: 8px; border-radius: 2px; flex: none; } +.legend b { color: var(--text); font-weight: 600; font-variant-numeric: tabular-nums; } + +.dim-note { color: var(--muted); font-size: 0.85rem; } + +/* A setting you can see is a setting you should be able to change. */ +.toggle { + display: inline-flex; + align-items: center; + gap: 0.45rem; + margin-top: 0.15rem; + padding: 0.2rem 0.6rem 0.2rem 0.3rem; + border-radius: 999px; + font-size: 0.9rem; + font-weight: 600; + color: var(--muted); +} +.toggle .knob { + width: 26px; + height: 15px; + border-radius: 999px; + background: var(--border); + position: relative; + transition: background 0.12s ease; + flex: none; +} +.toggle .knob::after { + content: ""; + position: absolute; + top: 2px; + left: 2px; + width: 11px; + height: 11px; + border-radius: 50%; + background: var(--muted); + transition: transform 0.12s ease, background 0.12s ease; +} +.toggle.on { color: var(--warn); border-color: var(--warn); } +.toggle.on .knob { background: rgba(210, 153, 34, 0.35); } +.toggle.on .knob::after { transform: translateX(11px); background: var(--warn); } +.toggle:disabled { opacity: 0.6; cursor: progress; } + +.empty { color: var(--muted); padding: 1.25rem 1rem; text-align: center; font-size: 0.875rem; } + +.scroll-y { max-height: 420px; overflow-y: auto; } + +/* The request table carries more columns than fit on a laptop. Scroll it + * rather than let the panel clip columns off the right edge. */ +.scroll-x { overflow-x: auto; } diff --git a/public/dashboard.html b/public/dashboard.html index 90c033d..4d05ab9 100644 --- a/public/dashboard.html +++ b/public/dashboard.html @@ -3,83 +3,436 @@ - ghc-proxy · Dashboard - + ghc-proxy · Overview + - -

GitHub Copilot API Proxy

-
-

Supported models

- - - -
IDDisplay nameOwned by
-
- Browse requests → -  |  - Metrics dashboard → + + + +
+ +
+
+

Consumption

+ +
+
+
+ + + + +
+
+

Traffic

+
+
+
+

Proxy

+
+
+
+ + +
+
+

Prompt cache

+ +
+
+
+ + + + + + + + + + + + + +
ModelDistributionRequestsInputHit rateCachedWrittenSaved
+
No cache activity recorded yet.
+
+
+
+ + +
+ Supported models +
+ + + +
IDDisplay nameOwned by
+
+
+ diff --git a/public/requests.html b/public/requests.html index ae580c8..0ebfb6b 100644 --- a/public/requests.html +++ b/public/requests.html @@ -4,70 +4,156 @@ ghc-proxy · Requests + - -

Requests ← Dashboard

-
- - - -
- - - - - - - - - - - - - - - -
TimeSessionEndpointModelStatusInNewOutIdleProbesDuration
-
- - - + + + +
+
+
+

Requests

+ + + + + + +
+
+ + + + + + + + + + + + +
TimeEndpointModelStatusInOutIdleDuration
+ +
+ + + +
+
+
diff --git a/scripts/anthropic_cache_probe.py b/scripts/anthropic_cache_probe.py new file mode 100644 index 0000000..286f05a --- /dev/null +++ b/scripts/anthropic_cache_probe.py @@ -0,0 +1,79 @@ +"""Why does the Anthropic surface never populate the prompt cache? + +The proxy forwards `cache_control` and Copilot echoes the full cache-accounting +shape back -- all zeros. This tries the remaining client-side levers one at a +time so the failing one can be named rather than guessed at. +""" + +import json +import sys +import urllib.error +import urllib.request + +BASE = f"http://127.0.0.1:{sys.argv[1] if len(sys.argv) > 1 else '8399'}" +MODEL = sys.argv[2] if len(sys.argv) > 2 else "claude-haiku-4.5" + + +def attempt(label, system, beta=None, ttl=None): + payload = { + "model": MODEL, + "max_tokens": 8, + "system": system, + "messages": [{"role": "user", "content": "Reply with the word ok."}], + } + headers = {"content-type": "application/json"} + if beta: + headers["anthropic-beta"] = beta + req = urllib.request.Request( + BASE + "/v1/messages", data=json.dumps(payload).encode(), headers=headers + ) + try: + with urllib.request.urlopen(req, timeout=120) as r: + u = json.loads(r.read())["usage"] + except urllib.error.HTTPError as e: + print(f"{label:<42}{e.code} {e.read()[:90].decode(errors='replace')}") + return + print( + f"{label:<42}in={u.get('input_tokens'):>6} " + f"write={u.get('cache_creation_input_tokens'):>6} " + f"read={u.get('cache_read_input_tokens'):>6}" + ) + + +def block(chars, ttl=None): + cc = {"type": "ephemeral"} + if ttl: + cc["ttl"] = ttl + return [{"type": "text", "text": ("cache me. " * 30000)[:chars], "cache_control": cc}] + + +print(f"model: {MODEL}\n") +# The text the other scripts use. Same character count as the block above but +# far fewer tokens, which is the whole question. +DEMO = ("The proxy records what the upstream reports. " * 520)[:22880] +# Each pair is sent twice: a write can only be observed as a read on the retry. +for label, kwargs in [ + ("4k chars, default beta", dict(system=block(4000))), + ("23k chars, default beta", dict(system=block(23000))), + ("92k chars, default beta", dict(system=block(92000))), + ( + "23k chars, prompt-caching beta", + dict(system=block(23000), beta="prompt-caching-2024-07-31"), + ), + ("23k chars, 1h ttl", dict(system=block(23000, ttl="1h"))), + ("23k chars, plain string system", dict(system=("cache me. " * 30000)[:23000])), + ( + "22.9k chars of demo filler", + dict( + system=[ + { + "type": "text", + "text": DEMO, + "cache_control": {"type": "ephemeral"}, + } + ] + ), + ), +]: + attempt(f"{label} (1st)", **kwargs) + attempt(f"{label} (2nd)", **kwargs) diff --git a/scripts/cache_demo.py b/scripts/cache_demo.py new file mode 100644 index 0000000..53830e2 --- /dev/null +++ b/scripts/cache_demo.py @@ -0,0 +1,96 @@ +"""Exercises the prompt cache on models that price it differently. + +The point is not that caching works -- that is already covered -- but that the +proxy reports a saving only where the model itself published rates to compute +one from. Run against a proxy with body capture off; only the recorded +statistics matter. +""" + +import json +import sys +import urllib.request + +BASE = sys.argv[1] if len(sys.argv) > 1 else "http://127.0.0.1:8399" + +# Long enough to clear Anthropic's minimum cacheable prefix. The content is +# irrelevant; only its length and its being byte-identical across turns matter. +FILLER = ("The proxy records what the upstream reports. " * 520)[:22880] + + +def post(path, payload): + req = urllib.request.Request( + BASE + path, + data=json.dumps(payload).encode(), + headers={"content-type": "application/json"}, + ) + with urllib.request.urlopen(req, timeout=180) as r: + return json.loads(r.read()) + + +def anthropic(turn): + return post( + "/v1/messages", + { + "model": "claude-haiku-4.5", + "max_tokens": 32, + "system": [ + { + "type": "text", + "text": FILLER, + "cache_control": {"type": "ephemeral"}, + } + ], + "messages": [{"role": "user", "content": f"Reply with the number {turn}."}], + }, + ) + + +def chat(model): + return post( + "/v1/chat/completions", + { + "model": model, + "max_tokens": 32, + "messages": [ + {"role": "system", "content": FILLER}, + {"role": "user", "content": "Reply with the word ok."}, + ], + }, + ) + + +def responses(model): + return post( + "/v1/responses", + { + "model": model, + "max_output_tokens": 32, + "instructions": FILLER, + "input": "Reply with the word ok.", + }, + ) + + +print("anthropic turn 1 (populates the cache)") +anthropic(1) +print("anthropic turn 2 (should read it back)") +anthropic(2) +print("gemini-3.5-flash (prices cache writes at zero)") +chat("gemini-3.5-flash") +# Responses-only, and observed to publish no cache_write rate at all -- the +# case that a flat zero in the Written column would misrepresent. +print("gpt-5.5 (publishes no cache-write rate)") +try: + responses("gpt-5.5") +except Exception as e: # noqa: BLE001 - a refusal is still a data point + print(f" gpt-5.5: {e}") + +cache = json.loads(urllib.request.urlopen(BASE + "/api/cache", timeout=30).read()) +print() +print(f"{'model':<24}{'in':>9}{'read':>9}{'write':>9}{'writes?':>9}{'saved nAIU':>14}") +for m in cache["by_model"]: + print( + f"{m['model']:<24}{m['input_tokens']:>9}{m['cache_read_tokens']:>9}" + f"{m['cache_creation_tokens']:>9}{str(m['prices_cache_write']):>9}" + f"{m['saved_nano_aiu']:>14}" + ) diff --git a/scripts/cache_sweep.py b/scripts/cache_sweep.py new file mode 100644 index 0000000..78f2724 --- /dev/null +++ b/scripts/cache_sweep.py @@ -0,0 +1,168 @@ +"""Sweeps models across every surface and reports which ones yielded cache facts. + +Answers one question: when the dashboard shows a dash, is that because the +model reported nothing, or because the proxy failed to read what it reported? +Needs body capture on, so the raw upstream payload can be inspected. +""" + +import json +import sys +import urllib.error +import urllib.request + +BASE = f"http://127.0.0.1:{sys.argv[1] if len(sys.argv) > 1 else '8399'}" + +# Long enough to be worth caching, and identical across calls so a second pass +# can read the first one back. +FILLER = ("The proxy records what the upstream reports. " * 520)[:22880] + + +def call(path, payload): + req = urllib.request.Request( + BASE + path, + data=json.dumps(payload).encode(), + headers={"content-type": "application/json"}, + ) + try: + with urllib.request.urlopen(req, timeout=180) as r: + r.read() + return None + except urllib.error.HTTPError as e: + return f"{e.code}" + except Exception as e: # noqa: BLE001 + return str(e)[:40] + + +def chat(model, stream): + return call( + "/v1/chat/completions", + { + "model": model, + "max_tokens": 24, + "stream": stream, + "messages": [ + {"role": "system", "content": FILLER}, + {"role": "user", "content": "Reply with the word ok."}, + ], + }, + ) + + +def messages(model, stream): + return call( + "/v1/messages", + { + "model": model, + "max_tokens": 24, + "stream": stream, + "system": [ + { + "type": "text", + "text": FILLER, + "cache_control": {"type": "ephemeral"}, + } + ], + "messages": [{"role": "user", "content": "Reply with the word ok."}], + }, + ) + + +def responses(model, stream): + return call( + "/v1/responses", + { + "model": model, + "max_output_tokens": 24, + "stream": stream, + "instructions": FILLER, + "input": "Reply with the word ok.", + }, + ) + + +def gemini(model, stream): + action = "streamGenerateContent" if stream else "generateContent" + return call( + f"/v1beta/models/{model}:{action}", + { + "systemInstruction": {"parts": [{"text": FILLER}]}, + "contents": [{"role": "user", "parts": [{"text": "Reply with the word ok."}]}], + "generationConfig": {"maxOutputTokens": 24}, + }, + ) + + +# `/api/models` is the plain OpenAI list; only the full catalogue carries +# `supported_endpoints`, which is what decides where each model can be called. +with urllib.request.urlopen(BASE + "/v1/models/full/", timeout=30) as r: + catalog = json.loads(r.read()) +supported = { + m["id"]: set(m.get("supported_endpoints") or []) + for m in (catalog.get("data") or catalog.get("models") or []) +} + +PLAN = [ + ("chat", "/chat/completions", chat), + ("messages", "/v1/messages", messages), + ("responses", "/responses", responses), +] + +# One representative per family, so the sweep stays inside the rate limit. +WANTED = [ + "claude-haiku-4.5", + "claude-sonnet-4.5", + "claude-opus-4.6", + "gemini-3.5-flash", + "gemini-3-pro", + "gpt-5-mini", + "gpt-5.4", + "gpt-5.5", + "gpt-5.3-codex", + "grok-4.5", +] + +for model in WANTED: + caps = supported.get(model) + if caps is None: + print(f" {model}: not in catalog") + continue + for label, endpoint, fn in PLAN: + if endpoint not in caps: + continue + for stream in (False, True): + err = fn(model, stream) + tag = f"{model} {label}{' stream' if stream else ''}" + print(f" {tag}: {err or 'ok'}") + if "/chat/completions" in caps and model.startswith("gemini"): + for stream in (False, True): + err = gemini(model, stream) + print(f" {model} gemini{' stream' if stream else ''}: {err or 'ok'}") + +print() +audit = json.loads(urllib.request.urlopen(BASE + "/api/audit?per_page=200", timeout=30).read()) +print(f"{'endpoint':<26}{'model':<20}{'body':>10}{'copilot_usage':>15}{'details':>9}") +seen = set() +for i in audit["items"]: + body = i.get("response_body") + # An uncaptured body is its own state; calling it a stream would hide the + # fact that this row proves nothing either way. + shape = "none" if not body else "json" if body.startswith("{") else "stream" + key = (i["endpoint"], i["model"], shape) + if key in seen: + continue + seen.add(key) + print( + f"{i['endpoint']:<26}{i['model']:<20}{shape:>10}" + f"{str('copilot_usage' in (body or '')):>15}" + f"{str('token_details' in (body or '')):>9}" + ) + +print() +cache = json.loads(urllib.request.urlopen(BASE + "/api/cache", timeout=30).read()) +print(f"{'model':<24}{'in':>8}{'read':>8}{'write':>8}{'writes?':>9}{'saved':>14}") +for m in cache["by_model"]: + print( + f"{m['model']:<24}{m['input_tokens']:>8}{m['cache_read_tokens']:>8}" + f"{m['cache_creation_tokens']:>8}{str(m['prices_cache_write']):>9}" + f"{str(m['saved_nano_aiu']):>14}" + ) diff --git a/scripts/cache_write_probe.py b/scripts/cache_write_probe.py new file mode 100644 index 0000000..b5a2ae1 --- /dev/null +++ b/scripts/cache_write_probe.py @@ -0,0 +1,111 @@ +"""Does the Written column ever carry data outside the Anthropic surface? + +Sends a prefix well past the minimum cacheable size twice per surface: the +first call should populate the cache, the second read it back. If a surface +never reports a write, the column is dead weight there and the dashboard should +say so rather than print a zero. +""" + +import json +import sys +import urllib.error +import urllib.request + +BASE = f"http://127.0.0.1:{sys.argv[1] if len(sys.argv) > 1 else '8399'}" + +# ~28k tokens: past every observed minimum, so a surface that can write will. +BIG = ("cache me. " * 40000)[:92000] + + +def call(path, payload): + req = urllib.request.Request( + BASE + path, + data=json.dumps(payload).encode(), + headers={"content-type": "application/json"}, + ) + try: + with urllib.request.urlopen(req, timeout=180) as r: + r.read() + return None + except urllib.error.HTTPError as e: + return str(e.code) + except Exception as e: # noqa: BLE001 + return str(e)[:40] + + +def chat(model): + return call( + "/v1/chat/completions", + { + "model": model, + "max_tokens": 8, + "messages": [ + {"role": "system", "content": BIG}, + {"role": "user", "content": "Reply with the word ok."}, + ], + }, + ) + + +def messages(model): + return call( + "/v1/messages", + { + "model": model, + "max_tokens": 8, + "system": [ + {"type": "text", "text": BIG, "cache_control": {"type": "ephemeral"}} + ], + "messages": [{"role": "user", "content": "Reply with the word ok."}], + }, + ) + + +def responses(model): + return call( + "/v1/responses", + { + "model": model, + "max_output_tokens": 16, + "instructions": BIG, + "input": "Reply with the word ok.", + }, + ) + + +PLAN = [ + ("claude-haiku-4.5", "/v1/messages", messages), + ("claude-haiku-4.5", "/chat/completions", chat), + ("gemini-3.5-flash", "/chat/completions", chat), + ("gpt-5-mini", "/chat/completions", chat), + ("gpt-5-mini", "/responses", responses), + ("gpt-5.5", "/responses", responses), +] + +before = { + i["id"] + for i in json.loads( + urllib.request.urlopen(BASE + "/api/requests?per_page=500", timeout=30).read() + )["items"] +} + +for model, endpoint, fn in PLAN: + for turn in (1, 2): + err = fn(model) + if err: + print(f" {model} {endpoint} turn {turn}: {err}") + +items = json.loads( + urllib.request.urlopen(BASE + "/api/requests?per_page=500", timeout=30).read() +)["items"] +fresh = [i for i in items if i["id"] not in before] +fresh.reverse() + +print() +print(f"{'endpoint':<22}{'model':<20}{'in':>8}{'read':>8}{'write':>8}{'prices?':>9}") +for i in fresh: + print( + f"{i['endpoint']:<22}{i['model']:<20}{i['input_tokens']:>8}" + f"{i['cache_read_input_tokens']:>8}{i['cache_creation_input_tokens']:>8}" + f"{str(i.get('prices_cache_write')):>9}" + ) diff --git a/scripts/protocol_check.py b/scripts/protocol_check.py new file mode 100644 index 0000000..6f56e61 --- /dev/null +++ b/scripts/protocol_check.py @@ -0,0 +1,475 @@ +"""Protocol conformance sweep against a running ghc-proxy. + +The proxy is a translating gateway: a client speaks Anthropic, OpenAI chat +completions, the OpenAI Responses API or Gemini, and the upstream only ever +speaks chat completions or Responses. The property that matters is therefore +not "did it return 200" but "did it return the shape the *client* asked for" — +a Gemini client must get `candidates`, never `choices`, no matter what the +upstream sent back. + +Run with the proxy already listening: + + python scripts/protocol_check.py --port 8399 + +Every check is an assertion about the response the client actually receives. +Nothing here inspects the dashboard; this is about the wire. +""" + +from __future__ import annotations + +import argparse +import json +import sys +import urllib.error +import urllib.request +from dataclasses import dataclass, field + +TIMEOUT = 180 + + +@dataclass +class Result: + name: str + ok: bool + detail: str = "" + + +@dataclass +class Report: + results: list[Result] = field(default_factory=list) + + def check(self, name: str, ok: bool, detail: str = "") -> bool: + self.results.append(Result(name, ok, detail)) + return ok + + @property + def failed(self) -> list[Result]: + return [r for r in self.results if not r.ok] + + +def post(base: str, path: str, body: dict) -> tuple[int, str, str]: + """Returns (status, content_type, text). Never raises on HTTP errors.""" + req = urllib.request.Request( + base + path, + data=json.dumps(body).encode(), + headers={"Content-Type": "application/json"}, + method="POST", + ) + try: + with urllib.request.urlopen(req, timeout=TIMEOUT) as r: + return r.status, r.headers.get("Content-Type", ""), r.read().decode("utf-8", "replace") + except urllib.error.HTTPError as e: + return e.code, e.headers.get("Content-Type", ""), e.read().decode("utf-8", "replace") + + +def get(base: str, path: str) -> tuple[int, str]: + try: + with urllib.request.urlopen(base + path, timeout=TIMEOUT) as r: + return r.status, r.read().decode("utf-8", "replace") + except urllib.error.HTTPError as e: + return e.code, e.read().decode("utf-8", "replace") + + +def sse_events(text: str) -> list[tuple[str | None, str]]: + """Splits an SSE body into (event name, data payload) pairs.""" + out = [] + for frame in text.split("\n\n"): + name, data = None, [] + for line in frame.splitlines(): + if line.startswith("event:"): + name = line[6:].strip() + elif line.startswith("data:"): + data.append(line[5:].strip()) + if data or name: + out.append((name, "\n".join(data))) + return out + + +def sse_types(text: str) -> list[str]: + """The `type` of every JSON frame, which is how all three streaming + protocols label what a frame is.""" + types = [] + for name, data in sse_events(text): + if data == "[DONE]": + types.append("[DONE]") + continue + try: + j = json.loads(data) + except ValueError: + continue + types.append(j.get("type") or name or ("chat.chunk" if "choices" in j else "?")) + return types + + +# -------------------------------------------------------------------------- +# Anthropic /v1/messages +# -------------------------------------------------------------------------- + +def check_anthropic(base: str, rep: Report, model: str) -> None: + p = "anthropic" + + status, _, text = post(base, "/v1/messages", { + "model": model, "max_tokens": 60, + "messages": [{"role": "user", "content": "Reply with exactly: ok"}], + }) + j = json.loads(text) if status == 200 else {} + rep.check(f"{p}: sync 200", status == 200, text[:200]) + rep.check(f"{p}: sync type=message", j.get("type") == "message", str(j.get("type"))) + rep.check(f"{p}: sync role=assistant", j.get("role") == "assistant", str(j.get("role"))) + rep.check(f"{p}: sync content is block array", + isinstance(j.get("content"), list) and bool(j["content"]), + json.dumps(j.get("content"))[:160]) + rep.check(f"{p}: sync has stop_reason", j.get("stop_reason") is not None, str(j.get("stop_reason"))) + u = j.get("usage") or {} + rep.check(f"{p}: sync usage has input/output tokens", + "input_tokens" in u and "output_tokens" in u, json.dumps(u)[:160]) + rep.check(f"{p}: sync does NOT leak choices", "choices" not in j, "leaked OpenAI shape") + + status, ctype, text = post(base, "/v1/messages", { + "model": model, "max_tokens": 60, "stream": True, + "messages": [{"role": "user", "content": "Count 1 to 3."}], + }) + types = sse_types(text) + rep.check(f"{p}: stream 200", status == 200, text[:200]) + rep.check(f"{p}: stream content-type", "text/event-stream" in ctype, ctype) + for want in ("message_start", "content_block_start", "content_block_delta", + "content_block_stop", "message_delta", "message_stop"): + rep.check(f"{p}: stream has {want}", want in types, ",".join(types[:12])) + rep.check(f"{p}: stream ends with message_stop", + "message_stop" in types and types.index("message_stop") >= len(types) - 2, + ",".join(types[-3:])) + rep.check(f"{p}: stream frames carry event: names", + all(n for n, d in sse_events(text) if d and d != "[DONE]"), + "a frame was missing its event: line") + + status, _, text = post(base, "/v1/messages", { + "model": model, "max_tokens": 200, + "tools": [{ + "name": "get_weather", + "description": "Look up the weather", + "input_schema": {"type": "object", "properties": {"city": {"type": "string"}}, + "required": ["city"]}, + }], + "messages": [{"role": "user", "content": "Weather in Paris? Call the tool."}], + }) + j = json.loads(text) if status == 200 else {} + blocks = j.get("content") or [] + tool_use = [b for b in blocks if b.get("type") == "tool_use"] + rep.check(f"{p}: tools produce tool_use block", bool(tool_use), json.dumps(blocks)[:200]) + if tool_use: + rep.check(f"{p}: tool_use has name+input+id", + all(k in tool_use[0] for k in ("name", "input", "id")), + json.dumps(tool_use[0])[:200]) + rep.check(f"{p}: stop_reason=tool_use", j.get("stop_reason") == "tool_use", + str(j.get("stop_reason"))) + + # Multi-turn with a tool_result the client sends back. + if tool_use: + tu = tool_use[0] + status, _, text = post(base, "/v1/messages", { + "model": model, "max_tokens": 120, + "tools": [{"name": "get_weather", "description": "Look up the weather", + "input_schema": {"type": "object", + "properties": {"city": {"type": "string"}}, + "required": ["city"]}}], + "messages": [ + {"role": "user", "content": "Weather in Paris? Call the tool."}, + {"role": "assistant", "content": blocks}, + {"role": "user", "content": [{ + "type": "tool_result", "tool_use_id": tu["id"], + "content": [{"type": "text", "text": "18C and clear"}], + }]}, + ], + }) + rep.check(f"{p}: tool_result round-trip accepted", status == 200, text[:200]) + + status, _, text = post(base, "/v1/messages/count_tokens", { + "model": model, + "messages": [{"role": "user", "content": "hello world"}], + }) + j = json.loads(text) if status == 200 else {} + rep.check(f"{p}: count_tokens 200", status == 200, text[:200]) + rep.check(f"{p}: count_tokens returns input_tokens", + isinstance(j.get("input_tokens"), int) and j["input_tokens"] > 0, + json.dumps(j)[:160]) + + +# -------------------------------------------------------------------------- +# OpenAI chat completions /v1/chat/completions +# -------------------------------------------------------------------------- + +def check_chat(base: str, rep: Report, model: str) -> None: + p = "chat" + + status, _, text = post(base, "/v1/chat/completions", { + "model": model, "max_tokens": 60, + "messages": [{"role": "user", "content": "Reply with exactly: ok"}], + }) + j = json.loads(text) if status == 200 else {} + rep.check(f"{p}: sync 200", status == 200, text[:200]) + rep.check(f"{p}: sync has choices[]", isinstance(j.get("choices"), list) and bool(j["choices"]), + json.dumps(j)[:160]) + if j.get("choices"): + c = j["choices"][0] + rep.check(f"{p}: choice has message.role=assistant", + (c.get("message") or {}).get("role") == "assistant", json.dumps(c)[:160]) + rep.check(f"{p}: choice has finish_reason", c.get("finish_reason") is not None, + str(c.get("finish_reason"))) + u = j.get("usage") or {} + rep.check(f"{p}: usage has prompt/completion tokens", + "prompt_tokens" in u and "completion_tokens" in u, json.dumps(u)[:160]) + rep.check(f"{p}: sync does NOT leak Anthropic shape", "content" not in j, "leaked content[]") + + status, ctype, text = post(base, "/v1/chat/completions", { + "model": model, "max_tokens": 60, "stream": True, + "stream_options": {"include_usage": True}, + "messages": [{"role": "user", "content": "Count 1 to 3."}], + }) + frames = [d for _, d in sse_events(text) if d] + rep.check(f"{p}: stream 200", status == 200, text[:200]) + rep.check(f"{p}: stream content-type", "text/event-stream" in ctype, ctype) + rep.check(f"{p}: stream terminates with [DONE]", frames and frames[-1] == "[DONE]", + frames[-1][:80] if frames else "no frames") + parsed = [json.loads(d) for d in frames if d != "[DONE]"] + rep.check(f"{p}: every frame has choices[]", all("choices" in d for d in parsed), + "a frame was missing choices") + text_seen = "".join( + (c.get("delta") or {}).get("content") or "" + for d in parsed for c in d.get("choices") or [] + ) + rep.check(f"{p}: stream produced text", bool(text_seen.strip()), repr(text_seen[:80])) + rep.check(f"{p}: stream reports a finish_reason", + any(c.get("finish_reason") for d in parsed for c in d.get("choices") or []), + "none present") + + status, _, text = post(base, "/v1/chat/completions", { + "model": model, "max_tokens": 200, + "tools": [{"type": "function", "function": { + "name": "get_weather", "description": "Look up the weather", + "parameters": {"type": "object", "properties": {"city": {"type": "string"}}, + "required": ["city"]}}}], + "messages": [{"role": "user", "content": "Weather in Paris? Call the tool."}], + }) + j = json.loads(text) if status == 200 else {} + calls = ((j.get("choices") or [{}])[0].get("message") or {}).get("tool_calls") or [] + rep.check(f"{p}: tools produce tool_calls", bool(calls), json.dumps(j)[:200]) + if calls: + fn = calls[0].get("function") or {} + rep.check(f"{p}: tool_call has name", bool(fn.get("name")), json.dumps(calls[0])[:160]) + ok_args = False + try: + ok_args = isinstance(json.loads(fn.get("arguments") or ""), dict) + except ValueError: + pass + rep.check(f"{p}: tool_call arguments parse as JSON", ok_args, repr(fn.get("arguments"))[:160]) + + +# -------------------------------------------------------------------------- +# OpenAI Responses /v1/responses +# -------------------------------------------------------------------------- + +def check_responses(base: str, rep: Report, model: str) -> None: + p = "responses" + + status, _, text = post(base, "/v1/responses", { + "model": model, + "input": [{"role": "user", "content": "Reply with exactly: ok"}], + }) + j = json.loads(text) if status == 200 else {} + rep.check(f"{p}: sync 200", status == 200, text[:200]) + rep.check(f"{p}: sync has output[]", isinstance(j.get("output"), list), json.dumps(j)[:160]) + rep.check(f"{p}: sync has status", j.get("status") is not None, str(j.get("status"))) + u = j.get("usage") or {} + rep.check(f"{p}: usage has input/output tokens", + "input_tokens" in u and "output_tokens" in u, json.dumps(u)[:160]) + + status, ctype, text = post(base, "/v1/responses", { + "model": model, "stream": True, + "input": [{"role": "user", "content": "Count 1 to 3."}], + }) + types = sse_types(text) + rep.check(f"{p}: stream 200", status == 200, text[:200]) + rep.check(f"{p}: stream content-type", "text/event-stream" in ctype, ctype) + for want in ("response.created", "response.output_text.delta", "response.completed"): + rep.check(f"{p}: stream has {want}", want in types, ",".join(types[:10])) + rep.check(f"{p}: stream ends at response.completed", + types and types[-1] in ("response.completed", "[DONE]"), + ",".join(types[-3:])) + deltas = "".join( + json.loads(d).get("delta") or "" + for n, d in sse_events(text) + if d and d != "[DONE]" and json.loads(d).get("type") == "response.output_text.delta" + ) + rep.check(f"{p}: stream produced text", bool(deltas.strip()), repr(deltas[:80])) + + +# -------------------------------------------------------------------------- +# Gemini /v1beta/models/{model}:generateContent +# -------------------------------------------------------------------------- + +def check_gemini(base: str, rep: Report, model: str) -> None: + p = "gemini" + # Enough budget that internal reasoning cannot consume the whole allowance + # before any text is emitted — at 60 this turn intermittently finished on + # MAX_TOKENS with empty parts, which is a real upstream outcome and not + # something the shape checks should depend on. + body = { + "contents": [{"role": "user", "parts": [{"text": "Reply with exactly: ok"}]}], + "generationConfig": {"maxOutputTokens": 512}, + } + + status, _, text = post(base, f"/v1beta/models/{model}:generateContent", body) + j = json.loads(text) if status == 200 else {} + rep.check(f"{p}: sync 200", status == 200, text[:200]) + # The whole point of the Gemini surface: the client must never see the + # chat-completions shape the upstream actually returned. + rep.check(f"{p}: sync has candidates[]", + isinstance(j.get("candidates"), list) and bool(j["candidates"]), + json.dumps(j)[:200]) + rep.check(f"{p}: sync does NOT leak choices[]", "choices" not in j, "leaked OpenAI shape") + if j.get("candidates"): + cand = j["candidates"][0] + parts = (cand.get("content") or {}).get("parts") + finish = cand.get("finishReason") + rep.check(f"{p}: candidate has content.parts[]", isinstance(parts, list), + json.dumps(cand)[:200]) + # A turn cut off by the token limit owes no content; anything else does. + rep.check(f"{p}: candidate carries content unless truncated", + bool(parts) or finish == "MAX_TOKENS", + json.dumps(cand)[:200]) + rep.check(f"{p}: candidate has finishReason", finish is not None, str(finish)) + um = j.get("usageMetadata") or {} + rep.check(f"{p}: has usageMetadata with token counts", + "promptTokenCount" in um and "candidatesTokenCount" in um, json.dumps(um)[:160]) + + status, ctype, text = post(base, f"/v1beta/models/{model}:streamGenerateContent", body) + frames = [d for _, d in sse_events(text) if d and d != "[DONE]"] + rep.check(f"{p}: stream 200", status == 200, text[:200]) + rep.check(f"{p}: stream content-type", "text/event-stream" in ctype, ctype) + parsed = [] + for d in frames: + try: + parsed.append(json.loads(d)) + except ValueError: + pass + rep.check(f"{p}: stream frames have candidates[]", + bool(parsed) and all("candidates" in d for d in parsed), + json.dumps(parsed[0])[:200] if parsed else "no frames") + rep.check(f"{p}: stream frames do NOT leak choices[]", + all("choices" not in d for d in parsed), "leaked OpenAI shape") + said = "".join( + p_.get("text") or "" + for d in parsed + for c in d.get("candidates") or [] + for p_ in ((c.get("content") or {}).get("parts") or []) + ) + truncated = any(c.get("finishReason") == "MAX_TOKENS" + for d in parsed for c in d.get("candidates") or []) + rep.check(f"{p}: stream produced text", bool(said.strip()) or truncated, repr(said[:80])) + + status, _, text = post(base, f"/v1beta/models/{model}:countTokens", body) + j = json.loads(text) if status == 200 else {} + rep.check(f"{p}: countTokens returns totalTokens", + status == 200 and isinstance(j.get("totalTokens"), int) and j["totalTokens"] > 0, + text[:160]) + + +# -------------------------------------------------------------------------- +# Cross-cutting +# -------------------------------------------------------------------------- + +def check_errors(base: str, rep: Report) -> None: + p = "errors" + + # An unknown model must not come back as a 200 SSE stream that never + # produces an event — the failure the client sees has to be the failure + # that happened. + status, ctype, text = post(base, "/v1/messages", { + "model": "definitely-not-a-model", "max_tokens": 10, "stream": True, + "messages": [{"role": "user", "content": "hi"}], + }) + rep.check(f"{p}: unknown model on stream is not 200", status != 200, f"{status} {ctype}") + rep.check(f"{p}: unknown model returns JSON error, not SSE", + "text/event-stream" not in ctype, ctype) + try: + rep.check(f"{p}: error body has error.message", + bool((json.loads(text).get("error") or {}).get("message")), text[:160]) + except ValueError: + rep.check(f"{p}: error body is JSON", False, text[:160]) + + # A model that only lives on /responses must say so rather than 500. + status, _, text = post(base, "/v1/chat/completions", { + "model": "gpt-5.5", "max_tokens": 10, + "messages": [{"role": "user", "content": "hi"}], + }) + rep.check(f"{p}: responses-only model rejected with 4xx on chat", 400 <= status < 500, f"{status} {text[:120]}") + rep.check(f"{p}: rejection names the right endpoint", "/v1/responses" in text, text[:200]) + + +def check_discovery(base: str, rep: Report) -> None: + p = "discovery" + status, text = get(base, "/v1/models") + j = json.loads(text) if status == 200 else {} + rep.check(f"{p}: /v1/models 200", status == 200, text[:160]) + rep.check(f"{p}: models list is OpenAI-shaped", + j.get("object") == "list" and isinstance(j.get("data"), list) and bool(j["data"]), + json.dumps(j)[:160]) + rep.check(f"{p}: every model has id and object=model", + all(m.get("id") and m.get("object") == "model" for m in j.get("data", [])), + "a model entry was malformed") + + status, text = get(base, "/health") + j = json.loads(text) if status == 200 else {} + rep.check(f"{p}: /health 200", status == 200, text[:160]) + rep.check(f"{p}: health reports ready + version", + j.get("ready") is not None and bool(j.get("version")), json.dumps(j)[:160]) + + +def main() -> int: + ap = argparse.ArgumentParser() + ap.add_argument("--port", type=int, default=8399) + ap.add_argument("--host", default="127.0.0.1") + ap.add_argument("--anthropic-model", default="claude-haiku-4.5") + ap.add_argument("--chat-model", default="gpt-4o-mini") + ap.add_argument("--responses-model", default="gpt-5.5") + ap.add_argument("--gemini-model", default="gemini-3.5-flash") + ap.add_argument("--only", default="", help="comma-separated subset of suites to run") + args = ap.parse_args() + + base = f"http://{args.host}:{args.port}" + rep = Report() + + suites = { + "discovery": lambda: check_discovery(base, rep), + "anthropic": lambda: check_anthropic(base, rep, args.anthropic_model), + "chat": lambda: check_chat(base, rep, args.chat_model), + "responses": lambda: check_responses(base, rep, args.responses_model), + "gemini": lambda: check_gemini(base, rep, args.gemini_model), + "errors": lambda: check_errors(base, rep), + } + wanted = [s.strip() for s in args.only.split(",") if s.strip()] or list(suites) + + for name in wanted: + fn = suites.get(name) + if not fn: + print(f"unknown suite: {name}", file=sys.stderr) + return 2 + try: + fn() + except Exception as e: # a crashed suite is a failed suite, not a crashed run + rep.check(f"{name}: suite raised", False, f"{type(e).__name__}: {e}") + + width = max(len(r.name) for r in rep.results) + for r in rep.results: + mark = "PASS" if r.ok else "FAIL" + line = f"[{mark}] {r.name.ljust(width)}" + if not r.ok and r.detail: + line += f" <- {r.detail}" + print(line) + + print(f"\n{len(rep.results) - len(rep.failed)}/{len(rep.results)} passed") + return 1 if rep.failed else 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/stats_inventory.py b/scripts/stats_inventory.py new file mode 100644 index 0000000..12717bd --- /dev/null +++ b/scripts/stats_inventory.py @@ -0,0 +1,123 @@ +"""Inventory the statistics upstream puts in terminal stream events. + +Every surface ends a turn with a frame carrying more than the token counts the +proxy already records — Copilot's own billing units, latency checkpoints, +reasoning-token splits. This walks the captured debug bodies and reports which +of those fields actually appear, per endpoint, so what gets surfaced is chosen +from evidence rather than from guesses about the schema. + +Run against a proxy that has served traffic with body capture on. +""" + +from __future__ import annotations + +import argparse +import json +import urllib.request +from collections import defaultdict + + +def frames(raw: str) -> list[dict]: + """Every JSON frame in a body, whichever transport carried it.""" + out = [] + if raw.lstrip().startswith(("event:", "data:")): + for block in raw.split("\n\n"): + for line in block.splitlines(): + if line.startswith("data:"): + payload = line[5:].strip() + if payload and payload != "[DONE]": + try: + out.append(json.loads(payload)) + except ValueError: + pass + return out + lines = [l for l in raw.splitlines() if l.strip()] + if len(lines) > 1: + for line in lines: + try: + out.append(json.loads(line)) + except ValueError: + pass + return out + try: + out.append(json.loads(raw)) + except ValueError: + pass + return out + + +def leaves(obj, prefix="") -> list[tuple[str, object]]: + """Flattens to dotted paths, collapsing list indices so paths aggregate.""" + found = [] + if isinstance(obj, dict): + for k, v in obj.items(): + found += leaves(v, f"{prefix}.{k}" if prefix else k) + elif isinstance(obj, list): + for v in obj[:1]: + found += leaves(v, f"{prefix}[]") + else: + found.append((prefix, obj)) + return found + + +# Paths the proxy already records. Everything else is a candidate. +KNOWN = { + "usage.input_tokens", "usage.output_tokens", + "usage.cache_read_input_tokens", "usage.cache_creation_input_tokens", + "usage.prompt_tokens", "usage.completion_tokens", "usage.total_tokens", + "response.usage.input_tokens", "response.usage.output_tokens", +} + +# Fragments that mark a value as a statistic worth considering. +INTERESTING = ("usage", "token", "latency", "ms", "duration", "nano", "aiu", + "cost", "batch", "reasoning", "cached", "tier", "fingerprint") + + +def main() -> int: + ap = argparse.ArgumentParser() + ap.add_argument("--port", type=int, default=8399) + ap.add_argument("--host", default="127.0.0.1") + ap.add_argument("--per-page", type=int, default=100) + args = ap.parse_args() + + url = f"http://{args.host}:{args.port}/api/requests?page=1&per_page={args.per_page}" + with urllib.request.urlopen(url, timeout=30) as r: + items = json.load(r).get("items") or [] + + # path -> endpoint -> (count, sample) + seen: dict[str, dict[str, tuple[int, object]]] = defaultdict(lambda: defaultdict(lambda: (0, None))) + bodies = 0 + + for rec in items: + raw = rec.get("response_body") + if not raw: + continue + bodies += 1 + ep = rec.get("endpoint", "?") + for fr in frames(raw): + for path, value in leaves(fr): + low = path.lower() + if not any(f in low for f in INTERESTING): + continue + if path in KNOWN: + continue + n, sample = seen[path][ep] + seen[path][ep] = (n + 1, value if sample is None else sample) + + if not bodies: + print("No captured bodies. Turn on body capture and send some traffic first.") + return 1 + + print(f"{bodies} captured bodies across {len({i.get('endpoint') for i in items})} endpoints\n") + width = max((len(p) for p in seen), default=10) + for path in sorted(seen): + eps = seen[path] + total = sum(n for n, _ in eps.values()) + sample = next((s for _, s in eps.values() if s is not None), None) + where = ",".join(sorted(eps)) + print(f"{path.ljust(width)} x{total:<5} {str(sample)[:40]:<42} {where}") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/ws_check.py b/scripts/ws_check.py new file mode 100644 index 0000000..f66a8b4 --- /dev/null +++ b/scripts/ws_check.py @@ -0,0 +1,221 @@ +"""Conformance sweep for the proxy's `ws:/responses` surface, across every +model that advertises it. + +The catalogue is the source of truth for which models support this transport, +so the model list is discovered rather than hard-coded — a model added or +withdrawn upstream changes what gets tested without editing this file. + +Each model is checked for the properties that make the transport usable: +the event vocabulary matches the SSE Responses API, sequence numbers are +monotonic, the turn terminates, text actually arrives, and usage is reported. +Models that do *not* advertise the transport are checked for a clean refusal — +an unsupported model must be told so, not left waiting on a socket. + +Requires the proxy running with `websockets` installed: + + python scripts/ws_check.py --port 8399 +""" + +from __future__ import annotations + +import argparse +import asyncio +import json +import sys +import urllib.request + +import websockets + +# Events the upstream emits for a plain text turn. Anything here that never +# arrives means the transport is not carrying a full Responses stream. +REQUIRED_EVENTS = ("response.created", "response.output_text.delta", "response.completed") + + +def catalogue(base: str) -> list[dict]: + with urllib.request.urlopen(base + "/v1/models/full/", timeout=30) as r: + return json.load(r).get("data") or [] + + +def ws_models(base: str) -> tuple[list[str], list[str]]: + """(models advertising ws:/responses, models that do not).""" + yes, no = [], [] + for m in catalogue(base): + eps = m.get("supported_endpoints") or [] + (yes if "ws:/responses" in eps else no).append(m["id"]) + return yes, no + + +class Check: + def __init__(self) -> None: + self.rows: list[tuple[str, bool, str]] = [] + + def add(self, name: str, ok: bool, detail: str = "") -> None: + self.rows.append((name, ok, detail)) + + @property + def failed(self) -> list[tuple[str, bool, str]]: + return [r for r in self.rows if not r[1]] + + +async def collect(url: str, payload: dict, limit: int, timeout: float) -> list[dict]: + """Sends one frame and returns every JSON event received until the turn ends.""" + events: list[dict] = [] + async with websockets.connect(url, max_size=None) as ws: + await ws.send(json.dumps(payload)) + while len(events) < limit: + try: + raw = await asyncio.wait_for(ws.recv(), timeout=timeout) + except asyncio.TimeoutError: + break + except websockets.ConnectionClosed: + break + text = raw if isinstance(raw, str) else raw.decode("utf-8", "replace") + try: + ev = json.loads(text) + except ValueError: + events.append({"type": "__non_json__", "raw": text[:200]}) + continue + events.append(ev) + if ev.get("type") in ("response.completed", "response.failed", + "response.incomplete", "error"): + break + return events + + +async def check_model(url: str, model: str, chk: Check, timeout: float) -> None: + p = model + payload = { + "type": "response.create", + "model": model, + "input": "Reply with exactly: ok", + "stream": True, + } + try: + events = await collect(url, payload, limit=400, timeout=timeout) + except Exception as e: # noqa: BLE001 + chk.add(f"{p}: connects", False, f"{type(e).__name__}: {e}") + return + + chk.add(f"{p}: connects", True) + types = [e.get("type") for e in events] + + err = next((e for e in events if e.get("type") == "error"), None) + if err: + chk.add(f"{p}: no error frame", False, json.dumps(err.get("error"))[:160]) + return + chk.add(f"{p}: no error frame", True) + + for want in REQUIRED_EVENTS: + chk.add(f"{p}: emits {want}", want in types, ",".join(t for t in types[:8] if t)) + + chk.add(f"{p}: terminates", bool(types) and types[-1] in + ("response.completed", "response.incomplete"), str(types[-1] if types else None)) + + seqs = [e["sequence_number"] for e in events if isinstance(e.get("sequence_number"), int)] + chk.add(f"{p}: sequence numbers monotonic", + seqs == sorted(seqs) and len(set(seqs)) == len(seqs), + f"{seqs[:12]}") + + said = "".join(e.get("delta") or "" for e in events + if e.get("type") == "response.output_text.delta") + chk.add(f"{p}: produced text", bool(said.strip()), repr(said[:60])) + + final = next((e for e in reversed(events) + if e.get("type") in ("response.completed", "response.incomplete")), None) + usage = ((final or {}).get("response") or {}).get("usage") or {} + chk.add(f"{p}: reports usage", + isinstance(usage.get("input_tokens"), int) and isinstance(usage.get("output_tokens"), int), + json.dumps(usage)[:120]) + # The model that answered should be the one asked for, or the mapping target. + served = ((final or {}).get("response") or {}).get("model") + chk.add(f"{p}: echoes a model", bool(served), str(served)) + + +async def check_refusal(url: str, model: str, chk: Check, timeout: float) -> None: + """A model without the transport must be refused, promptly and in the + upstream's own error shape — not left hanging.""" + p = f"{model} (unsupported)" + payload = {"type": "response.create", "model": model, "input": "hi", "stream": True} + try: + events = await collect(url, payload, limit=10, timeout=timeout) + except Exception as e: # noqa: BLE001 + chk.add(f"{p}: refused cleanly", False, f"{type(e).__name__}: {e}") + return + err = next((e for e in events if e.get("type") == "error"), None) + chk.add(f"{p}: refused with an error frame", err is not None, + ",".join(str(e.get("type")) for e in events[:4])) + if err: + msg = (err.get("error") or {}).get("message", "") + chk.add(f"{p}: refusal names the alternative", "/v1/responses" in msg, msg[:160]) + + +async def check_bad_frames(url: str, chk: Check, timeout: float) -> None: + """Malformed input must produce an error frame rather than a hung socket.""" + cases = [ + ("not JSON", "this is not json"), + ("missing type", json.dumps({"model": "gpt-5.5", "input": "hi"})), + ("wrong type", json.dumps({"type": "nonsense", "model": "gpt-5.5", "input": "hi"})), + ] + for label, frame in cases: + try: + async with websockets.connect(url, max_size=None) as ws: + await ws.send(frame) + raw = await asyncio.wait_for(ws.recv(), timeout=timeout) + ev = json.loads(raw if isinstance(raw, str) else raw.decode()) + chk.add(f"bad frame ({label}) -> error", ev.get("type") == "error", + json.dumps(ev)[:160]) + except asyncio.TimeoutError: + chk.add(f"bad frame ({label}) -> error", False, "socket hung, no reply") + except Exception as e: # noqa: BLE001 + chk.add(f"bad frame ({label}) -> error", False, f"{type(e).__name__}: {e}") + + +async def main_async(args: argparse.Namespace) -> int: + base = f"http://{args.host}:{args.port}" + url = f"ws://{args.host}:{args.port}{args.path}" + + supported, unsupported = ws_models(base) + if args.models: + supported = [m.strip() for m in args.models.split(",") if m.strip()] + print(f"catalogue advertises ws:/responses for {len(supported)} model(s):") + for m in supported: + print(f" {m}") + print() + + chk = Check() + await check_bad_frames(url, chk, args.timeout) + for m in supported: + await check_model(url, m, chk, args.timeout) + await asyncio.sleep(args.delay) + if unsupported and not args.skip_refusal: + await check_refusal(url, unsupported[0], chk, args.timeout) + + width = max(len(r[0]) for r in chk.rows) + for name, ok, detail in chk.rows: + line = f"[{'PASS' if ok else 'FAIL'}] {name.ljust(width)}" + if not ok and detail: + line += f" <- {detail}" + print(line) + print(f"\n{len(chk.rows) - len(chk.failed)}/{len(chk.rows)} passed") + return 1 if chk.failed else 0 + + +def main() -> int: + ap = argparse.ArgumentParser() + ap.add_argument("--host", default="127.0.0.1") + ap.add_argument("--port", type=int, default=8399) + ap.add_argument("--path", default="/v1/responses") + ap.add_argument("--models", default="", help="override the discovered model list") + ap.add_argument("--timeout", type=float, default=90.0) + ap.add_argument("--delay", type=float, default=1.0, + help="pause between models; bursts of upstream traffic get rate limited") + ap.add_argument("--skip-refusal", action="store_true") + args = ap.parse_args() + try: + return asyncio.run(main_async(args)) + except KeyboardInterrupt: + return 130 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/ws_explore.py b/scripts/ws_explore.py new file mode 100644 index 0000000..f70a418 --- /dev/null +++ b/scripts/ws_explore.py @@ -0,0 +1,154 @@ +"""Learn what the Copilot `ws:/responses` surface speaks. + +The catalogue advertises `ws:/responses` for several models and the endpoint +answers `101 Switching Protocols`, but there is no documentation for it. This +opens one connection, sends one Responses-API request, and prints every frame +that comes back, so the framing can be read off the wire instead of guessed. + +Nothing here prints the Copilot token. +""" + +from __future__ import annotations + +import argparse +import asyncio +import json +import os +import pathlib +import urllib.request + +import websockets + +UA = "GithubCopilot/1.155.0" +EDITOR = "vscode/1.104.0" + + +def github_token() -> str: + env = os.environ.get("GH_TOKEN") or os.environ.get("GITHUB_TOKEN") + if env: + return env.strip() + p = pathlib.Path.home() / "AppData/Roaming/ghc-tunnel/github_token.txt" + if not p.exists(): + p = pathlib.Path.home() / ".config/ghc-tunnel/github_token.txt" + return p.read_text().strip() + + +def copilot_token() -> tuple[str, str]: + req = urllib.request.Request( + "https://api.github.com/copilot_internal/v2/token", + headers={ + "Authorization": "token " + github_token(), + "Accept": "application/json", + "Editor-Version": EDITOR, + "User-Agent": UA, + }, + ) + d = json.load(urllib.request.urlopen(req, timeout=30)) + api = (d.get("endpoints") or {}).get("api") or "https://api.githubcopilot.com" + return d["token"], api + + +def summarize(frame: str, full: bool) -> str: + try: + j = json.loads(frame) + except ValueError: + return f"(non-JSON, {len(frame)}B) {frame[:200]!r}" + if full: + return json.dumps(j, indent=2)[:2000] + t = j.get("type") or "?" + bits = [f"type={t}"] + for k in ("delta", "status", "sequence_number", "output_index", "content_index", "error", "code", "message"): + if k in j and not isinstance(j[k], (dict, list)): + v = str(j[k]) + bits.append(f"{k}={v[:80]!r}" if k == "delta" else f"{k}={v[:80]}") + if "response" in j and isinstance(j["response"], dict): + r = j["response"] + bits.append(f"response.status={r.get('status')}") + if r.get("usage"): + bits.append(f"usage={json.dumps(r['usage'])[:120]}") + if not isinstance(j.get("type"), str): + bits.append(f"keys={sorted(j.keys())[:10]}") + return " ".join(bits) + + +async def run(model: str, envelope: str, full: bool, path: str, limit: int, + raw: str | None, query: str, init: str | None) -> None: + token, api = copilot_token() + host = api.split("://", 1)[-1].rstrip("/") + url = f"wss://{host}{path}" + (f"?{query}" if query else "") + + headers = { + "Authorization": f"Bearer {token}", + "User-Agent": UA, + "Editor-Version": EDITOR, + "Editor-Plugin-Version": "copilot-chat/0.26.7", + "Copilot-Integration-Id": "vscode-chat", + "Openai-Intent": "conversation-edits", + } + + body = { + "model": model, + "stream": True, + "input": [{"role": "user", "content": "Reply with exactly: ok"}], + } + payloads = { + "bare": body, + "typed": {"type": "response.create", "response": body}, + "wrapped": {"type": "request", "payload": body}, + "topmodel": {"type": "response.create", "model": model, "response": body}, + } + payload = json.loads(raw) if raw else payloads[envelope] + + print(f"connect {url}") + if init: + print(f"init {init[:200]}") + print(f"send {json.dumps(payload)[:300]}\n") + + async with websockets.connect(url, additional_headers=headers, max_size=None) as ws: + if init: + await ws.send(init) + await ws.send(json.dumps(payload)) + n = 0 + try: + while n < limit: + frame = await asyncio.wait_for(ws.recv(), timeout=60) + n += 1 + text = frame if isinstance(frame, str) else frame.decode("utf-8", "replace") + print(f"[{n:>3}] {summarize(text, full)}") + try: + if json.loads(text).get("type") in ("response.completed", "response.failed", + "response.incomplete", "error"): + break + except ValueError: + pass + except asyncio.TimeoutError: + print("(timed out waiting for the next frame)") + except websockets.ConnectionClosed as e: + print(f"(closed: code={e.code} reason={e.reason!r})") + print(f"\n{n} frames") + + +def main() -> int: + ap = argparse.ArgumentParser() + ap.add_argument("--model", default="gpt-5.5") + ap.add_argument("--envelope", default="bare", + choices=("bare", "typed", "wrapped", "topmodel")) + ap.add_argument("--path", default="/responses") + ap.add_argument("--query", default="", help="URL query string, e.g. model=gpt-5.5") + ap.add_argument("--raw", default=None, help="send this exact JSON instead of a preset") + ap.add_argument("--raw-file", default=None, + help="send the JSON in this file (avoids shell quoting)") + ap.add_argument("--init", default=None, help="send this JSON first, before the payload") + ap.add_argument("--full", action="store_true", help="print whole frames, not a summary") + ap.add_argument("--limit", type=int, default=40) + args = ap.parse_args() + raw = args.raw + if args.raw_file: + raw = pathlib.Path(args.raw_file).read_text(encoding="utf-8") + asyncio.run(run(args.model, args.envelope, args.full, args.path, args.limit, + raw, args.query, args.init)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/ws_probe.py b/scripts/ws_probe.py new file mode 100644 index 0000000..6da5ef4 --- /dev/null +++ b/scripts/ws_probe.py @@ -0,0 +1,127 @@ +"""Probe whether the Copilot API accepts a WebSocket upgrade on /responses. + +The model catalogue advertises `ws:/responses` for several models, but GitHub +publishes no documentation for the inference API at all, and the token +response's `endpoints` object carries no websocket URL. So the only way to +learn whether this surface exists — and what it speaks — is to ask it. + +This performs a raw RFC 6455 handshake and reports what came back. It sends no +payload and holds no connection open; it is a question, not a client. + +Nothing here prints the Copilot token. +""" + +from __future__ import annotations + +import argparse +import base64 +import json +import os +import pathlib +import socket +import ssl +import sys +import urllib.request + +UA = "GithubCopilot/1.155.0" +EDITOR = "vscode/1.104.0" + + +def github_token() -> str: + env = os.environ.get("GH_TOKEN") or os.environ.get("GITHUB_TOKEN") + if env: + return env.strip() + p = pathlib.Path.home() / "AppData/Roaming/ghc-tunnel/github_token.txt" + if not p.exists(): + p = pathlib.Path.home() / ".config/ghc-tunnel/github_token.txt" + return p.read_text().strip() + + +def copilot_token() -> tuple[str, dict]: + req = urllib.request.Request( + "https://api.github.com/copilot_internal/v2/token", + headers={ + "Authorization": "token " + github_token(), + "Accept": "application/json", + "Editor-Version": EDITOR, + "User-Agent": UA, + }, + ) + d = json.load(urllib.request.urlopen(req, timeout=30)) + return d["token"], d.get("endpoints") or {} + + +def handshake(host: str, path: str, token: str, extra: dict[str, str]) -> tuple[str, dict[str, str], bytes]: + """Raw RFC 6455 upgrade. Returns (status line, response headers, trailing body).""" + key = base64.b64encode(os.urandom(16)).decode() + headers = { + "Host": host, + "Upgrade": "websocket", + "Connection": "Upgrade", + "Sec-WebSocket-Key": key, + "Sec-WebSocket-Version": "13", + "Authorization": f"Bearer {token}", + "User-Agent": UA, + "Editor-Version": EDITOR, + "Editor-Plugin-Version": "copilot-chat/0.26.7", + "Copilot-Integration-Id": "vscode-chat", + "Origin": f"https://{host}", + **extra, + } + raw = f"GET {path} HTTP/1.1\r\n" + "".join(f"{k}: {v}\r\n" for k, v in headers.items()) + "\r\n" + + ctx = ssl.create_default_context() + with socket.create_connection((host, 443), timeout=20) as sock: + with ctx.wrap_socket(sock, server_hostname=host) as tls: + tls.sendall(raw.encode()) + buf = b"" + while b"\r\n\r\n" not in buf and len(buf) < 65536: + chunk = tls.recv(4096) + if not chunk: + break + buf += chunk + + head, _, rest = buf.partition(b"\r\n\r\n") + lines = head.decode("utf-8", "replace").split("\r\n") + status = lines[0] if lines else "(no response)" + resp_headers = {} + for line in lines[1:]: + k, _, v = line.partition(":") + if k: + resp_headers[k.strip().lower()] = v.strip() + return status, resp_headers, rest + + +def main() -> int: + ap = argparse.ArgumentParser() + ap.add_argument("--paths", default="/responses,/v1/responses,/ws/responses,/realtime", + help="comma-separated paths to probe") + args = ap.parse_args() + + token, endpoints = copilot_token() + api = endpoints.get("api") or "https://api.githubcopilot.com" + host = api.split("://", 1)[-1].rstrip("/") + print(f"api host: {host}\n") + + interesting = ("upgrade", "connection", "sec-websocket-accept", "sec-websocket-protocol", + "content-type", "x-request-id", "allow") + + for path in [p.strip() for p in args.paths.split(",") if p.strip()]: + try: + status, hdrs, body = handshake(host, path, token, {}) + except Exception as e: # noqa: BLE001 - a probe reports failures, it does not raise + print(f"{path:<20} ERROR {type(e).__name__}: {e}") + continue + shown = {k: v for k, v in hdrs.items() if k in interesting} + print(f"{path:<20} {status}") + for k, v in shown.items(): + print(f"{'':<20} {k}: {v}") + if body[:400].strip(): + print(f"{'':<20} body: {body[:300].decode('utf-8', 'replace')!r}") + print() + + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/src/server.rs b/src/server.rs index bb47773..3ea63c9 100644 --- a/src/server.rs +++ b/src/server.rs @@ -11,7 +11,10 @@ use crate::util; use crate::util::TokenUsage; use axum::{ body::{Body, Bytes}, - extract::{DefaultBodyLimit, Path, Query, Request, State}, + extract::{ + ws::{Message, WebSocket, WebSocketUpgrade}, + DefaultBodyLimit, Path, Query, Request, State, + }, http::{HeaderMap, HeaderValue, StatusCode}, middleware::{self, Next}, response::{IntoResponse, Response}, @@ -33,8 +36,8 @@ pub fn router(state: SharedState) -> Router { .route("/models/{model_id}", get(get_model)) .route("/v1/chat/completions", post(chat_completions)) .route("/chat/completions", post(chat_completions)) - .route("/v1/responses", post(responses)) - .route("/responses", post(responses)) + .route("/v1/responses", post(responses).get(ws_responses)) + .route("/responses", post(responses).get(ws_responses)) .route("/v1/messages", post(messages)) .route("/v1/messages/count_tokens", post(count_tokens)) .route("/v1/embeddings", post(embeddings)) @@ -45,11 +48,14 @@ pub fn router(state: SharedState) -> Router { .route("/", get(dashboard)) .route("/requests", get(requests_page)) .route("/metrics/dashboard", get(metrics_page)) + .route("/app.css", get(stylesheet)) .route("/api/stats", get(api_stats)) + .route("/api/cache", get(api_cache)) .route("/api/requests", get(api_requests)) .route("/api/audit", get(api_audit)) .route("/api/audit/summary", get(api_audit_summary)) .route("/api/config/reload", post(api_reload_config)) + .route("/api/config/debug", post(api_set_debug)) .route("/api/models", get(get_models)) .route("/v1beta/models/{model_action}", post(gemini_generate)) .route("/openapi.json", get(openapi_spec)) @@ -63,8 +69,14 @@ pub fn router(state: SharedState) -> Router { } /// Whether a request path is an LLM API endpoint that should be guarded by the -/// optional API key. The dashboard UI, static assets, and metrics endpoints are -/// intentionally left open so local monitoring keeps working without a key. +/// optional API key. The dashboard UI, static assets, and read-only metrics +/// endpoints are intentionally left open so local monitoring keeps working +/// without a key. +/// +/// `/api/config/` is guarded despite being part of the dashboard: those routes +/// mutate the running process, and one of them turns on body capture, which +/// writes whatever the client sent — credentials included — into the request +/// log. That is not something an unauthenticated caller should be able to do. fn is_protected_path(path: &str) -> bool { const PREFIXES: &[&str] = &[ "/v1/", @@ -73,6 +85,7 @@ fn is_protected_path(path: &str) -> bool { "/embeddings", "/models", "/v1beta/", + "/api/config/", ]; PREFIXES.iter().any(|p| path.starts_with(p)) } @@ -451,6 +464,9 @@ fn record_failure( output_tokens_final: None, cache_read_input_tokens: 0, cache_creation_input_tokens: 0, + reasoning_tokens: None, + billed_nano_aiu: None, + cache_saved_nano_aiu: None, premium_multiplier: None, upstream_idle_max_ms: None, keepalive_probes: None, @@ -489,6 +505,10 @@ fn record_failure( #[derive(Debug, Default)] struct DirectStreamState { usage: TokenUsage, + /// What Copilot billed, attached to the terminal event beside `usage`. + billed_nano_aiu: Option, + /// What the cache was worth, from the same event's per-token rates. + cache_saved_nano_aiu: Option, /// Whether `message_delta` — the only event carrying an authoritative /// `output_tokens` — has arrived. Until it does, the count taken from /// `message_start` is an opening placeholder, so a record finalized early @@ -502,6 +522,10 @@ struct DirectStreamState { impl DirectStreamState { /// Folds one decoded SSE event into the running state. fn observe(&mut self, v: &Value) { + // Rides along on the terminal event rather than inside `usage`. + if let Some(n) = util::copilot_billed_nano_aiu(v) { + self.billed_nano_aiu = Some(n); + } match v.get("type").and_then(|t| t.as_str()) { Some("message_start") => { // The input buckets are only ever stated here, and @@ -563,6 +587,10 @@ struct StreamRecorder { /// is still the `message_start` placeholder — from one that saw the real /// total in `message_delta`. usage_final: bool, + /// What Copilot billed for this turn, once its terminal event states it. + billed_nano_aiu: Option, + /// What the cache was worth this turn, from the model's own rates. + cache_saved_nano_aiu: Option, resp_size: usize, idle: util::IdleTracker, /// Shared with the keepalive layer wrapping this stream, so the record can @@ -602,6 +630,9 @@ impl StreamRecorder { output_tokens_final: None, cache_read_input_tokens: 0, cache_creation_input_tokens: 0, + reasoning_tokens: None, + billed_nano_aiu: None, + cache_saved_nano_aiu: None, premium_multiplier, upstream_idle_max_ms: None, keepalive_probes: None, @@ -626,6 +657,8 @@ impl StreamRecorder { start, usage: TokenUsage::default(), usage_final: false, + billed_nano_aiu: None, + cache_saved_nano_aiu: None, resp_size: 0, idle: util::IdleTracker::new(Instant::now()), probes, @@ -658,6 +691,10 @@ impl StreamRecorder { rec.output_tokens_final = Some(self.usage_final); rec.cache_read_input_tokens = self.usage.cache_read_input_tokens; rec.cache_creation_input_tokens = self.usage.cache_creation_input_tokens; + rec.reasoning_tokens = + (self.usage.reasoning_tokens > 0).then_some(self.usage.reasoning_tokens); + rec.billed_nano_aiu = self.billed_nano_aiu; + rec.cache_saved_nano_aiu = self.cache_saved_nano_aiu; rec.upstream_idle_max_ms = Some(self.idle.max_idle_ms_including_now()); rec.keepalive_probes = Some(self.probes.load(std::sync::atomic::Ordering::Relaxed)); rec.duration = elapsed_secs(self.start); @@ -997,6 +1034,8 @@ async fn chat_completions(State(state): State, body: Bytes) -> Resp if status.is_success() { let parsed: Value = serde_json::from_str(&text).unwrap_or(Value::Null); let usage = util::openai_usage(&parsed.get("usage").cloned().unwrap_or(json!({}))); + let billed = util::copilot_billed_nano_aiu(&parsed); + let cache_saved = util::cache_saving_nano_aiu(&parsed, &usage); let (tool_count, tool_names) = extract_tools_from_request(&req); let cost = calculate_cost(&translated, &usage); let premium_multiplier = state.model_premium_multiplier(&translated).await; @@ -1014,6 +1053,9 @@ async fn chat_completions(State(state): State, body: Bytes) -> Resp output_tokens_final: None, cache_read_input_tokens: usage.cache_read_input_tokens, cache_creation_input_tokens: usage.cache_creation_input_tokens, + reasoning_tokens: (usage.reasoning_tokens > 0).then_some(usage.reasoning_tokens), + billed_nano_aiu: billed, + cache_saved_nano_aiu: cache_saved, premium_multiplier, upstream_idle_max_ms: None, keepalive_probes: None, @@ -1145,6 +1187,8 @@ async fn responses(State(state): State, body: Bytes) -> Response { if status.is_success() { let parsed: Value = serde_json::from_str(&text).unwrap_or(Value::Null); let usage = util::responses_usage(&parsed.get("usage").cloned().unwrap_or(json!({}))); + let billed = util::copilot_billed_nano_aiu(&parsed); + let cache_saved = util::cache_saving_nano_aiu(&parsed, &usage); let (tool_count, tool_names) = extract_tools_from_request(&req); let cost = calculate_cost(&translated, &usage); let premium_multiplier = state.model_premium_multiplier(&translated).await; @@ -1162,6 +1206,9 @@ async fn responses(State(state): State, body: Bytes) -> Response { output_tokens_final: None, cache_read_input_tokens: usage.cache_read_input_tokens, cache_creation_input_tokens: usage.cache_creation_input_tokens, + reasoning_tokens: (usage.reasoning_tokens > 0).then_some(usage.reasoning_tokens), + billed_nano_aiu: billed, + cache_saved_nano_aiu: cache_saved, premium_multiplier, upstream_idle_max_ms: None, keepalive_probes: None, @@ -1187,9 +1234,460 @@ async fn responses(State(state): State, body: Bytes) -> Response { } // --------------------------------------------------------------------------- -// Anthropic messages +// Responses over WebSocket // --------------------------------------------------------------------------- +/// Message type a client must send to start a turn. The upstream rejects +/// anything else with `unsupported message type`. +const WS_RESPONSE_CREATE: &str = "response.create"; + +/// The catalog's name for this surface. +const WS_RESPONSES_ENDPOINT: &str = "ws:/responses"; + +/// Sends a `type: error` frame in the shape the upstream itself uses, so a +/// client needs no special handling for failures the proxy originates. +async fn ws_error(socket: &mut WebSocket, code: &str, message: &str) { + let frame = json!({"type": "error", "error": {"code": code, "message": message}}); + let _ = socket.send(Message::Text(frame.to_string().into())).await; +} + +/// The Responses API over WebSocket. +/// +/// Several models advertise `ws:/responses` in the catalog and nothing else +/// besides `/responses`; this exposes that transport to clients. The protocol +/// is the same `response.*` event vocabulary as the SSE path, one event per +/// text frame — only the transport differs, so a client already written +/// against the streaming Responses API needs no new parsing. +/// +/// Undocumented upstream: the request frame must be flat, with `model` at the +/// top level. Nesting it under `response` is rejected. +async fn ws_responses(ws: WebSocketUpgrade, State(state): State) -> Response { + ws.on_upgrade(move |socket| handle_ws_responses(socket, state)) +} + +async fn handle_ws_responses(mut socket: WebSocket, state: SharedState) { + // The turn does not begin until the client sends something, so there is no + // work to charge for and no record to write if it just connects and leaves. + let first = loop { + match socket.recv().await { + Some(Ok(Message::Text(t))) => break t.to_string(), + Some(Ok(Message::Binary(b))) => break String::from_utf8_lossy(&b).into_owned(), + // Keepalive traffic before the request is normal; axum answers pings. + Some(Ok(_)) => continue, + Some(Err(_)) | None => return, + } + }; + + let start = Instant::now(); + let req_size = first.len(); + + let Ok(req) = serde_json::from_str::(&first) else { + // Recorded like any other pre-flight failure: a request that leaves no + // trace is a request nobody can diagnose. + record_failure( + &state, + "ws:/responses", + "", + None, + 400, + crate::store::failure::PRECONDITION, + req_size, + capture_str(&state, &first), + None, + start, + None, + ); + ws_error(&mut socket, "bad_request", "frame is not JSON").await; + return; + }; + let msg_type = req.get("type").and_then(Value::as_str).unwrap_or(""); + if msg_type != WS_RESPONSE_CREATE { + record_failure( + &state, + "ws:/responses", + req.get("model").and_then(Value::as_str).unwrap_or_default(), + None, + 400, + crate::store::failure::PRECONDITION, + req_size, + capture_json(&state, &req), + None, + start, + None, + ); + ws_error( + &mut socket, + "bad_request", + &format!("unsupported message type: {msg_type}"), + ) + .await; + return; + } + + let original_model = req + .get("model") + .and_then(Value::as_str) + .unwrap_or_default() + .to_string(); + let translated = translate::translate(&state.model_mappings(), &original_model); + + // The transport is only available for models that advertise it, and saying + // so beats an opaque upstream rejection. + if !state + .model_supports_endpoint(&translated, WS_RESPONSES_ENDPOINT) + .await + { + record_failure( + &state, + "ws:/responses", + &original_model, + Some(&translated), + 400, + crate::store::failure::PRECONDITION, + req_size, + capture_json(&state, &req), + None, + start, + None, + ); + ws_error( + &mut socket, + "unsupported_api_for_model", + &format!( + "Model '{original_model}' does not support {WS_RESPONSES_ENDPOINT}. \ + Use POST /v1/responses instead." + ), + ) + .await; + return; + } + + if let Err(e) = state.ensure_copilot_token().await { + record_failure( + &state, + "ws:/responses", + &original_model, + Some(&translated), + 500, + crate::store::failure::PRECONDITION, + req_size, + capture_json(&state, &req), + None, + start, + None, + ); + ws_error(&mut socket, "internal_error", &e).await; + return; + } + + // Forward the model the mapping resolved to, not the alias the client used. + let mut upstream_req = req.clone(); + if let Some(m) = upstream_req.get_mut("model") { + *m = json!(translated); + } + + let (url, headers) = state.ws_responses_upstream().await; + log_debug_request(&state, "ws:/responses", &upstream_req); + + let mut request = + match tokio_tungstenite::tungstenite::client::IntoClientRequest::into_client_request( + url.as_str(), + ) { + Ok(r) => r, + Err(e) => { + ws_error( + &mut socket, + "internal_error", + &format!("bad upstream url: {e}"), + ) + .await; + return; + } + }; + // The handshake carries the same credentials as the HTTP path. + for (k, v) in headers.iter() { + request.headers_mut().insert(k.clone(), v.clone()); + } + + let upstream = match tokio_tungstenite::connect_async(request).await { + Ok((s, _)) => s, + Err(e) => { + record_failure( + &state, + "ws:/responses", + &original_model, + Some(&translated), + 502, + crate::store::failure::CONNECT, + req_size, + capture_json(&state, &req), + None, + start, + None, + ); + ws_error(&mut socket, "connect_error", &format!("upstream: {e}")).await; + return; + } + }; + + pump_ws_responses( + socket, + upstream, + state, + original_model, + translated, + req, + req_size, + start, + ) + .await; +} + +/// Relays frames in both directions until the turn ends, accumulating what the +/// record needs on the way past. Usage only becomes known at +/// `response.completed`, so the totals are folded in as the events go by rather +/// than by re-parsing the transcript afterwards. +#[allow(clippy::too_many_arguments)] +async fn pump_ws_responses( + mut client: WebSocket, + upstream: tokio_tungstenite::WebSocketStream< + tokio_tungstenite::MaybeTlsStream, + >, + state: SharedState, + original_model: String, + translated: String, + req: Value, + req_size: usize, + start: Instant, +) { + use futures_util::{SinkExt, StreamExt}; + use tokio_tungstenite::tungstenite::Message as WsMessage; + + let (mut up_tx, mut up_rx) = upstream.split(); + + // The first frame was consumed to route the request; send the rewritten one. + let mut outbound = req.clone(); + if let Some(m) = outbound.get_mut("model") { + *m = json!(translated); + } + if up_tx + .send(WsMessage::Text(outbound.to_string().into())) + .await + .is_err() + { + ws_error( + &mut client, + "connect_error", + "upstream closed before the request was sent", + ) + .await; + return; + } + + let mut usage = TokenUsage::default(); + // Copilot reports what it billed only on the terminal event. + let mut billed: Option = None; + let mut cache_saved: Option = None; + let mut status = 200u16; + let mut failure: Option<&'static str> = None; + let mut resp_size = 0usize; + let mut transcript = String::new(); + let capture = state.is_debug(); + let mut idle = util::IdleTracker::new(Instant::now()); + + loop { + tokio::select! { + // Upstream → client. This is where the answer comes from, so it is + // also where the turn ends. + frame = up_rx.next() => match frame { + Some(Ok(WsMessage::Text(t))) => { + idle.mark_now(); + resp_size += t.len(); + if capture { + transcript.push_str(&t); + transcript.push('\n'); + } + let done = ws_absorb_event( + &t, + &mut usage, + &mut billed, + &mut cache_saved, + &mut status, + &mut failure, + ); + if client.send(Message::Text(t.as_str().into())).await.is_err() { + failure.get_or_insert(crate::store::failure::CLIENT_DISCONNECTED); + status = 499; + break; + } + if done { + break; + } + } + Some(Ok(WsMessage::Binary(b))) => { + idle.mark_now(); + resp_size += b.len(); + if client.send(Message::Binary(b.to_vec().into())).await.is_err() { + failure.get_or_insert(crate::store::failure::CLIENT_DISCONNECTED); + status = 499; + break; + } + } + Some(Ok(WsMessage::Close(_))) | None => { + // A close before `response.completed` truncated the turn. + if failure.is_none() && status == 200 && usage.output_tokens == 0 { + failure = Some(crate::store::failure::STREAM_INTERRUPTED); + status = 502; + } + break; + } + Some(Ok(_)) => {} + Some(Err(e)) => { + tracing::warn!("[ws:/responses] upstream error: {e}"); + failure = Some(crate::store::failure::STREAM_INTERRUPTED); + status = 502; + let _ = client + .send(Message::Text( + json!({"type": "error", "error": {"code": "stream_interrupted", + "message": e.to_string()}}) + .to_string() + .into(), + )) + .await; + break; + } + }, + + // Client → upstream. Kept open for the whole turn: this transport + // is bidirectional, and a client may cancel or send another + // `response.create` on the same connection. + frame = client.recv() => match frame { + Some(Ok(Message::Text(t))) => { + if up_tx.send(WsMessage::Text(t.as_str().into())).await.is_err() { + break; + } + } + Some(Ok(Message::Binary(b))) => { + if up_tx.send(WsMessage::Binary(b.to_vec().into())).await.is_err() { + break; + } + } + Some(Ok(Message::Close(_))) | None => { + failure.get_or_insert(crate::store::failure::CLIENT_DISCONNECTED); + status = 499; + let _ = up_tx.send(WsMessage::Close(None)).await; + break; + } + Some(Ok(_)) => {} + Some(Err(_)) => { + failure.get_or_insert(crate::store::failure::CLIENT_DISCONNECTED); + status = 499; + break; + } + }, + } + } + + let _ = up_tx.send(WsMessage::Close(None)).await; + + let (tool_count, tool_names) = extract_tools_from_request(&req); + let cost = calculate_cost(&translated, &usage); + let premium_multiplier = state.model_premium_multiplier(&translated).await; + if capture { + log_debug_response(&state, "ws:/responses", &transcript); + } + state.store.add(RequestRecord { + id: uuid::Uuid::new_v4().to_string(), + timestamp: now_iso(), + endpoint: "ws:/responses".into(), + model: original_model.clone(), + translated_model: (translated != original_model).then_some(translated), + status_code: status, + request_size: req_size, + response_size: resp_size, + input_tokens: usage.input_tokens, + output_tokens: usage.output_tokens, + // The upstream sends usage only in the terminal event, so reaching it + // is exactly what makes the count authoritative. + output_tokens_final: Some(failure.is_none()), + cache_read_input_tokens: usage.cache_read_input_tokens, + cache_creation_input_tokens: usage.cache_creation_input_tokens, + reasoning_tokens: (usage.reasoning_tokens > 0).then_some(usage.reasoning_tokens), + billed_nano_aiu: billed, + cache_saved_nano_aiu: cache_saved, + premium_multiplier, + upstream_idle_max_ms: Some(idle.max_idle_ms_including_now()), + keepalive_probes: None, + duration: elapsed_secs(start), + // Both halves use the flag as it stood when the stream opened. A socket + // can stay open for minutes, so re-reading it here would let a toggle + // mid-turn produce a record with the request captured and the response + // missing — half a transcript is more misleading than none. + request_body: capture.then(|| req.to_string()), + response_body: capture.then_some(transcript), + message_count: None, + tool_count: (tool_count > 0).then_some(tool_count), + tool_names: (tool_count > 0).then_some(tool_names), + stop_reason: None, + tools_called: None, + is_agent_initiated: None, + session_id: None, + prompt_cache_hit: cache_disposition(&usage), + failure_kind: failure.map(String::from), + estimated_cost_usd: Some(cost), + }); +} + +/// Folds one upstream event into the running record state. Returns true when +/// the event terminates the turn. +fn ws_absorb_event( + text: &str, + usage: &mut TokenUsage, + billed: &mut Option, + cache_saved: &mut Option, + status: &mut u16, + failure: &mut Option<&'static str>, +) -> bool { + let Ok(v) = serde_json::from_str::(text) else { + return false; + }; + // Attached to the terminal event, beside the response rather than inside it. + if let Some(n) = util::copilot_billed_nano_aiu(&v) { + *billed = Some(n); + } + let done = match v.get("type").and_then(Value::as_str).unwrap_or("") { + "response.completed" | "response.incomplete" => { + if let Some(u) = v.pointer("/response/usage") { + *usage = util::responses_usage(u); + } + true + } + "response.failed" => { + if let Some(u) = v.pointer("/response/usage") { + *usage = util::responses_usage(u); + } + *status = 502; + *failure = Some(crate::store::failure::UPSTREAM_STATUS); + true + } + "error" => { + *status = 502; + *failure = Some(crate::store::failure::UPSTREAM_STATUS); + true + } + _ => false, + }; + // After the arms above, so the token counts it is derived from are the ones + // the terminal event just stated. + if let Some(n) = util::cache_saving_nano_aiu(&v, usage) { + *cache_saved = Some(n); + } + done +} + +// --------------------------------------------------------------------------- +// Anthropic messages// --------------------------------------------------------------------------- + async fn messages( State(state): State, client_headers: HeaderMap, @@ -1413,6 +1911,8 @@ async fn messages_direct( ); let usage = parsed.get("usage").cloned().unwrap_or(json!({})); let usage = util::anthropic_usage(&usage); + let billed = util::copilot_billed_nano_aiu(&parsed); + let cache_saved = util::cache_saving_nano_aiu(&parsed, &usage); let (tool_count, tool_names) = extract_tools_from_request(&req); let tools_called: Vec = parsed .get("content") @@ -1446,6 +1946,9 @@ async fn messages_direct( output_tokens_final: None, cache_read_input_tokens: usage.cache_read_input_tokens, cache_creation_input_tokens: usage.cache_creation_input_tokens, + reasoning_tokens: (usage.reasoning_tokens > 0).then_some(usage.reasoning_tokens), + billed_nano_aiu: billed, + cache_saved_nano_aiu: cache_saved, premium_multiplier: state.model_premium_multiplier(&translated).await, upstream_idle_max_ms: None, keepalive_probes: None, @@ -1588,6 +2091,8 @@ async fn messages_translated( &serde_json::to_string(&parsed).unwrap_or_default(), ); let usage = util::openai_usage(&parsed.get("usage").cloned().unwrap_or(json!({}))); + let billed = util::copilot_billed_nano_aiu(&parsed); + let cache_saved = util::cache_saving_nano_aiu(&parsed, &usage); let (tool_count, tool_names) = extract_tools_from_request(&openai_req); state.store.add(RequestRecord { id: uuid::Uuid::new_v4().to_string(), @@ -1605,6 +2110,9 @@ async fn messages_translated( output_tokens_final: None, cache_read_input_tokens: usage.cache_read_input_tokens, cache_creation_input_tokens: usage.cache_creation_input_tokens, + reasoning_tokens: (usage.reasoning_tokens > 0).then_some(usage.reasoning_tokens), + billed_nano_aiu: billed, + cache_saved_nano_aiu: cache_saved, premium_multiplier: state.model_premium_multiplier(&translated).await, upstream_idle_max_ms: None, keepalive_probes: None, @@ -1848,6 +2356,8 @@ async fn gemini_generate( let parsed: Value = serde_json::from_str(&text).unwrap_or(Value::Null); let gemini_resp = gemini::openai_to_gemini(&parsed); let usage = util::openai_usage(&parsed.get("usage").cloned().unwrap_or(json!({}))); + let billed = util::copilot_billed_nano_aiu(&parsed); + let cache_saved = util::cache_saving_nano_aiu(&parsed, &usage); let cost = calculate_cost(&translated, &usage); let premium_multiplier = state.model_premium_multiplier(&translated).await; state.store.add(RequestRecord { @@ -1864,6 +2374,9 @@ async fn gemini_generate( output_tokens_final: None, cache_read_input_tokens: usage.cache_read_input_tokens, cache_creation_input_tokens: usage.cache_creation_input_tokens, + reasoning_tokens: (usage.reasoning_tokens > 0).then_some(usage.reasoning_tokens), + billed_nano_aiu: billed, + cache_saved_nano_aiu: cache_saved, premium_multiplier, upstream_idle_max_ms: None, keepalive_probes: None, @@ -1969,6 +2482,9 @@ async fn stream_gemini( let mut byte_stream = upstream.bytes_stream(); let mut lines = util::SseLineBuffer::new(); let mut usage = TokenUsage::default(); + // Copilot reports what it billed only on the terminal event. + let mut billed: Option = None; + let mut cache_saved: Option = None; let mut resp_size = 0usize; let mut finish: Option = None; let mut debug_raw: Vec = Vec::new(); @@ -1989,6 +2505,8 @@ async fn stream_gemini( if let Some(u) = v.get("usage") { if !u.is_null() { usage.merge_stream_update(util::openai_usage(u)); + billed = util::copilot_billed_nano_aiu(&v).or(billed); + cache_saved = util::cache_saving_nano_aiu(&v, &usage).or(cache_saved); } } if let Some(choice) = v.get("choices").and_then(|c| c.as_array()).and_then(|a| a.first()) { @@ -2034,7 +2552,8 @@ async fn stream_gemini( output_tokens: usage.output_tokens, output_tokens_final: None, cache_read_input_tokens: usage.cache_read_input_tokens, - cache_creation_input_tokens: usage.cache_creation_input_tokens, + cache_creation_input_tokens: usage.cache_creation_input_tokens, reasoning_tokens: (usage.reasoning_tokens > 0).then_some(usage.reasoning_tokens), billed_nano_aiu: billed, + cache_saved_nano_aiu: cache_saved, premium_multiplier: state.model_premium_multiplier(&translated).await, upstream_idle_max_ms: Some(idle.max_idle_ms_including_now()), keepalive_probes: None, @@ -2100,6 +2619,8 @@ async fn embeddings(State(state): State, body: Bytes) -> Response { if status.is_success() { let parsed: Value = serde_json::from_str(&text).unwrap_or(Value::Null); let usage = util::openai_usage(&parsed.get("usage").cloned().unwrap_or(json!({}))); + let billed = util::copilot_billed_nano_aiu(&parsed); + let cache_saved = util::cache_saving_nano_aiu(&parsed, &usage); state.store.add(RequestRecord { id: uuid::Uuid::new_v4().to_string(), timestamp: now_iso(), @@ -2114,6 +2635,9 @@ async fn embeddings(State(state): State, body: Bytes) -> Response { output_tokens_final: None, cache_read_input_tokens: usage.cache_read_input_tokens, cache_creation_input_tokens: usage.cache_creation_input_tokens, + reasoning_tokens: (usage.reasoning_tokens > 0).then_some(usage.reasoning_tokens), + billed_nano_aiu: billed, + cache_saved_nano_aiu: cache_saved, // Embeddings are not billed as premium requests. premium_multiplier: None, upstream_idle_max_ms: None, @@ -2194,6 +2718,10 @@ async fn health( "models_loaded": model_count, "requests_served": stats.request_count, "auth_required": state.api_key().is_some(), + // Whether request/response bodies are being captured. Surfaced so the + // dashboard can say why a request has no body to show, and so an + // operator can notice capture was left on. + "debug": state.is_debug(), // Reported by the upstream on every response, so this costs no extra // API call. Empty until the first request has been proxied. "quota": state.quota_snapshot(), @@ -2295,6 +2823,9 @@ async fn stream_openai( let mut byte_stream = upstream.bytes_stream(); let mut lines = util::SseLineBuffer::new(); let mut usage = TokenUsage::default(); + // Copilot reports what it billed only on the terminal event. + let mut billed: Option = None; + let mut cache_saved: Option = None; let mut resp_size = 0usize; let mut debug_raw: Vec = Vec::new(); let mut interrupted: Option = None; @@ -2324,6 +2855,8 @@ async fn stream_openai( Ok(v) => { if let Some(u) = v.get("usage") { usage.merge_stream_update(util::openai_usage(u)); + billed = util::copilot_billed_nano_aiu(&v).or(billed); + cache_saved = util::cache_saving_nano_aiu(&v, &usage).or(cache_saved); } resp_size += data.len(); yield Ok(Bytes::from(format!("data: {data}\n\n"))); @@ -2350,6 +2883,8 @@ async fn stream_openai( if let Ok(v) = serde_json::from_str::(data) { if let Some(u) = v.get("usage") { usage.merge_stream_update(util::openai_usage(u)); + billed = util::copilot_billed_nano_aiu(&v).or(billed); + cache_saved = util::cache_saving_nano_aiu(&v, &usage).or(cache_saved); } } resp_size += data.len(); @@ -2387,7 +2922,8 @@ async fn stream_openai( output_tokens: usage.output_tokens, output_tokens_final: None, cache_read_input_tokens: usage.cache_read_input_tokens, - cache_creation_input_tokens: usage.cache_creation_input_tokens, + cache_creation_input_tokens: usage.cache_creation_input_tokens, reasoning_tokens: (usage.reasoning_tokens > 0).then_some(usage.reasoning_tokens), billed_nano_aiu: billed, + cache_saved_nano_aiu: cache_saved, premium_multiplier: state.model_premium_multiplier(&translated).await, upstream_idle_max_ms: Some(idle.max_idle_ms_including_now()), keepalive_probes: None, @@ -2457,6 +2993,9 @@ async fn stream_responses( let mut byte_stream = upstream.bytes_stream(); let mut lines = util::SseLineBuffer::new(); let mut usage = TokenUsage::default(); + // Copilot reports what it billed only on the terminal event. + let mut billed: Option = None; + let mut cache_saved: Option = None; let mut resp_size = 0usize; let mut debug_raw: Vec = Vec::new(); let mut interrupted: Option = None; @@ -2483,6 +3022,8 @@ async fn stream_responses( completed = true; let raw = v.get("response").and_then(|r| r.get("usage")).cloned().unwrap_or(json!({})); usage = util::responses_usage(&raw); + billed = util::copilot_billed_nano_aiu(&v).or(billed); + cache_saved = util::cache_saving_nano_aiu(&v, &usage).or(cache_saved); } } } @@ -2520,7 +3061,8 @@ async fn stream_responses( output_tokens: usage.output_tokens, output_tokens_final: None, cache_read_input_tokens: usage.cache_read_input_tokens, - cache_creation_input_tokens: usage.cache_creation_input_tokens, + cache_creation_input_tokens: usage.cache_creation_input_tokens, reasoning_tokens: (usage.reasoning_tokens > 0).then_some(usage.reasoning_tokens), billed_nano_aiu: billed, + cache_saved_nano_aiu: cache_saved, premium_multiplier: state.model_premium_multiplier(&translated).await, upstream_idle_max_ms: Some(idle.max_idle_ms_including_now()), keepalive_probes: None, @@ -2645,6 +3187,8 @@ async fn stream_anthropic_direct( // this generator: `Drop` can only report what the recorder // itself holds. recorder.usage = st.usage; + recorder.billed_nano_aiu = st.billed_nano_aiu; + recorder.cache_saved_nano_aiu = st.cache_saved_nano_aiu; recorder.usage_final = st.usage_final; } } @@ -2750,6 +3294,9 @@ async fn stream_anthropic_translated( let mut conv = AnthropicStreamState::new(); let mut chunks: Vec = Vec::new(); let mut usage = TokenUsage::default(); + // Copilot reports what it billed only on the terminal event. + let mut billed: Option = None; + let mut cache_saved: Option = None; let mut resp_size = 0usize; let mut debug_raw: Vec = Vec::new(); let mut interrupted: Option = None; @@ -2771,6 +3318,8 @@ async fn stream_anthropic_translated( }; if let Some(u) = v.get("usage") { usage.merge_stream_update(util::openai_usage(u)); + billed = util::copilot_billed_nano_aiu(&v).or(billed); + cache_saved = util::cache_saving_nano_aiu(&v, &usage).or(cache_saved); } chunks.push(v.clone()); for event in conv.process(&v) { @@ -2829,7 +3378,8 @@ async fn stream_anthropic_translated( output_tokens: usage.output_tokens, output_tokens_final: None, cache_read_input_tokens: usage.cache_read_input_tokens, - cache_creation_input_tokens: usage.cache_creation_input_tokens, + cache_creation_input_tokens: usage.cache_creation_input_tokens, reasoning_tokens: (usage.reasoning_tokens > 0).then_some(usage.reasoning_tokens), billed_nano_aiu: billed, + cache_saved_nano_aiu: cache_saved, premium_multiplier: state.model_premium_multiplier(&translated).await, upstream_idle_max_ms: Some(idle.max_idle_ms_including_now()), keepalive_probes: None, @@ -2986,7 +3536,16 @@ where // --------------------------------------------------------------------------- async fn dashboard() -> Response { - serve_asset("dashboard.html", include_str!("../public/dashboard.html")) + serve_asset( + include_str!("../public/dashboard.html"), + "text/html; charset=utf-8", + ) +} + +/// Design system shared by the three dashboard pages. Served separately rather +/// than inlined into each page so the pages cannot drift apart visually. +async fn stylesheet() -> Response { + serve_asset(include_str!("../public/app.css"), "text/css; charset=utf-8") } /// Serves a machine-readable OpenAPI v3 specification describing the proxy's @@ -3099,19 +3658,23 @@ async fn openapi_spec() -> Response { } async fn requests_page() -> Response { - serve_asset("requests.html", include_str!("../public/requests.html")) + serve_asset( + include_str!("../public/requests.html"), + "text/html; charset=utf-8", + ) } async fn metrics_page() -> Response { - serve_asset("metrics.html", include_str!("../public/metrics.html")) + serve_asset( + include_str!("../public/metrics.html"), + "text/html; charset=utf-8", + ) } -fn serve_asset(_name: &str, contents: &'static str) -> Response { +fn serve_asset(contents: &'static str, content_type: &'static str) -> Response { let mut resp = Response::new(Body::from(contents)); - resp.headers_mut().insert( - "Content-Type", - HeaderValue::from_static("text/html; charset=utf-8"), - ); + resp.headers_mut() + .insert("Content-Type", HeaderValue::from_static(content_type)); resp } @@ -3361,10 +3924,150 @@ async fn api_reload_config(State(state): State) -> Response { .into_response() } +/// Turns request/response body capture on or off without a restart. +/// +/// Capturing bodies is the only way to see what a client actually sent, but it +/// used to require stopping the proxy and relaunching it with `--debug` — by +/// which point the request you wanted to inspect is long gone. The flag is read +/// live on every request, so flipping it here applies from the next call. +/// +/// Deliberately not written back to `config.yaml`: capture puts prompts, tool +/// output and any credentials they carry into memory and the log, so it should +/// lapse on restart rather than stay on because someone forgot. +async fn api_set_debug(State(state): State, body: Option>) -> Response { + let requested = body + .as_ref() + .and_then(|Json(v)| v.get("debug")) + .and_then(Value::as_bool); + let Some(debug) = requested else { + return ( + StatusCode::BAD_REQUEST, + Json(json!({ + "error": "expected a JSON body of {\"debug\": true} or {\"debug\": false}" + })), + ) + .into_response(); + }; + state.set_debug(debug); + let action = if debug { "enabled" } else { "disabled" }; + tracing::info!("[debug] body capture {action} from the dashboard"); + Json(json!({ "ok": true, "debug": debug })).into_response() +} + async fn api_stats(State(state): State) -> Response { Json(state.store.stats()).into_response() } +/// Running totals for one model's prompt-cache behaviour. +#[derive(Default)] +struct CacheAgg { + requests: u64, + input: u64, + read: u64, + write: u64, + /// Net effect on the bill, in nano-AI-units, from the model's own rates. + saved_nano_aiu: i64, + /// Whether any response for this model reported rates to compute the above + /// from. Without it a genuine zero — nothing was cached, so nothing was + /// saved — is indistinguishable from an unpriced model. + priced: bool, +} + +/// Prompt-cache statistics. +/// +/// The hit rate is the early warning for a broken prompt prefix: on an agent +/// workload it should sit high and stable, and a sudden drop means the prompt +/// stopped matching and every turn is paying full input price again. Reporting +/// it per model is what makes that actionable — a single global number cannot +/// tell you *which* conversation broke. +/// +/// Totals come from the all-time running counters. The per-model breakdown is +/// derived from the retained ring buffer, so it describes the most recent +/// `sampled_requests` calls rather than every one ever served; the response +/// says so explicitly rather than letting the two silently disagree. +async fn api_cache(State(state): State) -> Response { + let stats = state.store.stats(); + + let (by_model, hit, write_only, uncached, sampled) = state.store.with_records(|records| { + let mut by_model: HashMap = HashMap::new(); + let (mut hit, mut write_only, mut uncached, mut sampled) = (0u64, 0u64, 0u64, 0u64); + + for r in records { + // A rejected attempt has no tokens and no model worth a row; counting + // it would add an empty-named entry and drag every disposition + // toward "uncached". + if r.failed() { + continue; + } + sampled += 1; + match r.prompt_cache_hit { + Some(true) => hit += 1, + Some(false) => write_only += 1, + None => uncached += 1, + } + + // Price against the model that actually served the request, not the + // alias the client asked for. + let model = r.translated_model.as_deref().unwrap_or(&r.model); + let agg = by_model.entry(model.to_string()).or_default(); + agg.requests += 1; + agg.input += r.input_tokens; + agg.read += r.cache_read_input_tokens; + agg.write += r.cache_creation_input_tokens; + // Derived upstream from the rates that response reported, so a + // model Copilot includes at no charge contributes nothing rather + // than an imagined saving. + agg.saved_nano_aiu += r.cache_saved_nano_aiu.unwrap_or(0); + agg.priced |= r.cache_saved_nano_aiu.is_some(); + } + (by_model, hit, write_only, uncached, sampled) + }); + + let mut models: Vec = by_model + .into_iter() + .map(|(model, a)| { + json!({ + "model": model, + "requests": a.requests, + "input_tokens": a.input, + "cache_read_tokens": a.read, + "cache_creation_tokens": a.write, + "fresh_tokens": a.input.saturating_sub(a.read + a.write), + "hit_rate": if a.input > 0 { a.read as f64 / a.input as f64 } else { 0.0 }, + "saved_nano_aiu": a.priced.then_some(a.saved_nano_aiu), + }) + }) + .collect(); + // Biggest prompt first: that is where a broken prefix costs the most. + models.sort_by(|a, b| { + let key = |v: &Value| v.get("input_tokens").and_then(Value::as_u64).unwrap_or(0); + key(b).cmp(&key(a)) + }); + + let total_in = stats.total_input_tokens; + let cached = stats.total_cache_read_tokens + stats.total_cache_creation_tokens; + Json(json!({ + "totals": { + "input_tokens": total_in, + "cache_read_tokens": stats.total_cache_read_tokens, + "cache_creation_tokens": stats.total_cache_creation_tokens, + "fresh_tokens": total_in.saturating_sub(cached), + "hit_rate": if total_in > 0 { + stats.total_cache_read_tokens as f64 / total_in as f64 + } else { 0.0 }, + "request_count": stats.request_count, + }, + "dispositions": { + "served_from_cache": hit, + "wrote_to_cache": write_only, + "no_cache": uncached, + }, + "sampled_requests": sampled, + "by_model": models, + })) + .into_response() +} + /// Largest page size a dashboard API will honour. Keeps a hostile `per_page` /// from forcing an unbounded clone of the request store. const MAX_PAGE_SIZE: usize = 500; @@ -3400,7 +4103,7 @@ async fn api_requests( let session = params.get("session").map(|s| s.as_str()); let (items, total) = if failed_only || session.is_some() { state.store.filtered_page(per_page, offset, |r| { - if failed_only && r.status_code < 400 && r.failure_kind.is_none() { + if failed_only && !r.failed() { return false; } match session { @@ -3659,6 +4362,19 @@ mod tests { assert!(!is_protected_path("/requests")); } + /// Read-only dashboard APIs stay open so local monitoring works without a + /// key, but anything that mutates the running process must not. Turning on + /// body capture writes whatever the client sent — credentials included — + /// into the request log, so it is guarded like an LLM endpoint. + #[test] + fn protected_paths_cover_config_mutations() { + assert!(is_protected_path("/api/config/debug")); + assert!(is_protected_path("/api/config/reload")); + assert!(!is_protected_path("/api/cache")); + assert!(!is_protected_path("/api/requests")); + assert!(!is_protected_path("/api/audit/summary")); + } + #[test] fn presented_key_from_bearer() { let mut h = HeaderMap::new(); @@ -4022,6 +4738,7 @@ mod tests { cache_read_input_tokens: 1000, cache_creation_input_tokens: 1000, output_tokens: 1000, + reasoning_tokens: 0, }; let (base_in, base_out) = model_rates("claude-opus-4.8"); let expected = base_in // uncached remainder @@ -4042,6 +4759,7 @@ mod tests { cache_read_input_tokens: 101_940, cache_creation_input_tokens: 1_731, output_tokens: 561, + reasoning_tokens: 0, }; let cache_aware = calculate_cost("claude-opus-5", &usage); let (base_in, base_out) = model_rates("claude-opus-5"); diff --git a/src/state.rs b/src/state.rs index b451ef7..eacef06 100644 --- a/src/state.rs +++ b/src/state.rs @@ -269,6 +269,15 @@ impl AppState { self.config.read().unwrap().debug } + /// Turns body capture on or off for the running process. Debug is read + /// live on every request, so this takes effect immediately and without a + /// restart — but it is deliberately not written back to `config.yaml`, so + /// a proxy that is restarted returns to its configured default rather than + /// silently keeping capture on forever. + pub fn set_debug(&self, debug: bool) { + self.config.write().unwrap().debug = debug; + } + pub fn max_connection_retries(&self) -> u32 { self.config.read().unwrap().max_connection_retries } @@ -442,6 +451,25 @@ impl AppState { } } + /// Upstream URL and headers for the catalog's `ws:/responses` surface. + /// + /// The WebSocket lives on the same host as the HTTP API and authenticates + /// identically — the token response carries no separate websocket endpoint, + /// and `endpoints.api` is the only host involved. Discovered by probing; + /// GitHub publishes no documentation for the inference API. + pub async fn ws_responses_upstream(&self) -> (String, HeaderMap) { + let base = self.copilot_base_url(); + let ws = match base.split_once("://") { + Some(("https", rest)) => format!("wss://{rest}"), + Some(("http", rest)) => format!("ws://{rest}"), + _ => base.clone(), + }; + ( + format!("{}/responses", ws.trim_end_matches('/')), + self.copilot_headers(false).await, + ) + } + /// Refreshes the Copilot token if it is missing or within 60 seconds of /// expiry. pub async fn ensure_copilot_token(&self) -> Result<(), String> { diff --git a/src/store.rs b/src/store.rs index 6647b04..893239b 100644 --- a/src/store.rs +++ b/src/store.rs @@ -34,6 +34,23 @@ pub struct RequestRecord { pub cache_read_input_tokens: u64, /// Portion of `input_tokens` this request wrote into the cache. pub cache_creation_input_tokens: u64, + /// Output tokens the model spent reasoning without emitting. Billed, but + /// never visible in the answer — which is how a turn can cost a full budget + /// and show nothing. + #[serde(skip_serializing_if = "Option::is_none")] + pub reasoning_tokens: Option, + /// What Copilot itself billed, in nano-AI-units, from the `copilot_usage` + /// object on the response. Authoritative, unlike `estimated_cost_usd` + /// beside it, which is a list-price guess that cannot know a model is + /// included at no charge. + #[serde(skip_serializing_if = "Option::is_none")] + pub billed_nano_aiu: Option, + /// What the prompt cache was worth this turn, in nano-AI-units, computed + /// from the model's own rates. Negative on a turn that populated a cache + /// it did not get to read. `None` when the model reported no rates — not + /// every surface charges for the same token types. + #[serde(skip_serializing_if = "Option::is_none")] + pub cache_saved_nano_aiu: Option, /// Copilot premium-request cost, from the model catalog's /// `billing.multiplier`. `None` when the model is unknown or the endpoint /// carries no premium cost — never guessed as 1.0. @@ -96,6 +113,19 @@ pub struct RequestRecord { pub estimated_cost_usd: Option, } +impl RequestRecord { + /// Whether this attempt never produced a usable answer. + /// + /// One definition, because three places ask: the failures-only filter, the + /// statistics that must exclude these, and the dashboard. A non-2xx status + /// is the obvious case; `failure_kind` catches the ones that carry a + /// success status but did not succeed — a stream cut off mid-answer, or a + /// client that hung up. + pub fn failed(&self) -> bool { + self.status_code >= 400 || self.failure_kind.is_some() + } +} + /// Values for [`RequestRecord::failure_kind`], ordered by how far the request /// got before dying. pub mod failure { @@ -114,13 +144,26 @@ pub mod failure { /// Aggregate statistics returned by `/api/stats`. #[derive(Debug, Default, Clone, Serialize)] pub struct Stats { + /// Requests that produced a usable answer. Failed attempts are counted + /// separately rather than here: they consume nothing, and folding them in + /// would dilute every rate derived from this — a burst of rejected calls + /// would look like a collapse in cache hit rate. pub request_count: u64, + /// Attempts that never produced a usable answer. + pub failed_requests: u64, pub total_input_tokens: u64, pub total_output_tokens: u64, /// Input tokens served from cache, across every recorded request. pub total_cache_read_tokens: u64, /// Input tokens written into the cache, across every recorded request. pub total_cache_creation_tokens: u64, + /// Output tokens spent on reasoning that was never emitted. + pub total_reasoning_tokens: u64, + /// What Copilot billed in total, in nano-AI-units. Since the move to + /// usage-based billing this, not the premium-request count, is the meter. + pub total_nano_aiu: u64, + /// Net effect of the prompt cache on the bill, in nano-AI-units. + pub cache_saved_nano_aiu: i64, /// Copilot premium requests consumed. Fractional because some models /// bill at a discounted multiplier. pub premium_requests: f64, @@ -153,11 +196,21 @@ impl RequestStore { /// Records a completed request and updates aggregate statistics. pub fn add(&self, record: RequestRecord) { let mut inner = self.inner.lock().unwrap(); - inner.stats.request_count += 1; + if record.failed() { + inner.stats.failed_requests += 1; + } else { + inner.stats.request_count += 1; + } + // Token and billing totals count either way. A stream cut off partway + // still consumed what it consumed, and hiding that would understate + // the bill. inner.stats.total_input_tokens += record.input_tokens; inner.stats.total_output_tokens += record.output_tokens; inner.stats.total_cache_read_tokens += record.cache_read_input_tokens; inner.stats.total_cache_creation_tokens += record.cache_creation_input_tokens; + inner.stats.total_reasoning_tokens += record.reasoning_tokens.unwrap_or(0); + inner.stats.total_nano_aiu += record.billed_nano_aiu.unwrap_or(0); + inner.stats.cache_saved_nano_aiu += record.cache_saved_nano_aiu.unwrap_or(0); // Only count what the catalog actually priced; an unknown multiplier // is left out rather than assumed to be a full premium request. if let Some(multiplier) = record.premium_multiplier { @@ -244,6 +297,9 @@ mod tests { output_tokens_final: None, cache_read_input_tokens: 0, cache_creation_input_tokens: 0, + reasoning_tokens: None, + billed_nano_aiu: None, + cache_saved_nano_aiu: None, premium_multiplier: None, upstream_idle_max_ms: None, session_id: None, diff --git a/src/util.rs b/src/util.rs index 5cbdef5..a18e4ca 100644 --- a/src/util.rs +++ b/src/util.rs @@ -354,6 +354,10 @@ pub struct TokenUsage { /// Portion of `input_tokens` written into the cache by this request. pub cache_creation_input_tokens: u64, pub output_tokens: u64, + /// Output tokens spent on reasoning the upstream never sends. Billed like + /// any other output token, which is what makes a turn that produced no + /// visible text still cost something. + pub reasoning_tokens: u64, } impl TokenUsage { @@ -378,6 +382,69 @@ impl TokenUsage { } } +/// Copilot's own billing figure for a request, in nano-AI-units, read from the +/// `copilot_usage` object the upstream attaches to terminal stream events and +/// non-streaming responses. +/// +/// This is what GitHub actually charges — since the move to usage-based billing +/// the meter is token consumption, not premium requests — and it is worth more +/// than the list-price estimate beside it: the estimate has no way to know that +/// a model is included at no cost, and prices `gpt-4o-mini` at a few +/// microdollars where Copilot bills exactly zero. +/// +/// `total_nano_aiu` is the sum of `token_count * cost_per_batch / batch_size` +/// over `token_details`, verified against every surface. The total is used +/// directly rather than recomputed, so a change in how Copilot slices the +/// details cannot silently skew the number. +pub fn copilot_billed_nano_aiu(root: &Value) -> Option { + root.get("copilot_usage")?.get("total_nano_aiu")?.as_u64() +} + +/// What the prompt cache was worth on this turn, in nano-AI-units. +/// +/// Copilot states its own per-token rate for every token type it charges, in +/// `copilot_usage.token_details`, so this is computed from what the model +/// actually costs rather than from a price list and an assumed discount. +/// +/// That distinction matters because cache pricing is per model, and the entries +/// are not the same on every one. Observed on the wire: `claude-haiku-4.5` +/// reports a `cache_write` rate above its input rate, `gemini-3.5-flash` +/// reports one of zero, `gpt-5.5` reports no `cache_write` entry at all, and +/// `gpt-4o-mini` reports zero for everything because Copilot includes it at no +/// charge. A single hard-coded multiplier would claim savings on a model that +/// is free. +/// +/// Positive means the cache paid for itself this turn. Negative is the normal +/// shape of a first turn, which pays a premium to populate a cache that later +/// turns read back. +pub fn cache_saving_nano_aiu(root: &Value, usage: &TokenUsage) -> Option { + let details = root + .get("copilot_usage")? + .get("token_details")? + .as_array()?; + let rate = |kind: &str| -> Option { + details + .iter() + .find(|d| d.get("token_type").and_then(Value::as_str) == Some(kind)) + .and_then(|d| { + let per_batch = d.get("cost_per_batch")?.as_f64()?; + let batch = d.get("batch_size")?.as_f64()?; + (batch > 0.0).then_some(per_batch / batch) + }) + }; + // Without an input rate there is nothing for the cache rates to be cheaper + // than, so there is no saving to state. + let input = rate("input")?; + let mut net = 0.0; + if let Some(read) = rate("cache_read") { + net += usage.cache_read_input_tokens as f64 * (input - read); + } + if let Some(write) = rate("cache_write") { + net -= usage.cache_creation_input_tokens as f64 * (write - input); + } + Some(net.round() as i64) +} + /// Reads a `u64` field, treating absent/malformed values as zero. fn usage_field(usage: &Value, key: &str) -> u64 { usage.get(key).and_then(|t| t.as_u64()).unwrap_or(0) @@ -398,6 +465,7 @@ pub fn anthropic_usage(usage: &Value) -> TokenUsage { cache_read_input_tokens: cache_read, cache_creation_input_tokens: cache_creation, output_tokens: usage_field(usage, "output_tokens"), + reasoning_tokens: 0, } } @@ -408,6 +476,21 @@ pub fn anthropic_usage(usage: &Value) -> TokenUsage { /// would double-count. OpenAI has no cache-write concept, so that bucket stays /// zero. pub fn openai_usage(usage: &Value) -> TokenUsage { + // Where the surface puts `reasoning_tokens` tells you whether it is already + // inside `completion_tokens`. o-series report it under + // `completion_tokens_details`, as a breakdown of a total that includes it. + // The translated Gemini surface reports it at the top level and *excludes* + // it — `prompt + completion + reasoning == total_tokens` there. + // + // `output_tokens` is normalized to the true total either way, for the same + // reason `input_tokens` is: a caller comparing two models should not have + // to know which convention each one follows. + let nested = usage + .get("completion_tokens_details") + .map(|d| usage_field(d, "reasoning_tokens")) + .unwrap_or(0); + let flat = usage_field(usage, "reasoning_tokens"); + let completion = usage_field(usage, "completion_tokens"); TokenUsage { input_tokens: usage_field(usage, "prompt_tokens"), cache_read_input_tokens: usage @@ -415,7 +498,12 @@ pub fn openai_usage(usage: &Value) -> TokenUsage { .map(|d| usage_field(d, "cached_tokens")) .unwrap_or(0), cache_creation_input_tokens: 0, - output_tokens: usage_field(usage, "completion_tokens"), + output_tokens: if nested > 0 { + completion + } else { + completion + flat + }, + reasoning_tokens: if nested > 0 { nested } else { flat }, } } @@ -435,6 +523,10 @@ pub fn responses_usage(usage: &Value) -> TokenUsage { .unwrap_or(0), cache_creation_input_tokens: 0, output_tokens: usage_field(usage, "output_tokens"), + reasoning_tokens: usage + .get("output_tokens_details") + .map(|d| usage_field(d, "reasoning_tokens")) + .unwrap_or(0), } } @@ -690,6 +782,139 @@ mod tests { assert_eq!(u.output_tokens, 300); } + /// The two surfaces disagree about whether reasoning is inside the + /// completion count, so `output_tokens` has to be normalized the way + /// `input_tokens` already is. Observed on the wire: a Responses turn + /// reported `input 11 + output 17 == total 28` with 10 of that output being + /// reasoning, while the translated Gemini surface reported + /// `prompt 5 + completion 1 + reasoning 97 == total 103`. + #[test] + fn reasoning_tokens_normalize_into_the_output_total() { + // Nested: already part of `completion_tokens`. + let nested = openai_usage(&serde_json::json!({ + "prompt_tokens": 11, + "completion_tokens": 17, + "completion_tokens_details": {"reasoning_tokens": 10} + })); + assert_eq!(nested.output_tokens, 17); + assert_eq!(nested.reasoning_tokens, 10); + + // Flat: disjoint from `completion_tokens`, so it has to be added. + let flat = openai_usage(&serde_json::json!({ + "prompt_tokens": 5, + "completion_tokens": 1, + "total_tokens": 103, + "reasoning_tokens": 97 + })); + assert_eq!(flat.output_tokens, 98); + assert_eq!(flat.reasoning_tokens, 97); + assert_eq!(flat.input_tokens + flat.output_tokens, 103); + + // A turn with no reasoning is unaffected. + let none = openai_usage(&serde_json::json!({ + "prompt_tokens": 10, "completion_tokens": 20 + })); + assert_eq!(none.output_tokens, 20); + assert_eq!(none.reasoning_tokens, 0); + } + + /// The Responses API states reasoning as a breakdown of `output_tokens`, + /// never as an addition to it. + #[test] + fn responses_usage_keeps_reasoning_inside_the_output_total() { + let u = responses_usage(&serde_json::json!({ + "input_tokens": 11, + "output_tokens": 17, + "output_tokens_details": {"reasoning_tokens": 10} + })); + assert_eq!(u.output_tokens, 17); + assert_eq!(u.reasoning_tokens, 10); + } + + #[test] + fn copilot_billing_is_read_from_the_response() { + let v = serde_json::json!({ + "copilot_usage": {"total_nano_aiu": 56_500_000, "token_details": []} + }); + assert_eq!(copilot_billed_nano_aiu(&v), Some(56_500_000)); + // A model Copilot includes at no charge reports zero, which is a real + // figure and must not be confused with "not reported". + let free = serde_json::json!({"copilot_usage": {"total_nano_aiu": 0}}); + assert_eq!(copilot_billed_nano_aiu(&free), Some(0)); + assert_eq!(copilot_billed_nano_aiu(&serde_json::json!({})), None); + } + + /// Builds a `copilot_usage.token_details` block from per-token rates, + /// expressed the way Copilot does: a cost per batch plus a batch size. + fn details(rates: &[(&str, f64)]) -> Value { + let entries: Vec = rates + .iter() + .map(|(kind, per_token)| { + serde_json::json!({ + "token_type": kind, + "batch_size": 1_000.0, + "cost_per_batch": per_token * 1_000.0, + }) + }) + .collect(); + serde_json::json!({"copilot_usage": {"token_details": entries}}) + } + + #[test] + fn cache_saving_uses_the_rates_the_model_itself_reported() { + let usage = TokenUsage { + cache_read_input_tokens: 10_000, + cache_creation_input_tokens: 2_000, + ..Default::default() + }; + // Anthropic-style: reads are a tenth of input, writes carry a premium. + let v = details(&[ + ("input", 100_000.0), + ("cache_read", 10_000.0), + ("cache_write", 125_000.0), + ]); + // 10_000 × (100_000 − 10_000) − 2_000 × (125_000 − 100_000) + assert_eq!(cache_saving_nano_aiu(&v, &usage), Some(850_000_000)); + } + + #[test] + fn a_model_that_prices_no_cache_writes_is_not_charged_for_them() { + let usage = TokenUsage { + cache_read_input_tokens: 10_000, + // Reported by the surface, but this model publishes no write rate, + // so there is nothing to subtract for them. + cache_creation_input_tokens: 2_000, + ..Default::default() + }; + let v = details(&[("input", 500_000.0), ("cache_read", 50_000.0)]); + assert_eq!(cache_saving_nano_aiu(&v, &usage), Some(4_500_000_000)); + } + + #[test] + fn a_model_copilot_includes_at_no_charge_saves_nothing() { + let usage = TokenUsage { + cache_read_input_tokens: 10_000, + ..Default::default() + }; + // Every rate is zero, so caching cannot have saved anything. Reporting + // a saving here would invent one from a published price list. + let v = details(&[("input", 0.0), ("cache_read", 0.0)]); + assert_eq!(cache_saving_nano_aiu(&v, &usage), Some(0)); + } + + #[test] + fn no_reported_rates_means_no_figure_rather_than_zero() { + let usage = TokenUsage { + cache_read_input_tokens: 10_000, + ..Default::default() + }; + // No `token_details` at all, and details that omit the input rate the + // others are measured against. Neither can be priced. + assert_eq!(cache_saving_nano_aiu(&serde_json::json!({}), &usage), None); + let no_input = details(&[("cache_read", 50_000.0)]); + assert_eq!(cache_saving_nano_aiu(&no_input, &usage), None); + } + #[test] fn idle_tracker_reports_the_longest_upstream_gap() { let t0 = std::time::Instant::now(); From cef65862a0b738760d98b05791e7409af0b02813 Mon Sep 17 00:00:00 2001 From: "Shangxiang Fan (from Dev Box)" Date: Wed, 29 Jul 2026 19:47:57 +0800 Subject: [PATCH 4/6] perf(anthropic): measure bodies instead of re-serialising them Both `/v1/messages` paths serialised whole JSON trees purely to obtain byte counts, and did it inside their four-attempt retry loops: let req_size = serde_json::to_vec(¤t).map(|v| v.len())... The bytes were already in hand. Every other endpoint in the file -- including the translated `/v1beta/models` path -- already does `let text = resp.text().await; let resp_size = text.len();` and parses from that string. These two were the outliers. The figures were also wrong, in two ways: - `req_size` measured the proxy's version of the request, after system prompt injection, the tool-result suffix and the model rename. Every other endpoint records what the client sent. The dashboard's "Sent" total was adding two different definitions together. - `response_size` measured a re-serialisation of the parsed tree, which differs from the bytes that came off the wire in whitespace, key order and number formatting. Now the client's `body.len()` is taken once in `messages` and threaded in, and the response is read as text once, measured, logged and parsed from that single copy -- removing one full serialisation per retry attempt and two per successful response. The debug-log call was the third copy: it built the string eagerly to pass to `log_debug_response`, which discards it unless debug logging is on. Verified against the wire: a 103-byte request that returns 865 bytes now records exactly 103 and 865. --- src/server.rs | 53 ++++++++++++++++++++++++++++++++------------------- 1 file changed, 33 insertions(+), 20 deletions(-) diff --git a/src/server.rs b/src/server.rs index 3ea63c9..3274359 100644 --- a/src/server.rs +++ b/src/server.rs @@ -1722,10 +1722,23 @@ async fn messages( req = anthropic::apply_tool_result_suffix(&req, &cfg); let client_beta = client_beta_header(&client_headers); + // What the client sent, counted once. The retry loops downstream mutate the + // request, so measuring it there would report the proxy's version of it and + // re-serialise the whole body on every attempt to do so. + let req_size = body.len(); if state.use_direct_anthropic(&translated).await { - messages_direct(state, req, original_model, translated, client_beta, start).await + messages_direct( + state, + req, + original_model, + translated, + client_beta, + req_size, + start, + ) + .await } else { - messages_translated(state, req, original_model, translated, start).await + messages_translated(state, req, original_model, translated, req_size, start).await } } @@ -1771,6 +1784,7 @@ async fn messages_direct( original_model: String, translated: String, client_beta: Option, + req_size: usize, start: Instant, ) -> Response { let vision = anthropic::has_image(&req); @@ -1802,7 +1816,6 @@ async fn messages_direct( for _ in 0..4 { let mut sanitized = anthropic::sanitize_anthropic_request(¤t); sanitized = anthropic::adjust_thinking_budget(&sanitized); - let req_size = serde_json::to_vec(¤t).map(|v| v.len()).unwrap_or(0); let payload = serde_json::to_vec(&sanitized).unwrap_or_default(); log_debug_request(&state, "/v1/messages", &sanitized); @@ -1903,12 +1916,13 @@ async fn messages_direct( }; let status = resp.status(); if status.is_success() { - let parsed: Value = resp.json().await.unwrap_or(Value::Null); - log_debug_response( - &state, - "/v1/messages", - &serde_json::to_string(&parsed).unwrap_or_default(), - ); + // Read once, then parse from that. Asking reqwest for JSON and + // re-serialising the tree costs two passes and reports a byte count + // that is not the one that came off the wire. + let text = resp.text().await.unwrap_or_default(); + let resp_size = text.len(); + log_debug_response(&state, "/v1/messages", &text); + let parsed: Value = serde_json::from_str(&text).unwrap_or(Value::Null); let usage = parsed.get("usage").cloned().unwrap_or(json!({})); let usage = util::anthropic_usage(&usage); let billed = util::copilot_billed_nano_aiu(&parsed); @@ -1940,7 +1954,7 @@ async fn messages_direct( translated_model: (translated != original_model).then_some(translated.clone()), status_code: status.as_u16(), request_size: req_size, - response_size: serde_json::to_vec(&parsed).map(|v| v.len()).unwrap_or(0), + response_size: resp_size, input_tokens: usage.input_tokens, output_tokens: usage.output_tokens, output_tokens_final: None, @@ -2014,6 +2028,7 @@ async fn messages_translated( req: Value, original_model: String, translated: String, + req_size: usize, start: Instant, ) -> Response { let vision = anthropic::has_image(&req); @@ -2040,7 +2055,6 @@ async fn messages_translated( let url = format!("{}/chat/completions", state.copilot_base_url()); let mut headers = state.copilot_headers(vision).await; set_initiator(&mut headers, agent); - let req_size = serde_json::to_vec(¤t).map(|v| v.len()).unwrap_or(0); let payload = serde_json::to_vec(&openai_req).unwrap_or_default(); log_debug_request(&state, "/v1/messages", &openai_req); @@ -2083,13 +2097,14 @@ async fn messages_translated( }; let status = resp.status(); if status.is_success() { - let parsed: Value = resp.json().await.unwrap_or(Value::Null); + // Read once, then parse from that: `resp.json()` followed by a + // re-serialisation costs two extra passes over the body and reports + // a size that is not the one that came off the wire. + let text = resp.text().await.unwrap_or_default(); + let resp_size = text.len(); + log_debug_response(&state, "/v1/messages", &text); + let parsed: Value = serde_json::from_str(&text).unwrap_or(Value::Null); let anthropic_resp = anthropic::openai_to_anthropic(&parsed); - log_debug_response( - &state, - "/v1/messages", - &serde_json::to_string(&parsed).unwrap_or_default(), - ); let usage = util::openai_usage(&parsed.get("usage").cloned().unwrap_or(json!({}))); let billed = util::copilot_billed_nano_aiu(&parsed); let cache_saved = util::cache_saving_nano_aiu(&parsed, &usage); @@ -2102,9 +2117,7 @@ async fn messages_translated( translated_model: (translated != original_model).then_some(translated.clone()), status_code: status.as_u16(), request_size: req_size, - response_size: serde_json::to_vec(&anthropic_resp) - .map(|v| v.len()) - .unwrap_or(0), + response_size: resp_size, input_tokens: usage.input_tokens, output_tokens: usage.output_tokens, output_tokens_final: None, From c72ce8aa56e6a85e24036e5031b96ce92956d0b5 Mon Sep 17 00:00:00 2001 From: "Shangxiang Fan (from Dev Box)" Date: Wed, 29 Jul 2026 20:02:41 +0800 Subject: [PATCH 5/6] docs(changelog): record the billing figure, the size fix, and repair two stale claims Three corrections to the unreleased section. A missing newline had merged the body-capture and read-timeout entries into one bullet. The prompt-cache entry still described savings as a read discount, a write premium and a net figure, which later work in the same section replaced with a single figure from the model's own rates. The dashboard entry still described a hero of premium requests and a recent-requests preview, neither of which shipped. Adds the two changes that had no entry: Copilot's own billed AI units, now recorded per request and shown as the headline, and the /v1/messages size accounting that measured the proxy's rewritten request rather than the client's. --- CHANGELOG.md | 50 ++++++++++++++++++++++++++++++++++++++++---------- 1 file changed, 40 insertions(+), 10 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index b52df8b..ddcefae 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -59,13 +59,23 @@ All notable changes to this project will be documented in this file. tool-result round trips, stream terminators (`message_stop`, `[DONE]`, `response.completed`), token counting, and the error paths — an unknown model must not arrive as a `200` SSE stream that never produces an event. +- **What Copilot actually billed, per request.** Every response carries + `copilot_usage.total_nano_aiu` — the AI units the turn cost — alongside a + `token_details` breakdown giving the per-token rate for each token type + charged. The total was verified against those rates on five responses. It is + now recorded on every request across all six paths (four protocols × + streamed/not, plus WebSocket), totalled in `/api/stats` as `total_nano_aiu`, + and is the dashboard's headline figure, replacing an estimate derived from + published list prices that could not know a model is included at no charge. + Reasoning tokens are recorded beside it. - **Prompt-cache statistics** at `GET /api/cache` and on the dashboard: where input tokens came from (served from cache / written to cache / fresh), the - disposition of recent requests, and a per-model hit rate. A single global - number cannot tell you *which* conversation stopped matching its prefix, which - is the failure that quietly multiplies the bill. Savings are reported in both - directions — the read discount, the write premium, and the net — because - caching is not free. + disposition of recent requests, and a per-model hit rate with its own + distribution bar. A single global number cannot tell you *which* conversation + stopped matching its prefix, which is the failure that quietly multiplies the + bill. What the cache was worth is reported as `saved_nano_aiu`, net of the + write premium, because caching is not free — see below for where the rates + come from. - **Body capture can be toggled at runtime**, from the request browser or via `POST /api/config/debug`. It previously required restarting the proxy with `--debug`, by which point the request you wanted to inspect was gone; the flag @@ -74,7 +84,8 @@ All notable changes to this project will be documented in this file. memory and the log and should lapse on restart. `GET /health` now reports it. The `/api/config/` routes are guarded by the API key when one is configured — read-only dashboard endpoints stay open, but turning on body capture must not - be something an unauthenticated caller can do.- `upstream_read_timeout_seconds` (default `900`, `0` disables, env + be something an unauthenticated caller can do. +- `upstream_read_timeout_seconds` (default `900`, `0` disables, env `GHC_PROXY_UPSTREAM_READ_TIMEOUT`). Bounds the silence *between* reads from an upstream response rather than the total duration, so long streaming answers are unaffected. Without it a half-open connection yields no data, no error and @@ -139,10 +150,11 @@ All notable changes to this project will be documented in this file. eight identically-weighted stat cards followed by the full 78-row model catalogue, so the number that tracks spend had no more prominence than `bytes_received`, and reference data dominated a page you open to check usage. - It now leads with premium requests, input tokens (with the cache-hit share) - and output tokens; quota bars per SKU sit directly beneath, since that is the - same quantity seen from the other end. Traffic and proxy health follow, then a - recent-requests preview, with the model catalogue folded away. + It now leads with what Copilot billed in AI units, input tokens (with the + cache-hit share) and output tokens (with the reasoning share); quota bars per + SKU sit directly beneath, since that is the same quantity seen from the other + end. Traffic, proxy health and the prompt-cache panel follow, with the model + catalogue folded away and the request list on its own tab. - The three pages share one stylesheet at `/app.css` instead of three drifting inline copies, and carry the same persistent nav with a live readiness and version indicator — previously each page offered only a lone @@ -248,6 +260,24 @@ All notable changes to this project will be documented in this file. customized mappings in current ones ### Fixed +- **Recorded request and response sizes on `/v1/messages` measured the wrong + bytes, and cost a full serialisation each to get.** Both paths built a byte + count by serialising the whole JSON tree — + `serde_json::to_vec(¤t).map(|v| v.len())` — inside their four-attempt + retry loops, and the response was serialised twice more: once for a debug log + that discards it unless logging is on, once to measure it. + + The numbers were also not the ones every other endpoint records. + `request_size` measured the proxy's version of the request, after system + prompt injection, the tool-result suffix and the model rename, so the + dashboard's "Sent" total was adding two definitions together. `response_size` + measured a re-serialisation of the parsed tree, which differs from the wire + bytes in whitespace, key order and number formatting. + + Both now follow the pattern the rest of the file already used: the client's + `body.len()` taken once, and the response read as text once, then measured, + logged and parsed from that single copy. A 103-byte request returning 865 + bytes now records exactly 103 and 865. - **Output token totals were not comparable across surfaces.** The two conventions for reasoning tokens disagree about whether they are already counted: a Responses turn reports `input 11 + output 17 == total 28` with 10 From 1fb274463af52ee917c4bc3b8e9ab99a7060791f Mon Sep 17 00:00:00 2001 From: "Shangxiang Fan (from Dev Box)" Date: Wed, 29 Jul 2026 20:22:50 +0800 Subject: [PATCH 6/6] feat(models): default to Opus 5, and document what actually ships The catalog gained `claude-opus-5` and `claude-sonnet-5`. Both are generally available and match `claude-opus-4.8` on every published capability -- 1M context, 64k output, billing multiplier 1, vision -- so `claude-opus-5` becomes the built-in target. The table gained the `opus5` / `5[1m]` aliases and the `claude-opus-5*`, `claude-sonnet-5*` and `claude-sonnet-4.6` prefixes. The v4 migration only ADDS keys. It does not lift mappings that still hold the previous built-in default, because a value equal to an old default is indistinguishable from a version pinned on purpose -- and pinning is common. A real config carried `claude-opus-4-7: claude-opus-4.7` beside `haiku: claude-opus-4.7`; a lifting pass rewrote both. `--setup` or `--default` is how a user asks for new defaults. Docs brought up to what ships: - README: the Responses API over WebSocket, billed AI units and the three-page dashboard, runtime body capture, and the Gemini parameters that do and do not survive translation. Endpoint table gained `GET /v1/responses` (Upgrade), `/api/cache`, `/api/config/debug`, `/api/stats` and `/app.css`, with a note that `/api/config/` is guarded by the API key. - docs/index.md: the same additions to Highlights. - docs/configuration.md: `config_version: 4`, current mapping targets, a schema-version table, and an explicit statement that an upgrade only ever adds. - docs/claude-code.md: the mapping example uses Opus 5, and the tips cover the three dashboard pages, the per-model cache breakdown with its ~4k-token minimum prefix, and how to turn body capture on. - docs/getting-started.md: the startup banner now matches what the proxy prints (it was missing the health-check line). - docs/api.md: `/app.css`, and the model example is no longer a superseded model. `scripts/alias_check.py` resolves every alias against the live upstream, since a mapping is only correct if the name it produces is one Copilot accepts. --- CHANGELOG.md | 16 +++++ README.md | 43 +++++++++++--- docs/api.md | 5 +- docs/claude-code.md | 27 +++++++-- docs/configuration.md | 31 ++++++++-- docs/getting-started.md | 7 +++ docs/index.md | 5 +- scripts/alias_check.py | 60 +++++++++++++++++++ src/config.rs | 125 ++++++++++++++++++++++++++++++++++++++-- 9 files changed, 295 insertions(+), 24 deletions(-) create mode 100644 scripts/alias_check.py diff --git a/CHANGELOG.md b/CHANGELOG.md index ddcefae..ef5b653 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -248,6 +248,22 @@ All notable changes to this project will be documented in this file. `--no-auto-upgrade`, or `GHC_PROXY_AUTO_UPGRADE=0` — worth doing when the binary is managed by a package manager, or lives in a build output directory that `cargo build`/`cargo clean` also writes to. +- **Default model mappings point at Opus 5 and Sonnet 5.** The catalog now + carries `claude-opus-5` and `claude-sonnet-5`; both are generally available and + match `claude-opus-4.8` on every published capability — 1M context, 64k + output, billing multiplier 1, vision. `claude-opus-5` is the new built-in + target, and the table gained the `opus5` / `5[1m]` aliases along with the + `claude-opus-5*`, `claude-sonnet-5*` and `claude-sonnet-4.6` prefixes. + + **Existing config files are only added to, never rewritten.** A mapping + already on disk keeps pointing where it was told to, even when it holds what + used to be the built-in default. A value equal to an old default is + indistinguishable from a version pinned on purpose, and pinning is common — + a config in the wild carried `claude-opus-4-7: claude-opus-4.7` beside + `haiku: claude-opus-4.7`, both of which a "lift stale defaults" pass would + silently retarget. Run `--setup`, or `--default`, to adopt the new defaults. +- `config_version` bumped to `4`, so existing `config.yaml` files gain the new + aliases on the next start. - `config_version` bumped to `3`, so existing `config.yaml` files gain `upstream_read_timeout_seconds` and the new `auto_upgrade` default on the next start. diff --git a/README.md b/README.md index 6655631..720fa97 100644 --- a/README.md +++ b/README.md @@ -42,7 +42,12 @@ file. upstream model supports it, otherwise translated through chat completions). - **Gemini-compatible** `/v1beta/models/{model}:generateContent`, `:streamGenerateContent`, and `:countTokens` endpoints (translated through chat - completions). + completions). `generationConfig` parameters with an OpenAI counterpart are + carried across — `temperature`, `topP`, `maxOutputTokens`, `stopSequences`, + `candidateCount`, `seed`, the penalties, and `responseMimeType`/`responseSchema` + as `response_format`. `topK` and `safetySettings` have no counterpart and are + dropped, which the dashboard states rather than leaving you to assume they + took effect. - **GitHub Models inference** — requests whose model id uses the `publisher/model` form (e.g. `openai/gpt-4o`) are transparently routed to the [GitHub Models](https://models.github.ai) API instead of Copilot, authenticated @@ -59,7 +64,23 @@ file. - **Copilot token management** with automatic refresh. - **Orphaned `tool_use_id` recovery** — retries with offending tool results stripped when the upstream returns the corresponding 400 error. -- **Request analytics dashboard** at `/` and a request browser at `/requests`. +- **The Responses API over WebSocket.** Ten models advertise a `ws:/responses` + transport in the Copilot catalog. `GET /v1/responses` with an `Upgrade: + websocket` header relays to it, recorded like any other request. A model that + does not advertise the transport is refused with an error frame naming the + endpoint that does work. +- **Request analytics dashboard** across three pages: an overview at `/` led by + what Copilot actually billed (in AI units, read from `copilot_usage`, not + estimated from a price list), a request browser at `/requests` where each + exchange renders as a conversation with the raw frames a tab away, and a + metrics UI at `/metrics/dashboard`. Prompt-cache statistics per model at + `GET /api/cache` show where input tokens came from and what the cache was + worth. Failed attempts are counted separately rather than diluting the + statistics. +- **Body capture toggled at runtime** from the dashboard or + `POST /api/config/debug`, so you can start recording request and response + bodies without a restart. Not persisted to `config.yaml`, since capture puts + prompts into memory and the log and should lapse on restart. - **Interactive setup wizard** (`--setup`, or first launch in a terminal): GitHub sign-in, live model catalog, and model-mapping configuration. - **1M-context support** — forwards the `anthropic-beta: context-1m-2025-08-07` @@ -158,7 +179,7 @@ Config file: `~/.ghc-tunnel/config.yaml` (`%APPDATA%/ghc-tunnel/config.yaml` on Windows). It is generated on first run or with `--config`. ```yaml -config_version: 2 +config_version: 4 address: 127.0.0.1 port: 8314 debug: false @@ -169,11 +190,11 @@ copilot_version: "0.48.1" auto_upgrade: true # self-update on startup; false to disable model_mappings: exact: - opus: claude-opus-4.8 - sonnet: claude-opus-4.8 + opus: claude-opus-5 + sonnet: claude-opus-5 haiku: claude-haiku-4.5 prefix: - claude-sonnet-4-: claude-opus-4.8 + claude-sonnet-4-: claude-opus-5 github_models: enabled: true # route publisher/model ids to GitHub Models # org: my-org # attribute inference to an organization @@ -241,6 +262,7 @@ the dashboard and model listings. |----------|-------------| | `POST /v1/chat/completions` | OpenAI chat completions | | `POST /v1/responses` | OpenAI responses API (Codex) | +| `GET /v1/responses` *(with `Upgrade: websocket`)* | Responses API over WebSocket, for models advertising `ws:/responses` | | `GET /v1/models` | List available models | | `GET /v1/models/{model}` | Retrieve a single model (aliases resolved) | | `POST /v1/messages` | Anthropic messages API | @@ -250,13 +272,20 @@ the dashboard and model listings. | `POST /v1beta/models/{model}:countTokens` | Gemini token counting | | `GET /health` | Liveness/readiness probe (`?strict=true` for 503 when not ready) | | `GET /openapi.json` | OpenAPI v3 specification | -| `GET /` | Web dashboard | +| `GET /` | Web dashboard — overview | | `GET /metrics/dashboard` | Metrics dashboard UI | | `GET /metrics` | OpenMetrics endpoint | | `GET /requests` | Request browser | +| `GET /app.css` | Stylesheet shared by the three dashboard pages | | `POST /api/config/reload` | Reload config.yaml without restart | +| `POST /api/config/debug` | Turn body capture on or off (`{"debug": true}`) | +| `GET /api/stats` | Running totals, including what Copilot billed | +| `GET /api/cache` | Prompt-cache statistics, overall and per model | | `GET /api/models` | All supported models (used by the dashboard) | +The `/api/config/` routes are guarded by the API key when one is configured; +the read-only dashboard endpoints stay open. + ## Example Usage ### OpenAI SDK diff --git a/docs/api.md b/docs/api.md index a978f35..201fce1 100644 --- a/docs/api.md +++ b/docs/api.md @@ -32,10 +32,11 @@ require a key on the LLM endpoints; see [Authentication](#authentication) below. | `GET /v1/models/full/` | Raw upstream model catalog with capabilities | | `GET /usage` | Copilot plan and quota usage | | `GET /health` | Liveness/readiness probe | -| `GET /` | Web analytics dashboard | +| `GET /` | Web analytics dashboard — overview | | `GET /metrics/dashboard` | Metrics dashboard UI | | `GET /metrics` | OpenMetrics exposition endpoint | | `GET /requests` | Request browser | +| `GET /app.css` | Stylesheet shared by the three dashboard pages | | `GET /api/stats` | Dashboard statistics (JSON) | | `GET /api/cache` | Prompt-cache statistics, overall and per model | | `GET /api/requests` | Recent requests (JSON) | @@ -210,7 +211,7 @@ the mapped Copilot model. Unknown ids return `404` with an OpenAI-style error body. ```bash -curl http://127.0.0.1:8314/v1/models/claude-opus-4.8 +curl http://127.0.0.1:8314/v1/models/claude-opus-5 ``` ## Token counting diff --git a/docs/claude-code.md b/docs/claude-code.md index 2a074a9..0bcb140 100644 --- a/docs/claude-code.md +++ b/docs/claude-code.md @@ -50,17 +50,23 @@ export ANTHROPIC_API_KEY="ghc-proxy" Claude Code sends specific model names (for example `claude-opus-4-7[1m]`). Use [model mappings](configuration.md#model-mappings) to route those to whichever -Copilot model you want. For example, to always use Claude Opus 4.8 with its +Copilot model you want. For example, to always use Claude Opus 5 with its native 1M context: ```yaml model_mappings: exact: - claude-opus-4-8: claude-opus-4.8 + opus: claude-opus-5 + "5[1m]": claude-opus-5 prefix: - claude-opus-4-7: claude-opus-4.8 + claude-opus-4-7: claude-opus-5 + claude-opus-4-8: claude-opus-5 ``` +The built-in defaults already do this for every Claude spelling. They apply to +a *new* config file only — an existing one keeps the targets it has, so if you +wrote yours before Opus 5 shipped, either edit it or re-run `--setup`. + Restart the proxy after editing `config.yaml` — mappings are read at startup. ## Codex CLI @@ -109,7 +115,18 @@ at `/v1beta/models/{model}:generateContent` (plus streaming and token counting). ## Tips - Use `GET /usage` (or `ghc-proxy check-usage`) to monitor your Copilot quota. -- The dashboard at `http://127.0.0.1:8314/` shows live request statistics and - the full list of supported models. +- The dashboard has three pages: the overview at `http://127.0.0.1:8314/` leads + with what Copilot billed for the session in AI units, alongside quota, traffic + and prompt-cache statistics; `/requests` browses each exchange as a + conversation; `/metrics/dashboard` shows the raw metric list. +- Prompt caching is where an agent session gets cheap or expensive. The cache + panel breaks the hit rate down per model, because a single global number + cannot tell you *which* conversation stopped matching its prefix. Copilot + needs a minimum cacheable prefix — around 4k tokens — before any of it is + eligible, so short prompts legitimately show nothing. +- To inspect what a tool is actually sending, turn on body capture from the + overview or with `curl -X POST http://127.0.0.1:8314/api/config/debug -d + '{"debug": true}'`. It takes effect on the next request and lapses on + restart. - If a tool reports a model is unavailable, check `GET /v1/models` for the exact model id and add a mapping. diff --git a/docs/configuration.md b/docs/configuration.md index d05ee08..f038072 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -21,7 +21,7 @@ Windows). Generated on first run, with `--config`, or through the setup wizard. ```yaml # Schema version for migration/write-back behavior -config_version: 2 +config_version: 4 # Server settings address: 127.0.0.1 @@ -42,11 +42,11 @@ auto_upgrade: true # Model name mappings: exact (full match) and prefix (starts-with) model_mappings: exact: - opus: claude-opus-4.8 - sonnet: claude-sonnet-4.6 + opus: claude-opus-5 + sonnet: claude-sonnet-5 haiku: claude-haiku-4.5 prefix: - claude-sonnet-4-: claude-opus-4.8 + claude-sonnet-4-: claude-opus-5 # GitHub Models (https://models.github.ai) inference # Route publisher/model ids (e.g. openai/gpt-4o) to GitHub Models instead of @@ -85,6 +85,14 @@ Incoming model names are rewritten before the request is forwarded upstream: Exact matches take priority over prefix matches. Unmapped names pass through unchanged. Use the live catalog at `GET /v1/models` to discover valid targets. +The built-in mappings point every Claude spelling at the newest generally +available Opus — currently `claude-opus-5` — and every Haiku spelling at +`claude-haiku-4.5`. Anthropic writes the same version two ways (`4.8` and +`4-8`), so both forms are listed. + +These are defaults for a *new* config file. A file you already have is never +rewritten to follow them: see [Schema upgrades](#schema-upgrades). + ### Account type Controls the upstream base URL only: @@ -125,6 +133,21 @@ this number; on the next start the proxy fills the missing properties with their default values and rewrites `config.yaml` automatically, preserving every value you have set. No flag is needed for this. +**An upgrade only ever adds.** A model mapping already in the file keeps +pointing where you told it to, even when a newer release changes the built-in +default for that alias. There is no way to tell a mapping left at an old +default apart from one deliberately pinned to that version, and pinning is +common — so nothing existing is rewritten. To adopt the new defaults, run +`--setup` or start it from built-in defaults with `--default`. + +Schema versions so far: + +| Version | Introduced | +|---------|------------| +| 2 | Opus 4.8 aliases | +| 3 | `upstream_read_timeout_seconds`; `auto_upgrade` defaulting to true | +| 4 | Opus 5 and Sonnet 5 aliases | + `--update-config` remains for the other, non-schema write-backs (for example restoring the built-in `model_mappings` when the file has none). diff --git a/docs/getting-started.md b/docs/getting-started.md index 0ed26a8..8a40338 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -42,6 +42,7 @@ Once running, the proxy prints the endpoints it serves: ```text Starting GitHub Copilot API Proxy on 127.0.0.1:8314 Dashboard: http://127.0.0.1:8314/ +Health check: http://127.0.0.1:8314/health Metrics UI: http://127.0.0.1:8314/metrics/dashboard OpenMetrics: http://127.0.0.1:8314/metrics Reload config: POST http://127.0.0.1:8314/api/config/reload @@ -52,6 +53,10 @@ Gemini API: http://127.0.0.1:8314/v1beta/models/{model}:generateContent OpenAPI spec: http://127.0.0.1:8314/openapi.json ``` +The dashboard link is the overview; `/requests` browses individual exchanges. +See the [API reference](api.md) for the rest, including the WebSocket transport +and the statistics endpoints. + ## Authentication A GitHub token is resolved in this order: @@ -122,3 +127,5 @@ persisted `machine_id.txt` used for client disguise. - Tune behavior in the [Configuration](configuration.md) reference. - Explore the [API Reference](api.md) and client examples. - Wire up [Claude Code & Codex](claude-code.md). +- Watch what it costs: the overview at `/` leads with the AI units Copilot + billed, and `/api/cache` breaks prompt-cache effectiveness down per model. diff --git a/docs/index.md b/docs/index.md index 99e970c..a035526 100644 --- a/docs/index.md +++ b/docs/index.md @@ -33,6 +33,7 @@ so OpenAI- and Anthropic-native clients work unmodified. ## Highlights - **OpenAI-compatible** `/v1/chat/completions` and `/v1/responses` (Codex) endpoints. +- **The Responses API over WebSocket** — `GET /v1/responses` with `Upgrade: websocket`, for the models whose catalog advertises `ws:/responses`. - **Anthropic-compatible** `/v1/messages` endpoint with native passthrough or translation. - **Gemini-compatible** `/v1beta/models/{model}:generateContent` (+ streaming and token counting). - **Embeddings**, **model listing**, **token counting**, and a **usage** endpoint. @@ -43,7 +44,9 @@ so OpenAI- and Anthropic-native clients work unmodified. - **Optional API-key auth** on the LLM endpoints (Bearer / `x-api-key` / `x-goog-api-key`), off by default. - **One-click client setup** for Claude Code, Codex, and the Gemini CLI. - **OpenAPI spec** at `/openapi.json`. -- **Analytics dashboard** at `/` and a request browser at `/requests`. +- **Three-page dashboard** — an overview at `/` led by what Copilot actually billed, a request browser at `/requests` that renders each exchange as a conversation, and metrics at `/metrics/dashboard`. +- **Spend and cache analytics** — billed AI units read from `copilot_usage` rather than estimated, and per-model prompt-cache statistics at `/api/cache`. +- **Body capture toggled at runtime** via `POST /api/config/debug`, so you can start recording bodies without a restart. ## Quick start diff --git a/scripts/alias_check.py b/scripts/alias_check.py new file mode 100644 index 0000000..5e086e8 --- /dev/null +++ b/scripts/alias_check.py @@ -0,0 +1,60 @@ +"""Checks that every Claude alias resolves to a model the upstream will serve. + +A mapping table is only correct if the name it produces is one Copilot accepts, +so this asks the live upstream rather than comparing strings. + +Note this exercises the *running proxy's* `config.yaml`, not the built-in +defaults. A hand-tuned file may be missing a prefix the defaults carry, in which +case the alias falls through unmapped and Copilot rejects it -- that is a +finding about that file. The defaults themselves are covered by the unit tests +in `src/translate.rs`. +""" + +import json +import sys +import urllib.error +import urllib.request + +BASE = f"http://127.0.0.1:{sys.argv[1] if len(sys.argv) > 1 else '8399'}" + +ALIASES = [ + "opus", + "sonnet", + "haiku", + "opus5", + "5[1m]", + "opus4-8", + "claude-opus-4.6", + "claude-opus-4.8", + "claude-opus-5", + "claude-sonnet-4-5", + "claude-sonnet-4.6", + "claude-sonnet-5", + "claude-sonnet-4-20250101", + "claude-haiku-4.5", +] + +failures = 0 +for alias in ALIASES: + req = urllib.request.Request( + BASE + "/v1/messages", + data=json.dumps( + { + "model": alias, + "max_tokens": 8, + "messages": [{"role": "user", "content": "Reply with the word ok."}], + } + ).encode(), + headers={"content-type": "application/json"}, + ) + try: + with urllib.request.urlopen(req, timeout=120) as r: + served = json.loads(r.read()).get("model", "?") + print(f" {alias:<26} -> {served}") + except urllib.error.HTTPError as e: + failures += 1 + print(f" {alias:<26} -> HTTP {e.code} {e.read()[:120].decode(errors='replace')}") + +print() +print("all aliases served" if failures == 0 else f"{failures} alias(es) failed") +sys.exit(1 if failures else 0) diff --git a/src/config.rs b/src/config.rs index 8ed8c5b..db55cfa 100644 --- a/src/config.rs +++ b/src/config.rs @@ -21,12 +21,16 @@ pub const COPILOT_VERSION: &str = "0.48.1"; /// Config schema version used to detect when defaults/options changed and a /// persisted config should be rewritten with migrated values. /// -/// Bumped to 3 for `upstream_read_timeout_seconds` and the `auto_upgrade` -/// default flip, so existing files gain both on the next start. -pub const CONFIG_VERSION: u32 = 3; +/// Bumped to 4 for the Opus 5 / Sonnet 5 model mappings, so existing files gain +/// the new aliases and have superseded targets lifted on the next start. +pub const CONFIG_VERSION: u32 = 4; /// Default model name that Claude "opus"/"sonnet" requests are mapped to. -pub const DEFAULT_OPUS: &str = "claude-opus-4.8"; +/// +/// The catalog carries `claude-opus-4.6` through `claude-opus-5`; this is the +/// newest generally-available one, and it matches 4.8 on every published +/// capability -- 1M context, 64k output, billing multiplier 1, vision. +pub const DEFAULT_OPUS: &str = "claude-opus-5"; /// Default model name that Claude "haiku" requests are mapped to. pub const DEFAULT_HAIKU: &str = "claude-haiku-4.5"; @@ -307,11 +311,17 @@ pub fn default_model_mappings() -> ModelMappings { let opus = DEFAULT_OPUS.to_string(); let haiku = DEFAULT_HAIKU.to_string(); let mut exact = BTreeMap::new(); - for k in ["opus", "sonnet", "opus4-7", "opus4-8", "4-7[1m]", "4-8[1m]"] { + for k in [ + "opus", "sonnet", "opus4-7", "opus4-8", "opus5", "4-7[1m]", "4-8[1m]", "5[1m]", + ] { exact.insert(k.to_string(), opus.clone()); } exact.insert("haiku".to_string(), haiku.clone()); + // Every spelling of a Claude model resolves to the current best one. The + // list is exhaustive rather than pattern-based because a request naming a + // model that has since been superseded should still be served, and because + // Anthropic writes the same version two ways (`4.8` and `4-8`). let mut prefix = BTreeMap::new(); for k in [ "claude-sonnet-4-", @@ -319,6 +329,7 @@ pub fn default_model_mappings() -> ModelMappings { "claude-opus-4.6-", "claude-opus-4.7-", "claude-opus-4.8-", + "claude-opus-5-", "claude-opus-4-5-", "claude-opus-4-6-", "claude-opus-4-7-", @@ -327,16 +338,21 @@ pub fn default_model_mappings() -> ModelMappings { "claude-opus-4.6", "claude-opus-4.7", "claude-opus-4.8", + "claude-opus-5", "claude-opus-4-6", "claude-opus-4-7", "claude-opus-4-8", "claude-opus-4-6[1m]", "claude-opus-4-7[1m]", "claude-opus-4-8[1m]", + "claude-opus-5[1m]", "claude-sonnet-4-7", "claude-sonnet-4-8", "claude-sonnet-4-6", "claude-sonnet-4-5", + "claude-sonnet-4.6", + "claude-sonnet-5-", + "claude-sonnet-5", ] { prefix.insert(k.to_string(), opus.clone()); } @@ -838,6 +854,39 @@ fn migrate_config(cfg: &mut Config) -> bool { } } + if cfg.config_version < 4 { + // Opus 5 and Sonnet 5 entered the catalog. Add their aliases so the new + // names resolve, and change nothing else. + // + // Deliberately no "lift stale defaults to the new one" pass. A value + // that equals the previous built-in default is indistinguishable from a + // version the user pinned on purpose -- and pinning is common: a real + // config in the wild carried `claude-opus-4-7: claude-opus-4.7` beside + // `haiku: claude-opus-4.7`, both of which such a pass would silently + // rewrite. Existing mappings keep pointing where they were told to; + // `--setup` or `--default` is how a user asks for the new defaults. + let opus = DEFAULT_OPUS.to_string(); + for k in ["opus5", "5[1m]"] { + cfg.model_mappings + .exact + .entry(k.to_string()) + .or_insert_with(|| opus.clone()); + } + for k in [ + "claude-opus-5-", + "claude-opus-5", + "claude-opus-5[1m]", + "claude-sonnet-5-", + "claude-sonnet-5", + "claude-sonnet-4.6", + ] { + cfg.model_mappings + .prefix + .entry(k.to_string()) + .or_insert_with(|| opus.clone()); + } + } + // Any older schema version is lifted to the current one; the caller // persists the re-rendered document so newly introduced properties appear // on disk with their defaults. @@ -1049,4 +1098,70 @@ mod tests { Some("my-model") ); } + + #[test] + fn opus_5_migration_adds_aliases_without_touching_existing_ones() { + let yaml = "config_version: 3\n"; + let mut cfg: Config = serde_norway::from_str(yaml).expect("v3 config parses"); + // A hand-tuned file: version-specific pins, and a tier alias pointed + // somewhere the defaults would never put it. Both shapes were observed + // in a real config, and a migration that "lifts stale defaults" rewrites + // both, because a pin and a stale default look identical. + cfg.model_mappings + .exact + .insert("opus".to_string(), "claude-opus-4.8".to_string()); + cfg.model_mappings + .exact + .insert("haiku".to_string(), "claude-opus-4.7".to_string()); + cfg.model_mappings + .prefix + .insert("claude-opus-4-7".to_string(), "claude-opus-4.7".to_string()); + cfg.model_mappings.prefix.insert( + "claude-sonnet-4.6".to_string(), + "pinned-by-hand".to_string(), + ); + + assert!(migrate_config(&mut cfg)); + assert_eq!(cfg.config_version, CONFIG_VERSION); + + // Nothing already in the file is rewritten, including a key this + // migration would otherwise seed. + for (k, want) in [("opus", "claude-opus-4.8"), ("haiku", "claude-opus-4.7")] { + assert_eq!( + cfg.model_mappings.exact.get(k).map(String::as_str), + Some(want), + "exact alias {k} must not be rewritten" + ); + } + for (k, want) in [ + ("claude-opus-4-7", "claude-opus-4.7"), + ("claude-sonnet-4.6", "pinned-by-hand"), + ] { + assert_eq!( + cfg.model_mappings.prefix.get(k).map(String::as_str), + Some(want), + "prefix {k} must not be rewritten" + ); + } + + // Names that were not in the file are seeded with the new default. + assert_eq!( + cfg.model_mappings.exact.get("opus5").map(String::as_str), + Some(DEFAULT_OPUS) + ); + assert_eq!( + cfg.model_mappings + .prefix + .get("claude-opus-5") + .map(String::as_str), + Some(DEFAULT_OPUS) + ); + assert_eq!( + cfg.model_mappings + .prefix + .get("claude-sonnet-5") + .map(String::as_str), + Some(DEFAULT_OPUS) + ); + } }