diff --git a/.gitignore b/.gitignore
index 1b59c51..9069a41 100644
--- a/.gitignore
+++ b/.gitignore
@@ -1,3 +1,7 @@
/target
**/*.rs.bk
Cargo.lock.orig
+
+# Python bytecode from scripts/
+__pycache__/
+*.py[cod]
diff --git a/CHANGELOG.md b/CHANGELOG.md
index aad37cc..ef5b653 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -4,7 +4,269 @@ All notable changes to this project will be documented in this file.
## [Unreleased]
+### Added
+- **Live quota with no extra API call.** Copilot attaches per-SKU quota to every
+ response (`x-quota-snapshot-chat`, `-completions`, `-premium_interactions`),
+ carrying entitlement, overage, overage-permitted, percent remaining and reset
+ date. Those are now parsed on every proxied response and surfaced on:
+ - `GET /health` under `quota`
+ - `GET /metrics` as `ghc_proxy_quota_percent_remaining`,
+ `ghc_proxy_quota_entitlement` and `ghc_proxy_quota_overage`, labelled by `sku`
+ - `GET /usage` under `live`, alongside the existing authoritative response
+
+ Previously quota was only available from `/usage`, which costs a separate
+ `/copilot_internal/user` request. The headers are undocumented, so a value
+ that stops parsing is ignored rather than reported as zero, and the previous
+ reading is left in place.
+- **Observability for streams and failures.** Records now carry
+ `upstream_idle_max_ms` (longest silence between upstream chunks, which is what
+ assigns blame for a stall), `keepalive_probes`, `failure_kind`, `session_id`
+ parsed from `metadata.user_id`, and `output_tokens_final` so a count that is
+ still the `message_start` placeholder renders as `—` instead of being
+ presented as fact. The dashboard gained the matching columns, a failures-only
+ filter, and premium-request/cache-hit stats.
+- **The Responses API over WebSocket.** Eight models advertise `ws:/responses`
+ in the catalog — six of them alongside `/responses` and nothing else — and the
+ proxy had no way to reach it. `GET /v1/responses` with an upgrade now exposes
+ that transport, relayed to the matching upstream socket.
+
+ The surface is undocumented — GitHub publishes no reference for the inference
+ API at all, and the token response carries no websocket URL — so the protocol
+ was established by probing: it lives on the same host as the HTTP API, speaks
+ the same `response.*` events one per text frame, and takes a *flat* request
+ frame with `model` at the top level. Nesting the body under `response` is
+ rejected, as is omitting `type`. `scripts/ws_probe.py` and
+ `scripts/ws_explore.py` are the tools that established this, kept for when it
+ changes.
+
+ A model that does not advertise the transport is refused with an error frame
+ naming the alternative, rather than left waiting on the socket. Refusals and
+ malformed frames are recorded with a `failure_kind` like every other path, so
+ a failed attempt leaves something to diagnose from. WebSocket turns appear in
+ the dashboard under `ws:/responses`, and their transcript renders through the
+ same reassembly as SSE — only the transport differs.
+- `scripts/ws_check.py` sweeps every model the catalog says supports the
+ transport, discovering the list rather than hard-coding it, and asserts the
+ event vocabulary, monotonic sequence numbers, termination, text and usage.
+- `scripts/replay.py` replays a captured request at the upstream or back through
+ the proxy and times the SSE stream event by event, with no stall watchdog.
+- `scripts/protocol_check.py` asserts protocol conformance against a running
+ proxy. The property that matters for a translating gateway is not "did it
+ return 200" but "did it return the shape the *client* asked for", so every
+ check is an assertion about the client-facing response: a Gemini caller must
+ receive `candidates`, never the `choices` the upstream actually sent. Covers
+ all four surfaces in both streaming and non-streaming modes, tool calls and
+ tool-result round trips, stream terminators (`message_stop`, `[DONE]`,
+ `response.completed`), token counting, and the error paths — an unknown model
+ must not arrive as a `200` SSE stream that never produces an event.
+- **What Copilot actually billed, per request.** Every response carries
+ `copilot_usage.total_nano_aiu` — the AI units the turn cost — alongside a
+ `token_details` breakdown giving the per-token rate for each token type
+ charged. The total was verified against those rates on five responses. It is
+ now recorded on every request across all six paths (four protocols ×
+ streamed/not, plus WebSocket), totalled in `/api/stats` as `total_nano_aiu`,
+ and is the dashboard's headline figure, replacing an estimate derived from
+ published list prices that could not know a model is included at no charge.
+ Reasoning tokens are recorded beside it.
+- **Prompt-cache statistics** at `GET /api/cache` and on the dashboard: where
+ input tokens came from (served from cache / written to cache / fresh), the
+ disposition of recent requests, and a per-model hit rate with its own
+ distribution bar. A single global number cannot tell you *which* conversation
+ stopped matching its prefix, which is the failure that quietly multiplies the
+ bill. What the cache was worth is reported as `saved_nano_aiu`, net of the
+ write premium, because caching is not free — see below for where the rates
+ come from.
+- **Body capture can be toggled at runtime**, from the request browser or via
+ `POST /api/config/debug`. It previously required restarting the proxy with
+ `--debug`, by which point the request you wanted to inspect was gone; the flag
+ is read live, so it applies from the next call. It is not written back to
+ `config.yaml`, since capture puts prompts and any credentials they carry into
+ memory and the log and should lapse on restart. `GET /health` now reports it.
+ The `/api/config/` routes are guarded by the API key when one is configured —
+ read-only dashboard endpoints stay open, but turning on body capture must not
+ be something an unauthenticated caller can do.
+- `upstream_read_timeout_seconds` (default `900`, `0` disables, env
+ `GHC_PROXY_UPSTREAM_READ_TIMEOUT`). Bounds the silence *between* reads from an
+ upstream response rather than the total duration, so long streaming answers
+ are unaffected. Without it a half-open connection yields no data, no error and
+ no end-of-stream: the request hangs forever and the stream-interrupted
+ handling never runs. The default clears the longest silence measured against
+ the real upstream (329.5s, while a tool call's argument JSON was buffered) with
+ room to spare.
+
### Changed
+- **The cache panel is per model, and says why a cell is empty.** The panel
+ opened with one aggregate stacked bar, which cannot say *which* model stopped
+ matching — the question a collapsing hit rate raises. The bar is now a
+ Distribution column, one per model, and what remains at the top is the colour
+ key those bars need. Below the table, a footnote separates the two reasons a
+ row comes back empty: a prompt too short to be eligible at all, and one long
+ enough but never sent before, since nothing can be read back on first sight.
+
+ **`Written` only appears when something was written.** Only the Anthropic
+ surface with an explicit `cache_control` marker ever bills a cache write.
+ Every other surface caches implicitly: the first call reads nothing, the next
+ reads the whole prefix back, and no write is billed. Measured across twelve
+ calls with a 27k-token prefix, exactly one reported a write. The column and
+ its swatch are hidden outright on a workload that never writes, rather than
+ showing a wall of zeros for an event that cannot happen.
+
+ Measured while investigating: `claude-haiku-4.5` cached a 6902-token prefix
+ but not a 4082-token one, so a prompt in the low thousands legitimately
+ produces an empty row. Every surface — chat, messages, responses, Gemini and
+ WebSocket, streamed and not — was confirmed to carry
+ `copilot_usage.token_details`, so an empty row is never the proxy failing to
+ read what upstream reported.
+- **Cache savings come from the model's own rates, not a price list.** The
+ per-model saving was `list price × (1 − 0.1)` for reads and
+ `× (1.25 − 1)` for writes — one hard-coded discount applied to every model.
+ Copilot states its actual per-token rate for every token type it charges, in
+ `copilot_usage.token_details`, and the entries are not the same on every
+ model: `claude-haiku-4.5` prices cache writes above its input rate,
+ `gemini-3.5-flash` prices them at zero, `gpt-5.5` does not price them at all,
+ and `gpt-4o-mini` prices nothing because Copilot includes it. The old maths
+ claimed a saving on models that are free.
+
+ `/api/cache` now reports `saved_nano_aiu` in the same AI units the rest of
+ the dashboard bills in, computed per request from the rates that request
+ reported, and drops `saved_usd` / `write_premium_usd` / `net_saved_usd`.
+
+ A `null` `saved_nano_aiu` now means no response reported rates to compute one
+ from, distinct from `0`, which is a real figure: nothing was cached, so
+ nothing was saved.
+- **Failed attempts no longer count as consumption.** `request_count` counted
+ every record, so a burst of rejected calls inflated the request total, added
+ an empty-named row to the per-model cache table, and dragged every cache
+ disposition toward "uncached" — a rate computed over requests that consumed
+ nothing. Statistics now cover requests that produced an answer, `/api/stats`
+ reports `failed_requests` separately, and the overview links to them. Token
+ and billing totals still count either way: a stream cut off partway consumed
+ what it consumed, and hiding that would understate the bill.
+
+ The predicate — non-2xx, or a `failure_kind` on an otherwise successful
+ status — now has one definition shared by the statistics, the failures-only
+ filter and the dashboard, instead of three.
+- **Dashboard restructured around consumption.** The landing page opened with
+ eight identically-weighted stat cards followed by the full 78-row model
+ catalogue, so the number that tracks spend had no more prominence than
+ `bytes_received`, and reference data dominated a page you open to check usage.
+ It now leads with what Copilot billed in AI units, input tokens (with the
+ cache-hit share) and output tokens (with the reasoning share); quota bars per
+ SKU sit directly beneath, since that is the same quantity seen from the other
+ end. Traffic, proxy health and the prompt-cache panel follow, with the model
+ catalogue folded away and the request list on its own tab.
+ - The three pages share one stylesheet at `/app.css` instead of three drifting
+ inline copies, and carry the same persistent nav with a live readiness and
+ version indicator — previously each page offered only a lone
+ `← Dashboard` link.
+ - **Wide screens are used.** The old 1180px cap left a 2560px monitor half
+ empty while the twelve-column request table still scrolled sideways. Width
+ is now per-page: the request table gets 2000px and the metric list 1600px,
+ since both are data a wide screen genuinely helps you read, while the
+ overview keeps a 1280px measure because a handful of large numbers only
+ drift apart when stretched. Message text keeps a 100ch measure so prose
+ stays readable at full width; tool payloads are code and stay unconstrained.
+ - `/health` data (per-SKU quota, Copilot token expiry, uptime, readiness,
+ models loaded) was reachable but shown nowhere; it is now on the overview.
+ The request list stays on its own tab — the overview only reports how many
+ requests failed and links across.
+ - **Debug bodies render as a conversation.** `--debug` stores request and
+ response bodies verbatim, which is the right thing to store and the wrong
+ thing to read: finding what the model actually said meant scrolling past
+ tool schemas, content-filter blocks and, for streams, every SSE frame. The
+ detail view now reconstructs the exchange — system/user/assistant turns,
+ tool calls and results as collapsible cards, token and stop-reason chips —
+ and reassembles streamed fragments into the message they describe. All
+ three wire formats are handled, and none of them agree: Anthropic numbers
+ content blocks and tags every delta, chat completions hides tool arguments
+ inside `choices[].delta.tool_calls[]`, and the Responses API uses a flat
+ `response.*` vocabulary where the delta is a bare string. Reasoning is
+ picked up under every name the families use (`reasoning_text`,
+ `reasoning_content`, `response.reasoning_summary_text.delta`) rather than
+ dropped, and a turn that produced no visible output says whether it ran out
+ of token budget instead of reporting an empty stream. The raw bodies remain
+ one click away on a `Raw` tab.
+ - A third tab shows what the wire actually carried, which is the question when
+ a stream stalls, repeats, or ends somewhere unexpected. For a stream it
+ lists **every SSE frame** — sequence number, event name and a one-line gist,
+ each expandable to the full payload — so the `[DONE]` sentinel, a missing
+ `message_stop` or a duplicated index is visible instead of buried in a
+ scroll of raw text. For a non-streaming completion it lists **every
+ top-level field**, including the ones the conversation view has no place for
+ (`content_filter_results`, `prompt_filter_results`, `system_fingerprint`,
+ `service_tier`, `copilot_usage`).
+ - The request table's full nanosecond ISO timestamp consumed a third of the
+ row width; it renders as local wall-clock time with the exact value in the
+ tooltip, and the table scrolls horizontally rather than letting the panel
+ clip columns off the right edge, with the expanded detail pinned to the left
+ so it stays readable while the table scrolls. Empty results say so instead
+ of showing a header row with nothing under it.
+ - **Each request now says what happened, not just what was said.** The
+ response section opened with a row of bare chips: the numbers were there,
+ but not their meaning. It now leads with the outcome, normalized across
+ surfaces and explained — `length`, `max_tokens`, `max_output_tokens` and
+ `MAX_TOKENS` are the same event on four different APIs, and all four mean
+ the answer on screen is not the whole answer. Endings are colour-coded by
+ whether they are normal (`end_turn`, `stop`, `completed`), truncated, or a
+ refusal or filter block.
+
+ Beneath it sit the facts each surface actually reports, omitted when
+ absent: reasoning effort, service tier, prompt-cache retention, response
+ verbosity and output item types on the Responses API; the backend build
+ fingerprint on chat completions, which is worth noticing because a change
+ there can shift results for identical input; cache-write TTL split and
+ inference region on Anthropic; rejected speculative tokens, which are
+ billed. Content-filter verdicts appear only when something was actually
+ flagged — an all-clear on four categories on every request is noise.
+ - The `Probes` column is gone. Keepalive only runs on the Anthropic streaming
+ path, so four of the five request paths rendered a dash that read as "zero
+ probes sent" when it meant "not measured here". The count now rides in the
+ `Idle` tooltip, which is where it was useful anyway — the pair is what
+ assigns blame for a stall.
+ - **Protocol-specific parameters moved out of the list and into the detail.**
+ The three surfaces agree on very little, so a table column for any of them
+ is structurally empty for the other two: `New` (cache writes) only exists on
+ Anthropic, and `Session` only when the client encodes one into
+ `metadata.user_id` the way Claude Code does. Both columns are gone; the
+ detail now carries a panel naming the surface and listing what that request
+ actually set — `top_k`, `stop_sequences`, `cache_control` marks and
+ `thinking` for Anthropic; `seed`, penalties, `response_format` and
+ `logprobs` for chat completions; `reasoning.effort`, `max_output_tokens`,
+ `store` and `truncation` for the Responses API; `generationConfig` and
+ `safetySettings` for Gemini. Absent parameters are omitted rather than shown
+ as blanks. A Gemini request is labelled `Gemini → OpenAI Chat Completions
+ (translated)`, because the proxy rewrites it before forwarding and the
+ recorded body is the rewritten one — `topK` and `safetySettings` have no
+ counterpart and are dropped, which the panel says instead of leaving the
+ reader to assume they took effect. Gemini's `contents`/`parts`,
+ `functionCall` and `functionResponse` shapes render like every other
+ surface's.
+- **`auto_upgrade` now defaults to `true`**, including for config files written
+ before the setting existed. The proxy checks GitHub releases on startup and
+ replaces its own binary when a newer version is published; the replacement
+ takes effect on the next start. Disable with `auto_upgrade: false`,
+ `--no-auto-upgrade`, or `GHC_PROXY_AUTO_UPGRADE=0` — worth doing when the
+ binary is managed by a package manager, or lives in a build output directory
+ that `cargo build`/`cargo clean` also writes to.
+- **Default model mappings point at Opus 5 and Sonnet 5.** The catalog now
+ carries `claude-opus-5` and `claude-sonnet-5`; both are generally available and
+ match `claude-opus-4.8` on every published capability — 1M context, 64k
+ output, billing multiplier 1, vision. `claude-opus-5` is the new built-in
+ target, and the table gained the `opus5` / `5[1m]` aliases along with the
+ `claude-opus-5*`, `claude-sonnet-5*` and `claude-sonnet-4.6` prefixes.
+
+ **Existing config files are only added to, never rewritten.** A mapping
+ already on disk keeps pointing where it was told to, even when it holds what
+ used to be the built-in default. A value equal to an old default is
+ indistinguishable from a version pinned on purpose, and pinning is common —
+ a config in the wild carried `claude-opus-4-7: claude-opus-4.7` beside
+ `haiku: claude-opus-4.7`, both of which a "lift stale defaults" pass would
+ silently retarget. Run `--setup`, or `--default`, to adopt the new defaults.
+- `config_version` bumped to `4`, so existing `config.yaml` files gain the new
+ aliases on the next start.
+- `config_version` bumped to `3`, so existing `config.yaml` files gain
+ `upstream_read_timeout_seconds` and the new `auto_upgrade` default on the next
+ start.
- **Config schema upgrades apply automatically.** When a release introduces new
`config.yaml` properties (signalled by a `config_version` bump), the missing
keys are now filled with their defaults and written back to `config.yaml` on
@@ -13,6 +275,83 @@ All notable changes to this project will be documented in this file.
Opus 4.8 alias backfill is now scoped to pre-v2 files so it cannot overwrite
customized mappings in current ones
+### Fixed
+- **Recorded request and response sizes on `/v1/messages` measured the wrong
+ bytes, and cost a full serialisation each to get.** Both paths built a byte
+ count by serialising the whole JSON tree —
+ `serde_json::to_vec(¤t).map(|v| v.len())` — inside their four-attempt
+ retry loops, and the response was serialised twice more: once for a debug log
+ that discards it unless logging is on, once to measure it.
+
+ The numbers were also not the ones every other endpoint records.
+ `request_size` measured the proxy's version of the request, after system
+ prompt injection, the tool-result suffix and the model rename, so the
+ dashboard's "Sent" total was adding two definitions together. `response_size`
+ measured a re-serialisation of the parsed tree, which differs from the wire
+ bytes in whitespace, key order and number formatting.
+
+ Both now follow the pattern the rest of the file already used: the client's
+ `body.len()` taken once, and the response read as text once, then measured,
+ logged and parsed from that single copy. A 103-byte request returning 865
+ bytes now records exactly 103 and 865.
+- **Output token totals were not comparable across surfaces.** The two
+ conventions for reasoning tokens disagree about whether they are already
+ counted: a Responses turn reports `input 11 + output 17 == total 28` with 10
+ of that output being reasoning, while the translated Gemini surface reports
+ `prompt 5 + completion 1 + reasoning 97 == total 103`. Taken at face value the
+ dashboard showed a reasoning share of 262%. `output_tokens` is now normalized
+ to the true total on both, the way `input_tokens` already was.
+- **Gemini requests lost parameters that had exact counterparts.** The
+ translation to chat completions carried `temperature`, `topP`,
+ `maxOutputTokens` and `stopSequences` and dropped the rest of
+ `generationConfig` in silence, so a client's `seed`, `candidateCount`,
+ `presencePenalty`, `frequencyPenalty` or `responseMimeType` had no effect and
+ nothing said why. They now map to `seed`, `n`, `presence_penalty`,
+ `frequency_penalty` and `response_format` (with `responseSchema` carried over
+ as a `json_schema` format). `topK` and `safetySettings` genuinely have no
+ counterpart in the chat completions schema and are still dropped — the
+ dashboard names them rather than describing the loss in general terms, and a
+ test asserts they are not invented.
+- **Non-2xx upstreams on `/v1/messages` reached the client as an empty `200`
+ stream.** `messages_direct` only intercepted `400`; a `401`, `403`, `429` or
+ `5xx` fell through and was wrapped in a `200 text/event-stream` whose body was
+ a JSON error object, so the client waited on a stream that never produced an
+ event and reported a stall instead of the auth or rate-limit failure that
+ actually happened. This was the one path Claude Code takes. All five streaming
+ paths now gate on a single `is_streamable_status()` predicate
+- **A client that disconnected mid-stream left no record.** axum drops the
+ response body on disconnect, which drops the generator, so the `store.add`
+ after the loop never ran. Recording now happens from `Drop`
+- Pre-flight failures (token refresh, rate gate, connect errors) returned early
+ without recording; they are captured with a `failure_kind`
+- **Keepalive probes could be silenced exactly during a stall.** The boundary
+ flag was set from the last chunk, so a TCP split mid-event left it stuck until
+ a new chunk arrived — and an upstream that went quiet right then produced no
+ probes at all. Partial events are now held back so the downstream always sits
+ on an event boundary, and the Anthropic path sends `event: ping` rather than
+ an SSE comment, since comments are discarded by the parser and never reset a
+ client's idle watchdog
+- **Token counts were read from one bucket.** Anthropic's `input_tokens`,
+ `cache_read_input_tokens` and `cache_creation_input_tokens` are disjoint, so a
+ fully-cached Claude Code turn reported single digits for a 348,483-token
+ prompt. The three protocols slice the total differently and now have separate
+ extractors; cost reprices the cached buckets instead of charging every input
+ token at the full rate
+- **Reassembling one large SSE event was quadratic.** The line buffer rescanned
+ the whole retained buffer on every chunk, so a single event that arrived in
+ many pieces was re-read from the start each time. A 4 MB event delivered in
+ 4 KB chunks took **7.9 seconds** of pure CPU; the search now resumes where the
+ previous one stopped, bringing it to milliseconds. Correctness was never
+ affected — only the cost of getting there.
+- The line buffer could grow without bound if an upstream never emitted a
+ newline. A single line is now capped at 64 MB.
+- Stopped scraping the Arch User Repository for the latest VS Code version. The
+ AUR maintainers asked projects to stop; `dynamic_vscode_version` now uses
+ Microsoft's own `update.code.visualstudio.com` release API and ignores
+ non-`major.minor.patch` builds.
+- Removed `scripts/__pycache__` from version control and added the matching
+ `.gitignore` entries.
+
## [1.3.0] - 2026-07-27
### Added
diff --git a/Cargo.lock b/Cargo.lock
index 5b59bdb..9775012 100644
--- a/Cargo.lock
+++ b/Cargo.lock
@@ -104,6 +104,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "31b698c5f9a010f6573133b09e0de5408834d0c82f8d7475a89fc1867a71cd90"
dependencies = [
"axum-core",
+ "base64",
"bytes",
"form_urlencoded",
"futures-util",
@@ -122,8 +123,10 @@ dependencies = [
"serde_json",
"serde_path_to_error",
"serde_urlencoded",
+ "sha1",
"sync_wrapper",
"tokio",
+ "tokio-tungstenite 0.29.0",
"tower",
"tower-layer",
"tower-service",
@@ -394,6 +397,12 @@ dependencies = [
"syn 2.0.118",
]
+[[package]]
+name = "data-encoding"
+version = "2.11.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "a4ae5f15dda3c708c0ade84bfee31ccab44a3da4f88015ed22f63732abe300c8"
+
[[package]]
name = "der"
version = "0.7.10"
@@ -759,6 +768,7 @@ dependencies = [
"serde_norway",
"tiktoken-rs",
"tokio",
+ "tokio-tungstenite 0.28.0",
"toml",
"tower",
"tower-http 0.7.0",
@@ -1905,6 +1915,17 @@ dependencies = [
"serde",
]
+[[package]]
+name = "sha1"
+version = "0.10.7"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "a978451301f4db1d02937a4ab3ccce137717b81826e79b7d49ffe3244a13c3b8"
+dependencies = [
+ "cfg-if",
+ "cpufeatures",
+ "digest",
+]
+
[[package]]
name = "sha2"
version = "0.10.9"
@@ -2237,6 +2258,34 @@ dependencies = [
"tokio",
]
+[[package]]
+name = "tokio-tungstenite"
+version = "0.28.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "d25a406cddcc431a75d3d9afc6a7c0f7428d4891dd973e4d54c56b46127bf857"
+dependencies = [
+ "futures-util",
+ "log",
+ "rustls",
+ "rustls-native-certs",
+ "rustls-pki-types",
+ "tokio",
+ "tokio-rustls",
+ "tungstenite 0.28.0",
+]
+
+[[package]]
+name = "tokio-tungstenite"
+version = "0.29.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "8f72a05e828585856dacd553fba484c242c46e391fb0e58917c942ee9202915c"
+dependencies = [
+ "futures-util",
+ "log",
+ "tokio",
+ "tungstenite 0.29.0",
+]
+
[[package]]
name = "tokio-util"
version = "0.7.18"
@@ -2428,6 +2477,41 @@ version = "0.2.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e421abadd41a4225275504ea4d6566923418b7f05506fbc9c0fe86ba7396114b"
+[[package]]
+name = "tungstenite"
+version = "0.28.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "8628dcc84e5a09eb3d8423d6cb682965dea9133204e8fb3efee74c2a0c259442"
+dependencies = [
+ "bytes",
+ "data-encoding",
+ "http",
+ "httparse",
+ "log",
+ "rand",
+ "rustls",
+ "rustls-pki-types",
+ "sha1",
+ "thiserror",
+ "utf-8",
+]
+
+[[package]]
+name = "tungstenite"
+version = "0.29.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "6c01152af293afb9c7c2a57e4b559c5620b421f6d133261c60dd2d0cdb38e6b8"
+dependencies = [
+ "bytes",
+ "data-encoding",
+ "http",
+ "httparse",
+ "log",
+ "rand",
+ "sha1",
+ "thiserror",
+]
+
[[package]]
name = "typenum"
version = "1.20.1"
@@ -2522,6 +2606,12 @@ version = "2.1.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "daf8dba3b7eb870caf1ddeed7bc9d2a049f3cfdfae7cb521b087cc33ae4c49da"
+[[package]]
+name = "utf-8"
+version = "0.7.6"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "09cc8ee72d2a9becf2f2febe0205bbed8fc6615b7cb429ad062dc7b7ddd036a9"
+
[[package]]
name = "utf8-zero"
version = "0.8.1"
diff --git a/Cargo.toml b/Cargo.toml
index a3ece90..530e6d6 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -16,8 +16,11 @@ name = "ghc-proxy"
path = "src/main.rs"
[dependencies]
-axum = { version = "0.8", features = ["json"] }
+axum = { version = "0.8", features = ["json", "ws"] }
tokio = { version = "1", features = ["full"] }
+# Upstream transport for the catalog's `ws:/responses` surface. rustls with the
+# platform trust store, to match reqwest and keep enterprise CAs working.
+tokio-tungstenite = { version = "0.28", default-features = false, features = ["connect", "rustls-tls-native-roots"] }
tower = { version = "0.5", features = ["util"] }
tower-http = { version = "0.7", features = ["fs"] }
reqwest = { version = "0.13", default-features = false, features = ["json", "rustls", "stream", "form"] }
@@ -38,3 +41,6 @@ self_update = { version = "0.44", default-features = false, features = ["reqwest
toml = "1.1"
[dev-dependencies]
+# `test-util` (not part of tokio's `full`) enables the paused clock used to
+# exercise the SSE keepalive timing deterministically.
+tokio = { version = "1", features = ["full", "test-util"] }
diff --git a/README.md b/README.md
index 6ba6341..720fa97 100644
--- a/README.md
+++ b/README.md
@@ -42,7 +42,12 @@ file.
upstream model supports it, otherwise translated through chat completions).
- **Gemini-compatible** `/v1beta/models/{model}:generateContent`,
`:streamGenerateContent`, and `:countTokens` endpoints (translated through chat
- completions).
+ completions). `generationConfig` parameters with an OpenAI counterpart are
+ carried across — `temperature`, `topP`, `maxOutputTokens`, `stopSequences`,
+ `candidateCount`, `seed`, the penalties, and `responseMimeType`/`responseSchema`
+ as `response_format`. `topK` and `safetySettings` have no counterpart and are
+ dropped, which the dashboard states rather than leaving you to assume they
+ took effect.
- **GitHub Models inference** — requests whose model id uses the
`publisher/model` form (e.g. `openai/gpt-4o`) are transparently routed to the
[GitHub Models](https://models.github.ai) API instead of Copilot, authenticated
@@ -59,7 +64,23 @@ file.
- **Copilot token management** with automatic refresh.
- **Orphaned `tool_use_id` recovery** — retries with offending tool results
stripped when the upstream returns the corresponding 400 error.
-- **Request analytics dashboard** at `/` and a request browser at `/requests`.
+- **The Responses API over WebSocket.** Ten models advertise a `ws:/responses`
+ transport in the Copilot catalog. `GET /v1/responses` with an `Upgrade:
+ websocket` header relays to it, recorded like any other request. A model that
+ does not advertise the transport is refused with an error frame naming the
+ endpoint that does work.
+- **Request analytics dashboard** across three pages: an overview at `/` led by
+ what Copilot actually billed (in AI units, read from `copilot_usage`, not
+ estimated from a price list), a request browser at `/requests` where each
+ exchange renders as a conversation with the raw frames a tab away, and a
+ metrics UI at `/metrics/dashboard`. Prompt-cache statistics per model at
+ `GET /api/cache` show where input tokens came from and what the cache was
+ worth. Failed attempts are counted separately rather than diluting the
+ statistics.
+- **Body capture toggled at runtime** from the dashboard or
+ `POST /api/config/debug`, so you can start recording request and response
+ bodies without a restart. Not persisted to `config.yaml`, since capture puts
+ prompts into memory and the log and should lapse on restart.
- **Interactive setup wizard** (`--setup`, or first launch in a terminal):
GitHub sign-in, live model catalog, and model-mapping configuration.
- **1M-context support** — forwards the `anthropic-beta: context-1m-2025-08-07`
@@ -91,6 +112,7 @@ ghc-proxy [options]
--fetch-version Fetch the latest VS Code version at startup
--no-fetch-version Disable dynamic VS Code version fetching
--auto-upgrade Auto-upgrade app when a newer release is available
+ (default: on)
--no-auto-upgrade Disable app auto-upgrade
--update-config Persist non-schema config write-backs (schema upgrades apply automatically)
-v, --version Show version
@@ -157,22 +179,22 @@ Config file: `~/.ghc-tunnel/config.yaml` (`%APPDATA%/ghc-tunnel/config.yaml`
on Windows). It is generated on first run or with `--config`.
```yaml
-config_version: 2
+config_version: 4
address: 127.0.0.1
port: 8314
debug: false
account_type: individual # individual | business | enterprise
-vscode_version: "1.123.0"
+vscode_version: "1.130.0"
api_version: "2025-05-01"
copilot_version: "0.48.1"
-auto_upgrade: false
+auto_upgrade: true # self-update on startup; false to disable
model_mappings:
exact:
- opus: claude-opus-4.8
- sonnet: claude-opus-4.8
+ opus: claude-opus-5
+ sonnet: claude-opus-5
haiku: claude-haiku-4.5
prefix:
- claude-sonnet-4-: claude-opus-4.8
+ claude-sonnet-4-: claude-opus-5
github_models:
enabled: true # route publisher/model ids to GitHub Models
# org: my-org # attribute inference to an organization
@@ -181,6 +203,7 @@ system_prompt_remove: []
system_prompt_add: []
tool_result_suffix_remove: []
max_connection_retries: 3
+upstream_read_timeout_seconds: 900 # max silence from upstream; 0 disables
# Optional: require this key on all LLM endpoints (Bearer / x-api-key /
# x-goog-api-key). Omit or leave empty to disable authentication.
@@ -239,6 +262,7 @@ the dashboard and model listings.
|----------|-------------|
| `POST /v1/chat/completions` | OpenAI chat completions |
| `POST /v1/responses` | OpenAI responses API (Codex) |
+| `GET /v1/responses` *(with `Upgrade: websocket`)* | Responses API over WebSocket, for models advertising `ws:/responses` |
| `GET /v1/models` | List available models |
| `GET /v1/models/{model}` | Retrieve a single model (aliases resolved) |
| `POST /v1/messages` | Anthropic messages API |
@@ -248,13 +272,20 @@ the dashboard and model listings.
| `POST /v1beta/models/{model}:countTokens` | Gemini token counting |
| `GET /health` | Liveness/readiness probe (`?strict=true` for 503 when not ready) |
| `GET /openapi.json` | OpenAPI v3 specification |
-| `GET /` | Web dashboard |
+| `GET /` | Web dashboard — overview |
| `GET /metrics/dashboard` | Metrics dashboard UI |
| `GET /metrics` | OpenMetrics endpoint |
| `GET /requests` | Request browser |
+| `GET /app.css` | Stylesheet shared by the three dashboard pages |
| `POST /api/config/reload` | Reload config.yaml without restart |
+| `POST /api/config/debug` | Turn body capture on or off (`{"debug": true}`) |
+| `GET /api/stats` | Running totals, including what Copilot billed |
+| `GET /api/cache` | Prompt-cache statistics, overall and per model |
| `GET /api/models` | All supported models (used by the dashboard) |
+The `/api/config/` routes are guarded by the API key when one is configured;
+the read-only dashboard endpoints stay open.
+
## Example Usage
### OpenAI SDK
diff --git a/docs/api.md b/docs/api.md
index b89baea..201fce1 100644
--- a/docs/api.md
+++ b/docs/api.md
@@ -32,16 +32,137 @@ require a key on the LLM endpoints; see [Authentication](#authentication) below.
| `GET /v1/models/full/` | Raw upstream model catalog with capabilities |
| `GET /usage` | Copilot plan and quota usage |
| `GET /health` | Liveness/readiness probe |
-| `GET /` | Web analytics dashboard |
+| `GET /` | Web analytics dashboard — overview |
| `GET /metrics/dashboard` | Metrics dashboard UI |
| `GET /metrics` | OpenMetrics exposition endpoint |
| `GET /requests` | Request browser |
+| `GET /app.css` | Stylesheet shared by the three dashboard pages |
| `GET /api/stats` | Dashboard statistics (JSON) |
+| `GET /api/cache` | Prompt-cache statistics, overall and per model |
| `GET /api/requests` | Recent requests (JSON) |
| `GET /api/audit` | Filtered audit records |
| `GET /api/audit/summary` | Aggregated audit summary |
| `POST /api/config/reload` | Reload `config.yaml` without restart |
+| `POST /api/config/debug` | Turn request/response body capture on or off |
| `GET /openapi.json` | OpenAPI v3 specification of the LLM endpoints |
+| `GET /v1/responses` (Upgrade) | Responses API over WebSocket |
+
+## Responses over WebSocket
+
+Several models advertise `ws:/responses` in `supported_endpoints` alongside
+`/responses`. `GET /v1/responses` (or `/responses`) with a WebSocket upgrade
+exposes that transport.
+
+The protocol is the streaming Responses API with a different carrier: the same
+`response.*` event vocabulary, one event per text frame, so a client already
+written against the SSE stream needs no new parsing. Send one frame to start a
+turn:
+
+```json
+{"type": "response.create", "model": "gpt-5.5", "input": "...", "stream": true}
+```
+
+The frame is flat — `model` sits at the top level beside the request fields, not
+nested under a `response` object. Omitting `type` or nesting the body is
+rejected.
+
+A model that does not advertise the transport is refused with an error frame
+naming the alternative rather than left waiting on the socket:
+
+```json
+{"type": "error", "error": {"code": "unsupported_api_for_model",
+ "message": "Model 'claude-opus-4.6' does not support ws:/responses. Use POST /v1/responses instead."}}
+```
+
+WebSocket turns are recorded like any other request, under the endpoint
+`ws:/responses`.
+
+Read-only dashboard endpoints are reachable without an API key so local
+monitoring keeps working. The `/api/config/` routes are not: they mutate the
+running process, and one of them turns on body capture, which writes whatever
+the client sent — credentials included — into the request log.
+
+## Statistics and failed attempts
+
+`GET /api/stats` counts requests that produced an answer. Attempts that did not
+— a non-2xx status, or a `failure_kind` on an otherwise successful one — are
+reported separately as `failed_requests` and excluded from the rest, so a burst
+of rejected calls cannot dilute a rate computed over requests that consumed
+nothing. Token and billing totals count either way: a stream cut off partway
+consumed what it consumed.
+
+## Prompt cache statistics
+
+`GET /api/cache` reports where input tokens came from. The hit rate is the early
+warning for a broken prompt prefix: on an agent workload it should sit high and
+stable, and a sudden drop means the prompt stopped matching and every turn is
+paying full input price again.
+
+```json
+{
+ "totals": {
+ "input_tokens": 9397,
+ "cache_read_tokens": 4682,
+ "cache_creation_tokens": 4682,
+ "fresh_tokens": 33,
+ "hit_rate": 0.498,
+ "request_count": 4
+ },
+ "dispositions": { "served_from_cache": 1, "wrote_to_cache": 1, "no_cache": 2 },
+ "sampled_requests": 4,
+ "by_model": [
+ {
+ "model": "gemini-3.5-flash",
+ "requests": 4,
+ "input_tokens": 9397,
+ "cache_read_tokens": 4682,
+ "cache_creation_tokens": 0,
+ "fresh_tokens": 33,
+ "hit_rate": 0.498,
+ "saved_nano_aiu": 547830000
+ }
+ ]
+}
+```
+
+`totals` are the all-time running counters. `by_model` is derived from the
+retained ring buffer, so it describes the most recent `sampled_requests` calls
+rather than every one ever served.
+
+### What a zero means
+
+Copilot states its own per-token rates on each response, in
+`copilot_usage.token_details`. `saved_nano_aiu` is computed from those rates
+rather than from a price list, so a model Copilot includes at no charge
+contributes nothing instead of an imagined discount. Positive means the cache
+paid for itself; negative is the normal shape of a turn that populated a cache
+it has not read back yet. `null` means no response for this model reported the
+rates to compute one from — distinct from `0`, which is a real figure: nothing
+was cached, so nothing was saved.
+
+`cache_creation_tokens` is only ever non-zero on the Anthropic surface with an
+explicit `cache_control` marker. Every other surface caches implicitly: the
+first call reads nothing, the next reads the whole prefix back, and no write is
+billed. Measured across twelve calls with a 27k-token prefix, exactly one
+reported a write, on `/v1/messages`. The dashboard hides the column outright
+when nothing wrote.
+
+A model with no cache activity at all is usually not a fault either: Copilot
+needs a minimum cacheable prefix before any of the prompt is eligible.
+`claude-haiku-4.5` was observed caching a 6902-token prefix but not a
+4082-token one.
+
+## Body capture
+
+`POST /api/config/debug` with `{"debug": true}` or `{"debug": false}` turns
+request/response body capture on or off for the running process. The flag is
+read live on every request, so it applies from the next call — no restart, and
+no need to have predicted in advance that you would want the bodies.
+
+It is deliberately not written back to `config.yaml`. Capture puts prompts, tool
+output and any credentials they carry into memory and the log, so it lapses on
+restart rather than staying on because someone forgot. `GET /health` reports the
+current value as `debug`.
Streaming (SSE) is supported on the chat, responses, and messages endpoints by
setting `"stream": true` in the request body. The Gemini surface streams via the
@@ -75,6 +196,12 @@ has loaded. A degraded proxy still answers `200` with `ready: false` so probes
can distinguish "process alive" from "able to serve traffic". Add `?strict=true`
to get `503 Service Unavailable` instead when the proxy is not ready.
+The `quota` object holds the most recent per-SKU allowance reported by the
+upstream. Copilot attaches it to every response, so it is current without any
+extra API call — but it stays empty until the first request has been proxied.
+The same figures are exported on `/metrics` as `ghc_proxy_quota_*` gauges
+labelled by `sku`.
+
## Retrieve a model
`GET /v1/models/{model}` returns a single catalog entry in the OpenAI shape,
@@ -84,7 +211,7 @@ the mapped Copilot model. Unknown ids return `404` with an OpenAI-style error
body.
```bash
-curl http://127.0.0.1:8314/v1/models/claude-opus-4.8
+curl http://127.0.0.1:8314/v1/models/claude-opus-5
```
## Token counting
@@ -239,7 +366,14 @@ counts, estimated cost, and prompt-cache hit rate.
- **SSE keepalive** — a `: keepalive` comment is emitted after 15 seconds of
silence so extended thinking does not trip the ~60 second idle timeout
enforced by the upstream load balancer. Comments are ignored by every
- spec-compliant SSE client.
+ spec-compliant SSE client. The comment is only injected at an event boundary:
+ if the upstream goes quiet *part way through* writing an event, splicing one
+ in would corrupt it, so that case is left to the read timeout below.
+- **Read timeout** — `upstream_read_timeout_seconds` (default 120) bounds how
+ long an upstream response may stay silent between reads. It does not cap the
+ total duration, so long answers stream normally, but a half-open connection
+ is turned into a reported stream error instead of hanging the request
+ forever. Set it to `0` to disable.
- **`anthropic-beta` passthrough** — the client's beta flags are forwarded and
merged with the ones the proxy derives (`context-1m-2025-08-07` for extended
context models, `context-management-2025-06-27` when the request uses
diff --git a/docs/claude-code.md b/docs/claude-code.md
index 2a074a9..0bcb140 100644
--- a/docs/claude-code.md
+++ b/docs/claude-code.md
@@ -50,17 +50,23 @@ export ANTHROPIC_API_KEY="ghc-proxy"
Claude Code sends specific model names (for example `claude-opus-4-7[1m]`). Use
[model mappings](configuration.md#model-mappings) to route those to whichever
-Copilot model you want. For example, to always use Claude Opus 4.8 with its
+Copilot model you want. For example, to always use Claude Opus 5 with its
native 1M context:
```yaml
model_mappings:
exact:
- claude-opus-4-8: claude-opus-4.8
+ opus: claude-opus-5
+ "5[1m]": claude-opus-5
prefix:
- claude-opus-4-7: claude-opus-4.8
+ claude-opus-4-7: claude-opus-5
+ claude-opus-4-8: claude-opus-5
```
+The built-in defaults already do this for every Claude spelling. They apply to
+a *new* config file only — an existing one keeps the targets it has, so if you
+wrote yours before Opus 5 shipped, either edit it or re-run `--setup`.
+
Restart the proxy after editing `config.yaml` — mappings are read at startup.
## Codex CLI
@@ -109,7 +115,18 @@ at `/v1beta/models/{model}:generateContent` (plus streaming and token counting).
## Tips
- Use `GET /usage` (or `ghc-proxy check-usage`) to monitor your Copilot quota.
-- The dashboard at `http://127.0.0.1:8314/` shows live request statistics and
- the full list of supported models.
+- The dashboard has three pages: the overview at `http://127.0.0.1:8314/` leads
+ with what Copilot billed for the session in AI units, alongside quota, traffic
+ and prompt-cache statistics; `/requests` browses each exchange as a
+ conversation; `/metrics/dashboard` shows the raw metric list.
+- Prompt caching is where an agent session gets cheap or expensive. The cache
+ panel breaks the hit rate down per model, because a single global number
+ cannot tell you *which* conversation stopped matching its prefix. Copilot
+ needs a minimum cacheable prefix — around 4k tokens — before any of it is
+ eligible, so short prompts legitimately show nothing.
+- To inspect what a tool is actually sending, turn on body capture from the
+ overview or with `curl -X POST http://127.0.0.1:8314/api/config/debug -d
+ '{"debug": true}'`. It takes effect on the next request and lapses on
+ restart.
- If a tool reports a model is unavailable, check `GET /v1/models` for the exact
model id and add a mapping.
diff --git a/docs/configuration.md b/docs/configuration.md
index b92fa38..f038072 100644
--- a/docs/configuration.md
+++ b/docs/configuration.md
@@ -21,7 +21,7 @@ Windows). Generated on first run, with `--config`, or through the setup wizard.
```yaml
# Schema version for migration/write-back behavior
-config_version: 2
+config_version: 4
# Server settings
address: 127.0.0.1
@@ -32,21 +32,21 @@ debug: false
account_type: individual
# Header version strings (mimic the VS Code Copilot Chat client)
-vscode_version: "1.123.0"
+vscode_version: "1.130.0"
api_version: "2025-05-01"
copilot_version: "0.48.1"
-# Self-update behavior
-auto_upgrade: false
+# Self-update behavior (enabled by default)
+auto_upgrade: true
# Model name mappings: exact (full match) and prefix (starts-with)
model_mappings:
exact:
- opus: claude-opus-4.8
- sonnet: claude-sonnet-4.6
+ opus: claude-opus-5
+ sonnet: claude-sonnet-5
haiku: claude-haiku-4.5
prefix:
- claude-sonnet-4-: claude-opus-4.8
+ claude-sonnet-4-: claude-opus-5
# GitHub Models (https://models.github.ai) inference
# Route publisher/model ids (e.g. openai/gpt-4o) to GitHub Models instead of
@@ -64,6 +64,11 @@ tool_result_suffix_remove: []
# Retry: max retries for upstream connection errors (0 = none)
max_connection_retries: 3
+# Max seconds of silence from an upstream response before it is treated as
+# dead. Bounds silence, not total duration, so long streams are fine.
+# 0 disables the timeout.
+upstream_read_timeout_seconds: 900
+
# Optional: require this key on all LLM endpoints (Bearer / x-api-key /
# x-goog-api-key). Omit or leave empty to disable authentication.
# api_key: my-secret-key
@@ -80,6 +85,14 @@ Incoming model names are rewritten before the request is forwarded upstream:
Exact matches take priority over prefix matches. Unmapped names pass through
unchanged. Use the live catalog at `GET /v1/models` to discover valid targets.
+The built-in mappings point every Claude spelling at the newest generally
+available Opus — currently `claude-opus-5` — and every Haiku spelling at
+`claude-haiku-4.5`. Anthropic writes the same version two ways (`4.8` and
+`4-8`), so both forms are listed.
+
+These are defaults for a *new* config file. A file you already have is never
+rewritten to follow them: see [Schema upgrades](#schema-upgrades).
+
### Account type
Controls the upstream base URL only:
@@ -120,6 +133,21 @@ this number; on the next start the proxy fills the missing properties with
their default values and rewrites `config.yaml` automatically, preserving every
value you have set. No flag is needed for this.
+**An upgrade only ever adds.** A model mapping already in the file keeps
+pointing where you told it to, even when a newer release changes the built-in
+default for that alias. There is no way to tell a mapping left at an old
+default apart from one deliberately pinned to that version, and pinning is
+common — so nothing existing is rewritten. To adopt the new defaults, run
+`--setup` or start it from built-in defaults with `--default`.
+
+Schema versions so far:
+
+| Version | Introduced |
+|---------|------------|
+| 2 | Opus 4.8 aliases |
+| 3 | `upstream_read_timeout_seconds`; `auto_upgrade` defaulting to true |
+| 4 | Opus 5 and Sonnet 5 aliases |
+
`--update-config` remains for the other, non-schema write-backs (for example
restoring the built-in `model_mappings` when the file has none).
@@ -149,6 +177,7 @@ ghc-proxy [options]
--fetch-version Fetch the latest VS Code version at startup
--no-fetch-version Disable dynamic VS Code version fetching
--auto-upgrade Auto-upgrade app when a newer release is available
+ (default: on)
--no-auto-upgrade Disable app auto-upgrade
--update-config Persist non-schema config write-backs (schema upgrades apply automatically)
-v, --version Show version
@@ -169,10 +198,11 @@ Every config field has a `GHC_PROXY_*` override:
| `GHC_PROXY_API_VERSION` | `X-GitHub-Api-Version` string |
| `GHC_PROXY_COPILOT_VERSION` | Copilot Chat plugin version string |
| `GHC_PROXY_MAX_CONNECTION_RETRIES` | Max connection retries |
+| `GHC_PROXY_UPSTREAM_READ_TIMEOUT` | Max seconds of upstream silence (`0` disables) |
| `GHC_PROXY_REDIRECT_ANTHROPIC` | Always translate Anthropic via chat completions |
| `GHC_PROXY_SHOW_TOKEN` | Log tokens on refresh (`true`/`1`) |
| `GHC_PROXY_DYNAMIC_VSCODE_VERSION` | Fetch latest VS Code version (`true`/`1`) |
-| `GHC_PROXY_AUTO_UPGRADE` | Auto-upgrade app on startup (`true`/`1`) |
+| `GHC_PROXY_AUTO_UPGRADE` | Auto-upgrade app on startup (`true`/`1`); set `0` to disable |
| `GHC_PROXY_RATE_LIMIT_SECONDS` | Minimum seconds between requests |
| `GHC_PROXY_RATE_LIMIT_WAIT` | Wait instead of rejecting when limited (`true`/`1`) |
| `GHC_PROXY_MANUAL_APPROVE` | Require manual approval per request (`true`/`1`) |
@@ -215,3 +245,26 @@ stale clients:
Enable `dynamic_vscode_version` (or `--fetch-version`) to refresh the VS Code
version automatically at startup.
+
+## Self-update
+
+`auto_upgrade` is **enabled by default**, including for config files written
+before the setting existed. On startup the proxy checks GitHub releases and, if
+a newer version is published, downloads and replaces its own binary. The
+replacement takes effect on the **next** start — the running process keeps
+serving the old code until it restarts.
+
+Disable it with any of:
+
+```yaml
+auto_upgrade: false
+```
+
+```bash
+ghc-proxy --no-auto-upgrade
+GHC_PROXY_AUTO_UPGRADE=0 ghc-proxy
+```
+
+Turn it off if the binary is managed by a package manager, or if it lives
+somewhere the process should not rewrite — for example a build output directory
+that `cargo build` or `cargo clean` also writes to.
diff --git a/docs/getting-started.md b/docs/getting-started.md
index 0ed26a8..8a40338 100644
--- a/docs/getting-started.md
+++ b/docs/getting-started.md
@@ -42,6 +42,7 @@ Once running, the proxy prints the endpoints it serves:
```text
Starting GitHub Copilot API Proxy on 127.0.0.1:8314
Dashboard: http://127.0.0.1:8314/
+Health check: http://127.0.0.1:8314/health
Metrics UI: http://127.0.0.1:8314/metrics/dashboard
OpenMetrics: http://127.0.0.1:8314/metrics
Reload config: POST http://127.0.0.1:8314/api/config/reload
@@ -52,6 +53,10 @@ Gemini API: http://127.0.0.1:8314/v1beta/models/{model}:generateContent
OpenAPI spec: http://127.0.0.1:8314/openapi.json
```
+The dashboard link is the overview; `/requests` browses individual exchanges.
+See the [API reference](api.md) for the rest, including the WebSocket transport
+and the statistics endpoints.
+
## Authentication
A GitHub token is resolved in this order:
@@ -122,3 +127,5 @@ persisted `machine_id.txt` used for client disguise.
- Tune behavior in the [Configuration](configuration.md) reference.
- Explore the [API Reference](api.md) and client examples.
- Wire up [Claude Code & Codex](claude-code.md).
+- Watch what it costs: the overview at `/` leads with the AI units Copilot
+ billed, and `/api/cache` breaks prompt-cache effectiveness down per model.
diff --git a/docs/index.md b/docs/index.md
index 99e970c..a035526 100644
--- a/docs/index.md
+++ b/docs/index.md
@@ -33,6 +33,7 @@ so OpenAI- and Anthropic-native clients work unmodified.
## Highlights
- **OpenAI-compatible** `/v1/chat/completions` and `/v1/responses` (Codex) endpoints.
+- **The Responses API over WebSocket** — `GET /v1/responses` with `Upgrade: websocket`, for the models whose catalog advertises `ws:/responses`.
- **Anthropic-compatible** `/v1/messages` endpoint with native passthrough or translation.
- **Gemini-compatible** `/v1beta/models/{model}:generateContent` (+ streaming and token counting).
- **Embeddings**, **model listing**, **token counting**, and a **usage** endpoint.
@@ -43,7 +44,9 @@ so OpenAI- and Anthropic-native clients work unmodified.
- **Optional API-key auth** on the LLM endpoints (Bearer / `x-api-key` / `x-goog-api-key`), off by default.
- **One-click client setup** for Claude Code, Codex, and the Gemini CLI.
- **OpenAPI spec** at `/openapi.json`.
-- **Analytics dashboard** at `/` and a request browser at `/requests`.
+- **Three-page dashboard** — an overview at `/` led by what Copilot actually billed, a request browser at `/requests` that renders each exchange as a conversation, and metrics at `/metrics/dashboard`.
+- **Spend and cache analytics** — billed AI units read from `copilot_usage` rather than estimated, and per-model prompt-cache statistics at `/api/cache`.
+- **Body capture toggled at runtime** via `POST /api/config/debug`, so you can start recording bodies without a restart.
## Quick start
diff --git a/public/app.css b/public/app.css
new file mode 100644
index 0000000..42f2ebe
--- /dev/null
+++ b/public/app.css
@@ -0,0 +1,372 @@
+/* ghc-proxy dashboard — shared design system.
+ *
+ * Served from /app.css so the three pages share one visual language instead of
+ * carrying three drifting copies of the same rules. */
+
+:root {
+ color-scheme: dark;
+
+ --bg: #0d1117;
+ --surface: #161b22;
+ --surface-2: #21262d;
+ --border: #30363d;
+ --border-soft: #21262d;
+ --text: #e6edf3;
+ --muted: #8b949e;
+ --accent: #58a6ff;
+ --ok: #3fb950;
+ --warn: #d29922;
+ --danger: #f85149;
+
+ --radius: 8px;
+ --radius-sm: 6px;
+ --mono: ui-monospace, SFMono-Regular, Menlo, Consolas, monospace;
+
+ /* Page width is per-page, not global: the overview is a handful of large
+ * numbers and short labels, which just drift apart when stretched, while the
+ * request table and the metric list are data that a wide screen genuinely
+ * helps you read. Capped rather than fluid so an ultrawide does not turn a
+ * table row into a journey for the eye. */
+ --page-max: 1280px;
+}
+
+body[data-page="requests"] { --page-max: 2000px; }
+body[data-page="metrics"] { --page-max: 1600px; }
+
+* { box-sizing: border-box; }
+
+body {
+ font-family: system-ui, -apple-system, "Segoe UI", Roboto, sans-serif;
+ margin: 0;
+ background: var(--bg);
+ color: var(--text);
+ font-size: 14px;
+ line-height: 1.5;
+}
+
+a { color: var(--accent); text-decoration: none; }
+a:hover { text-decoration: underline; }
+
+.wrap {
+ max-width: var(--page-max);
+ margin: 0 auto;
+ padding: 0 1.5rem 4rem;
+}
+
+/* ---------------------------------------------------------------- nav ---- */
+/* Persistent across all three pages: the "trunk test" wants every page to
+ * answer what this is, where you are, and what the other sections are. */
+
+.nav {
+ position: sticky;
+ top: 0;
+ z-index: 10;
+ background: rgba(13, 17, 23, 0.85);
+ backdrop-filter: blur(8px);
+ border-bottom: 1px solid var(--border);
+ margin-bottom: 1.75rem;
+}
+
+.nav-inner {
+ max-width: var(--page-max);
+ margin: 0 auto;
+ padding: 0 1.5rem;
+ display: flex;
+ align-items: center;
+ gap: 1.5rem;
+ height: 52px;
+}
+
+.brand {
+ font-weight: 600;
+ color: var(--text);
+ letter-spacing: -0.01em;
+ white-space: nowrap;
+}
+.brand:hover { text-decoration: none; }
+
+.nav-links { display: flex; gap: 0.25rem; flex: 1; }
+
+.nav-links a {
+ color: var(--muted);
+ padding: 0.3rem 0.7rem;
+ border-radius: var(--radius-sm);
+ font-size: 0.875rem;
+}
+.nav-links a:hover { color: var(--text); background: var(--surface-2); text-decoration: none; }
+
+/* Current section, set via
diff --git a/scripts/__pycache__/replay.cpython-313.pyc b/scripts/__pycache__/replay.cpython-313.pyc
deleted file mode 100644
index 27145d0..0000000
Binary files a/scripts/__pycache__/replay.cpython-313.pyc and /dev/null differ
diff --git a/scripts/alias_check.py b/scripts/alias_check.py
new file mode 100644
index 0000000..5e086e8
--- /dev/null
+++ b/scripts/alias_check.py
@@ -0,0 +1,60 @@
+"""Checks that every Claude alias resolves to a model the upstream will serve.
+
+A mapping table is only correct if the name it produces is one Copilot accepts,
+so this asks the live upstream rather than comparing strings.
+
+Note this exercises the *running proxy's* `config.yaml`, not the built-in
+defaults. A hand-tuned file may be missing a prefix the defaults carry, in which
+case the alias falls through unmapped and Copilot rejects it -- that is a
+finding about that file. The defaults themselves are covered by the unit tests
+in `src/translate.rs`.
+"""
+
+import json
+import sys
+import urllib.error
+import urllib.request
+
+BASE = f"http://127.0.0.1:{sys.argv[1] if len(sys.argv) > 1 else '8399'}"
+
+ALIASES = [
+ "opus",
+ "sonnet",
+ "haiku",
+ "opus5",
+ "5[1m]",
+ "opus4-8",
+ "claude-opus-4.6",
+ "claude-opus-4.8",
+ "claude-opus-5",
+ "claude-sonnet-4-5",
+ "claude-sonnet-4.6",
+ "claude-sonnet-5",
+ "claude-sonnet-4-20250101",
+ "claude-haiku-4.5",
+]
+
+failures = 0
+for alias in ALIASES:
+ req = urllib.request.Request(
+ BASE + "/v1/messages",
+ data=json.dumps(
+ {
+ "model": alias,
+ "max_tokens": 8,
+ "messages": [{"role": "user", "content": "Reply with the word ok."}],
+ }
+ ).encode(),
+ headers={"content-type": "application/json"},
+ )
+ try:
+ with urllib.request.urlopen(req, timeout=120) as r:
+ served = json.loads(r.read()).get("model", "?")
+ print(f" {alias:<26} -> {served}")
+ except urllib.error.HTTPError as e:
+ failures += 1
+ print(f" {alias:<26} -> HTTP {e.code} {e.read()[:120].decode(errors='replace')}")
+
+print()
+print("all aliases served" if failures == 0 else f"{failures} alias(es) failed")
+sys.exit(1 if failures else 0)
diff --git a/scripts/anthropic_cache_probe.py b/scripts/anthropic_cache_probe.py
new file mode 100644
index 0000000..286f05a
--- /dev/null
+++ b/scripts/anthropic_cache_probe.py
@@ -0,0 +1,79 @@
+"""Why does the Anthropic surface never populate the prompt cache?
+
+The proxy forwards `cache_control` and Copilot echoes the full cache-accounting
+shape back -- all zeros. This tries the remaining client-side levers one at a
+time so the failing one can be named rather than guessed at.
+"""
+
+import json
+import sys
+import urllib.error
+import urllib.request
+
+BASE = f"http://127.0.0.1:{sys.argv[1] if len(sys.argv) > 1 else '8399'}"
+MODEL = sys.argv[2] if len(sys.argv) > 2 else "claude-haiku-4.5"
+
+
+def attempt(label, system, beta=None, ttl=None):
+ payload = {
+ "model": MODEL,
+ "max_tokens": 8,
+ "system": system,
+ "messages": [{"role": "user", "content": "Reply with the word ok."}],
+ }
+ headers = {"content-type": "application/json"}
+ if beta:
+ headers["anthropic-beta"] = beta
+ req = urllib.request.Request(
+ BASE + "/v1/messages", data=json.dumps(payload).encode(), headers=headers
+ )
+ try:
+ with urllib.request.urlopen(req, timeout=120) as r:
+ u = json.loads(r.read())["usage"]
+ except urllib.error.HTTPError as e:
+ print(f"{label:<42}{e.code} {e.read()[:90].decode(errors='replace')}")
+ return
+ print(
+ f"{label:<42}in={u.get('input_tokens'):>6} "
+ f"write={u.get('cache_creation_input_tokens'):>6} "
+ f"read={u.get('cache_read_input_tokens'):>6}"
+ )
+
+
+def block(chars, ttl=None):
+ cc = {"type": "ephemeral"}
+ if ttl:
+ cc["ttl"] = ttl
+ return [{"type": "text", "text": ("cache me. " * 30000)[:chars], "cache_control": cc}]
+
+
+print(f"model: {MODEL}\n")
+# The text the other scripts use. Same character count as the block above but
+# far fewer tokens, which is the whole question.
+DEMO = ("The proxy records what the upstream reports. " * 520)[:22880]
+# Each pair is sent twice: a write can only be observed as a read on the retry.
+for label, kwargs in [
+ ("4k chars, default beta", dict(system=block(4000))),
+ ("23k chars, default beta", dict(system=block(23000))),
+ ("92k chars, default beta", dict(system=block(92000))),
+ (
+ "23k chars, prompt-caching beta",
+ dict(system=block(23000), beta="prompt-caching-2024-07-31"),
+ ),
+ ("23k chars, 1h ttl", dict(system=block(23000, ttl="1h"))),
+ ("23k chars, plain string system", dict(system=("cache me. " * 30000)[:23000])),
+ (
+ "22.9k chars of demo filler",
+ dict(
+ system=[
+ {
+ "type": "text",
+ "text": DEMO,
+ "cache_control": {"type": "ephemeral"},
+ }
+ ]
+ ),
+ ),
+]:
+ attempt(f"{label} (1st)", **kwargs)
+ attempt(f"{label} (2nd)", **kwargs)
diff --git a/scripts/cache_demo.py b/scripts/cache_demo.py
new file mode 100644
index 0000000..53830e2
--- /dev/null
+++ b/scripts/cache_demo.py
@@ -0,0 +1,96 @@
+"""Exercises the prompt cache on models that price it differently.
+
+The point is not that caching works -- that is already covered -- but that the
+proxy reports a saving only where the model itself published rates to compute
+one from. Run against a proxy with body capture off; only the recorded
+statistics matter.
+"""
+
+import json
+import sys
+import urllib.request
+
+BASE = sys.argv[1] if len(sys.argv) > 1 else "http://127.0.0.1:8399"
+
+# Long enough to clear Anthropic's minimum cacheable prefix. The content is
+# irrelevant; only its length and its being byte-identical across turns matter.
+FILLER = ("The proxy records what the upstream reports. " * 520)[:22880]
+
+
+def post(path, payload):
+ req = urllib.request.Request(
+ BASE + path,
+ data=json.dumps(payload).encode(),
+ headers={"content-type": "application/json"},
+ )
+ with urllib.request.urlopen(req, timeout=180) as r:
+ return json.loads(r.read())
+
+
+def anthropic(turn):
+ return post(
+ "/v1/messages",
+ {
+ "model": "claude-haiku-4.5",
+ "max_tokens": 32,
+ "system": [
+ {
+ "type": "text",
+ "text": FILLER,
+ "cache_control": {"type": "ephemeral"},
+ }
+ ],
+ "messages": [{"role": "user", "content": f"Reply with the number {turn}."}],
+ },
+ )
+
+
+def chat(model):
+ return post(
+ "/v1/chat/completions",
+ {
+ "model": model,
+ "max_tokens": 32,
+ "messages": [
+ {"role": "system", "content": FILLER},
+ {"role": "user", "content": "Reply with the word ok."},
+ ],
+ },
+ )
+
+
+def responses(model):
+ return post(
+ "/v1/responses",
+ {
+ "model": model,
+ "max_output_tokens": 32,
+ "instructions": FILLER,
+ "input": "Reply with the word ok.",
+ },
+ )
+
+
+print("anthropic turn 1 (populates the cache)")
+anthropic(1)
+print("anthropic turn 2 (should read it back)")
+anthropic(2)
+print("gemini-3.5-flash (prices cache writes at zero)")
+chat("gemini-3.5-flash")
+# Responses-only, and observed to publish no cache_write rate at all -- the
+# case that a flat zero in the Written column would misrepresent.
+print("gpt-5.5 (publishes no cache-write rate)")
+try:
+ responses("gpt-5.5")
+except Exception as e: # noqa: BLE001 - a refusal is still a data point
+ print(f" gpt-5.5: {e}")
+
+cache = json.loads(urllib.request.urlopen(BASE + "/api/cache", timeout=30).read())
+print()
+print(f"{'model':<24}{'in':>9}{'read':>9}{'write':>9}{'writes?':>9}{'saved nAIU':>14}")
+for m in cache["by_model"]:
+ print(
+ f"{m['model']:<24}{m['input_tokens']:>9}{m['cache_read_tokens']:>9}"
+ f"{m['cache_creation_tokens']:>9}{str(m['prices_cache_write']):>9}"
+ f"{m['saved_nano_aiu']:>14}"
+ )
diff --git a/scripts/cache_sweep.py b/scripts/cache_sweep.py
new file mode 100644
index 0000000..78f2724
--- /dev/null
+++ b/scripts/cache_sweep.py
@@ -0,0 +1,168 @@
+"""Sweeps models across every surface and reports which ones yielded cache facts.
+
+Answers one question: when the dashboard shows a dash, is that because the
+model reported nothing, or because the proxy failed to read what it reported?
+Needs body capture on, so the raw upstream payload can be inspected.
+"""
+
+import json
+import sys
+import urllib.error
+import urllib.request
+
+BASE = f"http://127.0.0.1:{sys.argv[1] if len(sys.argv) > 1 else '8399'}"
+
+# Long enough to be worth caching, and identical across calls so a second pass
+# can read the first one back.
+FILLER = ("The proxy records what the upstream reports. " * 520)[:22880]
+
+
+def call(path, payload):
+ req = urllib.request.Request(
+ BASE + path,
+ data=json.dumps(payload).encode(),
+ headers={"content-type": "application/json"},
+ )
+ try:
+ with urllib.request.urlopen(req, timeout=180) as r:
+ r.read()
+ return None
+ except urllib.error.HTTPError as e:
+ return f"{e.code}"
+ except Exception as e: # noqa: BLE001
+ return str(e)[:40]
+
+
+def chat(model, stream):
+ return call(
+ "/v1/chat/completions",
+ {
+ "model": model,
+ "max_tokens": 24,
+ "stream": stream,
+ "messages": [
+ {"role": "system", "content": FILLER},
+ {"role": "user", "content": "Reply with the word ok."},
+ ],
+ },
+ )
+
+
+def messages(model, stream):
+ return call(
+ "/v1/messages",
+ {
+ "model": model,
+ "max_tokens": 24,
+ "stream": stream,
+ "system": [
+ {
+ "type": "text",
+ "text": FILLER,
+ "cache_control": {"type": "ephemeral"},
+ }
+ ],
+ "messages": [{"role": "user", "content": "Reply with the word ok."}],
+ },
+ )
+
+
+def responses(model, stream):
+ return call(
+ "/v1/responses",
+ {
+ "model": model,
+ "max_output_tokens": 24,
+ "stream": stream,
+ "instructions": FILLER,
+ "input": "Reply with the word ok.",
+ },
+ )
+
+
+def gemini(model, stream):
+ action = "streamGenerateContent" if stream else "generateContent"
+ return call(
+ f"/v1beta/models/{model}:{action}",
+ {
+ "systemInstruction": {"parts": [{"text": FILLER}]},
+ "contents": [{"role": "user", "parts": [{"text": "Reply with the word ok."}]}],
+ "generationConfig": {"maxOutputTokens": 24},
+ },
+ )
+
+
+# `/api/models` is the plain OpenAI list; only the full catalogue carries
+# `supported_endpoints`, which is what decides where each model can be called.
+with urllib.request.urlopen(BASE + "/v1/models/full/", timeout=30) as r:
+ catalog = json.loads(r.read())
+supported = {
+ m["id"]: set(m.get("supported_endpoints") or [])
+ for m in (catalog.get("data") or catalog.get("models") or [])
+}
+
+PLAN = [
+ ("chat", "/chat/completions", chat),
+ ("messages", "/v1/messages", messages),
+ ("responses", "/responses", responses),
+]
+
+# One representative per family, so the sweep stays inside the rate limit.
+WANTED = [
+ "claude-haiku-4.5",
+ "claude-sonnet-4.5",
+ "claude-opus-4.6",
+ "gemini-3.5-flash",
+ "gemini-3-pro",
+ "gpt-5-mini",
+ "gpt-5.4",
+ "gpt-5.5",
+ "gpt-5.3-codex",
+ "grok-4.5",
+]
+
+for model in WANTED:
+ caps = supported.get(model)
+ if caps is None:
+ print(f" {model}: not in catalog")
+ continue
+ for label, endpoint, fn in PLAN:
+ if endpoint not in caps:
+ continue
+ for stream in (False, True):
+ err = fn(model, stream)
+ tag = f"{model} {label}{' stream' if stream else ''}"
+ print(f" {tag}: {err or 'ok'}")
+ if "/chat/completions" in caps and model.startswith("gemini"):
+ for stream in (False, True):
+ err = gemini(model, stream)
+ print(f" {model} gemini{' stream' if stream else ''}: {err or 'ok'}")
+
+print()
+audit = json.loads(urllib.request.urlopen(BASE + "/api/audit?per_page=200", timeout=30).read())
+print(f"{'endpoint':<26}{'model':<20}{'body':>10}{'copilot_usage':>15}{'details':>9}")
+seen = set()
+for i in audit["items"]:
+ body = i.get("response_body")
+ # An uncaptured body is its own state; calling it a stream would hide the
+ # fact that this row proves nothing either way.
+ shape = "none" if not body else "json" if body.startswith("{") else "stream"
+ key = (i["endpoint"], i["model"], shape)
+ if key in seen:
+ continue
+ seen.add(key)
+ print(
+ f"{i['endpoint']:<26}{i['model']:<20}{shape:>10}"
+ f"{str('copilot_usage' in (body or '')):>15}"
+ f"{str('token_details' in (body or '')):>9}"
+ )
+
+print()
+cache = json.loads(urllib.request.urlopen(BASE + "/api/cache", timeout=30).read())
+print(f"{'model':<24}{'in':>8}{'read':>8}{'write':>8}{'writes?':>9}{'saved':>14}")
+for m in cache["by_model"]:
+ print(
+ f"{m['model']:<24}{m['input_tokens']:>8}{m['cache_read_tokens']:>8}"
+ f"{m['cache_creation_tokens']:>8}{str(m['prices_cache_write']):>9}"
+ f"{str(m['saved_nano_aiu']):>14}"
+ )
diff --git a/scripts/cache_write_probe.py b/scripts/cache_write_probe.py
new file mode 100644
index 0000000..b5a2ae1
--- /dev/null
+++ b/scripts/cache_write_probe.py
@@ -0,0 +1,111 @@
+"""Does the Written column ever carry data outside the Anthropic surface?
+
+Sends a prefix well past the minimum cacheable size twice per surface: the
+first call should populate the cache, the second read it back. If a surface
+never reports a write, the column is dead weight there and the dashboard should
+say so rather than print a zero.
+"""
+
+import json
+import sys
+import urllib.error
+import urllib.request
+
+BASE = f"http://127.0.0.1:{sys.argv[1] if len(sys.argv) > 1 else '8399'}"
+
+# ~28k tokens: past every observed minimum, so a surface that can write will.
+BIG = ("cache me. " * 40000)[:92000]
+
+
+def call(path, payload):
+ req = urllib.request.Request(
+ BASE + path,
+ data=json.dumps(payload).encode(),
+ headers={"content-type": "application/json"},
+ )
+ try:
+ with urllib.request.urlopen(req, timeout=180) as r:
+ r.read()
+ return None
+ except urllib.error.HTTPError as e:
+ return str(e.code)
+ except Exception as e: # noqa: BLE001
+ return str(e)[:40]
+
+
+def chat(model):
+ return call(
+ "/v1/chat/completions",
+ {
+ "model": model,
+ "max_tokens": 8,
+ "messages": [
+ {"role": "system", "content": BIG},
+ {"role": "user", "content": "Reply with the word ok."},
+ ],
+ },
+ )
+
+
+def messages(model):
+ return call(
+ "/v1/messages",
+ {
+ "model": model,
+ "max_tokens": 8,
+ "system": [
+ {"type": "text", "text": BIG, "cache_control": {"type": "ephemeral"}}
+ ],
+ "messages": [{"role": "user", "content": "Reply with the word ok."}],
+ },
+ )
+
+
+def responses(model):
+ return call(
+ "/v1/responses",
+ {
+ "model": model,
+ "max_output_tokens": 16,
+ "instructions": BIG,
+ "input": "Reply with the word ok.",
+ },
+ )
+
+
+PLAN = [
+ ("claude-haiku-4.5", "/v1/messages", messages),
+ ("claude-haiku-4.5", "/chat/completions", chat),
+ ("gemini-3.5-flash", "/chat/completions", chat),
+ ("gpt-5-mini", "/chat/completions", chat),
+ ("gpt-5-mini", "/responses", responses),
+ ("gpt-5.5", "/responses", responses),
+]
+
+before = {
+ i["id"]
+ for i in json.loads(
+ urllib.request.urlopen(BASE + "/api/requests?per_page=500", timeout=30).read()
+ )["items"]
+}
+
+for model, endpoint, fn in PLAN:
+ for turn in (1, 2):
+ err = fn(model)
+ if err:
+ print(f" {model} {endpoint} turn {turn}: {err}")
+
+items = json.loads(
+ urllib.request.urlopen(BASE + "/api/requests?per_page=500", timeout=30).read()
+)["items"]
+fresh = [i for i in items if i["id"] not in before]
+fresh.reverse()
+
+print()
+print(f"{'endpoint':<22}{'model':<20}{'in':>8}{'read':>8}{'write':>8}{'prices?':>9}")
+for i in fresh:
+ print(
+ f"{i['endpoint']:<22}{i['model']:<20}{i['input_tokens']:>8}"
+ f"{i['cache_read_input_tokens']:>8}{i['cache_creation_input_tokens']:>8}"
+ f"{str(i.get('prices_cache_write')):>9}"
+ )
diff --git a/scripts/protocol_check.py b/scripts/protocol_check.py
new file mode 100644
index 0000000..6f56e61
--- /dev/null
+++ b/scripts/protocol_check.py
@@ -0,0 +1,475 @@
+"""Protocol conformance sweep against a running ghc-proxy.
+
+The proxy is a translating gateway: a client speaks Anthropic, OpenAI chat
+completions, the OpenAI Responses API or Gemini, and the upstream only ever
+speaks chat completions or Responses. The property that matters is therefore
+not "did it return 200" but "did it return the shape the *client* asked for" —
+a Gemini client must get `candidates`, never `choices`, no matter what the
+upstream sent back.
+
+Run with the proxy already listening:
+
+ python scripts/protocol_check.py --port 8399
+
+Every check is an assertion about the response the client actually receives.
+Nothing here inspects the dashboard; this is about the wire.
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import sys
+import urllib.error
+import urllib.request
+from dataclasses import dataclass, field
+
+TIMEOUT = 180
+
+
+@dataclass
+class Result:
+ name: str
+ ok: bool
+ detail: str = ""
+
+
+@dataclass
+class Report:
+ results: list[Result] = field(default_factory=list)
+
+ def check(self, name: str, ok: bool, detail: str = "") -> bool:
+ self.results.append(Result(name, ok, detail))
+ return ok
+
+ @property
+ def failed(self) -> list[Result]:
+ return [r for r in self.results if not r.ok]
+
+
+def post(base: str, path: str, body: dict) -> tuple[int, str, str]:
+ """Returns (status, content_type, text). Never raises on HTTP errors."""
+ req = urllib.request.Request(
+ base + path,
+ data=json.dumps(body).encode(),
+ headers={"Content-Type": "application/json"},
+ method="POST",
+ )
+ try:
+ with urllib.request.urlopen(req, timeout=TIMEOUT) as r:
+ return r.status, r.headers.get("Content-Type", ""), r.read().decode("utf-8", "replace")
+ except urllib.error.HTTPError as e:
+ return e.code, e.headers.get("Content-Type", ""), e.read().decode("utf-8", "replace")
+
+
+def get(base: str, path: str) -> tuple[int, str]:
+ try:
+ with urllib.request.urlopen(base + path, timeout=TIMEOUT) as r:
+ return r.status, r.read().decode("utf-8", "replace")
+ except urllib.error.HTTPError as e:
+ return e.code, e.read().decode("utf-8", "replace")
+
+
+def sse_events(text: str) -> list[tuple[str | None, str]]:
+ """Splits an SSE body into (event name, data payload) pairs."""
+ out = []
+ for frame in text.split("\n\n"):
+ name, data = None, []
+ for line in frame.splitlines():
+ if line.startswith("event:"):
+ name = line[6:].strip()
+ elif line.startswith("data:"):
+ data.append(line[5:].strip())
+ if data or name:
+ out.append((name, "\n".join(data)))
+ return out
+
+
+def sse_types(text: str) -> list[str]:
+ """The `type` of every JSON frame, which is how all three streaming
+ protocols label what a frame is."""
+ types = []
+ for name, data in sse_events(text):
+ if data == "[DONE]":
+ types.append("[DONE]")
+ continue
+ try:
+ j = json.loads(data)
+ except ValueError:
+ continue
+ types.append(j.get("type") or name or ("chat.chunk" if "choices" in j else "?"))
+ return types
+
+
+# --------------------------------------------------------------------------
+# Anthropic /v1/messages
+# --------------------------------------------------------------------------
+
+def check_anthropic(base: str, rep: Report, model: str) -> None:
+ p = "anthropic"
+
+ status, _, text = post(base, "/v1/messages", {
+ "model": model, "max_tokens": 60,
+ "messages": [{"role": "user", "content": "Reply with exactly: ok"}],
+ })
+ j = json.loads(text) if status == 200 else {}
+ rep.check(f"{p}: sync 200", status == 200, text[:200])
+ rep.check(f"{p}: sync type=message", j.get("type") == "message", str(j.get("type")))
+ rep.check(f"{p}: sync role=assistant", j.get("role") == "assistant", str(j.get("role")))
+ rep.check(f"{p}: sync content is block array",
+ isinstance(j.get("content"), list) and bool(j["content"]),
+ json.dumps(j.get("content"))[:160])
+ rep.check(f"{p}: sync has stop_reason", j.get("stop_reason") is not None, str(j.get("stop_reason")))
+ u = j.get("usage") or {}
+ rep.check(f"{p}: sync usage has input/output tokens",
+ "input_tokens" in u and "output_tokens" in u, json.dumps(u)[:160])
+ rep.check(f"{p}: sync does NOT leak choices", "choices" not in j, "leaked OpenAI shape")
+
+ status, ctype, text = post(base, "/v1/messages", {
+ "model": model, "max_tokens": 60, "stream": True,
+ "messages": [{"role": "user", "content": "Count 1 to 3."}],
+ })
+ types = sse_types(text)
+ rep.check(f"{p}: stream 200", status == 200, text[:200])
+ rep.check(f"{p}: stream content-type", "text/event-stream" in ctype, ctype)
+ for want in ("message_start", "content_block_start", "content_block_delta",
+ "content_block_stop", "message_delta", "message_stop"):
+ rep.check(f"{p}: stream has {want}", want in types, ",".join(types[:12]))
+ rep.check(f"{p}: stream ends with message_stop",
+ "message_stop" in types and types.index("message_stop") >= len(types) - 2,
+ ",".join(types[-3:]))
+ rep.check(f"{p}: stream frames carry event: names",
+ all(n for n, d in sse_events(text) if d and d != "[DONE]"),
+ "a frame was missing its event: line")
+
+ status, _, text = post(base, "/v1/messages", {
+ "model": model, "max_tokens": 200,
+ "tools": [{
+ "name": "get_weather",
+ "description": "Look up the weather",
+ "input_schema": {"type": "object", "properties": {"city": {"type": "string"}},
+ "required": ["city"]},
+ }],
+ "messages": [{"role": "user", "content": "Weather in Paris? Call the tool."}],
+ })
+ j = json.loads(text) if status == 200 else {}
+ blocks = j.get("content") or []
+ tool_use = [b for b in blocks if b.get("type") == "tool_use"]
+ rep.check(f"{p}: tools produce tool_use block", bool(tool_use), json.dumps(blocks)[:200])
+ if tool_use:
+ rep.check(f"{p}: tool_use has name+input+id",
+ all(k in tool_use[0] for k in ("name", "input", "id")),
+ json.dumps(tool_use[0])[:200])
+ rep.check(f"{p}: stop_reason=tool_use", j.get("stop_reason") == "tool_use",
+ str(j.get("stop_reason")))
+
+ # Multi-turn with a tool_result the client sends back.
+ if tool_use:
+ tu = tool_use[0]
+ status, _, text = post(base, "/v1/messages", {
+ "model": model, "max_tokens": 120,
+ "tools": [{"name": "get_weather", "description": "Look up the weather",
+ "input_schema": {"type": "object",
+ "properties": {"city": {"type": "string"}},
+ "required": ["city"]}}],
+ "messages": [
+ {"role": "user", "content": "Weather in Paris? Call the tool."},
+ {"role": "assistant", "content": blocks},
+ {"role": "user", "content": [{
+ "type": "tool_result", "tool_use_id": tu["id"],
+ "content": [{"type": "text", "text": "18C and clear"}],
+ }]},
+ ],
+ })
+ rep.check(f"{p}: tool_result round-trip accepted", status == 200, text[:200])
+
+ status, _, text = post(base, "/v1/messages/count_tokens", {
+ "model": model,
+ "messages": [{"role": "user", "content": "hello world"}],
+ })
+ j = json.loads(text) if status == 200 else {}
+ rep.check(f"{p}: count_tokens 200", status == 200, text[:200])
+ rep.check(f"{p}: count_tokens returns input_tokens",
+ isinstance(j.get("input_tokens"), int) and j["input_tokens"] > 0,
+ json.dumps(j)[:160])
+
+
+# --------------------------------------------------------------------------
+# OpenAI chat completions /v1/chat/completions
+# --------------------------------------------------------------------------
+
+def check_chat(base: str, rep: Report, model: str) -> None:
+ p = "chat"
+
+ status, _, text = post(base, "/v1/chat/completions", {
+ "model": model, "max_tokens": 60,
+ "messages": [{"role": "user", "content": "Reply with exactly: ok"}],
+ })
+ j = json.loads(text) if status == 200 else {}
+ rep.check(f"{p}: sync 200", status == 200, text[:200])
+ rep.check(f"{p}: sync has choices[]", isinstance(j.get("choices"), list) and bool(j["choices"]),
+ json.dumps(j)[:160])
+ if j.get("choices"):
+ c = j["choices"][0]
+ rep.check(f"{p}: choice has message.role=assistant",
+ (c.get("message") or {}).get("role") == "assistant", json.dumps(c)[:160])
+ rep.check(f"{p}: choice has finish_reason", c.get("finish_reason") is not None,
+ str(c.get("finish_reason")))
+ u = j.get("usage") or {}
+ rep.check(f"{p}: usage has prompt/completion tokens",
+ "prompt_tokens" in u and "completion_tokens" in u, json.dumps(u)[:160])
+ rep.check(f"{p}: sync does NOT leak Anthropic shape", "content" not in j, "leaked content[]")
+
+ status, ctype, text = post(base, "/v1/chat/completions", {
+ "model": model, "max_tokens": 60, "stream": True,
+ "stream_options": {"include_usage": True},
+ "messages": [{"role": "user", "content": "Count 1 to 3."}],
+ })
+ frames = [d for _, d in sse_events(text) if d]
+ rep.check(f"{p}: stream 200", status == 200, text[:200])
+ rep.check(f"{p}: stream content-type", "text/event-stream" in ctype, ctype)
+ rep.check(f"{p}: stream terminates with [DONE]", frames and frames[-1] == "[DONE]",
+ frames[-1][:80] if frames else "no frames")
+ parsed = [json.loads(d) for d in frames if d != "[DONE]"]
+ rep.check(f"{p}: every frame has choices[]", all("choices" in d for d in parsed),
+ "a frame was missing choices")
+ text_seen = "".join(
+ (c.get("delta") or {}).get("content") or ""
+ for d in parsed for c in d.get("choices") or []
+ )
+ rep.check(f"{p}: stream produced text", bool(text_seen.strip()), repr(text_seen[:80]))
+ rep.check(f"{p}: stream reports a finish_reason",
+ any(c.get("finish_reason") for d in parsed for c in d.get("choices") or []),
+ "none present")
+
+ status, _, text = post(base, "/v1/chat/completions", {
+ "model": model, "max_tokens": 200,
+ "tools": [{"type": "function", "function": {
+ "name": "get_weather", "description": "Look up the weather",
+ "parameters": {"type": "object", "properties": {"city": {"type": "string"}},
+ "required": ["city"]}}}],
+ "messages": [{"role": "user", "content": "Weather in Paris? Call the tool."}],
+ })
+ j = json.loads(text) if status == 200 else {}
+ calls = ((j.get("choices") or [{}])[0].get("message") or {}).get("tool_calls") or []
+ rep.check(f"{p}: tools produce tool_calls", bool(calls), json.dumps(j)[:200])
+ if calls:
+ fn = calls[0].get("function") or {}
+ rep.check(f"{p}: tool_call has name", bool(fn.get("name")), json.dumps(calls[0])[:160])
+ ok_args = False
+ try:
+ ok_args = isinstance(json.loads(fn.get("arguments") or ""), dict)
+ except ValueError:
+ pass
+ rep.check(f"{p}: tool_call arguments parse as JSON", ok_args, repr(fn.get("arguments"))[:160])
+
+
+# --------------------------------------------------------------------------
+# OpenAI Responses /v1/responses
+# --------------------------------------------------------------------------
+
+def check_responses(base: str, rep: Report, model: str) -> None:
+ p = "responses"
+
+ status, _, text = post(base, "/v1/responses", {
+ "model": model,
+ "input": [{"role": "user", "content": "Reply with exactly: ok"}],
+ })
+ j = json.loads(text) if status == 200 else {}
+ rep.check(f"{p}: sync 200", status == 200, text[:200])
+ rep.check(f"{p}: sync has output[]", isinstance(j.get("output"), list), json.dumps(j)[:160])
+ rep.check(f"{p}: sync has status", j.get("status") is not None, str(j.get("status")))
+ u = j.get("usage") or {}
+ rep.check(f"{p}: usage has input/output tokens",
+ "input_tokens" in u and "output_tokens" in u, json.dumps(u)[:160])
+
+ status, ctype, text = post(base, "/v1/responses", {
+ "model": model, "stream": True,
+ "input": [{"role": "user", "content": "Count 1 to 3."}],
+ })
+ types = sse_types(text)
+ rep.check(f"{p}: stream 200", status == 200, text[:200])
+ rep.check(f"{p}: stream content-type", "text/event-stream" in ctype, ctype)
+ for want in ("response.created", "response.output_text.delta", "response.completed"):
+ rep.check(f"{p}: stream has {want}", want in types, ",".join(types[:10]))
+ rep.check(f"{p}: stream ends at response.completed",
+ types and types[-1] in ("response.completed", "[DONE]"),
+ ",".join(types[-3:]))
+ deltas = "".join(
+ json.loads(d).get("delta") or ""
+ for n, d in sse_events(text)
+ if d and d != "[DONE]" and json.loads(d).get("type") == "response.output_text.delta"
+ )
+ rep.check(f"{p}: stream produced text", bool(deltas.strip()), repr(deltas[:80]))
+
+
+# --------------------------------------------------------------------------
+# Gemini /v1beta/models/{model}:generateContent
+# --------------------------------------------------------------------------
+
+def check_gemini(base: str, rep: Report, model: str) -> None:
+ p = "gemini"
+ # Enough budget that internal reasoning cannot consume the whole allowance
+ # before any text is emitted — at 60 this turn intermittently finished on
+ # MAX_TOKENS with empty parts, which is a real upstream outcome and not
+ # something the shape checks should depend on.
+ body = {
+ "contents": [{"role": "user", "parts": [{"text": "Reply with exactly: ok"}]}],
+ "generationConfig": {"maxOutputTokens": 512},
+ }
+
+ status, _, text = post(base, f"/v1beta/models/{model}:generateContent", body)
+ j = json.loads(text) if status == 200 else {}
+ rep.check(f"{p}: sync 200", status == 200, text[:200])
+ # The whole point of the Gemini surface: the client must never see the
+ # chat-completions shape the upstream actually returned.
+ rep.check(f"{p}: sync has candidates[]",
+ isinstance(j.get("candidates"), list) and bool(j["candidates"]),
+ json.dumps(j)[:200])
+ rep.check(f"{p}: sync does NOT leak choices[]", "choices" not in j, "leaked OpenAI shape")
+ if j.get("candidates"):
+ cand = j["candidates"][0]
+ parts = (cand.get("content") or {}).get("parts")
+ finish = cand.get("finishReason")
+ rep.check(f"{p}: candidate has content.parts[]", isinstance(parts, list),
+ json.dumps(cand)[:200])
+ # A turn cut off by the token limit owes no content; anything else does.
+ rep.check(f"{p}: candidate carries content unless truncated",
+ bool(parts) or finish == "MAX_TOKENS",
+ json.dumps(cand)[:200])
+ rep.check(f"{p}: candidate has finishReason", finish is not None, str(finish))
+ um = j.get("usageMetadata") or {}
+ rep.check(f"{p}: has usageMetadata with token counts",
+ "promptTokenCount" in um and "candidatesTokenCount" in um, json.dumps(um)[:160])
+
+ status, ctype, text = post(base, f"/v1beta/models/{model}:streamGenerateContent", body)
+ frames = [d for _, d in sse_events(text) if d and d != "[DONE]"]
+ rep.check(f"{p}: stream 200", status == 200, text[:200])
+ rep.check(f"{p}: stream content-type", "text/event-stream" in ctype, ctype)
+ parsed = []
+ for d in frames:
+ try:
+ parsed.append(json.loads(d))
+ except ValueError:
+ pass
+ rep.check(f"{p}: stream frames have candidates[]",
+ bool(parsed) and all("candidates" in d for d in parsed),
+ json.dumps(parsed[0])[:200] if parsed else "no frames")
+ rep.check(f"{p}: stream frames do NOT leak choices[]",
+ all("choices" not in d for d in parsed), "leaked OpenAI shape")
+ said = "".join(
+ p_.get("text") or ""
+ for d in parsed
+ for c in d.get("candidates") or []
+ for p_ in ((c.get("content") or {}).get("parts") or [])
+ )
+ truncated = any(c.get("finishReason") == "MAX_TOKENS"
+ for d in parsed for c in d.get("candidates") or [])
+ rep.check(f"{p}: stream produced text", bool(said.strip()) or truncated, repr(said[:80]))
+
+ status, _, text = post(base, f"/v1beta/models/{model}:countTokens", body)
+ j = json.loads(text) if status == 200 else {}
+ rep.check(f"{p}: countTokens returns totalTokens",
+ status == 200 and isinstance(j.get("totalTokens"), int) and j["totalTokens"] > 0,
+ text[:160])
+
+
+# --------------------------------------------------------------------------
+# Cross-cutting
+# --------------------------------------------------------------------------
+
+def check_errors(base: str, rep: Report) -> None:
+ p = "errors"
+
+ # An unknown model must not come back as a 200 SSE stream that never
+ # produces an event — the failure the client sees has to be the failure
+ # that happened.
+ status, ctype, text = post(base, "/v1/messages", {
+ "model": "definitely-not-a-model", "max_tokens": 10, "stream": True,
+ "messages": [{"role": "user", "content": "hi"}],
+ })
+ rep.check(f"{p}: unknown model on stream is not 200", status != 200, f"{status} {ctype}")
+ rep.check(f"{p}: unknown model returns JSON error, not SSE",
+ "text/event-stream" not in ctype, ctype)
+ try:
+ rep.check(f"{p}: error body has error.message",
+ bool((json.loads(text).get("error") or {}).get("message")), text[:160])
+ except ValueError:
+ rep.check(f"{p}: error body is JSON", False, text[:160])
+
+ # A model that only lives on /responses must say so rather than 500.
+ status, _, text = post(base, "/v1/chat/completions", {
+ "model": "gpt-5.5", "max_tokens": 10,
+ "messages": [{"role": "user", "content": "hi"}],
+ })
+ rep.check(f"{p}: responses-only model rejected with 4xx on chat", 400 <= status < 500, f"{status} {text[:120]}")
+ rep.check(f"{p}: rejection names the right endpoint", "/v1/responses" in text, text[:200])
+
+
+def check_discovery(base: str, rep: Report) -> None:
+ p = "discovery"
+ status, text = get(base, "/v1/models")
+ j = json.loads(text) if status == 200 else {}
+ rep.check(f"{p}: /v1/models 200", status == 200, text[:160])
+ rep.check(f"{p}: models list is OpenAI-shaped",
+ j.get("object") == "list" and isinstance(j.get("data"), list) and bool(j["data"]),
+ json.dumps(j)[:160])
+ rep.check(f"{p}: every model has id and object=model",
+ all(m.get("id") and m.get("object") == "model" for m in j.get("data", [])),
+ "a model entry was malformed")
+
+ status, text = get(base, "/health")
+ j = json.loads(text) if status == 200 else {}
+ rep.check(f"{p}: /health 200", status == 200, text[:160])
+ rep.check(f"{p}: health reports ready + version",
+ j.get("ready") is not None and bool(j.get("version")), json.dumps(j)[:160])
+
+
+def main() -> int:
+ ap = argparse.ArgumentParser()
+ ap.add_argument("--port", type=int, default=8399)
+ ap.add_argument("--host", default="127.0.0.1")
+ ap.add_argument("--anthropic-model", default="claude-haiku-4.5")
+ ap.add_argument("--chat-model", default="gpt-4o-mini")
+ ap.add_argument("--responses-model", default="gpt-5.5")
+ ap.add_argument("--gemini-model", default="gemini-3.5-flash")
+ ap.add_argument("--only", default="", help="comma-separated subset of suites to run")
+ args = ap.parse_args()
+
+ base = f"http://{args.host}:{args.port}"
+ rep = Report()
+
+ suites = {
+ "discovery": lambda: check_discovery(base, rep),
+ "anthropic": lambda: check_anthropic(base, rep, args.anthropic_model),
+ "chat": lambda: check_chat(base, rep, args.chat_model),
+ "responses": lambda: check_responses(base, rep, args.responses_model),
+ "gemini": lambda: check_gemini(base, rep, args.gemini_model),
+ "errors": lambda: check_errors(base, rep),
+ }
+ wanted = [s.strip() for s in args.only.split(",") if s.strip()] or list(suites)
+
+ for name in wanted:
+ fn = suites.get(name)
+ if not fn:
+ print(f"unknown suite: {name}", file=sys.stderr)
+ return 2
+ try:
+ fn()
+ except Exception as e: # a crashed suite is a failed suite, not a crashed run
+ rep.check(f"{name}: suite raised", False, f"{type(e).__name__}: {e}")
+
+ width = max(len(r.name) for r in rep.results)
+ for r in rep.results:
+ mark = "PASS" if r.ok else "FAIL"
+ line = f"[{mark}] {r.name.ljust(width)}"
+ if not r.ok and r.detail:
+ line += f" <- {r.detail}"
+ print(line)
+
+ print(f"\n{len(rep.results) - len(rep.failed)}/{len(rep.results)} passed")
+ return 1 if rep.failed else 0
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/scripts/stats_inventory.py b/scripts/stats_inventory.py
new file mode 100644
index 0000000..12717bd
--- /dev/null
+++ b/scripts/stats_inventory.py
@@ -0,0 +1,123 @@
+"""Inventory the statistics upstream puts in terminal stream events.
+
+Every surface ends a turn with a frame carrying more than the token counts the
+proxy already records — Copilot's own billing units, latency checkpoints,
+reasoning-token splits. This walks the captured debug bodies and reports which
+of those fields actually appear, per endpoint, so what gets surfaced is chosen
+from evidence rather than from guesses about the schema.
+
+Run against a proxy that has served traffic with body capture on.
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import urllib.request
+from collections import defaultdict
+
+
+def frames(raw: str) -> list[dict]:
+ """Every JSON frame in a body, whichever transport carried it."""
+ out = []
+ if raw.lstrip().startswith(("event:", "data:")):
+ for block in raw.split("\n\n"):
+ for line in block.splitlines():
+ if line.startswith("data:"):
+ payload = line[5:].strip()
+ if payload and payload != "[DONE]":
+ try:
+ out.append(json.loads(payload))
+ except ValueError:
+ pass
+ return out
+ lines = [l for l in raw.splitlines() if l.strip()]
+ if len(lines) > 1:
+ for line in lines:
+ try:
+ out.append(json.loads(line))
+ except ValueError:
+ pass
+ return out
+ try:
+ out.append(json.loads(raw))
+ except ValueError:
+ pass
+ return out
+
+
+def leaves(obj, prefix="") -> list[tuple[str, object]]:
+ """Flattens to dotted paths, collapsing list indices so paths aggregate."""
+ found = []
+ if isinstance(obj, dict):
+ for k, v in obj.items():
+ found += leaves(v, f"{prefix}.{k}" if prefix else k)
+ elif isinstance(obj, list):
+ for v in obj[:1]:
+ found += leaves(v, f"{prefix}[]")
+ else:
+ found.append((prefix, obj))
+ return found
+
+
+# Paths the proxy already records. Everything else is a candidate.
+KNOWN = {
+ "usage.input_tokens", "usage.output_tokens",
+ "usage.cache_read_input_tokens", "usage.cache_creation_input_tokens",
+ "usage.prompt_tokens", "usage.completion_tokens", "usage.total_tokens",
+ "response.usage.input_tokens", "response.usage.output_tokens",
+}
+
+# Fragments that mark a value as a statistic worth considering.
+INTERESTING = ("usage", "token", "latency", "ms", "duration", "nano", "aiu",
+ "cost", "batch", "reasoning", "cached", "tier", "fingerprint")
+
+
+def main() -> int:
+ ap = argparse.ArgumentParser()
+ ap.add_argument("--port", type=int, default=8399)
+ ap.add_argument("--host", default="127.0.0.1")
+ ap.add_argument("--per-page", type=int, default=100)
+ args = ap.parse_args()
+
+ url = f"http://{args.host}:{args.port}/api/requests?page=1&per_page={args.per_page}"
+ with urllib.request.urlopen(url, timeout=30) as r:
+ items = json.load(r).get("items") or []
+
+ # path -> endpoint -> (count, sample)
+ seen: dict[str, dict[str, tuple[int, object]]] = defaultdict(lambda: defaultdict(lambda: (0, None)))
+ bodies = 0
+
+ for rec in items:
+ raw = rec.get("response_body")
+ if not raw:
+ continue
+ bodies += 1
+ ep = rec.get("endpoint", "?")
+ for fr in frames(raw):
+ for path, value in leaves(fr):
+ low = path.lower()
+ if not any(f in low for f in INTERESTING):
+ continue
+ if path in KNOWN:
+ continue
+ n, sample = seen[path][ep]
+ seen[path][ep] = (n + 1, value if sample is None else sample)
+
+ if not bodies:
+ print("No captured bodies. Turn on body capture and send some traffic first.")
+ return 1
+
+ print(f"{bodies} captured bodies across {len({i.get('endpoint') for i in items})} endpoints\n")
+ width = max((len(p) for p in seen), default=10)
+ for path in sorted(seen):
+ eps = seen[path]
+ total = sum(n for n, _ in eps.values())
+ sample = next((s for _, s in eps.values() if s is not None), None)
+ where = ",".join(sorted(eps))
+ print(f"{path.ljust(width)} x{total:<5} {str(sample)[:40]:<42} {where}")
+ return 0
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/scripts/ws_check.py b/scripts/ws_check.py
new file mode 100644
index 0000000..f66a8b4
--- /dev/null
+++ b/scripts/ws_check.py
@@ -0,0 +1,221 @@
+"""Conformance sweep for the proxy's `ws:/responses` surface, across every
+model that advertises it.
+
+The catalogue is the source of truth for which models support this transport,
+so the model list is discovered rather than hard-coded — a model added or
+withdrawn upstream changes what gets tested without editing this file.
+
+Each model is checked for the properties that make the transport usable:
+the event vocabulary matches the SSE Responses API, sequence numbers are
+monotonic, the turn terminates, text actually arrives, and usage is reported.
+Models that do *not* advertise the transport are checked for a clean refusal —
+an unsupported model must be told so, not left waiting on a socket.
+
+Requires the proxy running with `websockets` installed:
+
+ python scripts/ws_check.py --port 8399
+"""
+
+from __future__ import annotations
+
+import argparse
+import asyncio
+import json
+import sys
+import urllib.request
+
+import websockets
+
+# Events the upstream emits for a plain text turn. Anything here that never
+# arrives means the transport is not carrying a full Responses stream.
+REQUIRED_EVENTS = ("response.created", "response.output_text.delta", "response.completed")
+
+
+def catalogue(base: str) -> list[dict]:
+ with urllib.request.urlopen(base + "/v1/models/full/", timeout=30) as r:
+ return json.load(r).get("data") or []
+
+
+def ws_models(base: str) -> tuple[list[str], list[str]]:
+ """(models advertising ws:/responses, models that do not)."""
+ yes, no = [], []
+ for m in catalogue(base):
+ eps = m.get("supported_endpoints") or []
+ (yes if "ws:/responses" in eps else no).append(m["id"])
+ return yes, no
+
+
+class Check:
+ def __init__(self) -> None:
+ self.rows: list[tuple[str, bool, str]] = []
+
+ def add(self, name: str, ok: bool, detail: str = "") -> None:
+ self.rows.append((name, ok, detail))
+
+ @property
+ def failed(self) -> list[tuple[str, bool, str]]:
+ return [r for r in self.rows if not r[1]]
+
+
+async def collect(url: str, payload: dict, limit: int, timeout: float) -> list[dict]:
+ """Sends one frame and returns every JSON event received until the turn ends."""
+ events: list[dict] = []
+ async with websockets.connect(url, max_size=None) as ws:
+ await ws.send(json.dumps(payload))
+ while len(events) < limit:
+ try:
+ raw = await asyncio.wait_for(ws.recv(), timeout=timeout)
+ except asyncio.TimeoutError:
+ break
+ except websockets.ConnectionClosed:
+ break
+ text = raw if isinstance(raw, str) else raw.decode("utf-8", "replace")
+ try:
+ ev = json.loads(text)
+ except ValueError:
+ events.append({"type": "__non_json__", "raw": text[:200]})
+ continue
+ events.append(ev)
+ if ev.get("type") in ("response.completed", "response.failed",
+ "response.incomplete", "error"):
+ break
+ return events
+
+
+async def check_model(url: str, model: str, chk: Check, timeout: float) -> None:
+ p = model
+ payload = {
+ "type": "response.create",
+ "model": model,
+ "input": "Reply with exactly: ok",
+ "stream": True,
+ }
+ try:
+ events = await collect(url, payload, limit=400, timeout=timeout)
+ except Exception as e: # noqa: BLE001
+ chk.add(f"{p}: connects", False, f"{type(e).__name__}: {e}")
+ return
+
+ chk.add(f"{p}: connects", True)
+ types = [e.get("type") for e in events]
+
+ err = next((e for e in events if e.get("type") == "error"), None)
+ if err:
+ chk.add(f"{p}: no error frame", False, json.dumps(err.get("error"))[:160])
+ return
+ chk.add(f"{p}: no error frame", True)
+
+ for want in REQUIRED_EVENTS:
+ chk.add(f"{p}: emits {want}", want in types, ",".join(t for t in types[:8] if t))
+
+ chk.add(f"{p}: terminates", bool(types) and types[-1] in
+ ("response.completed", "response.incomplete"), str(types[-1] if types else None))
+
+ seqs = [e["sequence_number"] for e in events if isinstance(e.get("sequence_number"), int)]
+ chk.add(f"{p}: sequence numbers monotonic",
+ seqs == sorted(seqs) and len(set(seqs)) == len(seqs),
+ f"{seqs[:12]}")
+
+ said = "".join(e.get("delta") or "" for e in events
+ if e.get("type") == "response.output_text.delta")
+ chk.add(f"{p}: produced text", bool(said.strip()), repr(said[:60]))
+
+ final = next((e for e in reversed(events)
+ if e.get("type") in ("response.completed", "response.incomplete")), None)
+ usage = ((final or {}).get("response") or {}).get("usage") or {}
+ chk.add(f"{p}: reports usage",
+ isinstance(usage.get("input_tokens"), int) and isinstance(usage.get("output_tokens"), int),
+ json.dumps(usage)[:120])
+ # The model that answered should be the one asked for, or the mapping target.
+ served = ((final or {}).get("response") or {}).get("model")
+ chk.add(f"{p}: echoes a model", bool(served), str(served))
+
+
+async def check_refusal(url: str, model: str, chk: Check, timeout: float) -> None:
+ """A model without the transport must be refused, promptly and in the
+ upstream's own error shape — not left hanging."""
+ p = f"{model} (unsupported)"
+ payload = {"type": "response.create", "model": model, "input": "hi", "stream": True}
+ try:
+ events = await collect(url, payload, limit=10, timeout=timeout)
+ except Exception as e: # noqa: BLE001
+ chk.add(f"{p}: refused cleanly", False, f"{type(e).__name__}: {e}")
+ return
+ err = next((e for e in events if e.get("type") == "error"), None)
+ chk.add(f"{p}: refused with an error frame", err is not None,
+ ",".join(str(e.get("type")) for e in events[:4]))
+ if err:
+ msg = (err.get("error") or {}).get("message", "")
+ chk.add(f"{p}: refusal names the alternative", "/v1/responses" in msg, msg[:160])
+
+
+async def check_bad_frames(url: str, chk: Check, timeout: float) -> None:
+ """Malformed input must produce an error frame rather than a hung socket."""
+ cases = [
+ ("not JSON", "this is not json"),
+ ("missing type", json.dumps({"model": "gpt-5.5", "input": "hi"})),
+ ("wrong type", json.dumps({"type": "nonsense", "model": "gpt-5.5", "input": "hi"})),
+ ]
+ for label, frame in cases:
+ try:
+ async with websockets.connect(url, max_size=None) as ws:
+ await ws.send(frame)
+ raw = await asyncio.wait_for(ws.recv(), timeout=timeout)
+ ev = json.loads(raw if isinstance(raw, str) else raw.decode())
+ chk.add(f"bad frame ({label}) -> error", ev.get("type") == "error",
+ json.dumps(ev)[:160])
+ except asyncio.TimeoutError:
+ chk.add(f"bad frame ({label}) -> error", False, "socket hung, no reply")
+ except Exception as e: # noqa: BLE001
+ chk.add(f"bad frame ({label}) -> error", False, f"{type(e).__name__}: {e}")
+
+
+async def main_async(args: argparse.Namespace) -> int:
+ base = f"http://{args.host}:{args.port}"
+ url = f"ws://{args.host}:{args.port}{args.path}"
+
+ supported, unsupported = ws_models(base)
+ if args.models:
+ supported = [m.strip() for m in args.models.split(",") if m.strip()]
+ print(f"catalogue advertises ws:/responses for {len(supported)} model(s):")
+ for m in supported:
+ print(f" {m}")
+ print()
+
+ chk = Check()
+ await check_bad_frames(url, chk, args.timeout)
+ for m in supported:
+ await check_model(url, m, chk, args.timeout)
+ await asyncio.sleep(args.delay)
+ if unsupported and not args.skip_refusal:
+ await check_refusal(url, unsupported[0], chk, args.timeout)
+
+ width = max(len(r[0]) for r in chk.rows)
+ for name, ok, detail in chk.rows:
+ line = f"[{'PASS' if ok else 'FAIL'}] {name.ljust(width)}"
+ if not ok and detail:
+ line += f" <- {detail}"
+ print(line)
+ print(f"\n{len(chk.rows) - len(chk.failed)}/{len(chk.rows)} passed")
+ return 1 if chk.failed else 0
+
+
+def main() -> int:
+ ap = argparse.ArgumentParser()
+ ap.add_argument("--host", default="127.0.0.1")
+ ap.add_argument("--port", type=int, default=8399)
+ ap.add_argument("--path", default="/v1/responses")
+ ap.add_argument("--models", default="", help="override the discovered model list")
+ ap.add_argument("--timeout", type=float, default=90.0)
+ ap.add_argument("--delay", type=float, default=1.0,
+ help="pause between models; bursts of upstream traffic get rate limited")
+ ap.add_argument("--skip-refusal", action="store_true")
+ args = ap.parse_args()
+ try:
+ return asyncio.run(main_async(args))
+ except KeyboardInterrupt:
+ return 130
+
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/scripts/ws_explore.py b/scripts/ws_explore.py
new file mode 100644
index 0000000..f70a418
--- /dev/null
+++ b/scripts/ws_explore.py
@@ -0,0 +1,154 @@
+"""Learn what the Copilot `ws:/responses` surface speaks.
+
+The catalogue advertises `ws:/responses` for several models and the endpoint
+answers `101 Switching Protocols`, but there is no documentation for it. This
+opens one connection, sends one Responses-API request, and prints every frame
+that comes back, so the framing can be read off the wire instead of guessed.
+
+Nothing here prints the Copilot token.
+"""
+
+from __future__ import annotations
+
+import argparse
+import asyncio
+import json
+import os
+import pathlib
+import urllib.request
+
+import websockets
+
+UA = "GithubCopilot/1.155.0"
+EDITOR = "vscode/1.104.0"
+
+
+def github_token() -> str:
+ env = os.environ.get("GH_TOKEN") or os.environ.get("GITHUB_TOKEN")
+ if env:
+ return env.strip()
+ p = pathlib.Path.home() / "AppData/Roaming/ghc-tunnel/github_token.txt"
+ if not p.exists():
+ p = pathlib.Path.home() / ".config/ghc-tunnel/github_token.txt"
+ return p.read_text().strip()
+
+
+def copilot_token() -> tuple[str, str]:
+ req = urllib.request.Request(
+ "https://api.github.com/copilot_internal/v2/token",
+ headers={
+ "Authorization": "token " + github_token(),
+ "Accept": "application/json",
+ "Editor-Version": EDITOR,
+ "User-Agent": UA,
+ },
+ )
+ d = json.load(urllib.request.urlopen(req, timeout=30))
+ api = (d.get("endpoints") or {}).get("api") or "https://api.githubcopilot.com"
+ return d["token"], api
+
+
+def summarize(frame: str, full: bool) -> str:
+ try:
+ j = json.loads(frame)
+ except ValueError:
+ return f"(non-JSON, {len(frame)}B) {frame[:200]!r}"
+ if full:
+ return json.dumps(j, indent=2)[:2000]
+ t = j.get("type") or "?"
+ bits = [f"type={t}"]
+ for k in ("delta", "status", "sequence_number", "output_index", "content_index", "error", "code", "message"):
+ if k in j and not isinstance(j[k], (dict, list)):
+ v = str(j[k])
+ bits.append(f"{k}={v[:80]!r}" if k == "delta" else f"{k}={v[:80]}")
+ if "response" in j and isinstance(j["response"], dict):
+ r = j["response"]
+ bits.append(f"response.status={r.get('status')}")
+ if r.get("usage"):
+ bits.append(f"usage={json.dumps(r['usage'])[:120]}")
+ if not isinstance(j.get("type"), str):
+ bits.append(f"keys={sorted(j.keys())[:10]}")
+ return " ".join(bits)
+
+
+async def run(model: str, envelope: str, full: bool, path: str, limit: int,
+ raw: str | None, query: str, init: str | None) -> None:
+ token, api = copilot_token()
+ host = api.split("://", 1)[-1].rstrip("/")
+ url = f"wss://{host}{path}" + (f"?{query}" if query else "")
+
+ headers = {
+ "Authorization": f"Bearer {token}",
+ "User-Agent": UA,
+ "Editor-Version": EDITOR,
+ "Editor-Plugin-Version": "copilot-chat/0.26.7",
+ "Copilot-Integration-Id": "vscode-chat",
+ "Openai-Intent": "conversation-edits",
+ }
+
+ body = {
+ "model": model,
+ "stream": True,
+ "input": [{"role": "user", "content": "Reply with exactly: ok"}],
+ }
+ payloads = {
+ "bare": body,
+ "typed": {"type": "response.create", "response": body},
+ "wrapped": {"type": "request", "payload": body},
+ "topmodel": {"type": "response.create", "model": model, "response": body},
+ }
+ payload = json.loads(raw) if raw else payloads[envelope]
+
+ print(f"connect {url}")
+ if init:
+ print(f"init {init[:200]}")
+ print(f"send {json.dumps(payload)[:300]}\n")
+
+ async with websockets.connect(url, additional_headers=headers, max_size=None) as ws:
+ if init:
+ await ws.send(init)
+ await ws.send(json.dumps(payload))
+ n = 0
+ try:
+ while n < limit:
+ frame = await asyncio.wait_for(ws.recv(), timeout=60)
+ n += 1
+ text = frame if isinstance(frame, str) else frame.decode("utf-8", "replace")
+ print(f"[{n:>3}] {summarize(text, full)}")
+ try:
+ if json.loads(text).get("type") in ("response.completed", "response.failed",
+ "response.incomplete", "error"):
+ break
+ except ValueError:
+ pass
+ except asyncio.TimeoutError:
+ print("(timed out waiting for the next frame)")
+ except websockets.ConnectionClosed as e:
+ print(f"(closed: code={e.code} reason={e.reason!r})")
+ print(f"\n{n} frames")
+
+
+def main() -> int:
+ ap = argparse.ArgumentParser()
+ ap.add_argument("--model", default="gpt-5.5")
+ ap.add_argument("--envelope", default="bare",
+ choices=("bare", "typed", "wrapped", "topmodel"))
+ ap.add_argument("--path", default="/responses")
+ ap.add_argument("--query", default="", help="URL query string, e.g. model=gpt-5.5")
+ ap.add_argument("--raw", default=None, help="send this exact JSON instead of a preset")
+ ap.add_argument("--raw-file", default=None,
+ help="send the JSON in this file (avoids shell quoting)")
+ ap.add_argument("--init", default=None, help="send this JSON first, before the payload")
+ ap.add_argument("--full", action="store_true", help="print whole frames, not a summary")
+ ap.add_argument("--limit", type=int, default=40)
+ args = ap.parse_args()
+ raw = args.raw
+ if args.raw_file:
+ raw = pathlib.Path(args.raw_file).read_text(encoding="utf-8")
+ asyncio.run(run(args.model, args.envelope, args.full, args.path, args.limit,
+ raw, args.query, args.init))
+ return 0
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/scripts/ws_probe.py b/scripts/ws_probe.py
new file mode 100644
index 0000000..6da5ef4
--- /dev/null
+++ b/scripts/ws_probe.py
@@ -0,0 +1,127 @@
+"""Probe whether the Copilot API accepts a WebSocket upgrade on /responses.
+
+The model catalogue advertises `ws:/responses` for several models, but GitHub
+publishes no documentation for the inference API at all, and the token
+response's `endpoints` object carries no websocket URL. So the only way to
+learn whether this surface exists — and what it speaks — is to ask it.
+
+This performs a raw RFC 6455 handshake and reports what came back. It sends no
+payload and holds no connection open; it is a question, not a client.
+
+Nothing here prints the Copilot token.
+"""
+
+from __future__ import annotations
+
+import argparse
+import base64
+import json
+import os
+import pathlib
+import socket
+import ssl
+import sys
+import urllib.request
+
+UA = "GithubCopilot/1.155.0"
+EDITOR = "vscode/1.104.0"
+
+
+def github_token() -> str:
+ env = os.environ.get("GH_TOKEN") or os.environ.get("GITHUB_TOKEN")
+ if env:
+ return env.strip()
+ p = pathlib.Path.home() / "AppData/Roaming/ghc-tunnel/github_token.txt"
+ if not p.exists():
+ p = pathlib.Path.home() / ".config/ghc-tunnel/github_token.txt"
+ return p.read_text().strip()
+
+
+def copilot_token() -> tuple[str, dict]:
+ req = urllib.request.Request(
+ "https://api.github.com/copilot_internal/v2/token",
+ headers={
+ "Authorization": "token " + github_token(),
+ "Accept": "application/json",
+ "Editor-Version": EDITOR,
+ "User-Agent": UA,
+ },
+ )
+ d = json.load(urllib.request.urlopen(req, timeout=30))
+ return d["token"], d.get("endpoints") or {}
+
+
+def handshake(host: str, path: str, token: str, extra: dict[str, str]) -> tuple[str, dict[str, str], bytes]:
+ """Raw RFC 6455 upgrade. Returns (status line, response headers, trailing body)."""
+ key = base64.b64encode(os.urandom(16)).decode()
+ headers = {
+ "Host": host,
+ "Upgrade": "websocket",
+ "Connection": "Upgrade",
+ "Sec-WebSocket-Key": key,
+ "Sec-WebSocket-Version": "13",
+ "Authorization": f"Bearer {token}",
+ "User-Agent": UA,
+ "Editor-Version": EDITOR,
+ "Editor-Plugin-Version": "copilot-chat/0.26.7",
+ "Copilot-Integration-Id": "vscode-chat",
+ "Origin": f"https://{host}",
+ **extra,
+ }
+ raw = f"GET {path} HTTP/1.1\r\n" + "".join(f"{k}: {v}\r\n" for k, v in headers.items()) + "\r\n"
+
+ ctx = ssl.create_default_context()
+ with socket.create_connection((host, 443), timeout=20) as sock:
+ with ctx.wrap_socket(sock, server_hostname=host) as tls:
+ tls.sendall(raw.encode())
+ buf = b""
+ while b"\r\n\r\n" not in buf and len(buf) < 65536:
+ chunk = tls.recv(4096)
+ if not chunk:
+ break
+ buf += chunk
+
+ head, _, rest = buf.partition(b"\r\n\r\n")
+ lines = head.decode("utf-8", "replace").split("\r\n")
+ status = lines[0] if lines else "(no response)"
+ resp_headers = {}
+ for line in lines[1:]:
+ k, _, v = line.partition(":")
+ if k:
+ resp_headers[k.strip().lower()] = v.strip()
+ return status, resp_headers, rest
+
+
+def main() -> int:
+ ap = argparse.ArgumentParser()
+ ap.add_argument("--paths", default="/responses,/v1/responses,/ws/responses,/realtime",
+ help="comma-separated paths to probe")
+ args = ap.parse_args()
+
+ token, endpoints = copilot_token()
+ api = endpoints.get("api") or "https://api.githubcopilot.com"
+ host = api.split("://", 1)[-1].rstrip("/")
+ print(f"api host: {host}\n")
+
+ interesting = ("upgrade", "connection", "sec-websocket-accept", "sec-websocket-protocol",
+ "content-type", "x-request-id", "allow")
+
+ for path in [p.strip() for p in args.paths.split(",") if p.strip()]:
+ try:
+ status, hdrs, body = handshake(host, path, token, {})
+ except Exception as e: # noqa: BLE001 - a probe reports failures, it does not raise
+ print(f"{path:<20} ERROR {type(e).__name__}: {e}")
+ continue
+ shown = {k: v for k, v in hdrs.items() if k in interesting}
+ print(f"{path:<20} {status}")
+ for k, v in shown.items():
+ print(f"{'':<20} {k}: {v}")
+ if body[:400].strip():
+ print(f"{'':<20} body: {body[:300].decode('utf-8', 'replace')!r}")
+ print()
+
+ return 0
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/src/config.rs b/src/config.rs
index dce1c04..db55cfa 100644
--- a/src/config.rs
+++ b/src/config.rs
@@ -20,10 +20,17 @@ pub const API_VERSION: &str = "2025-05-01";
pub const COPILOT_VERSION: &str = "0.48.1";
/// Config schema version used to detect when defaults/options changed and a
/// persisted config should be rewritten with migrated values.
-pub const CONFIG_VERSION: u32 = 2;
+///
+/// Bumped to 4 for the Opus 5 / Sonnet 5 model mappings, so existing files gain
+/// the new aliases and have superseded targets lifted on the next start.
+pub const CONFIG_VERSION: u32 = 4;
/// Default model name that Claude "opus"/"sonnet" requests are mapped to.
-pub const DEFAULT_OPUS: &str = "claude-opus-4.8";
+///
+/// The catalog carries `claude-opus-4.6` through `claude-opus-5`; this is the
+/// newest generally-available one, and it matches 4.8 on every published
+/// capability -- 1M context, 64k output, billing multiplier 1, vision.
+pub const DEFAULT_OPUS: &str = "claude-opus-5";
/// Default model name that Claude "haiku" requests are mapped to.
pub const DEFAULT_HAIKU: &str = "claude-haiku-4.5";
@@ -135,6 +142,16 @@ pub struct Config {
pub tool_result_suffix_remove: Vec,
#[serde(default = "default_max_retries")]
pub max_connection_retries: u32,
+ /// Maximum seconds of silence allowed between two reads from an upstream
+ /// response before the request is treated as dead. This bounds *silence*,
+ /// not total duration, so long streaming answers are unaffected. `0`
+ /// disables the timeout.
+ ///
+ /// Without it a half-open connection never errors and never ends: the
+ /// request hangs forever, the client hangs with it, and the stream-
+ /// interrupted handling never runs because no error is ever observed.
+ #[serde(default = "default_read_timeout")]
+ pub upstream_read_timeout_seconds: u64,
/// When true, never route to the upstream `/v1/messages` endpoint; always
/// translate Anthropic requests through the OpenAI chat completions API.
#[serde(default)]
@@ -147,9 +164,14 @@ pub struct Config {
/// the `Editor-Version` header (falling back to `vscode_version`).
#[serde(default)]
pub dynamic_vscode_version: bool,
- /// When true, check GitHub releases and auto-upgrade this binary when a
- /// newer version is available.
- #[serde(default)]
+ /// Check GitHub releases on startup and replace this binary when a newer
+ /// version is available. Enabled by default, including for config files
+ /// written before the key existed; disable with `auto_upgrade: false`,
+ /// `--no-auto-upgrade`, or `GHC_PROXY_AUTO_UPGRADE=0`.
+ ///
+ /// The replacement takes effect on the next start; the running process
+ /// keeps serving the old code until then.
+ #[serde(default = "default_true")]
pub auto_upgrade: bool,
/// Minimum number of seconds between successive proxied requests. `None`
/// disables rate limiting.
@@ -197,6 +219,19 @@ fn default_copilot_version() -> String {
fn default_max_retries() -> u32 {
3
}
+/// Deliberately far above the longest silence a healthy upstream produces.
+///
+/// An earlier 120s default was wrong: it was reasoned from the ~60s idle window
+/// the upstream load balancer enforces, but measurement against the real API
+/// showed Copilot buffers `input_json_delta` until a tool call's argument JSON
+/// is complete and then flushes it in one burst. A 35,899-token answer went
+/// **329.5 seconds** without emitting a byte, and the silence scales with
+/// output size at roughly 9.5ms/token — so 120s would have aborted perfectly
+/// healthy large tool calls. 15 minutes clears the worst measured case with
+/// room to spare while still bounding a genuinely dead connection.
+fn default_read_timeout() -> u64 {
+ 900
+}
impl Default for Config {
fn default() -> Self {
@@ -215,10 +250,11 @@ impl Default for Config {
system_prompt_add: Vec::new(),
tool_result_suffix_remove: Vec::new(),
max_connection_retries: default_max_retries(),
+ upstream_read_timeout_seconds: default_read_timeout(),
redirect_anthropic: false,
show_token: false,
dynamic_vscode_version: false,
- auto_upgrade: false,
+ auto_upgrade: true,
rate_limit_seconds: None,
rate_limit_wait: false,
manual_approve: false,
@@ -275,11 +311,17 @@ pub fn default_model_mappings() -> ModelMappings {
let opus = DEFAULT_OPUS.to_string();
let haiku = DEFAULT_HAIKU.to_string();
let mut exact = BTreeMap::new();
- for k in ["opus", "sonnet", "opus4-7", "opus4-8", "4-7[1m]", "4-8[1m]"] {
+ for k in [
+ "opus", "sonnet", "opus4-7", "opus4-8", "opus5", "4-7[1m]", "4-8[1m]", "5[1m]",
+ ] {
exact.insert(k.to_string(), opus.clone());
}
exact.insert("haiku".to_string(), haiku.clone());
+ // Every spelling of a Claude model resolves to the current best one. The
+ // list is exhaustive rather than pattern-based because a request naming a
+ // model that has since been superseded should still be served, and because
+ // Anthropic writes the same version two ways (`4.8` and `4-8`).
let mut prefix = BTreeMap::new();
for k in [
"claude-sonnet-4-",
@@ -287,6 +329,7 @@ pub fn default_model_mappings() -> ModelMappings {
"claude-opus-4.6-",
"claude-opus-4.7-",
"claude-opus-4.8-",
+ "claude-opus-5-",
"claude-opus-4-5-",
"claude-opus-4-6-",
"claude-opus-4-7-",
@@ -295,16 +338,21 @@ pub fn default_model_mappings() -> ModelMappings {
"claude-opus-4.6",
"claude-opus-4.7",
"claude-opus-4.8",
+ "claude-opus-5",
"claude-opus-4-6",
"claude-opus-4-7",
"claude-opus-4-8",
"claude-opus-4-6[1m]",
"claude-opus-4-7[1m]",
"claude-opus-4-8[1m]",
+ "claude-opus-5[1m]",
"claude-sonnet-4-7",
"claude-sonnet-4-8",
"claude-sonnet-4-6",
"claude-sonnet-4-5",
+ "claude-sonnet-4.6",
+ "claude-sonnet-5-",
+ "claude-sonnet-5",
] {
prefix.insert(k.to_string(), opus.clone());
}
@@ -385,6 +433,8 @@ pub fn render_config_yaml(cfg: &Config) -> String {
let _ = writeln!(s, "vscode_version: \"{}\"", cfg.vscode_version);
let _ = writeln!(s, "api_version: \"{}\"", cfg.api_version);
let _ = writeln!(s, "copilot_version: \"{}\"", cfg.copilot_version);
+ s.push_str("# Check GitHub releases on startup and replace this binary when a newer\n");
+ s.push_str("# version is available. Takes effect on the next start.\n");
let _ = writeln!(s, "auto_upgrade: {}", cfg.auto_upgrade);
s.push('\n');
s.push_str("# Model Name Mappings\n");
@@ -447,6 +497,14 @@ pub fn render_config_yaml(cfg: &Config) -> String {
s.push_str("# Retry Settings\n");
s.push_str("# Max retries for upstream connection errors (0 = no retries)\n");
let _ = writeln!(s, "max_connection_retries: {}", cfg.max_connection_retries);
+ s.push_str("# Max seconds of silence from an upstream response before it is treated as\n");
+ s.push_str("# dead. Bounds silence, not total duration, so long streams are fine.\n");
+ s.push_str("# 0 disables the timeout.\n");
+ let _ = writeln!(
+ s,
+ "upstream_read_timeout_seconds: {}",
+ cfg.upstream_read_timeout_seconds
+ );
if cfg.redirect_anthropic {
s.push('\n');
s.push_str(
@@ -646,6 +704,21 @@ pub fn load_config_with_options(write_back_on_migration: bool) -> Config {
);
}
}
+ if let Ok(val) = std::env::var("GHC_PROXY_UPSTREAM_READ_TIMEOUT") {
+ match val.parse::() {
+ Ok(secs) => {
+ cfg.upstream_read_timeout_seconds = secs;
+ tracing::info!(
+ "✓ Overriding upstream_read_timeout_seconds from GHC_PROXY_UPSTREAM_READ_TIMEOUT: {}",
+ secs
+ );
+ }
+ Err(_) => tracing::warn!(
+ "Invalid GHC_PROXY_UPSTREAM_READ_TIMEOUT value '{}': expected a number",
+ val
+ ),
+ }
+ }
if let Ok(val) = std::env::var("GHC_PROXY_REDIRECT_ANTHROPIC") {
cfg.redirect_anthropic = val.eq_ignore_ascii_case("true") || val == "1";
tracing::info!(
@@ -781,6 +854,39 @@ fn migrate_config(cfg: &mut Config) -> bool {
}
}
+ if cfg.config_version < 4 {
+ // Opus 5 and Sonnet 5 entered the catalog. Add their aliases so the new
+ // names resolve, and change nothing else.
+ //
+ // Deliberately no "lift stale defaults to the new one" pass. A value
+ // that equals the previous built-in default is indistinguishable from a
+ // version the user pinned on purpose -- and pinning is common: a real
+ // config in the wild carried `claude-opus-4-7: claude-opus-4.7` beside
+ // `haiku: claude-opus-4.7`, both of which such a pass would silently
+ // rewrite. Existing mappings keep pointing where they were told to;
+ // `--setup` or `--default` is how a user asks for the new defaults.
+ let opus = DEFAULT_OPUS.to_string();
+ for k in ["opus5", "5[1m]"] {
+ cfg.model_mappings
+ .exact
+ .entry(k.to_string())
+ .or_insert_with(|| opus.clone());
+ }
+ for k in [
+ "claude-opus-5-",
+ "claude-opus-5",
+ "claude-opus-5[1m]",
+ "claude-sonnet-5-",
+ "claude-sonnet-5",
+ "claude-sonnet-4.6",
+ ] {
+ cfg.model_mappings
+ .prefix
+ .entry(k.to_string())
+ .or_insert_with(|| opus.clone());
+ }
+ }
+
// Any older schema version is lifted to the current one; the caller
// persists the re-rendered document so newly introduced properties appear
// on disk with their defaults.
@@ -792,6 +898,54 @@ fn migrate_config(cfg: &mut Config) -> bool {
mod tests {
use super::*;
+ #[test]
+ fn auto_upgrade_is_on_by_default_including_for_legacy_configs() {
+ assert!(Config::default().auto_upgrade);
+
+ // Config files written before the key existed must also opt in, which
+ // is what `#[serde(default)]` would have got wrong.
+ let legacy = "address: 127.0.0.1\nport: 8314\nmax_connection_retries: 3\n";
+ let cfg: Config = serde_norway::from_str(legacy).unwrap();
+ assert!(cfg.auto_upgrade);
+
+ // An explicit opt-out is still honoured, and survives a round trip
+ // through the generated config file.
+ let off: Config = serde_norway::from_str("auto_upgrade: false\n").unwrap();
+ assert!(!off.auto_upgrade);
+ let rendered: Config = serde_norway::from_str(&render_config_yaml(&off)).unwrap();
+ assert!(!rendered.auto_upgrade);
+ }
+
+ #[test]
+ fn read_timeout_defaults_are_sane() {
+ // Silence, not total duration — a long stream must never be cut off by
+ // this, but a genuinely dead connection must be.
+ let cfg = Config::default();
+ assert_eq!(cfg.upstream_read_timeout_seconds, 900);
+ // The longest silence measured against the real upstream was 329.5s,
+ // while a tool call's argument JSON was buffered. The default must
+ // clear that with margin or it aborts healthy requests.
+ assert!(cfg.upstream_read_timeout_seconds > 330);
+ // The rendered config round-trips the value, including the 0 (disabled)
+ // case, so operators can turn it off.
+ let off = Config {
+ upstream_read_timeout_seconds: 0,
+ ..Default::default()
+ };
+ let parsed: Config = serde_norway::from_str(&render_config_yaml(&off)).unwrap();
+ assert_eq!(parsed.upstream_read_timeout_seconds, 0);
+ let parsed: Config = serde_norway::from_str(&render_config_yaml(&cfg)).unwrap();
+ assert_eq!(parsed.upstream_read_timeout_seconds, 900);
+ }
+
+ #[test]
+ fn read_timeout_missing_from_legacy_config_uses_default() {
+ // Existing config files predate the key and must keep working.
+ let legacy = "address: 127.0.0.1\nport: 8314\nmax_connection_retries: 3\n";
+ let cfg: Config = serde_norway::from_str(legacy).unwrap();
+ assert_eq!(cfg.upstream_read_timeout_seconds, 900);
+ }
+
#[test]
fn github_models_defaults_enabled() {
let gm = GithubModels::default();
@@ -944,4 +1098,70 @@ mod tests {
Some("my-model")
);
}
+
+ #[test]
+ fn opus_5_migration_adds_aliases_without_touching_existing_ones() {
+ let yaml = "config_version: 3\n";
+ let mut cfg: Config = serde_norway::from_str(yaml).expect("v3 config parses");
+ // A hand-tuned file: version-specific pins, and a tier alias pointed
+ // somewhere the defaults would never put it. Both shapes were observed
+ // in a real config, and a migration that "lifts stale defaults" rewrites
+ // both, because a pin and a stale default look identical.
+ cfg.model_mappings
+ .exact
+ .insert("opus".to_string(), "claude-opus-4.8".to_string());
+ cfg.model_mappings
+ .exact
+ .insert("haiku".to_string(), "claude-opus-4.7".to_string());
+ cfg.model_mappings
+ .prefix
+ .insert("claude-opus-4-7".to_string(), "claude-opus-4.7".to_string());
+ cfg.model_mappings.prefix.insert(
+ "claude-sonnet-4.6".to_string(),
+ "pinned-by-hand".to_string(),
+ );
+
+ assert!(migrate_config(&mut cfg));
+ assert_eq!(cfg.config_version, CONFIG_VERSION);
+
+ // Nothing already in the file is rewritten, including a key this
+ // migration would otherwise seed.
+ for (k, want) in [("opus", "claude-opus-4.8"), ("haiku", "claude-opus-4.7")] {
+ assert_eq!(
+ cfg.model_mappings.exact.get(k).map(String::as_str),
+ Some(want),
+ "exact alias {k} must not be rewritten"
+ );
+ }
+ for (k, want) in [
+ ("claude-opus-4-7", "claude-opus-4.7"),
+ ("claude-sonnet-4.6", "pinned-by-hand"),
+ ] {
+ assert_eq!(
+ cfg.model_mappings.prefix.get(k).map(String::as_str),
+ Some(want),
+ "prefix {k} must not be rewritten"
+ );
+ }
+
+ // Names that were not in the file are seeded with the new default.
+ assert_eq!(
+ cfg.model_mappings.exact.get("opus5").map(String::as_str),
+ Some(DEFAULT_OPUS)
+ );
+ assert_eq!(
+ cfg.model_mappings
+ .prefix
+ .get("claude-opus-5")
+ .map(String::as_str),
+ Some(DEFAULT_OPUS)
+ );
+ assert_eq!(
+ cfg.model_mappings
+ .prefix
+ .get("claude-sonnet-5")
+ .map(String::as_str),
+ Some(DEFAULT_OPUS)
+ );
+ }
}
diff --git a/src/gemini.rs b/src/gemini.rs
index 08e586a..2d1953c 100644
--- a/src/gemini.rs
+++ b/src/gemini.rs
@@ -162,6 +162,11 @@ pub fn gemini_to_openai(req: &Value, model: &str, stream: bool) -> Value {
out.insert("stream".into(), Value::Bool(stream));
// generationConfig -> top-level OpenAI sampling params.
+ //
+ // Only `topK` and `safetySettings` have no counterpart in the chat
+ // completions schema; everything else here maps exactly, and dropping it
+ // silently meant a client's `seed` or `candidateCount` had no effect and no
+ // explanation.
if let Some(gc) = req
.get("generationConfig")
.or_else(|| req.get("generation_config"))
@@ -181,6 +186,49 @@ pub fn gemini_to_openai(req: &Value, model: &str, stream: bool) -> Value {
if let Some(v) = gc.get("stopSequences").or_else(|| gc.get("stop_sequences")) {
out.insert("stop".into(), v.clone());
}
+ if let Some(v) = gc
+ .get("candidateCount")
+ .or_else(|| gc.get("candidate_count"))
+ {
+ out.insert("n".into(), v.clone());
+ }
+ if let Some(v) = gc.get("seed") {
+ out.insert("seed".into(), v.clone());
+ }
+ if let Some(v) = gc
+ .get("presencePenalty")
+ .or_else(|| gc.get("presence_penalty"))
+ {
+ out.insert("presence_penalty".into(), v.clone());
+ }
+ if let Some(v) = gc
+ .get("frequencyPenalty")
+ .or_else(|| gc.get("frequency_penalty"))
+ {
+ out.insert("frequency_penalty".into(), v.clone());
+ }
+ // Gemini asks for JSON with a mime type; chat completions with a
+ // response_format object. A schema, when supplied, carries over as a
+ // json_schema format.
+ let mime = gc
+ .get("responseMimeType")
+ .or_else(|| gc.get("response_mime_type"))
+ .and_then(Value::as_str);
+ if mime == Some("application/json") {
+ let schema = gc
+ .get("responseSchema")
+ .or_else(|| gc.get("response_schema"));
+ out.insert(
+ "response_format".into(),
+ match schema {
+ Some(s) => json!({
+ "type": "json_schema",
+ "json_schema": {"name": "response", "schema": s}
+ }),
+ None => json!({"type": "json_object"}),
+ },
+ );
+ }
}
// tools.functionDeclarations -> OpenAI function tools.
@@ -349,6 +397,92 @@ mod tests {
assert_eq!(out["top_p"], 0.9);
}
+ /// Everything in `generationConfig` with an exact chat-completions
+ /// counterpart has to carry over. These used to be dropped silently, so a
+ /// client's `seed` had no effect and nothing said why.
+ #[test]
+ fn generation_config_maps_the_rest() {
+ let req = json!({
+ "contents": [{"role": "user", "parts": [{"text": "hi"}]}],
+ "generationConfig": {
+ "candidateCount": 2,
+ "seed": 42,
+ "presencePenalty": 0.3,
+ "frequencyPenalty": 0.4,
+ "stopSequences": ["END"]
+ }
+ });
+ let out = gemini_to_openai(&req, "m", false);
+ assert_eq!(out["n"], 2);
+ assert_eq!(out["seed"], 42);
+ assert_eq!(out["presence_penalty"], 0.3);
+ assert_eq!(out["frequency_penalty"], 0.4);
+ assert_eq!(out["stop"], json!(["END"]));
+ }
+
+ #[test]
+ fn response_mime_type_becomes_response_format() {
+ let plain = gemini_to_openai(
+ &json!({
+ "contents": [{"role": "user", "parts": [{"text": "hi"}]}],
+ "generationConfig": {"responseMimeType": "application/json"}
+ }),
+ "m",
+ false,
+ );
+ assert_eq!(plain["response_format"], json!({"type": "json_object"}));
+
+ let schema = json!({"type": "object", "properties": {"a": {"type": "string"}}});
+ let with_schema = gemini_to_openai(
+ &json!({
+ "contents": [{"role": "user", "parts": [{"text": "hi"}]}],
+ "generationConfig": {
+ "responseMimeType": "application/json",
+ "responseSchema": schema
+ }
+ }),
+ "m",
+ false,
+ );
+ assert_eq!(with_schema["response_format"]["type"], "json_schema");
+ assert_eq!(
+ with_schema["response_format"]["json_schema"]["schema"],
+ schema
+ );
+
+ // A plain-text turn must not acquire a response_format it never asked for.
+ let text = gemini_to_openai(
+ &json!({
+ "contents": [{"role": "user", "parts": [{"text": "hi"}]}],
+ "generationConfig": {"responseMimeType": "text/plain"}
+ }),
+ "m",
+ false,
+ );
+ assert!(text.get("response_format").is_none());
+ }
+
+ /// `topK` and `safetySettings` have no counterpart in the chat completions
+ /// schema. Asserting they stay out keeps the dashboard's "dropped in
+ /// translation" note honest.
+ #[test]
+ fn params_without_counterparts_are_not_invented() {
+ let out = gemini_to_openai(
+ &json!({
+ "contents": [{"role": "user", "parts": [{"text": "hi"}]}],
+ "generationConfig": {"topK": 32},
+ "safetySettings": [{"category": "HARM_CATEGORY_HARASSMENT",
+ "threshold": "BLOCK_NONE"}]
+ }),
+ "m",
+ false,
+ );
+ assert!(out.get("top_k").is_none());
+ assert!(out.get("topK").is_none());
+ assert!(out.get("safetySettings").is_none());
+ assert!(out.get("safety_settings").is_none());
+ }
+
#[test]
fn function_declarations_become_tools() {
let req = json!({
diff --git a/src/main.rs b/src/main.rs
index 259a1e5..965d537 100644
--- a/src/main.rs
+++ b/src/main.rs
@@ -124,6 +124,7 @@ Options:
--fetch-version Fetch the latest VS Code version at startup
--no-fetch-version Disable dynamic VS Code version fetching
--auto-upgrade Auto-upgrade app when a newer release is available
+ (default: on)
--no-auto-upgrade Disable app auto-upgrade
--update-config Persist non-schema config write-backs (schema upgrades apply automatically)
-v, --version Show version
@@ -141,7 +142,7 @@ Environment Variables:
GHC_PROXY_REDIRECT_ANTHROPIC Redirect Anthropic requests (true/1)
GHC_PROXY_SHOW_TOKEN Log tokens on refresh (true/1)
GHC_PROXY_DYNAMIC_VSCODE_VERSION Fetch latest VS Code version (true/1)
- GHC_PROXY_AUTO_UPGRADE Auto-upgrade app on startup (true/1)
+ GHC_PROXY_AUTO_UPGRADE Auto-upgrade app on startup (true/1); 0 disables
GHC_PROXY_RATE_LIMIT_SECONDS Minimum seconds between requests
GHC_PROXY_RATE_LIMIT_WAIT Wait instead of rejecting when limited (true/1)
GHC_PROXY_MANUAL_APPROVE Require manual approval per request (true/1)
diff --git a/src/server.rs b/src/server.rs
index b1752d2..3274359 100644
--- a/src/server.rs
+++ b/src/server.rs
@@ -11,7 +11,10 @@ use crate::util;
use crate::util::TokenUsage;
use axum::{
body::{Body, Bytes},
- extract::{DefaultBodyLimit, Path, Query, Request, State},
+ extract::{
+ ws::{Message, WebSocket, WebSocketUpgrade},
+ DefaultBodyLimit, Path, Query, Request, State,
+ },
http::{HeaderMap, HeaderValue, StatusCode},
middleware::{self, Next},
response::{IntoResponse, Response},
@@ -33,8 +36,8 @@ pub fn router(state: SharedState) -> Router {
.route("/models/{model_id}", get(get_model))
.route("/v1/chat/completions", post(chat_completions))
.route("/chat/completions", post(chat_completions))
- .route("/v1/responses", post(responses))
- .route("/responses", post(responses))
+ .route("/v1/responses", post(responses).get(ws_responses))
+ .route("/responses", post(responses).get(ws_responses))
.route("/v1/messages", post(messages))
.route("/v1/messages/count_tokens", post(count_tokens))
.route("/v1/embeddings", post(embeddings))
@@ -45,11 +48,14 @@ pub fn router(state: SharedState) -> Router {
.route("/", get(dashboard))
.route("/requests", get(requests_page))
.route("/metrics/dashboard", get(metrics_page))
+ .route("/app.css", get(stylesheet))
.route("/api/stats", get(api_stats))
+ .route("/api/cache", get(api_cache))
.route("/api/requests", get(api_requests))
.route("/api/audit", get(api_audit))
.route("/api/audit/summary", get(api_audit_summary))
.route("/api/config/reload", post(api_reload_config))
+ .route("/api/config/debug", post(api_set_debug))
.route("/api/models", get(get_models))
.route("/v1beta/models/{model_action}", post(gemini_generate))
.route("/openapi.json", get(openapi_spec))
@@ -63,8 +69,14 @@ pub fn router(state: SharedState) -> Router {
}
/// Whether a request path is an LLM API endpoint that should be guarded by the
-/// optional API key. The dashboard UI, static assets, and metrics endpoints are
-/// intentionally left open so local monitoring keeps working without a key.
+/// optional API key. The dashboard UI, static assets, and read-only metrics
+/// endpoints are intentionally left open so local monitoring keeps working
+/// without a key.
+///
+/// `/api/config/` is guarded despite being part of the dashboard: those routes
+/// mutate the running process, and one of them turns on body capture, which
+/// writes whatever the client sent — credentials included — into the request
+/// log. That is not something an unauthenticated caller should be able to do.
fn is_protected_path(path: &str) -> bool {
const PREFIXES: &[&str] = &[
"/v1/",
@@ -73,6 +85,7 @@ fn is_protected_path(path: &str) -> bool {
"/embeddings",
"/models",
"/v1beta/",
+ "/api/config/",
];
PREFIXES.iter().any(|p| path.starts_with(p))
}
@@ -451,6 +464,9 @@ fn record_failure(
output_tokens_final: None,
cache_read_input_tokens: 0,
cache_creation_input_tokens: 0,
+ reasoning_tokens: None,
+ billed_nano_aiu: None,
+ cache_saved_nano_aiu: None,
premium_multiplier: None,
upstream_idle_max_ms: None,
keepalive_probes: None,
@@ -489,6 +505,10 @@ fn record_failure(
#[derive(Debug, Default)]
struct DirectStreamState {
usage: TokenUsage,
+ /// What Copilot billed, attached to the terminal event beside `usage`.
+ billed_nano_aiu: Option,
+ /// What the cache was worth, from the same event's per-token rates.
+ cache_saved_nano_aiu: Option,
/// Whether `message_delta` — the only event carrying an authoritative
/// `output_tokens` — has arrived. Until it does, the count taken from
/// `message_start` is an opening placeholder, so a record finalized early
@@ -502,6 +522,10 @@ struct DirectStreamState {
impl DirectStreamState {
/// Folds one decoded SSE event into the running state.
fn observe(&mut self, v: &Value) {
+ // Rides along on the terminal event rather than inside `usage`.
+ if let Some(n) = util::copilot_billed_nano_aiu(v) {
+ self.billed_nano_aiu = Some(n);
+ }
match v.get("type").and_then(|t| t.as_str()) {
Some("message_start") => {
// The input buckets are only ever stated here, and
@@ -563,6 +587,10 @@ struct StreamRecorder {
/// is still the `message_start` placeholder — from one that saw the real
/// total in `message_delta`.
usage_final: bool,
+ /// What Copilot billed for this turn, once its terminal event states it.
+ billed_nano_aiu: Option,
+ /// What the cache was worth this turn, from the model's own rates.
+ cache_saved_nano_aiu: Option,
resp_size: usize,
idle: util::IdleTracker,
/// Shared with the keepalive layer wrapping this stream, so the record can
@@ -602,6 +630,9 @@ impl StreamRecorder {
output_tokens_final: None,
cache_read_input_tokens: 0,
cache_creation_input_tokens: 0,
+ reasoning_tokens: None,
+ billed_nano_aiu: None,
+ cache_saved_nano_aiu: None,
premium_multiplier,
upstream_idle_max_ms: None,
keepalive_probes: None,
@@ -626,6 +657,8 @@ impl StreamRecorder {
start,
usage: TokenUsage::default(),
usage_final: false,
+ billed_nano_aiu: None,
+ cache_saved_nano_aiu: None,
resp_size: 0,
idle: util::IdleTracker::new(Instant::now()),
probes,
@@ -658,6 +691,10 @@ impl StreamRecorder {
rec.output_tokens_final = Some(self.usage_final);
rec.cache_read_input_tokens = self.usage.cache_read_input_tokens;
rec.cache_creation_input_tokens = self.usage.cache_creation_input_tokens;
+ rec.reasoning_tokens =
+ (self.usage.reasoning_tokens > 0).then_some(self.usage.reasoning_tokens);
+ rec.billed_nano_aiu = self.billed_nano_aiu;
+ rec.cache_saved_nano_aiu = self.cache_saved_nano_aiu;
rec.upstream_idle_max_ms = Some(self.idle.max_idle_ms_including_now());
rec.keepalive_probes = Some(self.probes.load(std::sync::atomic::Ordering::Relaxed));
rec.duration = elapsed_secs(self.start);
@@ -997,6 +1034,8 @@ async fn chat_completions(State(state): State, body: Bytes) -> Resp
if status.is_success() {
let parsed: Value = serde_json::from_str(&text).unwrap_or(Value::Null);
let usage = util::openai_usage(&parsed.get("usage").cloned().unwrap_or(json!({})));
+ let billed = util::copilot_billed_nano_aiu(&parsed);
+ let cache_saved = util::cache_saving_nano_aiu(&parsed, &usage);
let (tool_count, tool_names) = extract_tools_from_request(&req);
let cost = calculate_cost(&translated, &usage);
let premium_multiplier = state.model_premium_multiplier(&translated).await;
@@ -1014,6 +1053,9 @@ async fn chat_completions(State(state): State, body: Bytes) -> Resp
output_tokens_final: None,
cache_read_input_tokens: usage.cache_read_input_tokens,
cache_creation_input_tokens: usage.cache_creation_input_tokens,
+ reasoning_tokens: (usage.reasoning_tokens > 0).then_some(usage.reasoning_tokens),
+ billed_nano_aiu: billed,
+ cache_saved_nano_aiu: cache_saved,
premium_multiplier,
upstream_idle_max_ms: None,
keepalive_probes: None,
@@ -1145,6 +1187,8 @@ async fn responses(State(state): State, body: Bytes) -> Response {
if status.is_success() {
let parsed: Value = serde_json::from_str(&text).unwrap_or(Value::Null);
let usage = util::responses_usage(&parsed.get("usage").cloned().unwrap_or(json!({})));
+ let billed = util::copilot_billed_nano_aiu(&parsed);
+ let cache_saved = util::cache_saving_nano_aiu(&parsed, &usage);
let (tool_count, tool_names) = extract_tools_from_request(&req);
let cost = calculate_cost(&translated, &usage);
let premium_multiplier = state.model_premium_multiplier(&translated).await;
@@ -1162,6 +1206,9 @@ async fn responses(State(state): State, body: Bytes) -> Response {
output_tokens_final: None,
cache_read_input_tokens: usage.cache_read_input_tokens,
cache_creation_input_tokens: usage.cache_creation_input_tokens,
+ reasoning_tokens: (usage.reasoning_tokens > 0).then_some(usage.reasoning_tokens),
+ billed_nano_aiu: billed,
+ cache_saved_nano_aiu: cache_saved,
premium_multiplier,
upstream_idle_max_ms: None,
keepalive_probes: None,
@@ -1187,9 +1234,460 @@ async fn responses(State(state): State, body: Bytes) -> Response {
}
// ---------------------------------------------------------------------------
-// Anthropic messages
+// Responses over WebSocket
// ---------------------------------------------------------------------------
+/// Message type a client must send to start a turn. The upstream rejects
+/// anything else with `unsupported message type`.
+const WS_RESPONSE_CREATE: &str = "response.create";
+
+/// The catalog's name for this surface.
+const WS_RESPONSES_ENDPOINT: &str = "ws:/responses";
+
+/// Sends a `type: error` frame in the shape the upstream itself uses, so a
+/// client needs no special handling for failures the proxy originates.
+async fn ws_error(socket: &mut WebSocket, code: &str, message: &str) {
+ let frame = json!({"type": "error", "error": {"code": code, "message": message}});
+ let _ = socket.send(Message::Text(frame.to_string().into())).await;
+}
+
+/// The Responses API over WebSocket.
+///
+/// Several models advertise `ws:/responses` in the catalog and nothing else
+/// besides `/responses`; this exposes that transport to clients. The protocol
+/// is the same `response.*` event vocabulary as the SSE path, one event per
+/// text frame — only the transport differs, so a client already written
+/// against the streaming Responses API needs no new parsing.
+///
+/// Undocumented upstream: the request frame must be flat, with `model` at the
+/// top level. Nesting it under `response` is rejected.
+async fn ws_responses(ws: WebSocketUpgrade, State(state): State) -> Response {
+ ws.on_upgrade(move |socket| handle_ws_responses(socket, state))
+}
+
+async fn handle_ws_responses(mut socket: WebSocket, state: SharedState) {
+ // The turn does not begin until the client sends something, so there is no
+ // work to charge for and no record to write if it just connects and leaves.
+ let first = loop {
+ match socket.recv().await {
+ Some(Ok(Message::Text(t))) => break t.to_string(),
+ Some(Ok(Message::Binary(b))) => break String::from_utf8_lossy(&b).into_owned(),
+ // Keepalive traffic before the request is normal; axum answers pings.
+ Some(Ok(_)) => continue,
+ Some(Err(_)) | None => return,
+ }
+ };
+
+ let start = Instant::now();
+ let req_size = first.len();
+
+ let Ok(req) = serde_json::from_str::(&first) else {
+ // Recorded like any other pre-flight failure: a request that leaves no
+ // trace is a request nobody can diagnose.
+ record_failure(
+ &state,
+ "ws:/responses",
+ "",
+ None,
+ 400,
+ crate::store::failure::PRECONDITION,
+ req_size,
+ capture_str(&state, &first),
+ None,
+ start,
+ None,
+ );
+ ws_error(&mut socket, "bad_request", "frame is not JSON").await;
+ return;
+ };
+ let msg_type = req.get("type").and_then(Value::as_str).unwrap_or("");
+ if msg_type != WS_RESPONSE_CREATE {
+ record_failure(
+ &state,
+ "ws:/responses",
+ req.get("model").and_then(Value::as_str).unwrap_or_default(),
+ None,
+ 400,
+ crate::store::failure::PRECONDITION,
+ req_size,
+ capture_json(&state, &req),
+ None,
+ start,
+ None,
+ );
+ ws_error(
+ &mut socket,
+ "bad_request",
+ &format!("unsupported message type: {msg_type}"),
+ )
+ .await;
+ return;
+ }
+
+ let original_model = req
+ .get("model")
+ .and_then(Value::as_str)
+ .unwrap_or_default()
+ .to_string();
+ let translated = translate::translate(&state.model_mappings(), &original_model);
+
+ // The transport is only available for models that advertise it, and saying
+ // so beats an opaque upstream rejection.
+ if !state
+ .model_supports_endpoint(&translated, WS_RESPONSES_ENDPOINT)
+ .await
+ {
+ record_failure(
+ &state,
+ "ws:/responses",
+ &original_model,
+ Some(&translated),
+ 400,
+ crate::store::failure::PRECONDITION,
+ req_size,
+ capture_json(&state, &req),
+ None,
+ start,
+ None,
+ );
+ ws_error(
+ &mut socket,
+ "unsupported_api_for_model",
+ &format!(
+ "Model '{original_model}' does not support {WS_RESPONSES_ENDPOINT}. \
+ Use POST /v1/responses instead."
+ ),
+ )
+ .await;
+ return;
+ }
+
+ if let Err(e) = state.ensure_copilot_token().await {
+ record_failure(
+ &state,
+ "ws:/responses",
+ &original_model,
+ Some(&translated),
+ 500,
+ crate::store::failure::PRECONDITION,
+ req_size,
+ capture_json(&state, &req),
+ None,
+ start,
+ None,
+ );
+ ws_error(&mut socket, "internal_error", &e).await;
+ return;
+ }
+
+ // Forward the model the mapping resolved to, not the alias the client used.
+ let mut upstream_req = req.clone();
+ if let Some(m) = upstream_req.get_mut("model") {
+ *m = json!(translated);
+ }
+
+ let (url, headers) = state.ws_responses_upstream().await;
+ log_debug_request(&state, "ws:/responses", &upstream_req);
+
+ let mut request =
+ match tokio_tungstenite::tungstenite::client::IntoClientRequest::into_client_request(
+ url.as_str(),
+ ) {
+ Ok(r) => r,
+ Err(e) => {
+ ws_error(
+ &mut socket,
+ "internal_error",
+ &format!("bad upstream url: {e}"),
+ )
+ .await;
+ return;
+ }
+ };
+ // The handshake carries the same credentials as the HTTP path.
+ for (k, v) in headers.iter() {
+ request.headers_mut().insert(k.clone(), v.clone());
+ }
+
+ let upstream = match tokio_tungstenite::connect_async(request).await {
+ Ok((s, _)) => s,
+ Err(e) => {
+ record_failure(
+ &state,
+ "ws:/responses",
+ &original_model,
+ Some(&translated),
+ 502,
+ crate::store::failure::CONNECT,
+ req_size,
+ capture_json(&state, &req),
+ None,
+ start,
+ None,
+ );
+ ws_error(&mut socket, "connect_error", &format!("upstream: {e}")).await;
+ return;
+ }
+ };
+
+ pump_ws_responses(
+ socket,
+ upstream,
+ state,
+ original_model,
+ translated,
+ req,
+ req_size,
+ start,
+ )
+ .await;
+}
+
+/// Relays frames in both directions until the turn ends, accumulating what the
+/// record needs on the way past. Usage only becomes known at
+/// `response.completed`, so the totals are folded in as the events go by rather
+/// than by re-parsing the transcript afterwards.
+#[allow(clippy::too_many_arguments)]
+async fn pump_ws_responses(
+ mut client: WebSocket,
+ upstream: tokio_tungstenite::WebSocketStream<
+ tokio_tungstenite::MaybeTlsStream,
+ >,
+ state: SharedState,
+ original_model: String,
+ translated: String,
+ req: Value,
+ req_size: usize,
+ start: Instant,
+) {
+ use futures_util::{SinkExt, StreamExt};
+ use tokio_tungstenite::tungstenite::Message as WsMessage;
+
+ let (mut up_tx, mut up_rx) = upstream.split();
+
+ // The first frame was consumed to route the request; send the rewritten one.
+ let mut outbound = req.clone();
+ if let Some(m) = outbound.get_mut("model") {
+ *m = json!(translated);
+ }
+ if up_tx
+ .send(WsMessage::Text(outbound.to_string().into()))
+ .await
+ .is_err()
+ {
+ ws_error(
+ &mut client,
+ "connect_error",
+ "upstream closed before the request was sent",
+ )
+ .await;
+ return;
+ }
+
+ let mut usage = TokenUsage::default();
+ // Copilot reports what it billed only on the terminal event.
+ let mut billed: Option = None;
+ let mut cache_saved: Option = None;
+ let mut status = 200u16;
+ let mut failure: Option<&'static str> = None;
+ let mut resp_size = 0usize;
+ let mut transcript = String::new();
+ let capture = state.is_debug();
+ let mut idle = util::IdleTracker::new(Instant::now());
+
+ loop {
+ tokio::select! {
+ // Upstream → client. This is where the answer comes from, so it is
+ // also where the turn ends.
+ frame = up_rx.next() => match frame {
+ Some(Ok(WsMessage::Text(t))) => {
+ idle.mark_now();
+ resp_size += t.len();
+ if capture {
+ transcript.push_str(&t);
+ transcript.push('\n');
+ }
+ let done = ws_absorb_event(
+ &t,
+ &mut usage,
+ &mut billed,
+ &mut cache_saved,
+ &mut status,
+ &mut failure,
+ );
+ if client.send(Message::Text(t.as_str().into())).await.is_err() {
+ failure.get_or_insert(crate::store::failure::CLIENT_DISCONNECTED);
+ status = 499;
+ break;
+ }
+ if done {
+ break;
+ }
+ }
+ Some(Ok(WsMessage::Binary(b))) => {
+ idle.mark_now();
+ resp_size += b.len();
+ if client.send(Message::Binary(b.to_vec().into())).await.is_err() {
+ failure.get_or_insert(crate::store::failure::CLIENT_DISCONNECTED);
+ status = 499;
+ break;
+ }
+ }
+ Some(Ok(WsMessage::Close(_))) | None => {
+ // A close before `response.completed` truncated the turn.
+ if failure.is_none() && status == 200 && usage.output_tokens == 0 {
+ failure = Some(crate::store::failure::STREAM_INTERRUPTED);
+ status = 502;
+ }
+ break;
+ }
+ Some(Ok(_)) => {}
+ Some(Err(e)) => {
+ tracing::warn!("[ws:/responses] upstream error: {e}");
+ failure = Some(crate::store::failure::STREAM_INTERRUPTED);
+ status = 502;
+ let _ = client
+ .send(Message::Text(
+ json!({"type": "error", "error": {"code": "stream_interrupted",
+ "message": e.to_string()}})
+ .to_string()
+ .into(),
+ ))
+ .await;
+ break;
+ }
+ },
+
+ // Client → upstream. Kept open for the whole turn: this transport
+ // is bidirectional, and a client may cancel or send another
+ // `response.create` on the same connection.
+ frame = client.recv() => match frame {
+ Some(Ok(Message::Text(t))) => {
+ if up_tx.send(WsMessage::Text(t.as_str().into())).await.is_err() {
+ break;
+ }
+ }
+ Some(Ok(Message::Binary(b))) => {
+ if up_tx.send(WsMessage::Binary(b.to_vec().into())).await.is_err() {
+ break;
+ }
+ }
+ Some(Ok(Message::Close(_))) | None => {
+ failure.get_or_insert(crate::store::failure::CLIENT_DISCONNECTED);
+ status = 499;
+ let _ = up_tx.send(WsMessage::Close(None)).await;
+ break;
+ }
+ Some(Ok(_)) => {}
+ Some(Err(_)) => {
+ failure.get_or_insert(crate::store::failure::CLIENT_DISCONNECTED);
+ status = 499;
+ break;
+ }
+ },
+ }
+ }
+
+ let _ = up_tx.send(WsMessage::Close(None)).await;
+
+ let (tool_count, tool_names) = extract_tools_from_request(&req);
+ let cost = calculate_cost(&translated, &usage);
+ let premium_multiplier = state.model_premium_multiplier(&translated).await;
+ if capture {
+ log_debug_response(&state, "ws:/responses", &transcript);
+ }
+ state.store.add(RequestRecord {
+ id: uuid::Uuid::new_v4().to_string(),
+ timestamp: now_iso(),
+ endpoint: "ws:/responses".into(),
+ model: original_model.clone(),
+ translated_model: (translated != original_model).then_some(translated),
+ status_code: status,
+ request_size: req_size,
+ response_size: resp_size,
+ input_tokens: usage.input_tokens,
+ output_tokens: usage.output_tokens,
+ // The upstream sends usage only in the terminal event, so reaching it
+ // is exactly what makes the count authoritative.
+ output_tokens_final: Some(failure.is_none()),
+ cache_read_input_tokens: usage.cache_read_input_tokens,
+ cache_creation_input_tokens: usage.cache_creation_input_tokens,
+ reasoning_tokens: (usage.reasoning_tokens > 0).then_some(usage.reasoning_tokens),
+ billed_nano_aiu: billed,
+ cache_saved_nano_aiu: cache_saved,
+ premium_multiplier,
+ upstream_idle_max_ms: Some(idle.max_idle_ms_including_now()),
+ keepalive_probes: None,
+ duration: elapsed_secs(start),
+ // Both halves use the flag as it stood when the stream opened. A socket
+ // can stay open for minutes, so re-reading it here would let a toggle
+ // mid-turn produce a record with the request captured and the response
+ // missing — half a transcript is more misleading than none.
+ request_body: capture.then(|| req.to_string()),
+ response_body: capture.then_some(transcript),
+ message_count: None,
+ tool_count: (tool_count > 0).then_some(tool_count),
+ tool_names: (tool_count > 0).then_some(tool_names),
+ stop_reason: None,
+ tools_called: None,
+ is_agent_initiated: None,
+ session_id: None,
+ prompt_cache_hit: cache_disposition(&usage),
+ failure_kind: failure.map(String::from),
+ estimated_cost_usd: Some(cost),
+ });
+}
+
+/// Folds one upstream event into the running record state. Returns true when
+/// the event terminates the turn.
+fn ws_absorb_event(
+ text: &str,
+ usage: &mut TokenUsage,
+ billed: &mut Option,
+ cache_saved: &mut Option,
+ status: &mut u16,
+ failure: &mut Option<&'static str>,
+) -> bool {
+ let Ok(v) = serde_json::from_str::(text) else {
+ return false;
+ };
+ // Attached to the terminal event, beside the response rather than inside it.
+ if let Some(n) = util::copilot_billed_nano_aiu(&v) {
+ *billed = Some(n);
+ }
+ let done = match v.get("type").and_then(Value::as_str).unwrap_or("") {
+ "response.completed" | "response.incomplete" => {
+ if let Some(u) = v.pointer("/response/usage") {
+ *usage = util::responses_usage(u);
+ }
+ true
+ }
+ "response.failed" => {
+ if let Some(u) = v.pointer("/response/usage") {
+ *usage = util::responses_usage(u);
+ }
+ *status = 502;
+ *failure = Some(crate::store::failure::UPSTREAM_STATUS);
+ true
+ }
+ "error" => {
+ *status = 502;
+ *failure = Some(crate::store::failure::UPSTREAM_STATUS);
+ true
+ }
+ _ => false,
+ };
+ // After the arms above, so the token counts it is derived from are the ones
+ // the terminal event just stated.
+ if let Some(n) = util::cache_saving_nano_aiu(&v, usage) {
+ *cache_saved = Some(n);
+ }
+ done
+}
+
+// ---------------------------------------------------------------------------
+// Anthropic messages// ---------------------------------------------------------------------------
+
async fn messages(
State(state): State,
client_headers: HeaderMap,
@@ -1224,10 +1722,23 @@ async fn messages(
req = anthropic::apply_tool_result_suffix(&req, &cfg);
let client_beta = client_beta_header(&client_headers);
+ // What the client sent, counted once. The retry loops downstream mutate the
+ // request, so measuring it there would report the proxy's version of it and
+ // re-serialise the whole body on every attempt to do so.
+ let req_size = body.len();
if state.use_direct_anthropic(&translated).await {
- messages_direct(state, req, original_model, translated, client_beta, start).await
+ messages_direct(
+ state,
+ req,
+ original_model,
+ translated,
+ client_beta,
+ req_size,
+ start,
+ )
+ .await
} else {
- messages_translated(state, req, original_model, translated, start).await
+ messages_translated(state, req, original_model, translated, req_size, start).await
}
}
@@ -1273,6 +1784,7 @@ async fn messages_direct(
original_model: String,
translated: String,
client_beta: Option,
+ req_size: usize,
start: Instant,
) -> Response {
let vision = anthropic::has_image(&req);
@@ -1304,7 +1816,6 @@ async fn messages_direct(
for _ in 0..4 {
let mut sanitized = anthropic::sanitize_anthropic_request(¤t);
sanitized = anthropic::adjust_thinking_budget(&sanitized);
- let req_size = serde_json::to_vec(¤t).map(|v| v.len()).unwrap_or(0);
let payload = serde_json::to_vec(&sanitized).unwrap_or_default();
log_debug_request(&state, "/v1/messages", &sanitized);
@@ -1405,14 +1916,17 @@ async fn messages_direct(
};
let status = resp.status();
if status.is_success() {
- let parsed: Value = resp.json().await.unwrap_or(Value::Null);
- log_debug_response(
- &state,
- "/v1/messages",
- &serde_json::to_string(&parsed).unwrap_or_default(),
- );
+ // Read once, then parse from that. Asking reqwest for JSON and
+ // re-serialising the tree costs two passes and reports a byte count
+ // that is not the one that came off the wire.
+ let text = resp.text().await.unwrap_or_default();
+ let resp_size = text.len();
+ log_debug_response(&state, "/v1/messages", &text);
+ let parsed: Value = serde_json::from_str(&text).unwrap_or(Value::Null);
let usage = parsed.get("usage").cloned().unwrap_or(json!({}));
let usage = util::anthropic_usage(&usage);
+ let billed = util::copilot_billed_nano_aiu(&parsed);
+ let cache_saved = util::cache_saving_nano_aiu(&parsed, &usage);
let (tool_count, tool_names) = extract_tools_from_request(&req);
let tools_called: Vec = parsed
.get("content")
@@ -1440,12 +1954,15 @@ async fn messages_direct(
translated_model: (translated != original_model).then_some(translated.clone()),
status_code: status.as_u16(),
request_size: req_size,
- response_size: serde_json::to_vec(&parsed).map(|v| v.len()).unwrap_or(0),
+ response_size: resp_size,
input_tokens: usage.input_tokens,
output_tokens: usage.output_tokens,
output_tokens_final: None,
cache_read_input_tokens: usage.cache_read_input_tokens,
cache_creation_input_tokens: usage.cache_creation_input_tokens,
+ reasoning_tokens: (usage.reasoning_tokens > 0).then_some(usage.reasoning_tokens),
+ billed_nano_aiu: billed,
+ cache_saved_nano_aiu: cache_saved,
premium_multiplier: state.model_premium_multiplier(&translated).await,
upstream_idle_max_ms: None,
keepalive_probes: None,
@@ -1511,6 +2028,7 @@ async fn messages_translated(
req: Value,
original_model: String,
translated: String,
+ req_size: usize,
start: Instant,
) -> Response {
let vision = anthropic::has_image(&req);
@@ -1537,7 +2055,6 @@ async fn messages_translated(
let url = format!("{}/chat/completions", state.copilot_base_url());
let mut headers = state.copilot_headers(vision).await;
set_initiator(&mut headers, agent);
- let req_size = serde_json::to_vec(¤t).map(|v| v.len()).unwrap_or(0);
let payload = serde_json::to_vec(&openai_req).unwrap_or_default();
log_debug_request(&state, "/v1/messages", &openai_req);
@@ -1580,14 +2097,17 @@ async fn messages_translated(
};
let status = resp.status();
if status.is_success() {
- let parsed: Value = resp.json().await.unwrap_or(Value::Null);
+ // Read once, then parse from that: `resp.json()` followed by a
+ // re-serialisation costs two extra passes over the body and reports
+ // a size that is not the one that came off the wire.
+ let text = resp.text().await.unwrap_or_default();
+ let resp_size = text.len();
+ log_debug_response(&state, "/v1/messages", &text);
+ let parsed: Value = serde_json::from_str(&text).unwrap_or(Value::Null);
let anthropic_resp = anthropic::openai_to_anthropic(&parsed);
- log_debug_response(
- &state,
- "/v1/messages",
- &serde_json::to_string(&parsed).unwrap_or_default(),
- );
let usage = util::openai_usage(&parsed.get("usage").cloned().unwrap_or(json!({})));
+ let billed = util::copilot_billed_nano_aiu(&parsed);
+ let cache_saved = util::cache_saving_nano_aiu(&parsed, &usage);
let (tool_count, tool_names) = extract_tools_from_request(&openai_req);
state.store.add(RequestRecord {
id: uuid::Uuid::new_v4().to_string(),
@@ -1597,14 +2117,15 @@ async fn messages_translated(
translated_model: (translated != original_model).then_some(translated.clone()),
status_code: status.as_u16(),
request_size: req_size,
- response_size: serde_json::to_vec(&anthropic_resp)
- .map(|v| v.len())
- .unwrap_or(0),
+ response_size: resp_size,
input_tokens: usage.input_tokens,
output_tokens: usage.output_tokens,
output_tokens_final: None,
cache_read_input_tokens: usage.cache_read_input_tokens,
cache_creation_input_tokens: usage.cache_creation_input_tokens,
+ reasoning_tokens: (usage.reasoning_tokens > 0).then_some(usage.reasoning_tokens),
+ billed_nano_aiu: billed,
+ cache_saved_nano_aiu: cache_saved,
premium_multiplier: state.model_premium_multiplier(&translated).await,
upstream_idle_max_ms: None,
keepalive_probes: None,
@@ -1848,6 +2369,8 @@ async fn gemini_generate(
let parsed: Value = serde_json::from_str(&text).unwrap_or(Value::Null);
let gemini_resp = gemini::openai_to_gemini(&parsed);
let usage = util::openai_usage(&parsed.get("usage").cloned().unwrap_or(json!({})));
+ let billed = util::copilot_billed_nano_aiu(&parsed);
+ let cache_saved = util::cache_saving_nano_aiu(&parsed, &usage);
let cost = calculate_cost(&translated, &usage);
let premium_multiplier = state.model_premium_multiplier(&translated).await;
state.store.add(RequestRecord {
@@ -1864,6 +2387,9 @@ async fn gemini_generate(
output_tokens_final: None,
cache_read_input_tokens: usage.cache_read_input_tokens,
cache_creation_input_tokens: usage.cache_creation_input_tokens,
+ reasoning_tokens: (usage.reasoning_tokens > 0).then_some(usage.reasoning_tokens),
+ billed_nano_aiu: billed,
+ cache_saved_nano_aiu: cache_saved,
premium_multiplier,
upstream_idle_max_ms: None,
keepalive_probes: None,
@@ -1943,6 +2469,7 @@ async fn stream_gemini(
Ok(r) => r,
Err(e) => return gemini_error(StatusCode::GATEWAY_TIMEOUT, e.to_string()),
};
+ state.record_quota_headers(upstream.headers());
let status = upstream.status().as_u16();
// Surface a non-2xx upstream (JSON error, not SSE) as a normal error.
if !is_streamable_status(status) {
@@ -1968,6 +2495,9 @@ async fn stream_gemini(
let mut byte_stream = upstream.bytes_stream();
let mut lines = util::SseLineBuffer::new();
let mut usage = TokenUsage::default();
+ // Copilot reports what it billed only on the terminal event.
+ let mut billed: Option = None;
+ let mut cache_saved: Option = None;
let mut resp_size = 0usize;
let mut finish: Option = None;
let mut debug_raw: Vec = Vec::new();
@@ -1988,6 +2518,8 @@ async fn stream_gemini(
if let Some(u) = v.get("usage") {
if !u.is_null() {
usage.merge_stream_update(util::openai_usage(u));
+ billed = util::copilot_billed_nano_aiu(&v).or(billed);
+ cache_saved = util::cache_saving_nano_aiu(&v, &usage).or(cache_saved);
}
}
if let Some(choice) = v.get("choices").and_then(|c| c.as_array()).and_then(|a| a.first()) {
@@ -2033,7 +2565,8 @@ async fn stream_gemini(
output_tokens: usage.output_tokens,
output_tokens_final: None,
cache_read_input_tokens: usage.cache_read_input_tokens,
- cache_creation_input_tokens: usage.cache_creation_input_tokens,
+ cache_creation_input_tokens: usage.cache_creation_input_tokens, reasoning_tokens: (usage.reasoning_tokens > 0).then_some(usage.reasoning_tokens), billed_nano_aiu: billed,
+ cache_saved_nano_aiu: cache_saved,
premium_multiplier: state.model_premium_multiplier(&translated).await,
upstream_idle_max_ms: Some(idle.max_idle_ms_including_now()),
keepalive_probes: None,
@@ -2099,6 +2632,8 @@ async fn embeddings(State(state): State, body: Bytes) -> Response {
if status.is_success() {
let parsed: Value = serde_json::from_str(&text).unwrap_or(Value::Null);
let usage = util::openai_usage(&parsed.get("usage").cloned().unwrap_or(json!({})));
+ let billed = util::copilot_billed_nano_aiu(&parsed);
+ let cache_saved = util::cache_saving_nano_aiu(&parsed, &usage);
state.store.add(RequestRecord {
id: uuid::Uuid::new_v4().to_string(),
timestamp: now_iso(),
@@ -2113,6 +2648,9 @@ async fn embeddings(State(state): State, body: Bytes) -> Response {
output_tokens_final: None,
cache_read_input_tokens: usage.cache_read_input_tokens,
cache_creation_input_tokens: usage.cache_creation_input_tokens,
+ reasoning_tokens: (usage.reasoning_tokens > 0).then_some(usage.reasoning_tokens),
+ billed_nano_aiu: billed,
+ cache_saved_nano_aiu: cache_saved,
// Embeddings are not billed as premium requests.
premium_multiplier: None,
upstream_idle_max_ms: None,
@@ -2146,8 +2684,18 @@ async fn usage(State(state): State) -> Response {
if let Err(e) = state.ensure_copilot_token().await {
return error_response(StatusCode::INTERNAL_SERVER_ERROR, e);
}
+ // The live per-SKU snapshot rides along on every proxied response, so it is
+ // already current; the upstream call below adds the plan name and the
+ // detailed per-category breakdown it does not carry.
+ let live = state.quota_snapshot();
match state.fetch_usage().await {
- Ok(v) => Json(crate::state::summarize_usage(&v)).into_response(),
+ Ok(v) => {
+ let mut summary = crate::state::summarize_usage(&v);
+ if !live.is_empty() {
+ summary["live"] = serde_json::to_value(&live).unwrap_or(Value::Null);
+ }
+ Json(summary).into_response()
+ }
Err(e) => error_response(StatusCode::BAD_GATEWAY, e),
}
}
@@ -2183,6 +2731,13 @@ async fn health(
"models_loaded": model_count,
"requests_served": stats.request_count,
"auth_required": state.api_key().is_some(),
+ // Whether request/response bodies are being captured. Surfaced so the
+ // dashboard can say why a request has no body to show, and so an
+ // operator can notice capture was left on.
+ "debug": state.is_debug(),
+ // Reported by the upstream on every response, so this costs no extra
+ // API call. Empty until the first request has been proxied.
+ "quota": state.quota_snapshot(),
});
let strict = params
.get("strict")
@@ -2243,6 +2798,7 @@ async fn stream_openai(
Ok(r) => r,
Err(e) => return error_response(StatusCode::GATEWAY_TIMEOUT, e.to_string()),
};
+ state.record_quota_headers(upstream.headers());
let status = upstream.status().as_u16();
// A non-2xx upstream (e.g. GitHub Models returning 401/403 as JSON when
// the token lacks the `models: read` permission) is not an SSE stream —
@@ -2269,6 +2825,7 @@ async fn stream_openai(
}
break upstream;
};
+ state.record_quota_headers(upstream.headers());
let status = upstream.status().as_u16();
let model = translated.clone();
// Shared with the keepalive wrapper so the record can report how
@@ -2279,6 +2836,9 @@ async fn stream_openai(
let mut byte_stream = upstream.bytes_stream();
let mut lines = util::SseLineBuffer::new();
let mut usage = TokenUsage::default();
+ // Copilot reports what it billed only on the terminal event.
+ let mut billed: Option = None;
+ let mut cache_saved: Option = None;
let mut resp_size = 0usize;
let mut debug_raw: Vec = Vec::new();
let mut interrupted: Option = None;
@@ -2308,6 +2868,8 @@ async fn stream_openai(
Ok(v) => {
if let Some(u) = v.get("usage") {
usage.merge_stream_update(util::openai_usage(u));
+ billed = util::copilot_billed_nano_aiu(&v).or(billed);
+ cache_saved = util::cache_saving_nano_aiu(&v, &usage).or(cache_saved);
}
resp_size += data.len();
yield Ok(Bytes::from(format!("data: {data}\n\n")));
@@ -2334,6 +2896,8 @@ async fn stream_openai(
if let Ok(v) = serde_json::from_str::(data) {
if let Some(u) = v.get("usage") {
usage.merge_stream_update(util::openai_usage(u));
+ billed = util::copilot_billed_nano_aiu(&v).or(billed);
+ cache_saved = util::cache_saving_nano_aiu(&v, &usage).or(cache_saved);
}
}
resp_size += data.len();
@@ -2371,7 +2935,8 @@ async fn stream_openai(
output_tokens: usage.output_tokens,
output_tokens_final: None,
cache_read_input_tokens: usage.cache_read_input_tokens,
- cache_creation_input_tokens: usage.cache_creation_input_tokens,
+ cache_creation_input_tokens: usage.cache_creation_input_tokens, reasoning_tokens: (usage.reasoning_tokens > 0).then_some(usage.reasoning_tokens), billed_nano_aiu: billed,
+ cache_saved_nano_aiu: cache_saved,
premium_multiplier: state.model_premium_multiplier(&translated).await,
upstream_idle_max_ms: Some(idle.max_idle_ms_including_now()),
keepalive_probes: None,
@@ -2419,6 +2984,7 @@ async fn stream_responses(
Ok(r) => r,
Err(e) => return error_response(StatusCode::GATEWAY_TIMEOUT, e.to_string()),
};
+ state.record_quota_headers(upstream.headers());
let status = upstream.status().as_u16();
// A non-2xx upstream returns a JSON error body, not an SSE stream. Forward
// it as a normal error response instead of wrapping it in a 200 "stream",
@@ -2440,6 +3006,9 @@ async fn stream_responses(
let mut byte_stream = upstream.bytes_stream();
let mut lines = util::SseLineBuffer::new();
let mut usage = TokenUsage::default();
+ // Copilot reports what it billed only on the terminal event.
+ let mut billed: Option = None;
+ let mut cache_saved: Option = None;
let mut resp_size = 0usize;
let mut debug_raw: Vec = Vec::new();
let mut interrupted: Option = None;
@@ -2466,6 +3035,8 @@ async fn stream_responses(
completed = true;
let raw = v.get("response").and_then(|r| r.get("usage")).cloned().unwrap_or(json!({}));
usage = util::responses_usage(&raw);
+ billed = util::copilot_billed_nano_aiu(&v).or(billed);
+ cache_saved = util::cache_saving_nano_aiu(&v, &usage).or(cache_saved);
}
}
}
@@ -2503,7 +3074,8 @@ async fn stream_responses(
output_tokens: usage.output_tokens,
output_tokens_final: None,
cache_read_input_tokens: usage.cache_read_input_tokens,
- cache_creation_input_tokens: usage.cache_creation_input_tokens,
+ cache_creation_input_tokens: usage.cache_creation_input_tokens, reasoning_tokens: (usage.reasoning_tokens > 0).then_some(usage.reasoning_tokens), billed_nano_aiu: billed,
+ cache_saved_nano_aiu: cache_saved,
premium_multiplier: state.model_premium_multiplier(&translated).await,
upstream_idle_max_ms: Some(idle.max_idle_ms_including_now()),
keepalive_probes: None,
@@ -2550,6 +3122,7 @@ async fn stream_anthropic_direct(
session_id,
start,
} = meta;
+ state.record_quota_headers(upstream.headers());
let status = upstream.status().as_u16();
// Without this the error body is wrapped in a 200 `text/event-stream`, so
// the client waits on a stream that never produces an event and reports a
@@ -2627,6 +3200,8 @@ async fn stream_anthropic_direct(
// this generator: `Drop` can only report what the recorder
// itself holds.
recorder.usage = st.usage;
+ recorder.billed_nano_aiu = st.billed_nano_aiu;
+ recorder.cache_saved_nano_aiu = st.cache_saved_nano_aiu;
recorder.usage_final = st.usage_final;
}
}
@@ -2692,6 +3267,7 @@ async fn stream_anthropic_translated(
Ok(r) => r,
Err(e) => return anthropic_error(StatusCode::GATEWAY_TIMEOUT, e.to_string()),
};
+ state.record_quota_headers(upstream.headers());
let status = upstream.status().as_u16();
// Surface a non-2xx upstream (JSON error, not SSE) as a normal error.
if !is_streamable_status(status) {
@@ -2731,6 +3307,9 @@ async fn stream_anthropic_translated(
let mut conv = AnthropicStreamState::new();
let mut chunks: Vec = Vec::new();
let mut usage = TokenUsage::default();
+ // Copilot reports what it billed only on the terminal event.
+ let mut billed: Option = None;
+ let mut cache_saved: Option = None;
let mut resp_size = 0usize;
let mut debug_raw: Vec = Vec::new();
let mut interrupted: Option = None;
@@ -2752,6 +3331,8 @@ async fn stream_anthropic_translated(
};
if let Some(u) = v.get("usage") {
usage.merge_stream_update(util::openai_usage(u));
+ billed = util::copilot_billed_nano_aiu(&v).or(billed);
+ cache_saved = util::cache_saving_nano_aiu(&v, &usage).or(cache_saved);
}
chunks.push(v.clone());
for event in conv.process(&v) {
@@ -2810,7 +3391,8 @@ async fn stream_anthropic_translated(
output_tokens: usage.output_tokens,
output_tokens_final: None,
cache_read_input_tokens: usage.cache_read_input_tokens,
- cache_creation_input_tokens: usage.cache_creation_input_tokens,
+ cache_creation_input_tokens: usage.cache_creation_input_tokens, reasoning_tokens: (usage.reasoning_tokens > 0).then_some(usage.reasoning_tokens), billed_nano_aiu: billed,
+ cache_saved_nano_aiu: cache_saved,
premium_multiplier: state.model_premium_multiplier(&translated).await,
upstream_idle_max_ms: Some(idle.max_idle_ms_including_now()),
keepalive_probes: None,
@@ -2967,7 +3549,16 @@ where
// ---------------------------------------------------------------------------
async fn dashboard() -> Response {
- serve_asset("dashboard.html", include_str!("../public/dashboard.html"))
+ serve_asset(
+ include_str!("../public/dashboard.html"),
+ "text/html; charset=utf-8",
+ )
+}
+
+/// Design system shared by the three dashboard pages. Served separately rather
+/// than inlined into each page so the pages cannot drift apart visually.
+async fn stylesheet() -> Response {
+ serve_asset(include_str!("../public/app.css"), "text/css; charset=utf-8")
}
/// Serves a machine-readable OpenAPI v3 specification describing the proxy's
@@ -3080,19 +3671,23 @@ async fn openapi_spec() -> Response {
}
async fn requests_page() -> Response {
- serve_asset("requests.html", include_str!("../public/requests.html"))
+ serve_asset(
+ include_str!("../public/requests.html"),
+ "text/html; charset=utf-8",
+ )
}
async fn metrics_page() -> Response {
- serve_asset("metrics.html", include_str!("../public/metrics.html"))
+ serve_asset(
+ include_str!("../public/metrics.html"),
+ "text/html; charset=utf-8",
+ )
}
-fn serve_asset(_name: &str, contents: &'static str) -> Response {
+fn serve_asset(contents: &'static str, content_type: &'static str) -> Response {
let mut resp = Response::new(Body::from(contents));
- resp.headers_mut().insert(
- "Content-Type",
- HeaderValue::from_static("text/html; charset=utf-8"),
- );
+ resp.headers_mut()
+ .insert("Content-Type", HeaderValue::from_static(content_type));
resp
}
@@ -3255,6 +3850,45 @@ async fn metrics_openmetrics(State(state): State) -> Response {
state.uptime_secs()
));
+ // Quota comes from headers the upstream attaches to every response, so
+ // scraping this endpoint never costs an extra API call.
+ let quotas = state.quota_snapshot();
+ if !quotas.is_empty() {
+ out.push_str(
+ "# HELP ghc_proxy_quota_percent_remaining Percent of the entitlement still available.\n",
+ );
+ out.push_str("# TYPE ghc_proxy_quota_percent_remaining gauge\n");
+ for (sku, q) in "as {
+ out.push_str(&format!(
+ "ghc_proxy_quota_percent_remaining{{sku=\"{}\"}} {}\n",
+ metrics_label_escape(sku),
+ q.percent_remaining
+ ));
+ }
+
+ out.push_str(
+ "# HELP ghc_proxy_quota_entitlement Allowance for the period; negative means unlimited.\n",
+ );
+ out.push_str("# TYPE ghc_proxy_quota_entitlement gauge\n");
+ for (sku, q) in "as {
+ out.push_str(&format!(
+ "ghc_proxy_quota_entitlement{{sku=\"{}\"}} {}\n",
+ metrics_label_escape(sku),
+ q.entitlement
+ ));
+ }
+
+ out.push_str("# HELP ghc_proxy_quota_overage Amount consumed beyond the entitlement.\n");
+ out.push_str("# TYPE ghc_proxy_quota_overage gauge\n");
+ for (sku, q) in "as {
+ out.push_str(&format!(
+ "ghc_proxy_quota_overage{{sku=\"{}\"}} {}\n",
+ metrics_label_escape(sku),
+ q.overage
+ ));
+ }
+ }
+
out.push_str(
"# HELP ghc_proxy_estimated_cost_usd_total Total estimated request cost in USD.\n",
);
@@ -3303,10 +3937,150 @@ async fn api_reload_config(State(state): State) -> Response {
.into_response()
}
+/// Turns request/response body capture on or off without a restart.
+///
+/// Capturing bodies is the only way to see what a client actually sent, but it
+/// used to require stopping the proxy and relaunching it with `--debug` — by
+/// which point the request you wanted to inspect is long gone. The flag is read
+/// live on every request, so flipping it here applies from the next call.
+///
+/// Deliberately not written back to `config.yaml`: capture puts prompts, tool
+/// output and any credentials they carry into memory and the log, so it should
+/// lapse on restart rather than stay on because someone forgot.
+async fn api_set_debug(State(state): State, body: Option>) -> Response {
+ let requested = body
+ .as_ref()
+ .and_then(|Json(v)| v.get("debug"))
+ .and_then(Value::as_bool);
+ let Some(debug) = requested else {
+ return (
+ StatusCode::BAD_REQUEST,
+ Json(json!({
+ "error": "expected a JSON body of {\"debug\": true} or {\"debug\": false}"
+ })),
+ )
+ .into_response();
+ };
+ state.set_debug(debug);
+ let action = if debug { "enabled" } else { "disabled" };
+ tracing::info!("[debug] body capture {action} from the dashboard");
+ Json(json!({ "ok": true, "debug": debug })).into_response()
+}
+
async fn api_stats(State(state): State) -> Response {
Json(state.store.stats()).into_response()
}
+/// Running totals for one model's prompt-cache behaviour.
+#[derive(Default)]
+struct CacheAgg {
+ requests: u64,
+ input: u64,
+ read: u64,
+ write: u64,
+ /// Net effect on the bill, in nano-AI-units, from the model's own rates.
+ saved_nano_aiu: i64,
+ /// Whether any response for this model reported rates to compute the above
+ /// from. Without it a genuine zero — nothing was cached, so nothing was
+ /// saved — is indistinguishable from an unpriced model.
+ priced: bool,
+}
+
+/// Prompt-cache statistics.
+///
+/// The hit rate is the early warning for a broken prompt prefix: on an agent
+/// workload it should sit high and stable, and a sudden drop means the prompt
+/// stopped matching and every turn is paying full input price again. Reporting
+/// it per model is what makes that actionable — a single global number cannot
+/// tell you *which* conversation broke.
+///
+/// Totals come from the all-time running counters. The per-model breakdown is
+/// derived from the retained ring buffer, so it describes the most recent
+/// `sampled_requests` calls rather than every one ever served; the response
+/// says so explicitly rather than letting the two silently disagree.
+async fn api_cache(State(state): State) -> Response {
+ let stats = state.store.stats();
+
+ let (by_model, hit, write_only, uncached, sampled) = state.store.with_records(|records| {
+ let mut by_model: HashMap = HashMap::new();
+ let (mut hit, mut write_only, mut uncached, mut sampled) = (0u64, 0u64, 0u64, 0u64);
+
+ for r in records {
+ // A rejected attempt has no tokens and no model worth a row; counting
+ // it would add an empty-named entry and drag every disposition
+ // toward "uncached".
+ if r.failed() {
+ continue;
+ }
+ sampled += 1;
+ match r.prompt_cache_hit {
+ Some(true) => hit += 1,
+ Some(false) => write_only += 1,
+ None => uncached += 1,
+ }
+
+ // Price against the model that actually served the request, not the
+ // alias the client asked for.
+ let model = r.translated_model.as_deref().unwrap_or(&r.model);
+ let agg = by_model.entry(model.to_string()).or_default();
+ agg.requests += 1;
+ agg.input += r.input_tokens;
+ agg.read += r.cache_read_input_tokens;
+ agg.write += r.cache_creation_input_tokens;
+ // Derived upstream from the rates that response reported, so a
+ // model Copilot includes at no charge contributes nothing rather
+ // than an imagined saving.
+ agg.saved_nano_aiu += r.cache_saved_nano_aiu.unwrap_or(0);
+ agg.priced |= r.cache_saved_nano_aiu.is_some();
+ }
+ (by_model, hit, write_only, uncached, sampled)
+ });
+
+ let mut models: Vec = by_model
+ .into_iter()
+ .map(|(model, a)| {
+ json!({
+ "model": model,
+ "requests": a.requests,
+ "input_tokens": a.input,
+ "cache_read_tokens": a.read,
+ "cache_creation_tokens": a.write,
+ "fresh_tokens": a.input.saturating_sub(a.read + a.write),
+ "hit_rate": if a.input > 0 { a.read as f64 / a.input as f64 } else { 0.0 },
+ "saved_nano_aiu": a.priced.then_some(a.saved_nano_aiu),
+ })
+ })
+ .collect();
+ // Biggest prompt first: that is where a broken prefix costs the most.
+ models.sort_by(|a, b| {
+ let key = |v: &Value| v.get("input_tokens").and_then(Value::as_u64).unwrap_or(0);
+ key(b).cmp(&key(a))
+ });
+
+ let total_in = stats.total_input_tokens;
+ let cached = stats.total_cache_read_tokens + stats.total_cache_creation_tokens;
+ Json(json!({
+ "totals": {
+ "input_tokens": total_in,
+ "cache_read_tokens": stats.total_cache_read_tokens,
+ "cache_creation_tokens": stats.total_cache_creation_tokens,
+ "fresh_tokens": total_in.saturating_sub(cached),
+ "hit_rate": if total_in > 0 {
+ stats.total_cache_read_tokens as f64 / total_in as f64
+ } else { 0.0 },
+ "request_count": stats.request_count,
+ },
+ "dispositions": {
+ "served_from_cache": hit,
+ "wrote_to_cache": write_only,
+ "no_cache": uncached,
+ },
+ "sampled_requests": sampled,
+ "by_model": models,
+ }))
+ .into_response()
+}
+
/// Largest page size a dashboard API will honour. Keeps a hostile `per_page`
/// from forcing an unbounded clone of the request store.
const MAX_PAGE_SIZE: usize = 500;
@@ -3342,7 +4116,7 @@ async fn api_requests(
let session = params.get("session").map(|s| s.as_str());
let (items, total) = if failed_only || session.is_some() {
state.store.filtered_page(per_page, offset, |r| {
- if failed_only && r.status_code < 400 && r.failure_kind.is_none() {
+ if failed_only && !r.failed() {
return false;
}
match session {
@@ -3601,6 +4375,19 @@ mod tests {
assert!(!is_protected_path("/requests"));
}
+ /// Read-only dashboard APIs stay open so local monitoring works without a
+ /// key, but anything that mutates the running process must not. Turning on
+ /// body capture writes whatever the client sent — credentials included —
+ /// into the request log, so it is guarded like an LLM endpoint.
+ #[test]
+ fn protected_paths_cover_config_mutations() {
+ assert!(is_protected_path("/api/config/debug"));
+ assert!(is_protected_path("/api/config/reload"));
+ assert!(!is_protected_path("/api/cache"));
+ assert!(!is_protected_path("/api/requests"));
+ assert!(!is_protected_path("/api/audit/summary"));
+ }
+
#[test]
fn presented_key_from_bearer() {
let mut h = HeaderMap::new();
@@ -3964,6 +4751,7 @@ mod tests {
cache_read_input_tokens: 1000,
cache_creation_input_tokens: 1000,
output_tokens: 1000,
+ reasoning_tokens: 0,
};
let (base_in, base_out) = model_rates("claude-opus-4.8");
let expected = base_in // uncached remainder
@@ -3984,6 +4772,7 @@ mod tests {
cache_read_input_tokens: 101_940,
cache_creation_input_tokens: 1_731,
output_tokens: 561,
+ reasoning_tokens: 0,
};
let cache_aware = calculate_cost("claude-opus-5", &usage);
let (base_in, base_out) = model_rates("claude-opus-5");
diff --git a/src/state.rs b/src/state.rs
index c5b0588..eacef06 100644
--- a/src/state.rs
+++ b/src/state.rs
@@ -6,13 +6,13 @@ use crate::auth;
use crate::config::{self, Config, ModelMappings};
use crate::store::RequestStore;
use reqwest::header::{HeaderMap, HeaderName, HeaderValue};
+use std::collections::BTreeMap;
use std::sync::Arc;
use std::sync::RwLock as StdRwLock;
use std::time::{Duration, Instant, SystemTime, UNIX_EPOCH};
use tokio::sync::{Mutex, RwLock};
-/// Mutable token state guarded by a mutex.
-#[derive(Default)]
+/// Mutable token state guarded by a mutex.#[derive(Default)]
pub struct TokenState {
pub github_token: String,
pub copilot_token: Option,
@@ -37,10 +37,108 @@ pub struct AppState {
pub session_id: String,
/// Instant the process started serving, used to report uptime on `/health`.
pub started_at: Instant,
+ /// Latest per-SKU quota reported by the upstream, keyed by SKU name.
+ pub quotas: StdRwLock>,
}
pub type SharedState = Arc;
+/// Prefix of the per-SKU quota headers Copilot attaches to every response.
+const QUOTA_HEADER_PREFIX: &str = "x-quota-snapshot-";
+
+/// A quota snapshot for one billing SKU, as reported by the upstream on every
+/// response.
+///
+/// Copilot returns these on each request (`x-quota-snapshot-chat`,
+/// `-completions`, `-premium_interactions`), so live quota costs nothing —
+/// unlike `/copilot_internal/user`, which is a separate API call.
+#[derive(Debug, Clone, Default, serde::Serialize)]
+pub struct QuotaSnapshot {
+ /// Allowance for the period. Negative means unlimited.
+ pub entitlement: f64,
+ /// Amount consumed beyond the entitlement.
+ pub overage: f64,
+ /// Whether going past the entitlement is allowed at all.
+ pub overage_permitted: bool,
+ /// Percentage of the entitlement still available.
+ pub percent_remaining: f64,
+ /// True when the SKU has no cap.
+ pub unlimited: bool,
+ /// When the allowance resets, if the upstream reported it.
+ #[serde(skip_serializing_if = "Option::is_none")]
+ pub reset_date: Option,
+}
+
+/// Decodes the `%XX` escapes used in the header's date field.
+fn percent_decode(s: &str) -> String {
+ let bytes = s.as_bytes();
+ let mut out = Vec::with_capacity(bytes.len());
+ let mut i = 0;
+ while i < bytes.len() {
+ if bytes[i] == b'%' && i + 2 < bytes.len() {
+ let hex = std::str::from_utf8(&bytes[i + 1..i + 3]).ok();
+ if let Some(byte) = hex.and_then(|h| u8::from_str_radix(h, 16).ok()) {
+ out.push(byte);
+ i += 3;
+ continue;
+ }
+ }
+ out.push(bytes[i]);
+ i += 1;
+ }
+ String::from_utf8_lossy(&out).into_owned()
+}
+
+/// Parses a quota header value such as
+/// `ent=-1&ov=0.0&ovPerm=true&rem=100.0&rst=2026-08-01T00%3A00%3A00Z`.
+///
+/// Returns `None` when nothing recognizable is present, so an upstream that
+/// stops sending these (they are undocumented) simply leaves the cache empty
+/// rather than reporting zeroed quota.
+pub fn parse_quota_snapshot(value: &str) -> Option {
+ let mut snapshot = QuotaSnapshot::default();
+ let mut saw_field = false;
+ for pair in value.split('&') {
+ let Some((key, raw)) = pair.split_once('=') else {
+ continue;
+ };
+ match key.trim() {
+ "ent" => {
+ if let Ok(v) = raw.parse::() {
+ snapshot.entitlement = v;
+ snapshot.unlimited = v < 0.0;
+ saw_field = true;
+ }
+ }
+ "ov" => {
+ if let Ok(v) = raw.parse::() {
+ snapshot.overage = v;
+ saw_field = true;
+ }
+ }
+ "ovPerm" => {
+ snapshot.overage_permitted = raw.eq_ignore_ascii_case("true");
+ saw_field = true;
+ }
+ "rem" => {
+ if let Ok(v) = raw.parse::() {
+ snapshot.percent_remaining = v;
+ saw_field = true;
+ }
+ }
+ "rst" => {
+ let decoded = percent_decode(raw);
+ if !decoded.is_empty() {
+ snapshot.reset_date = Some(decoded);
+ saw_field = true;
+ }
+ }
+ _ => {}
+ }
+ }
+ saw_field.then_some(snapshot)
+}
+
fn now_secs() -> u64 {
SystemTime::now()
.duration_since(UNIX_EPOCH)
@@ -58,14 +156,20 @@ fn now_millis() -> u128 {
impl AppState {
pub fn new(config: Config, github_token: String) -> Self {
- // No global request timeout: SSE responses are long-lived streams. Only
- // the connect phase and idle pooled connections are bounded so a dead
- // upstream cannot wedge a request forever.
- let http = reqwest::Client::builder()
+ // No global request timeout: SSE responses are long-lived streams, and
+ // capping their total duration would cut off legitimate long answers.
+ // Instead the connect phase, idle pooled connections, and the gap
+ // *between* reads are bounded. The read timeout is what turns a
+ // half-open upstream into an error the streaming paths can report,
+ // rather than a request that hangs forever.
+ let mut builder = reqwest::Client::builder()
.connect_timeout(Duration::from_secs(30))
- .pool_idle_timeout(Duration::from_secs(90))
- .build()
- .expect("failed to build HTTP client");
+ .pool_idle_timeout(Duration::from_secs(90));
+ if config.upstream_read_timeout_seconds > 0 {
+ builder =
+ builder.read_timeout(Duration::from_secs(config.upstream_read_timeout_seconds));
+ }
+ let http = builder.build().expect("failed to build HTTP client");
AppState {
http,
config: StdRwLock::new(config),
@@ -81,9 +185,41 @@ impl AppState {
machine_id: auth::load_or_create_machine_id(),
session_id: format!("{}{}", uuid::Uuid::new_v4(), now_millis()),
started_at: Instant::now(),
+ quotas: StdRwLock::new(BTreeMap::new()),
+ }
+ }
+
+ /// Records the per-SKU quota headers attached to an upstream response.
+ ///
+ /// Called for every Copilot response, so quota stays current without the
+ /// extra `/copilot_internal/user` round trip. Headers that do not parse are
+ /// ignored, leaving the previous value in place.
+ pub fn record_quota_headers(&self, headers: &HeaderMap) {
+ let mut parsed: Vec<(String, QuotaSnapshot)> = Vec::new();
+ for (name, value) in headers {
+ let Some(sku) = name.as_str().strip_prefix(QUOTA_HEADER_PREFIX) else {
+ continue;
+ };
+ let Ok(value) = value.to_str() else { continue };
+ if let Some(snapshot) = parse_quota_snapshot(value) {
+ parsed.push((sku.to_string(), snapshot));
+ }
+ }
+ if parsed.is_empty() {
+ return;
+ }
+ if let Ok(mut quotas) = self.quotas.write() {
+ for (sku, snapshot) in parsed {
+ quotas.insert(sku, snapshot);
+ }
}
}
+ /// The most recent per-SKU quota reported by the upstream.
+ pub fn quota_snapshot(&self) -> BTreeMap {
+ self.quotas.read().map(|q| q.clone()).unwrap_or_default()
+ }
+
/// Seconds this process has been running.
pub fn uptime_secs(&self) -> u64 {
self.started_at.elapsed().as_secs()
@@ -133,6 +269,15 @@ impl AppState {
self.config.read().unwrap().debug
}
+ /// Turns body capture on or off for the running process. Debug is read
+ /// live on every request, so this takes effect immediately and without a
+ /// restart — but it is deliberately not written back to `config.yaml`, so
+ /// a proxy that is restarted returns to its configured default rather than
+ /// silently keeping capture on forever.
+ pub fn set_debug(&self, debug: bool) {
+ self.config.write().unwrap().debug = debug;
+ }
+
pub fn max_connection_retries(&self) -> u32 {
self.config.read().unwrap().max_connection_retries
}
@@ -306,6 +451,25 @@ impl AppState {
}
}
+ /// Upstream URL and headers for the catalog's `ws:/responses` surface.
+ ///
+ /// The WebSocket lives on the same host as the HTTP API and authenticates
+ /// identically — the token response carries no separate websocket endpoint,
+ /// and `endpoints.api` is the only host involved. Discovered by probing;
+ /// GitHub publishes no documentation for the inference API.
+ pub async fn ws_responses_upstream(&self) -> (String, HeaderMap) {
+ let base = self.copilot_base_url();
+ let ws = match base.split_once("://") {
+ Some(("https", rest)) => format!("wss://{rest}"),
+ Some(("http", rest)) => format!("ws://{rest}"),
+ _ => base.clone(),
+ };
+ (
+ format!("{}/responses", ws.trim_end_matches('/')),
+ self.copilot_headers(false).await,
+ )
+ }
+
/// Refreshes the Copilot token if it is missing or within 60 seconds of
/// expiry.
pub async fn ensure_copilot_token(&self) -> Result<(), String> {
@@ -700,7 +864,183 @@ fn premium_multiplier_from_catalog(catalog: &serde_json::Value, model: &str) ->
#[cfg(test)]
mod tests {
use super::*;
+ use futures_util::StreamExt;
use serde_json::json;
+ use tokio::io::AsyncWriteExt;
+
+ #[test]
+ fn parses_a_real_quota_header() {
+ // Captured verbatim from a live Copilot response.
+ let q = parse_quota_snapshot(
+ "ent=-1&ov=0.0&ovPerm=true&rem=100.0&rst=2026-08-01T00%3A00%3A00Z",
+ )
+ .expect("header should parse");
+ assert_eq!(q.entitlement, -1.0);
+ assert!(q.unlimited, "a negative entitlement means unlimited");
+ assert_eq!(q.overage, 0.0);
+ assert!(q.overage_permitted);
+ assert_eq!(q.percent_remaining, 100.0);
+ // The date arrives percent-encoded and must be decoded.
+ assert_eq!(q.reset_date.as_deref(), Some("2026-08-01T00:00:00Z"));
+ }
+
+ #[test]
+ fn parses_a_capped_sku() {
+ let q = parse_quota_snapshot("ent=300&ov=12.5&ovPerm=false&rem=42.5").unwrap();
+ assert_eq!(q.entitlement, 300.0);
+ assert!(!q.unlimited);
+ assert_eq!(q.overage, 12.5);
+ assert!(!q.overage_permitted);
+ assert_eq!(q.percent_remaining, 42.5);
+ assert!(q.reset_date.is_none());
+ }
+
+ #[test]
+ fn unrecognized_quota_headers_are_ignored() {
+ // These headers are undocumented; if the shape changes we must report
+ // nothing rather than a confident zero.
+ assert!(parse_quota_snapshot("").is_none());
+ assert!(parse_quota_snapshot("garbage").is_none());
+ assert!(parse_quota_snapshot("unknown=1&other=2").is_none());
+ // A partially recognizable value still yields what it does contain.
+ assert_eq!(
+ parse_quota_snapshot("rem=7.5&junk=x")
+ .unwrap()
+ .percent_remaining,
+ 7.5
+ );
+ }
+
+ #[test]
+ fn quota_headers_are_recorded_per_sku() {
+ let state = AppState::new(Config::default(), "t".into());
+ assert!(state.quota_snapshot().is_empty());
+
+ let mut h = HeaderMap::new();
+ h.insert(
+ "x-quota-snapshot-chat",
+ HeaderValue::from_static("ent=-1&ov=0.0&ovPerm=false&rem=100.0"),
+ );
+ h.insert(
+ "x-quota-snapshot-premium_interactions",
+ HeaderValue::from_static("ent=300&ov=0.0&ovPerm=true&rem=88.0"),
+ );
+ h.insert(
+ "content-type",
+ HeaderValue::from_static("text/event-stream"),
+ );
+ state.record_quota_headers(&h);
+
+ let q = state.quota_snapshot();
+ assert_eq!(q.len(), 2, "only the quota headers are captured: {q:?}");
+ assert_eq!(q["chat"].percent_remaining, 100.0);
+ assert_eq!(q["premium_interactions"].percent_remaining, 88.0);
+ assert_eq!(q["premium_interactions"].entitlement, 300.0);
+
+ // A later response updates in place.
+ let mut h2 = HeaderMap::new();
+ h2.insert(
+ "x-quota-snapshot-premium_interactions",
+ HeaderValue::from_static("ent=300&ov=0.0&ovPerm=true&rem=87.0"),
+ );
+ state.record_quota_headers(&h2);
+ let q = state.quota_snapshot();
+ assert_eq!(q["premium_interactions"].percent_remaining, 87.0);
+ // Untouched SKUs keep their previous value.
+ assert_eq!(q["chat"].percent_remaining, 100.0);
+ }
+
+ #[test]
+ fn responses_without_quota_headers_leave_the_cache_alone() {
+ let state = AppState::new(Config::default(), "t".into());
+ let mut h = HeaderMap::new();
+ h.insert(
+ "x-quota-snapshot-chat",
+ HeaderValue::from_static("ent=-1&rem=55.0"),
+ );
+ state.record_quota_headers(&h);
+
+ // An upstream that stops sending them must not wipe what we know.
+ state.record_quota_headers(&HeaderMap::new());
+ assert_eq!(state.quota_snapshot()["chat"].percent_remaining, 55.0);
+ }
+
+ /// Serves one request: SSE headers plus half an event, then stalls forever
+ /// without ever closing the socket. Returns the bound address.
+ async fn stalling_upstream() -> String {
+ let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
+ let addr = listener.local_addr().unwrap();
+ tokio::spawn(async move {
+ if let Ok((mut sock, _)) = listener.accept().await {
+ let mut buf = [0u8; 4096];
+ let _ = tokio::io::AsyncReadExt::read(&mut sock, &mut buf).await;
+ let _ = sock
+ .write_all(
+ b"HTTP/1.1 200 OK\r\n\
+ Content-Type: text/event-stream\r\n\
+ Transfer-Encoding: chunked\r\n\r\n",
+ )
+ .await;
+ // A chunk that stops in the middle of a `data:` line.
+ let partial: &[u8] = b"data: {\"partial\":";
+ let _ = sock
+ .write_all(format!("{:X}\r\n", partial.len()).as_bytes())
+ .await;
+ let _ = sock.write_all(partial).await;
+ let _ = sock.write_all(b"\r\n").await;
+ let _ = sock.flush().await;
+ std::future::pending::<()>().await;
+ }
+ });
+ format!("http://{addr}/")
+ }
+
+ /// The streaming paths only notice a dead upstream when the body stream
+ /// yields an error. Without a read timeout a half-open connection yields
+ /// neither data nor error nor end-of-stream, so the request hangs forever
+ /// and none of the stream-interrupted handling ever runs.
+ #[tokio::test]
+ async fn read_timeout_surfaces_a_stalled_upstream_as_an_error() {
+ let url = stalling_upstream().await;
+ let cfg = Config {
+ upstream_read_timeout_seconds: 1,
+ ..Default::default()
+ };
+ let state = AppState::new(cfg, "token".into());
+
+ let resp = state.http.get(&url).send().await.expect("headers arrive");
+ assert!(resp.status().is_success());
+
+ let mut stream = resp.bytes_stream();
+ let first = stream.next().await.expect("first chunk").expect("is data");
+ assert_eq!(&first[..], b"data: {\"partial\":");
+
+ let outcome = tokio::time::timeout(Duration::from_secs(15), stream.next()).await;
+ let item = outcome.expect("must not hang past the read timeout");
+ let err = item.expect("stream must yield an item").unwrap_err();
+ assert!(
+ err.is_timeout(),
+ "expected a timeout error, got: {err:?} ({err})"
+ );
+ }
+
+ /// Setting the timeout to zero disables it, for operators who would rather
+ /// let a stream run indefinitely.
+ #[tokio::test]
+ async fn read_timeout_can_be_disabled() {
+ let url = stalling_upstream().await;
+ let cfg = Config {
+ upstream_read_timeout_seconds: 0,
+ ..Default::default()
+ };
+ let state = AppState::new(cfg, "token".into());
+ let resp = state.http.get(&url).send().await.expect("headers arrive");
+ let mut stream = resp.bytes_stream();
+ let _ = stream.next().await;
+ let outcome = tokio::time::timeout(Duration::from_secs(3), stream.next()).await;
+ assert!(outcome.is_err(), "expected the stream to still be hanging");
+ }
+
#[test]
fn premium_multiplier_reads_billing_from_the_catalog() {
// Shape taken from a real `GET /models` response.
diff --git a/src/store.rs b/src/store.rs
index 6647b04..893239b 100644
--- a/src/store.rs
+++ b/src/store.rs
@@ -34,6 +34,23 @@ pub struct RequestRecord {
pub cache_read_input_tokens: u64,
/// Portion of `input_tokens` this request wrote into the cache.
pub cache_creation_input_tokens: u64,
+ /// Output tokens the model spent reasoning without emitting. Billed, but
+ /// never visible in the answer — which is how a turn can cost a full budget
+ /// and show nothing.
+ #[serde(skip_serializing_if = "Option::is_none")]
+ pub reasoning_tokens: Option,
+ /// What Copilot itself billed, in nano-AI-units, from the `copilot_usage`
+ /// object on the response. Authoritative, unlike `estimated_cost_usd`
+ /// beside it, which is a list-price guess that cannot know a model is
+ /// included at no charge.
+ #[serde(skip_serializing_if = "Option::is_none")]
+ pub billed_nano_aiu: Option,
+ /// What the prompt cache was worth this turn, in nano-AI-units, computed
+ /// from the model's own rates. Negative on a turn that populated a cache
+ /// it did not get to read. `None` when the model reported no rates — not
+ /// every surface charges for the same token types.
+ #[serde(skip_serializing_if = "Option::is_none")]
+ pub cache_saved_nano_aiu: Option,
/// Copilot premium-request cost, from the model catalog's
/// `billing.multiplier`. `None` when the model is unknown or the endpoint
/// carries no premium cost — never guessed as 1.0.
@@ -96,6 +113,19 @@ pub struct RequestRecord {
pub estimated_cost_usd: Option,
}
+impl RequestRecord {
+ /// Whether this attempt never produced a usable answer.
+ ///
+ /// One definition, because three places ask: the failures-only filter, the
+ /// statistics that must exclude these, and the dashboard. A non-2xx status
+ /// is the obvious case; `failure_kind` catches the ones that carry a
+ /// success status but did not succeed — a stream cut off mid-answer, or a
+ /// client that hung up.
+ pub fn failed(&self) -> bool {
+ self.status_code >= 400 || self.failure_kind.is_some()
+ }
+}
+
/// Values for [`RequestRecord::failure_kind`], ordered by how far the request
/// got before dying.
pub mod failure {
@@ -114,13 +144,26 @@ pub mod failure {
/// Aggregate statistics returned by `/api/stats`.
#[derive(Debug, Default, Clone, Serialize)]
pub struct Stats {
+ /// Requests that produced a usable answer. Failed attempts are counted
+ /// separately rather than here: they consume nothing, and folding them in
+ /// would dilute every rate derived from this — a burst of rejected calls
+ /// would look like a collapse in cache hit rate.
pub request_count: u64,
+ /// Attempts that never produced a usable answer.
+ pub failed_requests: u64,
pub total_input_tokens: u64,
pub total_output_tokens: u64,
/// Input tokens served from cache, across every recorded request.
pub total_cache_read_tokens: u64,
/// Input tokens written into the cache, across every recorded request.
pub total_cache_creation_tokens: u64,
+ /// Output tokens spent on reasoning that was never emitted.
+ pub total_reasoning_tokens: u64,
+ /// What Copilot billed in total, in nano-AI-units. Since the move to
+ /// usage-based billing this, not the premium-request count, is the meter.
+ pub total_nano_aiu: u64,
+ /// Net effect of the prompt cache on the bill, in nano-AI-units.
+ pub cache_saved_nano_aiu: i64,
/// Copilot premium requests consumed. Fractional because some models
/// bill at a discounted multiplier.
pub premium_requests: f64,
@@ -153,11 +196,21 @@ impl RequestStore {
/// Records a completed request and updates aggregate statistics.
pub fn add(&self, record: RequestRecord) {
let mut inner = self.inner.lock().unwrap();
- inner.stats.request_count += 1;
+ if record.failed() {
+ inner.stats.failed_requests += 1;
+ } else {
+ inner.stats.request_count += 1;
+ }
+ // Token and billing totals count either way. A stream cut off partway
+ // still consumed what it consumed, and hiding that would understate
+ // the bill.
inner.stats.total_input_tokens += record.input_tokens;
inner.stats.total_output_tokens += record.output_tokens;
inner.stats.total_cache_read_tokens += record.cache_read_input_tokens;
inner.stats.total_cache_creation_tokens += record.cache_creation_input_tokens;
+ inner.stats.total_reasoning_tokens += record.reasoning_tokens.unwrap_or(0);
+ inner.stats.total_nano_aiu += record.billed_nano_aiu.unwrap_or(0);
+ inner.stats.cache_saved_nano_aiu += record.cache_saved_nano_aiu.unwrap_or(0);
// Only count what the catalog actually priced; an unknown multiplier
// is left out rather than assumed to be a full premium request.
if let Some(multiplier) = record.premium_multiplier {
@@ -244,6 +297,9 @@ mod tests {
output_tokens_final: None,
cache_read_input_tokens: 0,
cache_creation_input_tokens: 0,
+ reasoning_tokens: None,
+ billed_nano_aiu: None,
+ cache_saved_nano_aiu: None,
premium_multiplier: None,
upstream_idle_max_ms: None,
session_id: None,
diff --git a/src/util.rs b/src/util.rs
index dad7ae8..a18e4ca 100644
--- a/src/util.rs
+++ b/src/util.rs
@@ -19,9 +19,23 @@ use std::time::Duration;
/// partial character simply waits for the rest of its bytes. Anything left over
/// when the stream ends is returned by [`SseLineBuffer::flush`], so a final
/// event that arrives without a trailing newline is not lost either.
+/// Largest single SSE line the buffer will accumulate before giving up.
+///
+/// Real events are kilobytes; this is far above anything legitimate. It exists
+/// so a broken or hostile upstream that never emits a newline cannot grow the
+/// buffer until the process runs out of memory.
+pub const MAX_SSE_LINE_BYTES: usize = 64 * 1024 * 1024;
+
#[derive(Default)]
pub struct SseLineBuffer {
+ /// Bytes of the line currently being assembled. Never contains a newline:
+ /// `push` always drains past the last one it finds, which is what makes
+ /// "only the newly arrived bytes can complete a line" true.
buf: Vec,
+ /// Set when a single line exceeded [`MAX_SSE_LINE_BYTES`]. The stream can
+ /// no longer be parsed correctly from this point, so callers treat it the
+ /// same as a dropped connection rather than emitting a truncated event.
+ poisoned: bool,
}
impl SseLineBuffer {
@@ -29,20 +43,51 @@ impl SseLineBuffer {
Self::default()
}
+ /// Whether an oversized line forced the buffer to give up. Once true the
+ /// remaining stream is unparseable and the caller should report the
+ /// response as incomplete.
+ pub fn is_poisoned(&self) -> bool {
+ self.poisoned
+ }
+
/// Appends a chunk and returns every complete line it completed, with the
/// trailing `\r` (CRLF streams) already stripped.
pub fn push(&mut self, chunk: &[u8]) -> Vec {
+ if self.poisoned {
+ return Vec::new();
+ }
+ // Whatever is already buffered was searched on an earlier call and, by
+ // the invariant above, holds no newline. So only the bytes that just
+ // arrived can terminate a line, and the search starts where they do —
+ // every byte is examined exactly once across the whole stream.
+ let search_from = self.buf.len();
self.buf.extend_from_slice(chunk);
+
let mut lines = Vec::new();
let mut start = 0usize;
- while let Some(pos) = self.buf[start..].iter().position(|&b| b == b'\n') {
- let end = start + pos;
+ let mut cursor = search_from;
+ while let Some(pos) = self.buf[cursor..].iter().position(|&b| b == b'\n') {
+ let end = cursor + pos;
+ // A line spans from the end of the previous one, which may reach
+ // back into bytes buffered by earlier calls.
lines.push(decode_line(&self.buf[start..end]));
start = end + 1;
+ cursor = start;
}
if start > 0 {
self.buf.drain(..start);
}
+ if self.buf.len() > MAX_SSE_LINE_BYTES {
+ tracing::error!(
+ "[sse] discarding a {} byte line with no newline (limit {} bytes); \
+ treating the stream as broken",
+ self.buf.len(),
+ MAX_SSE_LINE_BYTES
+ );
+ self.buf.clear();
+ self.buf.shrink_to_fit();
+ self.poisoned = true;
+ }
lines
}
@@ -245,6 +290,9 @@ pub async fn post_with_retry(
.await;
match result {
Ok(resp) => {
+ // Every Copilot response carries the current quota, so recording
+ // it here keeps it fresh for free on every request.
+ state.record_quota_headers(resp.headers());
let status = resp.status().as_u16();
// If response is successful or a non-retryable error, return it
if status < 400 || !is_retryable_error(status) {
@@ -306,6 +354,10 @@ pub struct TokenUsage {
/// Portion of `input_tokens` written into the cache by this request.
pub cache_creation_input_tokens: u64,
pub output_tokens: u64,
+ /// Output tokens spent on reasoning the upstream never sends. Billed like
+ /// any other output token, which is what makes a turn that produced no
+ /// visible text still cost something.
+ pub reasoning_tokens: u64,
}
impl TokenUsage {
@@ -330,6 +382,69 @@ impl TokenUsage {
}
}
+/// Copilot's own billing figure for a request, in nano-AI-units, read from the
+/// `copilot_usage` object the upstream attaches to terminal stream events and
+/// non-streaming responses.
+///
+/// This is what GitHub actually charges — since the move to usage-based billing
+/// the meter is token consumption, not premium requests — and it is worth more
+/// than the list-price estimate beside it: the estimate has no way to know that
+/// a model is included at no cost, and prices `gpt-4o-mini` at a few
+/// microdollars where Copilot bills exactly zero.
+///
+/// `total_nano_aiu` is the sum of `token_count * cost_per_batch / batch_size`
+/// over `token_details`, verified against every surface. The total is used
+/// directly rather than recomputed, so a change in how Copilot slices the
+/// details cannot silently skew the number.
+pub fn copilot_billed_nano_aiu(root: &Value) -> Option {
+ root.get("copilot_usage")?.get("total_nano_aiu")?.as_u64()
+}
+
+/// What the prompt cache was worth on this turn, in nano-AI-units.
+///
+/// Copilot states its own per-token rate for every token type it charges, in
+/// `copilot_usage.token_details`, so this is computed from what the model
+/// actually costs rather than from a price list and an assumed discount.
+///
+/// That distinction matters because cache pricing is per model, and the entries
+/// are not the same on every one. Observed on the wire: `claude-haiku-4.5`
+/// reports a `cache_write` rate above its input rate, `gemini-3.5-flash`
+/// reports one of zero, `gpt-5.5` reports no `cache_write` entry at all, and
+/// `gpt-4o-mini` reports zero for everything because Copilot includes it at no
+/// charge. A single hard-coded multiplier would claim savings on a model that
+/// is free.
+///
+/// Positive means the cache paid for itself this turn. Negative is the normal
+/// shape of a first turn, which pays a premium to populate a cache that later
+/// turns read back.
+pub fn cache_saving_nano_aiu(root: &Value, usage: &TokenUsage) -> Option {
+ let details = root
+ .get("copilot_usage")?
+ .get("token_details")?
+ .as_array()?;
+ let rate = |kind: &str| -> Option {
+ details
+ .iter()
+ .find(|d| d.get("token_type").and_then(Value::as_str) == Some(kind))
+ .and_then(|d| {
+ let per_batch = d.get("cost_per_batch")?.as_f64()?;
+ let batch = d.get("batch_size")?.as_f64()?;
+ (batch > 0.0).then_some(per_batch / batch)
+ })
+ };
+ // Without an input rate there is nothing for the cache rates to be cheaper
+ // than, so there is no saving to state.
+ let input = rate("input")?;
+ let mut net = 0.0;
+ if let Some(read) = rate("cache_read") {
+ net += usage.cache_read_input_tokens as f64 * (input - read);
+ }
+ if let Some(write) = rate("cache_write") {
+ net -= usage.cache_creation_input_tokens as f64 * (write - input);
+ }
+ Some(net.round() as i64)
+}
+
/// Reads a `u64` field, treating absent/malformed values as zero.
fn usage_field(usage: &Value, key: &str) -> u64 {
usage.get(key).and_then(|t| t.as_u64()).unwrap_or(0)
@@ -350,6 +465,7 @@ pub fn anthropic_usage(usage: &Value) -> TokenUsage {
cache_read_input_tokens: cache_read,
cache_creation_input_tokens: cache_creation,
output_tokens: usage_field(usage, "output_tokens"),
+ reasoning_tokens: 0,
}
}
@@ -360,6 +476,21 @@ pub fn anthropic_usage(usage: &Value) -> TokenUsage {
/// would double-count. OpenAI has no cache-write concept, so that bucket stays
/// zero.
pub fn openai_usage(usage: &Value) -> TokenUsage {
+ // Where the surface puts `reasoning_tokens` tells you whether it is already
+ // inside `completion_tokens`. o-series report it under
+ // `completion_tokens_details`, as a breakdown of a total that includes it.
+ // The translated Gemini surface reports it at the top level and *excludes*
+ // it — `prompt + completion + reasoning == total_tokens` there.
+ //
+ // `output_tokens` is normalized to the true total either way, for the same
+ // reason `input_tokens` is: a caller comparing two models should not have
+ // to know which convention each one follows.
+ let nested = usage
+ .get("completion_tokens_details")
+ .map(|d| usage_field(d, "reasoning_tokens"))
+ .unwrap_or(0);
+ let flat = usage_field(usage, "reasoning_tokens");
+ let completion = usage_field(usage, "completion_tokens");
TokenUsage {
input_tokens: usage_field(usage, "prompt_tokens"),
cache_read_input_tokens: usage
@@ -367,7 +498,12 @@ pub fn openai_usage(usage: &Value) -> TokenUsage {
.map(|d| usage_field(d, "cached_tokens"))
.unwrap_or(0),
cache_creation_input_tokens: 0,
- output_tokens: usage_field(usage, "completion_tokens"),
+ output_tokens: if nested > 0 {
+ completion
+ } else {
+ completion + flat
+ },
+ reasoning_tokens: if nested > 0 { nested } else { flat },
}
}
@@ -387,6 +523,10 @@ pub fn responses_usage(usage: &Value) -> TokenUsage {
.unwrap_or(0),
cache_creation_input_tokens: 0,
output_tokens: usage_field(usage, "output_tokens"),
+ reasoning_tokens: usage
+ .get("output_tokens_details")
+ .map(|d| usage_field(d, "reasoning_tokens"))
+ .unwrap_or(0),
}
}
@@ -511,6 +651,57 @@ mod tests {
assert!(b.flush().is_none());
}
+ #[test]
+ fn sse_buffer_handles_a_huge_single_event_split_into_many_chunks() {
+ // One 4 MB `data:` line delivered in 4 KB pieces, the shape a very
+ // large tool-call payload or final Responses event takes.
+ let payload = "x".repeat(4 * 1024 * 1024);
+ let line = format!("data: {{\"t\":\"{payload}\"}}\n");
+ let bytes = line.as_bytes();
+
+ let start = std::time::Instant::now();
+ let mut b = SseLineBuffer::new();
+ let mut lines = Vec::new();
+ for chunk in bytes.chunks(4096) {
+ lines.extend(b.push(chunk));
+ }
+ let elapsed = start.elapsed();
+
+ assert_eq!(lines.len(), 1);
+ let v: Value = serde_json::from_str(sse_data(&lines[0]).unwrap()).unwrap();
+ assert_eq!(v["t"].as_str().unwrap().len(), payload.len());
+ assert!(b.flush().is_none());
+
+ // Each byte must be examined a bounded number of times. Rescanning the
+ // whole retained buffer on every chunk makes this quadratic and takes
+ // orders of magnitude longer.
+ assert!(
+ elapsed < std::time::Duration::from_secs(5),
+ "reassembling one large event took {elapsed:?}; the buffer is rescanning"
+ );
+ }
+
+ #[test]
+ fn sse_buffer_gives_up_on_an_unbounded_line_instead_of_growing_forever() {
+ // An upstream that never emits a newline must not be able to grow the
+ // buffer until the process dies.
+ let mut b = SseLineBuffer::new();
+ let mib = vec![b'x'; 1024 * 1024];
+ let mut pushed = 0usize;
+ while !b.is_poisoned() {
+ b.push(&mib);
+ pushed += mib.len();
+ assert!(
+ pushed <= MAX_SSE_LINE_BYTES + 2 * mib.len(),
+ "buffer grew past the cap without giving up"
+ );
+ }
+ assert!(b.is_poisoned());
+ // Once poisoned it stops accumulating entirely.
+ assert!(b.push(b"data: anything\n").is_empty());
+ assert!(b.is_poisoned());
+ }
+
#[test]
fn sse_data_extracts_payload_only() {
assert_eq!(sse_data("data: {}"), Some("{}"));
@@ -591,6 +782,139 @@ mod tests {
assert_eq!(u.output_tokens, 300);
}
+ /// The two surfaces disagree about whether reasoning is inside the
+ /// completion count, so `output_tokens` has to be normalized the way
+ /// `input_tokens` already is. Observed on the wire: a Responses turn
+ /// reported `input 11 + output 17 == total 28` with 10 of that output being
+ /// reasoning, while the translated Gemini surface reported
+ /// `prompt 5 + completion 1 + reasoning 97 == total 103`.
+ #[test]
+ fn reasoning_tokens_normalize_into_the_output_total() {
+ // Nested: already part of `completion_tokens`.
+ let nested = openai_usage(&serde_json::json!({
+ "prompt_tokens": 11,
+ "completion_tokens": 17,
+ "completion_tokens_details": {"reasoning_tokens": 10}
+ }));
+ assert_eq!(nested.output_tokens, 17);
+ assert_eq!(nested.reasoning_tokens, 10);
+
+ // Flat: disjoint from `completion_tokens`, so it has to be added.
+ let flat = openai_usage(&serde_json::json!({
+ "prompt_tokens": 5,
+ "completion_tokens": 1,
+ "total_tokens": 103,
+ "reasoning_tokens": 97
+ }));
+ assert_eq!(flat.output_tokens, 98);
+ assert_eq!(flat.reasoning_tokens, 97);
+ assert_eq!(flat.input_tokens + flat.output_tokens, 103);
+
+ // A turn with no reasoning is unaffected.
+ let none = openai_usage(&serde_json::json!({
+ "prompt_tokens": 10, "completion_tokens": 20
+ }));
+ assert_eq!(none.output_tokens, 20);
+ assert_eq!(none.reasoning_tokens, 0);
+ }
+
+ /// The Responses API states reasoning as a breakdown of `output_tokens`,
+ /// never as an addition to it.
+ #[test]
+ fn responses_usage_keeps_reasoning_inside_the_output_total() {
+ let u = responses_usage(&serde_json::json!({
+ "input_tokens": 11,
+ "output_tokens": 17,
+ "output_tokens_details": {"reasoning_tokens": 10}
+ }));
+ assert_eq!(u.output_tokens, 17);
+ assert_eq!(u.reasoning_tokens, 10);
+ }
+
+ #[test]
+ fn copilot_billing_is_read_from_the_response() {
+ let v = serde_json::json!({
+ "copilot_usage": {"total_nano_aiu": 56_500_000, "token_details": []}
+ });
+ assert_eq!(copilot_billed_nano_aiu(&v), Some(56_500_000));
+ // A model Copilot includes at no charge reports zero, which is a real
+ // figure and must not be confused with "not reported".
+ let free = serde_json::json!({"copilot_usage": {"total_nano_aiu": 0}});
+ assert_eq!(copilot_billed_nano_aiu(&free), Some(0));
+ assert_eq!(copilot_billed_nano_aiu(&serde_json::json!({})), None);
+ }
+
+ /// Builds a `copilot_usage.token_details` block from per-token rates,
+ /// expressed the way Copilot does: a cost per batch plus a batch size.
+ fn details(rates: &[(&str, f64)]) -> Value {
+ let entries: Vec = rates
+ .iter()
+ .map(|(kind, per_token)| {
+ serde_json::json!({
+ "token_type": kind,
+ "batch_size": 1_000.0,
+ "cost_per_batch": per_token * 1_000.0,
+ })
+ })
+ .collect();
+ serde_json::json!({"copilot_usage": {"token_details": entries}})
+ }
+
+ #[test]
+ fn cache_saving_uses_the_rates_the_model_itself_reported() {
+ let usage = TokenUsage {
+ cache_read_input_tokens: 10_000,
+ cache_creation_input_tokens: 2_000,
+ ..Default::default()
+ };
+ // Anthropic-style: reads are a tenth of input, writes carry a premium.
+ let v = details(&[
+ ("input", 100_000.0),
+ ("cache_read", 10_000.0),
+ ("cache_write", 125_000.0),
+ ]);
+ // 10_000 × (100_000 − 10_000) − 2_000 × (125_000 − 100_000)
+ assert_eq!(cache_saving_nano_aiu(&v, &usage), Some(850_000_000));
+ }
+
+ #[test]
+ fn a_model_that_prices_no_cache_writes_is_not_charged_for_them() {
+ let usage = TokenUsage {
+ cache_read_input_tokens: 10_000,
+ // Reported by the surface, but this model publishes no write rate,
+ // so there is nothing to subtract for them.
+ cache_creation_input_tokens: 2_000,
+ ..Default::default()
+ };
+ let v = details(&[("input", 500_000.0), ("cache_read", 50_000.0)]);
+ assert_eq!(cache_saving_nano_aiu(&v, &usage), Some(4_500_000_000));
+ }
+
+ #[test]
+ fn a_model_copilot_includes_at_no_charge_saves_nothing() {
+ let usage = TokenUsage {
+ cache_read_input_tokens: 10_000,
+ ..Default::default()
+ };
+ // Every rate is zero, so caching cannot have saved anything. Reporting
+ // a saving here would invent one from a published price list.
+ let v = details(&[("input", 0.0), ("cache_read", 0.0)]);
+ assert_eq!(cache_saving_nano_aiu(&v, &usage), Some(0));
+ }
+
+ #[test]
+ fn no_reported_rates_means_no_figure_rather_than_zero() {
+ let usage = TokenUsage {
+ cache_read_input_tokens: 10_000,
+ ..Default::default()
+ };
+ // No `token_details` at all, and details that omit the input rate the
+ // others are measured against. Neither can be priced.
+ assert_eq!(cache_saving_nano_aiu(&serde_json::json!({}), &usage), None);
+ let no_input = details(&[("cache_read", 50_000.0)]);
+ assert_eq!(cache_saving_nano_aiu(&no_input, &usage), None);
+ }
+
#[test]
fn idle_tracker_reports_the_longest_upstream_gap() {
let t0 = std::time::Instant::now();