ask the gateway for usage on streaming requests, in all five engines

[brentrager]

Pearl th-5e59a5. One request field, five engines — and it invalidates the premise a lot of today's work was built on.

The gateway was never dropping usage

The OpenAI streaming API omits usage unless it is explicitly requested. Nothing here ever requested it. So the missing usage chunk was the gateway correctly honouring a request that never asked.

Verified against llm.smoo.ai (LiteLLM 1.95.0, groq-gpt-oss-120b), same prompt both ways:

	chunks	carrying usage
without stream_options	7	0
with stream_options	8	1

The usage-bearing chunk:

{"id":"chatcmpl-0c33d398-…","choices":[{"index":0,"delta":{}}],
 "usage":{"completion_tokens":8,"prompt_tokens":73,…}}

73 real prompt tokens, where the streaming estimator hardcodes 0.

What was built on the misattribution

Two char-count estimators, prompt_tokens pinned to 0, completion_tokens = content.len() / 4, the usage_estimated and cost_estimated flags, and a cross-language "is this measured?" convention — all compensating for one unset request parameter that nobody had tested.

The comment driving it blamed "LiteLLM at llm.smoo.ai currently drops it for smooth-* aliases" and cited pearl th-eff0d0, which does not exist in the pearl store.

Scope

One field per gateway client, sent only when streaming — meaningless otherwise, and leaving it off keeps a non-streaming request byte-identical to before. Python honours an explicit caller-supplied stream_options rather than overriding it.

The estimators are deliberately left in place: they are now a rare fallback rather than the common path, and deleting them is a separate call once real usage is confirmed flowing in prod.

A second finding from the same experiment — NOT fixed here

I captured the response headers while I was in there. The per-request cost headers ARE present on a streamed response, and all read 0.0:

x-litellm-response-cost-original: 0.0
x-litellm-response-cost-margin-amount: 0.0

So both halves of the open dispute were half right: the headers do survive streaming (they arrive), and the cost is unusable there (nothing is priced at header-flush time). parse_gateway_cost already maps 0 → None, so gateway cost is structurally unavailable on the streaming path, and the response_id → LiteLLM_SpendLogs.request_id join remains the only authoritative per-turn cost there. That makes response_id more valuable, not less.

Verification

Each engine has a test asserting the field reaches the wire, and that it is absent on a non-streaming request. Rust and Go confirmed to fail without the change:

a streaming request must ask for usage or the gateway sends none; got: POST /chat/completions HTTP/1.1

engine	result
Rust	649 passed, 0 failed · clippy -D warnings + fmt --check clean
Go	2 packages ok · go vet clean
Python	439 passed, 1 skipped
TypeScript	423 passed, 1 skipped · tsc --noEmit clean
.NET	412 passed, 0 failed across 2 assemblies

Next

Worth running a real turn after release and checking whether gen_ai_events.usage_input_tokens is finally non-zero — that's the experiment that closes th-5e59a5 and decides how much the provenance work in the other four cores is actually worth.

🤖 Generated with Claude Code

https://claude.ai/code/session_01LkCz96UfUxcai5RU4LwPnG

[changeset-bot]

🦋 Changeset detected

Latest commit: a1bbab9

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 2 packages

Name	Type
@smooai/smooth-operator-core	Minor
@smooai/smooth-operator-temporal	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR