Summary
Rate limiting on /passthrough/:provider/*rest is broken along two independent paths:
- RPM path (model layer). The model-layer rate limit — both the inline
rate_limit on a Model and rate_limit_policies with scope: model — is never consulted for passthrough requests. This holds even when the request body names exactly the rate-limited model. Key/team/member request limits do apply.
- TPM path (all layers). Token accounting never happens for passthrough traffic at any layer: a TPM limit configured on the api key (or any other scope) is never tripped by passthrough requests, and passthrough traffic consumes zero tokens from shared buckets. So even the layers whose request limits work on passthrough enforce only their RPM half.
Both failures are silent: configuration is accepted and distributed normally, the dashboard shows the limits as active, and no log or metric indicates that passthrough is exempt. The operator finds out from upstream bills or provider-side 429s.
Reported by a customer running embedding traffic through POST /passthrough/openai/embeddings/multimodals under a model-level 2 RPM / 1,000,000 TPM limit; both paths reproduced on the v0.3.1 release image.
Reproduction (v0.3.1)
Common setup: a ProviderKey with provider: <p>; one Model of that provider (upstream model_name: m); an ApiKey whose allowed_models includes it. Rate-limit windows are fixed per minute; each numbered experiment below runs in a fresh window.
A. Model-layer limits never consulted (RPM path)
Model configured with rate_limit: {rpm: 2, tpm: 1000000}, no key-level limit.
1) POST /passthrough/<p>/v1/chat/completions {"model": "m", ...} x4 -> 200, 200, 200, 200 (never 429)
POST /v1/chat/completions {"model": "m", ...} x3 -> 200, 200, 429 (same window)
The passthrough requests target the exact model the limit is configured on, are never blocked, and are not even counted — the typed requests still get their full 2-request allowance in the same window.
Control: with an api-key-level rate_limit: {rpm: 2}, the 3rd passthrough request returns 429 immediately — key-layer request limiting works on passthrough, so the gap is specific to the model layer.
B. Token usage never accumulates (TPM path)
Api key configured with rate_limit: {tpm: 20} (a layer whose request limiting demonstrably works on passthrough); each chat call consumes ~11 tokens.
2) POST /v1/chat/completions {"model": "m", ...} x3 -> 200, 200, 429 ("token limit exceeded (tokens)")
3) POST /passthrough/<p>/v1/chat/completions {"model": "m", ...} x5 -> 200, 200, 200, 200, 200 (never 429)
Identical body, identical upstream model, identical key and TPM budget: the typed endpoint trips the token limit on the 3rd request; passthrough never trips it no matter how many requests are sent.
Root cause
The two paths fail by different mechanisms in crates/aisix-proxy/src/passthrough.rs.
RPM path: the model identity is never passed to the limiter
// Reserve the rate-limit layers AFTER the input guardrail so a content
// block doesn't burn an RPM slot, matching the typed endpoints. Passthrough
// has no resolved model, so only the api-key/team/member layers apply.
let _reservation = crate::quota::enforce(&state, auth, None).await?;
With model_rl = None, quota::reserve_layers skips Layer 2 (model inline limits) entirely, and Layer 3 policies with PolicyScope::Model can never match (model_rl.is_some_and(...)). None is passed unconditionally, regardless of what the request body contains — hence the bypass covers requests that name the rate-limited model.
The comment is stale: since #449 the dispatch path does resolve a borrowed model entry for the provider segment, and that entry already anchors the client-IP allowlist (#697), the guardrail chain (#911), cooldown marking (#701), and the request timeout (#554). It also does not reflect that the request body is fully buffered at this point (body_bytes, read for the input guardrail scan), so the request's own model field is in principle available.
TPM path: commit_tokens is never called
The limiter is two-phase (crates/aisix-ratelimit/src/limiter.rs): pre_commit reserves a request slot at admission, and the caller reports actual token usage after the upstream response via Reservation::commit_tokens(tokens); dropping the reservation without committing releases it with zero token debit.
Typed endpoints parse the provider response usage and commit it. Passthrough is a raw tunnel: it parses neither request nor response (prompt_tokens = 0 by design, #699), binds the reservation to _reservation, and never calls commit_tokens. Every token bucket it touches — key, team, member, and (if path 1 were fixed) model — therefore stays at zero forever, and token limits can never trip.
Note the two mechanisms are independent: fixing path 1 alone would make model-layer request limits work on passthrough, but the TPM half of any layer stays inert until passthrough has a token-usage source. The response body is in fact fully buffered as well (output guardrails scan it), so usage extraction is not architecturally ruled out — but no such parsing exists today.
Environment
- Reproduced on
api7/aisix:0.3.1 (managed mode, control plane v0.3.1), code references at main (812fc72).
Summary
Rate limiting on
/passthrough/:provider/*restis broken along two independent paths:rate_limiton a Model andrate_limit_policieswithscope: model— is never consulted for passthrough requests. This holds even when the request body names exactly the rate-limited model. Key/team/member request limits do apply.Both failures are silent: configuration is accepted and distributed normally, the dashboard shows the limits as active, and no log or metric indicates that passthrough is exempt. The operator finds out from upstream bills or provider-side 429s.
Reported by a customer running embedding traffic through
POST /passthrough/openai/embeddings/multimodalsunder a model-level 2 RPM / 1,000,000 TPM limit; both paths reproduced on the v0.3.1 release image.Reproduction (v0.3.1)
Common setup: a ProviderKey with
provider: <p>; one Model of that provider (upstreammodel_name: m); an ApiKey whoseallowed_modelsincludes it. Rate-limit windows are fixed per minute; each numbered experiment below runs in a fresh window.A. Model-layer limits never consulted (RPM path)
Model configured with
rate_limit: {rpm: 2, tpm: 1000000}, no key-level limit.The passthrough requests target the exact model the limit is configured on, are never blocked, and are not even counted — the typed requests still get their full 2-request allowance in the same window.
Control: with an api-key-level
rate_limit: {rpm: 2}, the 3rd passthrough request returns 429 immediately — key-layer request limiting works on passthrough, so the gap is specific to the model layer.B. Token usage never accumulates (TPM path)
Api key configured with
rate_limit: {tpm: 20}(a layer whose request limiting demonstrably works on passthrough); each chat call consumes ~11 tokens.Identical body, identical upstream model, identical key and TPM budget: the typed endpoint trips the token limit on the 3rd request; passthrough never trips it no matter how many requests are sent.
Root cause
The two paths fail by different mechanisms in
crates/aisix-proxy/src/passthrough.rs.RPM path: the model identity is never passed to the limiter
With
model_rl = None,quota::reserve_layersskips Layer 2 (model inline limits) entirely, and Layer 3 policies withPolicyScope::Modelcan never match (model_rl.is_some_and(...)).Noneis passed unconditionally, regardless of what the request body contains — hence the bypass covers requests that name the rate-limited model.The comment is stale: since #449 the dispatch path does resolve a borrowed model entry for the provider segment, and that entry already anchors the client-IP allowlist (#697), the guardrail chain (#911), cooldown marking (#701), and the request timeout (#554). It also does not reflect that the request body is fully buffered at this point (
body_bytes, read for the input guardrail scan), so the request's ownmodelfield is in principle available.TPM path:
commit_tokensis never calledThe limiter is two-phase (
crates/aisix-ratelimit/src/limiter.rs):pre_commitreserves a request slot at admission, and the caller reports actual token usage after the upstream response viaReservation::commit_tokens(tokens); dropping the reservation without committing releases it with zero token debit.Typed endpoints parse the provider response
usageand commit it. Passthrough is a raw tunnel: it parses neither request nor response (prompt_tokens = 0by design, #699), binds the reservation to_reservation, and never callscommit_tokens. Every token bucket it touches — key, team, member, and (if path 1 were fixed) model — therefore stays at zero forever, and token limits can never trip.Note the two mechanisms are independent: fixing path 1 alone would make model-layer request limits work on passthrough, but the TPM half of any layer stays inert until passthrough has a token-usage source. The response body is in fact fully buffered as well (output guardrails scan it), so usage extraction is not architecturally ruled out — but no such parsing exists today.
Environment
api7/aisix:0.3.1(managed mode, control plane v0.3.1), code references atmain(812fc72).