Problem
Rotating a Vertex service-account key or an Azure AD client secret does not take effect for up to ~59 minutes. The gateway keeps authenticating upstream with the credential the operator just replaced, and reports success while doing it.
Both token minters cache the minted OAuth token under a key built from the credential's identity fields — which are exactly the fields rotation leaves unchanged:
crates/aisix-provider-vertex/src/token_mint.rs:170 — keyed by sa.client_email only. GCP rotation issues a new private_key under the same client_email, so the rotated SA hits the pre-rotation slot.crates/aisix-provider-azure-openai/src/aad_token_mint.rs:229 — keyed by (tenant_id, client_id) only. AAD secret rotation keeps the same app registration, so the rotated secret hits the pre-rotation slot.
The cached token stays valid for its full TTL (Google/Microsoft both return expires_in: 3600, minus our 60s safety margin), so the stale window is up to ~59 minutes. The other adapters are unaffected: openai/anthropic read api_key per request, and bedrock explicitly parses credentials per request (bridge.rs:839-843).
Why it matters
The point of rotating is usually that the old credential leaked or is being retired. An already-minted access token survives its issuing key's deletion, so during the stale window the gateway is still presenting authority derived from the credential the operator believes is out of use — with no error to signal it.
This is not specific to any one rotation workflow: the CP's documented rotation path today is "create a replacement provider key, repoint models, delete the old key", and that path hits the same cache. A new provider key carrying a fresh service_account_json for the same service account still resolves to the same client_email, so it too gets the old token.
authority_host has the same shape of problem on the Azure side: it selects which AAD cloud minted the token but is absent from the cache key, so two credentials that differ only by sovereign-cloud authority share one slot.
Fix
Key both caches on a fingerprint of the whole credential rather than a subset of its fields. Identical credentials still share a token slot (the intent behind the current key), while any rotated field misses.
Repro
Covered by the regression tests in the fix: with the pre-fix key, cache_remints_after_the_service_account_key_is_rotated sees 1 mint where 2 are expected — the rotated key silently reused the old token.
Found while implementing api7/AISIX-Cloud#1056 (in-place provider key secret update), but independent of it — the bug predates that work and affects the current rotation workflow too.
Problem
Rotating a Vertex service-account key or an Azure AD client secret does not take effect for up to ~59 minutes. The gateway keeps authenticating upstream with the credential the operator just replaced, and reports success while doing it.
Both token minters cache the minted OAuth token under a key built from the credential's identity fields — which are exactly the fields rotation leaves unchanged:
crates/aisix-provider-vertex/src/token_mint.rs:170— keyed bysa.client_emailonly. GCP rotation issues a newprivate_keyunder the sameclient_email, so the rotated SA hits the pre-rotation slot.crates/aisix-provider-azure-openai/src/aad_token_mint.rs:229— keyed by(tenant_id, client_id)only. AAD secret rotation keeps the same app registration, so the rotated secret hits the pre-rotation slot.The cached token stays valid for its full TTL (Google/Microsoft both return
expires_in: 3600, minus our 60s safety margin), so the stale window is up to ~59 minutes. The other adapters are unaffected: openai/anthropic readapi_keyper request, and bedrock explicitly parses credentials per request (bridge.rs:839-843).Why it matters
The point of rotating is usually that the old credential leaked or is being retired. An already-minted access token survives its issuing key's deletion, so during the stale window the gateway is still presenting authority derived from the credential the operator believes is out of use — with no error to signal it.
This is not specific to any one rotation workflow: the CP's documented rotation path today is "create a replacement provider key, repoint models, delete the old key", and that path hits the same cache. A new provider key carrying a fresh
service_account_jsonfor the same service account still resolves to the sameclient_email, so it too gets the old token.authority_hosthas the same shape of problem on the Azure side: it selects which AAD cloud minted the token but is absent from the cache key, so two credentials that differ only by sovereign-cloud authority share one slot.Fix
Key both caches on a fingerprint of the whole credential rather than a subset of its fields. Identical credentials still share a token slot (the intent behind the current key), while any rotated field misses.
Repro
Covered by the regression tests in the fix: with the pre-fix key,
cache_remints_after_the_service_account_key_is_rotatedsees 1 mint where 2 are expected — the rotated key silently reused the old token.Found while implementing api7/AISIX-Cloud#1056 (in-place provider key secret update), but independent of it — the bug predates that work and affects the current rotation workflow too.