Usage-key revocation lags up to 300s on other replicas (authz cache invalidation is per-instance)

[clawdbot-glitch003]

Observed

While testing lit-secrets against prod (api.chipotle.litprotocol.com, 2026-08-27):

1. POST /core/v1/remove_usage_api_key with the master key → 200.
2. Immediately after, POST /core/v1/lit_action with the removed usage key → 200, action executed.
3. ~1–2 minutes later the same call → rejected (HTTP 500, unknown key).

Cause (from reading the code)

accounts::remove_usage_api_key correctly calls blockchain_cache::invalidate_for_keys(api_key, usage_api_key) (lit-api-server/src/accounts/mod.rs:424), but blockchain_cache is an in-process moka cache with a per-key generation counter. The bump only happens on the replica that served the management request; every other replica keeps serving the cached can_execute_action / resolve_to_master result until CACHE_TTL_SECS = 300 expires.

Same applies to every mutation that relies on invalidate_for_account (group/action/PKP changes): consistency is per-instance.

Why it matters

For credential-style products (lit-secrets, lit-triggers agent keys) "revoke" is expected to be immediate. A 5-minute window where a leaked/revoked key still executes actions — and can still Decrypt — is a meaningful gap.

Options

• Shorter TTL for positive usage-key → account resolution (e.g. 30s), keep 300s for group/CID permission lookups.
• Cross-replica invalidation: publish bumps via a shared channel (the Apalis/SQLite job queue won't do; needs Redis/NATS or an on-chain event watcher — the contract already emits events for removeUsageApiKey).
• Cheapest: have the reverse proxy pin management + execution traffic for an API key to one replica (consistent hashing on X-Api-Key hash). Fixes the common case with no code change.

Also worth noting: the rejection surfaces as HTTP 500 rather than 401/403 (related to #604 structured errors).

[GTC6244]

Thanks for the detailed writeup and for reading the code. The mechanism you describe is accurate, but there's one important piece that changes the conclusion.

Cross-replica invalidation does exist (on-chain event watcher)

Your read of blockchain_cache is correct: it's a process-local moka cache keyed by a per-key generation counter (bump_generation), and remove_usage_api_key bumps it only on the serving replica (accounts/mod.rs:424, accounts/blockchain_cache.rs). CACHE_TTL_SECS = 300 is also correct.

However, that TTL is the fallback, not the normal cross-replica bound. There is an on-chain event listener — lit-api-server/src/account_events.rs, started from main.rs — that polls the AccountConfig contract every 10s (EVENT_POLL_INTERVAL) via eth_getLogs for WritesFacet mutation events, including UsageApiKeyRemoved (emitted at WritesFacet.sol:661), and bumps the generation for each affected apiKeyHash. Its module doc calls this out explicitly: it reflects changes "made … by another API-server instance … without waiting out the cache TTL."

So on a healthy fleet the cross-replica revocation bound is roughly block confirmation + up to the 10s poll interval, not 300s. The 300s window only applies when the listener is dead (it has a give-up path after MAX_LISTENER_RETRIES = 5) or for purely out-of-band ChainSecured contract writes.

Your observed ~1–2 min lag is longer than the ~10s the listener should give — that's worth investigating on our side. Likely candidates: the listener lagging/dead at the time of your test (2026-08-27), or block-confirmation delay before the event was queryable. If you can reproduce, that data point is useful.

On the HTTP 500

A cleanly revoked key surfaces as 403, not 500: once canExecuteAction returns false, the handler returns forbidden(..) (core_features.rs:58-63). A 500 only appears when the contract call itself errors (RPC down / revert). If you're seeing 500 for a revoked key, that points at RPC-layer errors rather than the revocation itself — still worth cleaner surfacing (relates to #604).

On the fix options

• Sticky consistent-hash routing per X-Api-Key isn't feasible in this deployment: it's one lit-api-server per Phala CVM behind a pinned dstack-ingress image (no user-editable nginx upstream/ip_hash); multi-replica routing lives in the Phala-managed gateway, outside this repo. It would also defeat horizontal scaling and wouldn't help out-of-band writes.
• The right lever is the listener that already exists. Actionable improvements we'll consider: (a) tighten the poll interval, (b) a shorter positive usage-key→account TTL (there's already a 30s NEGATIVE_CACHE_TTL_SECS for denials), and (c) alerting on the account_event_listener.up gauge hitting 0 so a dead listener (the actual 300s-window condition) is caught.

Re-titling this toward "reduce revocation propagation lag / harden the event listener," since the "no cross-replica invalidation" premise is out of date.