Back cache.rs with real Redis, add degraded-mode fallback and health check - #802
Merged
fejilaup-cloud merged 2 commits intoAug 25, 2026
Merged
Conversation
…alth check api-server/src/cache.rs documented a Redis-backed cache with graceful fallback, but was a pure in-process DashMap with no Redis client at all. Behind multiple instances, invalidate_on_contract_event only cleared the calling instance's local entries, so other instances could keep serving stale data indefinitely after a contract event. - Back get/set/exists/ttl_remaining/invalidate/invalidate_prefix/ invalidate_pattern/clear/stats with a real Redis connection pool (redis + r2d2) when REDIS_URL is configured and reachable, while keeping every public function signature unchanged. - invalidate/invalidate_prefix/invalidate_pattern issue real DEL / SCAN+DEL against Redis, so an invalidation from any instance is immediately visible to every instance sharing that Redis. - Track reachability in an AtomicBool degraded flag: falls back to the existing in-process DashMap cache when Redis is unset/unreachable, logs and increments a cache_backend_degraded_transitions_total counter on each transition (not silently), and auto-recovers via a background health-check thread that pings Redis every 10s. - Add Docker-backed integration tests (testcontainers, gated behind the redis-integration-tests feature) proving cross-instance invalidation against a real Redis, and the documented fallback path when Redis is unreachable. - Update the cache.rs module doc comment to describe the fallback behavior as it now actually works, including the degraded-mode signal. Closes AtomicIP#786
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
api-server/src/cache.rsdocumented a "Redis-based caching layer... falling back gracefully... when Redis is unavailable," but had no Redis client at all — a single in-processDashMapbacked every code path unconditionally. Behind a load balancer with multipleapi-serverinstances,invalidate_on_contract_eventonly cleared the calling instance's local entries, so a client whose read landed on a different instance could keep observing stale data indefinitely after a contract event that should have invalidated it.This backs the existing
cache.rsAPI surface with a real Redis connection pool (redis+r2d2), while preserving every public function signature so call sites elsewhere in the crate (handlers.rs) don't change:get/set/set_with_ttl/exists/ttl_remainingread/write through to Redis (GET/SETEX/EXISTS/TTL) when it's configured and reachable.invalidate/invalidate_prefix/invalidate_patternissue realDEL/SCAN+DELagainst Redis, so an invalidation performed by any instance is immediately visible to every instance sharing that Redis — no pub/sub needed, since all instances read the same shared store rather than keeping a local copy in front of it.AtomicBooldegraded flag. When Redis is unset or unreachable, every operation falls back to the pre-existing in-processDashMapcache — the server always starts and serves correct (if not shared) data. Each transition is logged (tracing::warn!/tracing::info!) and counted via acache_backend_degraded_transitions_totalmetric, edge-triggered so it fires once per transition rather than spamming on every failed op.cache.rsnow describes this behavior as it actually works, including the degraded-mode signal. (docs/api-reference.mdis the Soroban-contract function reference and never documented the HTTP cache layer, so there was nothing there to correct.)Tests
REDIS_URLunset in the default test run), plus a new test asserting the cache reports itself as degraded — not silently — when Redis isn't configured.testcontainers, gated behind theredis-integration-testsCargo feature so a plaincargo testnever needs Docker):tests/cache_redis_cross_instance.rs: writes a value, invalidates it, and asserts it's no longer served — against a real Redis container, proving the invalidation is visible cross-instance (not local-only).tests/cache_redis_fallback.rs: pointsREDIS_URLat an unreachable port and assertsget/setstill work correctly against local memory rather than erroring.cargo test --features redis-integration-tests --test cache_redis_cross_instance/--test cache_redis_fallback.cache's Redis pool is aLazythat readsREDIS_URLonce per process on first access — this is documented incache.rsand in both test files.Verification (local, since api-server isn't in this repo's CI)
api-serveris currently commented out of the rootCargo.tomlworkspace ("temporarily excluded: async-graphql version conflict"), so this repo's CI (.github/workflows/ci.yml,cargo build/test --workspace) does not build or testapi-serverat all, and is unaffected by this change either way.To verify the change itself, I installed a local Rust toolchain and built/tested
api-serverdirectly:cd api-server && cargo test --lib cache::— all 13 cache unit tests pass.cargo test --features redis-integration-tests --test cache_redis_cross_instanceand--test cache_redis_fallback— both pass against a real Redis container (Docker).rustfmt --checkon the changed files — clean.cargo clippy --lib --tests --features redis-integration-tests -- -W clippy::all— zero warnings oncache.rsor the new test files (remaining warnings are pre-existing, in unrelated files).Cargo.lockupdated for the two new top-level dependencies (redis,r2d2) and two new dev-dependencies (testcontainers,testcontainers-modules).Not verified, and out of scope for this PR:
api-server'smain.rs/bin target and several other modules (audit.rs,distributed_tracing.rs,rate_limit.rs) currently fail to compile onmainfor reasons unrelated to caching — a duplicate-axum-version conflict fromasync-graphql-axum(the same conflict the workspace comment references), atokio::sync::RwLockused as if it returns aResultinaudit.rs, anopentelemetry::trace::BoxedSpanpath that no longer exists in the pinnedopentelemetryversion, and aJoinHandlemoved through a shared reference in arate_limit.rstest. None of these touchcache.rs, and I left them alone to keep this PR scoped to #786. I confirmedcache.rsitself compiles and its tests pass by building in an isolated worktree with those specific pre-existing issues patched locally (not shipped in this diff), and by re-running against the realmain.rs/other modules to confirm the only remaining errors are exactly those pre-existing ones, unrelated to this change.Closes #786