Skip to content

Back cache.rs with real Redis, add degraded-mode fallback and health check - #802

Merged
fejilaup-cloud merged 2 commits into
AtomicIP:mainfrom
williamspatrivk-rgb:fix/786-redis-backed-cache
Aug 25, 2026
Merged

Back cache.rs with real Redis, add degraded-mode fallback and health check#802
fejilaup-cloud merged 2 commits into
AtomicIP:mainfrom
williamspatrivk-rgb:fix/786-redis-backed-cache

Conversation

@williamspatrivk-rgb

Copy link
Copy Markdown
Contributor

Summary

api-server/src/cache.rs documented a "Redis-based caching layer... falling back gracefully... when Redis is unavailable," but had no Redis client at all — a single in-process DashMap backed every code path unconditionally. Behind a load balancer with multiple api-server instances, invalidate_on_contract_event only cleared the calling instance's local entries, so a client whose read landed on a different instance could keep observing stale data indefinitely after a contract event that should have invalidated it.

This backs the existing cache.rs API surface with a real Redis connection pool (redis + r2d2), while preserving every public function signature so call sites elsewhere in the crate (handlers.rs) don't change:

  • get/set/set_with_ttl/exists/ttl_remaining read/write through to Redis (GET/SETEX/EXISTS/TTL) when it's configured and reachable.
  • invalidate/invalidate_prefix/invalidate_pattern issue real DEL / SCAN + DEL against Redis, so an invalidation performed by any instance is immediately visible to every instance sharing that Redis — no pub/sub needed, since all instances read the same shared store rather than keeping a local copy in front of it.
  • Reachability is tracked in an AtomicBool degraded flag. When Redis is unset or unreachable, every operation falls back to the pre-existing in-process DashMap cache — the server always starts and serves correct (if not shared) data. Each transition is logged (tracing::warn!/tracing::info!) and counted via a cache_backend_degraded_transitions_total metric, edge-triggered so it fires once per transition rather than spamming on every failed op.
  • A background thread pings Redis every 10s and flips the flag back once Redis is reachable again, so recovery is automatic.
  • The module doc comment on cache.rs now describes this behavior as it actually works, including the degraded-mode signal. (docs/api-reference.md is the Soroban-contract function reference and never documented the HTTP cache layer, so there was nothing there to correct.)

Tests

  • Existing unit tests updated/kept passing against the in-process fallback path (REDIS_URL unset in the default test run), plus a new test asserting the cache reports itself as degraded — not silently — when Redis isn't configured.
  • Two new Docker-backed integration tests (testcontainers, gated behind the redis-integration-tests Cargo feature so a plain cargo test never needs Docker):
    • tests/cache_redis_cross_instance.rs: writes a value, invalidates it, and asserts it's no longer served — against a real Redis container, proving the invalidation is visible cross-instance (not local-only).
    • tests/cache_redis_fallback.rs: points REDIS_URL at an unreachable port and asserts get/set still work correctly against local memory rather than erroring.
    • Run with: cargo test --features redis-integration-tests --test cache_redis_cross_instance / --test cache_redis_fallback.
    • Each lives in its own integration-test binary because cache's Redis pool is a Lazy that reads REDIS_URL once per process on first access — this is documented in cache.rs and in both test files.

Verification (local, since api-server isn't in this repo's CI)

api-server is currently commented out of the root Cargo.toml workspace ("temporarily excluded: async-graphql version conflict"), so this repo's CI (.github/workflows/ci.yml, cargo build/test --workspace) does not build or test api-server at all, and is unaffected by this change either way.

To verify the change itself, I installed a local Rust toolchain and built/tested api-server directly:

  • cd api-server && cargo test --lib cache:: — all 13 cache unit tests pass.
  • cargo test --features redis-integration-tests --test cache_redis_cross_instance and --test cache_redis_fallback — both pass against a real Redis container (Docker).
  • rustfmt --check on the changed files — clean.
  • cargo clippy --lib --tests --features redis-integration-tests -- -W clippy::all — zero warnings on cache.rs or the new test files (remaining warnings are pre-existing, in unrelated files).
  • Cargo.lock updated for the two new top-level dependencies (redis, r2d2) and two new dev-dependencies (testcontainers, testcontainers-modules).

Not verified, and out of scope for this PR: api-server's main.rs/bin target and several other modules (audit.rs, distributed_tracing.rs, rate_limit.rs) currently fail to compile on main for reasons unrelated to caching — a duplicate-axum-version conflict from async-graphql-axum (the same conflict the workspace comment references), a tokio::sync::RwLock used as if it returns a Result in audit.rs, an opentelemetry::trace::BoxedSpan path that no longer exists in the pinned opentelemetry version, and a JoinHandle moved through a shared reference in a rate_limit.rs test. None of these touch cache.rs, and I left them alone to keep this PR scoped to #786. I confirmed cache.rs itself compiles and its tests pass by building in an isolated worktree with those specific pre-existing issues patched locally (not shipped in this diff), and by re-running against the real main.rs/other modules to confirm the only remaining errors are exactly those pre-existing ones, unrelated to this change.

Closes #786

williamspatrivk-rgb and others added 2 commits August 25, 2026 03:02
…alth check

api-server/src/cache.rs documented a Redis-backed cache with graceful
fallback, but was a pure in-process DashMap with no Redis client at all.
Behind multiple instances, invalidate_on_contract_event only cleared the
calling instance's local entries, so other instances could keep serving
stale data indefinitely after a contract event.

- Back get/set/exists/ttl_remaining/invalidate/invalidate_prefix/
  invalidate_pattern/clear/stats with a real Redis connection pool
  (redis + r2d2) when REDIS_URL is configured and reachable, while
  keeping every public function signature unchanged.
- invalidate/invalidate_prefix/invalidate_pattern issue real DEL /
  SCAN+DEL against Redis, so an invalidation from any instance is
  immediately visible to every instance sharing that Redis.
- Track reachability in an AtomicBool degraded flag: falls back to the
  existing in-process DashMap cache when Redis is unset/unreachable,
  logs and increments a cache_backend_degraded_transitions_total
  counter on each transition (not silently), and auto-recovers via a
  background health-check thread that pings Redis every 10s.
- Add Docker-backed integration tests (testcontainers, gated behind the
  redis-integration-tests feature) proving cross-instance invalidation
  against a real Redis, and the documented fallback path when Redis is
  unreachable.
- Update the cache.rs module doc comment to describe the fallback
  behavior as it now actually works, including the degraded-mode
  signal.

Closes AtomicIP#786
@fejilaup-cloud
fejilaup-cloud merged commit 107c13a into AtomicIP:main Aug 25, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

cache.rs claims Redis-backed caching with fallback; implementation is pure in-process memory

2 participants