MOD-16399 Skip the rebuild when CLUSTERSET carries an unchanged topology - #104
Conversation
Expose MR_ClusterRefreshTopology(): refreshes the LibMR cluster view in response to a Redis server event instead of requiring a manual REFRESHCLUSTER on every primary. Gated by clusterCtx.isOss so it is a no-op outside OSS cluster mode. A reshard issues CLUSTER SETSLOT per slot, so the topology-change event can fire thousands of times in a burst; refreshing per event would tear down and rebuild every inter-shard connection thousands of times (MR_RefreshClusterData frees the whole cluster). So the refresh is trailing-debounced via an event-loop task: each event only bumps a counter, and a single refresh runs once the counter has been stable for one debounce window. Verified on a live 4-shard reshard: ~8000 events collapse to 1-2 refreshes while cross-shard queries stay complete. Also vendors the RedisModuleEvent_ClusterTopologyChange definitions (event id 20, subevents, info struct) into the local redismodule.h so consumers can subscribe; this mirrors redis/redis#15350 and will be reconciled by the normal redismodule.h sync. Relates-to: MOD-9152, RED-148990 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…esh on in-place reshard MR_ClusterRefreshTopology takes the change_flags reason bitmask from RedisModuleEvent_ClusterTopologyChange and picks the cheapest refresh that stays correct: - FLAG_NODE / FLAG_ROLE / FLAG_STATE (a node joined/left, a role flip, or an OK/FAIL transition -- the set of primaries may have changed) -> full MR_RefreshClusterData (reconnect to the new set of primaries). - FLAG_SLOT only (an in-place reshard: slots moved between primaries we are already connected to) -> new MR_UpdateClusterSlots, which repoints slot->node routing while reusing the existing connections, so in-flight fan-out / cross-shard queries are not aborted and slot-routed queries stay correct mid-reshard. The debounce accumulates the reason flags across the coalesced window and is race-safe: whichever refresh observes a rebuild-worthy flag does the full rebuild, so a membership/role/state change is never masked by the slot-map fast path. MR_UpdateClusterSlots self-upgrades to a full rebuild if an unknown shard appears. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…the reason flags MR_UpdateClusterSlots now reconciles against a fresh CLUSTER SLOTS and rebuilds (which aborts in-flight cross-shard executions) only when the set of slot-serving primaries actually changed -- a primary entered (unknown shard) or left (distinct-count shrank). Otherwise it repoints the slot map in place, preserving connections and in-flight commands. So an over-broad or spurious topology event -- a replica re-pointing, an OK<->FAIL flip, a slotless node joining -- no longer drops in-flight multi-key commands. The debounced handler always calls this single reconcile path; the event's reason flags become advisory (change_flags is no longer consulted). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A cluster-set rebuild drops all inter-shard connections and aborts every in-flight initiator execution with 'cluster topology changed'. The topology is re-broadcast on many events that do not change it (node events, shard reconnects, delivery retries), so multi-shard commands racing such a resend failed spuriously. Compare the incoming long-form command against the stored one (MYID excluded) and keep the current cluster when they match. Short-form commands still rebuild: their topology derives from the server's cluster state, not from the arguments. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
| } | ||
|
|
||
| /* Returns true when the incoming long-form cluster-set command carries the exact | ||
| * topology the current cluster was built from. The MYID slot is excluded: it names |
There was a problem hiding this comment.
Well.. for the same reasons we should do the same for the short-form (which probably means we need to refactor SetClusterDataShortForm() such that the topology data (or at least the part that's relevant for us) is returned and could be released and be compared to the one generated by the next short-form clusterset).
Fortunately, we can skip the case of comparing short to long form topology results, since no version of a module should get both formats.
There was a problem hiding this comment.
Done in 6f7b604. Rather than returning the intermediate node structures from SetClusterDataShortForm, the derived master set is serialized into a canonical snapshot string (one <id> <ip>:<port> <slot-ranges> line per master, sorted by node id so the nodes-list iteration order doesn't matter), stored on the Cluster, and compared against the snapshot freshly derived for the next short-form command. The AUTH password is intentionally not part of the snapshot — it arrives in the arguments, which are compared separately (so a password change still rebuilds). Long-vs-short comparison is naturally excluded by the argc check, per your note.
Verified empirically on a live 2-node OSS cluster: initial short form rebuilds, identical re-send skips, re-send after a CLUSTER SETSLOT move rebuilds (fresh snapshot differs from the stored one), identical re-send skips again. Full pytest suite still passes (34 run, 0 failed).
… skip - ClusterSetIsNewTopology replaces the double-negative ClusterSetCommandIsUnchanged - Arguments are compared with their lengths (memcmp) instead of strcmp - The short form now skips too: the derived master set is snapshotted into a canonical string (sorted by node id), stored on the cluster, and compared against the snapshot derived for the next short-form command Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Review feedback (keep it as simple as possible): every topology-change event now schedules one reconcile task directly on the event loop. MR_UpdateClusterSlots already makes a redundant reconcile harmless (one CLUSTER SLOTS read, connection-preserving unless the primary set changed), so the debounce only saved reconcile churn during legacy per-slot resharding bursts, at the cost of delaying convergence by the quiet window. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The in-place reconcile matched primaries by node id only, so a primary restarting at a new address with the same id (a pod reschedule that keeps nodes.conf) kept its stale ip/port and the reconnect loop redialed the dead address forever. Compare the address from the fresh CLUSTER SLOTS reply (port via RedisModule_GetClusterNodeInfo, matching MR_RefreshClusterData) and escalate to a full rebuild on mismatch. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Per review: drop the CLUSTER SLOTS RM_Call+parse in favor of the same cluster module API the short-form CLUSTERSET builds from (GetClusterNodesList/GetClusterNodeInfo/GetClusterNodeSlotRanges), expressed as a comparable topology-view struct; the decision is a plain master-set compare (ids + addresses). Falls back to a full refresh when the API is unavailable. Trims the over-long comments. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A cluster-set rebuild drops all inter-shard connections and aborts every in-flight initiator execution with 'cluster topology changed'. The topology is re-broadcast on many events that do not change it (node events, shard reconnects, delivery retries), so multi-shard commands racing such a resend failed spuriously. Compare the incoming long-form command against the stored one (MYID excluded) and keep the current cluster when they match. Short-form commands still rebuild: their topology derives from the server's cluster state, not from the arguments. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… skip - ClusterSetIsNewTopology replaces the double-negative ClusterSetCommandIsUnchanged - Arguments are compared with their lengths (memcmp) instead of strcmp - The short form now skips too: the derived master set is snapshotted into a canonical string (sorted by node id), stored on the cluster, and compared against the snapshot derived for the next short-form command Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Both CLUSTERSET forms now feed the same reconcile the topology-change event uses (MR_TryApplyTopologyInPlace): the command arguments (long form) or the server's cluster state (short form) are reduced to a topology view and diffed against the connected shard set. An unchanged set -- including a reshard that only moves slot ranges between the same shards -- is applied in place, keeping every connection, in-flight execution and the run id; a shard entering/leaving the set, or changing its address or credentials, still tears down and rebuilds. This supersedes the short-form string-snapshot compare (removed) and extends the long-form identical-args skip: resharding no longer aborts in-flight multi-shard commands. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
6f7b604 to
fafda41
Compare
|
Restacked per the review direction on #100 (and today's discussion): this PR now sits on top of #100 and moves both CLUSTERSET forms onto the same reconcile mechanism the topology-change event uses, instead of a separate string-snapshot compare. What changed vs. the previous head:
Validated: RTS Note: until #100 merges, this PR's diff-vs-master shows #100's commits too — review just the top commits ( |
The rewritten alternation started at mock shard id '2' -- argument- identical to the topology ShardMock.__enter__ had just applied -- so the identical-args skip made iteration 0 a no-op and GetConnection() blocked forever (45-minute CI cancel, buffered stdout hid the earlier results). Start the alternation at '3' so every iteration changes the shard set relative to the last-applied command, and give GetConnection a finite timeout so a regression fails visibly instead of wedging the job. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
CI hang root-caused (docker repro): the rewritten |
Build long-form CLUSTERSET into a candidate Cluster and feed it through the same MR_UpdateClusterTopologyIfNeeded path used by topology events and short form. Unchanged topology now preserves connections and in-flight executions; changed nodes, slots, credentials, or MYID still rebuild. Keep the existing CLUSTERSET stress test meaningful by alternating shard IDs and add a focused no-op/password-change regression.
Keep the existing short-form and topology-event comparison unchanged. Skip an identical long-form CLUSTERSET by comparing its stored arguments before parsing or rebuilding.
Build long-form CLUSTERSET into a candidate Cluster and feed it through the same MR_UpdateClusterTopologyIfNeeded path used by topology events and short form. Unchanged topology now preserves connections and in-flight executions; changed nodes, slots, credentials, or MYID still rebuild. Keep the existing CLUSTERSET stress test meaningful by alternating shard IDs and add a focused no-op/password-change regression.
Keep the existing short-form and topology-event comparison unchanged. Skip an identical long-form CLUSTERSET by comparing its stored arguments before parsing or rebuilding.
5d6182b to
83a5dea
Compare
Compare the long-form command with the stored arguments before parsing or rebuilding. Keep short-form and topology-event handling unchanged, and preserve the existing cluster for an identical DMC re-broadcast.
83a5dea to
c23cc6a
Compare
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
There are 2 total unresolved issues (including 1 from previous review).
Reviewed by Cursor Bugbot for commit c23cc6a. Configure here.
Reconnect every commit chain removed by the previous force-pushes while retaining the reviewed long-form-only implementation as the final tree.
Use RedisModuleString lengths with memcmp so embedded NUL bytes safely force a rebuild. Add a regression case covering the binary-string mismatch.
master moved SetClusterDataShortForm's topology build into MR_BuildCluster (RedisGears#104/RedisGears#105/RedisGears#117/RedisGears#119), so both guarded hunks moved with it: - the identity guard stays in SetClusterDataShortForm, next to the sibling slot-ranges-API guard; - the endpoint and slot-range guards move into MR_BuildCluster's node loop. Dropped this branch's 'no valid shards' reject: master's coveredSlots != NUMBER_OF_SLOTS check already rejects an under-covered topology, including the empty one.

Summary
CLUSTERSET.MYID; every topology-bearing argument must match.CLUSTERSET, Redis core topology events, and the existing internal cluster comparison unchanged frommaster.The final diff changes two files: 25 production lines plus focused test coverage. The previously published commit history is preserved.
Tests
testMassiveClusterSet: 1,000 alternating changed topologies passed.test_network.py: 19 tests completed, 0 failed.Note
Medium Risk
Touches cluster topology application in libMR; incorrect equality could skip needed rebuilds or miss updates, though comparison is narrow and tests cover identical vs changed args.
Overview
Long-form CLUSTERSET no longer tears down and rebuilds cluster state when the incoming topology matches what is already stored. Before parsing,
IsSameLongFormClusterSetcompares every argument except MYID (index 6) against the savedclusterSetCommand, usingRedisModule_StringPtrLenlengths so embedded NUL bytes are compared correctly. A match logs and returns OK without changing connections or the cluster run ID; any differing argument still goes throughSetClusterDataLongForm. Short-form CLUSTERSET and other topology paths are unchanged.Tests add
testIdenticalLongFormClusterSetIsNoOp(stable run ID on repeat, rebuild on byte-level arg change and shard-id change) and adjusttestMassiveClusterSetto alternate mock shard IDs so repeated updates still exercise rebuilds.Reviewed by Cursor Bugbot for commit 4d1759e. Bugbot is set up for automated code reviews on this repo. Configure here.