Skip to content

MOD-16399 Skip the rebuild when CLUSTERSET carries an unchanged topology - #104

Merged
gabsow merged 21 commits into
RedisGears:masterfrom
gabsow:mod-16399-skip-identical-clusterset
Jul 27, 2026
Merged

MOD-16399 Skip the rebuild when CLUSTERSET carries an unchanged topology#104
gabsow merged 21 commits into
RedisGears:masterfrom
gabsow:mod-16399-skip-identical-clusterset

Conversation

@gabsow

@gabsow gabsow commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Handle only long-form CLUSTERSET.
  • Before parsing or building cluster structures, compare the incoming arguments with the command stored on the current cluster.
  • Ignore the command name and MYID; every topology-bearing argument must match.
  • Preserve the current cluster, connections, and run ID for an identical command. Any changed argument follows the existing full rebuild path.
  • Leave short-form CLUSTERSET, Redis core topology events, and the existing internal cluster comparison unchanged from master.

The final diff changes two files: 25 production lines plus focused test coverage. The previously published commit history is preserved.

Tests

  • Identical long-form no-op test: passed.
  • testMassiveClusterSet: 1,000 alternating changed topologies passed.
  • Full standalone test_network.py: 19 tests completed, 0 failed.
  • RedisTimeSeries integration coverage: RedisTimeSeries#2088.

Note

Medium Risk
Touches cluster topology application in libMR; incorrect equality could skip needed rebuilds or miss updates, though comparison is narrow and tests cover identical vs changed args.

Overview
Long-form CLUSTERSET no longer tears down and rebuilds cluster state when the incoming topology matches what is already stored. Before parsing, IsSameLongFormClusterSet compares every argument except MYID (index 6) against the saved clusterSetCommand, using RedisModule_StringPtrLen lengths so embedded NUL bytes are compared correctly. A match logs and returns OK without changing connections or the cluster run ID; any differing argument still goes through SetClusterDataLongForm. Short-form CLUSTERSET and other topology paths are unchanged.

Tests add testIdenticalLongFormClusterSetIsNoOp (stable run ID on repeat, rebuild on byte-level arg change and shard-id change) and adjust testMassiveClusterSet to alternate mock shard IDs so repeated updates still exercise rebuilds.

Reviewed by Cursor Bugbot for commit 4d1759e. Bugbot is set up for automated code reviews on this repo. Configure here.

gabsow and others added 4 commits June 24, 2026 13:17
Expose MR_ClusterRefreshTopology(): refreshes the LibMR cluster view in response
to a Redis server event instead of requiring a manual REFRESHCLUSTER on every
primary. Gated by clusterCtx.isOss so it is a no-op outside OSS cluster mode.

A reshard issues CLUSTER SETSLOT per slot, so the topology-change event can fire
thousands of times in a burst; refreshing per event would tear down and rebuild
every inter-shard connection thousands of times (MR_RefreshClusterData frees the
whole cluster). So the refresh is trailing-debounced via an event-loop task: each
event only bumps a counter, and a single refresh runs once the counter has been
stable for one debounce window. Verified on a live 4-shard reshard: ~8000 events
collapse to 1-2 refreshes while cross-shard queries stay complete.

Also vendors the RedisModuleEvent_ClusterTopologyChange definitions (event id
20, subevents, info struct) into the local redismodule.h so consumers can
subscribe; this mirrors redis/redis#15350 and will be reconciled by the normal
redismodule.h sync.

Relates-to: MOD-9152, RED-148990

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…esh on in-place reshard

MR_ClusterRefreshTopology takes the change_flags reason bitmask from
RedisModuleEvent_ClusterTopologyChange and picks the cheapest refresh that stays
correct:

- FLAG_NODE / FLAG_ROLE / FLAG_STATE (a node joined/left, a role flip, or an
  OK/FAIL transition -- the set of primaries may have changed) -> full
  MR_RefreshClusterData (reconnect to the new set of primaries).
- FLAG_SLOT only (an in-place reshard: slots moved between primaries we are already
  connected to) -> new MR_UpdateClusterSlots, which repoints slot->node routing
  while reusing the existing connections, so in-flight fan-out / cross-shard queries
  are not aborted and slot-routed queries stay correct mid-reshard.

The debounce accumulates the reason flags across the coalesced window and is
race-safe: whichever refresh observes a rebuild-worthy flag does the full rebuild,
so a membership/role/state change is never masked by the slot-map fast path.
MR_UpdateClusterSlots self-upgrades to a full rebuild if an unknown shard appears.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…the reason flags

MR_UpdateClusterSlots now reconciles against a fresh CLUSTER SLOTS and rebuilds (which
aborts in-flight cross-shard executions) only when the set of slot-serving primaries
actually changed -- a primary entered (unknown shard) or left (distinct-count shrank).
Otherwise it repoints the slot map in place, preserving connections and in-flight
commands. So an over-broad or spurious topology event -- a replica re-pointing, an
OK<->FAIL flip, a slotless node joining -- no longer drops in-flight multi-key commands.

The debounced handler always calls this single reconcile path; the event's reason
flags become advisory (change_flags is no longer consulted).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A cluster-set rebuild drops all inter-shard connections and aborts every
in-flight initiator execution with 'cluster topology changed'. The topology
is re-broadcast on many events that do not change it (node events, shard
reconnects, delivery retries), so multi-shard commands racing such a resend
failed spuriously. Compare the incoming long-form command against the stored
one (MYID excluded) and keep the current cluster when they match. Short-form
commands still rebuild: their topology derives from the server's cluster
state, not from the arguments.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@gabsow
gabsow requested a review from galcohen-redislabs July 6, 2026 09:01
Comment thread src/cluster.c Outdated
Comment thread src/cluster.c
Comment thread src/cluster.c Outdated
}

/* Returns true when the incoming long-form cluster-set command carries the exact
* topology the current cluster was built from. The MYID slot is excluded: it names

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well.. for the same reasons we should do the same for the short-form (which probably means we need to refactor SetClusterDataShortForm() such that the topology data (or at least the part that's relevant for us) is returned and could be released and be compared to the one generated by the next short-form clusterset).
Fortunately, we can skip the case of comparing short to long form topology results, since no version of a module should get both formats.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 6f7b604. Rather than returning the intermediate node structures from SetClusterDataShortForm, the derived master set is serialized into a canonical snapshot string (one <id> <ip>:<port> <slot-ranges> line per master, sorted by node id so the nodes-list iteration order doesn't matter), stored on the Cluster, and compared against the snapshot freshly derived for the next short-form command. The AUTH password is intentionally not part of the snapshot — it arrives in the arguments, which are compared separately (so a password change still rebuilds). Long-vs-short comparison is naturally excluded by the argc check, per your note.

Verified empirically on a live 2-node OSS cluster: initial short form rebuilds, identical re-send skips, re-send after a CLUSTER SETSLOT move rebuilds (fresh snapshot differs from the stored one), identical re-send skips again. Full pytest suite still passes (34 run, 0 failed).

… skip

- ClusterSetIsNewTopology replaces the double-negative ClusterSetCommandIsUnchanged
- Arguments are compared with their lengths (memcmp) instead of strcmp
- The short form now skips too: the derived master set is snapshotted into a
  canonical string (sorted by node id), stored on the cluster, and compared
  against the snapshot derived for the next short-form command

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
gabsow and others added 6 commits July 13, 2026 12:00
Review feedback (keep it as simple as possible): every topology-change
event now schedules one reconcile task directly on the event loop.
MR_UpdateClusterSlots already makes a redundant reconcile harmless (one
CLUSTER SLOTS read, connection-preserving unless the primary set
changed), so the debounce only saved reconcile churn during legacy
per-slot resharding bursts, at the cost of delaying convergence by the
quiet window.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The in-place reconcile matched primaries by node id only, so a primary
restarting at a new address with the same id (a pod reschedule that
keeps nodes.conf) kept its stale ip/port and the reconnect loop redialed
the dead address forever. Compare the address from the fresh CLUSTER
SLOTS reply (port via RedisModule_GetClusterNodeInfo, matching
MR_RefreshClusterData) and escalate to a full rebuild on mismatch.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Per review: drop the CLUSTER SLOTS RM_Call+parse in favor of the same
cluster module API the short-form CLUSTERSET builds from
(GetClusterNodesList/GetClusterNodeInfo/GetClusterNodeSlotRanges),
expressed as a comparable topology-view struct; the decision is a plain
master-set compare (ids + addresses). Falls back to a full refresh when
the API is unavailable. Trims the over-long comments.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A cluster-set rebuild drops all inter-shard connections and aborts every
in-flight initiator execution with 'cluster topology changed'. The topology
is re-broadcast on many events that do not change it (node events, shard
reconnects, delivery retries), so multi-shard commands racing such a resend
failed spuriously. Compare the incoming long-form command against the stored
one (MYID excluded) and keep the current cluster when they match. Short-form
commands still rebuild: their topology derives from the server's cluster
state, not from the arguments.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… skip

- ClusterSetIsNewTopology replaces the double-negative ClusterSetCommandIsUnchanged
- Arguments are compared with their lengths (memcmp) instead of strcmp
- The short form now skips too: the derived master set is snapshotted into a
  canonical string (sorted by node id), stored on the cluster, and compared
  against the snapshot derived for the next short-form command

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Both CLUSTERSET forms now feed the same reconcile the topology-change
event uses (MR_TryApplyTopologyInPlace): the command arguments (long
form) or the server's cluster state (short form) are reduced to a
topology view and diffed against the connected shard set. An unchanged
set -- including a reshard that only moves slot ranges between the same
shards -- is applied in place, keeping every connection, in-flight
execution and the run id; a shard entering/leaving the set, or changing
its address or credentials, still tears down and rebuilds.

This supersedes the short-form string-snapshot compare (removed) and
extends the long-form identical-args skip: resharding no longer aborts
in-flight multi-shard commands.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@gabsow
gabsow force-pushed the mod-16399-skip-identical-clusterset branch from 6f7b604 to fafda41 Compare July 13, 2026 16:27
@gabsow

gabsow commented Jul 13, 2026

Copy link
Copy Markdown
Contributor Author

Restacked per the review direction on #100 (and today's discussion): this PR now sits on top of #100 and moves both CLUSTERSET forms onto the same reconcile mechanism the topology-change event uses, instead of a separate string-snapshot compare.

What changed vs. the previous head:

  • The short-form ShortFormTopologySnapshot string compare is gone — the short form now reduces the server's cluster state to the shared comparable view (TopologyViewEntry via BuildClusterApiView, from MOD-16382 Add MR_ClusterRefreshTopology for event-driven OSS topology refresh #100) and reconciles semantically.
  • The long form keeps the identical-args fast path, and any changed args are first reconciled through the same mechanism: a reshard that only moves slot ranges between the same shards is now applied in place — connections, in-flight executions and the run id survive (previously any non-identical payload rebuilt). A shard entering/leaving the set, or changing its address or credentials, still tears down and rebuilds.
  • Tests updated to the new spec + new coverage: moved-boundary → in-place (run id + live connection survive); re-send of moved args → no-op; changed shard id / changed password → rebuild; testMassiveClusterSet now alternates the shard id so it still stresses 1000 genuine rebuilds.

Validated: RTS test_asm 5/5 (incl. short-form path) and the MOD-16399 flow test (identical → run id kept; changed address → rebuilt) against this head.

Note: until #100 merges, this PR's diff-vs-master shows #100's commits too — review just the top commits (57a2dd1, e8a3578, fafda41).

Comment thread src/cluster.c Outdated
Comment thread src/cluster.c Outdated
The rewritten alternation started at mock shard id '2' -- argument-
identical to the topology ShardMock.__enter__ had just applied -- so the
identical-args skip made iteration 0 a no-op and GetConnection() blocked
forever (45-minute CI cancel, buffered stdout hid the earlier results).
Start the alternation at '3' so every iteration changes the shard set
relative to the last-applied command, and give GetConnection a finite
timeout so a regression fails visibly instead of wedging the job.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@gabsow

gabsow commented Jul 13, 2026

Copy link
Copy Markdown
Contributor Author

CI hang root-caused (docker repro): the rewritten testMassiveClusterSet alternation started at shard id '2' — argument-identical to the topology __enter__ had just applied — so the identical-args skip made iteration 0 a no-op and GetConnection() blocked forever on its timeout-less queue (the 45-min silent cancel). Fixed in 1b5d6c3: alternation starts at '3', and GetConnection now has a finite timeout so any such regression fails fast instead of wedging the job. The skip behaved correctly — the test's phase was wrong.

Comment thread src/cluster.c
gabsow added 2 commits July 27, 2026 13:09
Build long-form CLUSTERSET into a candidate Cluster and feed it through the same MR_UpdateClusterTopologyIfNeeded path used by topology events and short form. Unchanged topology now preserves connections and in-flight executions; changed nodes, slots, credentials, or MYID still rebuild.

Keep the existing CLUSTERSET stress test meaningful by alternating shard IDs and add a focused no-op/password-change regression.
Comment thread src/mr.c
gabsow added 3 commits July 27, 2026 13:41
Keep the existing short-form and topology-event comparison unchanged. Skip an identical long-form CLUSTERSET by comparing its stored arguments before parsing or rebuilding.
Build long-form CLUSTERSET into a candidate Cluster and feed it through the same MR_UpdateClusterTopologyIfNeeded path used by topology events and short form. Unchanged topology now preserves connections and in-flight executions; changed nodes, slots, credentials, or MYID still rebuild.

Keep the existing CLUSTERSET stress test meaningful by alternating shard IDs and add a focused no-op/password-change regression.
Keep the existing short-form and topology-event comparison unchanged. Skip an identical long-form CLUSTERSET by comparing its stored arguments before parsing or rebuilding.
@gabsow
gabsow force-pushed the mod-16399-skip-identical-clusterset branch from 5d6182b to 83a5dea Compare July 27, 2026 10:47
Compare the long-form command with the stored arguments before parsing or rebuilding. Keep short-form and topology-event handling unchanged, and preserve the existing cluster for an identical DMC re-broadcast.
@gabsow
gabsow force-pushed the mod-16399-skip-identical-clusterset branch from 83a5dea to c23cc6a Compare July 27, 2026 10:55

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

There are 2 total unresolved issues (including 1 from previous review).

Fix All in Cursor

Reviewed by Cursor Bugbot for commit c23cc6a. Configure here.

Comment thread src/cluster.c Outdated
gabsow added 2 commits July 27, 2026 17:16
Reconnect every commit chain removed by the previous force-pushes while retaining the reviewed long-form-only implementation as the final tree.
Use RedisModuleString lengths with memcmp so embedded NUL bytes safely force a rebuild. Add a regression case covering the binary-string mismatch.
Comment thread src/cluster.c Outdated
@gabsow
gabsow merged commit c6a5ed6 into RedisGears:master Jul 27, 2026
7 checks passed
gabsow added a commit to gabsow/LibMR that referenced this pull request Jul 30, 2026
master moved SetClusterDataShortForm's topology build into MR_BuildCluster
(RedisGears#104/RedisGears#105/RedisGears#117/RedisGears#119), so both guarded hunks moved with it:

- the identity guard stays in SetClusterDataShortForm, next to the sibling
  slot-ranges-API guard;
- the endpoint and slot-range guards move into MR_BuildCluster's node loop.

Dropped this branch's 'no valid shards' reject: master's coveredSlots !=
NUMBER_OF_SLOTS check already rejects an under-covered topology, including
the empty one.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants