MOD-17518 Don't register internal commands as internal on enterprise

[gabsow]

Problem

On an Enterprise database whose shards run in cluster mode, the libmr cluster never forms. Both masters sit in this loop at 1 Hz for as long as the database lives:

# <timeseries> Got bad hello response from 0000...000197 (10.0.101.246:25128), will try again in 1 second, ERR unknown command 'timeseries.HELLO'.

Every coordinated command then dies on the max-idle timeout, which is what fails test_asm_scale_in_out_with_workload (see MOD-17016).

The command is hidden, not missing. In the support packages the initiator gets unknown command from its own ip:port — LibMR deliberately connects to its own node for internal-function fan-out (cluster.c:281 + the lazy connect at cluster.c:252), so the process replying "unknown command" is the process that registered the command.

Cause

isOss is answering two unrelated questions at once — where does the topology come from and how do we authenticate / which commands are visible to us.

Since #98, an Enterprise binary running with cluster-enabled=yes is classified as OSS. That is correct for the topology source (it must read the native cluster view), but it also selected the internal command flag at cluster.c:1902. Meanwhile an Enterprise shard authenticates with the password DMC puts in the long-form CLUSTERSET (ADDR <pwd>@<ip>:<port>), and SendAuthCommandIfNeeded() returns as soon as it has used it — so the connection is a regular one, and internal-flagged commands are invisible on it. The retry path re-sends the same password, so it never recovers. LibMR's own comment at cluster.c:458-461 already describes this outcome.

Which databases are affected

cluster-enabled is per database on Enterprise, not a property of the deployment: RE sets it for the OSS cluster API and/or for ASM. So the trigger is "this database's shards run in cluster mode", and ASM is one route to it, not the only one — a database with the OSS cluster API enabled reaches the same broken combination with no ASM involved.

The support packages I have only contain the ASM route (oss_cluster=disabled, redis_cluster_enabled=enabled), so that is the only one directly observed; the OSS-cluster-API route follows from the same code path and is worth an explicit test.

The correlation in one package is exact: 4 of 344 shard processes log Running mode=cluster → exactly the ASM database's shards → exactly the only ones logging Detected redis oss (cluster-enabled=yes) → exactly the only ones with the HELLO storm. The other 340 log Detected redis enterprise (cluster-enabled=no) and are healthy.

Affected pins: v8.8.2 (a37b672) and master (c6a5ed6) include #98; v8.8.1 (44d5025) does not and is clean.

Change

Keep isOss for the topology source, and decide the connection/visibility model from whether this is an Enterprise binary:

• _proxy-filtered whenever rlec_version is present, regardless of cluster mode; internal only when we will actually authenticate as an internal connection.
• New clusterCtx.useInternalConn, used for the auth mode in SendAuthCommandIfNeeded() and for whether MR_ClusterGetPassword() may drop the password — so the flags and the auth can no longer disagree.
• Prefer the internal secret over a CLUSTERSET password when the commands are internal, since a password cannot make a connection internal.
• Log a failed AUTH. It was fire-and-forget with a NULL callback, which is why half an hour of broken handshake produced no diagnostic beyond unknown command.

CLUSTERSET itself also moves to _proxy-filtered on any Enterprise binary; previously a cluster-mode database exposed it to clients through the proxy.

Because the decision is keyed off the binary rather than the cluster mode, it covers both routes into cluster mode identically.

Effect

On the failing run this is sufficient: the long-form CLUSTERSET already supplies the password, the flags become _proxy-filtered (visible on a password-authenticated connection), HELLO succeeds, the fan-out forms.

Cost: #98's target scenario (an Enterprise binary running a plain OSS cluster in env0) loses the internal hiding — it gets _proxy-filtered and keeps internal-secret AUTH via the n->password == NULL path. Functionally unchanged, slightly less hardened.

Verification

cluster.c compiles clean. The behaviour needs a live cluster-mode Enterprise database to verify: a master shard log should show Detected redis oss (cluster-enabled=yes) and zero Got bad hello response. Today those two are mutually exclusive.

Two things I could not verify locally and would like a second opinion on:

1. That an Enterprise binary accepts _proxy-filtered while running with cluster-enabled=yes. If it does not, that branch should fall back to plain "readonly deny-script".
2. Whether Enterprise ever propagates a shared internal secret to a database's shards. The evidence says no — one package took the short-form path, so MR_ClusterGetPassword() returned NULL, the internal-secret AUTH was sent, and it still failed. If RE can share it, the opposite fix becomes viable (always prefer internal-secret AUTH and keep internal everywhere).

🤖 Generated with Claude Code

---

Note

Medium Risk
Changes cluster handshake, command ACL flags, and AUTH selection across OSS vs Enterprise and cluster-enabled combinations; wrong classification could still break HELLO or expose commands differently on proxy paths.

Overview
Fixes LibMR cluster formation on Enterprise shards with cluster-enabled=yes, where inter-shard HELLO failed with unknown command because commands were registered as internal while shard links authenticated with the CLUSTERSET password (a regular, non-internal connection).

isOss still only picks the topology source (native cluster vs DMC CLUSTERSET). A new commandsAreInternal flag decides whether module commands are registered internal and whether password-based auth is appropriate—true only for plain OSS binaries with internal-secret support, not when rlec_version is present.

On Enterprise binaries, cluster commands (including CLUSTERSET) now use _proxy-filtered instead of internal, so they remain visible on password-authenticated shard connections. SendAuthCommandIfNeeded prefers internal-secret AUTH when commands are internal; internal secret is optional (no assert). Failed AUTH responses are logged via MR_AuthResponseArrived. MR_ClusterGetPassword omits the stored password only when internal connections are required.

^{Reviewed by Cursor Bugbot for commit f7ff12c. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 86c8e8a. Configure here.}