Add MIGRATE_MODULE_PROPAGATE_END event to propagate module commands at the end of slot migration - #472
Open
sundb wants to merge 270 commits into
Open
Add MIGRATE_MODULE_PROPAGATE_END event to propagate module commands at the end of slot migration#472sundb wants to merge 270 commits into
sundb wants to merge 270 commits into
Conversation
Add key allocation sizes histograms based on previous memory accounting work in redis#14363 and redis#14451. The histograms are exposed via `INFO keysizes` and use logarithmic (power-of-2) bins, similar to current key sizes/length histogram implementation in the following fields: db0_distrib_lists_sizes:1=...,2=...,4=... db0_distrib_sets_sizes:1=...,2=...,4=... db0_distrib_hashes_sizes:1=...,2=...,4=... db0_distrib_zsets_sizes:1=...,2=...,4=... To avoid confusion with existing distrib_strings_sizes histograms which are based on string lengths we don't report allocation sizes histograms for strings. So far per key and per slot memory accounting code has been relying type specific functions (hashTypeAllocSize(), listTypeAllocSize(), zsetAllocSize(), etc) for computing data structure allocation sizes since it's faster and we only need to track size deltas and not the complete allocation size along with the kvobj and key length overhead. In order to keep the allocation sizes histogram consistent, memory accounting code has been switched to use kvobjAllocSize() instead which does return the total allocation size. Note that the feature is enabled with `key-bytes-stats` or `cluster-slot-stats` config in redis config file on startup.
…dis#14725) ## Summary Adds missing IDMP configuration parameters to redis.conf, previously ommitted in redis#14615 ## Changes - Added `stream-idmp-duration` configuration parameter with documentation - Added `stream-idmp-maxsize` configuration parameter with documentation - Both parameters were already implemented in the code (src/config.c, src/server.h) but were missing from redis.conf ## Configuration Parameters ### stream-idmp-duration - **Purpose**: Duration (in seconds) to remember IDMP identifiers for duplicate detection - **Range**: 1 to 86400 seconds (1 second to 24 hours) - **Default**: 100 seconds - **Modifiable**: Yes, via CONFIG SET at runtime ### stream-idmp-maxsize - **Purpose**: Maximum number of IDMP identifiers to track per producer per stream - **Range**: 1 to 10000 entries - **Default**: 100 entries - **Modifiable**: Yes, via CONFIG SET at runtime
…ock is available (redis#14713) This PR reduces per-command `ustime()` syscalls in `call()` by reusing cached time and batching wall-clock updates when HW monotonic time is available. ### What changed - Pass `server.ustime` to `enterExecutionUnit()` instead of calling `ustime()`. - Use HW monotonic clock to measure duration and accumulate it across commands. - Refresh cached time with `ustime()` only when accumulated duration > **10µs** or after **25 commands**. - Fallback to direct `ustime()` when HW monotonic clock isn’t available. ### Impact - `ustime` CPU: **4.58% → 0.25%**, which leads to ~4% boost on max QPS ### Notes - Time drift is bounded (≤10µs or 25 commands). - No behavior change on non-HW-monotonic systems. --------- Co-authored-by: Yuan Wang <yuan.wang@redis.com>
…redis#14727) ### Summary Adds `expired_keys_active` and `expired_subkeys_active` counters to track keys and hash fields expired by the active expiration cycle, distinguishing them from lazy expirations. These new metrics are exposed in INFO stats output. ### Motivation Currently, Redis tracks the total number of expired keys (expired_keys) and expired hash fields (expired_subkeys), but there's no way to differentiate between expirations triggered by active expire and lazy expire. --------- Co-authored-by: Moti Cohen <moti.cohen@redis.com>
in redis#14440, we remove the refcount check in [tryDeferFreeClientObject](redis@235e688#diff-252bce0cc340542712f0c1adf62e9035ea47a4a064321fbf40ec3dd4b814aaf2R1509), it is ok in 8.4 version, since after command execution, the refcount of a kvobject always is 1. but in redis#14608 (8.6 RC1) we change this assumption, increment refcount when a client refer a kvobject in reply, so now if the refcount of kvobject is more than 1, we may let the io thread call `decrRefCount`, there is data race, maybe it causes memory leak.
Adds startup-time security warnings when the default user permits unauthenticated access, with behavior dependent on protected-mode and bind settings. Warnings are skipped in Sentinel mode since it intentionally disables protected-mode by design. - No password + no protected-mode + no bind: warn about accepting connections from any IP/interface - No password + no protected-mode: warn about accepting connections from any IP on configured interface - No password + protected-mode enabled: warn about accepting connections from local clients
…ter (redis#14742) Some hotkeys cpu metrics display time in milliseconds others in microseconds. Change the metrics showing time of command executions to all use microseconds and use the `-us` postfix to show that. Also, disable the `SLOTS` param for `HOTKEYS START` if we are not in cluster mode.
…is#14739) Optimizes handling of clients with referenced replies by embedding the `pending_ref_reply_node` list node in `client` and avoiding per-operation node alloc/free. there is an improvement: ~2% on 4 and 16 io-threads. ~1% on 8 io-threads
This pull request vectorizes the 8-bit quantization vector-search path in a similar was as the non-quantization path. The assembly intrinsics are a bit more complicated than in the non-quantization path, since we are operating on 8-bit integers and we need to worry about preventing overflow. Thus, after loading the 8-bit integers, they are extended into 16-bits before multiplying and accumulating into 32-bit integers. --------- Co-authored-by: debing.sun <debing.sun@redis.com>
## Overview
This PR optimizes Redis Streams consumer group performance by replacing
the `pel_by_time` rax tree with a doubly-linked list, delivering
significant performance improvements for NACK updates and XREADGROUP
CLAIM operations while also reducing memory usage.
## The Problem
Consumer groups maintain a time-ordered index of pending entries using a
radix tree (`pel_by_time`). Every time a pending entry is reclaimed or
delivered, we need to update its delivery time, which currently
requires:
```c
raxRemovePelByTime(group->pel_by_time, old_time, &id); // O(k) where k=key length
nack->delivery_time = current_time;
raxInsertPelByTime(group->pel_by_time, current_time, &id); // O(k) where k=key length
```
## The Key Insight
**99% of delivery_time updates set the value to the current time** —
which means they're appending to the tail of a time-ordered structure.
We're using a radix tree (O(k) operations where k is key length, plus
tree traversal overhead) for what is essentially an append-only workload
(should be O(1)).
## The Solution
Replace the rax tree with a doubly-linked list embedded directly in each
`streamNACK`:
```c
typedef struct streamNACK {
mstime_t delivery_time;
uint64_t delivery_count;
streamConsumer *consumer;
listNode *cgroup_ref_node;
streamID id; // NEW
struct streamNACK *pel_prev; // NEW
struct streamNACK *pel_next; // NEW
} streamNACK;
```
Now updating a NACK becomes:
```c
pelListUpdate(group, nack, current_time); // O(1): unlink + append
```
## Why This Works
**Typical case (99%):** Delivery time = current time
- Unlink from current position: O(1) — just update 2-4 pointers
- Append to tail: O(1) — update tail pointer and link
**Edge case (1%):** XCLAIM with explicit past IDLE time
- Still handled correctly by `pelListInsertSorted()` which scans
backward from tail
- Rare enough that O(N) worst case doesn't matter
## Memory Reduction
The linked list approach uses less memory than the rax tree:
**What we add:**
- 3 new fields in `streamNACK`: `id` (16 bytes) + `pel_prev` (8 bytes) +
`pel_next` (8 bytes) = 32 bytes per entry
**What we remove:**
- Entire `pel_by_time` rax tree with its node overhead (~40-50 bytes per
entry)
**Net result:** Lower memory footprint per pending entry, plus better
cache locality from eliminating the separate tree structure.
## Performance Impact
### Theoretical Analysis
| Operation | Before | After |
|-----------|--------|-------|
| NACK update | O(k) × 2 + tree overhead | O(1) |
| CLAIM iteration | O(k) per entry + traversal | O(1) per entry |
*k = key length (32 bytes: timestamp + stream ID)*
For a consumer group with 10,000 pending entries claiming 100 oldest:
- **Before:** Tree traversal + key comparisons for each operation
- **After:** Simple pointer updates
**Key Findings:**
- **28% higher throughput** for XREADGROUP with CLAIM
- **22% lower average latency** (0.195ms → 0.152ms)
- **21% lower P99 latency** (0.212ms → 0.168ms)
- XADD performance unchanged (69K ops/sec both implementations)
The reason for the failure is that when starting server with bind *, the host will be set to *. At this time, when reconnect, the client will not recognize this host. So this fix skipped checking whether the server was ready.
…inary vector distance (redis#13962) This PR replaces the manual `popcount64()` implementation with `__builtin_popcountll()` for computing Hamming distance in binary vectors, when the underlying hardware supports the `POPCNT` instruction. The built-in version simplifies the code and enables the compiler to emit a single `POPCNT` instruction on supported CPUs, which is significantly faster than the manual bitwise method. You can verify the difference here: [https://godbolt.org/z/TxWMcE8M3](https://godbolt.org/z/TxWMcE8M3) — the manual version generates a long sequence of instructions (approximately 34 on modern HW) vs 1 instruction (popcnt) when using __builtin_popcountll() ## Portability across platforms This change maintains full portability across platforms and compilers. The use of `__builtin_popcountll()` is guarded by the `HAVE_POPCNT` macro, which is defined only when the compiler supports the target("popcnt") attribute. At runtime, we also check `__builtin_cpu_supports("popcnt")` to ensure the hardware provides support for the instruction. If not available, the implementation safely falls back to the original manual `popcount64()` logic. --------- Co-authored-by: debing.sun <debing.sun@redis.com>
…edis#14749) Follow redis#14680 Reply of `HOTKEYS GET` is an unordered collection of key-value pairs. It is more reasonable to be a map in resp3 instead of flat array.
…4751) Add LTRIM/LREM and RM_StringTruncate() memory tracking tests.
redis#14492) This PR adds SIMD vectorization for binary quantization distance calculation, similar to PR redis#14474. --------- Co-authored-by: debing.sun <debing.sun@redis.com>
This PR continues the work redis#14645, to further ensure sensitive user data is not exposed in logs when hide_user_data_from_log is enabled. - Redact empty key notices during RDB load. - Redact key names in eviction/expiration debug logs. - Block DEBUG SCRIPT output and suppress raw string dump in crash object debug when redaction is enabled. - Redact malformed MODULE LOAD argument snippets and unresolved module configuration logs. - Redact empty key notices during RDB load. - Redact key names during Lua globals allow‑list warnings.
Fix HOTKEYS to track each command in a MULTI/EXEC block.
… (redis#14729) Fix RDB Channel connections mistakenly discovered by Sentinel During fullsync, if the main replication connection is interrupted, but the rdbchannel connection is still active, it will be visible in the "info replication" output. Currently, the rdbchannel connection does not send `REPLCONF ip-address`, and in a meshed scenario, when the source IP addresses of both connections differ, Sentinel will treat them as separate replicas. This commit adds `REPLCONF ip-address` to rdbchannel replica handshake if `server.slave_announce_ip` is enabled. fixes: redis#14728 Co-authored-by: Ozan Tezcan <ozantezcan@gmail.com>
…alid slots (redis#14761) - When passing slots not within the range of a node to `HOTKEYS START SLOTS ...` the hotkey command now returns error. - Changed the cmd tips for the HOTKEYS subcommands so that they reflect the special nature of the cmd in cluster mode - i.e command should be issued against a single node only. Clients should not care about cluster management and aggregation of results. - Change reply schema to return Array of the maps. For a single node this will return array of 1 element. Getting results from multiple nodes will make it easy to concatenate the elements into one array.
…scope for vulnerability reports (redis#14747)
<!-- CURSOR_SUMMARY --> > [!NOTE] > **Low Risk** > Simple version bumps in build configuration; main risk is upstream module behavior/compatibility changes when building `v8.6.0`. > > **Overview** > Bumps the pinned build versions of the Redis data type modules `redisbloom`, `redisjson`, and `redistimeseries` from `v8.5.90` to `v8.6.0` via their Makefiles. > > <sup>Written by [Cursor Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit f1319fa. This will update automatically on new commits. Configure [here](https://cursor.com/dashboard?tab=bugbot).</sup> <!-- /CURSOR_SUMMARY -->
Key in the result of HOTKEYS `sampled-command-selected-slots-us` changed to `sampled-commands-selected-slots-us` in order to be aligned with the other key names.
# Summary Refactors IDMP producer defragmentation to use the generic `defragRadixTree` helper function instead of manually implementing radix tree iteration, improving code consistency and maintainability. ## Changes * Replaced manual rax iteration in `defragStreamIdmpProducers` with `defragRadixTree` call * Added `defragIdmpProducerCallback` function following the `raxDefragFunction` signature pattern * Removed unnecessary wrapper function and inlined the call directly at the call site * Made the callback function static since it's only used within `defrag.c` ## Benefits **Code Consistency:** Now follows the same pattern used by other stream defrag functions (`defragStreamConsumer`, `defragStreamConsumerGroup`, etc.) **Reduced Duplication:** Eliminates 20+ lines of boilerplate rax iteration code by leveraging the existing `defragRadixTree` helper **Maintainability:** All rax defragmentation logic (tree struct, head node, internal nodes, data pointers) is now centralized in `defragRadixTree` **Correctness:** Uses the proven, well-tested radix tree defragmentation logic consistently across all stream-related structures
- After redis#14440, the IO thread can parse all command in pipeline instead of only the header, so we should prefetch all pending commands when prefetching. - Since the main thread will access the `io_deferred_objects`, we should prefetch them. Up to 8% improvement for 100% writing scenarios, about 2% for other scenarios
Before this PR, `pipe2()` is only enabled on Linux and FreeBSD while `pipe2()` is available on *BSD. This PR enables `pipe2()` for the rest of *BSD: DragonFlyBSD, NetBSD and OpenBSD. - [pipe2 on DraonFlyBSD](https://man.dragonflybsd.org/?command=pipe§ion=2) - [__DragonFly_version for pipe2](https://github.com/DragonFlyBSD/DragonFlyBSD/blob/7485684fa5c3fadb6c7a1da0d8bb6ea5da4e0f2f/sys/sys/param.h#L121) - [pipe2 on NetBSD](https://man.netbsd.org/pipe.2) - [pipe2 on OpenBSD](https://man.openbsd.org/pipe.2)
Currently, redis uses select(2) on DragonFlyBSD while `kqueue` is available on DragonFlyBSD since FreeBSD 4.1, and DragonFlyBSD was originally forked from FreeBSD 4.8 `select(2)` is a pretty old technique that has many defects compared to `kqueue`, we should switch to `kqueue` on DragonFlyBSD. References: [DragonflyBSD - kqueue](https://man.dragonflybsd.org/?command=kqueue§ion=2) --------- Signed-off-by: Andy Pan <i@andypan.me> Co-authored-by: debing.sun <debing.sun@redis.com>
Prevent unsigned integer underflow when `mem_total` exceeds `zmalloc_used` in `getMemoryOverheadData()`. This can occur during early startup or due to timing mismatches in memory sampling. Add conditional check to set dataset to 0 when underflow would occur, matching the existing pattern used for `net_usage` calculation.
Currently, the SFLUSH command is permitted only when the specified slot range fully covers all slot ranges owned by the node. We want to enable it to perform partial slot flushes instead, and the reply should be list of ranges that were flushed. for implementation, we use TRIMSLOTS functionalities (added as part of ASM work) to support flush async. **NOTE**: Redis will reply `-TRYAGAIN Slot is being trimmed` error if clients send write commands to given slots during SFLUSH execution, clients should not send write commands to these slots before getting reply of SFLUSH command. And there is still an issue if we use active trim for `SFLUSH ASYNC` redis#14750 (comment), so we still mark this command `experimental`, we may remove this tag when we deprecate active slot trimming or don't support ASYNC option. --------- Co-authored-by: Ozan Tezcan <ozantezcan@gmail.com>
Use rax memory accounting for RedisModuleDict. Suggested-by: debing.sun <debing.sun@redis.com>
…nabled (redis#14786) Don't call kvobjAllocSize() in t_stream.c if memory tracking is not enabled. Reported-by: Sergei Georgiev <s_ggeorgiev@yahoo.com>
…s#15263) ## Summary The cluster bus PING/PONG/MEET packet parser validated extension padding and total length but never checked that string-carrying extensions are properly null-terminated, allowing a crafted packet to trigger out-of-bounds reads when the payload is later consumed as a C string. 1. **Null-termination check for hostname and human-nodename extensions (`cluster_legacy.c`)** Added a check inside the existing extension-validation loop in `clusterProcessPacket`: for `CLUSTERMSG_EXT_TYPE_HOSTNAME` and `CLUSTERMSG_EXT_TYPE_HUMAN_NODENAME` extension types, it verifies that the data portion is non-empty (`datalen > 0`) and that the last byte is `'\0'`. Packets failing this check are rejected with a warning log and an early return, the same way other malformed-extension cases are handled. 2. **Test (`hostnames.tcl`)** A new test exercises the rejection path by constructing a raw cluster-bus PING packet with a 32-byte hostname extension that contains no `'\0'`, sending it directly to a node's bus port, and verifying the packet is dropped (warning logged, hostname not updated in `CLUSTER NODES`). Two helper procs (`build_cluster_bus_ping` and `build_hostname_extension`) build the binary packet from scratch in Tcl, allowing fine-grained control over extension contents without needing a modified Redis sender.
…is#14704) Avoid zmalloc_size() in kvobjAllocSize() and use approximation instead. Since for ongoing key allocation histograms work (redis#14695) we need to call kvobjAllocSize() more often on hot paths, using zmalloc_size() would cause unnecessary performance overhead.
…s#15262) Fixes redis#15250 ## Summary When `redis-cli --cluster rebalance` is invoked with `--user <username>` but without `-a <password>`, the MIGRATE command constructed by redis-cli contains an extra unfilled argv slot that gets serialized as an empty string. The server interprets this empty string as a key with slot 0, triggering a CROSSSLOT error when it conflicts with the actual keys' slot. ## Root Cause In `src/redis-cli.c`, function `clusterManagerMigrateKeysInReply`: ```c int c = (replace ? 8 : 7); if (config.conn_info.auth) c += 2; if (config.conn_info.user) c += 1; // BUG: adds 1 for user even when auth is NULL size_t argc = c + reply->elements; ``` When `config.conn_info.user` is set but `config.conn_info.auth` is NULL, `c` is incremented by 1 for the user parameter. However, the argv filling logic later only sets the user inside the `if (config.conn_info.auth)` block (using AUTH2 with both user and password). This mismatch causes: - `argc` is 1 larger than the actual number of argv entries filled - The unfilled argv slot is serialized as `$0\r\n\r\n` (empty string) - Server's `migrateGetKeys` treats the empty string as a key → `keyHashSlot("",0)` returns slot 0 → CROSSSLOT ## Fix Consolidate the two separate increments into one: ```diff - if (config.conn_info.auth) c += 2; - if (config.conn_info.user) c += 1; + if (config.conn_info.auth) + c += config.conn_info.user ? 3 : 2; ``` This is consistent with the argv filling logic where both AUTH and AUTH2 cases are handled inside a single `if (config.conn_info.auth)` block. --------- Co-authored-by: 2030XiaoGe <2030XiaoGe@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 <noreply@owtffssent.com> Co-authored-by: debing.sun <debing.sun@redis.com>
Fixes redis#15269. This keeps `VADD ... CAS SETATTR` in sync with the normal insert path by incrementing `numattribs` when a CAS insert actually stores an attribute. Before this change, the CAS path attached the attribute to the node but left the set-wide counter unchanged. That made `VINFO` under-report `attributes-count`, and if the set-wide count stayed at zero, RDB save skipped attribute serialization for the whole vector set. --------- Co-authored-by: debing.sun <debing.sun@redis.com>
## Issue The Codecov workflow failed while running `codecov/codecov-action@57e3a136b779b570ffcdbf80b3bdc90e7fab3de2` (`v6`) because the downloaded Codecov CLI could not be verified with GPG. ```text Run codecov/codecov-action@57e3a13 ==> Running Action version 6.0.0 ... gpg: Can't check signature: No public key ==> Could not verify signature. Please contact Codecov if problem continues Exiting... ``` ## Change Update the Codecov upload step to use `codecov/codecov-action v7.0.0`, which includes the updated wrapper/signature verification behavior.
…#15308) ## Summary The stream `entries_added` field is stored as `uint64_t` but is consumed as `long long` in multiple downstream code paths - lag computation (`streamReplyWithCGLag`), distance estimation (`streamEstimateDistanceFromFirstEverEntry`), and `XINFO STREAM` output (`addReplyLongLong`). The same problem applies to each consumer group's `entries_read` field. A crafted RDB or RESTORE payload could set either field to a value exceeding `LLONG_MAX`, causing sign-flip in lag calculations and incorrect negative values reported to clients. 1. **Overflow and invariant check for `entries_added`** Added a validation in `rdbLoadObject` during stream loading that rejects payloads where `entries_added` exceeds `LLONG_MAX` (would overflow when cast to signed) or is less than the stream's current `length` (violates the invariant that total-ever-added must be >= live entries). This mirrors the same invariant already enforced by `xsetidCommand` and prevents corrupt values from propagating into lag arithmetic or AOF rewrites (which also narrow to `long long` via `rioWriteBulkLongLong`). 2. **Overflow and invariant check for consumer group `entries_read`** Added a validation during consumer group loading that rejects payloads where a group's `entries_read` exceeds `LLONG_MAX` (would overflow when cast to signed) or is greater than the stream's `entries_added` (a group cannot have read more entries than were ever added). The check skips the sentinel `SCG_INVALID_ENTRIES_READ` value, which legitimately represents an unknown lag. This prevents corrupt `entries_read` values from propagating into lag arithmetic. 3. **Tests** - A new test sends a hand-crafted RESTORE payload containing a valid 3-entry stream whose `entries_added` field is patched to `0xFFFFFFFF00000000` (exceeds `INT64_MAX`). The test disables checksum validation and `sanitize-dump-payload` to reach the RDB object loader, then verifies both the client-facing error (`Bad data format`) and the server log message (`Stream entries_added inconsistent with length`). - A second test sends a hand-crafted RESTORE payload containing a valid 2-entry stream with a consumer group whose `entries_read` field is patched to `0xFFFFFFFF00000000` (exceeds `INT64_MAX` and the stream's `entries_added`). It similarly verifies the client-facing error (`Bad data format`) and the server log message (`Stream cgroup entries_read inconsistent with entries_added`).
## Implement SUNIONCARD command Add a new `SUNIONCARD` command that returns the cardinality (number of distinct elements) of the union of multiple sets without returning the elements themselves. **Syntax:** `SUNIONCARD numkeys key [key ...] [APPROX] [LIMIT limit]` - **Exact mode (default):** Builds a temporary union set (reusing `sunionDiffGenericCommand` with new `cardinality_only` and `limit` parameters) and returns its size. Supports `LIMIT` with true early termination — stops adding elements as soon as cardinality reaches the limit. - **Approximate mode (`APPROX`):** Uses Redis's existing HyperLogLog internals (`createHLLObject`/`hllAdd`/`hllCount`) to estimate the union cardinality without materializing the full union. Standard HLL error of ~0.81%. With `LIMIT`, the HLL estimate is checked after processing sufficient elements to cross the boundary (calculated based on how far the current estimate is below LIMIT) for early termination. ## Changes - **`src/t_set.c`**: Extended `sunionDiffGenericCommand` with `cardinality_only`, `approx`, and `limit` parameters (following the same pattern as `sinterGenericCommand/SINTERCARD`). The `SET_OP_UNION` loop now accumulates into either a temporary set (exact) or a `HyperLogLog` (approx), with early termination once limit is reached, plus a `cardinality_only` reply branch that returns the count. All existing callers (`SUNION`, `SUNIONSTORE`, `SDIFF`, `SDIFFSTORE`, `SPOP`) pass `0, 0, 0` — no behavior change. Implemented `sunioncardCommand`: it parses arguments and delegates to `sunionDiffGenericCommand`, with `approx` selecting the HLL-based estimate. - **`src/hyperloglog.h`** (new): Extracted `struct hllhdr` from `hyperloglog.c` and exposes `createHLLObject`, `hllAdd`, `hllCount` so `t_set.c` can use the HLL API. --------- Co-authored-by: debing.sun <debing.sun@redis.com>
## Summary Bumps the pinned `MODULE_VERSION` for **RedisTimeSeries** from `v8.8.0` to `v8.9.80`. RedisBloom and RedisJSON are unchanged. | Module | From | To | |-----------------|---------|----------| | RedisTimeSeries | v8.8.0 | v8.9.80 | > **Note:** `v8.9.80` is the **8.10 LTS branch cut** (targets Redis 8.10) and has diverged from `v8.8.0`. The `v8.8.0...v8.9.80` ancestry diff contains 37 commits, but **7 are backports already shipped in `v8.8.0`** (verified with `git cherry`, see below); the **30 genuinely new** changes are listed here. ## Module changes ([v8.8.0 → v8.9.80](RedisTimeSeries/RedisTimeSeries@v8.8.0...v8.9.80)) - MOD-14124 — m commands acl ([redis#1913](RedisTimeSeries/RedisTimeSeries#1913)) - MOD-14250 — missing artifacts logs fix ([redis#1932](RedisTimeSeries/RedisTimeSeries#1932)) - MOD-14289 — Increase LibMR execution timeout for Valgrind builds ([redis#1933](RedisTimeSeries/RedisTimeSeries#1933)) - MOD-12726 — unique snapshot name + new output params for nighly event ([redis#1937](RedisTimeSeries/RedisTimeSeries#1937)) - MOD-14674 — oss refresh list ([redis#1935](RedisTimeSeries/RedisTimeSeries#1935)) - nightly build, upload snapshot artifact ([redis#1954](RedisTimeSeries/RedisTimeSeries#1954)) - MOD-14902 — existence check to prevent Timeseries crash when Redis is not aligned with new redis-core changes ([redis#1940](RedisTimeSeries/RedisTimeSeries#1940)) - MOD-14850 — Bump libmr ([redis#1962](RedisTimeSeries/RedisTimeSeries#1962)) - MOD-14715 — add apt retries/timeouts to Debian/Ubuntu Dockerfiles ([redis#1965](RedisTimeSeries/RedisTimeSeries#1965)) - MOD-15262 — Align TimeSeries with the RedisModule_GetUserUserName API changes ([redis#1975](RedisTimeSeries/RedisTimeSeries#1975)) - RED-180951 RED-180027 — fixing bugs and improving the code ([redis#2004](RedisTimeSeries/RedisTimeSeries#2004)) - MOD-14239 — Ignore known init-rdbchannel migration timeout under sanitizer ([redis#1979](RedisTimeSeries/RedisTimeSeries#1979)) - MOD-15105 — make one generic getUser function ([redis#1981](RedisTimeSeries/RedisTimeSeries#1981)) - MOD-15741 MOD-14091 — add dont-cache command tip to non-cacheable read commands ([redis#2031](RedisTimeSeries/RedisTimeSeries#2031)) - MOD-15731 — modules modernization alignment ([redis#2027](RedisTimeSeries/RedisTimeSeries#2027)) - MOD-15728 — Tal.ba/feat/blocking get ([redis#2028](RedisTimeSeries/RedisTimeSeries#2028)) - MOD-8187 — fix the aggregation bugs (fix reverse iterator) ([redis#2036](RedisTimeSeries/RedisTimeSeries#2036)) - MOD-15749 — Link to new version of LibMR with short-form CLUSTERSET and add tests ([redis#2034](RedisTimeSeries/RedisTimeSeries#2034)) - Fix REDISMODULE_MAIN typo + drop common.h forward-decl workarounds ([redis#2044](RedisTimeSeries/RedisTimeSeries#2044)) - CI: switch flow-linux arm64 runner to standard ubuntu-24.04-arm ([redis#2048](RedisTimeSeries/RedisTimeSeries#2048)) - MOD-15999 — Centralize Redis ref into a single source-of-truth file for CI ([redis#2045](RedisTimeSeries/RedisTimeSeries#2045)) - CI: trigger OSS cluster (multi-shard) benchmarks on demand ([redis#2050](RedisTimeSeries/RedisTimeSeries#2050)) - MOD-15899 — oom try calloc ([redis#2051](RedisTimeSeries/RedisTimeSeries#2051)) - MOD-8187 — fix non twa aggregation ([redis#2053](RedisTimeSeries/RedisTimeSeries#2053)) - MOD-16224 — Fix filter-blind neighbor lookups in EMPTY aggregation ([redis#2056](RedisTimeSeries/RedisTimeSeries#2056)) - MOD-15893 — Nrange new command ([redis#2052](RedisTimeSeries/RedisTimeSeries#2052)) - MOD-16160 — update blocking get syntax ([redis#2054](RedisTimeSeries/RedisTimeSeries#2054)) - MOD-16160 — Fix client-side cache tips for TS.BGET and TS.NRANGE/TS.NREVRANGE ([redis#2057](RedisTimeSeries/RedisTimeSeries#2057)) - Cut 8.10 LTS branch: bump version to 8.9.80, target Redis 8.10 ([27484f7](RedisTimeSeries/RedisTimeSeries@27484f7)) - Fix event-nightly CI on 8.10: read Redis ref from dedicated redis_ref field ([redis#2058](RedisTimeSeries/RedisTimeSeries#2058)) <details> <summary>7 commits excluded — already shipped in v8.8.0 as backports</summary> These show up in the `v8.8.0...v8.9.80` ancestry diff with different SHAs, but `git cherry` confirms their patch content is already in `v8.8.0`: they were backported to the 8.8 release branch (the "in 8.8 via" PR below), so they are **not** new in this bump. - MOD-9320 — Multiple aggregators single command ([redis#1916](RedisTimeSeries/RedisTimeSeries#1916)) → in 8.8 via [redis#1921](RedisTimeSeries/RedisTimeSeries#1921) - MOD-9320 — rename AGGREGATION token in m commands ([redis#1926](RedisTimeSeries/RedisTimeSeries#1926)) → in 8.8 via [redis#1927](RedisTimeSeries/RedisTimeSeries#1927) - MOD-13142 — Allow changing number of threads while pool hasn't started ([redis#1928](RedisTimeSeries/RedisTimeSeries#1928)) → in 8.8 via [redis#1934](RedisTimeSeries/RedisTimeSeries#1934) - MOD-14439 — Detect cluster topology changes during a multi-shard command ([redis#1930](RedisTimeSeries/RedisTimeSeries#1930)) → in 8.8 under the same PR - ci: skip Event CI when PR only changes src/version.h ([redis#1984](RedisTimeSeries/RedisTimeSeries#1984)) → in 8.8 via [redis#1986](RedisTimeSeries/RedisTimeSeries#1986) - MOD-15020 — remove bionic os support ([redis#1983](RedisTimeSeries/RedisTimeSeries#1983)) → in 8.8 via [redis#1996](RedisTimeSeries/RedisTimeSeries#1996) - MOD-14420 — fix count reducers return wrong NaN ([redis#2013](RedisTimeSeries/RedisTimeSeries#2013)) → in 8.8 via [redis#2016](RedisTimeSeries/RedisTimeSeries#2016) </details> ## Test plan - [ ] CI passes for RedisTimeSeries at the new pinned version - [ ] `make all` builds RedisTimeSeries cleanly against `unstable` - [ ] Module tests run under `make test` Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…tick (redis#15330) ## Motivation `getAllocatorFragmentation()` was called twice per `serverCron` tick: once by `cronUpdateMemoryStats()` for `INFO MEMORY`, then again by `computeDefragCycles()` for the threshold decision. Each call forces a jemalloc epoch refresh (cross-thread sync via IPI) plus a per-arena small-bin scan. The second call's result was almost always discarded by the threshold check. ## Change `struct defragFragCache` on `redisServer` (`src/server.h`) caches the Lua-arena-subtracted `(frag_pct, frag_bytes)` plus `server.cronloops` at the moment of measurement. - `defragFragCachePut(frag_bytes, allocated)` — called from `cronUpdateMemoryStats()` (`src/server.c`) after the existing arena walk; mirrors `getAllocatorFragmentation()`'s Lua subtraction. - `defragFragCacheTake(&pct, &bytes)` — called at the top of `computeDefragCycles()` (`src/defrag.c`); returns hit only when the recorded `cronloops` matches `server.cronloops` (same tick). Miss → falls through to a fresh `getAllocatorFragmentation()` call. Freshness gate is `server.cronloops`: it advances at the top of `serverCron()` and `whileBlockedCron()`, so any value published in a previous tick is automatically stale on the next `Take`. No explicit invalidate call needed. `initServerConfig()` seeds `cronloops = -1` so the first `Take` is a clean miss. `HAVE_DEFRAG=0` builds get no-op `Put`/`Take` stubs so the unconditional call site in `cronUpdateMemoryStats()` always links. ## Observability `DEBUG DEFRAG-FRAG-CACHE-STATS` exposes `defrag_frag_cache_hits` (testing/diagnostic surface only; not in `INFO`).
Add a new CMD_SCRIPT_RUNNER flag (bit 30) to identify commands that execute scripts or functions. The flag is applied to EVAL, EVALSHA, EVAL_RO, EVALSHA_RO, FCALL, and FCALL_RO. These marker help clients better understand command semantics, particularly in the context of Client Side Caching: clients can use the SCRIPT_RUNNER flag to decide which commands should not have their results cached, since script execution and consumer group reads are inherently stateful and non-repeatable.
Jira: https://redislabs.atlassian.net/browse/MOD-16291 Updates the bundled RediSearch module version used by Redis module builds from `v8.8.0` to `v8.9.80`. This picks up the tagged RediSearch 8.9.80 release. Validation: - Checked that `v8.9.80` exists in `RediSearch/RediSearch` and resolves to `280c3ce1bd65fd7306cae75a1d54bce3a05518e4`. - Checked the diff is limited to `modules/redisearch/Makefile`. - Ran `git diff --check`. <!-- CURSOR_SUMMARY --> --- > [!NOTE] > **Low Risk** > Single Makefile version pin with no application logic changes; main risk is behavioral differences in the upstream RediSearch 8.9.80 release at runtime. > > **Overview** > **RediSearch module version bump** from `v8.8.0` to `v8.9.80` via `MODULE_VERSION` in `modules/redisearch/Makefile`, so Redis module builds fetch and compile the tagged 8.9.80 release instead of 8.8.0. > > No other build flags or paths change in this PR. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 7bd4875. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup> <!-- /CURSOR_SUMMARY -->
…le (redis#15305) Corrects command introspection metadata for SET and DELEX so IFDEQ / IFDNE digest operands are documented as string, not integer.
…on (redis#15327) Adds two `RedisModuleEvent_ForkChild` subevents so a multi-threaded module can bring its background threads to a safe point before `fork()`: - `REDISMODULE_SUBEVENT_FORK_CHILD_PRE` — fired synchronously on the main thread just before `fork()`. The handler runs synchronously, so returning from it acknowledges the module is ready to fork. - `REDISMODULE_SUBEVENT_FORK_CHILD_CANCELLED` — fired if a fork preceded by `_PRE` did not happen (`fork()` failed), so the module can resume. On success the module resumes on the existing `FORK_CHILD_BORN`. `redisFork()` fires `_PRE` before `fork()` and `_CANCELLED` on `fork()` failure. ## Why Between `fork()` and `exec()` only async-signal-safe functions are safe to call. A multi-threaded module (search, timeseries/LibMR, gears…) whose background thread holds a lock — e.g. the allocator (malloc arena) lock — at the instant of `fork()` leaves the child holding that lock with no owning thread, so the child **deadlocks** the first time it takes it (e.g. during the RDB save it forked for). `_PRE` lets the module quiesce its threads to a safe, lock-free point first. Existing `FORK_CHILD` subscribers are unaffected — they ignore the new subevents (default case); no existing subevent value or the event id changes. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Ozan Tezcan <ozantezcan@gmail.com>
…edis#15242) ## What this PR does Adds a new module API, `RedisModule_AddPostNotificationJobForKey`, alongside the existing `RedisModule_AddPostNotificationJob`. Both schedule work to run after a keyspace-notification (KSN) handler returns (so the handler stays free of write side-effects), but the new API binds each job to a specific key and differs from the existing one in three ways: | | `AddPostNotificationJob` (existing) | `AddPostNotificationJobForKey` (new) | |---|---|---| | **Callback** | `(ctx, pd)` | `(ctx, key, pd)` — receives the bound key. | | **Firing schedule** | Once, at the end of the outermost execution unit (`firePostExecutionUnitJobs`). | At the tail of **every `call()`** — so each MULTI/EXEC and EVAL/FCALL sub-command is observed in turn — **plus** during AOF replay after each replayed command, **plus** the end-of-execution-unit drain (for standalone commands). | | **When allowed** | Refused while `server.loading` or on a read-only replica. | Permitted during AOF replay and on replicas (see below). Refused if called **outside a KSN handler**. | Motivation: RED-197766. A module needs to react to each step of an `HSET` + `HEXPIRE` sequence on the same hash inside a `MULTI/EXEC` block — the existing API can't deliver that because it only fires once at the end of EXEC. ## Design: one queue, distinguished by the bound key Per-key jobs share the existing post-notification queue (`modulePostExecUnitJobs`) rather than living in a separate one. A queued job (`RedisModulePostExecUnitJob`) carries an owned `key` reference and a callback union: - **Regular job** (`RM_AddPostNotificationJob`): `key == NULL`, `cb.callback` is used. Fires once at the end of the outermost execution unit and may write to the keyspace. - **Per-key job** (`RM_AddPostNotificationJobForKey`): `key != NULL`, `cb.key_callback` is used and receives the bound key. Fires between sub-commands and during AOF replay, and may **not** write to the keyspace. `key != NULL` is the sole discriminator — no separate type or flag. ### Firing sites `firePerKeyJobsBetweenSubcommands()` drains only the per-key jobs (those with `key != NULL`), leaving regular jobs for the end-of-unit drain. It is invoked at three places, all off the universal `afterCommand` hot path so standalone commands that never register a per-key job pay nothing: 1. `execCommand()` (`multi.c`) — after each MULTI/EXEC sub-command's `call()`, so per-key effects are observable between sibling sub-commands. 2. `scriptCall()` (`script.c`) — after each EVAL/FCALL sub-command's `call()`. 3. `loadSingleAppendOnlyFile()` (`aof.c`) — after each replayed single command, plus a final `firePostExecutionUnitJobs()` at the `cleanup:` path so a partially-applied command can't leave stragglers. EXEC's sub-commands go through `call()` during replay and are covered by the per-`call()` drain. Per-key jobs registered by a standalone command (no MULTI/EXEC, no script) are drained by the existing end-of-execution-unit `firePostExecutionUnitJobs()`, which walks the single queue in submission order. These drain sites are gated by an inline hint `server.fire_keyed_jobs_between_subcommands`, armed when a per-key job is enqueued and cleared after a drain, so the no-jobs case is a single well-predicted branch on a hot `server` field rather than a cross-TU call. ## AOF replay & replica support Unlike the regular API, `RM_AddPostNotificationJobForKey` does **not** mirror the `server.loading || repl_slave_ro` guard. The canonical consumer attaches **module key metadata** (`RM_SetKeyMeta` / `RM_GetKeyMeta`) to keys it observes via KSN. That metadata is neither stored in the AOF nor replicated; it must be rebuilt by re-running the same callback over the same KSN stream on every instance that applies the command — master, replica, and a process replaying its own AOF at startup. The per-key API is therefore permitted in all those contexts; the only registration guard is that it must be called from inside a KSN dispatch. **RDB load is intentionally outside this pattern.** RDB load decodes keys directly without running commands, so no KSN fires and per-key callbacks do not run. A test (`perkey-rdb`) pins this boundary. ## Runtime contract enforcement A per-key callback MUST only touch non-replicated, non-AOF-persisted state (canonically: module key metadata). Writing back into the keyspace would amplify the AOF on replay and diverge a replica from its master. This is now **enforced at runtime**, not merely documented — `server.firing_keyed_post_notif_jobs` is set for the duration of each per-key callback, and while it is set: - `RM_Call(...)` is refused: returns `NULL` with `errno == EINVAL`, or a `-ERR` call-reply when `CALL_REPLIES_AS_ERRORS` is requested. - `RM_NotifyKeyspaceEvent` / `RM_NotifyKeyspaceEventWithSubkeys` are refused (return `REDISMODULE_ERR`) — a nested notification could enqueue further per-key jobs mid-drain. Each refusal is logged once per drain (`LL_WARNING`, deduplicated via `keyedPostNotifRMCallWarned` / `keyedPostNotifNotifyWarned`). The canonical `RM_SetKeyMeta` path is unaffected (metadata does not propagate). ## Implementation summary - New per-key callback type `RedisModulePostNotifyJobPerKeyFunc` `(ctx, key, pd)`. The existing `RedisModulePostNotificationJobFunc` is renamed to `RedisModulePostNotifyJobFunc` with a backward-compatible typedef alias so existing module sources keep compiling. - `RedisModulePostExecUnitJob` gains an owned `key` field and a callback union. - `firePerKeyJobsBetweenSubcommands()` drains per-key jobs; `firePostExecutionUnitJobs()` drains the whole queue at end of unit. A shared `executePostExecUnitJob()` helper runs and frees a single job (and decrefs the owned key for per-key jobs). - New `server` fields: - `firing_keyed_post_notif_jobs` — set while a per-key callback runs; the no-write guard backing the `RM_Call` / `RM_NotifyKeyspaceEvent` refusals. - `fire_keyed_jobs_between_subcommands` — fast-path hint gating the explicit drains in `multi.c` / `script.c` / `aof.c`. - `in_keyspace_notification` — counter, incremented around the dispatch loop in `moduleNotifyKeyspaceEvent`. Defines the scope from which `RM_AddPostNotificationJobForKey` may be called (nested notifications nest cleanly). Calling the API outside this scope returns `REDISMODULE_ERR`. - `moduleUnregisterPostNotificationJobs()` drops any queued jobs belonging to a module being unloaded (wired into `moduleUnregisterCleanup`). - Because each KSN dispatch is single-key by construction, multi-key commands like `MSET`/`MGET` work transparently: they emit one `notifyKeyspaceEvent` per key, so the handler can register one per-key job per affected key. ## Testing A single test module, `pkmeta` (`tests/modules/postnotifications_perkey_metadata.c`), uses the canonical pattern: a KSN handler enqueues a per-key job, and the job attaches module key metadata via `RM_SetKeyMeta`. Metadata is neither AOF-persisted nor replicated, so its presence after a reload / propagation is direct evidence the callback re-ran on that instance. Module-internal counters (fire count, fire log, blocked-`RM_Call` count, blocked-notification count), kept out of the keyspace on purpose, carry the load-bearing assertions. All tests live in one suite, `tests/unit/moduleapi/postnotifications_perkey.tcl`: | Group | Tests | What they pin | |---|---|---| | `perkey-aof` | single command rebuilds metadata via AOF reload; MULTI/EXEC fires once per sub-command; HSET+HEXPIRE in MULTI/EXEC fires twice (each run for both `debug loadaof` and full restart) | One drain per replayed command/sub-command during AOF replay; the original RED-197766 scenario end-to-end. | | `perkey-rdb` | RDB-only restart does NOT rebuild metadata | Pins that RDB load is outside the firing pattern. | | `perkey-aof-replica` | AOF replay on a replica at startup rebuilds metadata | The carve-out removing the `repl_slave_ro` guard. | | `perkey-repl` | replica builds metadata from a propagated single command; replica fires per sub-command for propagated MULTI/EXEC | Both sides run the per-key job locally; no metadata crosses the replication stream. | | `perkey-misuse` | registration refused outside a KSN handler | The `in_keyspace_notification` scope guard. | | `perkey-order` | fires once per key in submission order across MULTI/EXEC; fires between sub-commands inside MULTI/EXEC; fires between commands inside a script (EVAL); multi-key command fires one job per key | Ordering and per-sub-command granularity for MULTI/EXEC, scripts, and multi-key commands. | | `perkey-contract` | RM_Call refused; refusal repeats per firing and never writes; RM_NotifyKeyspaceEvent refused; notification refusal repeats per firing | Runtime enforcement of the no-write contract. | | `perkey-expire` | lazy expire on read fires the per-key job; active expire (cron) fires it without a read | Expire-path coverage. | | `perkey-ptr-safety` | SMOVE first metadata attach must not UAF the source set | Owned-key lifetime safety. | The suite is registered in `runtest-moduleapi` and the module in `tests/modules/Makefile`. <!-- CURSOR_SUMMARY --> --- > [!NOTE] > **Medium Risk** > Touches module notification timing, MULTI/EXEC, scripting, and AOF replay—areas where ordering bugs could affect replication consistency, though keyspace writes from callbacks are blocked. > > **Overview** > Adds **`RedisModule_AddPostNotificationJobForKey`**, a sibling to `AddPostNotificationJob` that binds deferred work to a specific key from a keyspace-notification handler. Per-key jobs share the existing post-notification queue (`key != NULL`); regular jobs still drain only at the end of the execution unit, while per-key jobs also run after each sub-command in **MULTI/EXEC**, **EVAL/FCALL**, and **single-command AOF replay** (gated by `fire_keyed_jobs_between_subcommands` so the common path stays cheap). > > Runtime contract: per-key callbacks may update local module state (e.g. **`RM_SetKeyMeta`**) but **`RM_Call`** and **`RM_NotifyKeyspaceEvent`** are refused while they run; registration outside KSN dispatch fails via **`in_keyspace_notification`**. Unlike the regular API, per-key registration is allowed during AOF replay and on replicas so metadata can be rebuilt from the same KSN stream. Module unload now drops queued jobs for that module; AOF load **`cleanup`** flushes remaining regular jobs. > > New **`pkmeta`** test module and **`postnotifications_perkey`** suite cover ordering, AOF/replica rebuild, contract violations, and expire/evict edge cases. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 022ad9d. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup> <!-- /CURSOR_SUMMARY --> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Co-authored-by: Filipe Oliveira (Redis) <filipe@redis.com> Co-authored-by: Moti Cohen <moticless@gmail.com>
…er (redis#15309) Found by oranagra ## The problem `freeClient()` reverses the client's contribution to `stat_clients_type_memory` (exposed as `mem_clients_normal`) only when `c->conn` is set: ```c if (c->conn) server.stat_clients_type_memory[c->last_memory_type] -= c->last_memory_usage; ``` A replica caches its master on disconnect to attempt a partial resync. replicationCacheMaster() detaches the connection (unlinkClient) before caching, so by the time the cached master is freed by replicationDiscardCachedMaster(), `c->conn` is already NULL. The subtraction is skipped, so the cached master's last accounted size is never removed from mem_clients_normal. ## Fix Include the cached master in the subtraction by also reversing the contribution when the client being freed is `server.cached_master` (it is still set at the point `replicationDiscardCachedMaster()` calls `freeClient()`, and only cleared right after).
This PR is based on valkey-io/valkey#1213. sdssplitargs() builds each argument one byte at a time via sdscatlen(), which uses greedy doubling allocation. This leaves significant unused space in each SDS string — for a 600-byte value, ~400 bytes are wasted. Add sdsRemoveFreeSpace() after sdssplitargs() in processInlineBuffer() to trim each argument before wrapping it in an robj. Measured with 1M SET commands with 600-byte values piped via `redis-cli --pipe` in inline format: - Before: 1,021 MB vs 655 MB for RESP (55.9% overhead) - After: 655 MB, matching RESP baseline --------- Co-authored-by: muelstefamzn <muelstef@amazon.com>
Add a fixed-length-key rax mode that stores values directly in the leaf parent's child slots, avoiding the final leaf raxNode allocation. Introduce raxNewEx(metaSize, alloc_size, keyFixedLen) and keep the existing representation unchanged for variable-length trees. Update lookup, insert, remove, free, compression, and iterator paths to handle virtual leaves, including value updates through raxIteratorSetData(). Switch in-tree fixed-size rax users to the new mode. This reduces memory for dense fixed-length trees, with throughput changes within benchmark noise.
…redis#15362) `rewriteConfig()` accepts a path and reads the existing config from that path, but the writeback path used `server.configfile`. Since current callers all pass `server.configfile`, this does not affect the normal CONFIG REWRITE path, but using the passed-in path keeps the function behavior consistent with its interface and comments.
…is#15356) ## Problem `ACLLoadFromFile()` iterates `server.clients` to rebind users and drop pub/sub clients whose channel ACLs changed. Before this fix the loop only skipped `CLIENT_MASTER` clients before dereferencing `c->user->name`. Internal cluster connections (`AUTH "internal connection" <secret>`) set `c->user = NULL` and `CLIENT_INTERNAL`, but **not** `CLIENT_MASTER`. So if an internal connection is open when `ACL LOAD` runs, the loop dereferences a NULL `c->user` and crashes the node. ## Fix Guard on `c->user == NULL` instead of `CLIENT_MASTER`. This is a superset: master clients already carry `user = NULL`, so their handling is unchanged, while internal connections are now also skipped. Skipping internal connections is also semantically correct — they run unrestricted (no ACL user) and have no channel permissions to re-resolve. --------- Co-authored-by: debing.sun <debing.sun@redis.com>
## Issue `RedisModule_ConfigSetNumeric()` could crash when called with a non-existent config name. `moduleSetNumericConfig()` did not check whether `getMutableConfig()` returned `NULL` before dereferencing `config->type`, unlike the existing bool config setter path. ## Change Add the missing `NULL` check in `moduleSetNumericConfig()`, matching the existing handling in `moduleSetBoolConfig()`. Add TCL coverage for `configaccess.setnumeric nonexistent_config 1` to verify the module API returns an error instead of crashing. ## Test Passed: ```sh make ./runtest --single unit/moduleapi/configaccess ```
Add `SDIFFCARD` command — a cardinality-only variant of `SDIFF` that returns the number of elements in the set difference without sending the actual elements of the result. ### Motivation Similar to how `SINTERCARD` and `SUNIONCARD` provide cardinality queries for intersection and union, `SDIFFCARD` fills the gap for set difference. This is useful when clients only need to know the size of the difference rather than the full set of elements. ### Implementation - Reuses `sunionDiffGenericCommand` with `cardinality_only=1` and `SET_OP_DIFF`, consistent with how `SUNIONCARD` and `SINTERCARD` work. - For diff algorithm 1, skips `setTypeAddAux` entirely and just increments a counter, with early termination when `LIMIT` is reached. - For diff algorithm 2, positive-`LIMIT` cardinality-only requests use a dedicated 2b path that materializes the subtrahends, scans the first set, and stops at `LIMIT`; unlimited (`LIMIT 0` or omitted) requests use the normal algorithm-2 cardinality path and compute the full difference cardinality. - Adds a dedicated `sdiffcardGetKeys` key-extraction function following the same pattern as `sintercardGetKeys` / `sunioncardGetKeys`. - `LIMIT 0` means no limit. A positive `LIMIT` caps the returned cardinality, allowing early exit. ### Syntax SDIFFCARD numkeys key [key ...] [LIMIT limit] --------- Co-authored-by: debing.sun <debing.sun@redis.com>
## Module Update This PR updates **RedisJSON** to version **8.9.80**. ## Changes - Updated `modules/redisjson/Makefile` to set `MODULE_VERSION = v8.9.80` ## Release Information - **Module**: RedisJSON - **Version**: 8.9.80 - **Tag Date**: 2026-06-24 - **Makefile Path**: `modules/redisjson/Makefile` ## Verification - [ ] Version updated correctly in Makefile - [ ] Module builds successfully - [ ] Tests pass - [ ] No breaking changes --- *This PR was automatically created by the Redis Module Release Orchestrator.* <!-- CURSOR_SUMMARY --> --- > [!NOTE] > **Low Risk** > Single Makefile version bump with no application or security logic changes in this repository; risk is limited to upstream RedisJSON 8.9.80 behavior at build/runtime. > > **Overview** > Bumps the **RedisJSON** build pin from **v8.8.0** to **v8.9.80** by updating `MODULE_VERSION` in `modules/redisjson/Makefile`. > > That version is used by the shared module build (`common.mk`) to check out the matching **redisjson/redisjson** tag when fetching sources and producing `rejson.so`; no other files or build steps change in this PR. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 9bc0964. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup> <!-- /CURSOR_SUMMARY --> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
## Module Update This PR updates **RedisBloom** to version **8.9.80**. ## Changes - Updated `modules/redisbloom/Makefile` to set `MODULE_VERSION = v8.9.80` ## Release Information - **Module**: RedisBloom - **Version**: 8.9.80 - **Tag Date**: 2026-06-24 - **Makefile Path**: `modules/redisbloom/Makefile` ## Verification - [ ] Version updated correctly in Makefile - [ ] Module builds successfully - [ ] Tests pass - [ ] No breaking changes --- *This PR was automatically created by the Redis Module Release Orchestrator.* <!-- CURSOR_SUMMARY --> --- > [!NOTE] > **Low Risk** > Single version constant change with no application logic; risk is limited to upstream RedisBloom 8.9.80 behavior or compatibility in deployments that use this module. > > **Overview** > Bumps the **RedisBloom** build pin from **v8.8.0** to **v8.9.80** by updating `MODULE_VERSION` in `modules/redisbloom/Makefile`. > > That variable drives the upstream clone in `modules/common.mk`, so builds will fetch and compile the **8.9.80** tag from `redisbloom/redisbloom` instead of the previous release. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 0ebf5d7. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup> <!-- /CURSOR_SUMMARY --> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
## Summary Bumps the pinned `MODULE_VERSION` for **RedisTimeSeries** from `v8.9.80` to `v8.9.81`. RedisBloom and RedisJSON are unchanged. | Module | From | To | | --- | --- | --- | | RedisTimeSeries | v8.9.80 | v8.9.81 | > **Note:** the `v8.9.80...8.9.81` range is **diverged** — 10 new commits on `8.9.81`, and `8.9.81` is 2 commits *behind* `v8.9.80` (release-branch-only version/branch-cut commits, not forward-ported). The ancestry diff contains 10 commits and **all 10 are genuinely new** — `git cherry` marks every one `+`, so **0 are backports** already in `v8.9.80`. One in-range note: **`TS.BGET` was renamed to `TS.READ`** ([redis#2081](RedisTimeSeries/RedisTimeSeries#2081)) with a reworked blocking syntax, so the net-new command surface is `TS.READ`, not `TS.BGET`. ## Module changes ([v8.9.80 → v8.9.81](RedisTimeSeries/RedisTimeSeries@v8.9.80...8.9.81)) * Use dedicated redis_ref field in ramp.yml for CI git ref ([redis#2059](RedisTimeSeries/RedisTimeSeries#2059)) * pass VG=1 flag to the build as well ([redis#2060](RedisTimeSeries/RedisTimeSeries#2060)) * MOD-16160 — wake parked TS.BGET clients on key deletion ([redis#2062](RedisTimeSeries/RedisTimeSeries#2062)) * CI: bump GitHub Actions to Node24-compatible versions ([redis#2035](RedisTimeSeries/RedisTimeSeries#2035)) * MOD-15749 — Short-form CLUSTERSET: RAMP capability + LibMR error-return ([redis#2065](RedisTimeSeries/RedisTimeSeries#2065)) * CI: speed up Event CI (lean PR gate) and shard slow nightly jobs ([redis#2068](RedisTimeSeries/RedisTimeSeries#2068)) * MOD-6409 — TS.INFO memoryUsage (and MEMORY USAGE, via the mem_usage callback) ([redis#2067](RedisTimeSeries/RedisTimeSeries#2067)) * MOD-16266 — fix ci hang ([redis#2066](RedisTimeSeries/RedisTimeSeries#2066)) * MOD-15565 — mrange aggregation execution in inner shards ([redis#2074](RedisTimeSeries/RedisTimeSeries#2074)) * MOD-16505 — change TS.BGET command to TS.READ ([redis#2081](RedisTimeSeries/RedisTimeSeries#2081)) **0 commits excluded — all 10 are genuinely new** The `v8.9.80...8.9.81` ancestry diff contains exactly these 10 commits. `git cherry -v v8.9.80 8.9.81` marks every one `+` (no patch-equivalent change already in `v8.9.80`), so none are backports — unlike the `v8.8.0 → v8.9.80` bump, there is nothing to exclude here. ## Test plan * [ ] CI passes for RedisTimeSeries at the new pinned version * [ ] `make all` builds RedisTimeSeries cleanly against `unstable` * [ ] Module tests run under `make test`
### Overview
This PR adds two new options, `MAXCOUNT` and `MAXSIZE`, to the `XREAD`
and `XREADGROUP` stream commands. They cap the **cumulative** reply
across all streams named in a single command: `MAXCOUNT` bounds the
total number of entries returned, and `MAXSIZE` bounds the total reply
size in bytes. Unlike the existing `COUNT` option — which limits entries
on a per-stream basis — these new options apply a global budget for the
whole command, giving clients a reliable way to bound a multi-stream
read.
### Problem Statement
`XREAD`/`XREADGROUP` accept multiple streams in one call, and `COUNT`
only limits entries **per stream**. A read over N streams with `COUNT C`
can therefore return up to `N * C` entries, and the total reply size is
effectively unbounded — a single command can return a very large payload
depending on how many streams match and how large each entry is.
For clients that need to bound the work and memory of a single round
trip (e.g. fan-in consumers reading many streams, or latency-sensitive
paths), there is no way today to say "give me at most K entries total"
or "at most B bytes total" across all streams in one command. Working
around this requires issuing per-stream reads and aggregating
client-side, costing extra round trips.
### Solution
Introduces two cumulative caps, applied across all streams in the
command:
- `XREAD [COUNT count] [MAXCOUNT maxcount] [MAXSIZE maxsize] [BLOCK
milliseconds] STREAMS key [key ...] id [id ...]`
- `XREADGROUP GROUP group consumer [COUNT count] [MAXCOUNT maxcount]
[MAXSIZE maxsize] [BLOCK milliseconds] [CLAIM min-idle-time] [NOACK]
STREAMS key [key ...] id [id ...]`
Semantics:
- **`MAXCOUNT`** caps the total number of entries returned across all
streams. It must be a positive integer, and must be `>= COUNT` when both
are given (since it is a cumulative cap over a per-stream limit). When
`COUNT` is omitted, `MAXCOUNT` alone bounds the total.
- **`MAXSIZE`** caps the total reply size in bytes (tracked via
`c->net_output_bytes_curr_cmd`). It must be a positive integer.
- **At least one entry is always returned.** The budget is never
enforced until at least one entry has been emitted across the whole
reply, so a single message larger than `MAXSIZE` is still returned
rather than yielding an empty reply.
- Both caps work for new (`>`) reads and for history/PEL reads in
`XREADGROUP`. For `MAXSIZE`, the limit is checked *before* delivering
the next PEL/new entry, so a skipped entry is neither sent nor added to
the consumer's PEL.
**Implementation details:**
`MAXCOUNT` is enforced in `xreadCommand()`: a running `total_entries`
counter shrinks each stream's effective `count` to the remaining budget
and stops scanning further streams once the cap is reached. `MAXSIZE` is
enforced inside the emit loops of `streamReplyWithRange()` and
`streamReplyWithRangeFromConsumerPEL()`, guarded by `(emitted_before +
arraylen) > 0` to preserve the "always serve one entry" exception. All
existing callers (`XRANGE`, `XCLAIM`, `XAUTOCLAIM`, `XINFO`, and the
consumer-PEL path) are migrated to the new struct form.
### Examples
Given three streams, each with 100 entries:
**`MAXCOUNT` caps the cumulative entry count (vs. per-stream `COUNT`):**
```
> XREAD COUNT 50 STREAMS s1 s2 s3 0 0 0
# returns 150 entries total (50 per stream)
> XREAD COUNT 50 MAXCOUNT 80 STREAMS s1 s2 s3 0 0 0
# returns 80 entries total — capped across all streams
```
**`MAXCOUNT` without `COUNT`** — bounds the total directly, filling from
the first stream onward:
```
> XREAD MAXCOUNT 7 STREAMS s1 s2 s3 0 0 0
# returns 7 entries total, all from s1
```
**`MAXSIZE` bounds the reply size in bytes:**
```
> XREAD MAXSIZE 200 STREAMS s1 s2 s3 0 0 0
# returns fewer entries than an unbounded read, but always >= 1
```
**`MAXSIZE` always returns a single oversized message:**
```
> XADD bigstream 1-1 f <5000-byte value>
> XREAD MAXSIZE 50 STREAMS bigstream 0
1) 1) "bigstream"
2) 1) 1) "1-1"
2) 1) "f"
2) "<5000-byte value>"
# the single entry exceeds MAXSIZE but is still returned
```
**Both together — whichever bound triggers first wins:**
```
> XREAD MAXCOUNT 5 MAXSIZE 100000 STREAMS s1 s2 s3 0 0 0
# returns 5 entries (MAXCOUNT is the tighter bound)
```
`XREADGROUP` behaves identically for both new (`>`) reads and
history/PEL reads, and the caps are honored after a blocking client is
unblocked.
### Backward Compatibility
This is a fully additive change. `MAXCOUNT` and `MAXSIZE` are new
optional tokens recognized only within `XREAD`/`XREADGROUP`; when
omitted, behavior is exactly as before. No existing command, argument,
default, or reply format is modified, and no new command is introduced.
…t the end of slot migration
Co-authored-by: Ozan Tezcan <ozan.vx@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Background
During atomic slot migration (ASM), modules can already propagate commands to
the destination node at the beginning of the migration via the
REDISMODULE_SUBEVENT_CLUSTER_SLOT_MIGRATION_MIGRATE_MODULE_PROPAGATEevent andthe
RM_ClusterPropagateForSlotMigration()API. Those commands are delivered asthe first part of the migration stream, before the slot snapshot.
There was no equivalent hook to propagate commands at the end of the
migration, which some modules need to keep their own state consistent on the
destination side.
What this PR does
Adds a new sub-event
REDISMODULE_SUBEVENT_CLUSTER_SLOT_MIGRATION_MIGRATE_MODULE_PROPAGATE_END,fired on the source node at the very end of the migration — right before the
STREAM-EOFis sent to the destination. Modules use the sameRM_ClusterPropagateForSlotMigration()API to enqueue commands; they aredelivered as the last part of the migration stream.
MIGRATE_MODULE_PROPAGATEMIGRATE_MODULE_PROPAGATE_ENDSTREAM-EOF