IPC shared memory exhaustion after mass delete operations

[O3cheng]

IPC shared memory exhaustion after mass delete operations

Problem

After a large number of delete() calls, the IPC shared memory (ipc_shm) gets exhausted, causing subsequent broadcasts to fail with no memory and leading to cache inconsistency across workers.

Root Cause

Each delete() triggers a broadcast() which appends an event to shm:

local idx, err = self.dict:incr(INDEX_KEY, 1, 0)
self.dict:set(idx, marshalled_event)

However, poll() only reads events but never cleans up consumed entries from shm. The index keeps growing and consumed events are never removed, causing shm to grow indefinitely until exhaustion.

Additionally, there is a 1:1 relationship between delete() calls and broadcast events, meaning mass deletions produce an equal number of IPC events with no deduplication or batching.

Reproduction

local mlcache = require "resty.mlcache"

local cache = mlcache.new("my_cache", "cache_shm", {
    ipc_shm = "ipc_shm",  -- lua_shared_dict ipc_shm 1m;
})

for i = 1, 100000 do
    cache:delete("key_" .. i)
end

-- Broadcast fails due to shm exhaustion
local ok, err = cache:delete("important_key")
-- err: could not broadcast update: failed to insert event in shm: no memory

Impact

Broadcast failures cause cache inconsistency across workers
Workers that poll less frequently are more likely to miss events
No recovery mechanism once shm is full
Since lua_shared_dict is shared across all modules using the same dict name, shm exhaustion can also block writes from other unrelated modules that happen to use the same shared dict

Question

Do you have any plans or ideas for addressing the shm exhaustion caused by uncleaned IPC events?