Found this bug while reviewing #961
AteomDialer caches one grpc.ClientConn per worker pod UID in lru.New(256) (cmd/atelet/main.go:183). k8s.io/utils/lru without an eviction func just drops the oldest entry when the cache is full — and nothing anywhere calls Close() on the dropped conn. Cache entries are also never removed when a worker pod dies, so the cache fills with dead UIDs over time.
A ClientConn that's dropped but not closed keeps its goroutines and memory around; gRPC requires an explicit Close(). So every eviction is a leaked conn.
Today this is slow: the cache only sees pods that lifecycle RPCs explicitly target, so it takes >256 distinct worker pods dialed since atelet started before evictions begin, and an atelet restart resets the count. On a long-running atelet on a churny node it's a steady drip; mostly nobody notices.
Repro
In-process repro with the repo's exact deps (grpc v1.83.0, same k8s.io/utils/lru, same dial options as DialAteomPod): 32-entry cache, 100 fake ateom sockets swept per tick, one failed RPC per probe, 10 ticks.
- No eviction func (current code): goroutines grow 1 → 4001, heap 0.4 → 12.7 MiB, linear, ~400 goroutines leaked per tick. GC reclaims nothing.
- With
NewWithEvictionFunc closing evicted conns: flat at 129 goroutines / ~1 MiB across all ticks.
- gRPC's channel idle timeout (default 30 min) is not a rescue: after it fires, only 1 of each abandoned conn's 4 goroutines exits. ~3 goroutines and ~5 KiB heap per conn stay leaked permanently.
Fix
Use lru.NewWithEvictionFunc and close the evicted conn. One-liner, fixes the flaw at the source regardless of who dials.
Separately, #961's poller should probably not go through this cache at all (open/close per probe), but that's that PR's call — this issue is just: don't drop conns without hanging up.
Found this bug while reviewing #961
AteomDialercaches onegrpc.ClientConnper worker pod UID inlru.New(256)(cmd/atelet/main.go:183).k8s.io/utils/lruwithout an eviction func just drops the oldest entry when the cache is full — and nothing anywhere callsClose()on the dropped conn. Cache entries are also never removed when a worker pod dies, so the cache fills with dead UIDs over time.A
ClientConnthat's dropped but not closed keeps its goroutines and memory around; gRPC requires an explicitClose(). So every eviction is a leaked conn.Today this is slow: the cache only sees pods that lifecycle RPCs explicitly target, so it takes >256 distinct worker pods dialed since atelet started before evictions begin, and an atelet restart resets the count. On a long-running atelet on a churny node it's a steady drip; mostly nobody notices.
Repro
In-process repro with the repo's exact deps (grpc v1.83.0, same
k8s.io/utils/lru, same dial options asDialAteomPod): 32-entry cache, 100 fake ateom sockets swept per tick, one failed RPC per probe, 10 ticks.NewWithEvictionFuncclosing evicted conns: flat at 129 goroutines / ~1 MiB across all ticks.Fix
Use
lru.NewWithEvictionFuncand close the evicted conn. One-liner, fixes the flaw at the source regardless of who dials.Separately, #961's poller should probably not go through this cache at all (open/close per probe), but that's that PR's call — this issue is just: don't drop conns without hanging up.