fix: bound the voice conditioning cache to stop it exhausting VRAM

[malammar]

Summary

_conds_cache in engine.py is unbounded. It is keyed by (path, mtime, exaggeration) and cleared only by unload_model(), so every distinct reference voice a server ever handles pins GPU tensors for the lifetime of the process.

Each entry costs ~175 MB of VRAM. On a 6 GB card that is enough to take the server from healthy to failing every request within an hour of ordinary traffic.

This adds an LRU bound (CONDS_CACHE_MAX, default 4) and refreshes recency on cache hits so a rotating voice workload evicts least-recently-used rather than by insertion order.

How it fails today

Observed on a live deployment: a worker served correctly for ~45 minutes, then began returning 500 for every generation with CUDA out of memory: Tried to allocate 2.00 MiB, while /api/model-info still reported {"loaded": true} and /docs still returned 200.

The tell is the size of the failing allocation. Failing to obtain 2 MiB while holding ~260 MB reserved-but-free is not an oversized model, it is a cache that has eaten the working headroom.

Measurements

P106-100 (5.93 GiB usable), chatterbox-turbo, three 10-minute soaks cycling 28 reference voices at ~45 req/min with text lengths varying 40–520 chars:

CONDS_CACHE_MAX	requests	failed	live tensors at end	headroom
unbounded (today)	453	406 (90%)	5.46 GiB	0.11 GiB
2	89	1 (1.1%)	4.58 GiB	0.10–0.49 GiB
0	88	0	4.37 GiB	0.28–0.40 GiB

Model footprint on a fresh load is 4.09 GiB, so the +1.37 GiB of retained tensors is exactly 8 cached voices × ~175 MB.

Two controls confirm the cache is the cause rather than allocator fragmentation:

• Same load, one voice, unbounded cache: 50/50 requests succeeded, live tensors flat at 4.34 GiB. Voice variety is the variable, not request volume or text length.
• PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True alone does not fix it. It helps a separate fragmentation effect (reserved memory grows while live tensors stay flat), but it cannot free tensors the cache still references.

Cost of a smaller cache

A miss only re-encodes the reference audio. Measured on an RTX 4070 against a 3.8 MB reference:

request	miss	hit	delta
84 chars	1.23s	1.09s	+0.14s (+13%)
240 chars	3.30s	3.32s	within noise

It is a fixed per-request cost that does not scale with output length, so it is a small percentage on anything but very short requests. Cheap relative to 175 MB of VRAM per entry.

Choosing the default

4 entries (~700 MB) leaves a 6 GB card ~1.1 GiB for generation after the model. It is deliberately conservative: an oversized cache takes a card out of service entirely, while an undersized one costs ~0.14s on a miss. Deployments with VRAM to spare can raise it — 16 is comfortable on 12 GB — and orchestration layers can size it per device from torch.cuda.mem_get_info() at launch.

Happy to change the default, or gate the whole thing behind an opt-in flag if you would rather not alter behaviour for existing users. I would suggest against leaving it unbounded even so, since the failure is silent and looks like a hung GPU rather than a cache problem.

Testing

• LRU eviction, recency refresh on hit, re-store of an existing key, and CONDS_CACHE_MAX=0 verified directly against the eviction logic.
• Soaks above run against the real model on real hardware.
• No API change; default behaviour differs only once a process has cached more than 4 distinct voices.