FTW expert-bank load: resident set grows to ~2.2× the bank size — deterministic abort at ~67% on a 220G host
Summary. After converting a large MoE checkpoint to FTW (ft checkpoint — conversion itself completed cleanly, byte-verified), the dense-weight load phase is excellent: ~10G in ~38s with O_DIRECT active (no fallback warnings in the log). However, the expert-bank phase's resident set grows far beyond the bank bytes themselves. On our host it deterministically hit the memory ceiling at ~67% of bank progress (91.4G of 137G) with ≈205G consumed ≈ 2.2× the reported bank progress. Four consecutive runs, including from a fully cold page cache, aborted at the same point.
Environment. FreeToken v0.1.2. Threadripper PRO 3945WX (12C/24T), 224 GB DDR4-3200 ECC (220 GiB usable), 2× NVIDIA CMP 170HX 64 GB (sm_80, HBM2e; x16 link at Gen2). 300B-A-class MoE (FP8/FP4 origin; ~137G expert pool in host RAM). ft serve --tp-size 2 --moe-backend offload --moe-cache-rate 0.5 --expert-load serial over the FTW tree. OMP caps for #371-class oversubscription were active (OMP_NUM_THREADS=6 MKL_NUM_THREADS=6 OMP_WAIT_POLICY=PASSIVE), and an external watchdog (PSI full + MemAvailable floor 16G) aborted the load cleanly when the ceiling was reached — so no host starvation occurred during these runs.
What we ruled out:
- O_DIRECT fallback — no "O_DIRECT unsupported" warning in serve logs; the dense phase showed no cache growth.
- Page-cache ballast — identical cliff from a cold cache (page cache dropped to ~0; 215G available at start of load).
- Reclaim lag — an external sweeper (
echo 1 > /proc/sys/vm/drop_caches on a 5s cadence when cache > 20G) freed up to 100G in a single sweep, yet the steady-state growth continued: the dominant consumer is not droppable page cache. Process AnonPages stayed ~2.5G during the load; the growth presents like a mapped/pinned working set accumulating roughly linearly with bank fill (consistent with a pin-as-you-go pattern).
- Thread oversubscription — capped as above; no spin, no starvation, no oomd activity.
Observed profile (representative run; all four runs identical in shape):
- start: 215G
MemAvailable, page cache ~0
- by 67% bank progress (91.4G/137G):
MemAvailable 15.4G → ≈205G consumed ≈ 2.24× the bank progress
- between ~50% and the abort the host shed >100G of available memory in well under 15 minutes, with cache sweeps only slowing, not stopping, the decline
Ask:
- Is the FTW bank path expected to be bounded to "banks + a small working set" (like the safetensors parallel path's documented peak ≈ (prefetch+1) shards + banks)? If yes, is there a configuration we should use on a RAM-constrained host, or is this something a patch could address?
- If the working set is inherent to the flat-window scratch + pin pipeline, would you consider documenting the expected envelope (≈2.2× on our measurements) so users can size RAM accordingly? A host that "fits the model on paper" (137G banks on 220G RAM) cannot actually load it today.
Offer: happy to test candidate fixes on the same box, and to share any logs (serve logs, memory time series at 5s resolution, watchdog traces) in whatever format is most useful. Conversion plus four load attempts are all fully instrumented on our side.
FTW expert-bank load: resident set grows to ~2.2× the bank size — deterministic abort at ~67% on a 220G host
Summary. After converting a large MoE checkpoint to FTW (
ft checkpoint— conversion itself completed cleanly, byte-verified), the dense-weight load phase is excellent: ~10G in ~38s with O_DIRECT active (no fallback warnings in the log). However, the expert-bank phase's resident set grows far beyond the bank bytes themselves. On our host it deterministically hit the memory ceiling at ~67% of bank progress (91.4G of 137G) with ≈205G consumed ≈ 2.2× the reported bank progress. Four consecutive runs, including from a fully cold page cache, aborted at the same point.Environment. FreeToken v0.1.2. Threadripper PRO 3945WX (12C/24T), 224 GB DDR4-3200 ECC (220 GiB usable), 2× NVIDIA CMP 170HX 64 GB (sm_80, HBM2e; x16 link at Gen2). 300B-A-class MoE (FP8/FP4 origin; ~137G expert pool in host RAM).
ft serve --tp-size 2 --moe-backend offload --moe-cache-rate 0.5 --expert-load serialover the FTW tree. OMP caps for #371-class oversubscription were active (OMP_NUM_THREADS=6 MKL_NUM_THREADS=6 OMP_WAIT_POLICY=PASSIVE), and an external watchdog (PSIfull+MemAvailablefloor 16G) aborted the load cleanly when the ceiling was reached — so no host starvation occurred during these runs.What we ruled out:
echo 1 > /proc/sys/vm/drop_cacheson a 5s cadence when cache > 20G) freed up to 100G in a single sweep, yet the steady-state growth continued: the dominant consumer is not droppable page cache. ProcessAnonPagesstayed ~2.5G during the load; the growth presents like a mapped/pinned working set accumulating roughly linearly with bank fill (consistent with a pin-as-you-go pattern).Observed profile (representative run; all four runs identical in shape):
MemAvailable, page cache ~0MemAvailable15.4G → ≈205G consumed ≈ 2.24× the bank progressAsk:
Offer: happy to test candidate fixes on the same box, and to share any logs (serve logs, memory time series at 5s resolution, watchdog traces) in whatever format is most useful. Conversion plus four load attempts are all fully instrumented on our side.