Coalesce FilesystemStore durability flushes on macOS#2539
Conversation
Every FilesystemStore upload makes its blob durable with File::sync_all before publishing it, which on macOS issues fcntl(F_FULLFSYNC) — a full device-cache flush — once per blob. The flush serializes at the device and costs multiple milliseconds, so many-small-blob workloads spend nearly all their time flushing: a 4,400-file / 2 MB action input tree takes 14.5s cold (21.0s under contention), ~12s of which is F_FULLFSYNC, while tar -x of the same tree takes ~1.4s. Linux does not suffer from this because concurrent fsyncs coalesce in the filesystem journal's group commit; macOS has no journal-level coalescing and no syncfs(2), and F_BARRIERFSYNC measures just as slow per file (~15.6s). Apply the same group-commit idea in userspace, macOS-only, no config: F_FULLFSYNC's per-file part (pushing the file's pages to the device) is cheap plain fsync(2); the expensive device-cache drain is device-wide by nature. Each writer fsync(2)s its own blob (bounded by the crate's open-file permit machinery), then joins the current commit round; a single long-lived flusher task issues one F_FULLFSYNC per round on a dedicated sentinel file (a sibling of the content path, invisible to the startup scan) covering every previously-fsynced blob at once. Rounds self-batch like a journal — while one flush runs, later writers accumulate into the next round — plus a ~3ms accumulation window applied only when a round has multiple waiters, so a solo write pays about what it pays today. Durability is unchanged: a blob is only renamed into the content directory after a device-cache flush that started after its own data reached the device — the same guarantee as per-file F_FULLFSYNC. EINTR is retried exactly as std's sync_all does; the blocking fsync task owns a dup'd fd so caller cancellation can never touch a reused descriptor; a cancelled flusher round reports an error to its waiters rather than stranding them; and a store whose sentinel cannot be created (read-only volume) degrades to per-file sync_all instead of failing construction. The executable-variant copy path drops its flush entirely: the .exec directory is cleared on every startup, so those files are never trusted across a crash. Linux and Windows keep plain sync_all, byte-identical behavior. Measured on the benchmark above (output trees byte-verified every run): 14.5s -> ~3.0s alone and 21.0s -> ~5.6s contended (4.8x / 3.7x), within ~35% of the no-flush floor (2.2s / 4.1s). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UXVtatcR9YMecBiu9RjwpC
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
| // startup (see `new_with_timeout_and_rename_fn`), so a | ||
| // variant is never trusted across a crash — durability | ||
| // would buy nothing, and on macOS `sync_all` is a | ||
| // multi-ms full device-cache flush per executable. |
There was a problem hiding this comment.
Shouldn't this still get done on non-Mac unix systems?
There was a problem hiding this comment.
I don't think it buys anything on any platform: the .exec variant directory is unconditionally wiped on every startup (drop(fs::remove_dir_all(&executable_dir).await) in new_with_timeout_and_rename_fn, filesystem_store.rs:893), so a variant can never survive a crash to be trusted afterwards — the flush was guarding a file that's deleted before it could ever be read again. Process crashes don't need it either (page cache survives the process), and the ETXTBSY protocol only requires the writer fd to be closed before the rename, not flushed.
The fsync was only load-bearing if the variants were meant to persist across restarts — and they deliberately aren't (they're regenerable, and the startup clear exists so a stale variant can't leak across runs).
That said, it's cheap on Linux — if you'd prefer belt-and-suspenders on non-macOS I'm happy to restore it behind #[cfg(not(target_os = "macos"))]. Just flagging that it would be dead code by the above reasoning rather than a platform behavior difference.
2 participants
Coalesce FilesystemStore durability flushes on macOS (userspace group commit)
Problem
Every
FilesystemStoreupload makes its blob durable withFile::sync_allbefore publishing it into the content directory. On macOS,
sync_allissuesfcntl(F_FULLFSYNC)— a full device-cache flush — once per blob. The flushserializes at the device and costs multiple milliseconds, so workloads that
upload thousands of small blobs (a worker's fast tier materializing a
many-tiny-file action input tree) spend nearly all of their time in flushes:
a 4,400-file / 2 MB input tree takes 14.5 s cold (21.0 s under contention),
of which ~12 s is
F_FULLFSYNC, whiletar -xof the same tree takes ~1.4 s.Linux does not suffer from this because concurrent
fsynccalls coalesceinside the filesystem journal's group commit. macOS has no journal-level
coalescing and no
syncfs(2), andF_BARRIERFSYNCmeasures just as slowper-file here (~15.6 s).
Change: group commit in userspace (macOS only, no config)
F_FULLFSYNChas an exploitable asymmetry: writing one file's pages to thestorage device is per-file work (plain
fsync(2)— measured effectivelyfree), but the expensive device-cache drain is device-wide by nature. So a
FlushCoalescerinsideFilesystemStoreapplies the ext4 trick one layerup:
fsync(2)(cheap).single
F_FULLFSYNC(on a dedicated sentinel file in the content root)that makes every previously-
fsynced blob durable at once.writers accumulate into the next round. A ~3 ms accumulation window
(about the cost of one flush) fattens rounds further.
into the content directory.
Durability is unchanged: a blob is only published after a device-cache
flush that started after its own data reached the device — the same
guarantee as per-file
F_FULLFSYNC, at ~1/10th the flush count. A solo(unbatched) write pays one flush plus the 3 ms window, roughly what it pays
today. Linux and Windows keep plain
sync_all(their journals alreadygroup-commit; behavior byte-identical).
Measurements
Real-stack benchmark (DirectoryCache → FastSlowStore(FilesystemStore fast) →
GrpcStore → in-process ByteStream/CAS servers over TCP), 4,400-file / ~450 B
tree, Apple Silicon / APFS, output trees byte-verified every run:
F_FULLFSYNC(today)tar -xreference4.8× / 3.7× with full durability, within ~35% of the physically possible
no-flush floor (contended runs ranged 4.9–6.3 s across iterations).
Relationship to a possible sync_policy knob
A companion branch (not yet a PR) adds
FilesystemSpec.sync_policy("full" | "data" | "fsync" | "none", default "full") for deployments that
want to trade durability for the last ~25% on re-fetchable cache tiers.
The two compose — this coalescer would become the implementation of
fullon macOS — and neither depends on the other; happy to open it as a
follow-up if there's interest.
Tests
concurrent_upload_burst_round_trip_test: 200 concurrent uploads throughthe coalescer, all round-trip and survive a store restart.
behind the old
sync_allcall sites), including the Deadlock in NativeLink Worker due to File Permit Exhaustion #2051 open-filesemaphore-exhaustion regression test, which caught (and now guards) the
permit-deadlock hazard described below.
subtree-caching (Add opt-in subtree-keyed caching to the worker DirectoryCache #2541) PRs: zero conflicts, all 40 bazel tests across
config/store/worker green on the combined branch.
Post-review hardening (multi-angle review applied)
{content_path}.flush_sentinel,the
.exec-directory precedent): the startup scan's legacy root sweepwould otherwise migrate and delete a sentinel inside the content root on
every restart.
content volume degrades to per-file
sync_allinstead of failing storeconstruction (macOS-only regression otherwise).
try_clone), so callercancellation can never race a reused descriptor. It deliberately does
NOT take an open-file permit: the caller's
FileSlotalready holds one,and requiring a second deadlocks under a drained semaphore (caught by
the Deadlock in NativeLink Worker due to File Permit Exhaustion #2051 regression test, which pins the semaphore at one permit).
Concurrency is still bounded by the callers' own permits.
EINTRretried exactly as std'ssync_all(cvt_r) does, for bothfsync(2)andF_FULLFSYNC.instead of stranding them; the sentinel dirty-write is best-effort so an
ENOSPC on the 8-byte write cannot fail a whole round.
solo write on an idle store pays roughly today's latency.
store (simpler audit surface; serialization is inherent). Sharing one
flusher per device (
st_dev) would amortize across stores on onevolume but requires a runtime-agnostic flusher (dedicated OS thread) —
multiple tokio runtimes per process (tests) break a task-based global —
left as a documented follow-up.
.execdirectory is cleared on every startup, so durability there bought
nothing while costing one
F_FULLFSYNCper executable digest.Notes for reviewers
{content_path}.flush_sentinel(a sibling ofthe content root), and is re-dirtied (best-effort) each round so the
flush is never elided.
Notify; writers notify strictly after subscribing to their round, solost wakeups are impossible, and
FlushCoalescer::dropwakes the taskso it exits instead of parking forever.
This change is