Prefetch directory-cache tree protos with one GetTree stream#2546
Open
erneestoc wants to merge 2 commits into
Open
Prefetch directory-cache tree protos with one GetTree stream#2546erneestoc wants to merge 2 commits into
erneestoc wants to merge 2 commits into
Conversation
The directory cache is keyed only on the root Directory digest, so one changed file out of thousands forces a full re-materialization (measured 1400ms of tree walk + per-file hardlinks + proto decodes) even though a warm full-root hit is 69ms and nearly every blob is already in the fast tier. REAPI Merkle nodes are content-addressed, so unchanged subtrees are byte-identical across roots and can be reused wholesale. With `experimental_subtree_caching` enabled (default: false), `create_subdirectory` routes each subdirectory through the same core get-or-create flow root materializations use, keyed by the subtree's own Directory digest: a hit is one `hardlink_directory_tree` call; a miss constructs the subtree once into its own cache entry (single-flighted through the shared `construction_locks` map; deadlock-free since flights only wait on strict Merkle descendants) and recursion caches nested subtrees too. New `subtree_hits`/`subtree_misses`/`evictions` counters in `CacheStats` surface reuse and thrash. To keep eviction meaningful, an entry's recorded size covers only bytes not owned by a descendant entry, so the map's size total approximates unique materialized bytes instead of counting each file once per ancestor level. Because subtree caching turns several previously rare root-path hazards into routine events, the shared flow is also hardened (these fixes apply to root-only caching as well): - Unified core: `get_or_create_entry` + `try_materialize_from_cache` + `construct_and_materialize` now carry ONE copy of the hit/miss logic for both root and subtree materializations. - RAII refcount pins: `CachedDirectoryMetadata.ref_count` is an `Arc<AtomicUsize>` released by `EntryPin::drop`, so futures cancelled mid-await (e.g. `buffer_unordered` dropping siblings on the first error) can no longer leak pins and leave entries permanently unevictable. `last_access` is an atomic too, letting the hit path run under the cache READ lock instead of convoying on the write lock. - Temp+rename publish: constructions build into a unique `cache_root/.tmp-N` and atomically rename to the canonical path, so a failed construction can never poison its (stable, shared) digest with a partial tree; failures clean up inline and a `ScratchGuard` sweeps temps of cancelled futures. - Tombstone eviction: victims are renamed to `cache_root/.del-N` under the same write-lock critical section that removed them from the map (rename is metadata-cheap; deletion stays off-lock in the background), so background deletion can never race a re-construction publishing to the canonical path. `DirectoryCache::new` sweeps pre-existing `cache_root` content the same way (the map starts empty, so anything on disk is orphaned and would poison re-constructions). - Dead-entry invalidation + clean retry: when hardlinking from an entry fails, the entry is invalidated (guarded by refcount-handle identity so a freshly rebuilt entry is never nuked) and the partially populated destination is removed before reconstruction — previously the retry hit "destination already exists"/AlreadyExists and the dead map entry kept failing its digest forever. - `hardlink_directory_tree` now chmods created directories to 0o755 (unix), keeping the documented invariant under restrictive umasks. Tests cover one-file-changed churn reuse (exact hit/miss accounting plus byte verification), the default-off regression guard, cross-root subtree sharing, eviction safety under concurrent construction, construction- failure retry (no digest poisoning, no temp leftovers), pin drain after cancelled sibling futures, damaged-entry recovery, and non-depth- multiplied size accounting. Observability: DirectoryCache implements MetricsComponent (clonefile_hits, hardlink_hits, subtree_hits, subtree_misses, evictions, entries, total_size_bytes) and is wired into the worker metrics tree via a new directory_cache field on the running-actions-manager Metrics component (Weak ref, absent when the cache is disabled). entries/total_size_bytes are lock-free atomic mirrors maintained at insert/evict/invalidate. A rate-limited (1/min) tracing::info! "DirectoryCache summary" line from the hit/miss path covers deployments without a metrics pipeline. Config: the cache-wide fetch semaphore size (previously the hardcoded CONSTRUCT_DIRECTORY_MAX_FETCHES=64 from TraceMachina#2526) is now DirectoryCacheConfig::max_concurrent_fetches (serde default 64 — behavior-preserving; validated > 0 at construction with InvalidArgument). Low values fragment coalesced read batches, so deployments using experimental_read_batching on the underlying grpc store can raise it substantially (e.g. 512). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
On a directory-cache miss, the worker fetches Directory protos level by level: each level's protos are only discoverable from its parents, so a cold tree pays one worker->CAS round trip per tree DEPTH before file fetching can even start. Real input trees are commonly 10+ levels deep, so at millisecond RTTs this serialization alone costs tens to hundreds of milliseconds per cold action, and GetTree exists in REAPI precisely to avoid it. Add opt-in DirectoryCacheConfig.experimental_get_tree_prefetch: on a cache miss, issue a single GetTree stream (one round trip, one fetch permit) and key the returned protos by their re-computed digest (the stream carries protos, not digests; hashing uses the caller's digest function). construct_directory consults the map and falls back to the existing per-level fetch for any missing proto, any stream failure, or when the slow tier is not a grpc store — so behavior can only degrade to today's path, never fail because of the prefetch. Measured RPC-shape A/B (real CasServer over TCP, client-injected RTT, identical proto counts): depth-32 chain at 25ms RTT: 877ms -> 26ms (33.7x); wide 221-proto depth-3 tree at 25ms RTT: 164ms -> 30ms (5.5x). End-to-end validated against a real server: a 4,400-file / 221-proto tree constructs entirely from one prefetched stream (zero per-level fetches) with byte-identical output. Off by default: the serving CAS walks the tree against its own backing store to answer GetTree, so deployments should enable this only when directory protos are served from a fast tier. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UXVtatcR9YMecBiu9RjwpC
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Prefetch directory-cache tree protos with one GetTree stream
Problem
On a directory-cache miss, the worker discovers
Directoryprotos level bylevel: a child's digest is only known once its parent's proto has arrived,
so a cold tree pays one worker→CAS round trip per tree depth before
file fetching can even begin. Real input trees are commonly 10+ levels
deep; at millisecond RTTs this serialization costs tens to hundreds of
milliseconds per cold action. REAPI's
GetTreeexists precisely for this:one RPC streams every directory proto of a tree.
Change
Opt-in
DirectoryCacheConfig.experimental_get_tree_prefetch: on aroot-level cold miss, issue a single
GetTreestream (one round trip,one fetch permit) and key the returned protos by their re-computed digest —
the stream carries protos, not digests, so each is hashed with the caller's
digest function. Construction consults the map and falls back to the
existing per-level fetch for any missing proto, any stream failure, or a
non-gRPC slow tier: the feature can only degrade to today's path, never
fail because of it.
Interaction with subtree caching (#2541): prefetch triggers only when a
full tree construction begins. Subtree cache hits never construct and never
prefetch; subtree misses reached during a construction reuse the map
already fetched for the root.
Default off, for an honest reason: the serving CAS answers
GetTreebywalking the tree against its own backing store, so deployments should
enable this only when directory protos are served from a fast tier
(memory/filesystem), which is the common scheduler-side setup.
Measurements
RPC-shape A/B (real
CasServerover TCP, client-injected RTT, identicalproto counts on both paths):
End-to-end validated against a real server: a 4,400-file / 221-proto tree
constructs entirely from one prefetched stream (zero per-level fetches),
byte-identical output.
Tests
None, constructionfalls back and produces correct results (new test).
and cargo.
Notes for reviewers
Box::pin'd at call sites deliberately: inlined,it enlarges the callers' futures enough to overflow the stack in
small-stack (fastbuild) test builds.
GetTreepagination is honored via the streaming client; an incompletemap is tolerated per-proto by the fallback.
This change is