Add opt-in subtree-keyed caching to the worker DirectoryCache

[erneestoc]

Summary

The worker DirectoryCache is keyed only on the root Directory digest, so any input change is a full miss: one changed file out of 4400 re-materializes the entire tree. Measured on a churn-shaped workload (the dominant case for incremental builds): the one-file-changed reconstruction costs 1855 ms of almost purely local work — only 4 slow-store fetches; the rest is tree-walking, per-file hardlinks, and proto decodes for 4399 unchanged files.

This adds opt-in subtree caching: every subdirectory is additionally cached under its own Directory digest, and create_subdirectory consults the cache before constructing. REAPI's Merkle design makes this sound by construction — Directory nodes are content-addressed, so an unchanged subtree has an identical digest across roots (and across different actions' input trees). With the flag on, the one-file-changed case reuses all unchanged subtrees via whole-subtree hardlink_directory_tree (APFS clonefile on macOS) and constructs only the changed spine.

"directory_cache": {
  "experimental_subtree_caching": true,
  // Subtree entries multiply entry COUNT; raise max_entries accordingly
  // (e.g. 10x the default 1000).
}

Default off: existing behavior byte-for-byte (guarded by a regression test asserting zero subtree cache activity and correct results with the flag unset).

Measurements

4400-file tree (20 mids x 10 leaves x 22 files), latency-injected slow tier, A' = same tree with exactly one file changed:

	flag off	flag on
A cold construct	23.4 s	26.3 s (within run variance)
A warm (full root hit)	58 ms	106 ms
A' (1 file changed)	1855 ms	479 ms (3.9x)

The win scales with tree size and churn locality; cross-root sharing means common subtrees (SDKs, framework bundles) are reused across different actions' input roots, not just successive versions of one root.

Caveat on absolute local timings: captured under a test harness whose
capturing TRACE subscriber adds measurable overhead — treat the table as
same-harness A/B. Production evidence: with this flag + read coalescing on
macOS workers (4-build sweep, true cold), per-exec setup time -67% avg
cold and -48% avg incremental — the incremental number is the one this
feature targets.

Scope and companion changes

This PR makes repeat/churn materialization cheap (warm reuse). The
cold construct cost itself is addressed independently: on macOS, most
of it was per-blob F_FULLFSYNC in FilesystemStore, fixed by the flush
coalescing / sync_policy changes (cold 4,400-file construct ~14.5s ->
~3s in the same benchmark). All of these are orthogonal and compose;
subtree hits (~0.1-0.5s) remain far below even the improved cold path.

Design

• Same maps, same invariants: subtree entries live in the existing digest-keyed cache map and use the existing single-flight construction_locks — a root-level get_or_create and a subtree construction of the same digest share one flight and one entry. Cache paths use the existing cache_root/<digest> scheme, so a subtree entry can later serve a root-level request and vice versa.
• Deadlock-free by construction: a flight only ever waits on construction locks of its strict descendants; the content-addressed Merkle DAG is acyclic (a Directory cannot contain an ancestor — that would be a hash cycle).
• Eviction safety: on miss, the subtree is constructed into its own cache entry, inserted pinned (ref_count: 1) with the same digest-derived size accounting as root entries, hardlinked into the parent, then released — the existing ref_count == 0 eviction filter covers it, mirroring construct_and_materialize. Hardlink failures fall back to reconstruction (hit path) or direct in-place construction (post-construction path), mirroring the root-level fallbacks.
• Observability: subtree_hits / subtree_misses counters alongside the existing clonefile_hits/hardlink_hits.
• Known accounting caveat (conservative): bytes shared between a root entry and its subtree entries count toward max_size_bytes once per entry.

Testing

• Churn reuse: cold A then mutant A' → exactly (mids-1)+(leaves-1) subtree hits and exactly 2 misses (changed mid + changed leaf); full-tree byte verification including the changed file.
• Flag off: same sequence → zero subtree activity, byte-correct (default-behavior regression guard).
• Cross-root sharing: identical subtree under two different roots → 1 miss then 1 hit.
• Eviction safety under concurrency: max_entries: 3 vs 4 concurrent multi-entry trees on a multi-threaded runtime, all byte-verified.
• bazel test //nativelink-worker:integration //nativelink-config:unit_test green (clippy + rustfmt aspects); cargo check --tests and cargo fmt --check clean.
• Also validated merged together with the read-coalescing (Add opt-in small-blob read coalescing to GrpcStore #2540) and
  macOS flush-coalescing (Coalesce FilesystemStore durability flushes on macOS #2539) PRs: zero conflicts, all 40 bazel tests
  across config/store/worker green on the combined branch.

---

This change is 

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
nativelink	Ready	Preview, Comment	Jul 10, 2026 5:28pm
nativelink-aidm	Ready	Preview, Comment	Jul 10, 2026 5:28pm