Upstream tracking#165
Draft
grahamc wants to merge 3141 commits into
Draft
Conversation
…1fb-d1bc-4edd-b088-fbda88d8ec77 Release v3.21.4
Previously every RootValue was created with std::allocate_shared<Value *>(traceable_allocator<Value *>()), which costs a GC_MALLOC_UNCOLLECTABLE() / GC_FREE() pair per root value. Uncollectable allocations always take the global GC allocation lock, making this a significant source of mutex contention during parallel evaluation: the eval cache allocates a root value per AttrCursor (i.e. per attribute visited by 'nix search'), and the parallel evaluator allocates one per queued work item. Instead, carve root slots out of large uncollectable slabs (which are permanently part of the GC root set) and recycle them through a free list protected by a plain mutex, whose critical section is a few instructions rather than the entire GC allocator. Slots are cleared on release so they don't keep values alive; liveness semantics are unchanged. The slabs are never freed, so pool memory is bounded by the peak number of simultaneously live root values (8 bytes per slot). At 16 eval cores (with GC disabled via GC_INITIAL_HEAP_SIZE=40G), this cuts kernel time from ~8-9s to ~7s, context switches from 1.46M to 0.99M, and elapsed time from ~4.9s to ~4.3-4.5s for 'nix search nixpkgs --no-eval-cache fizzbuzz'. Assisted-by: Claude Fable 5 <noreply@anthropic.com>
RootValue (std::shared_ptr<Value *>) adds avoidable overhead per GC
root: a malloc'd control block, atomic reference count updates on
copies (a measured source of cache line contention during parallel
evaluation), and 16 bytes per handle. Most users never copy their
roots, so introduce UniqueRootValue, a move-only RAII wrapper around
a pointer to a root slot that returns the slot to the pool on
destruction.
The pool internals move from allocRootValue() into
allocRootValueSlot() / freeRootValueSlot(), shared by both handle
types, and the whole root value machinery moves into new files
root-value.{hh,cc}. The non-Boehm fallback now also goes through the
slot functions (new/delete) instead of a make_shared special case.
Converted to UniqueRootValue: ValMap, fileEvalCache, internalPrimOps,
genericClosure's work/result lists, EvalCache::value,
AttrCursor::_value, InstallableAttrPath::v, and the JSON parser
state. RootValue remains (documented as such) for roots captured in
std::function-backed lambdas, which require copyability.
Assisted-by: Claude Fable 5 <noreply@anthropic.com>
Replace the Sync<std::vector<Value **>> free slot pool with an intrusive linked list: each slot is a union of Value * (in use) and a pointer to the next free slot, with a Sync<Slot *> head. Allocation pops the head; freeing pushes in O(1). This avoids the auxiliary vector memory (8 bytes per free slot), the 4096-iteration push_back loop when carving a new slab, and most importantly the possibility of a vector reallocation (a malloc) while holding the pool lock on the free path. GC safety is unchanged: slabs remain uncollectable and conservatively scanned. In-use slots contain a Value * and root the value; free slots contain a pointer into a slab (or null), which is scanned harmlessly since slabs are never freed anyway. Writing the next pointer on free overwrites the Value *, which doubles as the "stop rooting the value" step. Assisted-by: Claude Fable 5 <noreply@anthropic.com>
ci: check that we actually released the tag
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Allocate RootValues more efficiently
Add a patch to boehmgc that makes GC_generic_malloc_many() hand out up to GC_many_blocks heap blocks worth of objects per acquisition of the global GC allocation lock, instead of exactly one 4 KiB block. All evaluator threads replenish their thread-local free lists (Values, Envs, Bindings) through this function, so during parallel evaluation they otherwise serialize on the allocation lock: at 12 eval cores, batching 64 blocks reduces futex syscalls by 6x (3.58M to 0.58M) for 'nix search nixpkgs --no-eval-cache fizzbuzz', and reduces both kernel time and elapsed time at higher core counts. The batch size defaults to GC_MANY_BLOCKS_DEFAULT (set to 64 here) and can be overridden at runtime via the GC_MALLOC_MANY_BLOCKS environment variable (1-64, where 1 restores the upstream behavior). Assisted-by: Claude Fable 5 <noreply@anthropic.com>
Flake lock file updates:
• Updated input 'nix':
'path:../..'
→ 'path:../..'
• Updated input 'nix/nixpkgs':
'https://api.flakehub.com/f/pinned/DeterminateSystems/secure-packages-25.11/0.1.915293%2Brev-36ee565fd8b26b3fd6847e9970af42f67ec517b2/019f3a3c-f2c0-72e3-af0e-4940993c93b4/source.tar.gz' (2026-07-07)
→ 'https://api.flakehub.com/f/pinned/DeterminateSystems/secure-packages-25.11/0.1.915296%2Brev-03ab51be14d9f1692a35003d8e128e7cefcf6092/019f4296-40f2-7350-859c-547b9fb7e2bf/source.tar.gz' (2026-07-08)
GCC has a silly warning that seems to only get triggered during a dev build and not in componentized builds because of compiler wrapper machinery. Suggested-by: randomizedcoder dave.seddon.ca@gmail.com <dave.seddon.ca@gmail.com> (cherry picked from commit a92c525)
…2ed-9b2e-4dc0-b002-983835a8ae32 Release v3.21.5
libutil: Fix GCC build with -Wchanges-meaning
fixup: build non-dsp nix on tags as well
build.yml: Enable provenance
Add initial CLAUDE.md
Declare GC_MANY_BLOCKS_DEFAULT, GC_MANY_BLOCKS_MAX and GC_many_blocks in private/gc_priv.h (following the GC_EXTERN convention for internal globals) instead of defining them in mallocx.c and repeating the constant 64 and an extern declaration in misc.c's environment variable parsing. Verified that the compile-time default still applies (futex calls drop 6.6x vs GC_MALLOC_MANY_BLOCKS=1) and the environment variable override still works. Assisted-by: Claude Fable 5 <noreply@anthropic.com>
boehmgc: Batch GC_generic_malloc_many() block allocation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
12 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Not intended to be merged directly. This PR is a convenience to show the diff between upstream Nix and Determinate Nix (the
mainbranch).Continuation of #4.