diff --git a/CHANGELOG.md b/CHANGELOG.md index 8bb1139..04f8311 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,6 +14,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - **`BStackChunk<'a>`/`BStackChunkIter<'a>` — fixed-stride view over `BStackSlice` (`alloc`).** `BStackSlice::chunks`/`rchunks` (mirrored on `BStackOwnedSlice`) return `(BStackChunk, BStackSlice)`: aligned view + remainder, pure offset arithmetic, no I/O. `as_slice`/`into_slice`/`with_stride` recover or re-chunk the aligned region. `PartialEq`/`Eq`/`Hash`/`PartialOrd`/`Ord` on `(chunk_len, region)`; no cross-type comparison with `BStackSlice`. Not an iterator itself: `iter()`/`IntoIterator` yield a `BStackChunkIter` (`DoubleEndedIterator` + `ExactSizeIterator` + `FusedIterator`), zero I/O per step. - **`BStackChunk` search/sort/select.** `binary_search_by`/`binary_search_by_key` (`alloc`): O(log n) chunk reads. `sort_by`/`sort_by_key`/`select_nth_by`/`select_nth_by_key` (`set` + `atomic`): one crash-atomic `BStack::process` call, in-place cycle-following permutation (O(1) scratch chunks, stack-allocated ≤128 B); `select_nth_*` mirrors `[T]::select_nth_unstable_by`. +### Changed + +- **`GhostTreeBstackAllocator` — smaller AVL critical section (Rust + C, `alloc`).** `alloc`/`dealloc`/`realloc` of non-tail blocks do less work while holding the allocator mutex. The rebalance up-pass no longer re-reads and re-writes each ancestor through a redundant balance-factor pass — the balance factor and height computed by the node write are threaded into `avl_rebalance` — and each node now caches its two child heights, so the up-pass and rotations write one node per level and read no children in the common in-balance case (down from ~2 writes plus several reads per level). Rust also swaps the per-op heap `Vec` path buffer for a stack array of the fixed `MAX_AVL_DEPTH` bound. Purely internal — no API or observable-behavior change beyond throughput (~25–33% lower per-op latency under real `F_FULLFSYNC`, `benches/alloc.rs`). +- **`GhostTreeBstackAllocator` version bumped to 0.1.3** (`alloc` + `set` features): Magic number updated from `ALGT\x00\x01\x02\x00` to `ALGT\x00\x01\x03\x00`. Reflects the new per-node child-height cache stored in the AVL node header's previously-reserved bytes. Existing 0.1.x files remain fully compatible (only the first 6 bytes are checked on open). + ## [0.4.1] - 2026-08-03 ### Added diff --git a/PLANNED.md b/PLANNED.md index 625a661..88f77df 100644 --- a/PLANNED.md +++ b/PLANNED.md @@ -113,24 +113,6 @@ Reference: https://github.com/williamwutq/bstack/pull/37 - **Recovery format.** Whether commit reuses the existing multi-write journal (`wip_aux = MultiWrite`) unchanged — the guard still reduces to a flat non-overlapping `[offset, data]` set — or needs its own `wip_aux` mode. - **Naming.** `BStackInPlaceGuard`/`inplace_guard()`/`commit()`/`is_inplace_guarded()` are working names. -## GhostTree allocator: multithreaded performance improvement - -**Feature flag:** `alloc` (optionally `atomic` for the `Sync` path) -**Breaking change:** No — internal implementation only. - -### Motivation - -`benches/alloc.rs` result shows `GhostTreeBstackAllocator` is already the fastest general-purpose allocator in the suite, but its scaling under concurrency has received less attention than its single-threaded design: throughput rises from 1t to 4t and then flattens through 16t. This is consistent with `GhostTreeBstackAllocator::lock` — the single mutex serializing all non-tail `alloc`/`dealloc`/`realloc` — capping throughput once contention saturates it. As the crate's best-performing allocator, it is also the one most likely to be used under concurrent load, so its scaling behavior warrants continued performance work, independent of any specific defect. - -The mutex's scope and implementation are not in question here (see the `NOT PLANNED` entry on `FirstFitBStackAllocator`'s mutex), and no tree-sharding or other data-structure redesign is intended — that would be a different allocator. The improvement surface is reducing the amount of work done per operation while the mutex is held. Two examples found by code inspection, illustrative rather than exhaustive: - -- `avl_insert`, `avl_find_best_fit_and_remove`, and `avl_remove_min` each allocate a `Vec::with_capacity(MAX_AVL_DEPTH)` path buffer under the mutex on every call, despite `MAX_AVL_DEPTH` being a fixed compile-time bound that a stack array could cover instead. -- In the up-pass of `avl_insert` and `avl_find_best_fit_and_remove`, `avl_write_and_update` calls `avl_height` on a child subtree whose height was already computed and written in the previous loop iteration, issuing an avoidable `BStack::get_into` (lock + syscall) to re-fetch it. - -### Open questions - -- **Validation.** Whether `mixed/uniform` workload is sufficient to measure improvement, or whether a contention-specific microbenchmark (concurrent non-tail alloc/dealloc only) is needed to isolate the critical-section-size effect. - ## External-merge-sort strategy and partial sort for `BStackChunk` **Feature flag:** `alloc` + `set` + `atomic`, same as the `BStackChunk::sort_by`/`select_nth_by` family it extends. diff --git a/c/bstack.c b/c/bstack.c index cfaddf6..1cff15b 100644 --- a/c/bstack.c +++ b/c/bstack.c @@ -153,10 +153,22 @@ static void win_set_errno(void) } } +/* When BSTACK_TEST_NO_DURABLE_SYNC is defined at compile time, plat_durable_sync + * becomes a no-op. This mirrors the Rust crate, whose durable_sync is a no-op in + * cfg(test)+debug builds (src/io_core.rs): an in-process test or fuzz run tears + * the store down logically and reopens it in-process, so skipping the physical + * sync changes neither observable behavior nor the on-disk bytes, yet on macOS + * F_FULLFSYNC otherwise dominates runtime (minutes → seconds). UNSAFE for any + * build that must survive a real crash — test/fuzz builds only, never production. */ static int plat_durable_sync(bstack_fd_t h) { +#ifdef BSTACK_TEST_NO_DURABLE_SYNC + (void)h; + return 0; +#else if (!FlushFileBuffers(h)) { win_set_errno(); return -1; } return 0; +#endif } static int plat_file_size(bstack_fd_t h, uint64_t *out) @@ -218,14 +230,21 @@ static int plat_ftruncate(bstack_fd_t h, uint64_t size) #else /* !_WIN32 */ +/* No-op under BSTACK_TEST_NO_DURABLE_SYNC — see the note on the Windows + * definition above. Test/fuzz builds only, never production. */ static int plat_durable_sync(bstack_fd_t fd) { +#ifdef BSTACK_TEST_NO_DURABLE_SYNC + (void)fd; + return 0; +#else # ifdef __APPLE__ if (fcntl(fd, F_FULLFSYNC) == 0) return 0; /* Device does not support F_FULLFSYNC — fall back to fdatasync. */ # endif return fdatasync(fd); +#endif } static int plat_file_size(bstack_fd_t fd, uint64_t *out) diff --git a/c/bstack_alloc.c b/c/bstack_alloc.c index d126b0c..e590354 100644 --- a/c/bstack_alloc.c +++ b/c/bstack_alloc.c @@ -2153,7 +2153,7 @@ static void bstack_alloc_lock_destroy(void *lock) * reliably detecting cycles created by a partial rotation crash. */ #define ALGT_MAX_AVL_DEPTH 128u -static const uint8_t algt_magic[8] = {'A','L','G','T',0,1,2,0}; +static const uint8_t algt_magic[8] = {'A','L','G','T',0,1,3,0}; static const uint8_t algt_magic_prefix[6] = {'A','L','G','T',0,1}; typedef struct { @@ -2162,6 +2162,8 @@ typedef struct { uint64_t left; uint64_t right; int went_left; + uint8_t lh; /* this node's cached child heights, read on the way */ + uint8_t rh; /* down; the sibling's is stable for the up-pass write */ } algt_path_entry_t; /* ---- alignment helpers ------------------------------------------------- */ @@ -2204,9 +2206,14 @@ static int algt_write_root(bstack_t *bs, uint64_t root) * [0..8) size (u64 LE) * [8] balance_factor (i8) * [9] height (u8) - * [10..16) reserved / zero + * [10] cached height of left child (u8) ─┐ denormalized so a parent read + * [11] cached height of right child (u8) ─┘ on the way down yields its + * [12..16) reserved / zero untouched sibling's height * [16..24) left child ptr (u64 LE) - * [24..32) right child ptr (u64 LE) */ + * [24..32) right child ptr (u64 LE) + * + * The cache is rebuilt from scratch by algt_coalesce_and_rebalance on open, so + * old-format arenas (whose [10..16) were zero) self-upgrade transparently. */ static int algt_read_node(bstack_t *bs, uint64_t ptr, uint64_t *out_size, int8_t *out_bf, uint8_t *out_height, @@ -2222,18 +2229,45 @@ static int algt_read_node(bstack_t *bs, uint64_t ptr, return 0; } -/* Write (size, left, right) to ptr, computing bf and height from children's - * stored heights in one pass. Sets *out_bf if non-NULL. Returns 0/-1. */ -static int algt_avl_write_and_update(bstack_t *bs, uint64_t ptr, - uint64_t size, uint64_t left, uint64_t right, int8_t *out_bf) +/* Read the node at ptr for a down-pass: size, left, right, and the node's cached + * child heights (*out_lh, *out_rh). Same single bstack_get as algt_read_node; + * the cache lets the up-pass skip re-reading the untouched sibling child. */ +static int algt_read_node_hc(bstack_t *bs, uint64_t ptr, + uint64_t *out_size, uint64_t *out_left, uint64_t *out_right, + uint8_t *out_lh, uint8_t *out_rh) +{ + uint8_t buf[32]; + if (bstack_get(bs, ptr, ptr + 32, buf) != 0) return -1; + *out_size = read_le64(buf); + *out_left = read_le64(buf + 16); + *out_right = read_le64(buf + 24); + *out_lh = buf[10]; + *out_rh = buf[11]; + return 0; +} + +/* Write (size, left, right) to ptr, computing bf and height in one pass. + * + * A child height passed as >= 0 (known_lh / known_rh) is used directly — the + * caller already knows it, e.g. from the node written in the previous up-pass + * step or from a sibling untouched by a rotation — which avoids a bstack_get + * (lock + syscall) to re-read that child. A negative value reads the height + * from the child. Sets *out_bf and *out_height when non-NULL. Returns 0/-1. */ +static int algt_avl_write_h(bstack_t *bs, uint64_t ptr, uint64_t size, + uint64_t left, uint64_t right, int known_lh, int known_rh, + int8_t *out_bf, uint8_t *out_height) { uint8_t lh = 0, rh = 0; - if (left != ALGT_NULL_PTR) { + if (known_lh >= 0) { + lh = (uint8_t)known_lh; + } else if (left != ALGT_NULL_PTR) { uint8_t buf[32]; if (bstack_get(bs, left, left + 32, buf) != 0) return -1; lh = buf[9]; } - if (right != ALGT_NULL_PTR) { + if (known_rh >= 0) { + rh = (uint8_t)known_rh; + } else if (right != ALGT_NULL_PTR) { uint8_t buf[32]; if (bstack_get(bs, right, right + 32, buf) != 0) return -1; rh = buf[9]; @@ -2245,81 +2279,113 @@ static int algt_avl_write_and_update(bstack_t *bs, uint64_t ptr, uint8_t buf[32]; memset(buf, 0, 32); write_le64(buf, size); - buf[8] = (uint8_t)bf; - buf[9] = height; + buf[8] = (uint8_t)bf; + buf[9] = height; + buf[10] = lh; /* cached left-child height */ + buf[11] = rh; /* cached right-child height */ write_le64(buf + 16, left); write_le64(buf + 24, right); if (bstack_set(bs, ptr, buf, 32) != 0) return -1; - if (out_bf) *out_bf = bf; + if (out_bf) *out_bf = bf; + if (out_height) *out_height = height; } return 0; } +/* Write (size, left, right) to ptr, reading both child heights. Thin wrapper + * over algt_avl_write_h. Sets *out_bf if non-NULL. Returns 0/-1. */ +static int algt_avl_write_and_update(bstack_t *bs, uint64_t ptr, + uint64_t size, uint64_t left, uint64_t right, int8_t *out_bf) +{ + return algt_avl_write_h(bs, ptr, size, left, right, -1, -1, out_bf, NULL); +} + /* ---- AVL helpers ------------------------------------------------------- */ -/* Right-rotate around node; return the new subtree root. */ -static int algt_avl_rotate_right(bstack_t *bs, uint64_t node, uint64_t *out_root) +/* Right-rotate around node; return the new subtree root and (if out_height is + * non-NULL) its height. */ +static int algt_avl_rotate_right(bstack_t *bs, uint64_t node, + uint64_t *out_root, uint8_t *out_height) { uint64_t node_sz, node_r, pivot, pivot_sz, pivot_l, pivot_r; - int8_t bf; uint8_t height; - if (algt_read_node(bs, node, &node_sz, &bf, &height, &pivot, &node_r) != 0) return -1; - if (algt_read_node(bs, pivot, &pivot_sz, &bf, &height, &pivot_l, &pivot_r) != 0) return -1; - if (algt_avl_write_and_update(bs, node, node_sz, pivot_r, node_r, NULL) != 0) return -1; - if (algt_avl_write_and_update(bs, pivot, pivot_sz, pivot_l, node, NULL) != 0) return -1; + uint8_t node_lh, node_rh, pivot_lh, pivot_rh, node_h; + /* Read both nodes' cached child heights so neither rewrite re-reads a child. + * (node's left child is pivot, whose height node_lh is not needed.) */ + if (algt_read_node_hc(bs, node, &node_sz, &pivot, &node_r, &node_lh, &node_rh) != 0) return -1; + if (algt_read_node_hc(bs, pivot, &pivot_sz, &pivot_l, &pivot_r, &pivot_lh, &pivot_rh) != 0) return -1; + (void)node_lh; + /* node's new children (pivot_r, node_r) → heights (pivot_rh, node_rh). */ + if (algt_avl_write_h(bs, node, node_sz, pivot_r, node_r, + (int)pivot_rh, (int)node_rh, NULL, &node_h) != 0) return -1; + /* pivot's new children (pivot_l, node) → heights (pivot_lh, node_h). */ + if (algt_avl_write_h(bs, pivot, pivot_sz, pivot_l, node, + (int)pivot_lh, (int)node_h, NULL, out_height) != 0) return -1; *out_root = pivot; return 0; } -/* Left-rotate around node; return the new subtree root. */ -static int algt_avl_rotate_left(bstack_t *bs, uint64_t node, uint64_t *out_root) +/* Left-rotate around node; return the new subtree root and (if out_height is + * non-NULL) its height. */ +static int algt_avl_rotate_left(bstack_t *bs, uint64_t node, + uint64_t *out_root, uint8_t *out_height) { uint64_t node_sz, node_l, pivot, pivot_sz, pivot_l, pivot_r; - int8_t bf; uint8_t height; - if (algt_read_node(bs, node, &node_sz, &bf, &height, &node_l, &pivot) != 0) return -1; - if (algt_read_node(bs, pivot, &pivot_sz, &bf, &height, &pivot_l, &pivot_r) != 0) return -1; - if (algt_avl_write_and_update(bs, node, node_sz, node_l, pivot_l, NULL) != 0) return -1; - if (algt_avl_write_and_update(bs, pivot, pivot_sz, node, pivot_r, NULL) != 0) return -1; + uint8_t node_lh, node_rh, pivot_lh, pivot_rh, node_h; + /* Read both nodes' cached child heights so neither rewrite re-reads a child. + * (node's right child is pivot, whose height node_rh is not needed.) */ + if (algt_read_node_hc(bs, node, &node_sz, &node_l, &pivot, &node_lh, &node_rh) != 0) return -1; + if (algt_read_node_hc(bs, pivot, &pivot_sz, &pivot_l, &pivot_r, &pivot_lh, &pivot_rh) != 0) return -1; + (void)node_rh; + /* node's new children (node_l, pivot_l) → heights (node_lh, pivot_lh). */ + if (algt_avl_write_h(bs, node, node_sz, node_l, pivot_l, + (int)node_lh, (int)pivot_lh, NULL, &node_h) != 0) return -1; + /* pivot's new children (node, pivot_r) → heights (node_h, pivot_rh). */ + if (algt_avl_write_h(bs, pivot, pivot_sz, node, pivot_r, + (int)node_h, (int)pivot_rh, NULL, out_height) != 0) return -1; *out_root = pivot; return 0; } -/* Fix imbalance at node (uses < -1 / > 1 to handle post-crash excess). */ -static int algt_avl_rebalance(bstack_t *bs, uint64_t node, uint64_t *out_root) +/* Fix imbalance at node (uses < -1 / > 1 to handle post-crash excess). + * + * The caller passes the bf and height already computed by the algt_avl_write_h + * that installed node's current children, so the common in-balance case needs + * no further I/O. Returns the (possibly new) subtree root and its height. */ +static int algt_avl_rebalance(bstack_t *bs, uint64_t node, int8_t bf, uint8_t height, + uint64_t *out_root, uint8_t *out_height) { - uint64_t size, left, right; - int8_t bf; uint8_t height; - if (algt_read_node(bs, node, &size, &bf, &height, &left, &right) != 0) return -1; - if (algt_avl_write_and_update(bs, node, size, left, right, &bf) != 0) return -1; - if (bf < -1) { - uint64_t left_sz, left_l, left_r; - int8_t left_bf; uint8_t left_h; + uint64_t size, left, right, left_sz, left_l, left_r; + int8_t nbf, left_bf; uint8_t nh, left_h; + if (algt_read_node(bs, node, &size, &nbf, &nh, &left, &right) != 0) return -1; if (algt_read_node(bs, left, &left_sz, &left_bf, &left_h, &left_l, &left_r) != 0) return -1; if (left_bf > 0) { /* Left-right: rotate left child left first */ uint64_t new_left; - if (algt_avl_rotate_left(bs, left, &new_left) != 0) return -1; + if (algt_avl_rotate_left(bs, left, &new_left, NULL) != 0) return -1; if (algt_avl_write_and_update(bs, node, size, new_left, right, NULL) != 0) return -1; } - return algt_avl_rotate_right(bs, node, out_root); + return algt_avl_rotate_right(bs, node, out_root, out_height); } if (bf > 1) { - uint64_t right_sz, right_l, right_r; - int8_t right_bf; uint8_t right_h; + uint64_t size, left, right, right_sz, right_l, right_r; + int8_t nbf, right_bf; uint8_t nh, right_h; + if (algt_read_node(bs, node, &size, &nbf, &nh, &left, &right) != 0) return -1; if (algt_read_node(bs, right, &right_sz, &right_bf, &right_h, &right_l, &right_r) != 0) return -1; if (right_bf < 0) { /* Right-left: rotate right child right first */ uint64_t new_right; - if (algt_avl_rotate_right(bs, right, &new_right) != 0) return -1; + if (algt_avl_rotate_right(bs, right, &new_right, NULL) != 0) return -1; if (algt_avl_write_and_update(bs, node, size, left, new_right, NULL) != 0) return -1; } - return algt_avl_rotate_left(bs, node, out_root); + return algt_avl_rotate_left(bs, node, out_root, out_height); } - *out_root = node; + *out_root = node; + *out_height = height; return 0; } @@ -2337,10 +2403,10 @@ static int algt_avl_insert(bstack_t *bs, uint64_t ptr, uint64_t size) current = root; while (current != ALGT_NULL_PTR) { uint64_t root_sz, left, right; - int8_t bf; uint8_t height; + uint8_t lh, rh; int went_left; if (path_len >= ALGT_MAX_AVL_DEPTH) { errno = EINVAL; return -1; } - if (algt_read_node(bs, current, &root_sz, &bf, &height, &left, &right) != 0) + if (algt_read_node_hc(bs, current, &root_sz, &left, &right, &lh, &rh) != 0) return -1; went_left = (size < root_sz || (size == root_sz && ptr < current)); path[path_len].ptr = current; @@ -2348,32 +2414,47 @@ static int algt_avl_insert(bstack_t *bs, uint64_t ptr, uint64_t size) path[path_len].left = left; path[path_len].right = right; path[path_len].went_left = went_left; + path[path_len].lh = lh; + path[path_len].rh = rh; path_len++; current = went_left ? left : right; } { uint8_t buf[32]; - memset(buf, 0, 32); + memset(buf, 0, 32); /* null children → cached child heights [10],[11] = 0 */ write_le64(buf, size); buf[9] = 1; if (bstack_set(bs, ptr, buf, 32) != 0) return -1; } + /* Up-pass: install the new child pointer in each ancestor and rebalance. + * Both child heights are known — the modified child from the previous + * iteration, the untouched sibling from this node's cache read on the way + * down — so algt_avl_write_h reads neither. The (bf, height) it returns is + * handed to algt_avl_rebalance, so the in-balance case does no further I/O: + * one write per level, zero reads. */ child = ptr; - for (i = (int)path_len - 1; i >= 0; i--) { - uint64_t new_left, new_right, new_child; - if (path[i].went_left) { - new_left = child; - new_right = path[i].right; - } else { - new_left = path[i].left; - new_right = child; + { + uint8_t child_h = 1; /* leaf height */ + for (i = (int)path_len - 1; i >= 0; i--) { + uint64_t new_left, new_right, new_child; + int known_lh, known_rh; + int8_t bf; uint8_t h, new_h; + if (path[i].went_left) { + new_left = child; new_right = path[i].right; + known_lh = (int)child_h; known_rh = (int)path[i].rh; + } else { + new_left = path[i].left; new_right = child; + known_lh = (int)path[i].lh; known_rh = (int)child_h; + } + if (algt_avl_write_h(bs, path[i].ptr, path[i].size, + new_left, new_right, known_lh, known_rh, &bf, &h) != 0) + return -1; + if (algt_avl_rebalance(bs, path[i].ptr, bf, h, &new_child, &new_h) != 0) return -1; + child = new_child; + child_h = new_h; } - if (algt_avl_write_and_update(bs, path[i].ptr, path[i].size, - new_left, new_right, NULL) != 0) return -1; - if (algt_avl_rebalance(bs, path[i].ptr, &new_child) != 0) return -1; - child = new_child; } return algt_write_root(bs, child); } @@ -2386,27 +2467,36 @@ static int algt_avl_insert(bstack_t *bs, uint64_t ptr, uint64_t size) static int algt_avl_remove_min(bstack_t *bs, uint64_t root, uint64_t *out_min_ptr, uint64_t *out_min_size, uint64_t *out_new_root) { - uint64_t stk_ptr [ALGT_MAX_AVL_DEPTH]; - uint64_t stk_size [ALGT_MAX_AVL_DEPTH]; - uint64_t stk_right[ALGT_MAX_AVL_DEPTH]; + uint64_t stk_ptr [ALGT_MAX_AVL_DEPTH]; + uint64_t stk_size [ALGT_MAX_AVL_DEPTH]; + uint64_t stk_right [ALGT_MAX_AVL_DEPTH]; + uint8_t stk_right_h[ALGT_MAX_AVL_DEPTH]; /* cached right-child heights */ size_t stk_len = 0; uint64_t current = root; for (;;) { uint64_t size, left, right; - int8_t bf; uint8_t height; - if (algt_read_node(bs, current, &size, &bf, &height, &left, &right) != 0) + uint8_t lh, rh; + if (algt_read_node_hc(bs, current, &size, &left, &right, &lh, &rh) != 0) return -1; + (void)lh; if (left == ALGT_NULL_PTR) { + /* Replace current with its right child, whose height is current's + * cached right height. `child` is always the left side going up; + * both child heights are known each step, so write_h reads neither. */ uint64_t child = right; + uint8_t child_h = rh; int i; for (i = (int)stk_len - 1; i >= 0; i--) { uint64_t new_child; - if (algt_avl_write_and_update(bs, stk_ptr[i], stk_size[i], - child, stk_right[i], NULL) != 0) + int8_t bf; uint8_t h, new_h; + if (algt_avl_write_h(bs, stk_ptr[i], stk_size[i], + child, stk_right[i], (int)child_h, (int)stk_right_h[i], + &bf, &h) != 0) return -1; - if (algt_avl_rebalance(bs, stk_ptr[i], &new_child) != 0) return -1; - child = new_child; + if (algt_avl_rebalance(bs, stk_ptr[i], bf, h, &new_child, &new_h) != 0) return -1; + child = new_child; + child_h = new_h; } *out_min_ptr = current; *out_min_size = size; @@ -2414,9 +2504,10 @@ static int algt_avl_remove_min(bstack_t *bs, uint64_t root, return 0; } if (stk_len >= ALGT_MAX_AVL_DEPTH) { errno = EINVAL; return -1; } - stk_ptr [stk_len] = current; - stk_size [stk_len] = size; - stk_right[stk_len] = right; + stk_ptr [stk_len] = current; + stk_size [stk_len] = size; + stk_right [stk_len] = right; + stk_right_h[stk_len] = rh; stk_len++; current = left; } @@ -2435,8 +2526,9 @@ static int algt_avl_find_best_fit_and_remove(bstack_t *bs, uint64_t min_size, size_t last_fit_idx = 0; uint64_t root, current; uint64_t found_ptr, found_size, found_left, found_right; - uint64_t replacement, child; - int i; + uint8_t found_lh, found_rh; + uint64_t child; + int i, child_h; if (algt_read_root(bs, &root) != 0) return -1; if (root == ALGT_NULL_PTR) { @@ -2448,10 +2540,10 @@ static int algt_avl_find_best_fit_and_remove(bstack_t *bs, uint64_t min_size, current = root; while (current != ALGT_NULL_PTR) { uint64_t root_sz, left, right; - int8_t bf; uint8_t height; + uint8_t lh, rh; int went_left; if (path_len >= ALGT_MAX_AVL_DEPTH) { errno = EINVAL; return -1; } - if (algt_read_node(bs, current, &root_sz, &bf, &height, &left, &right) != 0) + if (algt_read_node_hc(bs, current, &root_sz, &left, &right, &lh, &rh) != 0) return -1; if (root_sz >= min_size) { last_fit_idx = path_len; @@ -2465,6 +2557,8 @@ static int algt_avl_find_best_fit_and_remove(bstack_t *bs, uint64_t min_size, path[path_len].left = left; path[path_len].right = right; path[path_len].went_left = went_left; + path[path_len].lh = lh; + path[path_len].rh = rh; path_len++; current = went_left ? left : right; } @@ -2479,34 +2573,48 @@ static int algt_avl_find_best_fit_and_remove(bstack_t *bs, uint64_t min_size, found_size = path[last_fit_idx].size; found_left = path[last_fit_idx].left; found_right = path[last_fit_idx].right; + found_lh = path[last_fit_idx].lh; /* cached found_left height */ + found_rh = path[last_fit_idx].rh; /* cached found_right height */ + /* Seed the up-pass directly with the best-fit node's replacement. child_h + * is the replacement subtree's height — known in every case now (single + * child from the cache, successor from its rebalance). */ if (found_left == ALGT_NULL_PTR) { - replacement = found_right; + child = found_right; + child_h = (int)found_rh; } else if (found_right == ALGT_NULL_PTR) { - replacement = found_left; + child = found_left; + child_h = (int)found_lh; } else { uint64_t succ, succ_sz, new_right; + int8_t bf; uint8_t h, rh; if (algt_avl_remove_min(bs, found_right, &succ, &succ_sz, &new_right) != 0) return -1; - if (algt_avl_write_and_update(bs, succ, succ_sz, found_left, new_right, NULL) != 0) + if (algt_avl_write_h(bs, succ, succ_sz, found_left, new_right, -1, -1, &bf, &h) != 0) return -1; - if (algt_avl_rebalance(bs, succ, &replacement) != 0) return -1; + if (algt_avl_rebalance(bs, succ, bf, h, &child, &rh) != 0) return -1; + child_h = (int)rh; } - child = replacement; + /* Up-pass: both child heights are known each step — the modified child + * threaded from below, the untouched sibling from this node's cache — so + * algt_avl_write_h reads neither and the in-balance case does no further I/O. */ for (i = (int)last_fit_idx - 1; i >= 0; i--) { uint64_t new_left, new_right, new_child; + int known_lh, known_rh; + int8_t bf; uint8_t h, new_h; if (path[i].went_left) { - new_left = child; - new_right = path[i].right; + new_left = child; new_right = path[i].right; + known_lh = child_h; known_rh = (int)path[i].rh; } else { - new_left = path[i].left; - new_right = child; - } - if (algt_avl_write_and_update(bs, path[i].ptr, path[i].size, - new_left, new_right, NULL) != 0) return -1; - if (algt_avl_rebalance(bs, path[i].ptr, &new_child) != 0) return -1; - child = new_child; + new_left = path[i].left; new_right = child; + known_lh = (int)path[i].lh; known_rh = child_h; + } + if (algt_avl_write_h(bs, path[i].ptr, path[i].size, + new_left, new_right, known_lh, known_rh, &bf, &h) != 0) return -1; + if (algt_avl_rebalance(bs, path[i].ptr, bf, h, &new_child, &new_h) != 0) return -1; + child = new_child; + child_h = (int)new_h; } if (algt_write_root(bs, child) != 0) return -1; diff --git a/src/alloc/ghost_tree.rs b/src/alloc/ghost_tree.rs index 31cce9e..dba9834 100644 --- a/src/alloc/ghost_tree.rs +++ b/src/alloc/ghost_tree.rs @@ -13,7 +13,7 @@ use std::marker::PhantomData; #[cfg(feature = "atomic")] use std::sync::Mutex; -const ALGT_MAGIC: [u8; 8] = *b"ALGT\x00\x01\x02\x00"; +const ALGT_MAGIC: [u8; 8] = *b"ALGT\x00\x01\x03\x00"; const ALGT_MAGIC_PREFIX: [u8; 6] = *b"ALGT\x00\x01"; /// Payload offset of the magic number. @@ -39,6 +39,13 @@ const MAX_AVL_DEPTH: u32 = 128; const NODE_SIZE_OFF: u64 = 0; const NODE_BF_OFF: u64 = 8; // i8 balance factor const NODE_HEIGHT_OFF: u64 = 9; // u8 height (max ~59 for balanced; slightly more tolerated) +// Cached child heights, denormalized so a parent read during a down-pass yields +// the height of its untouched (sibling) child without a separate read of that +// child on the way back up. Maintained by every node write; rebuilt from +// scratch by `coalesce_and_rebalance` on open, so old-format arenas (whose +// reserved bytes were zero) self-upgrade transparently. +const NODE_LH_OFF: u64 = 10; // u8 cached height of the left child +const NODE_RH_OFF: u64 = 11; // u8 cached height of the right child const NODE_LEFT_OFF: u64 = 16; const NODE_RIGHT_OFF: u64 = 24; @@ -47,13 +54,21 @@ const NODE_RIGHT_OFF: u64 = 24; /// Used by [`avl_insert`](GhostTreeBstackAllocator::avl_insert) and /// [`avl_find_best_fit_and_remove`](GhostTreeBstackAllocator::avl_find_best_fit_and_remove) /// to record the path so that balance factors and heights can be updated on -/// the way back up without recursion. +/// the way back up without recursion. Recorded on a fixed-size stack array of +/// [`MAX_AVL_DEPTH`] entries, so it must be `Copy` and cheaply default-able. +#[derive(Clone, Copy, Default)] struct PathEntry { ptr: u64, size: u64, left: u64, right: u64, went_left: bool, + /// This node's cached child heights, read from its denormalized cache during + /// the down-pass. The sibling (untouched) child's height is stable across + /// the operation — nothing off the path is modified — so the up-pass writes + /// this node with both child heights known and reads neither. + lh: u8, + rh: u8, } /// A pure-AVL general-purpose allocator built on top of a [`BStack`]. @@ -278,13 +293,32 @@ impl GhostTreeBstackAllocator { Ok((size, bf, height, left, right)) } - /// Write a complete AVL node at `ptr`. + /// Read the node at `ptr` for a down-pass, returning `(size, left, right, + /// lh, rh)` where `lh`/`rh` are the node's cached child heights. Same single + /// `get` as [`read_node`](Self::read_node); the cache lets the up-pass skip + /// re-reading the untouched sibling child. + fn read_node_hc(&self, ptr: u64) -> io::Result<(u64, u64, u64, u8, u8)> { + let buf = &mut [0u8; 32]; + self.stack.get_into(ptr, buf)?; + let size = read_buf_le!(buf, NODE_SIZE_OFF => u64); + let left = read_buf_le!(buf, NODE_LEFT_OFF => u64); + let right = read_buf_le!(buf, NODE_RIGHT_OFF => u64); + let lh = read_buf_le!(buf, NODE_LH_OFF => u8); + let rh = read_buf_le!(buf, NODE_RH_OFF => u8); + Ok((size, left, right, lh, rh)) + } + + /// Write a complete AVL node at `ptr`, including the denormalized child-height + /// cache (`lh`, `rh` = heights of `left`, `right`). + #[allow(clippy::too_many_arguments)] // one serialization site; grouping into a struct would not aid clarity fn write_node( &self, ptr: u64, size: u64, bf: i8, height: u8, + lh: u8, + rh: u8, left: u64, right: u64, ) -> io::Result<()> { @@ -292,6 +326,8 @@ impl GhostTreeBstackAllocator { write_buf!(size => buf, NODE_SIZE_OFF); write_buf!(bf => buf, NODE_BF_OFF); write_buf!(height => buf, NODE_HEIGHT_OFF); + write_buf!(lh => buf, NODE_LH_OFF); + write_buf!(rh => buf, NODE_RH_OFF); write_buf!(left => buf, NODE_LEFT_OFF); write_buf!(right => buf, NODE_RIGHT_OFF); self.stack.set(ptr, buf)?; @@ -322,30 +358,48 @@ impl GhostTreeBstackAllocator { Ok(height) } - /// Write `(size, left, right)` to `ptr`, computing bf and height from the - /// children's stored heights in one pass. Returns the balance factor. + /// Write `(size, left, right)` to `ptr`, computing bf and height in one pass, + /// and return `(bf, height)`. /// - /// Replaces the `write_node(…, 0, 0, …) + avl_update_bf` pair: instead of - /// writing stale zeros and reading back, we read the two child heights once, - /// compute both fields, and write the node exactly once. + /// A child's height passed as `Some` is used directly — the caller already + /// knows it, e.g. from the node written in the previous up-pass step or from + /// a sibling untouched by a rotation — which avoids a `get_into` (lock + + /// syscall) to re-read that child. `None` reads the height from the child. #[inline] - fn avl_write_and_update(&self, ptr: u64, size: u64, left: u64, right: u64) -> io::Result { - let lh = self.avl_height(left)? as i16; - let rh = self.avl_height(right)? as i16; + fn avl_write_h( + &self, + ptr: u64, + size: u64, + left: u64, + right: u64, + lh: Option, + rh: Option, + ) -> io::Result<(i8, u8)> { + let lh = match lh { + Some(h) => h as i16, + None => self.avl_height(left)? as i16, + }; + let rh = match rh { + Some(h) => h as i16, + None => self.avl_height(right)? as i16, + }; let bf = (rh - lh) as i8; let height = (1 + lh.max(rh)) as u8; - self.write_node(ptr, size, bf, height, left, right)?; - Ok(bf) + self.write_node(ptr, size, bf, height, lh as u8, rh as u8, left, right)?; + Ok((bf, height)) } - /// Recompute bf and height for `node` from its children's stored heights, - /// write both back, and return the balance factor. - /// - /// O(1) — delegates to [`avl_write_and_update`](Self::avl_write_and_update). + /// Write `(size, left, right)` to `ptr`, reading both child heights, and + /// return `(bf, height)`. Thin wrapper over [`avl_write_h`](Self::avl_write_h). #[inline] - fn avl_update_bf(&self, node: u64) -> io::Result { - let (size, _, _, left, right) = self.read_node(node)?; - self.avl_write_and_update(node, size, left, right) + fn avl_write_and_update( + &self, + ptr: u64, + size: u64, + left: u64, + right: u64, + ) -> io::Result<(i8, u8)> { + self.avl_write_h(ptr, size, left, right, None, None) } /// Right-rotate around `node`; return the new subtree root. @@ -357,12 +411,24 @@ impl GhostTreeBstackAllocator { /// / \ / \ /// L M M R /// ``` - fn avl_rotate_right(&self, node: u64) -> io::Result { - let (node_sz, _, _, pivot, node_r) = self.read_node(node)?; - let (pivot_sz, _, _, pivot_l, pivot_r) = self.read_node(pivot)?; - self.avl_write_and_update(node, node_sz, pivot_r, node_r)?; - self.avl_write_and_update(pivot, pivot_sz, pivot_l, node)?; - Ok(pivot) + fn avl_rotate_right(&self, node: u64) -> io::Result<(u64, u8)> { + // Read both nodes' cached child heights so neither rewrite re-reads a + // child. (`node`'s left child is `pivot`, whose height is not needed.) + let (node_sz, pivot, node_r, _node_lh, node_rh) = self.read_node_hc(node)?; + let (pivot_sz, pivot_l, pivot_r, pivot_lh, pivot_rh) = self.read_node_hc(pivot)?; + // `node`'s new children (pivot_r, node_r) → heights (pivot_rh, node_rh). + let (_, node_h) = self.avl_write_h( + node, + node_sz, + pivot_r, + node_r, + Some(pivot_rh), + Some(node_rh), + )?; + // `pivot`'s new children (pivot_l, node) → heights (pivot_lh, node_h). + let (_, pivot_h) = + self.avl_write_h(pivot, pivot_sz, pivot_l, node, Some(pivot_lh), Some(node_h))?; + Ok((pivot, pivot_h)) } /// Left-rotate around `node`; return the new subtree root. @@ -374,28 +440,45 @@ impl GhostTreeBstackAllocator { /// / \ / \ /// M R L M /// ``` - fn avl_rotate_left(&self, node: u64) -> io::Result { - let (node_sz, _, _, node_l, pivot) = self.read_node(node)?; - let (pivot_sz, _, _, pivot_l, pivot_r) = self.read_node(pivot)?; - self.avl_write_and_update(node, node_sz, node_l, pivot_l)?; - self.avl_write_and_update(pivot, pivot_sz, node, pivot_r)?; - Ok(pivot) + fn avl_rotate_left(&self, node: u64) -> io::Result<(u64, u8)> { + // Read both nodes' cached child heights so neither rewrite re-reads a + // child. (`node`'s right child is `pivot`, whose height is not needed.) + let (node_sz, node_l, pivot, node_lh, _node_rh) = self.read_node_hc(node)?; + let (pivot_sz, pivot_l, pivot_r, pivot_lh, pivot_rh) = self.read_node_hc(pivot)?; + // `node`'s new children (node_l, pivot_l) → heights (node_lh, pivot_lh). + let (_, node_h) = self.avl_write_h( + node, + node_sz, + node_l, + pivot_l, + Some(node_lh), + Some(pivot_lh), + )?; + // `pivot`'s new children (node, pivot_r) → heights (node_h, pivot_rh). + let (_, pivot_h) = + self.avl_write_h(pivot, pivot_sz, node, pivot_r, Some(node_h), Some(pivot_rh))?; + Ok((pivot, pivot_h)) } /// Fix imbalance at `node` after an insert or remove, then return the - /// (possibly new) subtree root. Children must already be balanced. + /// (possibly new) subtree root and its height. Children must already be + /// balanced. + /// + /// The caller passes the `bf` and `height` already computed by the + /// [`avl_write_h`](Self::avl_write_h) that installed `node`'s current + /// children, so the common in-balance case needs no further I/O — it just + /// returns `(node, height)`. /// /// Uses `< -1` / `> 1` rather than `== -2` / `== 2` so that a node whose /// balance factor exceeds ±2 (possible after crash recovery) still gets /// corrected instead of silently passed over. - fn avl_rebalance(&self, node: u64) -> io::Result { - let bf = self.avl_update_bf(node)?; + fn avl_rebalance(&self, node: u64, bf: i8, height: u8) -> io::Result<(u64, u8)> { if bf < -1 { let (_, _, _, left, _) = self.read_node(node)?; let (_, left_bf, _, _, _) = self.read_node(left)?; if left_bf > 0 { // Left-right case: rotate left child left first. - let new_left = self.avl_rotate_left(left)?; + let (new_left, _) = self.avl_rotate_left(left)?; let (node_sz, _, _, _, node_r) = self.read_node(node)?; self.avl_write_and_update(node, node_sz, new_left, node_r)?; } @@ -405,13 +488,13 @@ impl GhostTreeBstackAllocator { let (_, right_bf, _, _, _) = self.read_node(right)?; if right_bf < 0 { // Right-left case: rotate right child right first. - let new_right = self.avl_rotate_right(right)?; + let (new_right, _) = self.avl_rotate_right(right)?; let (node_sz, _, _, node_l, _) = self.read_node(node)?; self.avl_write_and_update(node, node_sz, node_l, new_right)?; } self.avl_rotate_left(node) } else { - Ok(node) + Ok((node, height)) } } @@ -419,41 +502,55 @@ impl GhostTreeBstackAllocator { fn avl_insert(&self, ptr: u64, size: u64) -> io::Result<()> { let root = self.read_root()?; - // Down-pass: walk to the insertion position, recording the path. - let mut path: Vec = Vec::with_capacity(MAX_AVL_DEPTH as usize); + // Down-pass: walk to the insertion position, recording the path on a + // fixed-size stack array. `MAX_AVL_DEPTH` is a compile-time bound, so + // this avoids a heap allocation on every insert under the mutex. + let mut path = [PathEntry::default(); MAX_AVL_DEPTH as usize]; + let mut path_len = 0usize; let mut current = root; while current != NULL_PTR { - if path.len() >= MAX_AVL_DEPTH as usize { + if path_len >= MAX_AVL_DEPTH as usize { return Err(io::Error::new( io::ErrorKind::InvalidData, "AVL insert exceeded maximum depth: corrupted tree (possible cycle)", )); } - let (root_sz, _, _, left, right) = self.read_node(current)?; + let (root_sz, left, right, lh, rh) = self.read_node_hc(current)?; let went_left = (size, ptr) < (root_sz, current); - path.push(PathEntry { + path[path_len] = PathEntry { ptr: current, size: root_sz, left, right, went_left, - }); + lh, + rh, + }; + path_len += 1; current = if went_left { left } else { right }; } - // Write the new leaf. - self.write_node(ptr, size, 0, 1, NULL_PTR, NULL_PTR)?; + // Write the new leaf (height 1, null children → cached child heights 0). + self.write_node(ptr, size, 0, 1, 0, 0, NULL_PTR, NULL_PTR)?; - // Up-pass: propagate the new child pointer and rebalance each ancestor. + // Up-pass: install the new child pointer in each ancestor and rebalance. + // Both child heights are known — the modified child from the previous + // iteration, the untouched sibling from this node's cache read on the + // way down — so `avl_write_h` reads neither. The `(bf, height)` it + // returns is handed to `avl_rebalance`, so the in-balance case does no + // further I/O: one write per level, zero reads. let mut child = ptr; - for entry in path.iter().rev() { - let (new_left, new_right) = if entry.went_left { - (child, entry.right) + let mut child_h = 1u8; // leaf height + for entry in path[..path_len].iter().rev() { + let (new_left, new_right, lh, rh) = if entry.went_left { + (child, entry.right, Some(child_h), Some(entry.rh)) } else { - (entry.left, child) + (entry.left, child, Some(entry.lh), Some(child_h)) }; - self.avl_write_and_update(entry.ptr, entry.size, new_left, new_right)?; - child = self.avl_rebalance(entry.ptr)?; + let (bf, h) = self.avl_write_h(entry.ptr, entry.size, new_left, new_right, lh, rh)?; + let (new_root, new_h) = self.avl_rebalance(entry.ptr, bf, h)?; + child = new_root; + child_h = new_h; } self.write_root(child) } @@ -464,27 +561,45 @@ impl GhostTreeBstackAllocator { /// Returns `(min_ptr, min_size, new_subtree_root)`. The minimum node always /// has no left child, so its replacement is its right child (or [`NULL_PTR`]). fn avl_remove_min(&self, root: u64) -> io::Result<(u64, u64, u64)> { - // Walk left, recording (ptr, size, right_child) for each ancestor. - let mut path: Vec<(u64, u64, u64)> = Vec::with_capacity(MAX_AVL_DEPTH as usize); + // Walk left, recording (ptr, size, right_child, right_child_height) for + // each ancestor on a fixed-size stack array (no heap allocation under the + // mutex). The cached right-child height lets the up-pass write each + // ancestor with both child heights known. + let mut path = [(0u64, 0u64, 0u64, 0u8); MAX_AVL_DEPTH as usize]; + let mut path_len = 0usize; let mut current = root; loop { - let (size, _, _, left, right) = self.read_node(current)?; + let (size, left, right, _lh, rh) = self.read_node_hc(current)?; if left == NULL_PTR { - // `current` is the minimum; replace it with its right child. + // `current` is the minimum; replace it with its right child, whose + // height is `current`'s cached right height. `child` is always + // the left side going up; both child heights are known each step, + // so `avl_write_h` reads neither. let mut child = right; - for &(anc_ptr, anc_sz, anc_right) in path.iter().rev() { - self.avl_write_and_update(anc_ptr, anc_sz, child, anc_right)?; - child = self.avl_rebalance(anc_ptr)?; + let mut child_h = rh; + for &(anc_ptr, anc_sz, anc_right, anc_right_h) in path[..path_len].iter().rev() { + let (bf, h) = self.avl_write_h( + anc_ptr, + anc_sz, + child, + anc_right, + Some(child_h), + Some(anc_right_h), + )?; + let (new_root, new_h) = self.avl_rebalance(anc_ptr, bf, h)?; + child = new_root; + child_h = new_h; } return Ok((current, size, child)); } - if path.len() >= MAX_AVL_DEPTH as usize { + if path_len >= MAX_AVL_DEPTH as usize { return Err(io::Error::new( io::ErrorKind::InvalidData, "AVL min exceeded maximum depth: corrupted tree (possible cycle)", )); } - path.push((current, size, right)); + path[path_len] = (current, size, right, rh); + path_len += 1; current = left; } } @@ -504,37 +619,45 @@ impl GhostTreeBstackAllocator { return Ok(None); } - // Down-pass: record the full traversal path and the index of the last + // Down-pass: record the full traversal path (on a fixed-size stack + // array, no heap allocation under the mutex) and the index of the last // node that satisfies size >= min_size (the best fit). - let mut path: Vec = Vec::with_capacity(MAX_AVL_DEPTH as usize); + let mut path = [PathEntry::default(); MAX_AVL_DEPTH as usize]; + let mut path_len = 0usize; let mut last_fit_idx: Option = None; let mut current = root; while current != NULL_PTR { - if path.len() >= MAX_AVL_DEPTH as usize { + if path_len >= MAX_AVL_DEPTH as usize { return Err(io::Error::new( io::ErrorKind::InvalidData, "AVL find exceeded maximum depth: corrupted tree (possible cycle)", )); } - let (root_sz, _, _, left, right) = self.read_node(current)?; + let (root_sz, left, right, lh, rh) = self.read_node_hc(current)?; if root_sz >= min_size { - last_fit_idx = Some(path.len()); - path.push(PathEntry { + last_fit_idx = Some(path_len); + path[path_len] = PathEntry { ptr: current, size: root_sz, left, right, went_left: true, - }); + lh, + rh, + }; + path_len += 1; current = left; } else { - path.push(PathEntry { + path[path_len] = PathEntry { ptr: current, size: root_sz, left, right, went_left: false, - }); + lh, + rh, + }; + path_len += 1; current = right; } } @@ -548,30 +671,45 @@ impl GhostTreeBstackAllocator { let found_size = path[fit_idx].size; let found_left = path[fit_idx].left; let found_right = path[fit_idx].right; + // The found node's cached child heights (it was reached via a left + // descent, so its right subtree — path[fit_idx+1..] — is what the search + // exhausted; both children are untouched by the removal of this node). + let found_lh = path[fit_idx].lh; + let found_rh = path[fit_idx].rh; // Remove the best-fit node. The left subtree (path[fit_idx+1..]) was // searched and yielded nothing, so found_left is returned unchanged. - let replacement = if found_left == NULL_PTR { - found_right + // `repl_h` is the replacement subtree's height, seeding the up-pass — + // known in every case now (single child from the cache, successor from + // its rebalance). + let (replacement, repl_h): (u64, u8) = if found_left == NULL_PTR { + (found_right, found_rh) } else if found_right == NULL_PTR { - found_left + (found_left, found_lh) } else { // Two children: replace with in-order successor (min of right subtree). let (succ, succ_sz, new_right) = self.avl_remove_min(found_right)?; - self.avl_write_and_update(succ, succ_sz, found_left, new_right)?; - self.avl_rebalance(succ)? + let (bf, h) = self.avl_write_and_update(succ, succ_sz, found_left, new_right)?; + let (new_root, new_h) = self.avl_rebalance(succ, bf, h)?; + (new_root, new_h) }; - // Up-pass: update path[0..fit_idx] (path[fit_idx] was removed). + // Up-pass: update path[0..fit_idx] (path[fit_idx] was removed). Both + // child heights are known each step — the modified child threaded from + // below, the untouched sibling from this node's cache — so `avl_write_h` + // reads neither and the in-balance case does no further I/O. let mut child = replacement; + let mut child_h = repl_h; for entry in path[..fit_idx].iter().rev() { - let (new_left, new_right) = if entry.went_left { - (child, entry.right) + let (new_left, new_right, lh, rh) = if entry.went_left { + (child, entry.right, Some(child_h), Some(entry.rh)) } else { - (entry.left, child) + (entry.left, child, Some(entry.lh), Some(child_h)) }; - self.avl_write_and_update(entry.ptr, entry.size, new_left, new_right)?; - child = self.avl_rebalance(entry.ptr)?; + let (bf, h) = self.avl_write_h(entry.ptr, entry.size, new_left, new_right, lh, rh)?; + let (new_root, new_h) = self.avl_rebalance(entry.ptr, bf, h)?; + child = new_root; + child_h = new_h; } self.write_root(child)?; Ok(Some((found_ptr, found_size)))