diff --git a/CHANGELOG.md b/CHANGELOG.md index dcc9f04..93c0745 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,6 +14,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - **`BStackChunk<'a>`/`BStackChunkIter<'a>` — fixed-stride view over `BStackSlice` (`alloc`).** `BStackSlice::chunks`/`rchunks` (mirrored on `BStackOwnedSlice`) return `(BStackChunk, BStackSlice)`: aligned view + remainder, pure offset arithmetic, no I/O. `as_slice`/`into_slice`/`with_stride` recover or re-chunk the aligned region. `PartialEq`/`Eq`/`Hash`/`PartialOrd`/`Ord` on `(chunk_len, region)`; no cross-type comparison with `BStackSlice`. Not an iterator itself: `iter()`/`IntoIterator` yield a `BStackChunkIter` (`DoubleEndedIterator` + `ExactSizeIterator` + `FusedIterator`), zero I/O per step. - **`BStackChunk` search/sort/select.** `binary_search_by`/`binary_search_by_key` (`alloc`): O(log n) chunk reads. `sort_by`/`sort_by_key`/`select_nth_by`/`select_nth_by_key` (`set` + `atomic`): one crash-atomic `BStack::process` call, in-place cycle-following permutation (O(1) scratch chunks, stack-allocated ≤128 B); `select_nth_*` mirrors `[T]::select_nth_unstable_by`. - **`BStackRange`/`BStackSlice`/`BStackChunk` — overlap, adjacency, and merge queries (`alloc`).** `BStackRange::overlaps`/`adjacent_to` are pure `(offset, len)` arithmetic, no I/O. `merge`/`merge_adjacent` combine two ranges into one covering union: `merge` succeeds on overlap or when either range is empty (empty acts as an identity element, returned unchanged); `merge_adjacent` is stricter, requiring the ranges to touch end-to-end with both non-empty. `BStackSlice` mirrors all four, delegating to the underlying `BStackRange`, and additionally returns `None` from `merge`/`merge_adjacent` if the two slices are backed by different `BStack`s. `BStackChunk` adds `same_stride`/`same_phase` plus stride-aware `adjacent_to`/`overlaps`/`merge`/`merge_adjacent`, requiring `same_phase`. An exception is that `merge` treats an empty, same-stride chunk as a phase-agnostic identity element. Not provided on `BStackOwnedSlice` — ownership/allocation semantics leave "adjacent" and "mergeable" without a meaningful definition for an owned handle. +- **`SegregatedBStackAllocator` (`alloc` + `set`, experimental): segregated (binned) free-list allocator.** Generalises `CheckedSlabBStackAllocator` from one block size to 33 size classes sharing one arena — 16 linear (16‥256 B, step 16), 16 geometric (320‥4096 B, 4 per octave), and one shared oversized bucket — with the class derived from the request by register arithmetic (no tables) for O(1) classed alloc/dealloc. Each block carries an 8-byte overhead tag (in-use length / free physical size, the latter doubling as the class tag), so leaked blocks are reclaimable by a linear scan and double-frees are caught. A single `new(stack)` constructor initialises a fresh stack or reopens one, running recovery automatically; `recover()` is `unsafe` (requires a quiescent allocator). `Send` in all configurations; `Send + Sync` with `atomic`, no allocator-level lock — free-list pops/pushes ride `BStack::process_gen`/`inplace_gen`. Without `atomic` the in-place non-tail-shrink carve is unavailable (that `realloc` takes a move instead). Experimental: the on-disk format (`ALSG` magic) and API are not yet stable, and the background coalescer and deep in-use-leak GC are unimplemented. ### Changed diff --git a/PLANNED.md b/PLANNED.md index 7567b84..88f77df 100644 --- a/PLANNED.md +++ b/PLANNED.md @@ -142,96 +142,3 @@ Reference: https://github.com/williamwutq/bstack/pull/37 - **Unstable sort.** Whether to also provide `sort_unstable_by`/`sort_unstable_by_key`, mirroring `std`'s stable/unstable split — unstable sort does fewer chunk moves at the cost of stability, which may matter more once movement is spread across multiple sections/passes. - **Threshold sizing.** How the small/medium/large boundary and section size are chosen, and whether it should be caller-configurable given the memory/atomicity tradeoff is now explicit. - **Batch staging cost.** `set_batched` still stages every block's bytes into the multi-write journal before committing, so the final pass's staging cost scales with the number and size of the (new_offset, chunk_bytes) pairs it's fed — worth measuring against the current single-`process` cost before committing to this as the merge-commit strategy. - -## `SegregatedBStackAllocator` (`alloc` + `set`) - -**Feature flag:** `alloc` + `set`; `Send + Sync` only under `atomic` (see Thread safety). -**Breaking change:** No — new allocator type behind a new magic (`ALSG`); no existing allocator is modified. - -### Motivation - -`GhostTreeBstackAllocator` is the fastest *general-purpose* allocator in the crate, but it has structural cost - best-fit is a root-to-leaf AVL descent, issuing O(log N) dependent node reads on the critical path and additional rotation writes on mutation. Although PR #39 already squeezed that path for ~27% lower per-op latency, it did not remove its fundamental limitation. In addition, under `atomic`, the whole descent runs under the allocator `Mutex`, which means the chain also sets the length of the serialized critical section, capping threaded throughput. - -Results from the benchmark requires an alternative for higher throughput. From the benchmark, we have a few observations to follow. First, on-hot-path coalescing introduces latency: `first_fit` is the only allocator that merges adjacent frees per-`dealloc`, and it is both the slowest general-purpose allocator and the one with pathological tails under contention.Second, a slab allocator configured to the **correct size** is the fastest thing measured (`slab_16/uniform`), because its pop is a single fixed-offset, fixed-shape fused critical section. a **wrong-sized** slab is the slowest (`slab_128`, `slab_24`). - -The conclusion dictates the design of the modern `bstack` on disk allocator. It points out that the solution is to generalize `slab` from one size class to N. By keeping slab's superior performance and minimal critical section while allowing better size fitting for allocation requests, an allocator can process allocation and deallocation of arbitrary sizes at the same class of efficiency as `slab_16`. - -### Design - -- **Segregated (binned) free lists.** The header holds an array of `num_classes` free-list heads. Each is a singly linked list of free blocks, where the pointer `head == 0` meaning empty, and non-empty pointers should point to the next block. As the mimimum physical block size is 16, free blocks has enough space to store their `next_free` offset inline in their own payload, allowing for zero disk space overhead for live allocation (only the 8-byte state overhead, below). Each class is effectively an independent slab sharing one arena. - -- **Hot path is fixed-shape.** `alloc` computes the class from the requested size with register arithmetic, then reads exactly one head at the computed offset `head_base + 8*class` and pops. On an exact-class miss (`head == 0`) it `try_extend`s one fresh block of that class's size and returns it. This is the `pop, else extend` rule slabs already use. It deliberately does **not** search neighbouring classes on a miss, as a data-dependent search would lengthen the critical section under contention, leading to suboptimal concurrency performance. Not only that, it also could introduced the need for new locks in memory, further degrading performance. Class drift/fragmentation is the background coalescer's job, off the hot path. - -- **Size classes are physical block sizes, all multiples of 16.** On disk, we store a class size that includes the 8-byte overhead, so usable `data = class − 8`. Every class size is a multiple of 16. Defining classes on the physical size rather than data size allows splitting blocks to yield blocks that are exact class blocks, and coalescing two adjacent class blocks yields another multiple of 16, letting the overhead to be absorbed inside each piece. Therefore, the boundaries of blocks will stay clean over time, reducing internal fragmentation of the allocator. - - - **Linear floor:** block sizes 16, 32, 48, ..., up to `linear_max`, step 16. Pure power-of-two would waste too much space; the 16-step floor caps small-size waste at < 16 B. Index is direct: `class = (block >> 4) − 1`. - - **Geometric ceiling** (blocks above `linear_max`): each octave `[2^k, 2^{k+1})` is split into `2^subclass_bits` evenly-spaced subclasses, each itself a multiple of 16. `subclass_bits` is a **compile-time constant** (not a stored, caller-tunable field) — it sets *how many* subclasses each power-of-two octave is divided into (`subclass_bits = 2` → 4 per octave), which bounds worst-case internal waste to `2^{−subclass_bits}` of the request (≤ 25% at 2, ≤ 12.5% at 3) in exchange for more classes (larger header). Index via `clz` (octave) + the top `subclass_bits` of the remainder (subclass) — a formula, never a stored table. - -| block-size (class) region | class sizes (bytes, incl. 8 B overhead) | usable data | -|---------------------------|-----------------------------------------|--------------------| -| linear (step 16) | 16, 32, 48, … `linear_max` | 8, 24, 40, … | -| octave (256, 512] ÷4 | 320, 384, 448, 512 | 312, 376, 440, 504 | -| octave (512, 1024] ÷4 | 640, 768, 896, 1024 | … | -| … | … up to `max_class` | … | -| > `max_class` | oversized (raw multi-16 path) | — | - - Alloc rounds the *physical* need up to a class: `class = round_up(len + 8, 16)` then snap into the geometric region if above `linear_max`. Example: `len = 200` → need 208 B → linear class **208** (= 16×13, index 12), 200 B usable, exact fit. `len = 500` → need 508 → octave `(256,512]` → class **512**, 504 B usable. Minimum block is 16 (8 data), matching slab's `≥ 8` data floor. - -- **On-disk layout.** The size→class scheme is a fixed compile-time policy, not per-file state — by the same logic that keeps `subclass_bits` out of the header, `quantum` (16), `linear_max`, and the resulting `NUM_CLASSES` are all constants baked into the build and *encoded by the magic version*, so a format change bumps the magic rather than reinterpreting a stored field. The header therefore carries only the magic, the recovery flag, and the head array; `open` validates the magic and arena alignment. - - ``` - offset 0 reserved (user) 24 B - offset 24 magic "ALSG\x00\x01\x00\x00" 8 B # version encodes the fixed class scheme - offset 32 flags 4 B # bit0 = recovery_needed (CAS-updated) - offset 36 _reserved 4 B - offset 40 free_head[NUM_CLASSES] : u64 each # the only per-op-mutated region; - (pad to 32-B alignment) # last entry is the shared oversized list - arena start (32-B aligned) - ``` - - `NUM_CLASSES` = (linear classes) + (geometric classes) + 1 for the oversized bucket, all fixed at compile time. Each `free_head[c]` holds the block-start offset of that class's first free block, `0` = empty. - - Every arena block is `[ overhead(8) | data(block − 8) ]`; the returned pointer `ptr` is the data start (`block + 8`). The 8-byte overhead is a single tagged word: - -| bit(s) | in_use = 1 (high bit set) | in_use = 0 (high bit clear) | -|--------|----------------------------------------------------------|--------------------------------------| -| 63 | `1` — block is live | `0` — block is free | -| 0–62 | **slice length** — the exact bytes the caller was handed | **physical block size >> 4** — bytes | - - One word, two readings, tag in the top bit — the same high-bit-set convention `CheckedSlab` uses for "in use", so a raw scan reads the sign bit and knows immediately. The choices behind it: - - - **Live blocks store the caller's slice length, not the block size.** The block size is always recoverable as `classify_blocksize(len)` (a formula, no IO), so storing `len` is strictly more information: it lets `recover()` reconstruct exact `(ptr, len)` handles, lets `realloc` know the true occupancy of the current block, and still lets the scan stride by deriving the block size. This allows for smaller range to zero during realloc, if the zeroing strategy is zero-on-allocation and zero-on-grow, and also prevents bugs introduced by the caller by refusing to deallocate a partial slice or a slice originated from erroneous integer math. The physical size of the block can easily be obtained from the stored `len` (round up to the next class). - - **Free blocks store their actual physical size, divided by 16**, which doubles as the class tag: `classify(size)` gives the free-list bucket with no separate class field, and the recovery scan strides directly by it. This is why there is **no oversized bit** — "oversized" is simply what `classify()` returns for any `size > max_class`, collapsing every large size onto one shared free-list head instead of minting a class (and a header slot) for `2^56`-style sizes that would bloat `NUM_CLASSES`. A huge free block records its true size right here in its own overhead; nothing extra is stored out of line. Since every physical block is a multiple of 16, the stored valus is the size shifted right by 4, which fits in the 63 bits available. - - **`next_free` lives at `ptr` (data `[0..8]`).** This is different than the design of `CheckedSlabBStackAllocator` because the minimum block is 16 B (8 overhead + 8 data), every free block has at least 8 data bytes to hold the link, so the overhead word is free to keep the size/class tag *while* the block is free — the two never contend. Head slots and `next_free` both store block-start offsets; `0` is the empty/end sentinel (offset 0 is the header, never a block). - - So `dealloc` needs no stored class: read overhead → derive `size = classify_blocksize(len)` → `classify(size)` picks the head. And double-free is caught up front — if the high bit is already `0`, the block is already free → `InvalidInput` before any list write. - -- **Operations.** - -| operation | strategy | -|---|---| -| `alloc` (`len ≤ max_class`) | `c = classify(len)`; pop `head[c]`, else `try_extend` one class-`c` block; write overhead `in_use\|len` | -| `alloc` (`len > max_class`) | oversized: pop the oversized head if a block fits, else `try_extend` `round_up(len+8, 16)`; write overhead `in_use\|len` | -| `dealloc` | high bit clear → double-free `InvalidInput`; else `size = classify_blocksize(len)`, write overhead `free\|size` + `next_free` + splice to `head[classify(size)]` (one batch) | -| `dealloc` (oversized, at tail) | `try_discard` the whole block (single call) | -| `dealloc` (oversized, mid-arena) | push to oversized head — see Open questions (carve vs mark) | -| `realloc` (same class) | `len` still maps to this class → rewrite overhead `len` only; no list touch | -| `realloc` (grow at tail) | `try_extend_zeros` in place, rewrite `len` | -| `realloc` (cross-class-shrink) | rewrite overhead `len`, split maximum unused tail into free blocks of the same class (this is done to reduce the number of lists touched), push new free blocks onto `head[classify(size)]` (multi-transaction; `recovery_needed`-bracketed) | -| `realloc` (cross-class-grow) | alloc new class, copy, dealloc old (multi-transaction; `recovery_needed`-bracketed) | - -- **Atomicity mapping (`atomic`).** Reuses slab's primitives, so there is **no allocator-level lock on the alloc/dealloc/realloc paths**. The key lever: `BStack::set_batched` / `inplace_gen` (`set` + `atomic`) commit an arbitrary batch of *non-overlapping* writes as one crash-atomic multi-write-journal transaction. A free-list mutation's three writes — the overhead word (block `[0..8]`, `free\|size` ⇄ `in_use\|len`), `next_free` (data `[0..8]`), and `head[class]` (header) — are non-overlapping, so they land in a **single** transaction; there is no ordered "write A then write B" and thus no window between them. Pop is one `process_gen` holding `BStack`'s write lock across read-`head` → read-`next` → the batched write (closing the ABA window a `get`/`cas` pair would open); push is one `cross_exchange` (the specialized 2-write splice) or an `inplace_gen` batch when the overhead flip is bundled in. Tail grow/shrink use `try_extend_zeros` / `try_discard`, check-and-act under `BStack`'s own write lock. `recovery_needed` is a CAS on the flag word. The only retained `Mutex` is held by `recover()` alone, to keep recovery single-flight; the scan itself serializes against live ops through the `BStack` write lock it holds across one `process_gen` sequence, not through the `Mutex`. - -- **Crash consistency.** Under `atomic`, a single-block alloc/dealloc commits its overhead word, `next_free`, and `head[class]` as one crash-atomic `set_batched`/`inplace_gen` transaction, so a crash leaves the block either fully in-use or fully free-and-linked — **no torn state, no leak window**, and a double-free cannot be observed mid-op. Without `atomic` the same mutation is separate `set` calls (write `next_free`, then `head`), so a crash between them leaks ≤ 1 block without corrupting a list. Oversized tail discard is a single crash-atomic `discard`. The leaks `recover()` must reclaim therefore come only from (a) the non-atomic build and (b) genuinely multi-transaction ops — `realloc` grow-by-copy and background coalescing — which bracket their work with the `recovery_needed` flag. No recovery scan is *required* to reopen a cleanly-closed stack. - -- **Recovery.** `recover()` (auto-run by `open`, like `CheckedSlab`) linearly scans the arena, reading each overhead word: a live block (high bit set) strides by `classify_blocksize(len)`; a free block (high bit clear) strides by its stored `size` and is relinked onto `head[classify(size)]` from scratch (stored `next_free` pointers are rebuilt, not trusted). A partial tail block is truncated. Returns the count of blocks it could not classify with certainty (`0` = fully accounted for). - -- **Thread safety.** Always `Send`. Without `atomic`, **not `Sync`**: `head[class]` is read then written as separate `BStack` calls (TOCTOU, two callers could pop the same block). With `atomic`, **`Send + Sync`** via the fused `process_gen`/`cross_exchange` sequences above — no allocator mutex except the recovery single-flight guard. - -### Open questions - -- **Coalescer.** Should the background coalescer be a separate thread, caller triggered (e.g. `alloc::coalesce()`), periodically run by `alloc`/`dealloc`, or only involved in heavy ops (e.g. oversized pop miss)? -- **Oversized mid-arena free.** Carve the span into class blocks (immediately reusable, a few extra IO on a rare path) vs mark-and-leave for the coalescer (cheaper free, slower reclaim). Leaning carve, but it depends on how oversized-heavy the real workload is. -- **Class scheme constants.** `linear_max`, `subclass_bits`, `max_class` are chosen at build time (the quantum is fixed at 16), trading fit-density against `NUM_CLASSES`/header size. Open: the specific values, and whether more than one scheme (i.e. more than one magic version) is worth shipping. If per-call cost turns out size-sensitive at the top end (the `slab_128` slowdown hints it might), a larger `subclass_bits` beats wider octaves. -- **Oversized reuse.** The shared oversized free list holds variable-size blocks, so reuse isn't a fixed-offset pop: pop-if-head-fits-else-extend is O(1) but wastes non-head fits; a bounded first-fit walk reclaims more at some cold-path cost. Which, given how rare oversized is in the real workload. -- **Header bound.** `NUM_CLASSES` is a compile-time constant but still sets header size (e.g. 64 classes → 512 B of heads); pick `linear_max`/`max_class` so the head array and the oversized cutover stay sane for the workload's large tail. diff --git a/README.md b/README.md index 4fec97c..410569b 100644 --- a/README.md +++ b/README.md @@ -551,7 +551,7 @@ bstack = { version = "0.4", features = ["set"] } ### `alloc` -Enables the region-management layer on top of `BStack`: `BStackAllocator`, `BStackBulkAllocator`, `BStackUninitAllocator`, `BStackOwnedSliceAllocator`, `BStackAllocError`, `BStackBulkAllocError`, `BStackRange`, `BStackOwnedSlice`, `BStackSlice`, `BStackSliceReader`, `LinearBStackAllocator`, `GhostTreeBstackAllocator`, and `DebugCheckingAllocator`. Combined with `set`, also enables `BStackSliceWriter`, `FirstFitBStackAllocator`, `SlabBStackAllocator`, `CheckedSlabBStackAllocator`, `BStackByteVec`, and `BStackByteVecIter`. +Enables the region-management layer on top of `BStack`: `BStackAllocator`, `BStackBulkAllocator`, `BStackUninitAllocator`, `BStackOwnedSliceAllocator`, `BStackAllocError`, `BStackBulkAllocError`, `BStackRange`, `BStackOwnedSlice`, `BStackSlice`, `BStackSliceReader`, `LinearBStackAllocator`, `GhostTreeBstackAllocator`, and `DebugCheckingAllocator`. Combined with `set`, also enables `BStackSliceWriter`, `FirstFitBStackAllocator`, `SlabBStackAllocator`, `CheckedSlabBStackAllocator`, `SegregatedBStackAllocator` (experimental), `BStackByteVec`, and `BStackByteVecIter`. ```toml [dependencies] @@ -1087,6 +1087,21 @@ Constructor takes `data_size` (usable bytes per block; physical = `data_size + 8 `open` runs `recover()` automatically. Without `atomic`: `Send` only. With `atomic`: `Send + Sync` (same lock-free strategy as `SlabBStackAllocator`). +### `SegregatedBStackAllocator` (**experimental**, `alloc + set`) + +Segregated (binned) free-list allocator: the checked slab generalised to 33 size +classes (16 linear 16‥256 B, 16 geometric 320‥4096 B, one oversized bucket) +sharing one arena. Class computed by register arithmetic; O(1) classed +alloc/dealloc; 8-byte overhead tag per block. Single `new(stack)` constructor +(runs recovery automatically). Without `atomic`: `Send` only. With `atomic`: +`Send + Sync`, no allocator-level lock. + +> **Experimental.** The on-disk format and API are not yet stable, some resize +> paths differ between the `atomic` and non-`atomic` builds (in-place non-tail +> carve is `atomic`-only; the non-`atomic` build moves instead), and the +> background coalescer and deep in-use-leak GC are not yet implemented. See +> [`algos/ALLOCATOR.md`](algos/ALLOCATOR.md) for the full design. + ### `DebugCheckingAllocator` (`alloc`) Transparent debug wrapper that can be placed around any allocator. Tracks diff --git a/algos/ALLOCATOR.md b/algos/ALLOCATOR.md index 65bbe53..36c3703 100644 --- a/algos/ALLOCATOR.md +++ b/algos/ALLOCATOR.md @@ -326,3 +326,115 @@ With the `atomic` feature it **is `Sync`**. `alloc` / `dealloc` / `realloc` take | `CheckedSlabBStackAllocator::new(stack, data_size)` | **empty** | Writes the 48-byte allocator header; fails with `InvalidInput` if the stack already has data or `data_size < 8`. | | `CheckedSlabBStackAllocator::open(stack)` | **non-empty** | Reads and validates the stored header, then runs `recover()` automatically. Fails with `InvalidData` on magic mismatch, invalid block size, or misaligned arena. | | `CheckedSlabBStackAllocator::recover()` | any | Reclaims leaked blocks and discards orphaned tails left by an unclean shutdown. Returns the count of blocks that could not be classified with certainty (`0` = fully accounted for). Called automatically by `open`; exposed for explicit inspection or re-runs. | + +--- + +## `SegregatedBStackAllocator` (**experimental**, `alloc + set` features) + +> **Experimental.** The on-disk format and API are not yet stable, a few resize +> paths behave differently between the `atomic` and non-`atomic` builds, and the +> background coalescer and deep in-use-leak reclamation are not yet implemented. + +[`CheckedSlabBStackAllocator`] generalised from one block size to **33 size +classes** sharing one arena. Each class is an independent intrusive free list; +the class is derived from the request with register arithmetic (no tables), so +classed alloc/dealloc are O(1). Every block carries the same 8-byte overhead tag +as the checked slab. + +### Size-class scheme + +Fixed at compile time (encoded by the magic version), quantum 16: + +* **16 linear classes** — 16, 32, …, 256 (step 16). +* **16 geometric classes** — octaves `[256, 4096)`, 4 subclasses each: 320, 384, + 448, 512; 640, 768, 896, 1024; 1280, 1536, 1792, 2048; 2560, 3072, 3584, 4096. +* **1 shared oversized bucket** — above 4096 B. + +Sizes are physical block sizes (payload + 8) and multiples of 16; **33 free-list +heads** total (`NUM_CLASSES`), the last oversized. Helpers: `phys_need(len) = +round_up(len + 8, 16)`; `class_blocksize(need)` snaps up to the enclosing class; +`largest_class_le(v)` snaps down (for the carve). + +### On-disk layout + +```text +offset 0 reserved (user) 24 B +offset 24 magic "ALSG\x00\x01\x00\x00" 8 B +offset 32 _reserved 8 B +offset 40 free_head[33] : u64 264 B # last entry = oversized list +offset 304 arena start (16-B aligned) +``` + +Each block is `[ overhead(8) | data(block − 8) ]`; the caller pointer is the data +start (`block_start + 8`, always `16n + 8`). The overhead is one tagged word: +**high bit set ⇒ in use**, low 63 bits = the caller's exact length; **high bit +clear ⇒ free**, low 63 bits = physical size `>> 4` (also the class tag). A free +block stores `next_free` inline at the data start; live blocks carry no overhead +beyond the word. + +### Allocation policy + +1. Compute `class_blocksize(phys_need(len))` and the class index. +2. **Classed** (`≤ 4096`): pop the class head, claiming the block by writing + overhead and any copied prefix as one buffer; on a miss, grow a zero-filled + block at the tail with one sparse `extend`. +3. **Oversized** (`> 4096`): pop the head if its stored size is `≥ need`. If + exact, claim it; else claim `block` bytes and carve the excess into ≤ 3 class + free blocks. On a miss, extend. + +### Deallocation policy + +Read the overhead at `ptr − 8`; reject a clear high bit (double-free) or a stored +length disagreeing with the freed slice (partial free). An oversized block at the +tail is `discard`ed back to the stack; every other block is spliced onto its +class head. + +### Realloc + +| Case | Strategy | +|-----------------|----------------------------------------------------------------------------------------------------------| +| Same class | rewrite the overhead length; zero the grown tail on grow | +| Grow at tail | extend in place (zero-filled), then rewrite the length | +| Shrink at tail | rewrite the length, then discard the excess tail in place | +| Non-tail shrink | `atomic`: rewrite length + greedy-carve the freed tail as one in-place transaction. non-`atomic`: *move* | +| Non-tail grow | *move* — allocate the new class (reading the prefix into its claim buffer), then free the old block | + +The **greedy carve** takes the largest class `≤` the remainder, repeated: a region +`> 4096` becomes one oversized block; a classed region splits into ≤ 3 +distinct-class pieces. Fixed stack buffers, no heap. + +### Crash consistency + +Every path either commits atomically or leaves an orphaned-but-recoverable block. + +* **With `atomic`**: pops use `BStack::process_gen`; pushes and the in-place + non-tail carve use `BStack::inplace_gen` (multiple writes, one journalled + commit); tail grow/shrink use `try_extend_zeros` / `try_discard`. +* **Without `atomic`**: plain read-then-write / `len`-check-then-`extend`/`discard` + (each one durable write). The in-place non-tail carve is not fault-safe, so + non-tail shrink uses the *move*, and the oversized-reuse carve writes all freed + pieces first and the claiming header last. + +`recover()` rebuilds every free list with one linear scan of the overhead words: +a live block is strided by its class size; a free block is relinked onto +`head[classify(size)]` (reclaiming leaked blocks); a fully-zeroed tail is +discarded. The rebuilt head table is published as one crash-atomic contiguous +write. The scan trusts only the overhead words and is idempotent. In-use orphans +(e.g. the old block of a crashed *move*) are left live; a deep GC to reclaim them +is future work. `recover` requires a quiescent allocator and is `unsafe`; `new` +runs it before the handle escapes. + +### Thread safety + +Always **`Send`**. With `atomic`, also **`Send + Sync`** with no allocator-level +lock — operations drive `process_gen`/`inplace_gen` under `BStack`'s write lock; +no retained `Mutex`. Without `atomic`, **not `Sync`** (pops/pushes read a head +then write without holding a lock across the pair). + +### Constructors + +| Constructor | Stack | Effect | +|-----------------------------------------------|---------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `SegregatedBStackAllocator::new(stack)` | **empty** | Writes the header (magic + 33 zeroed heads) with one sparse extend to the arena start. | +| `SegregatedBStackAllocator::new(stack)` | **non-empty** | Validates the magic prefix and arena alignment, then runs `recover()` automatically before returning. Fails with `InvalidData`/`UnexpectedEof` on mismatch or misalignment. | +| `unsafe SegregatedBStackAllocator::recover()` | any | Reclaims leaks and discards orphaned tails. Returns the count of blocks that could not be classified with certainty (`0` = fully accounted for). `unsafe`: the caller must guarantee the allocator is quiescent (no concurrent operations). Called automatically by `new`. | diff --git a/src/alloc/mod.rs b/src/alloc/mod.rs index 971d860..8dfb084 100644 --- a/src/alloc/mod.rs +++ b/src/alloc/mod.rs @@ -75,6 +75,12 @@ //! * [`CheckedSlabBStackAllocator`] — crash-recoverable slab variant (`alloc` + `set`). //! 8-byte per-block header tracks state; double-frees caught. //! +//! * [`SegregatedBStackAllocator`] — **experimental** segregated (binned) +//! free-list allocator (`alloc` + `set`). Generalises the checked slab to 33 +//! size classes sharing one arena; 8-byte per-block header, O(1) classed +//! alloc/dealloc, crash-recoverable by linear scan. `Send` in all +//! configurations; `Send + Sync` with `atomic`. +//! //! # Debug wrapper //! //! * [`DebugCheckingAllocator`](DebugCheckingAllocator) — wraps any allocator @@ -113,7 +119,8 @@ //! ``` //! //! [`BStackSliceWriter`], [`FirstFitBStackAllocator`], [`SlabBStackAllocator`], -//! [`CheckedSlabBStackAllocator`], and [`BStackByteVec`] additionally require `set`: +//! [`CheckedSlabBStackAllocator`], [`SegregatedBStackAllocator`] (experimental), +//! and [`BStackByteVec`] additionally require `set`: //! //! ```toml //! bstack = { version = "0.1", features = ["alloc", "set"] } @@ -682,6 +689,8 @@ pub mod ghost_tree; pub mod guarded; pub mod linear; #[cfg(feature = "set")] +pub mod segregated; +#[cfg(feature = "set")] pub mod slab; #[cfg(feature = "set")] pub mod vec; @@ -699,6 +708,8 @@ pub use guarded::{BStackAtomicGuardedSlice, BStackAtomicGuardedSliceSubview}; pub use guarded::{BStackGuardedSlice, BStackGuardedSliceSubview}; pub use linear::LinearBStackAllocator; #[cfg(feature = "set")] +pub use segregated::SegregatedBStackAllocator; +#[cfg(feature = "set")] pub use slab::SlabBStackAllocator; #[cfg(feature = "set")] pub use vec::{BStackByteVec, BStackByteVecIter}; diff --git a/src/alloc/segregated.rs b/src/alloc/segregated.rs new file mode 100644 index 0000000..d6aaa32 --- /dev/null +++ b/src/alloc/segregated.rs @@ -0,0 +1,1522 @@ +//! Segregated (binned) free-list allocator for [`BStack`]-backed storage. +//! +//! Provides [`SegregatedBStackAllocator`], which generalises the fixed-block +//! slab to `NUM_CLASSES` size classes sharing one arena. Each class is an +//! independent intrusive free list; `alloc` computes the class from the request +//! with register arithmetic and pops a single fixed-shape head, falling back to +//! a tail extension on a miss (the slab `pop, else extend` rule). Requests above +//! the largest class collapse onto one shared *oversized* list. +//! +//! The size→class policy (quantum 16, `linear_max`, `subclass_bits`, +//! `max_class`, and the resulting `NUM_CLASSES`) is a compile-time constant +//! encoded by the magic version, not per-file state — a format change bumps the +//! magic rather than reinterpreting a stored field. +//! +//! Implemented: `new`/`open`, `alloc` (incl. oversized non-exact reuse with +//! excess carve), `dealloc`, `realloc` (in-place same-class, tail grow/shrink, +//! non-tail shrink via greedy carve, non-tail grow via move), and `recover` +//! (linear-scan free-list rebuild + leak reclaim). Still pending: the background +//! coalescer. +//! +//! # Feature flags +//! +//! Requires `set`; `atomic` is optional. The module compiles and is fully +//! functional under both. With `atomic`, free-list splices ride +//! `process_gen`/`inplace_gen` (write lock held across the dependent read → +//! modify → write), and the tail grow/shrink/oversized-discard paths use the +//! size-guarded `try_extend_zeros`/`try_discard` — one locked critical section +//! that fuses the tail check with the mutation — which also makes the allocator +//! `Sync`. Without `atomic`, those become a plain read-then-write / `len`-check +//! then `extend`/`discard`: still crash-safe (each issues a single `bstack` +//! write, and multi-write splices are leak-preferring), but the allocator is +//! `Send` and **not** `Sync`, so concurrent use must be externally synchronised. + +use super::{BStackAllocError, BStackAllocator, BStackOwnedSlice}; +use crate::BStack; +#[cfg(feature = "atomic")] +use crate::BStackGenOp; +#[cfg(not(feature = "atomic"))] +use std::cell::Cell; +#[cfg(not(feature = "atomic"))] +use std::marker::PhantomData; +use std::{fmt, io}; + +/// Magic: `ALSG` + major 0 + minor 1; the version encodes the fixed class scheme. +const ALSG_MAGIC: [u8; 8] = *b"ALSG\x00\x01\x00\x00"; +/// Compatibility prefix checked on open (`ALSG` + major 0 + minor 1). +const ALSG_MAGIC_PREFIX: [u8; 6] = *b"ALSG\x00\x01"; + +/// A segregated free-list allocator implementing [`BStackAllocator`] on top of a +/// [`BStack`]. +/// +/// # On-disk layout +/// +/// ```text +/// offset 0 reserved (user) 24 B +/// offset 24 magic "ALSG\x00\x01\x00\x00" 8 B +/// offset 32 _reserved 8 B +/// offset 40 free_head[NUM_CLASSES] : u64 # last entry = oversized list +/// arena start (16-B aligned; header ends 16-aligned already) +/// ``` +/// +/// Every arena block is `[ overhead(8) | data(block − 8) ]`; the caller pointer +/// is the data start (`block_start + 8`). The overhead is a single tagged word: +/// high bit set ⇒ in use, low 63 bits = the caller's exact slice length; high +/// bit clear ⇒ free, low 63 bits = physical block size `>> 4` (which doubles as +/// the class tag). A free block stores its `next_free` offset inline at the data +/// start, so live allocations carry no space overhead beyond the 8-byte word. +/// +/// # Thread safety +/// +/// `SegregatedBStackAllocator` is always **`Send`** — ownership can be transferred +/// to another thread safely. +/// +/// ``` +/// fn assert_send() {} +/// assert_send::(); +/// ``` +/// +/// Under `atomic`, it is also `Sync`: `alloc`/`dealloc` drive [`BStack::process_gen`] / +/// [`BStack::inplace_gen`] sequences that hold `BStack`'s write lock across the +/// dependent read/modify/write, so no allocator-level lock is taken. +/// [`recover`](Self::recover) is the one exception — it is `unsafe` and shifts the +/// no-concurrent-access obligation to the caller (see its `# Safety`). +/// +/// Without `atomic` the type is `!Sync` (this fails to compile); with `atomic` +/// it is `Sync` via its [`BStack`] (this compiles): +/// +#[cfg_attr(not(feature = "atomic"), doc = "```compile_fail")] +#[cfg_attr(feature = "atomic", doc = "```")] +/// fn assert_sync() {} +/// assert_sync::(); +/// ``` +#[cfg(feature = "set")] +pub struct SegregatedBStackAllocator { + stack: BStack, + #[cfg(not(feature = "atomic"))] + _not_sync: PhantomData>, +} + +#[cfg(feature = "set")] +impl SegregatedBStackAllocator { + // Class scheme (compile-time, encoded by the magic version) + /// Minimum physical block size and the size quantum; every block is a + /// multiple of this. + const QUANTUM: u64 = 16; + /// Per-block overhead prefix. + const OVERHEAD: u64 = 8; + /// Top of the linear (step-16) region. + const LINEAR_MAX: u64 = 256; + /// `log2(LINEAR_MAX)` — the first geometric octave. + const LINEAR_OCTAVE: u32 = 8; + /// `log2(MAX_CLASS)` — the last geometric octave. + const MAX_OCTAVE: u32 = 12; + /// Largest classed physical block; above this is the oversized bucket. + const MAX_CLASS: u64 = 1 << Self::MAX_OCTAVE; // 4096 + /// Subclasses per geometric octave: `2^SUBCLASS_BITS`. + const SUBCLASS_BITS: u32 = 2; + const SUBCLASSES: u64 = 1 << Self::SUBCLASS_BITS; // 4 + /// Linear classes: 16, 32, …, LINEAR_MAX. + const LINEAR_CLASSES: u64 = Self::LINEAR_MAX / Self::QUANTUM; // 16 + /// Geometric classes across octaves `[LINEAR_OCTAVE, MAX_OCTAVE)`. + const GEO_CLASSES: u64 = (Self::MAX_OCTAVE - Self::LINEAR_OCTAVE) as u64 * Self::SUBCLASSES; // 16 + /// Total heads: linear + geometric + 1 shared oversized bucket. + const NUM_CLASSES: u64 = Self::LINEAR_CLASSES + Self::GEO_CLASSES + 1; // 33 + /// Index of the shared oversized free-list head. + const OVERSIZED_CLASS: u64 = Self::NUM_CLASSES - 1; // 32 + + // Header layout (compile-time, fixed offsets) + /// Bytes before the allocator header reserved for caller use. + const OFFSET_SIZE: u64 = 24; + // offset 32: 8 reserved bytes (see the on-disk layout) — no field yet. + /// Offset of `free_head[0]`. + const FREE_HEAD_BASE: u64 = 40; + /// Payload offset of the first arena block: header rounded up to the 16-B + /// quantum (which every block start must satisfy for the pointer check). The + /// header already ends 16-aligned, so this is exact: `40 + 33*8 = 304`. + const ARENA_START: u64 = (Self::FREE_HEAD_BASE + Self::NUM_CLASSES * 8 + 15) & !15; + + /// Maximum pieces a greedy carve emits: a region `> MAX_CLASS` is one + /// oversized block, and any region `≤ MAX_CLASS` decomposes into `≤ 3` + /// distinct-class blocks under this scheme (verified over every multiple of + /// 16 up to `MAX_CLASS`). Lets [`commit_carve`](Self::commit_carve) stay + /// heap-free with fixed stack buffers. + const MAX_CARVE_PIECES: usize = 3; + + /// Free-list sentinel: `0` (offset 0 is the header, never a block). + const SENTINEL: u64 = 0; + /// High bit of the overhead word: set when a block is live. + const IN_USE_BIT: u64 = 0x8000_0000_0000_0000; + + /// Round a caller `len` up to the physical need `round_up(len + 8, 16)`. + #[inline] + fn phys_need(len: u64) -> io::Result { + let n = len + .checked_add(Self::OVERHEAD + Self::QUANTUM - 1) + .ok_or_else(|| { + io::Error::new( + io::ErrorKind::InvalidInput, + "allocation length overflows u64", + ) + })?; + Ok(n & !(Self::QUANTUM - 1)) + } + + /// Snap a physical need (`= round_up(len + 8, 16)`) up to its class block + /// size. Linear needs pass through; geometric needs round up to the octave + /// subclass width; oversized needs pass through (raw multiple-of-16 path). + #[inline] + fn class_blocksize(need: u64) -> u64 { + if need <= Self::LINEAR_MAX { + return need; + } + if need <= Self::MAX_CLASS { + let k = 63 - (need - 1).leading_zeros(); // octave: need ∈ (2^k, 2^{k+1}] + let w = 1u64 << (k - Self::SUBCLASS_BITS); // subclass width = 2^{k-2} + return (need + w - 1) & !(w - 1); + } + need + } + + /// Map a physical block size (multiple of 16, ≥ 16) to its free-list head + /// index. Sizes above `MAX_CLASS` collapse onto the oversized bucket. + #[inline] + fn classify(size: u64) -> u64 { + if size <= Self::LINEAR_MAX { + return (size >> 4) - 1; + } + if size <= Self::MAX_CLASS { + let k = 63 - (size - 1).leading_zeros(); + let sub = (size - 1 - (1u64 << k)) >> (k - Self::SUBCLASS_BITS); + return Self::LINEAR_CLASSES + + (k as u64 - Self::LINEAR_OCTAVE as u64) * Self::SUBCLASSES + + sub; + } + Self::OVERSIZED_CLASS + } + + /// Largest class block size `≤ v` (with `16 ≤ v ≤ MAX_CLASS`, `v` a multiple + /// of 16). Snaps `v` **down** to a class boundary — the dual of + /// [`class_blocksize`](Self::class_blocksize), used by the greedy carve. + #[inline] + fn largest_class_le(v: u64) -> u64 { + if v <= Self::LINEAR_MAX { + return v; // every multiple of 16 ≤ LINEAR_MAX is itself a class + } + let k = 63 - v.leading_zeros(); // 2^k ≤ v < 2^{k+1} + let w = 1u64 << (k - Self::SUBCLASS_BITS); // subclass width + v & !(w - 1) // round down to a multiple of w in the octave (a class) + } + + /// Payload offset of the free-list head for `class`. + #[inline] + fn head_off(class: u64) -> u64 { + Self::FREE_HEAD_BASE + class * core::mem::size_of::() as u64 + } + + /// Validate a caller data pointer and return its block base offset. + /// + /// A valid data pointer is `block_base + OVERHEAD`, where `block_base` is a + /// 16-aligned arena offset `>= ARENA_START`. Equivalently the pointer is of + /// the form `16·n + 8`: `ptr ≡ OVERHEAD (mod QUANTUM)` and + /// `ptr >= ARENA_START + OVERHEAD`. Reject anything else — a header-range, + /// underflowing, or mid-block pointer would let an operation reinterpret + /// interior bytes as an overhead word. + #[inline] + fn block_start_of(ptr: u64) -> io::Result { + if ptr < Self::ARENA_START + Self::OVERHEAD || ptr % Self::QUANTUM != Self::OVERHEAD { + return Err(io::Error::new( + io::ErrorKind::InvalidInput, + "slice start is not a valid block pointer", + )); + } + Ok(ptr - Self::OVERHEAD) + } + + /// Create a new allocator over `stack` or reopen an existing one. + /// + /// If `stack` is empty this writes the header and returns a fresh + /// allocator. Otherwise it validates the header and arena alignment and + /// runs recovery before returning the allocator. + /// + /// # Errors + /// + /// * [`io::ErrorKind::UnexpectedEof`] if the stack is too short to contain + /// the header or the arena is not a multiple of the block quantum. + /// * [`io::ErrorKind::InvalidData`] if the magic is wrong (not a Segregated + /// allocator of the expected version). + /// * Any [`io::Error`] from the underlying [`BStack`] operations. + pub fn new(stack: BStack) -> io::Result { + if stack.is_empty()? { + // Initialize a new stack: write the header and return a fresh allocator. + const OFFSET_OFFSET: usize = SegregatedBStackAllocator::OFFSET_SIZE as usize; + let mut hdr = [0u8; OFFSET_OFFSET + 8]; + hdr[OFFSET_OFFSET..].copy_from_slice(&ALSG_MAGIC); + // the reserved words and every free_head remain 0. + let _ = stack.extend_sparse(hdr, Self::ARENA_START)?; + return Ok(Self { + stack, + #[cfg(not(feature = "atomic"))] + _not_sync: PhantomData, + }); + } + + // Reopen an existing file + let stack_len = stack.len()?; + if stack_len < Self::FREE_HEAD_BASE { + return Err(io::Error::new( + io::ErrorKind::UnexpectedEof, + "stack too short to contain allocator header", + )); + } + + let mut magic = [0u8; 8]; + stack.get_into(Self::OFFSET_SIZE, &mut magic)?; + if magic[..ALSG_MAGIC_PREFIX.len()] != ALSG_MAGIC_PREFIX { + return Err(io::Error::new( + io::ErrorKind::InvalidData, + "invalid magic: not a SegregatedBStackAllocator file of the expected version", + )); + } + if stack_len < Self::ARENA_START { + // Make a zeroed free_head array + let needed = Self::ARENA_START - stack_len; + let _ = stack.extend(needed)?; + } else if (stack_len - Self::ARENA_START) % Self::QUANTUM != 0 { + // Every block is a multiple of QUANTUM, so the arena byte count is too. + return Err(io::Error::new( + io::ErrorKind::UnexpectedEof, + "arena is not a multiple of the block quantum", + )); + } + let allocator = Self { + stack, + #[cfg(not(feature = "atomic"))] + _not_sync: PhantomData, + }; + // SAFETY: `allocator` was just constructed and has not yet escaped this + // function, so no other thread can hold it — it is trivially quiescent. + unsafe { allocator.recover()? }; + Ok(allocator) + } + + /// Reclaim blocks leaked by an unclean shutdown and return the count that + /// could not be classified with certainty (`0` = fully accounted for). + /// + /// Rebuilds **every** free list from scratch by a single linear scan of the + /// arena's overhead words: a live block (high bit set) is strided over by its + /// physical size derived from the stored `len`; a free block (high bit clear, + /// non-zero) is relinked onto `head[classify(size)]` by its stored physical + /// `size`, which reclaims any block leaked by a crashed `alloc` pop/claim + /// (still free-tagged but reachable from no head). A fully-zeroed region — a + /// crashed tail `extend` whose overhead write never landed — is discarded as + /// an orphaned tail. + /// + /// Because the scan trusts only the overhead words (never the stored + /// `next_free` links) and [`new`](Self::new) runs it before any live + /// operation, it is **idempotent and crash-safe by re-running**: a crash + /// mid-rebuild leaves half-written links that the next `open`'s scan simply + /// rebuilds again. Blocks orphaned *in-use* (e.g. the old block of a crashed + /// realloc move) are not reclaimable by a bare scan and are left live; a deep + /// reachability GC that could reclaim them (needing a per-op journal of the + /// affected block, not just a dirty bit) is deferred to a later pass. + /// + /// The rebuilt head table is published as a single crash-atomic contiguous + /// [`BStack::set`], so the table never becomes half-updated. Stops at the first + /// unclassifiable overhead, counting the remaining arena as unsure. + /// + /// # Safety + /// + /// The caller must guarantee the allocator is **quiescent** for the duration + /// of the call: no other thread may run any `alloc`/`dealloc`/`realloc` (or + /// another `recover`) on this allocator concurrently. `recover` rebuilds every + /// free list from a single linear snapshot and then replaces the whole head + /// table wholesale; a concurrent operation between the snapshot and the flip + /// would be clobbered — resurrecting a just-allocated block onto a free list, + /// or dropping a just-freed one — which is memory-unsafe for the allocator's + /// consumers. Construction ([`new`](Self::new)) satisfies this trivially by + /// running it before the allocator handle escapes. + pub unsafe fn recover(&self) -> io::Result { + let stack_len = self.stack.len()?; + if stack_len <= Self::ARENA_START { + return Ok(0); + } + let mut heads = [0u64; Self::NUM_CLASSES as usize]; + let mut unsure = 0u64; + let mut p = Self::ARENA_START; + while p < stack_len { + let word = u64::from_le_bytes(read_bstack!(self.stack, p => u64)); + if word & Self::IN_USE_BIT != 0 { + // Live: stride by the physical size implied by the stored len. + let len = word & !Self::IN_USE_BIT; + let size = match Self::phys_need(len) { + Ok(need) => Self::class_blocksize(need), + Err(_) => { + unsure += (stack_len - p) / Self::QUANTUM; + break; + } + }; + if size == 0 || p + size > stack_len { + unsure += (stack_len - p) / Self::QUANTUM; + break; + } + p += size; + } else if word == 0 { + // Zeroed tail from a crashed `extend`: discard it (free blocks + // always store size >> 4 ≥ 1, so a zero word is never valid). + self.stack.discard(stack_len - p)?; + break; + } else { + // Free: relink by the stored physical size (reclaims leaks too). + let size = word << 4; + if size < Self::QUANTUM || size % Self::QUANTUM != 0 || p + size > stack_len { + unsure += (stack_len - p) / Self::QUANTUM; + break; + } + let c = Self::classify(size) as usize; + // Prepend: next_free ← current head of this class, then head ← p. + self.stack.set(p + Self::OVERHEAD, heads[c].to_le_bytes())?; + heads[c] = p; + p += size; + } + } + // Publish the rebuilt head table. + let mut head_bytes = [0u8; Self::NUM_CLASSES as usize * 8]; + for (c, &h) in heads.iter().enumerate() { + write_buf!(h => head_bytes, c * 8); + } + self.stack.set(Self::FREE_HEAD_BASE, head_bytes)?; + Ok(unsure) + } + + /// Pop the head of `class`, returning its block-start offset or `None`. + /// + /// The method is **not** thread-safe and must be externally synchronised if the + /// allocator is used concurrently. However, since it only issues one `bstack` + /// write, it is trivially crash-safe. + #[cfg(not(feature = "atomic"))] + fn pop_class(&self, class: u64) -> io::Result> { + let head_off = Self::head_off(class); + let head = u64::from_le_bytes(read_bstack!(self.stack, head_off => u64)); + if head == Self::SENTINEL { + return Ok(None); + } + let next = u64::from_le_bytes(read_bstack!(self.stack, head + Self::OVERHEAD => u64)); + self.stack.set(head_off, next.to_le_bytes())?; + Ok(Some(head)) + } + + /// Pop the head of `class`, returning its block-start offset or `None`. + /// + /// Drives one [`BStack::process_gen`] holding the write lock across + /// read-head → read-next → advance-head, closing the ABA window. The claim + /// (overhead flip + data scrub) is a separate write on the now-detached + /// block; a crash between leaks ≤ 1 block, reclaimed by `recover`. + #[cfg(feature = "atomic")] + fn pop_class(&self, class: u64) -> io::Result> { + let head_off = Self::head_off(class); + let mut head_buf = [0u8; 8]; + let mut next_buf = [0u8; 8]; + let mut step = 0u32; + let mut popped: Option = None; + self.stack.process_gen(|| { + let op = match step { + 0 => Some(BStackGenOp::Read { + offset: head_off, + // SAFETY: `head_buf` outlives this `process_gen` call. + buf: unsafe { core::mem::transmute::<&mut [u8], &mut [u8]>(&mut head_buf[..]) }, + }), + 1 => { + let head = u64::from_le_bytes(head_buf); + if head == Self::SENTINEL { + None + } else { + popped = Some(head); + Some(BStackGenOp::Read { + offset: head + Self::OVERHEAD, + // SAFETY: `next_buf` outlives this `process_gen` call. + buf: unsafe { + core::mem::transmute::<&mut [u8], &mut [u8]>(&mut next_buf[..]) + }, + }) + } + } + 2 => Some(BStackGenOp::Write { + offset: head_off, + // SAFETY: `next_buf` outlives this `process_gen` call. + data: unsafe { core::mem::transmute::<&[u8], &[u8]>(&next_buf[..]) }, + }), + _ => None, + }; + step += 1; + op + })?; + Ok(popped) + } + + /// Pop the oversized head if its stored physical size is **≥ `need`**, + /// returning `(block_start, actual_size)`; a head smaller than `need` is left + /// in place (the O(1) pop-if-head-fits rule — no search). The caller uses the + /// first `need` bytes and carves any excess (`actual_size − need`). + /// + /// The method is **not** thread-safe and must be externally synchronised if the + /// allocator is used concurrently. Since this only issues one `bstack` write, it + /// is trivially crash-safe. + #[cfg(not(feature = "atomic"))] + fn pop_oversized(&self, need: u64) -> io::Result> { + let head_off = Self::head_off(Self::OVERSIZED_CLASS); + let head = u64::from_le_bytes(read_bstack!(self.stack, head_off => u64)); + if head == Self::SENTINEL { + return Ok(None); + } + // overhead ‖ next_free are contiguous: fetch both in one 16-byte read. + let buf = read_bstack!(self.stack, head => 16); + let word = read_buf_le!(buf, 0 => u64); + // Free head must have the high bit clear; its size is word << 4. + let size = word << 4; + if word & Self::IN_USE_BIT != 0 || size < need { + return Ok(None); + } + let next = read_buf_le!(buf, 8 => u64); + self.stack.set(head_off, next.to_le_bytes())?; + Ok(Some((head, size))) + } + + /// Pop the oversized head if its stored physical size is **≥ `need`**, + /// returning `(block_start, actual_size)`; a head smaller than `need` is left + /// in place (the O(1) pop-if-head-fits rule — no search). The caller uses the + /// first `need` bytes and carves any excess (`actual_size − need`). + /// + /// A single [`BStack::process_gen`] holds the write lock across read-head → + /// read-(overhead‖next) → advance-head, removing any ABA window. The head + /// block's overhead word and inline `next_free` are contiguous, so they are + /// fetched in one 16-byte read. + #[cfg(feature = "atomic")] + fn pop_oversized(&self, need: u64) -> io::Result> { + let head_off = Self::head_off(Self::OVERSIZED_CLASS); + let mut head_buf = [0u8; 8]; + // overhead ‖ next_free of the head block, read as one 16-byte op. + let mut oh_next_buf = [0u8; 16]; + let mut step = 0usize; + let mut head = 0u64; + let mut size = 0u64; + let mut popped: Option<(u64, u64)> = None; + self.stack.process_gen(|| { + let op = match step { + 0 => Some(BStackGenOp::Read { + offset: head_off, + // SAFETY: `head_buf` outlives this call. + buf: unsafe { core::mem::transmute::<&mut [u8], &mut [u8]>(&mut head_buf[..]) }, + }), + 1 => { + head = u64::from_le_bytes(head_buf); + if head == Self::SENTINEL { + None + } else { + Some(BStackGenOp::Read { + offset: head, + // SAFETY: `oh_next_buf` outlives this call. + buf: unsafe { + core::mem::transmute::<&mut [u8], &mut [u8]>(&mut oh_next_buf[..]) + }, + }) + } + } + 2 => { + let word = read_buf_le!(oh_next_buf, 0 => u64); + // Free head must have the high bit clear; its size is word << 4. + size = word << 4; + if word & Self::IN_USE_BIT != 0 || size < need { + None + } else { + popped = Some((head, size)); + // head[oversized] ← next_free (the second half of the read). + Some(BStackGenOp::Write { + offset: head_off, + // SAFETY: `oh_next_buf` outlives this call; its [8..16] + // half is untouched after step 1's read resolved. + data: unsafe { + core::mem::transmute::<&[u8], &[u8]>(&oh_next_buf[8..]) + }, + }) + } + } + _ => None, + }; + step += 1; + op + })?; + Ok(popped) + } + + /// Build the `block`-byte claim buffer (`in_use | len` overhead, `copy_from` + /// prefix read straight in, rest zero) without writing it. + /// + /// Shared by both `atomic` and non-`atomic` paths. + fn claim_buf( + &self, + block: u64, + len: u64, + copy_from: Option<(u64, u64)>, + ) -> io::Result> { + let mut buf = vec![0u8; block as usize]; + write_buf!(Self::IN_USE_BIT | len => buf, 0); + if let Some((src, n)) = copy_from { + self.stack.get_into(src, &mut buf[8..8 + n as usize])?; + } + Ok(buf) + } + + /// Core allocation: place a `len`-byte block whose payload begins with `n` + /// bytes copied from payload offset `src` (`copy_from = Some((src, n))`, rest + /// zeroed) and return its data pointer (`block_start + OVERHEAD`). A free-list + /// hit reads the source straight into the single claim buffer; a miss extends + /// a zero-filled block and writes overhead (plus the copied prefix), relying + /// on `extend`'s zero-fill for the tail. `n` must not exceed the class payload + /// capacity for `len` (callers pass a prefix `≤ len`). + fn alloc_raw(&self, len: u64, copy_from: Option<(u64, u64)>) -> io::Result { + let need = Self::phys_need(len)?; + let block = Self::class_blocksize(need); + if block <= Self::MAX_CLASS { + let class = Self::classify(block); + if let Some(bs) = self.pop_class(class)? { + // In both atomic and non-atomic paths, a failure between the pop + // and the claim leaves the block free-tagged and reachable from + // the head, so a crash is recoverable by `recover`. + let buf = self.claim_buf(block, len, copy_from)?; + self.stack.set(bs, buf)?; + return Ok(bs + Self::OVERHEAD); + } + } else if let Some((bs, actual)) = self.pop_oversized(block)? { + let buf = self.claim_buf(block, len, copy_from)?; + if actual == block { + // See the same reasoning for crash recovery above. + self.stack.set(bs, buf)?; + } else { + // Non-exact reuse: claim `block` bytes and carve the excess, all + // as one crash-atomic transaction (claim buffer as the prefix) + self.commit_carve(bs, &buf, bs + block, actual - block)?; + } + return Ok(bs + Self::OVERHEAD); + } + // Miss: eagerly grow the whole zero-filled block in one sparse write. + // The write prefix is copied in before the tail is left zero by the + // sparse grow, so we do not need a separate `set` for the remainder. + let buf = self.claim_buf(block, len, copy_from)?; + let bs = self.stack.extend_sparse(&buf, block)?; + Ok(bs + Self::OVERHEAD) + } + + /// Push `block_start` (physical size `size`, head index `class`) onto its + /// free list, bundling the overhead flip. + fn push(&self, block_start: u64, size: u64, class: u64) -> io::Result<()> { + let head_off = Self::head_off(class); + let start_bytes = block_start.to_le_bytes(); + // overhead || next_free: contiguous fields (OVERHEAD == 8 == size_of head), + // so both are staged in one 16-byte buffer and written in a single op. + let mut overhead_buf = [0u8; 16]; + write_buf!(size >> 4 => overhead_buf, 0); // free tag: high bit clear + #[cfg(not(feature = "atomic"))] + { + // Non-atomic path: read head, write overhead+next_free, write head. + let head = u64::from_le_bytes(read_bstack!(self.stack, head_off => u64)); + write_buf!(head => overhead_buf, 8); + self.stack.set(block_start, overhead_buf)?; + // A crash between these two writes leaves the block free-tagged so it is + // recoverable by `recover`. + self.stack.set(head_off, start_bytes) + } + #[cfg(feature = "atomic")] + let mut step = 0u32; + #[cfg(feature = "atomic")] + self.stack.inplace_gen(|_res| { + let op = match step { + // Read the current head into next_free's half (no writes staged + // yet ⇒ committed value). + 0 => Some(BStackGenOp::Read { + offset: head_off, + // SAFETY: `overhead_buf` outlives this call. + buf: unsafe { + core::mem::transmute::<&mut [u8], &mut [u8]>(&mut overhead_buf[8..]) + }, + }), + // overhead ← free | size; next_free ← old head + 1 => Some(BStackGenOp::Write { + offset: block_start, + // SAFETY: `overhead_buf` outlives this call and is not + // mutated after step 0's read resolved. + data: unsafe { core::mem::transmute::<&[u8], &[u8]>(&overhead_buf[..]) }, + }), + // head[class] ← block_start + 2 => Some(BStackGenOp::Write { + offset: head_off, + // SAFETY: `start_bytes` outlives this call. + data: unsafe { core::mem::transmute::<&[u8], &[u8]>(&start_bytes[..]) }, + }), + _ => None, + }; + step += 1; + op + }) + } + + /// Commit a `prefix` write and free a contiguous `region` as **one** + /// crash-atomic [`BStack::inplace_gen`] transaction. + /// + /// `region` is greedily decomposed into class free blocks — the largest class + /// `≤` the remainder, repeated (a region `> MAX_CLASS` becomes one oversized + /// block). Every piece is a distinct class (greedy remainders strictly + /// shrink), so each free-list head is read once and rewritten once: the + /// transaction reads every involved head, writes `prefix`, writes each + /// piece's overhead + `next_free` (= that class's old head), and repoints each + /// head — all together. Bundling the `prefix` (the shrunk block's overhead, + /// or the used oversized block) with the carve means a crash leaves the block + /// either wholly un-shrunk or fully shrunk-and-freed, never a mid-arena gap. + /// + /// `region_size` must be a multiple of `QUANTUM` (`0` is a valid no-op that + /// just writes `prefix`); pieces number ≤ 3 for any classed region. + fn commit_carve( + &self, + prefix_off: u64, + prefix: &[u8], + region_start: u64, + region_size: u64, + ) -> io::Result<()> { + // Greedy decomposition into ≤ MAX_CARVE_PIECES pieces, held in fixed + // stack buffers (no heap on the carve path). + const N: usize = SegregatedBStackAllocator::MAX_CARVE_PIECES; + let mut block_offs = [0u64; N]; // piece block starts + let mut head_offs = [0u64; N]; // free-list head slot per piece + let mut blockoff_bytes = [[0u8; 8]; N]; // block start LE, for head writes + // Per-piece 16-byte buffer: [0..8]=overhead (free|size), [8..16]=next_free (old head) + let mut overhead_next = [[0u8; 16]; N]; + let mut k = 0usize; + let mut off = region_start; + let mut rem = region_size; + while rem > 0 { + let ps = if rem > Self::MAX_CLASS { + rem // one oversized block absorbs the whole remainder + } else { + Self::largest_class_le(rem) + }; + debug_assert!(k < N, "greedy carve exceeded MAX_CARVE_PIECES"); + block_offs[k] = off; + head_offs[k] = Self::head_off(Self::classify(ps)); + write_buf!(ps >> 4 => overhead_next[k], 0); + blockoff_bytes[k] = off.to_le_bytes(); + off += ps; + rem -= ps; + k += 1; + } + + // Non-atomic path: lay down every freed piece *before* the prefix. Until + // the prefix is written the carved region is not yet exposed as separate + // blocks — it still sits inside the block whose header `prefix_off` will + // change (a free block for the sole non-atomic caller, oversized non-exact + // reuse) — so these writes are invisible to `recover`, and the single + // prefix write is the commit point that exposes the already-valid pieces + // atomically. A fault before it leaves the whole region reclaimable as it + // was. This ordering is only fault-safe because that region is free excess, + // never live caller data (non-tail shrink, which owns its tail, takes the + // move path without `atomic`). + #[cfg(not(feature = "atomic"))] + { + for i in 0..k { + // Copy the prefilled overhead into a local buffer, then read the + // old head directly into the latter half before writing both. + let mut shared = overhead_next[i]; + // next_free ← current head of this class (read straight in). + self.stack.get_into(head_offs[i], &mut shared[8..])?; + // overhead || next_free, then head ← this block. + self.stack.set(block_offs[i], shared)?; + self.stack.set(head_offs[i], blockoff_bytes[i])?; + } + // Commit: expose the pieces (and, for a claim, mark the block in use). + self.stack.set(prefix_off, prefix)?; + Ok(()) + } + + // Atomic path with `inplace_gen` transaction: read each head, write the prefix, + // then write each piece's overhead, next_free, and head. A crash leaves the + // block either wholly un-shrunk or fully shrunk-and-freed, never a mid-arena gap. + // Steps: [0, k) read each head; k writes the prefix; then 3 writes per + // piece (overhead, next_free, head); then None commits the batch. + #[cfg(feature = "atomic")] + { + let mut step = 0usize; + // `overhead_next` already holds the per-piece overhead in its first + // 8 bytes; we will read each head directly into its second half. + self.stack.inplace_gen(|_res| { + let op = if step < k { + // Read the committed head of piece `step`'s class (no head writes + // staged yet ⇒ this is the current head, captured as next_free). + Some(BStackGenOp::Read { + offset: head_offs[step], + // SAFETY: `overhead_next` outlives this call; we read the + // old head straight into its upper 8 bytes so a later write + // can emit both overhead and next_free together. + buf: unsafe { + core::mem::transmute::<&mut [u8], &mut [u8]>( + &mut overhead_next[step][8..], + ) + }, + }) + } else if step == k { + Some(BStackGenOp::Write { + offset: prefix_off, + // SAFETY: `prefix` outlives this call. + data: unsafe { core::mem::transmute::<&[u8], &[u8]>(prefix) }, + }) + } else if step < k + 1 + 2 * k { + let j = step - (k + 1); + let i = j / 2; + Some(match j % 2 { + // overhead || next_free (combined 16-byte write) + 0 => BStackGenOp::Write { + offset: block_offs[i], + // SAFETY: `overhead_next` outlives this call. + data: unsafe { + core::mem::transmute::<&[u8], &[u8]>(&overhead_next[i][..]) + }, + }, + // head[class] ← this block + _ => BStackGenOp::Write { + offset: head_offs[i], + // SAFETY: `blockoff_bytes` outlives this call. + data: unsafe { + core::mem::transmute::<&[u8], &[u8]>(&blockoff_bytes[i][..]) + }, + }, + }) + } else { + None + }; + step += 1; + op + }) + } + } +} + +#[cfg(feature = "set")] +impl fmt::Debug for SegregatedBStackAllocator { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + f.debug_struct("SegregatedBStackAllocator") + .field("num_classes", &Self::NUM_CLASSES) + .finish_non_exhaustive() + } +} + +#[cfg(feature = "set")] +impl BStackAllocator for SegregatedBStackAllocator { + type Error = io::Error; + type Allocated<'a> = BStackOwnedSlice<'a, Self>; + + #[inline] + fn stack(&self) -> &BStack { + &self.stack + } + + #[inline] + fn into_stack(self) -> BStack { + self.stack + } + + /// Allocate `len` bytes. Computes the class, pops its head, else extends a + /// fresh class block; oversized requests reuse an exact-size head or extend. + #[inline] + fn alloc(&self, len: u64) -> io::Result> { + if len == 0 { + return Ok(BStackOwnedSlice::empty(self)); + } + let ptr = self.alloc_raw(len, None)?; + // SAFETY: `ptr` is the data start of a freshly allocated `len`-byte block. + Ok(unsafe { BStackOwnedSlice::from_raw_parts(self, ptr, len) }) + } + + /// Resize the region described by `slice` to `new_len` bytes. + /// + /// | Case | Strategy | + /// |------|----------| + /// | Same class (`new_len` maps to this block) | rewrite overhead `len`; zero the grown tail on grow | + /// | Grow at tail | extend the tail in place (zero-filled), then rewrite `len` | + /// | Shrink at tail | rewrite `len`, then discard the excess tail in place | + /// | Non-tail shrink | rewrite `len` + greedy-carve the freed tail into free blocks — one crash-atomic transaction (`atomic`); a move without `atomic` | + /// | Non-tail grow | alloc new class, copy, dealloc old | + /// + /// Every path only ever *leaks* on a mid-op failure (never corrupts): the + /// tail grow/shrink commit the physical size change leak-preferring so a + /// crash leaves an orphaned tail that [`recover`](Self::recover) reclaims, and + /// the non-tail shrink commits its length change and carve as a single + /// transaction so a crash leaves the block wholly un-shrunk or fully carved. + fn realloc<'a>( + &'a self, + slice: BStackOwnedSlice<'a, Self>, + new_len: u64, + ) -> Result, BStackAllocError<'a, Self>> { + if slice.is_empty() { + // Nothing backs an empty handle: realloc is just a fresh alloc. + return self.alloc(new_len).map_err(|source| { + BStackAllocError::with_handle(source, BStackOwnedSlice::empty(self)) + }); + } + if new_len == 0 { + // dealloc consumes `slice`; its BStackAllocError propagates unchanged. + self.dealloc(slice)?; + return Ok(BStackOwnedSlice::empty(self)); + } + let start = slice.start(); + let old_len = slice.len(); + // Validate the pointer up front so every path below (including the + // no-op resize) trusts a real block start. + let block_start = match Self::block_start_of(start) { + Ok(bs) => bs, + Err(source) => { + // SAFETY: the region is untouched and still owned by the caller. + let handle = unsafe { BStackOwnedSlice::from_raw_parts(self, start, old_len) }; + return Err(BStackAllocError::with_handle(source, handle)); + } + }; + if new_len == old_len { + // SAFETY: unchanged region. + return Ok(unsafe { BStackOwnedSlice::from_raw_parts(self, start, old_len) }); + } + + // The allocation to hand back on failure. Starts as the original block; + // becomes the new region once a move has committed and copied it (both + // are always distinct live regions safe to return). + let mut recovered = (start, old_len); + let result = (|| -> io::Result> { + let word = u64::from_le_bytes(read_bstack!(self.stack, block_start => u64)); + if word & Self::IN_USE_BIT == 0 { + return Err(io::Error::new( + io::ErrorKind::InvalidInput, + "cannot realloc a freed block", + )); + } + if word & !Self::IN_USE_BIT != old_len { + return Err(io::Error::new( + io::ErrorKind::InvalidInput, + "cannot realloc a partial or mismatched slice", + )); + } + let old_size = Self::class_blocksize(Self::phys_need(old_len)?); + let new_size = Self::class_blocksize(Self::phys_need(new_len)?); + + if new_size == old_size { + // Same class: the block already fits. Zero the newly-exposed + // bytes on grow (a prior shrink may have left stale data there), + // ordered before the length commit so a crash never exposes them. + if new_len > old_len { + self.stack.zero(start + old_len, new_len - old_len)?; + } + self.stack + .set(block_start, (Self::IN_USE_BIT | new_len).to_le_bytes())?; + // SAFETY: same physical block, new visible length. + return Ok(unsafe { BStackOwnedSlice::from_raw_parts(self, start, new_len) }); + } + + // Grow at the tail: extend the physical block in place, only when the + // block ends at the payload tail. Extend first (leak-preferring: a + // failure after it only leaks the extension, not corrupts), then zero + // the old slack, then commit the new length. Under `atomic`, + // `try_extend_zeros` fuses the tail check and the grow into one locked + // critical section; otherwise we check `len` then `extend` (which + // zero-fills the new region via `set_len`). + let old_end = block_start + old_size; // block exists ⇒ ≤ stack_len + let grew = new_size > old_size && { + #[cfg(feature = "atomic")] + { + self.stack.try_extend_zeros(old_end, new_size - old_size)? + } + #[cfg(not(feature = "atomic"))] + { + old_end == self.stack.len()? && { + self.stack.extend(new_size - old_size)?; + true + } + } + }; + if grew { + // Old block's slack [start+old_len, old_end) may hold stale bytes + // from a prior shrink; the extension past old_end is already zero. + let slack = (old_size - Self::OVERHEAD) - old_len; + if slack > 0 { + self.stack.zero(start + old_len, slack)?; + } + self.stack + .set(block_start, (Self::IN_USE_BIT | new_len).to_le_bytes())?; + // SAFETY: block extended in place to the new class at the tail. + return Ok(unsafe { BStackOwnedSlice::from_raw_parts(self, start, new_len) }); + } + + // Shrink at the tail: drop the excess physically in place. Commit the + // new length first (leak-preferring: a crash before the discard leaves + // an orphaned zero tail that `recover` reclaims, never corruption). + if new_size < old_size && old_end == self.stack.len()? { + self.stack + .set(block_start, (Self::IN_USE_BIT | new_len).to_le_bytes())?; + // Under `atomic`, `try_discard` re-checks the tail atomically; a + // lost race (concurrent tail extension) reverts the length and + // falls through to the carve/move paths. Without `atomic` the + // `len` guard above already established the tail, so discard plainly. + #[cfg(feature = "atomic")] + { + if self.stack.try_discard(old_end, old_size - new_size)? { + // SAFETY: block shrunk in place to the new class at the tail. + return Ok(unsafe { + BStackOwnedSlice::from_raw_parts(self, start, new_len) + }); + } + self.stack + .set(block_start, (Self::IN_USE_BIT | old_len).to_le_bytes())?; + } + #[cfg(not(feature = "atomic"))] + { + self.stack.discard(old_size - new_size)?; + // SAFETY: block shrunk in place to the new class at the tail. + return Ok(unsafe { BStackOwnedSlice::from_raw_parts(self, start, new_len) }); + } + } + + // Non-tail shrink: under `atomic`, keep the block at the new class and + // free the excess tail *in place* as one crash-atomic carve — no move, + // no copy, no mid-arena gap for `recover`, and `recovered` stays the + // untouched original (a fault leaves the block un-shrunk). + // + // Without `atomic` an in-place carve cannot be made fault-safe: shrinking + // the header and validating the freed tail are two non-adjacent writes, + // and with no journal to commit them together any fault between them + // either corrupts the caller's tail data (header still reads `old_len`) + // or leaves `recover` a garbage-headed tail to desync on. So the + // non-atomic build falls through to the move below, fault-safe step by + // step (each op atomic, a mid-move failure only ever leaks). + #[cfg(feature = "atomic")] + if new_size < old_size { + let prefix = (Self::IN_USE_BIT | new_len).to_le_bytes(); + self.commit_carve( + block_start, + &prefix, + block_start + new_size, + old_size - new_size, + )?; + // SAFETY: block shrunk in place; the freed tail is now free blocks. + return Ok(unsafe { BStackOwnedSlice::from_raw_parts(self, start, new_len) }); + } + + // Move: allocate the new class, having it read the surviving prefix + // straight from the old block into its claim buffer (no separate copy + // buffer or write), then free the old block. Handles the non-tail grow + // (both builds) and, without `atomic`, the non-tail shrink that fell + // through above. Each step is individually atomic; a mid-move failure + // leaks (never corrupts). + let copy_len = old_len.min(new_len); + let new_ptr = self.alloc_raw(new_len, Some((start, copy_len)))?; + // New region committed and populated; it is now the survivor. + recovered = (new_ptr, new_len); + // SAFETY: (start, old_len) still names the caller's live old block. + let old = unsafe { BStackOwnedSlice::from_raw_parts(self, start, old_len) }; + self.dealloc(old).map_err(|e| e.source)?; + // SAFETY: `new_ptr` is the data start of the freshly populated block. + Ok(unsafe { BStackOwnedSlice::from_raw_parts(self, new_ptr, new_len) }) + })(); + result.map_err(|source| BStackAllocError { + source, + // SAFETY: `recovered` names a live region owned by the caller. + handle: Some(unsafe { + BStackOwnedSlice::from_raw_parts(self, recovered.0, recovered.1) + }), + }) + } + + /// Release the region described by `slice`. + /// + /// Reads the overhead at `slice.start() − 8`: a clear high bit is a + /// double-free; a stored length that disagrees with `slice.len()` is a + /// partial/erroneous free. Otherwise an oversized block at the tail is + /// discarded in one call, and every other block is spliced onto its class + /// head via one crash-atomic [`BStack::inplace_gen`] transaction. + fn dealloc<'a>( + &'a self, + slice: BStackOwnedSlice<'a, Self>, + ) -> Result<(), BStackAllocError<'a, Self>> { + let start = slice.start(); + let len = slice.len(); + if slice.is_empty() { + return Ok(()); + } + // Set once a free-list splice is attempted: a torn/failed atomic push + // must not hand back a handle that could double-free. + let mut lost = false; + let result = (|| -> io::Result<()> { + let block_start = Self::block_start_of(start)?; + let word = u64::from_le_bytes(read_bstack!(self.stack, block_start => u64)); + if word & Self::IN_USE_BIT == 0 { + return Err(io::Error::new( + io::ErrorKind::InvalidInput, + "double free: block is already free", + )); + } + if word & !Self::IN_USE_BIT != len { + return Err(io::Error::new( + io::ErrorKind::InvalidInput, + "cannot free a partial or mismatched slice", + )); + } + let size = Self::class_blocksize(Self::phys_need(len)?); + let class = Self::classify(size); + + if size > Self::MAX_CLASS { + let end = block_start.checked_add(size).ok_or_else(|| { + io::Error::new(io::ErrorKind::InvalidInput, "block end overflows u64") + })?; + // Oversized tail block: hand its bytes back to the stack instead of + // the free list. Under `atomic`, `try_discard` fuses the tail check + // and the drop; otherwise check `len` then `discard`. + #[cfg(feature = "atomic")] + if self.stack.try_discard(end, size)? { + return Ok(()); + } + #[cfg(not(feature = "atomic"))] + if end == self.stack.len()? { + self.stack.discard(size)?; + return Ok(()); + } + } + lost = true; + self.push(block_start, size, class) + })(); + result.map_err(|source| BStackAllocError { + source, + handle: if lost { + None + } else { + // SAFETY: (start, len) still describes the caller's live block. + Some(unsafe { BStackOwnedSlice::from_raw_parts(self, start, len) }) + }, + }) + } +} + +#[cfg(all(test, feature = "set"))] +mod _assertions { + use super::SegregatedBStackAllocator; + fn _send() + where + SegregatedBStackAllocator: Send, + { + } + #[cfg(feature = "atomic")] + fn _sync() + where + SegregatedBStackAllocator: Sync, + { + } +} + +#[cfg(all(test, feature = "set"))] +mod tests { + use super::SegregatedBStackAllocator as Seg; + use crate::BStack; + use crate::alloc::BStackAllocator; + use std::sync::atomic::{AtomicU64, Ordering}; + + struct Guard(std::path::PathBuf); + impl Drop for Guard { + fn drop(&mut self) { + let _ = std::fs::remove_file(&self.0); + } + } + fn temp_path() -> std::path::PathBuf { + static COUNTER: AtomicU64 = AtomicU64::new(0); + let id = COUNTER.fetch_add(1, Ordering::Relaxed); + let pid = std::process::id(); + std::env::temp_dir().join(format!("bstack_seg_{pid}_{id}.bin")) + } + fn new_alloc() -> (Seg, Guard) { + let path = temp_path(); + let g = Guard(path.clone()); + (Seg::new(BStack::open(&path).unwrap()).unwrap(), g) + } + + // ── classification math ────────────────────────────────────────────────── + + #[test] + fn class_scheme_constants() { + assert_eq!(Seg::LINEAR_CLASSES, 16); + assert_eq!(Seg::GEO_CLASSES, 16); + assert_eq!(Seg::NUM_CLASSES, 33); + assert_eq!(Seg::OVERSIZED_CLASS, 32); + assert_eq!(Seg::ARENA_START, 304); + } + + #[test] + fn classify_boundaries() { + assert_eq!(Seg::classify(16), 0); + assert_eq!(Seg::classify(256), 15); // last linear + assert_eq!(Seg::classify(320), 16); // first geometric + assert_eq!(Seg::classify(512), 19); // top of octave 8 + assert_eq!(Seg::classify(640), 20); // first of octave 9 + assert_eq!(Seg::classify(4096), 31); // last geometric + assert_eq!(Seg::classify(4112), Seg::OVERSIZED_CLASS); // oversized + } + + #[test] + fn class_blocksize_snaps() { + // len 200 → need 208 → linear 208 (exact fit, data 200) + assert_eq!(Seg::class_blocksize(Seg::phys_need(200).unwrap()), 208); + // len 500 → need 512 → octave (256,512] → 512 + assert_eq!(Seg::class_blocksize(Seg::phys_need(500).unwrap()), 512); + // len 8 → need 16 → minimum block + assert_eq!(Seg::class_blocksize(Seg::phys_need(8).unwrap()), 16); + // oversized passes through as a raw multiple of 16 + assert_eq!(Seg::class_blocksize(Seg::phys_need(5000).unwrap()), 5008); + } + + #[test] + fn class_blocksize_roundtrips_through_classify() { + for len in [1u64, 8, 9, 24, 100, 200, 255, 300, 500, 1000, 3000, 4000] { + let size = Seg::class_blocksize(Seg::phys_need(len).unwrap()); + // The block is large enough and its class tag round-trips. + assert!(size >= len + Seg::OVERHEAD, "len {len} size {size}"); + assert_eq!(size % Seg::QUANTUM, 0); + assert!(Seg::classify(size) < Seg::OVERSIZED_CLASS); + } + } + + // ── behaviour ──────────────────────────────────────────────────────────── + + #[test] + fn seg_new_initialises_header() { + let (a, _g) = new_alloc(); + assert_eq!(a.stack().len().unwrap(), Seg::ARENA_START); + } + + #[test] + fn seg_alloc_zero_is_empty() { + let (a, _g) = new_alloc(); + assert!(a.alloc(0).unwrap().is_empty()); + } + + #[test] + fn seg_dealloc_then_alloc_reuses_same_class_block() { + let (a, _g) = new_alloc(); + // 100 and 104 both round up to need 112 → class 6 (block 112). + let s1 = a.alloc(100).unwrap(); + let off1 = s1.start(); + a.dealloc(s1).unwrap(); + let s2 = a.alloc(104).unwrap(); + assert_eq!( + s2.start(), + off1, + "block should be reused from the free list" + ); + } + + #[test] + fn seg_distinct_classes_do_not_alias() { + let (a, _g) = new_alloc(); + let s1 = a.alloc(100).unwrap(); // block 112, class 6 + let off1 = s1.start(); + a.dealloc(s1).unwrap(); + let s2 = a.alloc(500).unwrap(); // block 512, class 19 — different list + assert_ne!(s2.start(), off1); + } + + #[test] + fn seg_write_read_round_trip_and_reopen() { + let path = temp_path(); + let _g = Guard(path.clone()); + let off = { + let a = Seg::new(BStack::open(&path).unwrap()).unwrap(); + let mut s = a.alloc(300).unwrap(); + s.write(b"segregated allocator payload").unwrap(); + let off = s.start(); + drop(a); + off + }; + let a = Seg::new(BStack::open(&path).unwrap()).unwrap(); + let s = unsafe { crate::alloc::BStackSlice::from_raw_parts(a.stack(), off, 28) }; + assert_eq!(s.read().unwrap(), b"segregated allocator payload"); + } + + // ── realloc ────────────────────────────────────────────────────────────── + + #[test] + fn seg_realloc_same_class_grow_zeros_and_preserves() { + let (a, _g) = new_alloc(); + // 90 and 100 both map to block 112 (class 6): a same-class resize. + let mut s = a.alloc(90).unwrap(); + s.write([0xABu8; 90]).unwrap(); + let off = s.start(); + let s = a.realloc(s, 100).unwrap(); + assert_eq!(s.start(), off, "same-class grow stays in place"); + let data = s.read().unwrap(); + assert_eq!(&data[..90], &[0xABu8; 90], "prefix preserved"); + assert_eq!(&data[90..], &[0u8; 10], "grown tail zeroed"); + } + + #[test] + fn seg_realloc_same_class_shrink_then_grow_has_no_stale_bytes() { + let (a, _g) = new_alloc(); + let mut s = a.alloc(100).unwrap(); // block 112 + s.write([0xCDu8; 100]).unwrap(); + let s = a.realloc(s, 90).unwrap(); // same class, shrink (stale [90,100)) + let s = a.realloc(s, 100).unwrap(); // same class, grow back + let data = s.read().unwrap(); + assert_eq!(&data[..90], &[0xCDu8; 90]); + assert_eq!( + &data[90..], + &[0u8; 10], + "re-grown bytes must be zero, not stale" + ); + } + + #[test] + fn seg_realloc_tail_grow_in_place() { + let (a, _g) = new_alloc(); + let mut s = a.alloc(100).unwrap(); // block 112, at the tail + s.write([7u8; 100]).unwrap(); + let off = s.start(); + let len_before = a.stack().len().unwrap(); + let s = a.realloc(s, 200).unwrap(); // need 208 (class 12) > 112: cross-class + assert_eq!(s.start(), off, "tail grow extends in place"); + assert!(a.stack().len().unwrap() > len_before); + let data = s.read().unwrap(); + assert_eq!(&data[..100], &[7u8; 100]); + assert_eq!(&data[100..], &[0u8; 100], "grown region zeroed"); + } + + #[test] + fn seg_realloc_cross_class_grow_non_tail_moves() { + let (a, _g) = new_alloc(); + let mut s = a.alloc(100).unwrap(); // block 112 + s.write([9u8; 100]).unwrap(); + let off = s.start(); + let _pin = a.alloc(100).unwrap(); // pins the tail so `s` is interior + let s = a.realloc(s, 300).unwrap(); // cross-class, non-tail → move + assert_ne!(s.start(), off, "interior grow moves to a new block"); + let data = s.read().unwrap(); + assert_eq!(&data[..100], &[9u8; 100]); + assert_eq!(&data[100..], &[0u8; 200]); + } + + // In-place non-tail-shrink carve is atomic-only; without `atomic` the same + // realloc takes the fault-safe move path (covered by the round-trip/fault tests). + #[cfg(feature = "atomic")] + #[test] + fn seg_realloc_cross_class_shrink_non_tail_carves_in_place() { + let (a, _g) = new_alloc(); + let mut s = a.alloc(200).unwrap(); // block 208 (class 12) + s.write([0x5Au8; 200]).unwrap(); + let off = s.start(); + let base = off - Seg::OVERHEAD; + let _pin = a.alloc(100).unwrap(); // pins the tail so `s` is interior + let s = a.realloc(s, 100).unwrap(); // block 112; gap 96 → carve one 96 block + assert_eq!(s.start(), off, "non-tail shrink carves in place, no move"); + assert_eq!( + s.read().unwrap(), + vec![0x5Au8; 100], + "surviving prefix preserved" + ); + // The 96 excess (class 5) at base+112 is reusable. + let r = a.alloc(88).unwrap(); // class 5 (block 96) + assert_eq!(r.start(), base + 112 + Seg::OVERHEAD); + } + + #[test] + fn seg_realloc_to_zero_frees_and_from_empty_allocs() { + let (a, _g) = new_alloc(); + let s = a.alloc(64).unwrap(); + let empty = a.realloc(s, 0).unwrap(); + assert!(empty.is_empty()); + let grown = a.realloc(empty, 48).unwrap(); + assert_eq!(grown.len(), 48); + assert!(!grown.is_empty()); + } + + #[test] + fn seg_realloc_rejects_malformed_pointer_even_on_noop() { + let (a, _g) = new_alloc(); + // A misaligned pointer with new_len == old_len must still be rejected + // (the no-op path validates), handle returned intact. + let bad = unsafe { + crate::alloc::BStackOwnedSlice::from_raw_parts( + &a, + Seg::ARENA_START + Seg::OVERHEAD + 1, + 16, + ) + }; + let err = a.realloc(bad, 16).unwrap_err(); + assert_eq!(err.source.kind(), std::io::ErrorKind::InvalidInput); + assert!(err.handle.is_some()); + } + + #[test] + fn seg_realloc_tail_shrink_in_place() { + let (a, _g) = new_alloc(); + let mut s = a.alloc(200).unwrap(); // block 208, at the tail + s.write([0x3Cu8; 200]).unwrap(); + let off = s.start(); + let len_before = a.stack().len().unwrap(); + let s = a.realloc(s, 100).unwrap(); // new class 112 < 208, at tail → discard + assert_eq!(s.start(), off, "tail shrink stays in place"); + assert!( + a.stack().len().unwrap() < len_before, + "excess discarded from tail" + ); + assert_eq!(s.read().unwrap(), vec![0x3Cu8; 100]); + } + + // ── recover ────────────────────────────────────────────────────────────── + + #[test] + fn seg_recover_relinks_leaked_free_block() { + let (a, _g) = new_alloc(); + let a1 = a.alloc(100).unwrap(); // class 6 (block 112) + let off1 = a1.start(); + let _b = a.alloc(100).unwrap(); // pins a second class-6 block + a.dealloc(a1).unwrap(); // head[6] → a1 + // Simulate a leak: clear head[6] so a1 is free-tagged but unreachable. + a.stack().set(Seg::head_off(6), 0u64.to_le_bytes()).unwrap(); + assert_eq!( + unsafe { a.recover() }.unwrap(), + 0, + "arena fully accounted for" + ); + // a1 is relinked, so the next same-class alloc reuses it. + let r = a.alloc(90).unwrap(); + assert_eq!(r.start(), off1, "recover relinked the leaked block"); + } + + #[test] + fn seg_recover_discards_crashed_extend_tail() { + let (a, _g) = new_alloc(); + let _a1 = a.alloc(100).unwrap(); + let base = a.stack().len().unwrap(); + // Simulate a crashed tail extend: a zero-filled block whose overhead + // write never landed (word == 0 at `base`). + a.stack().extend(128).unwrap(); + assert_eq!(a.stack().len().unwrap(), base + 128); + assert_eq!(unsafe { a.recover() }.unwrap(), 0); + assert_eq!( + a.stack().len().unwrap(), + base, + "orphaned zero tail discarded" + ); + } + + #[test] + fn seg_recover_clean_arena_preserves_data_and_free_list() { + let (a, _g) = new_alloc(); + let mut keep = a.alloc(300).unwrap(); + keep.write(b"survives recover").unwrap(); + let freed = a.alloc(64).unwrap(); + let freed_off = freed.start(); + a.dealloc(freed).unwrap(); + assert_eq!(unsafe { a.recover() }.unwrap(), 0); + assert_eq!(&keep.read().unwrap()[..16], b"survives recover"); + // The free list still works: the freed block is reused. + let r = a.alloc(60).unwrap(); + assert_eq!(r.start(), freed_off); + } + + #[test] + fn seg_largest_class_le() { + assert_eq!(Seg::largest_class_le(16), 16); + assert_eq!(Seg::largest_class_le(256), 256); + assert_eq!(Seg::largest_class_le(272), 256); // just above the linear top + assert_eq!(Seg::largest_class_le(704), 640); + assert_eq!(Seg::largest_class_le(896), 896); // itself a class + assert_eq!(Seg::largest_class_le(4080), 3584); + assert_eq!(Seg::largest_class_le(4096), 4096); + } + + // In-place non-tail-shrink carve is atomic-only (see the sibling test above). + #[cfg(feature = "atomic")] + #[test] + fn seg_realloc_non_tail_shrink_carves_reusable_blocks() { + let (a, _g) = new_alloc(); + let mut s = a.alloc(1000).unwrap(); // block 1024 (class 23) + s.write([0x77u8; 1000]).unwrap(); + let off = s.start(); + let base = off - Seg::OVERHEAD; + let _pin = a.alloc(200).unwrap(); // class 12 — pins the tail, s is interior + let s = a.realloc(s, 300).unwrap(); // block 320; gap 704 → carve 640 + 64 + assert_eq!(s.start(), off, "non-tail shrink keeps the block in place"); + assert_eq!( + &s.read().unwrap()[..300], + &[0x77u8; 300], + "prefix preserved" + ); + // Carved 640 (class 20) at base+320, 64 (class 3) at base+960 — reusable. + let r640 = a.alloc(632).unwrap(); // class 20 + assert_eq!(r640.start(), base + 320 + Seg::OVERHEAD); + let r64 = a.alloc(56).unwrap(); // class 3 + assert_eq!(r64.start(), base + 960 + Seg::OVERHEAD); + assert_eq!( + unsafe { a.recover() }.unwrap(), + 0, + "arena fully accounted for after carve" + ); + } + + #[test] + fn seg_oversized_non_exact_reuse_carves_excess() { + let (a, _g) = new_alloc(); + let x = a.alloc(5000).unwrap(); // oversized, block 5008 + let off_x = x.start(); + let base = off_x - Seg::OVERHEAD; + let _pin = a.alloc(50).unwrap(); // pins the tail so X is interior + a.dealloc(x).unwrap(); // X → oversized free list (size 5008) + // Y needs block 4112 (oversized); reuses X (5008 ≥ 4112), carves 896. + let y = a.alloc(4090).unwrap(); + assert_eq!(y.start(), off_x, "oversized reuse hands back X's block"); + // The 896 excess is itself class 22 (greedy → one block, not 7×128). + let z = a.alloc(888).unwrap(); // class 22 (block 896) + assert_eq!(z.start(), base + 4112 + Seg::OVERHEAD); + assert_eq!(unsafe { a.recover() }.unwrap(), 0); + } + + #[test] + fn seg_double_free_detected() { + let (a, _g) = new_alloc(); + let s = a.alloc(64).unwrap(); + let (start, len) = (s.start(), s.len()); + a.dealloc(s).unwrap(); + let dup = unsafe { crate::alloc::BStackOwnedSlice::from_raw_parts(&a, start, len) }; + let err = a.dealloc(dup).unwrap_err(); + assert_eq!(err.source.kind(), std::io::ErrorKind::InvalidInput); + } + + #[test] + fn seg_dealloc_rejects_malformed_pointer() { + let (a, _g) = new_alloc(); + // Each malformed handle must be rejected without panic, with the handle + // handed back intact: (a) underflowing/header-range start, and + // (b) a mid-block start whose block_start is not QUANTUM-aligned. + for start in [4u64, Seg::ARENA_START + Seg::OVERHEAD + 1] { + let bogus = unsafe { crate::alloc::BStackOwnedSlice::from_raw_parts(&a, start, 16) }; + let err = a.dealloc(bogus).unwrap_err(); + assert_eq!(err.source.kind(), std::io::ErrorKind::InvalidInput); + assert!( + err.handle.is_some(), + "malformed free must return the handle" + ); + } + } + + #[test] + fn seg_oversized_tail_dealloc_shrinks() { + let (a, _g) = new_alloc(); + let base = a.stack().len().unwrap(); + let s = a.alloc(5000).unwrap(); // oversized, at the tail + assert!(a.stack().len().unwrap() > base); + a.dealloc(s).unwrap(); + assert_eq!( + a.stack().len().unwrap(), + base, + "tail oversized block discarded" + ); + } +} diff --git a/src/alloc_fault_tests.rs b/src/alloc_fault_tests.rs index 8291337..1e2ca5f 100644 --- a/src/alloc_fault_tests.rs +++ b/src/alloc_fault_tests.rs @@ -32,7 +32,7 @@ mod alloc_fault_tests { use crate::alloc::{ BStackOwnedSlice, BStackOwnedSliceAllocator, BStackRange, FirstFitBStackAllocator, - GhostTreeBstackAllocator, SlabBStackAllocator, + GhostTreeBstackAllocator, SegregatedBStackAllocator, SlabBStackAllocator, }; use crate::alloc_test_common::{ FuzzConfig, Guard, Operation, Payload, check_is_zero, gen_op, make_allocator, make_payload, @@ -313,4 +313,9 @@ mod alloc_fault_tests { make_allocator!(CheckedSlabBStackAllocator, 64), 0x6666 ); + fault_suite!( + segregated, + make_allocator!(SegregatedBStackAllocator), + 0x7777 + ); } diff --git a/src/alloc_fuzz_tests.rs b/src/alloc_fuzz_tests.rs index 6822cc6..24c1cbf 100644 --- a/src/alloc_fuzz_tests.rs +++ b/src/alloc_fuzz_tests.rs @@ -3,7 +3,7 @@ mod alloc_fuzz_tests { use crate::alloc::{ BStackOwnedSlice, BStackOwnedSliceAllocator, BStackRange, FirstFitBStackAllocator, - GhostTreeBstackAllocator, SlabBStackAllocator, + GhostTreeBstackAllocator, SegregatedBStackAllocator, SlabBStackAllocator, }; use crate::alloc_test_common::{ FuzzConfig, Guard, Operation, Payload, check_is_zero, gen_op, make_allocator, make_payload, @@ -371,6 +371,7 @@ mod alloc_fuzz_tests { check_slab_64, make_allocator!(CheckedSlabBStackAllocator, 64) ); + fuzz_suite!(segregated, make_allocator!(SegregatedBStackAllocator)); mod double_free { use super::*; @@ -383,5 +384,10 @@ mod alloc_fuzz_tests { fn check_slab_16() { super::run_double_free_error(make_allocator!(CheckedSlabBStackAllocator, 16)); } + + #[test] + fn segregated() { + super::run_double_free_error(make_allocator!(SegregatedBStackAllocator)); + } } } diff --git a/src/lib.rs b/src/lib.rs index 03f4f49..9a06d89 100644 --- a/src/lib.rs +++ b/src/lib.rs @@ -400,7 +400,8 @@ //! [`GhostTreeBstackAllocator`], and [`DebugCheckingAllocator`]. //! Combined with `set`, also enables [`BStackSliceWriter`], //! [`FirstFitBStackAllocator`], [`SlabBStackAllocator`], -//! [`CheckedSlabBStackAllocator`], and [`BStackByteVec`]. +//! [`CheckedSlabBStackAllocator`], [`SegregatedBStackAllocator`] +//! (experimental), and [`BStackByteVec`]. //! //! * **`atomic`** — Compound read-modify-write operations that hold the write //! lock across what would otherwise be separate calls. Combined with `set`, @@ -509,6 +510,23 @@ //! [`recover`](CheckedSlabBStackAllocator::recover) automatically). //! Requires both `alloc` and `set` features. //! +//! * [`SegregatedBStackAllocator`] — **experimental** segregated (binned) +//! free-list allocator. Generalises [`CheckedSlabBStackAllocator`] from one +//! block size to 33 size classes sharing a single arena: 16 linear classes +//! (16‥256 B, step 16), 16 geometric classes (320‥4096 B, 4 per octave), and +//! one shared oversized bucket. Each class is an independent intrusive free +//! list; the class is computed from the request with register arithmetic (no +//! tables), giving O(1) classed alloc/dealloc. Every block carries the same +//! 8-byte overhead tag as the checked slab, so leaked blocks are reclaimable by +//! a linear scan and double-frees are caught. A single [`new`](SegregatedBStackAllocator::new) +//! constructor initialises an empty stack or reopens one (running recovery +//! automatically). Requires both `alloc` and `set`; `Send` in all +//! configurations, `Send + Sync` with `atomic` (no allocator-level lock — +//! free-list splices ride [`BStack::process_gen`]/[`BStack::inplace_gen`]). +//! **Experimental:** the on-disk format and API may change, some resize paths +//! differ between the `atomic` and non-`atomic` builds, and the background +//! coalescer / deep in-use-leak GC are not yet implemented. +//! //! * [`DebugCheckingAllocator`](DebugCheckingAllocator) — transparent debug //! wrapper. Wraps any allocator whose `Allocated` type is [`BStackOwnedSlice`] //! and whose `Error` is [`io::Error`]. Tracks allocated and freed regions in @@ -632,14 +650,14 @@ pub use alloc::{ BStackChunkIter, BStackOwnedSlice, BStackOwnedSliceAllocator, BStackRange, BStackSlice, BStackSliceReader, BStackUninitAllocator, DebugCheckingAllocator, LinearBStackAllocator, }; +#[cfg(all(feature = "guarded", feature = "atomic"))] +pub use alloc::{BStackAtomicGuardedSlice, BStackAtomicGuardedSliceSubview}; #[cfg(all(feature = "alloc", feature = "set"))] pub use alloc::{ BStackByteVec, BStackByteVecIter, BStackSliceWriter, CheckedSlabBStackAllocator, - FirstFitBStackAllocator, GhostTreeBstackAllocator, SlabBStackAllocator, + FirstFitBStackAllocator, GhostTreeBstackAllocator, SegregatedBStackAllocator, + SlabBStackAllocator, }; - -#[cfg(all(feature = "guarded", feature = "atomic"))] -pub use alloc::{BStackAtomicGuardedSlice, BStackAtomicGuardedSliceSubview}; #[cfg(feature = "guarded")] pub use alloc::{BStackGuardedSlice, BStackGuardedSliceSubview};