Context
The criterion benchmark harness (crates/zerodb-oracle/benches/engine_comparison.rs — this file lives in the same crate as ZeroDB's differential test harness, the one place in the codebase allowed to link the real LMDB C library, because the benchmark also needs to run against real LMDB) is a single binary that runs the same workloads against both real LMDB (via heed, the Rust LMDB wrapper Meilisearch uses) and ZeroDB (via the heed-zerodb adapter) and compares timings. Right now the whole binary runs under Rust's default system allocator for both engines.
ZeroDB's write path currently allocates more than LMDB's per commit: every modified page gets its own heap-allocated box, plus extra HashMap/Vec churn in the in-memory dirty-page store (this is a known, separately tracked performance item, documented in docs/PERF-GAP-VS-LMDB.md — the dirty-page store is a HashMap of individually heap-allocated, zero-filled page boxes with a fresh sort on every commit, and the fix is to replace it with a pooled arena; this issue does not do that work, it just makes the benchmark honest in the meantime).
The mismatch: the real Meilisearch binary links jemalloc (via the tikv-jemallocator crate) on Linux, which is specifically good at the many-small-allocations pattern ZeroDB's write path produces. Benchmarking both engines under the plain system allocator instead of jemalloc understates how ZeroDB's write path will actually perform inside Meilisearch, because the allocator the benchmark uses doesn't match the allocator the consumer actually links.
Task
- Set
#[global_allocator] in the bench binary to a faster allocator (mimalloc or tikv-jemallocator), added as a dev-dependency so it stays confined to crates/zerodb-oracle/benches/ and does not become a dependency of the library crates.
- Re-run the criterion benchmarks and paste the before/after diff in the PR.
- Add a short docs note for consumers explaining that ZeroDB's write-path performance is sensitive to which global allocator the host application links, and that jemalloc/mimalloc are recommended over the system default for allocation-heavy workloads.
- If this allocator crate is ever promoted from a bench-only dev-dependency to a real dependency of a published crate, that requires going through the existing new-dependency process (the current allowlist is memmap2, libc, thiserror, crossbeam-utils, rand, proptest, arbitrary, criterion, with io-uring behind a feature) and a short design document (ADR) in docs/adr/ approved first.
Why it matters
Gives honest milli (Meilisearch's indexing engine, the main consumer of this storage layer) vs. LMDB numbers instead of numbers skewed by an allocator mismatch between the benchmark and production. It's also a cheap, low-risk win for any consumer that isn't already linking jemalloc, independent of the larger (and harder) work of removing the allocations from the write path itself.
Risk
Low — the change is confined to the bench binary via a dev-dependency and does not touch any library code path.
See also: docs/PERF-GAP-VS-LMDB.md — describes the dirty-page store's per-page heap allocations this benchmark change works around.
Context
The criterion benchmark harness (
crates/zerodb-oracle/benches/engine_comparison.rs— this file lives in the same crate as ZeroDB's differential test harness, the one place in the codebase allowed to link the real LMDB C library, because the benchmark also needs to run against real LMDB) is a single binary that runs the same workloads against both real LMDB (viaheed, the Rust LMDB wrapper Meilisearch uses) and ZeroDB (via theheed-zerodbadapter) and compares timings. Right now the whole binary runs under Rust's default system allocator for both engines.ZeroDB's write path currently allocates more than LMDB's per commit: every modified page gets its own heap-allocated box, plus extra
HashMap/Vecchurn in the in-memory dirty-page store (this is a known, separately tracked performance item, documented in docs/PERF-GAP-VS-LMDB.md — the dirty-page store is a HashMap of individually heap-allocated, zero-filled page boxes with a fresh sort on every commit, and the fix is to replace it with a pooled arena; this issue does not do that work, it just makes the benchmark honest in the meantime).The mismatch: the real Meilisearch binary links jemalloc (via the
tikv-jemallocatorcrate) on Linux, which is specifically good at the many-small-allocations pattern ZeroDB's write path produces. Benchmarking both engines under the plain system allocator instead of jemalloc understates how ZeroDB's write path will actually perform inside Meilisearch, because the allocator the benchmark uses doesn't match the allocator the consumer actually links.Task
#[global_allocator]in the bench binary to a faster allocator (mimallocortikv-jemallocator), added as a dev-dependency so it stays confined tocrates/zerodb-oracle/benches/and does not become a dependency of the library crates.Why it matters
Gives honest milli (Meilisearch's indexing engine, the main consumer of this storage layer) vs. LMDB numbers instead of numbers skewed by an allocator mismatch between the benchmark and production. It's also a cheap, low-risk win for any consumer that isn't already linking jemalloc, independent of the larger (and harder) work of removing the allocations from the write path itself.
Risk
Low — the change is confined to the bench binary via a dev-dependency and does not touch any library code path.
See also: docs/PERF-GAP-VS-LMDB.md — describes the dirty-page store's per-page heap allocations this benchmark change works around.