Context
ZeroDB is a pure-Rust embedded key-value store — a transactional B+tree over a memory-mapped file — built as a drop-in replacement for LMDB at the "heed" API level (heed is the Rust LMDB wrapper Meilisearch uses). It's consumed by Meilisearch's indexing core ("milli") and its vector-search index ("hannoy").
The maintenance tools in crates/zerodb-tools (check, and the dump/compaction path) currently walk the whole database tree on a single thread — see cmd_check in crates/zerodb-tools/src/commands.rs and the walker in crates/zerodb-core/src/check.rs. On a 100GB+ environment this takes multiple minutes.
The read-only memory-mapped snapshot these tools walk is immutable for the lifetime of the walk, and Rust's Sync trait lets the compiler prove it's safe to share across threads without extra locking. That means disjoint subtrees of the B+tree can safely be walked by different threads at the same time, with the compiler — not a human — verifying there's no unsafe aliasing. (In C, LMDB-style tools would need this sharding done by hand with pthreads, with no compile-time safety net.) None of this is exercised today: everything runs on one thread.
Note: rayon (the popular Rust data-parallelism crate) is not currently on this project's list of approved dependencies, so using it here needs a short design document (ADR) justifying the addition, unless the work is done with the standard library's scoped threads instead (std::thread::scope, stable since Rust 1.63, which allows threads to safely borrow data without an Arc/'static requirement).
Task
Parallelize the three tool operations that are naturally split into independent units of work:
check — walk the root page's child subtrees concurrently instead of one after another.- Page checksum verification — each page's CRC32C checksum is independent of every other page, so this is "embarrassingly parallel" (no coordination needed between units of work).
- Compaction (used by the copy/dump-and-reload path) — have multiple threads read and serialize entries in parallel, feeding a single sequential thread that appends the results to the output file in order (the output file itself must be written sequentially).
Implement this with std::thread::scope-based parallelism if that's sufficient; otherwise write an ADR proposing rayon as a new allowed dependency before adding it.
Why it matters
This is an operations/tooling improvement, not a query-latency one — it doesn't speed up reads or writes during normal Meilisearch operation. Its value is for the multi-minute check/compact runs operators run against large environments: parallelizing them turns "minutes" into "minutes divided by core count."
It also has a secondary durability benefit: check and compaction hold open a long-lived read transaction while they walk the database. Per this project's garbage-collection design, pages freed by a write transaction can only be reused once the oldest still-open read transaction has moved past it — so a long-running check blocks reclamation of freed pages for its entire duration, and the file's free-page list grows during that window. Making the walk faster shrinks that window and reduces this free-list bloat.
Risk
Low — this only touches read-only tooling code paths (crates/zerodb-tools, crates/zerodb-core/src/check.rs), not the core transactional engine, write path, or on-disk format.
See also: crates/zerodb-tools/src/commands.rs (check/compact command entry points), crates/zerodb-core/src/check.rs (single-threaded invariant walker)
Context
ZeroDB is a pure-Rust embedded key-value store — a transactional B+tree over a memory-mapped file — built as a drop-in replacement for LMDB at the "heed" API level (heed is the Rust LMDB wrapper Meilisearch uses). It's consumed by Meilisearch's indexing core ("milli") and its vector-search index ("hannoy").
The maintenance tools in
crates/zerodb-tools(check, and the dump/compaction path) currently walk the whole database tree on a single thread — seecmd_checkincrates/zerodb-tools/src/commands.rsand the walker incrates/zerodb-core/src/check.rs. On a 100GB+ environment this takes multiple minutes.The read-only memory-mapped snapshot these tools walk is immutable for the lifetime of the walk, and Rust's
Synctrait lets the compiler prove it's safe to share across threads without extra locking. That means disjoint subtrees of the B+tree can safely be walked by different threads at the same time, with the compiler — not a human — verifying there's no unsafe aliasing. (In C, LMDB-style tools would need this sharding done by hand with pthreads, with no compile-time safety net.) None of this is exercised today: everything runs on one thread.Note:
rayon(the popular Rust data-parallelism crate) is not currently on this project's list of approved dependencies, so using it here needs a short design document (ADR) justifying the addition, unless the work is done with the standard library's scoped threads instead (std::thread::scope, stable since Rust 1.63, which allows threads to safely borrow data without anArc/'staticrequirement).Task
Parallelize the three tool operations that are naturally split into independent units of work:
check— walk the root page's child subtrees concurrently instead of one after another.Implement this with
std::thread::scope-based parallelism if that's sufficient; otherwise write an ADR proposingrayonas a new allowed dependency before adding it.Why it matters
This is an operations/tooling improvement, not a query-latency one — it doesn't speed up reads or writes during normal Meilisearch operation. Its value is for the multi-minute
check/compact runs operators run against large environments: parallelizing them turns "minutes" into "minutes divided by core count."It also has a secondary durability benefit:
checkand compaction hold open a long-lived read transaction while they walk the database. Per this project's garbage-collection design, pages freed by a write transaction can only be reused once the oldest still-open read transaction has moved past it — so a long-runningcheckblocks reclamation of freed pages for its entire duration, and the file's free-page list grows during that window. Making the walk faster shrinks that window and reduces this free-list bloat.Risk
Low — this only touches read-only tooling code paths (
crates/zerodb-tools,crates/zerodb-core/src/check.rs), not the core transactional engine, write path, or on-disk format.See also: crates/zerodb-tools/src/commands.rs (check/compact command entry points), crates/zerodb-core/src/check.rs (single-threaded invariant walker)