Context
ZeroDB is a pure-Rust embedded key-value engine, drop-in compatible with LMDB at the "heed" API level (heed is the Rust LMDB wrapper Meilisearch uses today). Its consumers are milli (Meilisearch's indexing/search core) and hannoy (Meilisearch's HNSW vector-search library, an arroy successor).
Several engine-side features are already built, or are actively planned, but they produce zero real-world benefit until the calling code in milli or hannoy is changed to actually use them. That consumer-side work currently isn't written down anywhere, so it risks never happening or being re-derived from scratch later. This issue asks for a hand-off document (under docs/) giving the Meilisearch team concrete, actionable recipes.
Task
Write a recipe for each of the following:
- Scheduler durability policy. Meilisearch's task scheduler currently appears to do a full disk sync on every task-state flip. Recipe: switch to the durability flag that skips the per-commit meta-page fsync (a crash can then lose visibility of at most the single most recent transaction, never cause corruption) — or, once a future "bounded relaxed durability" engine mode lands (crash loses at most a configurable window of commits, never corruption) — combined with explicit sync checkpoints (
Env::sync(force) / Env::force_sync(), matching LMDB's mdb_env_sync with the force/no-force distinction heed doesn't expose) at fixed points, e.g. once per scheduler tick, instead of syncing on every flip. The sync mechanism is already implemented; only the policy choice is consumer-side.
- Fused read-modify-write adoption. milli's dominant incremental-indexing pattern is: read an existing value (e.g. a bitmap of document ids), merge in a change, write it back — two tree descents plus a temporary allocation per key. Once the proposed single-descent
update(key, closure) engine operation lands, migrate these get-then-put call sites to it.
- Batched multi-key lookup adoption. hannoy's per-hop nearest-neighbor id lookups and milli's per-chunk key lookups during indexing currently issue independent single-key reads. Once a batched multi-key lookup engine operation lands, adopt it at both call sites.
- Allocator/build parity. Document that the Meilisearch binary links jemalloc on Linux (note the implication for allocator choice), and document recommended build flags for the meilisearch binary itself — link-time optimization,
codegen-units=1, target-cpu.
- Disabling OS read-ahead on vector-heavy environments. Once the engine actually honors the flag that turns off OS read-ahead (currently accepted but silently ignored), pass it on hannoy's vector-search environments, whose HNSW graph traversal reads pages in a near-random pattern. This replaces hannoy's own hand-rolled prefetch-hint call.
Each recipe should state whether its engine-side dependency has already landed or is still pending, and link to the corresponding engine work item.
Why it matters
Items 1 and 4 unlock benefit from work already shipped in the engine — writing the recipe down turns a known win into something that actually gets deployed. Items 2, 3, and 5 describe the exact consumer-side change needed the moment the corresponding engine feature ships, so integration can start immediately instead of being rediscovered later. Item 2's read-merge-write pattern is milli's single most common write during incremental indexing; item 3's batched lookups speed up both the neighbor fetches hannoy repeats at every step of graph traversal and the chunked key lookups milli does per indexing batch.
Risk
Three of the five recipes (fused read-modify-write, batched multi-key lookup, and bounded relaxed durability) depend on engine features that have not landed yet. The document should clearly flag which recipes are actionable today versus blocked on upcoming engine work, so the Meilisearch team doesn't start integrating against an API that doesn't exist yet.
See also:
PLAN.md — Phase 3 roadmap listing these engine featurescrates/heed-zerodb/src/flags.rs — where the read-ahead-disable flag is currently a no-op
Context
ZeroDB is a pure-Rust embedded key-value engine, drop-in compatible with LMDB at the "heed" API level (heed is the Rust LMDB wrapper Meilisearch uses today). Its consumers are milli (Meilisearch's indexing/search core) and hannoy (Meilisearch's HNSW vector-search library, an arroy successor).
Several engine-side features are already built, or are actively planned, but they produce zero real-world benefit until the calling code in milli or hannoy is changed to actually use them. That consumer-side work currently isn't written down anywhere, so it risks never happening or being re-derived from scratch later. This issue asks for a hand-off document (under
docs/) giving the Meilisearch team concrete, actionable recipes.Task
Write a recipe for each of the following:
Env::sync(force)/Env::force_sync(), matching LMDB'smdb_env_syncwith the force/no-force distinction heed doesn't expose) at fixed points, e.g. once per scheduler tick, instead of syncing on every flip. The sync mechanism is already implemented; only the policy choice is consumer-side.update(key, closure)engine operation lands, migrate these get-then-put call sites to it.codegen-units=1,target-cpu.Each recipe should state whether its engine-side dependency has already landed or is still pending, and link to the corresponding engine work item.
Why it matters
Items 1 and 4 unlock benefit from work already shipped in the engine — writing the recipe down turns a known win into something that actually gets deployed. Items 2, 3, and 5 describe the exact consumer-side change needed the moment the corresponding engine feature ships, so integration can start immediately instead of being rediscovered later. Item 2's read-merge-write pattern is milli's single most common write during incremental indexing; item 3's batched lookups speed up both the neighbor fetches hannoy repeats at every step of graph traversal and the chunked key lookups milli does per indexing batch.
Risk
Three of the five recipes (fused read-modify-write, batched multi-key lookup, and bounded relaxed durability) depend on engine features that have not landed yet. The document should clearly flag which recipes are actionable today versus blocked on upcoming engine work, so the Meilisearch team doesn't start integrating against an API that doesn't exist yet.
See also:
PLAN.md— Phase 3 roadmap listing these engine featurescrates/heed-zerodb/src/flags.rs— where the read-ahead-disable flag is currently a no-op