add bloom filter pre-screen for external-internal id mapping

[Seth Ockerman (OckermanSethGVSU)]

What this does

Adds a blocked Bloom filter in front of PointMappings external → internal ID lookups.

When Qdrant processes an update, SegmentHolder::find_points_to_update_and_delete checks each segment for the point. With S segments, an existing point usually produces one hit and S - 1 misses, while a new point misses in every segment.

Those misses currently require walking a BTreeMap. The Bloom filter provides a cheap pre-check so most missing IDs can be rejected before touching the tree.

The BTreeMap is still the source of truth. A Bloom-filter hit only means "maybe present," so the normal lookup still runs. A miss means the ID is definitely absent.

The implementation uses the same split-block approach as Impala/Parquet. Each lookup touches a single small block, keeping the check cache-friendly.

Deletes and persistence

The filter tracks IDs that have ever been inserted, so deleted IDs may leave stale bits behind. This is safe: stale bits can only cause false positives, which fall through to the BTreeMap and return the correct result. Bits are never cleared, so the filter cannot introduce false negatives.

The filter is not persisted and does not change the on-disk format. On restart, PointMappings is rebuilt from the existing change log through set_link, which rebuilds the filter as part of normal recovery.

The filter starts small and grows as needed. Rebuilds are sized with extra headroom and also remove stale bits left by deleted points.

Measurements

End-to-end release build, whole server, 8M points:

Update pattern	Filter OFF	Filter ON	Result
Scattered IDs	118.8 s	62.1 s	1.91× faster
Sequential IDs	52.7 s	52.3 s	No meaningful change

The benefit depends heavily on ID locality. Sequential updates already walk nearby BTreeMap entries that tend to stay cached, so there is little to gain. Scattered IDs cause more random tree lookups and benefit much more.

Microbenchmark at 1M points per segment, pinned to one core:

num/miss     196 ns -> 14.3 ns
uuid/miss    286 ns -> 38.5 ns
num/hit      180 ns -> 204 ns
uuid/hit     245 ns -> 308 ns
reopen 200k  33.8 ms -> 39.7 ms

RAM overhead was about 2.1% at 1M points.

Turning it off

Set:

QDRANT_ID_TRACKER_BLOOM_FILTER=0

false, off, and no also work.

A disabled filter allocates nothing and always falls through to the BTreeMap, making it useful for A/B testing without deploying a different binary.

Testing

The change has been tested against normal segment tests as well as randomized ID-tracker operation sequences covering numeric and UUID IDs, deletes, re-links, slot reuse, and deferred-point configurations.

I also tested recovery and lifecycle behavior against a real binary, including:

• kill -9 followed by WAL recovery
• snapshot create/restore
• concurrent writers
• delete-heavy workloads
• restarting with the filter disabled and then enabled again

Known limits

• Only appendable segments use the filter; immutable/compressed ID trackers are unchanged.
• Adds about 2.1% ID-tracker RAM at 1M points.
• Mapping-log replay was about 17% slower in the tested startup benchmark.
• The biggest gains are on workloads with scattered IDs; sequential or highly local ID access sees little benefit.