Description
In cuvs_bench, OpenSearchBackend.search() keeps neighbors only from the last ef_search in a sweep, so recall is computed for just that one point — every other sweep point is dropped and the recall-vs-QPS curve collapses to a single dot.
The loop records per-point timing but no per-point recall and overwrites last_neighbors every iteration (opensearch.py#L960-L969). SearchResult has a single recall/neighbors field, so the orchestrator computes recall once, from those last-point neighbors (orchestrator.py#L265-L274).
TLDR:
For both built-in parameter sweep presets (--groups) in config/algos/opensearch_faiss_hnsw.yaml, i.e. test and base, both yield only one data point (value of ef_search) per build (value of m). No pareto frontiers built for each value of m.
Steps to Reproduce
With OpenSearch running and an opensearch_faiss_hnsw index built, run the test group (ef_search: [10, 20]):
BENCH_GROUPS=test DATASET=sift-128-euclidean K=10 \
docker compose -f docker-compose.multinode.yml --profile client run --rm bench
Two sweep points go in, one comes out: ef_search=20 recall@10 0.9967 QPS 2045.4
Omitted 1 timing-only search rows without recall.
The results JSON has one entry and the plot is a single point. (base, 4 m × 10 ef_search, likewise collapses from 40 points to 4)
Expected Behavior
Each ef_search should get its own recall (from that point's neighbors), so an M×E sweep produces M×E points and actual frontier curves.
Environment Details
- Environment location: Cloud (AWS EC2), Docker
- Method of cuVS install: from source, the Dockerfile does git clone https://github.com/rapidsai/cuvs.git (@ e626144) and adds python/cuvs_bench to the image
- docker compose --profile client build bench then ... run --rm bench
See https://github.com/jrbourbeau/cuvs/blob/9cf5c45ea07d884e07242f4727c027354e5232b0/deploy/AWS_MULTINODE_SINGLE_ROLE_SETUP.md
Additional context
Fix: compute recall inside the sweep loop per ef_search and store it per entry in per_search_param_results, rather than keeping only the last point's neighbors.
Description
In cuvs_bench, OpenSearchBackend.search() keeps neighbors only from the last ef_search in a sweep, so recall is computed for just that one point — every other sweep point is dropped and the recall-vs-QPS curve collapses to a single dot.
The loop records per-point timing but no per-point recall and overwrites last_neighbors every iteration (opensearch.py#L960-L969). SearchResult has a single recall/neighbors field, so the orchestrator computes recall once, from those last-point neighbors (orchestrator.py#L265-L274).
TLDR:
For both built-in parameter sweep presets (--groups) in config/algos/opensearch_faiss_hnsw.yaml, i.e. test and base, both yield only one data point (value of ef_search) per build (value of m). No pareto frontiers built for each value of m.
Steps to Reproduce
With OpenSearch running and an opensearch_faiss_hnsw index built, run the test group (ef_search: [10, 20]):
Two sweep points go in, one comes out: ef_search=20 recall@10 0.9967 QPS 2045.4
Omitted 1 timing-only search rows without recall.
The results JSON has one entry and the plot is a single point. (base, 4 m × 10 ef_search, likewise collapses from 40 points to 4)
Expected Behavior
Each ef_search should get its own recall (from that point's neighbors), so an M×E sweep produces M×E points and actual frontier curves.
Environment Details
See https://github.com/jrbourbeau/cuvs/blob/9cf5c45ea07d884e07242f4727c027354e5232b0/deploy/AWS_MULTINODE_SINGLE_ROLE_SETUP.md
Additional context
Fix: compute recall inside the sweep loop per ef_search and store it per entry in per_search_param_results, rather than keeping only the last point's neighbors.