perf(store): drive hybrid search from candidate set to avoid full-table scan

[arabold]

Summary

Follow-up to #454 / #455. The MATERIALIZED fix in #455 removed the pathological re-evaluation of the vector KNN, but the hybrid search query still scans the entire documents table on every search: the final SELECT reads FROM documents d … LEFT JOIN vec_distances … LEFT JOIN fts_scores and narrows to the match set with WHERE (v.id IS NOT NULL OR f.id IS NOT NULL). So search cost still scaled with total database size (a residual, much smaller cost than the original wedge, but real on large corpora).

This PR drives the final query from the candidate id set instead:

candidates AS (
  SELECT id FROM vec_distances
  UNION
  SELECT id FROM fts_scores
)
SELECT …
FROM candidates c
JOIN documents d ON d.id = c.id
JOIN pages p ON d.page_id = p.id
LEFT JOIN vec_distances v ON d.id = v.id
LEFT JOIN fts_scores f ON d.id = f.id
WHERE NOT EXISTS ( … structural … )

Cost is now proportional to the number of matches (~k + overfetch), not the size of the database. Documents are looked up by primary key.

Evidence

Synthesized a 72,581-chunk / 5 GB database (matching the scale in #454) and ran the exact old vs new statements with the same parameters:

	Result rows	Time (best of 3)	Query plan
Current (FROM documents)	151	233.6 ms	SCAN d (full documents table)
This PR (candidate-driven)	151	8.7 ms	SCAN candidates → SEARCH d … PRIMARY KEY

• Results are identical — same 151 rows with the same vec_score/fts_score (verified by comparing the full result sets).
• ~27× faster at 72k chunks, and the improvement grows with DB size because the full-table scan is gone.
• End-to-end through the HTTP API on the 72k-doc DB: searches ~0.2–0.5 s, and ping stays at ~1 ms while a search runs.

Notes / scope

• Behavior-preserving: purely a query-shape change; RRF ranking (done in JS on the returned rows) is unaffected.
• The FTS-only fallback path (when vector search is disabled) already drives from the FTS index and is left unchanged.
• No new dependencies; no schema/migration changes.

Testing

• npx vitest run src/store/DocumentStore.test.ts test/vector-search-e2e.test.ts → 76 passed.
• npm run typecheck clean; npm run build clean.

🤖 Generated with Claude Code

[Copilot]

Pull request overview

This PR optimizes the hybrid (vector + FTS) search query in DocumentStore by driving the final result selection from a small candidate-id set rather than scanning the full documents table, making query cost scale with match count instead of total DB size.

Changes:

• Adds a candidates CTE that unions ids from vec_distances and fts_scores.
• Updates the final SELECT to start from candidates and join into documents/pages, removing the prior full-table-scan shape.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.