fix(search): OR+prefix FTS query + title-weighted bm25 to fix natural-language recall

[omarhash95]

Problem

Search() runs mem_search through sanitizeFTS, which quotes each term and joins them with spaces — i.e. FTS5 implicit AND. Every term must match, so any natural-language / paraphrased / multi-term query returns nothing even when the answer is stored verbatim. Examples that returned No memories found on v1.16.1 against a populated DB:

• "how do I permanently remove a saved memory from engram"
• "should we adopt X for better memory recall"
• "How long have Mel ..." (where the stored text says Melanie)

The longer / more natural the query, the more certain the miss. (I measured ~0% recall on a multi-question retrieval slice with raw questions; short keyword queries worked, natural questions did not.)

Change

Two small, localized edits in internal/store/store.go:

1. sanitizeFTS now emits "t1"* OR "t2"* OR ... instead of space-joined AND:
   • OR restores recall (any term may match),
   • prefix (*) makes mel match melanie, delet match deleted,
   • internal quotes are escaped ("").
2. Search() ORDER BY uses weighted bm25(observations_fts, 10,2,1,1,1,8) (title 10×, topic_key 8×, content 2×) so the OR expansion still surfaces the most on-topic observation first rather than any doc that merely shares a common term.

Existing internal/store tests pass.

Result (same DB, before → after)

query	before	after
"should we adopt X for better memory recall"	0	correct obs at #1
"what did we learn about retrieval quality being poor"	0	correct obs at #1
"how do I permanently remove a saved memory"	0	relevant obs returned

Precise keyword queries still rank tightly because bm25 orders multi-term matches highest.

Tradeoff + happy to adjust

Defaulting to OR favors recall over precision, which is a behavior change. If you'd prefer to preserve strict-AND precision, two options I'm glad to implement instead:

• AND-first, OR-fallback: run the current implicit-AND query, and only expand to OR when it returns fewer than N results — best of both.
• Config flag to opt into the OR behavior.

I can also add unit tests for sanitizeFTS and a Search recall test if you'd like. Let me know which direction you prefer.

[Copilot]

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

This PR tunes SQLite FTS5 search to improve recall and ranking quality by expanding the MATCH expression and applying weighted BM25 ordering.

Changes:

• Replace ORDER BY fts.rank with a weighted bm25(...) ranking to prioritize title/topic matches.
• Change sanitizeFTS to generate an OR-joined, prefix (*) MATCH query for higher recall.
• Expand inline documentation explaining the new FTS tuning rationale.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Alan-TheGentleman]

Closing this because the search-recall problem is already being handled through #352 and #475 with a safer match_mode approach. Changing the default FTS behavior globally to OR plus prefix matching is too risky for the current contract. If more search tuning is needed after #475, we should do it as a follow-up issue with explicit behavior and ranking expectations.