fix(core): count non-Latin tokens when relaxing full-text queries

[gingeard]

Problem

relaxed_query_words() decides whether a strict AND full-text query may be retried as an OR query. Its eligibility check counts tokens with an ASCII-only pattern:

RELAXATION_ASCII_TOKEN_PATTERN = re.compile(r"[A-Za-z0-9]+")
...
tokens = RELAXATION_ASCII_TOKEN_PATTERN.findall(stripped.lower())
if len(tokens) < 3 or any(token.isdigit() for token in tokens):
    return None

A query written in any non-Latin alphabet yields zero tokens, trips the "fewer than three tokens" guard, and never relaxes — regardless of how long it is.

Only the hybrid path opts into relaxation, so the result is a silent degradation, not an error. The comment right above the call site describes exactly what then happens:

allow_relaxed: question-form queries rarely AND-match, and a dead FTS branch silently degrades hybrid to vector-only ranking.

That is precisely the state every non-Latin corpus is permanently in.

Reproduction

>>> from basic_memory.repository.search_query import relaxed_query_words
>>> relaxed_query_words("how to safely retry a payment")
['safely', 'retry', 'payment']
>>> relaxed_query_words("как безопасно повторить платёж")
None          # ← expected the same shape of result

End to end on a Russian corpus (17 notes, 176 observations), from the timing log:

fts_count=0  vector_count=21   ← lexical half of "hybrid" contributes nothing

Every question-form query behaved this way. Fusion (max(v, f) + 0.3 * min(v, f)) had one side pinned at zero, so ranking was vector-only. The practical cost is exact identifiers: Idempotency-Key, next_page and similar tokens are what FTS is good at and embeddings are blind to.

Why this looks unintended

This is the same defect #1022 already diagnosed, one script family at a time.

#994 introduced the relaxed retry because "questions rarely have every word in one document", and its guards are deliberate. #1022 then found that the eligibility check could not see the terms at all:

The old eligibility path only extracted ASCII alphanumeric tokens. Short CJK queries split by spaces therefore produced zero tokens, so strict FTS misses never reached the relaxed retry.

That fix added a CJK branch beside the gate rather than repairing the gate, so the ASCII assumption was known and worked around — but only for Han, kana and Hangul. Cyrillic, Greek, Hebrew, Arabic, Armenian and Georgian produce zero tokens for exactly the same reason, with no branch of their own.

#1022 asks that existing ASCII relaxation behaviour be preserved, including client-side state management and foo/bar baz qux. It is: on ASCII input the new tokenizer returns token-for-token what [A-Za-z0-9]+ returned, those two cases included. The CJK branch runs before this line and is untouched.

This also does not reopen #577. The fusion formula and every guard are unchanged; Cyrillic queries simply reach the path the project already accepts for English and CJK, where "fusion plus bm25 keep relaxed lexical candidates from dominating precision".

Fix

Token counting moves into relaxation_word_tokens: a token is a run of alphanumeric characters together with any combining marks attached to them.

That repairs two distinct ways the ASCII rule mis-counted.

Non-Latin letters now count. Cyrillic, Greek, Hebrew, Arabic, Armenian and Georgian queries reach the same three-token guard as Latin ones, instead of being read as zero tokens and rejected wholesale.

Combining marks stay inside their word. Marks are not alphanumeric, so treating them as separators cuts abugidas (Devanagari, Thai) and NFD-decomposed text into syllable fragments. अंतर्राष्ट्रीयकरण looked like seven tokens, cleared the three-token guard, and relaxed into a broad OR of one- and two-letter fragments — whose top lexical row then normalizes to 1.0 during hybrid fusion.

That second point comes from the Codex review on this PR, and it was right. My first push fixed only the ASCII gate and pinned the fragmenting as a known limitation, but relaxing a single word into fragments is worse than the silent no-op it replaced — so it is fixed rather than pinned. One word in those scripts is now one token, and the short-query guard rejects it as intended.

Every existing guard is untouched. Short queries, quoted queries, explicit booleans and pure-digit identifiers are still rejected, for Latin and non-Latin alike. The CJK branch runs before this line and is unaffected.

After the change the same corpus produces a live FTS branch and two-sided fusion.

Tests

35 tests in tests/repository/test_search_relaxation.py, 27 of them added here:

• 6 alphabetic scripts relax correctly (Russian, Ukrainian, Greek, Hebrew, Arabic, Armenian)
• 3 keep combining marks with their base character (Devanagari, Thai, Latin)
• 3 keep U+200C/U+200D join controls inside a word (Persian ZWNJ, Devanagari ZWJ conjunct)
• 5 keep word-internal apostrophes inside a word, without shielding a digit from the numeric guard
• 3 confirm one word in those scripts stays one token, so the short-query guard still rejects it
• 3 reject Unicode numeric tokens (Ⅻ, ½, Arabic-Indic digits)
• 4 confirm the existing guards still reject (short query, digit token, quoted, explicit boolean)

Measured over the search-related selection of tests/: 542 to 569 passing, with the same 33 pre-existing environment-dependent failures on the base commit and on this branch — identical test IDs on both sides, they need sqlite-vec / embedding models. ruff check, ruff format --check and pyright are clean on both touched files.

Not included

The second half of the same problem is morphology: SQLite FTS5 is configured with unicode61, which has no stemmer, so маркерного ведра does not match a query for маркерное ведро. That needs a stemming dependency and an index migration — a separate discussion, and I'd be glad to open an issue if there's interest.

[CLAassistant]

All committers have signed the CLA.

[chatgpt-codex-connector]

💡 Codex Review

basic-memory/src/basic_memory/repository/search_query.py

Line 37 in 5a760dc

RELAXATION_WORD_TOKEN_PATTERN = re.compile(r"[^\W_]+", re.UNICODE)

Preserve combining marks when counting query words

When a query uses Devanagari, Thai, vocalized Arabic/Hebrew, or decomposed Unicode text, this pattern splits a single word at every combining mark; for example, अंतर्राष्ट्रीयकरण becomes seven apparent tokens and passes the three-token safety guard. The fallback then ORs one- or two-letter prefixes, so even a one-word query can produce a very broad FTS candidate set whose top lexical match is normalized to 1.0 during hybrid fusion. The new test explicitly pins this fragmenting behavior, but grouping Unicode marks with their base characters would preserve the short-query guard without requiring full language-aware word segmentation.

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[gingeard]

Good catch, and it is now fixed in 5576b5f rather than pinned.

You are right that fragmenting is worse than the no-op it replaced: a single word cleared the three-token guard and relaxed into an OR of one- and two-letter fragments, and in hybrid fusion the top lexical row normalizes to 1.0. Pinning that with a test was the wrong call on my part.

Token counting now lives in relaxation_word_tokens, which builds a token from a run of alphanumeric characters plus any combining marks attached to them, so marks group with their base character without needing full language-aware segmentation. A leading mark cannot open a token, so stray marks do not form fragment-only terms.

अंतर्राष्ट्रीयकरण and การเข้าถึง are now one token each and the short-query guard rejects them; पहुंच कैसे रद्द करें relaxes into four whole words. The two tests that pinned the fragmenting are replaced by six that assert the intended behaviour, including an NFD-decomposed Latin case.

[phernandez]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5576b5fb9c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[phernandez]

The exact-head Codex review found two regressions in the eligibility guards, and both are valid: in-word U+200C/U+200D join controls currently split Persian/Indic words into extra tokens, and Unicode numeric tokens such as Ⅻ or ½ bypass the existing isdigit() identifier guard. Please keep join controls inside an existing token, use the Unicode-wide numeric classification for the guard, and add focused regressions. I’m withdrawing approval until those are addressed.

[gingeard]

Both are valid and both are fixed in 88b2453. Thanks for re-running the review on the exact head — I reproduced each one before changing anything.

Join controls. می‌روم خانه tokenized to ['می', 'روم', 'خانه'], so a two-word query cleared the three-token guard and relaxed. U+200C/U+200D are now read as word-internal, alongside combining marks: the same query is one token short of the guard and returns None. A trailing join control is stripped rather than kept, since a word that ends in one is really a word followed by a separator. Covered by a Persian ZWNJ case and an explicit Devanagari ZWJ conjunct.

Unicode numerics. SPEC Ⅻ design and spec ½ design both relaxed to three terms; Ⅻ is Nl and ½ is No, so isalnum() admits them while isdigit() does not see them. Both guards now use isnumeric(), and both queries return None.

One thing to flag, since it goes past the report: I applied the same classification to the CJK branch. It carried the identical hole — 季度 Ⅻ would have passed its digit guard — and leaving one branch on isdigit() seemed likely to resurface as the same bug. It changes behaviour in code this PR did not otherwise touch, so say the word and I will pull it back out into its own change.

Verification: 30 tests in test_search_relaxation.py, 6 of them new for these two findings. On ASCII input the tokenizer still returns token-for-token what [A-Za-z0-9]+ returned, including client-side state management and foo/bar baz qux. The search-related selection of tests/ goes 542 → 564 passing against the base commit, with the same 33 pre-existing environment-dependent failures on both sides. ruff check, ruff format --check and pyright are clean.

[gingeard]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 88b2453912

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[gingeard]

Valid, and fixed in 401d5d4. Same class as the previous two: п’ять проектів tokenized to ['п', 'ять', 'проектів'], so a two-word query cleared the guard, and the old ASCII path had rejected it outright.

Treating every apostrophe as word-internal would have opened a hole in the numeric guard — 16's is not isnumeric(), so SPEC 16's design would have started relaxing where SPEC 16 design is rejected. So an apostrophe continues a token only between two letters: п’ять and об'єкт stay whole, SPEC 16's design still splits on the digit and still returns None. Leading and trailing apostrophes remain separators, so rock 'n' roll music is unchanged.

One ASCII consequence worth your call. Contractions now form one token: don't touch this yields ["don't", "touch"] where the old pattern yielded ["don", "t", "touch"]. Merging can only lower the token count, so no query the guards used to reject can begin relaxing because of it, and both cases #1022 named — client-side state management, foo/bar baz qux — are byte-identical. But it is a real difference in emitted terms for English, so if you would rather keep strict parity there, I can restrict the rule to U+2019 and leave the ASCII apostrophe splitting. That would leave об'єкт broken, which is why I did not choose it.

35 tests now, 5 new for this finding. Search-related selection: 542 → 569 passing against the base commit, same 33 pre-existing environment-dependent failures on both sides. ruff, ruff format and pyright clean.

[gingeard]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 482241b70e

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[gingeard]

Confirmed and fixed in 1d5ed5d. This one was the serious one — thank you for pushing the review down to the renderer.

Reproduced against a real FTS5 table before changing anything:

don't* OR touch*      → sqlite3.OperationalError: fts5: syntax error near "'"
"don't"* OR touch*    → matches
п’ять* OR проектів*    → matches (U+2019 is not FTS5 syntax; only the ASCII apostrophe is)

Keeping apostrophes inside tokens let them reach _relaxed_fts_text, which interpolates each word straight into query syntax. The caller treats a syntax error as an empty result, so the relaxed retry silently contributed nothing — precisely the failure this PR exists to remove, reintroduced for English contractions and Ukrainian words. Before this PR no apostrophe could reach that renderer, so it is mine to fix.

Both renderers now quote a word only when it carries an apostrophe; every other term renders byte-identical, verified by test. Postgres receives the same tokens, so it gets the matching treatment in _relaxed_tsquery_text — with one caveat I would rather state than hide: its escaping is the documented tsquery lexeme form, but I had no live server here, so only the rendered string is asserted, not its execution. Worth a second pair of eyes.

On the test gap you identified: there is now a separate tests/repository/test_search_relaxed_rendering.py that executes the rendered SQLite expression against a real FTS5 table across five scripts, asserts the exact rendered output for both backends, and pins that the unquoted form raises — so removing the quoting fails loudly instead of silently returning nothing.

48 relaxation tests in total. Search-related selection: 542 → 582 passing against the current merge base, same 33 pre-existing environment-dependent failures on both sides. ruff, ruff format and pyright clean.

[gingeard]

@codex review

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. More of your lovely PRs please.

Reviewed commit: 1d5ed5d897

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[gingeard]

@phernandez ready for another look when you have time. Codex is clean on 1d5ed5d ("didn't find any major issues"), and all four findings from the review rounds are addressed:

1. Combining marks split abugidas and NFD text into syllable fragments — marks now stay with their base character.
2. Join controls U+200C/U+200D split Persian and Indic words — now read as word-internal.
3. Unicode numerics Ⅻ and ½ slipped past the identifier guard — both guards classify numbers Unicode-wide.
4. Apostrophes split Ukrainian words, and once kept, reached the renderer as FTS5 syntax and killed the relaxed retry with a parse error — tokens keep them, renderers quote them.

Findings 2–4 were regressions this PR introduced by widening token recognition, so each is covered by tests that fail without the fix. The rendering path now has its own file that executes the generated expression against a real FTS5 table across five scripts.

Three things are your call rather than mine, all flagged in the commits above:

• I aligned the CJK branch to the same numeric classification. It had the identical hole, but it is code this PR did not otherwise need to touch — happy to split it out.
• ASCII contractions now form one token (don't touch this → ["don't", "touch"]). It only ever lowers the token count, so nothing previously rejected can start relaxing, and the #1022 cases are byte-identical — but it is a real difference in emitted terms.
• The Postgres escaping mirrors SQLite and follows the documented tsquery form, but I had no live server here, so only the rendered string is asserted, never executed.

Numbers: 48 relaxation tests, search-related selection 542 → 582 passing against the current merge base, same 33 pre-existing environment-dependent failures on both sides. ruff check, ruff format --check, pyright clean. DCO and CLA green.