FastEmbed provider never applies query/passage asymmetry, silently breaking asymmetric models (multilingual-e5)

[itchyshin]

Thanks for basic-memory — it is the retrieval backbone of my lab's bilingual (EN/JP) research knowledge base, queried daily by both humans and agents, and the local-first design is exactly why we can run it on real research notes at all.

While tuning multilingual retrieval I found a small but consequential gap in the FastEmbed path, with a fix that looks well-contained. Details and measurements below.

Environment

• basic-memory v0.22.1, sqlite backend
• Embedding provider: FastEmbed (local)
• Corpus: bilingual (English/Japanese) research notes

What happens

Asymmetric-prefix embedding models — notably the multilingual-e5 family, which FastEmbed itself ships (intfloat/multilingual-e5-large) — run off-label on the FastEmbed path: queries and documents are embedded identically, so the model's query/passage distinction is never applied and retrieval quality collapses.

Why

Two code pointers:

1. src/basic_memory/repository/fastembed_provider.py: embed_query() simply calls embed_documents([text]). No query/passage asymmetry is ever applied. FastEmbed exposes query_embed() for exactly this purpose, but it is never called.
2. The config fields semantic_embedding_query_input_type / semantic_embedding_document_input_type exist, but are only honored by the litellm provider (litellm_provider.py) — never by the FastEmbed provider. So configuring them has no effect on the local path.

Measurements

All measured today on real note chunks, through basic-memory's exact code path:

• multilingual-e5-large without query prefix: true-target vs unrelated-junk cosine margin collapses to 0.031 (Japanese query) and 0.016 (English paraphrase query) — effectively no discrimination.
• Why e5 matters: for a bilingual corpus the multilingual-e5 family is essentially the only strong local retriever in FastEmbed's registry. The default bge-small-en-v1.5 is English-only — a Japanese query scores 0.52 against its true target vs 0.74 against unrelated Japanese text (it encodes the language, not the content) — and the paraphrase-multilingual family truncates at 128 tokens, below typical chunk length.

Suggested fix (deliberately minimal)

Either of these unlocks the e5 family with no schema or API changes:

1. Have FastEmbedEmbeddingProvider.embed_query() call the underlying model's query_embed() when available, falling back to embed(); and/or
2. Honor the existing *_input_type config fields as literal prefixes ("query: " / "passage: ") on the FastEmbed path, mirroring what the litellm provider already does.

Offer to help

We maintain a bilingual golden set (100 queries, EN+JP) over this corpus. Happy to test a branch against it and report recall numbers before/after — just point me at it. And thanks again; this tool carries a lot of weight for us, which is the only reason a 0.031 margin ever got noticed.

[Aryan-Pardeshi]

Thanks for the detailed report and the cosine-margin measurements — they made the root cause easy to confirm.

I've opened #1265, which routes embed_query through FastEmbed's query_embed() and shares the existing L2 normalization + dimension checks with the document path. Stored document vectors are unaffected, so no re-index is needed. On the config fields: semantic_embedding_document_input_type/query_input_type are LiteLLM request parameters, and FastEmbed's equivalent of the role split is query_embed vs embed itself, so the fix doesn't need extra config plumbing — happy to revisit if maintainers would rather expose it explicitly.

[phernandez]

I verified the proposed fix against the repository's FastEmbed 0.8.0. For intfloat/multilingual-e5-large, FastEmbed selects PooledEmbedding; its inherited query_embed() delegates directly to embed(), so PR #1265 does not yet apply the required query: / passage: prefixes and its test double masks the reported behavior. I requested changes on the PR to use the existing prefixing/provider-identity path, require document reindexing, and add an E5-shaped regression. Keeping this in v0.23 while that is addressed.