fix(qdrant): index the type payload field so recall filters stop 400ing - #223
Open
zackkatz wants to merge 1 commit into
Open
fix(qdrant): index the type payload field so recall filters stop 400ing#223zackkatz wants to merge 1 commit into
type payload field so recall filters stop 400ing#223zackkatz wants to merge 1 commit into
Conversation
…0ing
`_vector_search` always builds its filter with
`excluded_types=RECALL_EXCLUDED_TYPES`, and that setting defaults to
`MetaPattern` — a non-empty value — so every vector search sends Qdrant a
condition on the `type` payload field. `ensure_qdrant_collection` only ever
indexed `tags` and `tag_prefixes`, and Qdrant rejects an entire search with 400
when a filter references an unindexed field:
Bad request: Index required but not found for "type" of one of the
following types: [keyword].
`_vector_search` catches that broadly and returns `[]`, so recall silently
degrades to keyword-only rather than surfacing an error. On a live instance the
symptom is `vector_search.matched: false` on every query, `vector_count: null`
in `/health`, and semantically-phrased queries returning unrelated keyword
collisions — with nothing in the response indicating that vector search is
failing.
Add `type` to the ensured indexes, collapsing the duplicated enum/string
branches into one loop over a named tuple so the set of filtered fields stays
visible in one place.
Verified against a production instance (~25k points): creating the `type`
payload index restored `vector_search.matched: true` and `match_type: "vector"`
results immediately, with no other change.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Every vector search sends Qdrant a filter condition on the
typepayload field, but that field is never indexed — so Qdrant rejects the search and recall silently falls back to keyword-only.The bug
_vector_searchbuilds its filter withexcluded_types=RECALL_EXCLUDED_TYPESon every call:RECALL_EXCLUDED_TYPESdefaults to a non-empty value:So the
typecondition is present by default. Butensure_qdrant_collectiononly indexestagsandtag_prefixes, and Qdrant refuses to run a filter over an unindexed payload field:Why it is easy to miss
_vector_searchcatches broadly and returns[]:Recall keeps returning results, so nothing looks broken — the vector half is just gone. On a live instance the symptoms are:
vector_search.matched: falseon every recall, every hitmatch_type: "keyword"vector_count: nullandvector_dimensions.collection: nullin/healthNone of which names the actual cause. The 400 only appears in server logs.
The fix
Add
typeto the ensured payload indexes, and collapse the duplicated enum/string branches into one loop over a named tuple so the set of filtered fields is stated once:Verification
Reproduced and fixed on a live instance (~25k points, 768d collection, upgraded from a pre-0.16.0 build). Creating the
typepayload index by hand restoredvector_search.matched: trueandmatch_type: "vector"results immediately, with no other change — and/healthwent fromvector_count: nullback tovector_count: 24937.Two regression tests cover both the enum and string-schema branches. Both fail when
typeis removed from the tuple and pass with it (tamper-checked, not just observed green).Note:
tests/test_vector_size_safety.pyhas 12 pre-existing failures in a minimal env (AttributeError: module 'automem' has no attribute 'config'); they fail identically on an unmodified checkout. This PR takes that file from 9 passing to 11.