Skip to content

fix(qdrant): index the type payload field so recall filters stop 400ing - #223

Open
zackkatz wants to merge 1 commit into
verygoodplugins:mainfrom
zackkatz:fix/qdrant-type-payload-index
Open

fix(qdrant): index the type payload field so recall filters stop 400ing#223
zackkatz wants to merge 1 commit into
verygoodplugins:mainfrom
zackkatz:fix/qdrant-type-payload-index

Conversation

@zackkatz

Copy link
Copy Markdown
Contributor

Every vector search sends Qdrant a filter condition on the type payload field, but that field is never indexed — so Qdrant rejects the search and recall silently falls back to keyword-only.

The bug

_vector_search builds its filter with excluded_types=RECALL_EXCLUDED_TYPES on every call:

query_filter = build_qdrant_tag_filter(
    tag_filters, tag_mode, tag_match,
    excluded_types=RECALL_EXCLUDED_TYPES,
)

RECALL_EXCLUDED_TYPES defaults to a non-empty value:

RECALL_EXCLUDED_TYPES = frozenset(
    t.strip() for t in os.getenv("RECALL_EXCLUDED_TYPES", "MetaPattern").split(",") if t.strip()
)

So the type condition is present by default. But ensure_qdrant_collection only indexes tags and tag_prefixes, and Qdrant refuses to run a filter over an unindexed payload field:

Bad request: Index required but not found for "type" of one of the
following types: [keyword]. Help: Create an index for this key or use
a different filter.

Why it is easy to miss

_vector_search catches broadly and returns []:

except Exception:
    logger.exception("Qdrant search failed")
    return []

Recall keeps returning results, so nothing looks broken — the vector half is just gone. On a live instance the symptoms are:

  • vector_search.matched: false on every recall, every hit match_type: "keyword"
  • vector_count: null and vector_dimensions.collection: null in /health
  • semantically-phrased queries returning unrelated keyword collisions

None of which names the actual cause. The 400 only appears in server logs.

The fix

Add type to the ensured payload indexes, and collapse the duplicated enum/string branches into one loop over a named tuple so the set of filtered fields is stated once:

QDRANT_FILTERED_PAYLOAD_FIELDS = ("tags", "tag_prefixes", "type")

Verification

Reproduced and fixed on a live instance (~25k points, 768d collection, upgraded from a pre-0.16.0 build). Creating the type payload index by hand restored vector_search.matched: true and match_type: "vector" results immediately, with no other change — and /health went from vector_count: null back to vector_count: 24937.

Two regression tests cover both the enum and string-schema branches. Both fail when type is removed from the tuple and pass with it (tamper-checked, not just observed green).

Note: tests/test_vector_size_safety.py has 12 pre-existing failures in a minimal env (AttributeError: module 'automem' has no attribute 'config'); they fail identically on an unmodified checkout. This PR takes that file from 9 passing to 11.

…0ing

`_vector_search` always builds its filter with
`excluded_types=RECALL_EXCLUDED_TYPES`, and that setting defaults to
`MetaPattern` — a non-empty value — so every vector search sends Qdrant a
condition on the `type` payload field. `ensure_qdrant_collection` only ever
indexed `tags` and `tag_prefixes`, and Qdrant rejects an entire search with 400
when a filter references an unindexed field:

    Bad request: Index required but not found for "type" of one of the
    following types: [keyword].

`_vector_search` catches that broadly and returns `[]`, so recall silently
degrades to keyword-only rather than surfacing an error. On a live instance the
symptom is `vector_search.matched: false` on every query, `vector_count: null`
in `/health`, and semantically-phrased queries returning unrelated keyword
collisions — with nothing in the response indicating that vector search is
failing.

Add `type` to the ensured indexes, collapsing the duplicated enum/string
branches into one loop over a named tuple so the set of filtered fields stays
visible in one place.

Verified against a production instance (~25k points): creating the `type`
payload index restored `vector_search.matched: true` and `match_type: "vector"`
results immediately, with no other change.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant