Skip to content

Add FlatFieldVectorsWriter#asKnnVectorValues#16062

Open
ldematte wants to merge 5 commits into
apache:mainfrom
ldematte:flat-field-vectors-writer-knn-vector-values
Open

Add FlatFieldVectorsWriter#asKnnVectorValues#16062
ldematte wants to merge 5 commits into
apache:mainfrom
ldematte:flat-field-vectors-writer-knn-vector-values

Conversation

@ldematte
Copy link
Copy Markdown

@ldematte ldematte commented May 13, 2026

FlatFieldVectorsWriter<T> exposes its accumulated vectors via List<T> getVectors().
This is both too generic and too prescriptive:

  • List<T> binds the return type to specific vector representation (T = float[] / byte[] heap arrays, one vector per list element); if a FlatFieldVectorsWriter implementation is backed by a different data structure (off-heap / paged / memory-mapped storage), it has to materialize a heap array even when the downstream consumer would be happy with a non-materialized view.
  • it does not carry enough information with it (e.g. encoding, dimensions for sub-byte packed types, etc.). In fact, current consumes have to wrap that list into a KnnVectorValues to feed the scoring / graph-building path (see e.g. Lucene99HnswVectorsWriter.FieldWriter).

This PR proposes to move this step directly to FlatFieldVectorsWriter<T>, exposing its vectors also as KnnVectorValues.

Why KnnVectorValues?

KnnVectorValues is the shape current callers (e.g. scorers) actually use. This PR updates Lucene99HnswVectorsWriter to use the new function when it calls FlatVectorsScorer.getRandomVectorScorerSupplier, which already takes KnnVectorValues.
No new Lucene type is needed; an alternative would be to introduce a new Lucene interface (e.g. FlatVectorsView). That would be very similar to KnnVectorValues with minor variations.

Copy link
Copy Markdown
Contributor

@ChrisHegarty ChrisHegarty left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! LGTM. I'm going to label this for milestone 10.5, so a changes entry can be added there for it.

@ChrisHegarty ChrisHegarty added this to the 10.5.0 milestone May 13, 2026
@ldematte ldematte marked this pull request as ready for review May 13, 2026 16:55
Copy link
Copy Markdown
Member

@benwtrent benwtrent left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it seems like a natural improvement to the API, I like it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants