fix: truncate embeddings to one model window

[sjvans]

Summary

• embed only the first tokenizer/model input window
• remove averaging across independently embedded chunks
• keep [CLS] and [SEP] within the model's input limit
• document that long documents should be split and stored as multiple vectors above VECTOR_EMBEDDING

This addresses the averaging concern raised in #53: averaging sentence embeddings does not produce the embedding of the combined text and becomes increasingly misleading for smaller model windows.

Validation

• tokenizer and model-download unit tests pass
• real MiniLM test confirms that appending text beyond the first input window does not change the embedding
• ESLint, Prettier, and git diff --check pass

Stack

Targets AISQLiteService so merging this PR updates #53 directly.

[hyperspace-pr-bot]

Summary

The following content is AI-generated and provides a summary of the pull request:

---

Fix: Truncate VECTOR_EMBEDDING input to one model window

Category: Bug Fix

This PR removes the chunked-averaging approach from the VECTOR_EMBEDDING function in AISQLiteService, replacing it with single-window truncation. Averaging embeddings from independently processed chunks does not produce a semantically meaningful combined embedding and becomes increasingly misleading for smaller model windows.

What changed

• lib/vector_embedding/embedding.js: Refactored wordPieceTokenizer to stop at the model's maxLength boundary (keeping [CLS] and [SEP] within the limit) and return a single { tokens, ids } object instead of an array of chunks. Removed processChunkedEmbeddings (which averaged multi-chunk embeddings) and replaced it with a simpler processEmbedding that processes exactly one model input window.
• tests/vector-unit.test.js: Updated unit tests to reflect the new single-object return value and added a test asserting that output is identical for inputs that differ only beyond the first window.
• tests/vector.test.js: Added an integration test confirming that text appended beyond the first model input window does not change the resulting embedding.
• README.md: Documents that VECTOR_EMBEDDING embeds one model input window and advises splitting long documents into chunks stored as separate vectors.
• CHANGELOG.md: Added changelog entry noting the single-window embedding behavior.

Why

Averaging sentence embeddings across chunks does not produce the embedding of the combined text. Truncating to one window gives predictable, consistent results and aligns with standard practice — applications that need long-document retrieval should split documents and store one vector per chunk.

Related: #53

Have you...

• Added relevant entry to the change log?

---

• 🔄 Regenerate and Update Summary
• ✏️ Insert as PR Description (deletes this comment)
• 🗑️ Delete comment

PR Bot Information

Version: 1.29.49

• GithubContextProvider: feat: add AI-enabled SQLite service #53
• Correlation ID: 1ae64130-a171-11f1-9854-594348a7ae18
• LLM: anthropic--claude-4.6-sonnet
• Summary Prompt: Default Prompt
• Event Trigger: pull_request.opened
• File Content Strategy: Full file content
• Output Template: Repository PR Template

[hyperspace-pr-bot]

The PR cleanly removes the multi-chunk averaging logic and replaces it with a single-window truncation approach, and the documentation and tests are updated consistently. The one substantive issue found: the inner break tokenize fires mid-way through a pre-token's word-piece sub-tokens, which can leave a dangling prefix sub-token (e.g. 'un' without '##usual') as the last content token before [SEP]. This produces a sequence the model was not trained on, and could subtly skew embeddings near the truncation boundary.

PR Bot Information

Version: 1.29.49

• Correlation ID: 1ae64130-a171-11f1-9854-594348a7ae18
• File Content Strategy: Full file content
• LLM: anthropic--claude-4.6-sonnet
• Event Trigger: pull_request.opened