feat(pgvector): support pgvector halfvec indexes

[emerzon]

Summary

Adds PostgreSQL pgvector halfvec support for Metis vector stores so large embedding dimensions, such as 3072-dimensional embeddings, can still use HNSW indexes.

Root Cause

Metis already allows metis_engine.embed_dim: 3072, but its PostgreSQL backend always created normal vector stores and forwarded HNSW settings using normal vector operator classes such as vector_cosine_ops. pgvector normal-vector HNSW indexes cannot be created above the supported dimension limit, so large embeddings could fall back to unindexed vector search or log HNSW setup failures.

Changes

• Adds metis_engine.pgvector_use_halfvec with auto, true, and false behavior.
• Defaults auto to halfvec for embed_dim > 2000 and full-precision vector storage for smaller embeddings.
• Passes use_halfvec through to both code and docs PGVectorStore instances.
• Rewrites configured HNSW operator classes from vector_*_ops to matching halfvec_*_ops when halfvec is enabled.
• Preserves hnsw_kwargs: null as disabled HNSW instead of converting it to an empty dict.
• Documents the setting in README, provider docs, and packaged metis.yaml.

Validation

• uv run --extra lint ruff format --check .
• uv run --extra lint ruff check .
• uv run pytest
• uv run --extra postgres pytest --postgres tests/test_pg_backend_real_integration.py tests/test_pg_backend_mocked_unit.py
• uv run --extra postgres pytest --postgres

The full Postgres-enabled suite passed against a local pgvector/pgvector:pg16 container after creating the expected metis_user role, metis_db database, and vector extension.

[Copilot]

Pull request overview

Adds pgvector halfvec support in Metis’ PostgreSQL vector store to keep HNSW indexing available for high-dimensional embeddings (e.g., 3072), including config, runtime wiring, HNSW operator-class rewriting, and documentation.

Changes:

• Introduces metis_engine.pgvector_use_halfvec (auto/true/false) with auto-enable behavior for large embed_dim.
• Passes use_halfvec through PG backend construction and into PGVectorStore.from_params, rewriting hnsw_dist_method operator classes when needed.
• Updates tests and docs, and preserves hnsw_kwargs: null semantics as “disable HNSW”.

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
tests/test_pg_backend_mocked_unit.py	Adds mocked tests for use_halfvec passthrough, HNSW dist method rewrite, and hnsw_kwargs=None preservation.
tests/test_configuration.py	Adds config parsing tests for pgvector_use_halfvec and auto behavior based on embed_dim.
src/metis/vector_store/pgvector_store.py	Adds halfvec operator-class rewrite logic and propagates use_halfvec to PGVectorStore.
src/metis/metis.yaml	Documents and defaults pgvector_use_halfvec: auto in packaged config.
src/metis/configuration.py	Adds pgvector_use_halfvec_setting and sets runtime pgvector_use_halfvec based on config + embed_dim.
src/metis/cli/utils.py	Plumbs runtime pgvector_use_halfvec into PG backend construction (PGVectorStoreImpl).
README.md	Documents automatic halfvec selection and override knob.
docs/providers/vllm.md	Documents pgvector_use_halfvec in provider config example.
docs/providers/gemini.md	Documents pgvector_use_halfvec and rationale for 3072-dim embeddings.
docs/providers/embedding-provider.md	Adds explanation of pgvector_use_halfvec behavior and overrides.
docs/providers/anthropic.md	Documents pgvector_use_halfvec and rationale for 3072-dim embeddings.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.