Make the embed endpoint optional so lexical-only search works

[TheGreatAxios]

What

The engine already degrades to lexical-only cleanly at runtime — hybridSearch falls back to Postgres full-text search when dense retrieval fails, and embedChunks reports a clientError instead of throwing so capture stays durable. But loadMemoryConfig() hard-required EMBED_BASE_URL/EMBED_MODEL, so a host with a working pgvector Postgres and no embedding account couldn't construct the plane at all.

EngineConfig.embed / MemoryConfig are now optional end to end:

• loadMemoryConfig() builds embed only when both EMBED_BASE_URL and EMBED_MODEL are set (trimmed, matching requireEnv's existing behavior — a whitespace-only value is treated as unset); leaving both unset opts into lexical-only. Setting exactly one throws — that's a real misconfiguration, not an opt-in.
• hybridSearch (src/services/search.ts) skips the dense channel entirely when there's no embed client config — no HTTP call is dispatched — instead of attempting-and-failing on every query.
• captureDocument still writes documents when embed is unconfigured: chunks are stored, vectors are not, and the embed-model registry (ensureEmbedModel/activateEmbedModel) is never reached.
• Rerank behavior is unchanged — it was already fully optional and is the model this followed.

Degrade-flag decision

Reused the existing flags rather than inventing one, per the two states they already model:

• dense_unavailable — existing meaning, "dense contributed nothing this call" (a configured endpoint that's down, or a tenant with no active model yet).
• lexical_only — was already reserved in SearchDegradedReasonSchema (src/core/schemas/search.ts) but never produced by any code path; it now means the structural "no embed endpoint configured on this engine at all," distinct from a transient failure.

search reports degraded: ["dense_unavailable", "lexical_only"] on every call when unconfigured. lexical_only was added to DegradeFlag (hybrid-search.ts) and DEGRADE_FLAG_SET (degrade-metrics.ts, compile-enforced via satisfies). One side effect worth flagging: for a deliberately lexical-only host, this flag sits at a permanent ~100% windowed rate and will escalate to log.error in degrade-metrics.ts and stay there — accurate ("running degraded"), not a bug, but noisier than the module's other flags. Being filed as its own follow-up (config-state flags vs. failure flags need separating in that module), out of scope here.

add mirrors this shape. add's degraded was a bare boolean, which made it impossible for an integrator to write one "is this response degraded" check across both verbs. It's now CaptureDegradedReason[] ("embed_unavailable" / "lexical_only"), threaded through CaptureResult → DocumentStoreAddResult → MemoryAddResult → the /add HTTP route. embed_unavailable is capture's counterpart to dense_unavailable (the embed pass ran and failed — client error, timeout, or a rejected/dims-mismatched chunk); paired with lexical_only specifically when there's no endpoint configured at all. CaptureDegradedReason lives in core/embed-worker.ts (not services/capture.ts) so ports/types.ts can reference it the same way it already references DegradeFlag for DocumentStoreSearchResult.

Final wire shape, for a settings page coding against both:

// search
{ hits: [...], evidence: "weak", degraded?: DegradeFlag[] }        // e.g. ["dense_unavailable", "lexical_only"]
// add
{ documentId, versionId, degraded?: CaptureDegradedReason[] }       // e.g. ["embed_unavailable", "lexical_only"]

Both are arrays, both omitted when healthy — one Array.isArray(res.degraded) && res.degraded.length > 0 check works for either.

Migration note: dense_unavailable now also fires permanently for a deliberately lexical-only deployment, and add's degraded is no longer a boolean — a host with an existing alert/check on either should check for lexical_only alongside dense_unavailable, and switch any degraded === true check to an array check. Noted in CHANGELOG.md.

Capability discoverability

A host previously could only learn embeddings were unconfigured by running a search and inspecting degrade flags — meaning a lexical-only plane could run for months with nobody noticing recall was limited. Memory.capabilities.embeddingsConfigured (on the handle createMemory returns) now answers this at construction time, no query needed:

const memory = createMemory({ config });
memory.capabilities.embeddingsConfigured; // false for a lexical-only engine

Backed by an optional DocumentStore.capabilities (ports/types.ts) so a vendor store can report its own; a custom store that omits it defaults to embeddingsConfigured: true (the pre-CL-6287 assumption, since this SDK can't otherwise introspect a store it doesn't own). Both MemoryCapabilities and DocumentStoreCapabilities are exported from index.ts.

Vocabulary

Settled on "embed endpoint" (what the config actually is) over "embedding account" everywhere it appeared (config.ts, engine-client-config.ts, mount-config.ts, IMPLEMENTATION.md, capture.test.ts, CHANGELOG.md) — hybrid-search.ts/search.ts/transform.ts already used it consistently.

Not renaming lexical_only → dense_unconfigured: that string was already reserved in SearchDegradedReasonSchema before this work, so it isn't ours to rename in passing, even though the naming argument (name a mode, not a failure) is reasonable.

Opting into lexical-only

Unset both EMBED_BASE_URL and EMBED_MODEL (or omit them entirely). No other config change needed — add and lexical search work as before, RERANK_BASE_URL is independent and can still be set.

The replay/backfill pipeline (runTransform, promoteGeneration in services/transform.ts) still requires an embed endpoint, since re-deriving a corpus is inherently a re-embedding operation — it fails loudly and early with a clear message if run against an engine with none configured, rather than throwing deep inside buildEmbedClientConfig.

Backfill (explicitly out of scope)

Re-embedding documents captured while lexical-only, once an embed endpoint is later added, is not implemented here (per the issue). A follow-up would need:

• A scan for chunks with no row in any per-model embedding table for their tenant (the embed-model registry already tracks per-tenant active/ready tables, so this is a LEFT JOIN / anti-join against that).
• A way to invoke embedChunks/activateEmbedModel against already-captured, already-chunked rows without re-running adaptAndPlan/deriveVersionInTransaction (capture.ts's embed pass is currently only reachable from the post-transaction hook of captureDocument/deriveFromRawCapture).
• Batching/rate-limiting for a potentially large backlog, and a way to report progress (the transform_run table's shape may be reusable, but it's currently scoped to full re-derivation, not a vectors-only backfill).
• A decision on whether backfill activates the model as "live" (as capture does) or requires an explicit promote step (as replay does) — the two existing precedents disagree on this.

Testing

bun run typecheck and bun test ./src: 401 → 417 tests, all passing. Coverage added across the review rounds:

• mount-config.test.ts: both-set-or-both-unset validation, whitespace-only EMBED_BASE_URL treated as unset.
• capture.test.ts: toEmbedClientConfig(undefined) returns undefined; embedInsertedChunksWithConfig returns [] for no chunks and ["embed_unavailable", "lexical_only"] when unconfigured, without touching the embed-model registry.
• search.test.ts: hybridSearch against a fake Db/RawSql with embed unconfigured — returns lexical results, reports both degrade flags, and makes zero embed HTTP calls / zero embed-model-registry access.
• memory.test.ts: capabilities.embeddingsConfigured is true/false for engine construction with/without embed, and defaults true for a custom store that doesn't report it.
• transform.test.ts: runTransform and promoteGeneration's embed-absent guards, against a fake Db — both fail loudly with a clear message and never reach per-row derivation or the embed-model registry.

CL-6287