Skip to content

IMPLEMENTATION.md

Latest commit

 

History

History
751 lines (651 loc) · 44.1 KB

IMPLEMENTATION.md

File metadata and controls

751 lines (651 loc) · 44.1 KB
Raw

Corbits Memory — Implementation Reference

This is the detailed implementation reference: concrete files, tables, functions, and wire shapes. For the "why standalone" / boundaries story, read ARCHITECTURE.md first — this doc complements it, it does not repeat it.

Repo layout

src/
  index.ts                # createMemory / registerMemoryRoutes + distiller re-exports

  mount-config.ts         # MemoryConfig + loadMemoryConfig() — the mount config
  config.ts               # EngineConfig — the core vector-plane config (db + embed + rerank)
  memory.ts               # createMemory — add/search/list against store or pgvector
  grant-tags.ts           # resolveAccessTags + canAccessDocument (host grants)

  log.ts                  # getLogger(["memory"]) from @intx/log
  migrations.ts           # runMemoryMigrations(url)
  ports/                  # DocumentStore / SourceProvider + fakes
  routes/                 # the mounted routes
    mount.ts              # registerMemoryRoutes (HTTP)

    deps.ts               # RouteDeps, caller(c) (context identity), grantGuard
    add.ts, search.ts, list.ts, feed.ts
  tools/                  # Interchange defineTool factories (HTTP clients)
    add.ts, search.ts, list.ts, feed.ts, client.ts, install.ts
  distiller/              # Resident distiller (CL-5869) — workflow + tick helpers
    index.ts              # createResidentDistiller, runDistillTick, buildDistilledClaim
    workflow.ts           # defineWorkflow + defineAgent with memory tools
    tick.ts               # imperative distill tick (host injects distill())
    claim.ts              # pure claim body / gate / cursor helpers
  db/
    schema.ts             # Drizzle table defs (memory.* schema)
    client.ts             # createDb(config) -> { db (drizzle), sql (raw postgres-js) }
  services/
    capture.ts            # captureDocument, deriveFromRawCapture — the write path
    search.ts             # hybridSearch and every retrieval-candidate query
    timeline.ts           # listTimelineEvents — durable recent docs + grant-tag filter
    transform.ts          # transform_config CRUD + runTransform (replay)
    feed.ts               # capture feed cursor pull (CL-5868)
    retention.ts          # deprecate / tombstone / sweep ephemeral (CL-5871)
    share-grants.ts       # peer grant materialization on share (CL-5873)
  core/                   # framework-agnostic (chunking, embed/rerank, merge, schemas)
# DocumentStore adapters / tools live as sibling packages (not in this tree):
#   @corbits/mem0-memory-adapter         → github.com/corbitsdev/corbits-mem0-memory-adapter
#   @corbits/supermemory-memory-adapter  → github.com/corbitsdev/corbits-supermemory-memory-adapter
#   @corbits/linear-tools                → github.com/corbitsdev/corbits-linear-tools
migrations/               # pgvector schema, applied in filename order by scripts/db-setup.ts
scripts/db-setup.ts       # idempotent migration runner, tracked in `_migrations`
compose.yml               # pgvector + Ollama + reranker for local dev

The SDK has no server and no process entrypoint. createMemory takes

the host's Hono<TenantEnv> app plus { config, grants? } and mounts the routes; each reads identity from the context (caller(c)) and guards via grantGuard. Services take { db, sql, config } explicitly (no module-level singletons except the logger). Nothing has import-time side effects, so unit tests exercise the routes and services directly without a listening server.

Mounting does not verify the FTS language against the database at boot — see the memory_chunk section below. A host is expected to either run runMemoryMigrations itself (which verifies) or wire its own readiness probe to call verifyFtsLanguage; without one of those, a language mismatch surfaces as a runtime failure on the plane's first query, not at mount time.

Config

There are two config types, both in the SDK:

  • EngineConfig (src/config.ts) — the core vector-plane config the DB client and capture/search/transform services consume: databaseUrl, dbPoolMax, embed, rerank.

  • MemoryConfig (src/mount-config.ts) — what createMemory

    takes: just { memory: EngineConfig }. loadMemoryConfig() builds one from the environment; hosts may also construct it programmatically. Auth, tenancy, and grants are the host's — none of that is config here.

loadMemoryConfig() uses the same fail-loud helpers: requireEnv(name) throws if unset/empty, optionalEnv(name) returns undefined, intEnv(name, fallback) parses a positive integer or throws.

Var Required? Default Notes
DATABASE_URL yes — the engine's own pgvector Postgres
DB_POOL_MAX no 8 postgres-js pool size
FTS_LANGUAGE no english text search config for the lexical channel; fixed into the generated column at migration time — changing it later requires rebuilding the column (recipe below), and runMemoryMigrations fails loudly if config and column disagree. Unqualified pg_catalog config names only — a schema-qualified config (myschema.mycfg) is rejected explicitly, both when configuring and when read back from an already-migrated column.
EMBED_BASE_URL no — embed endpoint root, no path suffix; absent (with EMBED_MODEL also absent) => lexical-only, see below
EMBED_MODEL no — model id/name passed to the embed endpoint; must be set together with EMBED_BASE_URL (both or neither — one without the other throws)
EMBED_API_STYLE no "openai" "openai" | "tei" | "ollama"
EMBED_API_KEY no undefined forwarded as Authorization: Bearer <key>
RERANK_BASE_URL no undefined absent => search degrades to fusion-only
RERANK_MODEL no undefined defaults to bge-reranker-v2-m3 in the client
RERANK_API_KEY no undefined forwarded as Bearer token to the rerank endpoint

Lexical-only mode (CL-6287). EngineConfig.embed is optional — leave both EMBED_BASE_URL/EMBED_MODEL unset and the engine still constructs and serves add + lexical search against a pgvector Postgres with no embed endpoint configured. Dense retrieval is skipped rather than attempted (no doomed HTTP call on every query), add still captures documents (chunks stored, no vectors), and both verbs report a degraded reason array — never a bare boolean, so a host can write one "is this response degraded" check across both: search reports degraded: ["dense_unavailable", "lexical_only"]; add reports degraded: ["embed_unavailable", "lexical_only"] (or ["embed_unavailable"] alone when the endpoint IS configured but a specific embed pass failed — a client error, timeout, or rejected chunk). The embed-model registry (ensureEmbedModel/activateEmbedModel) is never reached in this mode.

Discoverability. A host does not have to run a search to learn recall is limited: memory.capabilities.embeddingsConfigured (on the Memory handle createMemory returns) is false for a lexical-only engine, true otherwise — known at construction, no query needed. A custom documentStore that doesn't report its own capabilities defaults to true (this SDK cannot introspect a vendor store it doesn't own); see DocumentStoreCapabilities (ports/types.ts) for how a vendor store opts in.

The replay/backfill pipeline (runTransform, promoteGeneration in services/transform.ts) still requires an embed endpoint — re-deriving a corpus is inherently a re-embedding operation — and fails loudly if run against an engine with none configured; re-embedding documents captured while lexical-only, once an endpoint is later added, is an open follow-up (not implemented).

The engine's EngineConfig.rerank carries no apiStyle field of its own; search.ts's toRerankClientConfig hardcodes apiStyle: "tei" when building the client config, i.e. the engine currently only wires a TEI-compatible reranker via env (Cohere/Voyage rerank backends are reachable only through a per-transform_config rerank.apiStyle, not through top-level env).

Model endpoints are trusted URLs — no SSRF guard, no self-host flag. Every embed/rerank endpoint the engine calls — its own EngineConfig.embed/ EngineConfig.rerank (the capture embed pass, the dense-search query embed, the embed-model-registry probe/activation, and toRerankClientConfig) and a transform_config's replay embed override (buildEmbedClientConfig in services/transform.ts) — is treated as a trusted URL, exactly like DATABASE_URL. There is no private-IP / SSRF filtering and no allowSelfHost knob anywhere: a self-hosted endpoint on localhost or a private IP is just a URL, indistinguishable from a managed provider. Model/replay endpoints are configured by the operator (env) or named by a trusted caller in a transform_config, never by an unauthenticated request — so operators who need egress control front the endpoints with an allowlisting proxy.

Data model (src/db/schema.ts + migrations/*.sql)

All tables are Drizzle-defined in db/schema.ts, DDL'd in migrations/ (applied by scripts/db-setup.ts, tracked in a _migrations ledger table so re-running is a no-op). No memory table has a foreign key into any control-plane table — tenant_id/principal_id/source refs are plain text.

memory_document

The stable logical row for a captured source. Unique on (tenant_id, adapter, external_ref) — this triple is the dedupe/identity key every capture upserts against. Document access is grant tags: access_tags text[] (resource strings in grant-pattern space; see docs/AUTHZ-DOCUMENT-ACCESS.md). attributes is a flat jsonb bag of scalars. last_seen_at bumps on every re-capture, even a content-hash NOOP.

memory_version

The versioned body of a document. version is a monotonic integer scoped to (document_id, generation) — not globally per-document — per the memory_version_document_generation_version_uniq unique index (baseline schema). status tracks 'active'|'superseded'|'deprecated'|'archived'|'tombstoned'; only one active row exists per (document_id, generation) at a time (the capture path enforces this by flipping the prior active row to superseded before inserting a new one). content_hash is the NOOP-check key. Attribution columns: created_by_principal_id, created_by_kind ('human'|'agent'|'system'|'adapter'), generator_agent_id. Authority columns (authority, actor_count, has_social_signal, source_class) are a snapshot computed once at capture time (computeAuthority, never recomputed retroactively). raw_capture_id points at the immutable source row this version was derived from. generation (default 'live') is the replay-generation tag: the normal add path always writes 'live'; a replay (runTransform) writes its own transform_run.id instead, so a replayed corpus's versions never collide with, or even become visible alongside, the live ones unless a caller explicitly searches that generation.

memory_chunk

An ordered slice of a version's text, keyed by (version_id, ordinal) (unique). Carries a generated-always text_fts tsvector column (GIN-indexed) that powers the lexical search channel — this is the only place FTS is computed; no separate FTS table exists. Its language comes from FTS_LANGUAGE at migration time; the query side binds the same configured language as a regconfig parameter. The invariant is verified twice, both read-only against the catalog: runMemoryMigrations checks after applying (the deploy step), and the memory plane runs the same check once, memoized, before its first query (the serving path) — so a mismatch or unmigrated schema fails loudly on first use regardless of who ran the migrations. The serving-path check only runs when something actually calls it — search()/capture() invoke it lazily and memoize the result, but nothing forces that first call to happen at boot. A host that mounts the engine without running runMemoryMigrations itself and without wiring a readiness probe will not learn about a language mismatch until the first real query or capture fails — not at startup. Hosts that want a boot-time guarantee must call the exported verifyFtsLanguage from their own readiness probe; it is not optional belt-and-suspenders, it is the only way to get a boot-time check if this SDK instance isn't the one that migrated. Chunks are never reused across versions — every new version gets a fresh full insert of its own chunks.

Changing FTS_LANGUAGE on an already-migrated database (the mismatch verifyFtsLanguage throws on) requires rebuilding the generated column — runMemoryMigrations only applies new files and will not retroactively alter an existing one. One-time recipe (verified against a live postgres:16 instance):

BEGIN;
DROP INDEX IF EXISTS memory_chunk_text_fts_idx;
ALTER TABLE memory_chunk DROP COLUMN text_fts;
ALTER TABLE memory_chunk ADD COLUMN text_fts tsvector
  GENERATED ALWAYS AS (to_tsvector('<new_language>', "text")) STORED;
COMMIT;

-- Separate statement/connection — CREATE INDEX CONCURRENTLY is rejected
-- inside any transaction block, unconditionally, since Postgres 8.2. It
-- cannot be combined with the BEGIN/COMMIT block above.
CREATE INDEX CONCURRENTLY memory_chunk_text_fts_idx ON memory_chunk USING gin (text_fts);

Both ALTER TABLE statements take an ACCESS EXCLUSIVE lock and force a full table rewrite (dropping then re-adding a STORED generated column always rewrites) — plan for a stall on memory_chunk for the duration on a populated database; run in a maintenance window. Only unqualified pg_catalog config names are supported; a schema-qualified config on this column is rejected explicitly by verifyFtsLanguage (with this same recipe in the error) rather than silently mis-parsed.

memory_entity / memory_edge

Lightweight graph rows. memory_entity has no unique constraint; dedupe on (tenant_id, kind, identifiers) is done in application code (upsertEntity in capture.ts, an exact-match linear scan per kind). Same for memory_edge (dedupe on the full (tenant_id, rel, from, to) tuple, upsertEdge). rel is constrained (DB CHECK + arktype, single source of truth in src/core/enums.ts) to mentions|about|authored_by|involves|part_of|derived_from|supports|contradicts|supersedes; from_type/to_type stored in the DB are document|version|chunk|entity. Adapter-facing edge hints may also use native as a planning-time principal ref; capture resolves it to an entity row (kind=principal) before insert. A lockstep test asserts the migration CHECK sets match the TS constants.

Provenance and lineage (claim-bearing substrate)

Two axes on memory.version, orthogonal to ranking priors:

Column / field Values Meaning
provenance stated | inferred | unknown How content was obtained. Capture defaults to stated; distilled claims write inferred. Existing rows default unknown.
source_class (lineage) native | imported | derived Data lineage. Adapters write native (or imported for bulk import); distilled claims write derived via AdaptedDocument.lineageClass.

Ranking priors (AdaptedDocument.sourceClass: native|thread|channel|call|record) feed computeAuthority only and are not written to the version source_class column — that was a latent CHECK violation fixed when the axes were split.

A derived claim is a normal version with provenance: inferred, lineageClass: derived, and a derived_from edge to the source version (or document). Core never runs inference; it only accepts the shape.

Temporal model

See docs/TEMPORAL.md. On memory.version:

Column Role
occurred_at Effective time the content refers to
ingested_at When the plane learned it (no separate asserted_at)
temporal_class event | deadline | state | lesson — ranking prior
valid_from / valid_until Optional validity window

Search multiplies fused scores by temporalRecencyMultiplier (class-aware). Timeline and search both filter generation (default live) so replay rows never leak into live views.

memory_embed_model

Per-tenant registry of embed models and their discovered dimensionality (discoverModelDims/probeEmbedDims — dims are never hard-coded, always probed live against the endpoint). Unique on (tenant_id, model_key), where model_key is sha256(baseUrl|modelId).slice(0,16) (computeModelKey).

Two write paths (CL-5872):

  • ensureEmbedModel — upserts the registry row with status = 'ready', creates the per-model table + indexes. Never flips the tenant's active dense table. Used by staged runTransform when the config's embed model differs from live.
  • activateEmbedModel — ensure then UPDATE … SET status = 'active'. Used only by the live capture path and by promoteGeneration when cutover should make the generation's embed model serve live dense search.

"Active" for live dense search is the most-recently-updated_at row with status = 'active' (resolveActiveEmbedTable). Replay dense search uses resolveEmbedTableByModelKey against the owning transform config's embed model_key — ready or active — so staged generations never steal live.

Optional archived_live_generation and archived_live_model_key on transform_run record which generation and dense model held live rows before promote (for demote/rollback of both corpus and dense search).

Dynamic per-model vector tables: memory_embedding_<key>

Not in db/schema.ts (no fixed shape — dimensionality varies by model) and not in any migration file. Created at runtime by activateEmbedModel (embed-model-registry.ts) the first time a given (baseUrl, modelId) pair is used:

CREATE TABLE IF NOT EXISTS memory_embedding_<key> (
  chunk_id text PRIMARY KEY,
  tenant_id text NOT NULL,
  embedding vector(<dims>),
  CONSTRAINT memory_embedding_<key>_chunk_fk
    FOREIGN KEY (chunk_id) REFERENCES memory_chunk (id) ON DELETE CASCADE
)

plus a (tenant_id, chunk_id) index (created on every activation, so it retrofits onto older tables). The FK completes the hard-delete cascade chain document -> version -> chunk -> embedding. CREATE TABLE IF NOT EXISTS cannot add the FK to a table created before it existed; that gap is accepted (no populated pre-FK installs exist). If hard deletes are ever introduced against such a table, run once, per embedding table:

ALTER TABLE memory_embedding_<key>
  ADD CONSTRAINT memory_embedding_<key>_chunk_fk
  FOREIGN KEY (chunk_id) REFERENCES memory_chunk (id) ON DELETE CASCADE;

plus an HNSW index: vector_cosine_ops up to 2000 dims (falling back to ivfflat if the Postgres/pgvector build lacks the hnsw access method), or a halfvec expression index (halfvec_cosine_ops, pgvector >= 0.7) for 2001–4000 dims — pgvector's vector-typed indexes cap at 2000 dims. Models above 4000 dims are rejected at activation (MAX_EMBED_DIMS) since no index type could serve them. MAX_EMBED_DIMS is now pinned to HALFVEC_INDEX_MAX_DIMS (4000, pgvector's halfvec index cap), narrowed down from the previous 4096 — a model in the 4001–4096 range that used to activate and only fail later (no index type covering it) now fails loudly at activation instead. The dense query's ORDER BY is generated by cosineDistanceExpr from the same module so it always matches the indexed expression. The table name is validated against EMBED_TABLE_NAME_PATTERN (/^"memory"\."embedding_[a-f0-9]{16}"$/) both when computed and again every time it's read back from memory_embed_model, before ever being string-interpolated into raw SQL — this is the only place in the codebase a computed identifier is spliced into DDL/DML.

raw_capture

The immutable, append-only substrate (the raw-capture layer). Stores the exact add/ingest request payload (adapter, occurred_at, document) as JSON in raw_text (there's also a raw_bytes bytea column for non-textual payloads, currently unused by any write path — everything captured today is JSON). Deduped on (tenant_id, source_hash), where source_hash is sha256(stableStringify({adapter, occurredAt, document})) — a byte-identical recapture reuses the existing row (insertOrReuseRawCapture) rather than inserting a duplicate; the table never has rows updated or deleted by ingestion.

transform_config

A named, versioned recipe of derivation + retrieval-tuning knobs (TransformConfigParams: chunk — only token.recursive is a valid strategy today — embed, optional rerank, authorityWeight, recencyHalfLifeDays, mmrLambda, overfetch). Unique on (tenant_id, name, version); re-POSTing the same name mints version = max(existing) + 1 rather than colliding (createTransformConfig).

transform_run

One execution of a transform_config against a (possibly scoped) slice of raw_capture. generation is this run's own id, unique (transform_run_generation_uniq) — so a generation always resolves back to exactly one run and therefore one config at search time (resolveGenerationSearchParams). status is 'running'|'completed'|'failed'; scope is jsonb ({ adapter?, since?, until? }, filtering on raw_capture.fetched_at). runTransform never throws for a mid-run derivation failure — it marks the run row 'failed' with error recorded and returns the run summary either way.

Capture flow (services/capture.ts — captureDocument)

  1. adaptAndPlan(document) (core/adapt-and-plan.ts, pure, no I/O): validates title/kind are non-empty, re-chunks every incoming AdaptedDocumentChunk through the chunker port (default chunkTokenRecursive, caps DEFAULT_CHUNK_CAPS = 700 max / 40 min / 60 overlap tokens — adaptAndPlan's internal rechunk always applies these defaults regardless of chunker, which is why the live path is unaffected by a replay's custom caps), and computes contentHash over title + kind + externalRef + stableStringify(attributes) + joined chunk text (core/hash.ts) — this hash is the NOOP-check key.
  2. insertOrReuseRawCapture: hashes the raw wire payload (computeSourceHash, independent of contentHash — this one covers the unadapted input including adapter/occurredAt) and looks it up by (tenantId, sourceHash); reuses the existing raw_capture row or inserts a new one, all inside the same transaction as the derived rows.
  3. deriveVersionInTransaction — the single derivation core shared by live capture and replay:
    • No existing memory_document for (tenantId, adapter, externalRef) → insert a new document row + a version at version = 1, supersedesVersionId = null.
    • Existing document, and its current active version (scoped to this generation) has the same contentHash → NOOP: bump last_seen_at only, write nothing else, return { status: "noop", chunks: 0 }.
    • Existing document, content changed → flip the prior active version to status: "superseded", insert a new version at version + 1 with supersedesVersionId pointing at it, update the document's mutable fields (title, access_tags, attributes, last_seen_at).
  4. insertChunksAndGraph: inserts every plan chunk fresh (chunks are never reused across versions), then best-effort upserts entity hints (upsertEntity) and edge hints (upsertEdge) — these are independent of version and never rolled back if a later step fails within the same transaction (they're just additional statements inside it).
  5. After the transaction commits — embedInsertedChunksWithConfig: resolves/activates the tenant's embed model (activateEmbedModel, probing dims live and creating the per-model vector table if needed), then embedChunks embeds and inserts vectors for the freshly-inserted chunks. This step is best-effort: an embed-client failure (timeout, HTTP error) or a per-chunk dims mismatch is logged via log.warn and never thrown — the chunk rows are already durable in Postgres, and the module's own comments note a later re-embed pass could pick up anything left unembedded (no such background pass exists yet in this repo; chunks left unembedded simply never populate the dense channel for the query — they're still found by lexical/FTS). Any of these failure modes sets degraded: true on the CaptureResult, surfaced by POST /api/tenants/:tenantId/memory/add as a degraded field in its response — the add still succeeded (chunks are durable and lexically searchable), only the dense/vector channel for those chunks is incomplete.

Request-size guards independent of embedding. index.ts caps the whole request body at MAX_REQUEST_BODY_BYTES = 10 MB (hono/body-limit, rejecting oversized bodies with 413 before they're even parsed as JSON); core/schemas/adapted-document.ts separately caps a single chunk's text at MAX_CHUNK_TEXT_CHARS = 100,000 chars, the number of chunks per document at MAX_CHUNKS_PER_DOCUMENT = 2,000, title at MAX_TITLE_CHARS = 500, and kind at MAX_KIND_CHARS = 200 — a payload well under the body-size limit could otherwise still carry pathologically many or large chunks.

The live path (captureInTransaction → deriveVersionInTransaction) always writes generation = LIVE_GENERATION ("live", core/generation.ts) and always resolves/reuses its own raw_capture row. deriveFromRawCapture is the same core function called directly by a replay with an existing raw_capture_id and the run's own generation — a replay never writes a new raw_capture row (the raw-capture corpus is read-only from that path).

Search pipeline (services/search.ts — hybridSearch)

Entry point, one query in, one ranked/citable hit list out. k is clamped to [1, MAX_K=100], default DEFAULT_HYBRID_TOP_K = 8. An empty query string is only accepted if kinds or entityIds is provided (structured-filter-only search); otherwise it throws MemorySearchInputError (400).

Living relevancy (CL-5867): after fusion, attachCorroborationCounts loads supports/contradicts edge counts per version. Ranking multiplies by corroborationFactor (bounded [0.7, 1.3]); evidence:strong also requires the gate in core/corroboration.ts (stated human or support count ≥ floor). Capture-time authority is never rewritten. See docs/RELEVANCY.md.

Wire attribution (CL-5870): attachDerivedFrom loads derived_from edges onto candidates; toHit emits provenance, source/temporal class, occurred_at, valid_until, corroboration counts, and derived_from. The plane maps these into additive SearchItem.attribution (and DocumentStoreSearchItem.attribution). List/timeline stays document-title oriented and does not attach the full attribution block (version-level fields are search/feed concerns).

Retention (CL-5871): memory.version.retention_class + migration 0007_retention.sql. Plane helpers in services/retention.ts (deprecateVersion, tombstoneDocument, hardDeleteDocument, sweepEphemeral, setRetentionClass). Search accepts includeDeprecated so lexical/dense can include status IN ('active','deprecated'). See docs/RETENTION.md.

Capture feed (CL-5868): memory.feed({ after, limit, excludeGenerator? }) and GET .../memory/feed pull live versions ordered by feed_seq (migration 0006_capture_feed.sql). Grant-checked like search. See docs/FEED.md.

  1. Generation resolution — generation defaults to LIVE_GENERATION. For any non-live generation, resolveGenerationSearchParams (transform.ts) looks up the owning transform_run → transform_config for the search tenant and pulls its tuning knobs (authorityWeight, recencyHalfLifeDays, mmrLambda, overfetch, rerank); every field it doesn't supply falls back to the engine's own defaults. Live search never pays for this lookup. Cross-tenant generation ids resolve to null (engine defaults) so embed overrides (including apiKey) cannot leak.
  2. Lexical channel — fetchLexicalCandidates: Postgres full-text search (ts_rank against plainto_tsquery in the configured FTS_LANGUAGE, bound as a regconfig parameter, over memory_chunk.text_fts), joined to memory_version (filtered to status = 'active' and the resolved generation) and memory_document (tenant-scoped only — document access is grant-tag post-filter in the plane), optionally further filtered by kinds and/or entityIds (via a sub-select against memory_edge). Overfetches up to overfetchLimit rows, non-deduped, per-chunk.
  3. Dense channel — fetchDenseCandidates: embeds the query (embedTexts), resolves the dense table:
    • live generation → resolveActiveEmbedTable (tenant active model)
    • staged generation → resolveEmbedTableByModelKey for the transform config's embed model (ready or active; never activates) runs a raw-SQL cosine-distance ANN query via cosineDistanceExpr (e.embedding <=> $vector up to 2000 dims, or the matching (e.embedding::halfvec(N)) <=> $vector::halfvec(N) expression above that so the halfvec HNSW index is used) against that table joined back to memory_chunk/memory_version/memory_document with the same tenant-only scope as the lexical channel (no mini-ACL in SQL). Returns null (not an error) when there's no active embed model yet or the query is empty; a thrown error from the embed call or the SQL itself is caught by the caller and also folds into null/degraded — the dense channel never fails the whole search.
  4. RRF fusion — fuseRrf (core/hybrid-search.ts): combines the lexical and dense per-channel rank orders (never raw scores — they're on incomparable scales) via Reciprocal Rank Fusion, score = Σ 1/(60 + rank).
  5. Per-document dedupe (non-reranked path only) — dedupeCandidatesPerDocument: collapses to the single highest-scoring chunk per documentId, using authorityWeightedScore (relevance * (1 + 0.5 * authority)) as the rank prior, tie-broken by recency.
  6. Rerank (only if a rerank endpoint is configured — engine env RERANK_BASE_URL, or the replay generation's transform_config.params.rerank): dedupe (without the authority prior — authority is applied later on this path, never twice), take the top RERANK_CANDIDATE_LIMIT = 50, call rerankDocuments (cross-encoder), sort by rerank score.
  7. Bounded authority/recency boosts — applyBoosts: normalizes the active-stage score (rerank score, or fused RRF score on the degraded path) to [0,1] within the batch, then multiplies by authorityBoostMultiplier and recencyBoostMultiplier, both clamped to [0.7, 1.3] — a boost can never let a weak match outrank a strong one on its own. Take the top MMR_POOL_SIZE = 20.
  8. MMR diversity pass — mmrRerank (core/mmr.ts): greedy pick maximizing relevance - λ * maxSimilarityToAlreadyPicked (λ = 0.7 default, or the replay config's mmrLambda), using vectors pulled fresh from the active embedding table (fetchChunkVectors). Items without a vector are never dropped — appended by score after every vector-bearing item is placed. Produces the final top-k order.
  9. Degrade path — if reranking fails (network error, non-2xx) or was never configured, the pipeline falls back to dedupeCandidatesPerDocument(mergedRows, true, authorityWeight).slice(0, k) (fused + authority-weighted order, no MMR) and reports degraded: ["rerank_unavailable"]. If dense retrieval failed/unconfigured, degraded includes "dense_unavailable" and lexical alone answers.
  10. Finishing — attachEntityIds joins in each surviving document's entity edges; toHit builds the wire SearchHit (citation/open-target resolution via openTarget, mapping known adapters — artifact, task, workflow_run, mail — to a deep-linkable {type, id}, else a generic {type: "memory", id: documentId}).
  11. Evidence — deriveHybridEvidence: "none" if zero hits; "weak" if the lexical channel contributed zero rows (a dense-only result never reports "strong"); otherwise deriveEvidence on the lexical rows — "strong" requires both the top-ranked hit's raw ts_rank ≥ STRONG_RANK_FLOOR (0.05) and its authority ≥ AUTHORITY_STRONG_FLOOR (0.3); else "weak".

Tenant isolation is unconditional and first in every query (tenant_id filtered before grant-tag / creator document access); every table/channel is scoped that way, with no exception.

Raw + replay (the replay pipeline — services/transform.ts)

raw_capture is the immutable substrate every replay reads from and never writes to. A transform_config is a named/versioned recipe (createTransformConfig, listTransformConfigs) capturing chunk/embed/rerank

  • retrieval-tuning knobs.

runTransform(configId, scope?):

  1. Loads the config, mints a new runId = the run's own generation.
  2. Inserts a transform_run row (status: 'running').
  3. Selects every raw_capture row in scope (adapter/since/until filters on fetched_at; an empty scope = a full tenant backfill).
  4. For each row: re-parses its stored JSON payload back into a CaptureInput (parseRawCapturePayload, itself validated through RawCapturePayloadSchema — re-hydrating from storage is treated as its own trust boundary, never a blind JSON.parse), then calls deriveFromRawCapture with the config's own chunker (chunkTokenRecursive with the config's caps) and embed client config, targeting the run's generation — never the live one. Embed tables are ensured (ensureEmbedModel), never activated, so live dense search is untouched until an explicit promote.
  5. On completion, updates the run row: status: 'completed', rawCount, versionCount. On any exception mid-loop, catches it, logs it, and marks the run 'failed' with error set — runTransform itself never throws to its caller; callers always get a run summary.
  6. resolveGenerationSearchParams is how hybridSearch later maps a generation back to its config's search-tuning knobs (authority weight, recency half-life, MMR λ, overfetch, rerank config) and dense modelKey for generation-scoped table resolution.

Promote / demote (staged cutover):

  • promoteGeneration — requires status = 'completed'. Snapshots the pre-promote active model_key, activates the staged embed model (sole active for the tenant; peers demoted to ready), then swaps generation tags (live → archive, staged → live). Records archived_live_generation + archived_live_model_key for demote. If the version swap fails after activate, re-activates the prior model_key (or clears active when there was no prior).
  • demoteGeneration — re-activates archived_live_model_key (fail-closed if the registry row is gone), or clears all active models when no prior was recorded, then restores archive → live and staged corpus back onto the run generation.

Plane methods (engine DocumentStore only): createTransformConfig, listTransformConfigs, runTransform, promoteGeneration, demoteGeneration on Memory. Custom/fake stores omit these methods.

Host privilege: transform/promote/demote take tenantId (and config/run ids) only — no principal, no HTTP routes. They rewrite live generation and the active dense embed model for a tenant. The host must treat them as admin operations (grant-gate before calling); never expose them unauthenticated to end-user agents.

Mounted routes

createMemory({ app }) registers these onto the host app. Identity is the request

principal read off the Interchange context (caller(c) → { scopeId: principal.tenantId, subjectId: principal.id }); clients never send tenant_id/principal_id — the handlers only read title/text/query/limit/access_tags/share. Each route is guarded with grantGuard(deps, action), which applies the host's requireGrant("memory", action) when provided (else a pass-through).

Machine callers (CL-6286): RouteDeps.callerResolver / createMemory({ callerResolver }) lets a host resolve identity for a caller that never goes through its tenant-session middleware — e.g. a workflow-run child authenticating with its own sidecar bearer token. Unset by default (every existing host is unaffected). When set, resolveCaller (deps.ts) runs ahead of requirePrincipal/grantGuard, calls the resolver, parses its return with arktype (non-empty tenantId/principalId — a malformed resolver return is a host bug and gets 500, not 401), and seats the result as the context principal/tenant so the exact same requireGrant/authorize path a browser caller gets applies to the machine caller too. Migrating a host off a hand-rolled parallel surface (like a createWorkflowMemoryRoutes-shaped workaround) onto callerResolver: that kind of surface commonly also carries a per-run write-rate limiter and a request payload cap that this package does not implement (see CL-6286's PR body for why) — re-home both as host middleware before deleting the old surface, or a migrating host silently loses them.

Method + path Grant action Request body Response
POST /api/tenants/:tenantId/memory/add add { title, text, access_tags?, share? } 200 { documentId, versionId }; 400 on validation
POST /api/tenants/:tenantId/memory/search search { query, limit?, kinds?, entity_ids?, sources?, includeEvidence? } (limit 1–50; kinds/entity_ids/sources narrow retrieval before fusion; unset or [] = unfiltered; includeEvidence adds a short evidence string when true) 200 { items[], evidence?, degraded? }; 400 on bad input
GET /api/tenants/:tenantId/memory/list search query ?limit= (1–100, string on the wire) 200 { events: [{ at, title, source, tenantId, principalId }] } — durable recent documents for the caller's scope, filtered with grant-tag access (canAccessDocument). One event per document (active live version).
GET /api/tenants/:tenantId/memory/feed search query ?after=&limit=&exclude_generator= 200 { entries[], nextCursor } — cursor pull of new live versions. See docs/FEED.md.
POST /api/tenants/:tenantId/memory/documents/:documentId/forget forget { reason? } 200 { documentId, versions }; 403 unless caller is the document's creator; 404 unknown document. Tombstones — content is redacted, not archived; see docs/RETENTION.md.
POST /api/tenants/:tenantId/memory/documents/:documentId/purge purge none 200 { documentId, deleted, reason? }; 403 unless caller is the document's creator; 404 unknown document. Hard-deletes the row — irreversible; refused while a durable version is untombstoned.
POST /api/tenants/:tenantId/memory/versions/:versionId/retention-class forget { retention_class } 200 { versionId, documentId, status }; 400 invalid class; 403 unless caller is the version's creator; 404 unknown version.

registerMemoryRoutes and createMemory({ app }) register these seven HTTP routes (add, search, list, feed, forget, purge, retention-class). Agent tools ship in this package as Interchange defineTool factories (@corbits/memory/tools / interchange.tools): thin HTTP clients that call the mounted routes with install env (memoryBaseUrl, memoryTenantId, memoryAuthToken). They do not import the plane. Host checklist: agent principal needs memory:add and/or memory:search grants; Bearer token only (no session cookie path); tool results are JSON strings; pass AbortSignal if you need hang protection — the client has no default timeout. OpenAPI→MCP remains an optional host bridge. The plane surface is add / search / list / close, plus optional transform methods when backed by the engine DocumentStore (createTransformConfig, listTransformConfigs, runTransform, promoteGeneration, demoteGeneration) and optional retention methods (tombstoneDocument, hardDeleteDocument, setRetentionClass, sweepEphemeral, deprecateVersion) — see docs/RETENTION.md. Inference stays on the host.

Share materialization (CL-5873)

share.principals on add still mints owner tags, and when the host grant store implements WritableGrantStore.putGrant:

  1. Appends memory.doc:<documentId> to the document's access_tags.
  2. Writes one allow/search grant per peer on that resource, origin system, with conditions.memoryShare audit payload.
  3. resolveGrantConfig merges MEMORY_SHARE_CONDITION_REGISTRY so those condition keys are not fail-closed-skipped by @intx/authz.

Without a writable store: tags only + warn log (peers need host grants). Audience widening uses splitAudienceWiden (write-narrow-then-widen) and shareWidenReceipt on version attributes after source-owner approval. Ask-on-read remains design-only (fail-closed).

Timeline wire fields (vs the old CaptureLog ring)

The process-local CaptureLog hardcoded source: "api" and recorded the HTTP caller's subjectId as principalId at capture time. The durable timeline maps different columns:

Wire field Source column / meaning
at memory_document.last_seen_at (ISO) — re-captures rise in the feed
title memory_document.title
source memory_document.adapter (HTTP add defaults to "http", not "api")
tenantId memory_document.tenant_id
principalId memory_version.created_by_principal_id of the active live version (empty string when null) — the capturing actor stored on the version, not the request principal of a later timeline read

Document access (grant tags)

Document access is Interchange authz — not a mini-ACL.

  • Write path: resolveAccessTags always writes memory.owner:<caller> and merges optional accessTags / share sugar (tenant, peer principals, explicit tags). Stored on memory.document.access_tags.
  • Read path (search + list): canAccessDocument — creator always allowed; otherwise authorize(grantStore, principal, tenant, tag, "search") for any tag on the document.
  • SQL retrieval is tenant-scoped only. Document access is grant-tag post-filter in the plane (canAccessDocument); there is no SQL mini-ACL.
  • HTTP POST /add accepts access_tags and/or share — not product acl modes or block lists.

See docs/AUTHZ-DOCUMENT-ACCESS.md.

Observability

The SDK does no logging or error-reporting setup of its own — that belongs to the host app. Routes log a one-line console.error on a 5xx-class failure and return a generic error body; the host's middleware owns request logging and any Sentry/OTel wiring.

Local dev

docker compose up -d                                   # pgvector + Ollama + reranker
docker compose exec ollama ollama pull nomic-embed-text
cp .env.example .env
bun install
bun run db:setup                                       # apply the memory schema, idempotent
bun run test                                           # unit suite (no external services)

compose.yml provisions the pgvector Postgres (memory db, host port 5434), an Ollama embeddings server (:11434), and a TEI reranker (:8085). The engine never embeds internally — when EMBED_BASE_URL is set, it must point at a real endpoint. A model endpoint is just a URL + capability options, trusted the same as DATABASE_URL:

  • Local default: Ollama at http://localhost:11434 (EMBED_API_STYLE=ollama, EMBED_MODEL=nomic-embed-text).
  • Paid provider: e.g. EMBED_BASE_URL=https://api.openai.com, EMBED_MODEL=text-embedding-3-small, EMBED_API_STYLE=openai, EMBED_API_KEY=sk-....

RERANK_BASE_URL is optional — unset runs lexical+dense+MMR without the cross-encoder (degraded: ["rerank_unavailable"], still ranked/citable hits).

EMBED_BASE_URL/EMBED_MODEL are optional too — unset both to run lexical-only (degraded: ["dense_unavailable", "lexical_only"], no dense channel, add still captures documents unvectorized). See the lexical-only note above.

Testing

bun test ./src (bun run test), coverage via bun run test:coverage. Every core/* module, most services, and the route/identity layer have colocated *.test.ts files exercising pure logic and mocked-boundary behavior — no external services required. The SDK ships with unit tests only.