Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions AGENTS.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,10 @@
# Agent guide — @corbits/memory

A library, not a service. `src/` is the whole product: a memory add / find /
ask / recent SDK that **mounts onto a host Interchange app**. There is no server,
A library, not a service. `src/` is the whole product: a memory **add / search /
list** SDK that **mounts onto a host Interchange app**. There is no server,
port, or process entrypoint here, and there never should be.


## Commands

```bash
Expand Down
139 changes: 63 additions & 76 deletions ARCHITECTURE.md
Original file line number Diff line number Diff line change
@@ -1,97 +1,84 @@
# Corbits Memory — Architecture

A memory add / find / ask / recent SDK that mounts onto an Interchange hub. The
host owns auth, tenancy, and the process; this library owns the memory /
vector plane and the routes that read and write it.
A memory **add / search / list** SDK that mounts onto an Interchange hub. The
host owns auth, tenancy, and the process; this library owns the durable memory
plane and the protected routes that read and write it.

## Why an SDK, not a service

The memory store was originally built inside a larger backend. It turned out
to be cleanly detachable, and then cleanly *mountable*:
The store was detachable from a larger backend, then mountable:

- No memory table has a foreign key into any control-plane table — every
cross-reference (`tenant_id`, `principal_id`, source refs) is plain `text`.
- Embedding and reranking go out as plain HTTP to configured model endpoints,
not through any agent runtime.
- The ACL rule is a self-contained scope stored on the row, not a join against
a grant engine.
- No memory table has a foreign key into any control-plane table — cross-refs
(`tenant_id`, `principal_id`, source refs) are plain `text`.
- Embedding and reranking go out as plain HTTP to configured model endpoints.
- Document access is Interchange grant tags on the row (`accessTags` + creator),
not a private ACL engine inside this package.

So the library needs nothing but a pgvector Postgres and an embed/rerank
endpoint. It ships as `createMemory(opts)` — pass `app` to register HTTP. The host passes its
It ships as `createMemory({ app, … })`: the host passes its Hono app and grant
store; the library registers routes, reads identity from request context, and
talks to its DocumentStore. No second server.

Hono app and its grant store; the library mounts its routes, reads identity from
the request context, and talks to its own vector store. No second server, no
HTTP hop.
## Product path

```
tools / ingestion → /api/tenants/:tenantId/memory/* → Memory plane → DocumentStore
Interchange auth + principal + grants
```

Mount is intentionally small. The host already has `app`, grants, and
principal middleware; memory only needs to be handed those and the vector
config (or an injected store).

## Boundaries

- **Runtime**: Bun + Hono, mounted on the host's app. **DB**: its own pgvector
Postgres (`KNOWLEDGE_DATABASE_URL`). **Types**: arktype at every route
boundary.
- **No auth of its own.** The SDK authenticates nothing. Interchange resolves
the caller (session, `cke_` API key, or MCP OAuth) and puts `principal` +
`tenant` on the request context; each mounted route reads identity from there
(`caller(c)` → `scopeId = principal.tenantId`, `subjectId = principal.id`).
- **Grants delegate to the host.** Pass `grants` (`{ grantStore,
conditionRegistry }`) and the SDK guards routes with Interchange's
`createRequireGrant`.
- **Dependencies** are public npm only — `@intx/hub-api` (`TenantEnv`,
`createRequireGrant`), `@intx/authz` (`authorize`), `@intx/log`, Hono,
Drizzle, arktype, `postgres`, `hono-openapi`. Eight total. LGPL-2.1-licensed —
see `LICENSE`.

## Identity — read from context, stored as data

1. **Who is calling** is the request principal, read off the Interchange
context. Clients never send `tenant_id`/`principal_id` — the handlers read
only content fields (title/text/query/limit/access_tags/share) and take identity from context.
2. **What is stored** is opaque data on every record: `tenant_id`,
`principal_id`, `created_by_kind` (human/agent/system), `source_class`, and
relations (the edge graph). Every query is scoped by `tenant_id` first; then
document access uses Interchange grant tags (`accessTags` + creator).

Cross-tenant isolation is enforced at query time by `tenant_id`; document-level
access is grant tags via `@intx/authz` (creator always allowed). This is the
trust model.

## Layers

- `raw_capture` — immutable, append-only original content. The replay substrate.
- `derived` — chunks / embeddings / authority / edges, all derived from
`raw_capture`.
- `transform_config` + replay — a named, versioned transform (chunk strategy,
embed model, rerank endpoint, authority weights, MMR λ) that rebuilds the
derived layer from raw without re-fetching source.
- **Runtime**: Bun + Hono, mounted on the host app. **DB**: own pgvector
Postgres (`KNOWLEDGE_DATABASE_URL`) unless `documentStore` is injected.
**Types**: arktype at every route boundary.
- **No auth of its own.** Interchange resolves the caller and puts `principal`
+ `tenant` on context; routes read identity from there
(`tenantId = principal.tenantId`, `principalId = principal.id`).
- **Grants delegate to the host.** Pass `grantStore` + `conditionRegistry`;
routes use `createRequireGrant("memory", action)`.
- **Dependencies**: `@intx/hub-api`, `@intx/authz`, `@intx/log`, Hono, Drizzle,
arktype, `postgres`, `hono-openapi`. LGPL-2.1 — see `LICENSE`.

## Mounted surface
## Identity — context in, data out

1. **Who is calling** is the request principal. Clients never send
`tenant_id` / `principal_id` on the body.
2. **What is stored** is opaque data: `tenant_id`, `principal_id`,
`created_by_kind`, `access_tags`, source refs. Queries scope by `tenant_id`
first; document access is grant tags + creator.

`createMemory({ app })` adds, under the host app:
## Layers (default pgvector store)

- `POST /api/memory/add` — ingest a note (raw + derive).
- `POST /api/memory/search` — hybrid retrieval: FTS + dense (pgvector) → RRF
fusion → cross-encoder rerank → bounded authority/recency boosts → MMR;
optional live `SourceProvider` merge (fail-soft).
- `GET /api/memory/list` — recent documents for the caller's scope,
filtered with the same grant-tag access as local search (`canAccessDocument`).
- `raw_capture` — immutable original content (replay substrate).
- `derived` — chunks / embeddings / authority / edges from raw.
- `transform_config` + replay — rebuild derived from raw without re-fetch.

Injected DocumentStores own their own persistence model; the plane still
exposes the same three verbs.

## Mounted surface

It also returns an in-process `Memory` (`add`, `search`, `list`, `close`).
There is no product `ask` / `remember` / `recall` and no host-injected
`generate` on the plane — inference is host-owned and ephemeral (call your
model, then `add` / `search`).
`createMemory({ app })` registers:

MCP is not part of this package — mount `@corbitsdev/hono-openapi-mcp` to expose
these routes as MCP tools.
- `POST /api/tenants/:tenantId/memory/add` — ingest (raw + derive on the default store).
- `POST /api/tenants/:tenantId/memory/search` — hybrid retrieval (FTS + dense → RRF → rerank →
authority/recency → MMR); optional live `SourceProvider` merge (fail-soft).
- `GET /api/tenants/:tenantId/memory/list` — recent documents, same grant-tag filter as local
search.

External ingestion (Linear, GitHub, …) is not a route here — the host
authenticates the forwarder to Interchange and calls `plane.add` / a
`SourceProvider` mapper, or mounts HTTP add after its own auth.
Returns an in-process `Memory` (`add`, `search`, `list`, `close`) for host
workers and ingestion modules that already resolved identity.

Legacy paths `/capture`, `/search` (old knowledge), `/timeline`, `/find`,
`/ask`, `/recent` are not mounted (hard cutover).
**Agent tools are not in this package.** Routes are OpenAPI-described
(`describeRoute`). The host mounts `@corbitsdev/hono-openapi-mcp` (or any
OpenAPI→tools bridge) so agents call these routes under Interchange auth.

## Provenance

The framework-agnostic core (chunk strategies, embed client + model registry,
authority weighting, hybrid search, MMR, rerank client, ingestion adapters) was
extracted from an internal RAG implementation and generalized. The persistence
and the mountable surface are native to this repo.
Framework-agnostic core (chunking, embed/rerank clients, hybrid search, MMR)
was extracted from an internal RAG implementation. Persistence and the
mountable surface are native to this repo.
22 changes: 11 additions & 11 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,15 +14,18 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
`registerMemoryRoutes`.

`createMemory`, `loadMemoryConfig`, `runMemoryMigrations`, `Memory`,
`MemoryConfig`, `MemoryError`. HTTP paths are under `/api/memory/`; grants are
`MemoryConfig`, `MemoryError`. HTTP paths are under
`/api/tenants/:tenantId/memory/`; grants are
`memory:add` / `memory:search`; access tags use `memory.owner:` / `memory.tenant:`
/ `memory.space:`. Postgres schema name remains `knowledge`. See `MIGRATION.md`.
/ `memory.space:`. Postgres schema name remains `knowledge`.
- **Breaking:** memory plane surface is `add` / `search` / `list` with
`principalId` + `tenantId` only. Removed product verbs: `find` (→`search`),
`recent` (→`list`), `ask`, `remember`, `recall`, and any `MemoryProvider` /
`generate` path. Inference is host-owned. See `MIGRATION.md`.
- **Breaking:** HTTP routes are `POST /api/memory/add`,
`POST /api/memory/search`, `GET /api/memory/list`. Old paths are not mounted.
`principalId` + `tenantId` only. Inference is host-owned (no answer endpoint
or personal-memory side-channel on the plane).

- **Breaking:** HTTP routes are `POST /api/tenants/:tenantId/memory/add`,
`POST /api/tenants/:tenantId/memory/search`,
`GET /api/tenants/:tenantId/memory/list` (inherits hub `resolveTenant`).
Old unscoped `/api/memory/*` paths are not mounted.
- **Breaking:** grant actions are `add` and `search` (was `capture` / `find` /
knowledge `search`). `list` uses the `search` grant. Capability resource is
`memory`. Document-tag checks use action `search`.
Expand All @@ -42,13 +45,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Optional `TextExtractor` + `file` XOR `content` on `add`
- `share` sugar on `add` (maps to access tags only: owner, tenant, peers)
- `access_tags` on `knowledge.document` (baseline schema; Postgres schema name unchanged)
- `MIGRATION.md` hard-cutover notes for in-repo consumers

### Removed

- Product `ask` / `remember` / `recall` and `MemoryProvider` side-channel
- Host-injected `generate` on the plane (use host inference + `add` / `search`)
- HTTP `POST /api/memory/ask`, `POST /api/memory/find`, `GET /api/memory/recent`
- Host-injected generate path on the plane (use host inference + `add` / `search`)

## [0.1.2] — 2026-07-31

Expand Down
15 changes: 9 additions & 6 deletions IMPLEMENTATION.md
Original file line number Diff line number Diff line change
Expand Up @@ -329,7 +329,7 @@ returns the run summary either way.
this repo; chunks left unembedded simply never populate the dense
channel for the query — they're still found by lexical/FTS). Any of these
failure modes sets `degraded: true` on the `CaptureResult`, surfaced by
`POST /api/memory/add` as a `degraded` field in its response — the add
`POST /api/tenants/:tenantId/memory/add` as a `degraded` field in its response — the add
still succeeded (chunks are durable and lexically searchable), only the
dense/vector channel for those chunks is incomplete.

Expand Down Expand Up @@ -486,12 +486,15 @@ Each route is guarded with `grantGuard(deps, action)`, which applies the host's

| Method + path | Grant action | Request body | Response |
|---|---|---|---|
| `POST /api/memory/add` | `add` | `{ title, text, access_tags?, share? }` | `200 { documentId }`; `400` on validation |
| `POST /api/memory/search` | `search` | `{ query, limit?, kinds?, entity_ids? }` (limit 1–50; `kinds`/`entity_ids` narrow every retrieval channel — lexical and dense — to a document `kind` or linked entity id before fusion; unset or `[]` = unfiltered) | `200 { items[], evidence?, degraded? }`; `400` on bad input |
| `GET /api/memory/list` | `search` | — | `200 { events: [{ at, title, source, tenantId, principalId }] }` — durable recent documents for the caller's scope, filtered with grant-tag access (`canAccessDocument`). One event per document (active live version). |
| `POST /api/tenants/:tenantId/memory/add` | `add` | `{ title, text, access_tags?, share? }` | `200 { documentId }`; `400` on validation |
| `POST /api/tenants/:tenantId/memory/search` | `search` | `{ query, limit?, kinds?, entity_ids? }` (limit 1–50; `kinds`/`entity_ids` narrow every retrieval channel — lexical and dense — to a document `kind` or linked entity id before fusion; unset or `[]` = unfiltered) | `200 { items[], evidence?, degraded? }`; `400` on bad input |
| `GET /api/tenants/:tenantId/memory/list` | `search` | — | `200 { events: [{ at, title, source, tenantId, principalId }] }` — durable recent documents for the caller's scope, filtered with grant-tag access (`canAccessDocument`). One event per document (active live version). |

`registerMemoryRoutes` and `createMemory({ app })` register the three HTTP routes.
Agent tools are a host concern — mount `@corbitsdev/hono-openapi-mcp` (or any
OpenAPI→tools bridge) against the same app. The plane surface is only
`add` / `search` / `list` (plus `close`); inference stays on the host.

`registerMemoryRoutes` and `createMemory({ app })` register the three HTTP routes. MCP is a separate package (`@corbitsdev/hono-openapi-mcp`).
There is no product `ask` / `remember` / `recall` HTTP or plane surface.


### Timeline wire fields (vs the old CaptureLog ring)
Expand Down
104 changes: 0 additions & 104 deletions MIGRATION.md

This file was deleted.

Loading
Loading