Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -11,3 +11,7 @@ coverage/
.hub-data/
.agent-state/
tmp/

# Dispatch orchestration scratch (local only)
dispatch/

14 changes: 8 additions & 6 deletions AGENTS.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# Agent guide β€” @corbits/knowledge-engine

A library, not a service. `src/` is the whole product: a knowledge capture +
search SDK that **mounts onto a host Interchange app**. There is no server,
A library, not a service. `src/` is the whole product: a knowledge add / find /
ask / recent SDK that **mounts onto a host Interchange app**. There is no server,
port, or process entrypoint here, and there never should be.

## Commands
Expand All @@ -20,10 +20,12 @@ CI runs `typecheck` + `test` β€” both must pass before any push.

- `src/index.ts` β€” public surface: `mountKnowledgeEngine`, `mountKnowledgeRoutes`, `createKnowledgePlane`
- `src/mount-config.ts` / `src/config.ts` β€” mount config + engine config
- `src/routes/` β€” Hono routes (capture, search, timeline)
- `src/services/` β€” capture / search / transform logic
- `src/core/` β€” embed/rerank clients, arktype schemas
- `src/db/` + `migrations/` β€” Drizzle schema + SQL migrations (pgvector)
- `src/routes/` β€” Hono routes (`add`, `find`, `ask`, `recent`)
- `src/services/` β€” capture / search / transform internals (not public verbs)
- `src/ports/` β€” `DocumentStore` / `SourceProvider` / `MemoryProvider` + fakes
- `src/core/` β€” embed/rerank clients, merge, arktype schemas
- `src/db/` + `migrations/` β€” Drizzle schema + SQL migrations (pgvector, `knowledge.*`)
- `packages/` β€” optional DocumentStore adapters (`knowledge-adapter-mem0`, `knowledge-adapter-supermemory`); pure fetch, no vendor SDKs in core. Linear tools live in sibling `@corbits/linear`.

## Non-negotiable invariants

Expand Down
34 changes: 19 additions & 15 deletions ARCHITECTURE.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Knowledge Engine β€” Architecture

A knowledge capture + retrieval SDK that mounts onto an Interchange hub. The
A knowledge add / find / ask / recent SDK that mounts onto an Interchange hub. The
host owns auth, tenancy, and the process; this library owns the knowledge /
vector plane and the routes that read and write it.

Expand Down Expand Up @@ -66,25 +66,29 @@ model.

`mountKnowledgeEngine` adds, under the host app:

- `POST /api/knowledge/capture` β€” ingest content (raw + derive).
- `POST /api/knowledge/search` β€” hybrid retrieval: FTS + dense (pgvector) β†’ RRF
fusion β†’ cross-encoder rerank β†’ bounded authority/recency boosts β†’ MMR.
- `GET /api/knowledge/timeline` β€” recent captures for the caller's scope,
filtered with the same document ACL as search (visibility + block list).

It also returns an in-process `KnowledgePlane` (`capture`, `search`, `ask`).
`ask()` is grant-checked in-process (callers bypass the HTTP `requireGrant`
guard), searches as the asking principal, grounds a prompt from hit `snippet`
text (search truncates snippets to ≀240 chars today β€” that is the MVP
grounding limit), and calls a host-injected `generate` function. The engine
owns no generation client; hosts wire `generate` to their inference layer.
- `POST /api/knowledge/add` β€” ingest a note (raw + derive).
- `POST /api/knowledge/find` β€” hybrid retrieval: FTS + dense (pgvector) β†’ RRF
fusion β†’ cross-encoder rerank β†’ bounded authority/recency boosts β†’ MMR;
optional live `SourceProvider` merge (fail-soft).
- `POST /api/knowledge/ask` β€” grant-checked as `knowledge:find`; retrieves as
the principal, grounds a prompt from hit snippets, calls host-injected
`generate`. Optional memory recall when `includeMemory` is true.
- `GET /api/knowledge/recent` β€” recent documents for the caller's scope,
filtered with the same document ACL as local find (visibility + block list).

It also returns an in-process `KnowledgePlane` (`add`, `find`, `ask`,
`recent`, optional `remember` / `recall`). `ask()` is grant-checked in-process
(callers bypass the HTTP `requireGrant` guard). The engine owns no generation
client; hosts wire `generate` to their inference layer.

MCP is not part of this package β€” mount `@corbitsdev/hono-openapi-mcp` to expose
these routes as MCP tools.

External ingestion (Linear, GitHub, …) is not a route here β€” the host
authenticates the forwarder to Interchange and it calls the capture route, or
the host calls `knowledge.capture()` directly.
authenticates the forwarder to Interchange and calls `plane.add` / a
`SourceProvider` mapper, or mounts HTTP add after its own auth.

Legacy paths `/capture`, `/search`, `/timeline` are not mounted (hard cutover).

## Provenance

Expand Down
22 changes: 13 additions & 9 deletions IMPLEMENTATION.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,29 +11,33 @@ src/
index.ts # mountKnowledgeEngine / mountKnowledgeRoutes
mount-config.ts # KnowledgeConfig + loadKnowledgeConfig() β€” the mount config
config.ts # EngineConfig β€” the core vector-plane config (db + embed + rerank)
knowledge.ts # createKnowledgePlane β€” capture/search/timeline against pgvector
knowledge.ts # createKnowledgePlane β€” add/find/ask/recent against store or pgvector
acl.ts # parseAcl + shared acl_block read-path helpers
log.ts # getLogger(["knowledge-engine"]) from @intx/log
migrations.ts # runKnowledgeMigrations(url)
ports/ # DocumentStore / SourceProvider / MemoryProvider + fakes
routes/ # the mounted routes
mount.ts # mountKnowledgeRoutes (HTTP)
deps.ts # RouteDeps, caller(c) (context identity), grantGuard
capture.ts, search.ts, timeline.ts
add.ts, find.ts, ask.ts, recent.ts
db/
schema.ts # Drizzle table defs for every fixed-shape table
schema.ts # Drizzle table defs (knowledge.* schema)
client.ts # createDb(config) -> { db (drizzle), sql (raw postgres-js) }
services/
capture.ts # captureDocument, deriveFromRawCapture β€” the write path
search.ts # hybridSearch and every retrieval-candidate query
timeline.ts # listTimelineEvents β€” durable recent docs + ACL filter
transform.ts # transform_config CRUD + runTransform (replay)
core/ # framework-agnostic, mostly pure (chunking, embed/rerank
# clients, authority, hybrid fusion, MMR, schemas)
core/ # framework-agnostic (chunking, embed/rerank, merge, schemas)
packages/
knowledge-adapter-mem0/ # DocumentStore backend (Mem0)
knowledge-adapter-supermemory/ # DocumentStore backend (Supermemory)
migrations/ # pgvector schema, applied in filename order by scripts/db-setup.ts
scripts/db-setup.ts # idempotent migration runner, tracked in `_migrations`
compose.yml # pgvector + Ollama + reranker for local dev
```


The SDK has no server and no process entrypoint. `mountKnowledgeEngine` takes
the host's `Hono<TenantEnv>` app plus `{ config, grants? }` and mounts the
routes; each reads identity from the context (`caller(c)`) and guards via
Expand Down Expand Up @@ -126,7 +130,7 @@ columns (`authority`, `actor_count`, `has_social_signal`, `source_class`) are
a **snapshot computed once at capture time** (`computeAuthority`, never
recomputed retroactively). `raw_capture_id` points at the immutable source row
this version was derived from. `generation` (added by migration 0009,
default `'live'`) is the replay-generation tag: the normal `/capture` path always writes
default `'live'`) is the replay-generation tag: the normal add path always writes
`'live'`; a replay (`runTransform`) writes its own `transform_run.id` instead,
so a replayed corpus's versions never collide with, or even become visible
alongside, the live ones unless a caller explicitly searches that generation.
Expand Down Expand Up @@ -245,7 +249,7 @@ string-interpolated into raw SQL β€” this is the only place in the codebase a
computed identifier is spliced into DDL/DML.

### `raw_capture`
The immutable, append-only substrate (the raw-capture layer). Stores the exact `/capture`
The immutable, append-only substrate (the raw-capture layer). Stores the exact add/ingest
request payload (`adapter`, `occurred_at`, `document`) as JSON in `raw_text`
(there's also a `raw_bytes bytea` column for non-textual payloads, currently
unused by any write path β€” everything captured today is JSON). Deduped on
Expand Down Expand Up @@ -320,7 +324,7 @@ returns the run summary either way.
this repo; chunks left unembedded simply never populate the dense
channel for the query β€” they're still found by lexical/FTS). Any of these
failure modes sets `degraded: true` on the `CaptureResult`, surfaced by
`POST /api/knowledge/capture` as a `degraded` field in its response β€” the capture
`POST /api/knowledge/add` as a `degraded` field in its response β€” the add
still succeeded (chunks are durable and lexically searchable), only the
dense/vector channel for those chunks is incomplete.

Expand Down Expand Up @@ -493,7 +497,7 @@ timeline maps different columns:
|---|---|
| `at` | `knowledge_document.last_seen_at` (ISO) β€” re-captures rise in the feed |
| `title` | `knowledge_document.title` |
| `source` | `knowledge_document.adapter` (HTTP capture defaults to `"mcp"`, not `"api"`) |
| `source` | `knowledge_document.adapter` (HTTP add defaults to `"http"`, not `"api"`) |
| `tenantId` | `knowledge_document.tenant_id` |
| `principalId` | `knowledge_version.created_by_principal_id` of the active live version (empty string when null) β€” the capturing actor stored on the version, not the request principal of a later timeline read |

Expand Down
Loading
Loading