Retention routes: forget, purge, retention-class (CL-6288)

AGENTS.md

@@ -22,7 +22,8 @@ CI runs `typecheck` + `test` — both must pass before any push.
 - `src/index.ts` — public surface: `createMemory` (optional `app` registers HTTP), `registerMemoryRoutes`
 
 - `src/mount-config.ts` / `src/config.ts` — mount config + engine config
-- `src/routes/` — Hono routes (`add`, `search`, `list`)
+- `src/routes/` — Hono routes (`add`, `search`, `list`, `feed`, retention
+  `forget`/`purge`/`retention-class`)
 - `src/tools/` — Interchange `defineTool` factories (`@corbits/memory/tools`);
   HTTP clients for mounted routes (env credentials; no in-process plane)
 - `src/services/` — capture / search / transform internals (not public verbs)

ARCHITECTURE.md

@@ -56,6 +56,20 @@ helpers are optional multi-writer / backfill — not the primary path.
   anything from the request body, is what `grantGuard` authorizes.
 - **Grants delegate to the host.** Pass `grantStore` + `conditionRegistry`;
   routes use `createRequireGrant("memory", action)`.
+- **Two authorization mechanisms, not one — know which is source of truth
+  for what.** (1) Grant tags decide *capability* (may this principal call
+  `add`/`search`/`forget`/`purge` at all — `requireGrant`) and *visibility*
+  (which documents a principal may see — `accessTags` + `canAccessDocument`
+  in `grant-tags.ts`, where a share grant legitimately widens who can find a
+  document). (2) A separate, imperative **ownership** check — the creator
+  lookup in `services/retention-ownership.ts`, called from `memory.ts` —
+  decides who may *forget or purge* a specific document, and is the sole
+  source of truth for "whose document is this": it is never derived from
+  grant tags and a share grant never satisfies it. `MemoryGrantRequirement.
+  installHint` (`grant-requirements.ts`) looks adjacent to this but is not:
+  it is advisory metadata for install tooling sizing a capability grant,
+  read by nothing at request time. Do not extend mechanism (1) expecting it
+  to cover ownership — extend `retention-ownership.ts` instead.
 - **Dependencies**: `@intx/hub-api`, `@intx/authz`, `@intx/log`, Hono, Drizzle,
   arktype, `postgres`, `hono-openapi`. LGPL-2.1 — see `LICENSE`.
 
@@ -85,9 +99,23 @@ exposes the same three verbs.
   authority/recency → MMR); optional live `SourceProvider` merge (fail-soft).
 - `GET /api/tenants/:tenantId/memory/list` — recent documents, same grant-tag filter as local
   search.
-
-Returns an in-process `Memory` (`add`, `search`, `list`, `close`) for host
-workers and ingestion modules that already resolved identity.
+- `POST /api/tenants/:tenantId/memory/documents/:documentId/forget` — tombstone
+  (grant `memory:forget`; creator-only, see below).
+- `POST /api/tenants/:tenantId/memory/documents/:documentId/purge` — hard
+  delete (grant `memory:purge`; creator-only; irreversible).
+- `POST /api/tenants/:tenantId/memory/versions/:versionId/retention-class` —
+  set retention class (grant `memory:forget`; creator-only).
+
+Forget and purge are deliberately separate routes and separate grant actions
+(never one route with a boolean flag) — a host wiring a "forget this" button
+cannot accidentally wire up permanent deletion. `sweepEphemeral` (TTL
+auto-deprecation) is **not** HTTP-routed: it is a maintenance sweep a host
+schedules on its own cron, not a user action; call it in-process against the
+returned `Memory`. See docs/RETENTION.md.
+
+Returns an in-process `Memory` (`add`, `search`, `list`, `close`, plus the
+optional retention writes) for host workers and ingestion modules that
+already resolved identity.
 
 **Agent tools live in this package** as thin HTTP clients
 (`@corbits/memory/tools` / `interchange.tools`): `defineTool` factories that

CHANGELOG.md

@@ -9,6 +9,18 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Added
 
+- Retention HTTP routes (CL-6288): `POST …/memory/documents/:documentId/forget`
+  (tombstone, grant `memory:forget`), `POST …/memory/documents/:documentId/purge`
+  (hard delete, grant `memory:purge`), and
+  `POST …/memory/versions/:versionId/retention-class` (grant `memory:forget`).
+  Forget and purge are separate routes with separate grant actions — never one
+  route with a boolean flag — and both are refused with 403 unless the caller
+  is the document/version's creator, independent of any share grant that lets
+  them merely see it. `sweepEphemeral` stays off the HTTP surface (maintenance
+  sweep, not a user action); a host schedules it on its own cron against the
+  in-process `Memory`. New `memory:forget` / `memory:purge` grant requirements
+  (`source: "creator"`) and `capabilityIdsForSurface()` so distiller/tools
+  installs no longer pick up routes-only capabilities by accident.
 - `RouteDeps.callerResolver` / `createMemory({ callerResolver })` — an
   optional host-supplied resolver from a request to a `{ tenantId,
   principalId }` scope, for a caller that never goes through the host's

IMPLEMENTATION.md

@@ -633,8 +633,13 @@ surface, or a migrating host silently loses them.
 | `POST /api/tenants/:tenantId/memory/add` | `add` | `{ title, text, access_tags?, share? }` | `200 { documentId, versionId }`; `400` on validation |
 | `POST /api/tenants/:tenantId/memory/search` | `search` | `{ query, limit?, kinds?, entity_ids?, sources?, includeEvidence? }` (limit 1–50; `kinds`/`entity_ids`/`sources` narrow retrieval before fusion; unset or `[]` = unfiltered; `includeEvidence` adds a short evidence string when true) | `200 { items[], evidence?, degraded? }`; `400` on bad input |
 | `GET /api/tenants/:tenantId/memory/list` | `search` | query `?limit=` (1–100, string on the wire) | `200 { events: [{ at, title, source, tenantId, principalId }] }` — durable recent documents for the caller's scope, filtered with grant-tag access (`canAccessDocument`). One event per document (active live version). |
+| `GET /api/tenants/:tenantId/memory/feed` | `search` | query `?after=&limit=&exclude_generator=` | `200 { entries[], nextCursor }` — cursor pull of new live versions. See `docs/FEED.md`. |
+| `POST /api/tenants/:tenantId/memory/documents/:documentId/forget` | `forget` | `{ reason? }` | `200 { documentId, versions }`; `403` unless caller is the document's creator; `404` unknown document. Tombstones — content is redacted, not archived; see docs/RETENTION.md. |
+| `POST /api/tenants/:tenantId/memory/documents/:documentId/purge` | `purge` | none | `200 { documentId, deleted, reason? }`; `403` unless caller is the document's creator; `404` unknown document. Hard-deletes the row — irreversible; refused while a `durable` version is untombstoned. |
+| `POST /api/tenants/:tenantId/memory/versions/:versionId/retention-class` | `forget` | `{ retention_class }` | `200 { versionId, documentId, status }`; `400` invalid class; `403` unless caller is the version's creator; `404` unknown version. |
 
-`registerMemoryRoutes` and `createMemory({ app })` register the three HTTP routes.
+`registerMemoryRoutes` and `createMemory({ app })` register these seven HTTP
+routes (add, search, list, feed, forget, purge, retention-class).
 Agent tools ship in this package as Interchange `defineTool` factories
 (`@corbits/memory/tools` / `interchange.tools`): thin HTTP clients that call the
 mounted routes with install env (`memoryBaseUrl`, `memoryTenantId`,
@@ -645,7 +650,10 @@ protection — the client has no default timeout. OpenAPI→MCP remains an optio
 host bridge. The plane surface is `add` / `search` / `list` / `close`, plus
 optional transform methods when backed by the engine DocumentStore
 (`createTransformConfig`, `listTransformConfigs`, `runTransform`,
-`promoteGeneration`, `demoteGeneration`). Inference stays on the host.
+`promoteGeneration`, `demoteGeneration`) and optional retention methods
+(`tombstoneDocument`, `hardDeleteDocument`, `setRetentionClass`,
+`sweepEphemeral`, `deprecateVersion`) — see docs/RETENTION.md. Inference stays
+on the host.
 
 ### Share materialization (CL-5873)
 

docs/RETENTION.md

@@ -7,7 +7,7 @@ Versions carry a **retention class** orthogonal to temporal ranking class
 | --- | --- |
 | `durable` | Long-lived claims; hard-delete blocked until tombstoned |
 | `standard` | Default working memory |
-| `ephemeral` | Short TTL; sweeper hard-deletes past `valid_until` (or 7d from `ingested_at`) |
+| `ephemeral` | Short TTL; sweeper deprecates past `valid_until` (or 7d from `ingested_at`) — hard delete is a separate explicit step |
 | `source_only` | Keep raw capture; derived versions may be dropped by host policy |
 
 Schema: `memory.version.retention_class` (migration `0007_retention.sql`).
@@ -21,7 +21,7 @@ CHECK constraint `version_retention_class_check` stays lockstep with
 | Deprecate | `memory.deprecateVersion` | `status=deprecated`, `deprecated_at` / reason |
 | Tombstone | `memory.tombstoneDocument` | All active/deprecated/superseded versions → `tombstoned`; chunk text redacted to `[redacted]` |
 | Hard delete | `memory.hardDeleteDocument` | Deletes document row (cascade); **refuses** if any non-tombstoned version is `durable` |
-| Sweep | `memory.sweepEphemeral` | Auto-deprecates ephemeral versions past `valid_until` (or 7d from `ingested_at`); host schedules, core is cron-free |
+| Sweep | `memory.sweepEphemeral` | Auto-**deprecates** (never deletes) ephemeral versions past `valid_until` (or 7d from `ingested_at`); host schedules, core is cron-free |
 | Set class | `memory.setRetentionClass` | Update `retention_class` on a version |
 
 Search and feed exclude non-active (and non-superseded for feed) rows by
@@ -30,3 +30,46 @@ versions intentionally (ops / audit). Hard-delete is a separate explicit
 verb — TTL never hard-deletes.
 
 Service module: `src/services/retention.ts`.
+
+## HTTP surface (CL-6288)
+
+| Route | Grant action | Plane verb |
+| --- | --- | --- |
+| `POST …/memory/documents/:documentId/forget` | `memory:forget` | `tombstoneDocument` |
+| `POST …/memory/documents/:documentId/purge` | `memory:purge` | `hardDeleteDocument` |
+| `POST …/memory/versions/:versionId/retention-class` | `memory:forget` | `setRetentionClass` |
+
+`deprecateVersion` and `sweepEphemeral` have no route (see below).
+
+**Tombstone vs. hard delete stay distinct verbs, distinct grants.** A UI
+offering "forget this" must never be one flag away from "shred this" by
+accident. `forget` (tombstone) is **not** an undo-able action: the document
+stops appearing in search/feed and its chunk text is overwritten with
+`[redacted]` — the original content does not survive, there is no
+un-tombstone/restore verb, and only version metadata (status, timestamps,
+retention class) remains for audit. `purge` (hard delete) goes further and
+removes the document row itself; it has its own grant action and is refused
+outright while a `durable`-class version on the document is untombstoned. The
+distinction that matters is *what's still queryable*: after `forget` a
+document row and its metadata still exist (for audit) but its content is
+gone; after `purge` nothing does. A host can grant `forget` broadly (every
+user gets a "forget this" button) while keeping `purge` to an operator role —
+but should not describe `forget` to end users as reversible.
+
+**Ownership, not just visibility.** `memory:search`/a document's `accessTags`
+say who can *see* a document — never who may forget or purge it. Every
+retention route additionally checks that the caller is the document's
+creator (`created_by_principal_id` — the document's first version for
+`forget`/`purge`, the specific version's own creator for `retention-class`),
+independent of any share grant. A peer who can search a shared document gets
+403 on `forget`/`purge`/`retention-class` for it. See
+`src/services/retention-ownership.ts` and the ownership tests in
+`src/memory.test.ts` / `src/routes/routes.test.ts`.
+
+**`sweepEphemeral` stays off the HTTP surface.** It is a maintenance sweep —
+"deprecate every ephemeral version past its TTL for this tenant" — not
+something a single user requests about their own data, and it has no natural
+per-caller grant (it does not take a `principalId` and touches every
+matching row tenant-wide). A host that wants it schedules a cron job calling
+`memory.sweepEphemeral({ tenantId })` in-process (the returned `Memory`
+already exposes it); the engine stays cron-free per `ARCHITECTURE.md`.

package.json

@@ -15,14 +15,26 @@
       {
         "resource": "memory",
         "action": "add",
-        "source": "tenant",
+        "installHint": "tenant",
         "surfaces": ["tools", "distiller", "routes"]
       },
       {
         "resource": "memory",
         "action": "search",
-        "source": "tenant",
+        "installHint": "tenant",
         "surfaces": ["tools", "distiller", "routes"]
+      },
+      {
+        "resource": "memory",
+        "action": "forget",
+        "installHint": "creator",
+        "surfaces": ["routes"]
+      },
+      {
+        "resource": "memory",
+        "action": "purge",
+        "installHint": "creator",
+        "surfaces": ["routes"]
       }
     ]
   },

src/core/schemas/non-blank-id.ts

@@ -0,0 +1,15 @@
+import { type } from "arktype";
+
+/**
+ * `"string >= 1"` is a LENGTH constraint, not a content one — `" "` has
+ * length 1 and would pass it, seating/accepting a whitespace-only id exactly
+ * like the empty-string case such a schema exists to reject. Require at
+ * least one non-whitespace character instead.
+ *
+ * Shared by the resolved-caller trust boundary (`routes/deps.ts`, CL-6286)
+ * and the retention path-param schemas (`http-bodies.ts`, CL-6288) so the
+ * fix lives in one schema instead of a comment repeated at each call site.
+ */
+export const NonBlankId = type("string").narrow(
+  (s, ctx) => s.trim().length > 0 || ctx.mustBe("non-blank (not just whitespace)"),
+);

src/distiller/workflow.ts

@@ -23,7 +23,7 @@ import { defineWorkflow, type WorkflowDefinition } from "@intx/workflow";
 import { memoryAdd } from "../tools/add.ts";
 import { memoryFeed } from "../tools/feed.ts";
 import { memorySearch } from "../tools/search.ts";
-import { MEMORY_CAPABILITY_IDS } from "../grant-requirements.ts";
+import { capabilityIdsForSurface } from "../grant-requirements.ts";
 import {
   RESIDENT_DISTILLER_AGENT_ID,
   RESIDENT_DISTILLER_CRON_DEFAULT,
@@ -97,7 +97,7 @@ export function createResidentDistiller(
       "Resident memory distiller — feed → classify → claim write",
     systemPrompt: opts.systemPrompt ?? DEFAULT_SYSTEM_PROMPT(generatorAgentId),
     tools,
-    capabilities: [...MEMORY_CAPABILITY_IDS],
+    capabilities: capabilityIdsForSurface("distiller"),
 
     inference: opts.inference,
     tags: {

src/grant-requirements.test.ts

@@ -3,27 +3,79 @@ import { readFileSync } from "node:fs";
 import { join } from "node:path";
 
 import {
+  capabilityIdsForSurface,
   MEMORY_CAPABILITY_IDS,
   MEMORY_GRANT_REQUIREMENTS,
 } from "./grant-requirements.ts";
 
 describe("MEMORY_GRANT_REQUIREMENTS", () => {
-  test("covers add + search on memory resource", () => {
+  test("covers add, search, forget, and purge on memory resource", () => {
     expect(MEMORY_GRANT_REQUIREMENTS.map((r) => r.action).sort()).toEqual([
       "add",
+      "forget",
+      "purge",
       "search",
     ]);
     for (const r of MEMORY_GRANT_REQUIREMENTS) {
       expect(r.resource).toBe("memory");
-      expect(r.source).toBe("tenant");
+    }
+  });
+
+  test("add/search hint tenant-wide and reach tools + distiller + routes", () => {
+    for (const action of ["add", "search"]) {
+      const r = MEMORY_GRANT_REQUIREMENTS.find((x) => x.action === action)!;
+      expect(r.installHint).toBe("tenant");
       expect(r.surfaces).toContain("tools");
       expect(r.surfaces).toContain("distiller");
+      expect(r.surfaces).toContain("routes");
+    }
+  });
+
+  test("forget/purge hint creator-scoped and are routes-only", () => {
+    for (const action of ["forget", "purge"]) {
+      const r = MEMORY_GRANT_REQUIREMENTS.find((x) => x.action === action)!;
+      expect(r.installHint).toBe("creator");
+      expect(r.surfaces).toEqual(["routes"]);
+    }
+  });
+
+  test("installHint is advisory only — the requirement shape carries no enforcement field", () => {
+    // The actual ownership check lives in services/retention-ownership.ts,
+    // wired imperatively into memory.ts, entirely independent of this hint.
+    // This test exists so a future reader who tightens grant-requirements.ts
+    // notices this comment rather than assuming installHint is load-bearing.
+    for (const r of MEMORY_GRANT_REQUIREMENTS) {
+      expect(Object.keys(r).sort()).toEqual([
+        "action",
+        "installHint",
+        "resource",
+        "surfaces",
+      ]);
     }
   });
 
   test("capability ids are resource:action", () => {
     expect([...MEMORY_CAPABILITY_IDS].sort()).toEqual([
       "memory:add",
+      "memory:forget",
+      "memory:purge",
+      "memory:search",
+    ]);
+  });
+
+  test("capabilityIdsForSurface excludes routes-only actions from distiller/tools", () => {
+    expect(capabilityIdsForSurface("distiller").sort()).toEqual([
+      "memory:add",
+      "memory:search",
+    ]);
+    expect(capabilityIdsForSurface("tools").sort()).toEqual([
+      "memory:add",
+      "memory:search",
+    ]);
+    expect(capabilityIdsForSurface("routes").sort()).toEqual([
+      "memory:add",
+      "memory:forget",
+      "memory:purge",
       "memory:search",
     ]);
   });
@@ -36,7 +88,7 @@ describe("MEMORY_GRANT_REQUIREMENTS", () => {
         grantRequirements?: Array<{
           resource: string;
           action: string;
-          source: string;
+          installHint: string;
           surfaces: string[];
         }>;
       };
@@ -46,7 +98,7 @@ describe("MEMORY_GRANT_REQUIREMENTS", () => {
       MEMORY_GRANT_REQUIREMENTS.map((r) => ({
         resource: r.resource,
         action: r.action,
-        source: r.source,
+        installHint: r.installHint,
         surfaces: [...r.surfaces],
       })),
     );