G7: make multi-tenant isolation a test, and close the two leaks it found

[brentrager]

Problem

G7 was filed as "Likely already passes for OLTP via organizationId; the test makes it a guarantee." It did not pass. Writing the suite found two live cross-tenant holes.

What's here

One shared conformance suite — rust/adapters/multitenancy_suite.rs, #[path]-included by each adapter's tests/multitenancy.rs, so the isolation property is asserted from exactly one source and runs against in-memory, Postgres (pgvector container) and DynamoDB (dynamodb-local). One adapter instance, two orgs — the multi-tenant pod shape, and the harshest variant. Every isolation assertion is paired with a positive control, so a backend that returns nothing cannot pass vacuously.

A server-level suite — smooth-operator-server/tests/multitenancy.rs — drives the real handler::handle_frame from an attacker authenticated to another org.

Now guaranteed by a test, on all three backends

• conversation listings are org-partitioned, by-org and by-org-and-user — asserted with the same user email owning one conversation in each org, so the isolation cannot be incidentally coming from a differing owner;
• the idempotency claim is per-org: two orgs using the same idempotency_key get two distinct conversations, where an org-blind claim would have handed org B org A's conversation row;
• message pages, participants and sessions ride their own org's conversation, and a session update in one org does not touch the other's;
• knowledge retrieval bound to org B never returns org A's document — including documents ingested through the org-blind knowledge() handle;
• checkpoints saved under one agent id are invisible under another.

🚨 Leak 1 — cross-tenant session access on every by-id path

Org was resolved per connection only to stamp new sessions. may_read_conversation (handler.rs) checked the owner email and never the org, and its deliberate ownerless-is-open rule (a conversation with no user participant carrying an email stays readable) is exactly the embeddable widget's default state.

An attacker authenticated to org B who learned an org-A session id could:

• read the session snapshot (get_session)
• read the whole conversation history (get_conversation_messages)
• drive a turn — replaying that history as LLM context and streaming the reply back (send_message)
• retitle the conversation (rename_conversation) — a write
• resume it, minting a session bound to the victim's org, which then flows into the turn's ToolProviderContext

The Lambda transport had no check at all: dispatch::get_session / send_message acted on whatever storage.get_session returned.

Fixed at the chokepoints — scoped_session and may_read_conversation take the connection's auth_org and refuse another tenant's row indistinguishably from not-found; the Lambda gained the same check off the frame's verified principal. A connection with no verified org (anonymous / tokenless — the widget's normal state) is unchanged.

Proof it fails without the fix (same_org stubbed to true, re-run):

test cross_org_get_session_is_not_found ... FAILED
test cross_org_send_message_never_runs_a_turn ... FAILED
test cross_org_resume_does_not_bind_the_foreign_conversation ... FAILED
test cross_org_rename_conversation_is_not_found_and_does_not_write ... FAILED
test cross_org_get_conversation_messages_is_not_found ... FAILED
test the_owning_org_still_reads_its_own_session ... ok      <- positive control

🚨 Leak 2 — knowledge is not tenant-isolated where the backend isn't org-partitioned

AclKnowledgeStore filtered by user/group only, on the assumption the wrapped store had already filtered by org. True for Postgres/DynamoDB; false for the in-memory adapter and any third-party adapter using the knowledge_for_access trait default.

Compounding it, POST /admin/connectors/{id}/index ingested through the org-blind knowledge() handle for every tenant — the same shape as G3: the seam existed and one caller went around it. On Postgres that wrote organization_id = NULL, which the org-filtered read can never match, so connector-ingested knowledge silently returned nothing; on DynamoDB it wrote whichever partition the adapter was constructed for.

• AclKnowledgeStore now records each document's owning org (from the org_id metadata the ingestion pipeline stamps, falling back to the org the ingesting handle is bound to) and enforces the tenant boundary before the ACL.
• DynamoKnowledgeBase honours AccessContext::organization_id for the query partition and the document's org_id for the ingest partition — mirroring what PgKnowledgeBase::with_access already did.
• PgKnowledgeBase::ingest prefers the document's org_id, so the org-blind handle still lands rows in the right tenant.
• The admin index run ingests through knowledge_for_access.

Proof each fails without its fix (revert in place, re-run):

• in-memory, before the ACL-store fix: CROSS-TENANT LEAK: org B's knowledge retrieval returned org A's document: [KnowledgeResult { chunk: "The alphamarkermem escalation code…" }, …]
• DynamoDB, before the query_org/ingest_org fix: CROSS-TENANT LEAK: org B's knowledge retrieval returned org A's document: [… document_id: "doc-a-ddb" …]
• Postgres, before the ingest fix: a document ingested through the org-blind handle but stamped with org A's org_id must be retrievable by org A — it was lost instead

Postgres was already isolated for retrieval via the access-bound seam — its change is the org-blind-ingest half only, and I say so rather than claiming more.

⚠️ Behavior change worth reading before merging

A retrieval whose AccessContext carries an org now sees only documents recorded as that org's — matching the Postgres backend's existing SQL pre-filter, so all three backends finally agree. A document ingested through the raw knowledge() handle with no org_id metadata belongs to no tenant and is therefore invisible to a turn that has one.

This is the fail-closed branch, deliberately: fail-open would have meant one tenant's unstamped documents reaching another. Migration is one line — stamp org_id, or ingest through storage.knowledge_for_access(&AccessContext::default().with_organization_id(org)). Two in-repo seeders (acl_trusted_mode.rs, scenario_parity.rs) are updated here and show the pattern; they are the only in-repo callers that were affected.

Residuals — true today, NOT guaranteed

Written up in docs/Planning/Feature Gaps.md §G7 rather than papered over:

• StorageAdapter's by-id reads (get_conversation, get_message, get_session, list_participants_by_conversation) take no org and are not org-checked at the adapter. Enforcement lives at the caller; a new caller can still forget.
• A connection with no verified org is not org-checked at all. Fail-closing it would deny the widget its own session; a deployment needing hard isolation must require auth.
• A conversation with no participants yet (create→first-frame race) has no derivable org, so the conversation-id path falls through to the ownership check. The session-id path is unaffected — a Session always carries its org.
• CheckpointStore has no org dimension: isolation rests entirely on agent-id uniqueness (the server mints a fresh UUID per Agent and never reads checkpoints back). A host reusing a stable agent id across tenants would commingle conversation state.
• S3 Vectors is UNVERIFIED — the index-per-org path is behind the s3-vectors feature and needs real AWS; the suite exercises the brute-force DynamoDB backend only.

Verification

• cargo test --workspace — 697 passed, 0 failed (rebased onto main @ 🦋 New version release #539/G1: fence the ingest→ACL chain at the pipeline seam, mark G1/G2/G9 shipped #535)
• cargo clippy --all-targets — clean, no warnings
• cargo fmt --all --check — clean
• Adapters actually exercised: in-memory, Postgres (pgvector/pg16 testcontainer), DynamoDB (dynamodb-local testcontainer). All three ran with Docker up.

Changeset: .changeset/g7-multitenancy-isolation.md (minor — behavior change above).

🤖 Generated with Claude Code

CI on the rebased head: rust ✅ · kind-deploy-smoke ✅ · anchor-guard ✅.

[changeset-bot]

🦋 Changeset detected

Latest commit: fca96ba

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 2 packages

Name	Type
@smooai/smooth-operator	Minor
@smooai/smooth-operator-web-chat-example	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR