Skip to content

A storage blip is not "session not found" - #545

Merged
brentrager merged 2 commits into
mainfrom
th-loadsession-error
Aug 23, 2026
Merged

A storage blip is not "session not found"#545
brentrager merged 2 commits into
mainfrom
th-loadsession-error

Conversation

@brentrager

Copy link
Copy Markdown
Contributor

Problem

AppState::load_session hydrates a session from storage when the local per-pod
registry misses (th-ca579c, #529) — the normal path for a returning visitor
whose WebSocket lands on a pod that has never seen their session. That read
collapsed Err into None:

Ok(None) => None,
Err(e) => {
    // A storage failure is NOT "no such session". Saying so would
    // turn a blip into an existence claim, and the caller renders
    // that to a human as a not-found error.
    tracing::warn!(...);
    None            // ← says exactly that
}

The comment states the rule and the code breaks it. Both outcomes reach
scoped_session, which every session-id-taking handler routes through, and a
None there becomes session '<id>' not found — in the chat bubble of a live
visitor on smoo.ai. A transient Postgres failure was indistinguishable from a
conversation that never existed. Seen in production today.

Shape

load_session returns anyhow::Result<Option<Session>> (matching the
StorageAdapter trait, which is already anyhow::Result), and scoped_session
propagates it. Three outcomes, distinct at the user-visible boundary:

outcome meaning what the user sees
Ok(Some(s)) the session, hydrated + primed unchanged
Ok(None) does not exist or is not yours unchanged not-found event, byte for byte
Err(_) storage could not answer retryable STORAGE_ERROR

Ok(None) keeping both cases fused is deliberate and load-bearing: splitting
"not yours" out would be an existence oracle for enumerating other users'
session ids (th-1b7ed0). The new Err arm leaks nothing — a storage failure is
independent of whether the id is real or ours.

STORAGE_ERROR is the code the rename path already emits for the same class of
failure; the message is fixed text ("session lookup is temporarily unavailable,
please try again") and the underlying error is logged server-side rather than
sent to the client.

What each error site says now

All six route through the one chokepoint, so all six switch together.

action genuinely unknown / not yours storage blip
get_session SESSION_NOT_FOUND (unchanged) STORAGE_ERROR
get_conversation_messages SESSION_NOT_FOUND (unchanged) STORAGE_ERROR
send_message SESSION_NOT_FOUND (unchanged) STORAGE_ERROR
verify_otp SESSION_NOT_FOUND (unchanged) STORAGE_ERROR (gate still fails closed — no code is checked)
confirm_tool_action NO_PENDING_CONFIRMATION (unchanged) STORAGE_ERROR
submit_interaction NO_PENDING_INTERACTION (unchanged) STORAGE_ERROR

The last two mattered as much as the first four: "no tool action is awaiting
confirmation" is also an existence claim, and it is false — the turn is still
parked, and a retry after the blip still approves it. submit_interaction only
peeks the park, so its retry is safe too.

Tests

rust/smooth-operator-server/tests/session_storage_blip.rs — a storage adapter
that delegates to the in-memory one but fails get_session on demand, driven
through the real handler::handle_frame:

  • a_storage_blip_is_never_reported_as_not_found — every one of the six
    actions must emit STORAGE_ERROR, never its not-found code, and never the
    words "not found". Failures accumulate across all six, so one run names every
    handler that regressed.
  • an_unknown_session_is_still_not_found_when_storage_is_healthy — the
    other half. A fix that made everything retryable would leave clients retrying
    an id that will never resolve.
  • a_healthy_hydrate_still_serves_the_session — the th-ca579c path is
    untouched: storage has it, the local registry has never seen it, it serves.

Negative controls, each run and each seen to fail before restoring:

  1. Err(e) => Ok(None) (the old conflation) → blip test fails, naming all six
    actions with the exact prod payload SESSION_NOT_FOUND / "session 's-ghost' not found".
  2. Ok(None) => bail!(...) (everything retryable) → the not-found test fails.
  3. hydration returns Ok(None) for a session storage has → the healthy-hydrate
    test fails.

Gate

cargo test -p smooai-smooth-operator-server green (every integration suite),
cargo fmt --all -- --check clean, cargo clippy --all-targets -- -D warnings
clean.

One unrelated failure on the full-workspace run:
smooai-smooth-operator-adapter-postgrestwo_orgs_are_isolated_on_one_postgres_adapter,
which panicked writing a message to its testcontainer Postgres. The dev box hit
100% disk mid-run and Docker went down with it (other testcontainer suites in
the same run had passed minutes earlier). That crate does not depend on
AppState or handler; this diff cannot reach it.

Not in scope

may_read_conversation has the same conflation on its participants read
(Err(_) => false, which scoped_session then renders as not-found). Unlike
load_session it is documented as deliberate ("an owner check that can't be
completed must not pass"), and un-fusing it means changing the signature for
four more callers, two of them a match guard and a listing loop with their own
right answer. Left alone here.

Shipping

Changeset added (.changeset/session-storage-blip-not-found.md). This crate is
published and smooai's chat-ws pins smooai-smooth-operator-server = "1.57"
without the changeset the fix would merge and never reach prod.

Host-facing API change: AppState::load_session now returns
anyhow::Result<Option<Session>>; hosts calling it directly add a ? or a
match.

load_session collapsed Err into None, so a transient Postgres failure on the
hydration read was indistinguishable from a session that never existed — and
every caller renders None as "session '<id>' not found", which a live visitor
reads in a chat bubble on smoo.ai.

load_session now returns anyhow::Result<Option<Session>> and scoped_session
propagates it: Ok(None) stays the indistinguishable not-found/not-yours answer
(no existence oracle), Err becomes a retryable STORAGE_ERROR at all six
session-id-taking handlers.
…hangeset

- accumulate per-action failures in the blip test so one run names every
  handler that regressed, not just the first
- changeset: this crate is published; without one the fix never ships
@changeset-bot

changeset-bot Bot commented Aug 23, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 905c37e

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 2 packages
Name Type
@smooai/smooth-operator Patch
@smooai/smooth-operator-web-chat-example Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@brentrager
brentrager merged commit 330680e into main Aug 23, 2026
3 checks passed
@brentrager
brentrager deleted the th-loadsession-error branch August 23, 2026 23:07
brentrager added a commit that referenced this pull request Aug 25, 2026
* Anonymous widget visitors were locked out of their own sessions

P0, live on smoo.ai: create_conversation_session { agentId, userEmail }
answered 200, and the very next send_message on the same socket answered
SESSION_NOT_FOUND for a session that existed. userEmail alone was the
trigger; the same create without it streamed fine.

The visitor's email lands on its own `user` participant, which makes the
conversation `owned`. A public widget visitor has no verified principal,
which on a multi-user deployment is UserScope::Denied, whose arm was
`!owned` — so the visitor was owner-checked against an identity it does
not have and locked out of the session it had just created. Its recovery
path created another session carrying the same email and was denied
identically, so real visitors saw an infinite retry loop rather than a
blip. th-909995 recurring for the emailful case, which
anonymous_scope()'s own "the anonymous widget flow keeps working"
comment assumed could not happen.

An anonymous connection can never satisfy an ownership check, so it now
skips that axis — but only for a read it reached BY ID, where the
unguessable id is the visitor's whole capability. Listing stays strict
for everyone: anonymous listing falls back to the SEED org, which is
where widget conversations pool, so widening it there would have leaked
visitors' chats to each other. A negative control caught exactly that
before it shipped. An authenticated principal with no email claim still
fails closed, the tenant boundary is untouched, and the fused
SESSION_NOT_FOUND from #545 still leaks nothing.

Fixed at the one may_read_conversation chokepoint, so all of
send_message, get_session, get_conversation_messages,
confirm_tool_action, submit_interaction, verify_otp and resume are
covered by the one change.

Tests: the create-WITH-userEmail-then-send round trip every existing
test skipped (they covered capture and ownership separately, never the
round trip), plus four negative controls.

* Narrow the anonymous exception to by-id reach, add changeset

A negative control caught the first cut widening list_conversations: an
anonymous connection could enumerate the SEED org's owned conversations,
which is exactly where widget conversations pool, so it would have leaked
visitors' chats to each other. The exception is now Reach::ById only.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant