A storage blip is not "session not found" - #545
Merged
Conversation
load_session collapsed Err into None, so a transient Postgres failure on the hydration read was indistinguishable from a session that never existed — and every caller renders None as "session '<id>' not found", which a live visitor reads in a chat bubble on smoo.ai. load_session now returns anyhow::Result<Option<Session>> and scoped_session propagates it: Ok(None) stays the indistinguishable not-found/not-yours answer (no existence oracle), Err becomes a retryable STORAGE_ERROR at all six session-id-taking handlers.
…hangeset - accumulate per-action failures in the blip test so one run names every handler that regressed, not just the first - changeset: this crate is published; without one the fix never ships
🦋 Changeset detectedLatest commit: 905c37e The changes in this PR will be included in the next version bump. This PR includes changesets to release 2 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
brentrager
added a commit
that referenced
this pull request
Aug 25, 2026
* Anonymous widget visitors were locked out of their own sessions
P0, live on smoo.ai: create_conversation_session { agentId, userEmail }
answered 200, and the very next send_message on the same socket answered
SESSION_NOT_FOUND for a session that existed. userEmail alone was the
trigger; the same create without it streamed fine.
The visitor's email lands on its own `user` participant, which makes the
conversation `owned`. A public widget visitor has no verified principal,
which on a multi-user deployment is UserScope::Denied, whose arm was
`!owned` — so the visitor was owner-checked against an identity it does
not have and locked out of the session it had just created. Its recovery
path created another session carrying the same email and was denied
identically, so real visitors saw an infinite retry loop rather than a
blip. th-909995 recurring for the emailful case, which
anonymous_scope()'s own "the anonymous widget flow keeps working"
comment assumed could not happen.
An anonymous connection can never satisfy an ownership check, so it now
skips that axis — but only for a read it reached BY ID, where the
unguessable id is the visitor's whole capability. Listing stays strict
for everyone: anonymous listing falls back to the SEED org, which is
where widget conversations pool, so widening it there would have leaked
visitors' chats to each other. A negative control caught exactly that
before it shipped. An authenticated principal with no email claim still
fails closed, the tenant boundary is untouched, and the fused
SESSION_NOT_FOUND from #545 still leaks nothing.
Fixed at the one may_read_conversation chokepoint, so all of
send_message, get_session, get_conversation_messages,
confirm_tool_action, submit_interaction, verify_otp and resume are
covered by the one change.
Tests: the create-WITH-userEmail-then-send round trip every existing
test skipped (they covered capture and ownership separately, never the
round trip), plus four negative controls.
* Narrow the anonymous exception to by-id reach, add changeset
A negative control caught the first cut widening list_conversations: an
anonymous connection could enumerate the SEED org's owned conversations,
which is exactly where widget conversations pool, so it would have leaked
visitors' chats to each other. The exception is now Reach::ById only.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
AppState::load_sessionhydrates a session from storage when the local per-podregistry misses (th-ca579c, #529) — the normal path for a returning visitor
whose WebSocket lands on a pod that has never seen their session. That read
collapsed
ErrintoNone:The comment states the rule and the code breaks it. Both outcomes reach
scoped_session, which every session-id-taking handler routes through, and aNonethere becomessession '<id>' not found— in the chat bubble of a livevisitor on smoo.ai. A transient Postgres failure was indistinguishable from a
conversation that never existed. Seen in production today.
Shape
load_sessionreturnsanyhow::Result<Option<Session>>(matching theStorageAdaptertrait, which is alreadyanyhow::Result), andscoped_sessionpropagates it. Three outcomes, distinct at the user-visible boundary:
Ok(Some(s))Ok(None)Err(_)STORAGE_ERROROk(None)keeping both cases fused is deliberate and load-bearing: splitting"not yours" out would be an existence oracle for enumerating other users'
session ids (th-1b7ed0). The new
Errarm leaks nothing — a storage failure isindependent of whether the id is real or ours.
STORAGE_ERRORis the code the rename path already emits for the same class offailure; the message is fixed text ("session lookup is temporarily unavailable,
please try again") and the underlying error is logged server-side rather than
sent to the client.
What each error site says now
All six route through the one chokepoint, so all six switch together.
get_sessionSESSION_NOT_FOUND(unchanged)STORAGE_ERRORget_conversation_messagesSESSION_NOT_FOUND(unchanged)STORAGE_ERRORsend_messageSESSION_NOT_FOUND(unchanged)STORAGE_ERRORverify_otpSESSION_NOT_FOUND(unchanged)STORAGE_ERROR(gate still fails closed — no code is checked)confirm_tool_actionNO_PENDING_CONFIRMATION(unchanged)STORAGE_ERRORsubmit_interactionNO_PENDING_INTERACTION(unchanged)STORAGE_ERRORThe last two mattered as much as the first four: "no tool action is awaiting
confirmation" is also an existence claim, and it is false — the turn is still
parked, and a retry after the blip still approves it.
submit_interactiononlypeeks the park, so its retry is safe too.
Tests
rust/smooth-operator-server/tests/session_storage_blip.rs— a storage adapterthat delegates to the in-memory one but fails
get_sessionon demand, driventhrough the real
handler::handle_frame:a_storage_blip_is_never_reported_as_not_found— every one of the sixactions must emit
STORAGE_ERROR, never its not-found code, and never thewords "not found". Failures accumulate across all six, so one run names every
handler that regressed.
an_unknown_session_is_still_not_found_when_storage_is_healthy— theother half. A fix that made everything retryable would leave clients retrying
an id that will never resolve.
a_healthy_hydrate_still_serves_the_session— the th-ca579c path isuntouched: storage has it, the local registry has never seen it, it serves.
Negative controls, each run and each seen to fail before restoring:
Err(e) => Ok(None)(the old conflation) → blip test fails, naming all sixactions with the exact prod payload
SESSION_NOT_FOUND / "session 's-ghost' not found".Ok(None) => bail!(...)(everything retryable) → the not-found test fails.Ok(None)for a session storage has → the healthy-hydratetest fails.
Gate
cargo test -p smooai-smooth-operator-servergreen (every integration suite),cargo fmt --all -- --checkclean,cargo clippy --all-targets -- -D warningsclean.
One unrelated failure on the full-workspace run:
smooai-smooth-operator-adapter-postgres→two_orgs_are_isolated_on_one_postgres_adapter,which panicked writing a message to its testcontainer Postgres. The dev box hit
100% disk mid-run and Docker went down with it (other testcontainer suites in
the same run had passed minutes earlier). That crate does not depend on
AppStateorhandler; this diff cannot reach it.Not in scope
may_read_conversationhas the same conflation on its participants read(
Err(_) => false, whichscoped_sessionthen renders as not-found). Unlikeload_sessionit is documented as deliberate ("an owner check that can't becompleted must not pass"), and un-fusing it means changing the signature for
four more callers, two of them a match guard and a listing loop with their own
right answer. Left alone here.
Shipping
Changeset added (
.changeset/session-storage-blip-not-found.md). This crate ispublished and smooai's chat-ws pins
smooai-smooth-operator-server = "1.57"—without the changeset the fix would merge and never reach prod.
Host-facing API change:
AppState::load_sessionnow returnsanyhow::Result<Option<Session>>; hosts calling it directly add a?or amatch.