Add cancel + Rich Interaction conformance scenarios (th-eae69d) - #513
Merged
Conversation
|
brentrager
added a commit
that referenced
this pull request
Aug 19, 2026
Per the ruling on #513: Rust's park ordering is correct (the four ports change, not the spec), and this must not merge red. So the new scenarios now carry an EXPIRING xfail marker that all five runners honour. A scenario names the languages it fails on today, with the reason and the pearl id right beside it: "knownDivergences": ["go", "typescript", "python", "dotnet"], "knownDivergencesReason": "th-eae69d — these four emit the raise tool's toolCall chunk BEFORE interaction_required ..." Both halves of the contract are implemented, and the second is the point: * a listed language that FAILS is reported (reason + the real assertion) and does not fail the build; * a listed language that PASSES FAILS the build — "remove <lang> from knownDivergences in <scenario> — it now passes". Without the xpass half the markers rot silently and we recreate the exact "green tests that prove nothing" problem this corpus exists to catch. A marker is a tracked bug with an expiry, never an accepted difference. Marked strictly to the measured matrix: the five interaction park scenarios for go/typescript/python/dotnet, and cancel-mid-turn for go/dotnet. The sixth interaction scenario (interaction-conversational-fallback) already passes everywhere and is deliberately unmarked. *testing.T and panics do not catch alike, so the five runners differ: rust tokio::spawn, so the scenario's panic surfaces as a JoinError go narrow *testing.T to a small parityT interface, then run a marked scenario against a recorder whose Fatalf panics with the message (testing.T's own Fatalf is terminal and cannot be un-failed) ts/py catch the assertion (vitest / pytest.xfail) dotnet catch XunitException Verified in both directions, per language: with the real corpus all five are green (Rust 18/18, Go/TS/.NET green with divergences logged, Python 13 passed + 5 xfailed), and with a passing scenario temporarily marked for all five, every runner fails with its "it now passes" message. gofmt, cargo fmt, clippy, tsc --noEmit and ruff are clean. Note for anyone iterating locally: `go test` caches results and does NOT invalidate on a scenario-JSON edit — use -count=1. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The shared conformance corpus every server replays had no cancel scenario
and no interaction scenario. Cancellation and Rich Interactions — the two
newest features — were cross-checked only by fixture SHAPE, never by
cross-language BEHAVIOR, so a port could be fully green on parity while
implementing neither correctly. That is why six reviewers independently
found the same divergences in four languages: nothing in CI was positioned
to catch them.
Eight scenarios, test-only, no runner or server changes:
cancel-mid-turn terminal cancelled/499 echoing the
TURN's requestId, then the slot is free
cancel-no-active-turn-noop a stray cancel emits nothing
interaction-park-resume identity_intake parks + resumes
interaction-invalid-retryable invalid values stay PARKED, twice
interaction-stale-id-rejected INTERACTION_MISMATCH, stays parked
interaction-declined declined:true resolves without values
interaction-conversational-fallback no capability -> no park
interaction-choices-park-resume the choices kind, same envelope
A mock turn finishes faster than a cancel can race it and the format has no
slow-tool directive, so cancel-mid-turn opens its in-flight window with a
write-confirmation park — the one pause the corpus can express.
These go in RED on four servers. That is the deliverable: the audit's
systemic findings become CI-visible facts. Two real divergences, detailed
in the README:
* Rust emits interaction_required BEFORE the raise tool's stream_chunk;
Go/TS/Python/.NET emit the chunk first. The scenarios assert Rust's
order — it is the reference, and it is the order the corpus already
pins for the other park type (hitl-write-confirmation), which the
other four also honor, making them internally inconsistent between
their two park paths. Whether the spec adopts chunk-first instead is
a protocol decision, not a test fix.
* Go and .NET keep running a cancelled turn: it consumes one more LLM
response after the cancel and is merely gagged, not aborted.
Cancellation is a mute button there, not a stop button. Confirmed by
re-running with an extra mockLlmScript entry — both then pass, so the
entry is provably eaten by the cancelled turn.
Verified on all five: Rust 18/18. TS, Python 13/18 (the 5 park scenarios).
Go, .NET 12/18 (those 5 plus cancel-mid-turn).
There is no per-language skip/allowlist/xfail mechanism in any of the five
runners and none can be added in JSON, so these land on all five at once.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Per the ruling on #513: Rust's park ordering is correct (the four ports change, not the spec), and this must not merge red. So the new scenarios now carry an EXPIRING xfail marker that all five runners honour. A scenario names the languages it fails on today, with the reason and the pearl id right beside it: "knownDivergences": ["go", "typescript", "python", "dotnet"], "knownDivergencesReason": "th-eae69d — these four emit the raise tool's toolCall chunk BEFORE interaction_required ..." Both halves of the contract are implemented, and the second is the point: * a listed language that FAILS is reported (reason + the real assertion) and does not fail the build; * a listed language that PASSES FAILS the build — "remove <lang> from knownDivergences in <scenario> — it now passes". Without the xpass half the markers rot silently and we recreate the exact "green tests that prove nothing" problem this corpus exists to catch. A marker is a tracked bug with an expiry, never an accepted difference. Marked strictly to the measured matrix: the five interaction park scenarios for go/typescript/python/dotnet, and cancel-mid-turn for go/dotnet. The sixth interaction scenario (interaction-conversational-fallback) already passes everywhere and is deliberately unmarked. *testing.T and panics do not catch alike, so the five runners differ: rust tokio::spawn, so the scenario's panic surfaces as a JoinError go narrow *testing.T to a small parityT interface, then run a marked scenario against a recorder whose Fatalf panics with the message (testing.T's own Fatalf is terminal and cannot be un-failed) ts/py catch the assertion (vitest / pytest.xfail) dotnet catch XunitException Verified in both directions, per language: with the real corpus all five are green (Rust 18/18, Go/TS/.NET green with divergences logged, Python 13 passed + 5 xfailed), and with a passing scenario temporarily marked for all five, every runner fails with its "it now passes" message. gofmt, cargo fmt, clippy, tsc --noEmit and ruff are clean. Note for anyone iterating locally: `go test` caches results and does NOT invalidate on a scenario-JSON edit — use -count=1. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
CI runs `go test -race`, and cancel-mid-turn trips it: a DATA RACE in core's MockLlmProvider.ChatStream, where the CANCELLED turn's goroutine and the NEXT turn's goroutine pop the same unguarded FIFO script concurrently. That is independent, mechanical proof of the divergence the scenario's assertion already catches — the cancelled turn is genuinely still executing, not merely mis-reporting. Reproduces deterministically, locally and in CI. A knownDivergences marker cannot and must not suppress it: the race detector fails the test outside the runner's assertion path. So vet-test (go/server) stays red until Go actually aborts a cancelled turn. Documented rather than worked around — guarding the mock's FIFO would silence the evidence and leave the bug. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The go marker on cancel-mid-turn bought nothing and would have backfired. It covers only the assertion; the data race (th-f2ac48) fails the test outside the runner's assertion path, so vet-test (go/server) was red with the marker anyway. And once fix-cancel-unwind lands, Go would start PASSING the assertion — firing the xpass guard and re-reding the build with 'remove go from knownDivergences' instead of going green. Unmarked, cancel-mid-turn turns green on its own the moment Go actually unwinds a cancelled turn. That is the proof the port fix worked. .NET stays marked: it has no race, so the marker is what keeps build-test green today, and its xpass guard is what will tell us when it is fixed. Also spells out in the README that hitl-write-confirmation — the scenario proving the four ports are internally inconsistent with their own other park path — is one ALL FIVE pass today, so the ordering ruling does not get re-litigated by the next reader. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
#514 made Go and .NET actually unwind a cancelled turn, and the corpus proved it twice, unaided: * Go was UNMARKED and went green on its own, scenario untouched, race detector included. The corpus said Go was broken, the port was fixed, and the corpus now agrees without anyone editing it. * .NET was marked, so the xpass guard FIRED on its first real opportunity: 'remove dotnet from knownDivergences in cancel-mid-turn — it now passes'. Marker removed; .NET is back to 19/19. That is the whole point of an expiring marker over a skip: it told us the moment it became a lie, instead of rotting into a green check that proved nothing. The five interaction-ordering markers (th-ef78d0) are untouched — those still genuinely fail and the ruling stands: Rust's order is correct. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
brentrager
force-pushed
the
test/conformance-cancel-interaction-scenarios
branch
from
August 19, 2026 23:39
a69729e to
9d87c3c
Compare
brentrager
added a commit
that referenced
this pull request
Aug 20, 2026
Mirrors the Rust reference in the preceding commit. Go, Python and .NET each kept the declared render capabilities in a per-connection map on the dispatcher (`FrameDispatcher.supports` / `_session_supports` / `_sessionSupports`), which a reconnect wipes and which nothing ever pruned. TypeScript already routed the value through its `SessionStore`, but stored it on the SESSION record — and a resume mints a new session, so it started empty just the same. Each port now persists it per CONVERSATION through the mechanism its own store already used for conversation-scoped facts, rather than a fifth invented one: Go adds `SetConversationSupports` beside `SetCurrentStep`, Python adds `get/set_client_supports` beside the workflow-step pointer, TypeScript adds a `convSupports` map beside `convOwner`/`convOrg`, .NET adds `Get/SetClientSupportsAsync` beside `Get/SetWorkflowStepAsync`. Every implementation of each store interface is updated, in-memory and Postgres. The per-connection maps are DELETED rather than kept as caches, so there is one source of truth and the leak goes with them. Distinguishing an omitted `supports` from an explicit `[]` is load-bearing — omitted inherits, `[]` replaces — and three of the four collapsed them. Go's frame field becomes `*[]string` (json.Unmarshal gives nil for both), .NET's `ParseSupports` returns `IReadOnlyList<string>?`, and the TS stores stopped dropping an empty list on the floor. Correction to the previous commit message: it claimed the four ports never parse `supports` and host no interactions framework. That was true of the commit this work branched from and is false on current main — #505/#509/#513/#520 landed the framework in Go, TypeScript, Python and .NET in the meantime. The pearl's original diagnosis was right; the rebase is what surfaced it. Each port adds a reconnect test that drives a SECOND, FRESH dispatcher over the same store — a single dispatcher would pass even with per-connection state — and each was verified to fail against its own pre-fix code before being kept. Verified: go build/vet/test + gofmt (go and go/server modules), ruff check + format + pytest (389), tsc + vitest (390), dotnet build + test (642 across five assemblies). Postgres-backed suites really ran; Docker was up.
brentrager
added a commit
that referenced
this pull request
Aug 21, 2026
* th-13df6d: keep Rich Interactions alive across a reconnect
`supports` — the client render-capability list that gates the entire Rich
Interactions framework — lived only on the session
(`Session.metadata.supports`, read by `AppState::session_capabilities`). A
reconnect IS a resume: the client re-opens the socket and re-issues
`create_conversation_session` with the same `conversationId`, which mints a NEW
session id. So unless the client re-declared `supports` every single time, the
server forgot it could render cards and every interaction kind quietly fell back
to conversational collection — no error, no event, nothing on the wire to
notice. Reconnects are routine (network blips, mobile backgrounding, deploys),
so a shipped feature was degrading in the field with no signal.
The session registry was already the wrong home, and this repo had said so once
before: th-c12df5 moved the workflow step pointer off it for exactly this reason
("this per-pod session map resets on reconnect/pod hop"). `supports` now rides
durable conversation metadata (`clientSupports`) the same way — same
read-modify-write shape as `persist_workflow_step`, same best-effort failure
mode — and a resume that omits the key inherits what the conversation last
declared.
A list the frame DOES declare always wins, including `[]`. That is now how a
text-only channel resuming a rich conversation opts out, so the spec's
`supports` description carries the rule and the generated TS/Go/Python/.NET
types are regenerated from it rather than restating it by hand. The inherit
direction is bounded anyway: a card a client cannot render times out
(`INTERACTION_TIMEOUT`) into the same conversational fallback the gate would
have chosen.
Verified across all five implementations first. The pearl's premise that the
four ports keep `supports` in a per-connection map does not hold — Go,
TypeScript, Python and .NET never parse `supports` at all and host no
interactions framework (`interaction_required` / `submit_interaction` /
`identity_form` appear only in their generated wire types), so there is nothing
there to persist yet. Rust is the only implementation with the behavior, so it
is the only one changed.
`reconnect_resuming_a_conversation_keeps_the_declared_capabilities` covers both
directions and fails on the pre-fix handler (verified by reverting).
* th-13df6d: move `supports` onto the conversation in all four ports
Mirrors the Rust reference in the preceding commit. Go, Python and .NET each
kept the declared render capabilities in a per-connection map on the dispatcher
(`FrameDispatcher.supports` / `_session_supports` / `_sessionSupports`), which a
reconnect wipes and which nothing ever pruned. TypeScript already routed the
value through its `SessionStore`, but stored it on the SESSION record — and a
resume mints a new session, so it started empty just the same.
Each port now persists it per CONVERSATION through the mechanism its own store
already used for conversation-scoped facts, rather than a fifth invented one:
Go adds `SetConversationSupports` beside `SetCurrentStep`, Python adds
`get/set_client_supports` beside the workflow-step pointer, TypeScript adds a
`convSupports` map beside `convOwner`/`convOrg`, .NET adds
`Get/SetClientSupportsAsync` beside `Get/SetWorkflowStepAsync`. Every
implementation of each store interface is updated, in-memory and Postgres.
The per-connection maps are DELETED rather than kept as caches, so there is one
source of truth and the leak goes with them.
Distinguishing an omitted `supports` from an explicit `[]` is load-bearing —
omitted inherits, `[]` replaces — and three of the four collapsed them. Go's
frame field becomes `*[]string` (json.Unmarshal gives nil for both), .NET's
`ParseSupports` returns `IReadOnlyList<string>?`, and the TS stores stopped
dropping an empty list on the floor.
Correction to the previous commit message: it claimed the four ports never parse
`supports` and host no interactions framework. That was true of the commit this
work branched from and is false on current main — #505/#509/#513/#520 landed the
framework in Go, TypeScript, Python and .NET in the meantime. The pearl's
original diagnosis was right; the rebase is what surfaced it.
Each port adds a reconnect test that drives a SECOND, FRESH dispatcher over the
same store — a single dispatcher would pass even with per-connection state — and
each was verified to fail against its own pre-fix code before being kept.
Verified: go build/vet/test + gofmt (go and go/server modules), ruff check +
format + pytest (389), tsc + vitest (390), dotnet build + test (642 across five
assemblies). Postgres-backed suites really ran; Docker was up.
* th-13df6d: don't float a reconnect to the top of the conversation sidebar
The TypeScript store's capability write also set conversations.updated_at.
That column is the sidebar's recency sort and is bumped when a MESSAGE lands
(appendMessage); a bare reconnect appends nothing, so this floated every
backgrounded tab to the top of the list on resume. The Go, Python and Rust
stores all leave it alone — a parity-relevant choice, now commented as one.
* th-13df6d: correct the .NET store's divergence note — the data is NOT in the shared place
The <remarks> claimed 'the DATA lives in the shared place under the shared keys
either way'. Only the key names are shared: this store writes
conversation_sessions.metadata, the other four write conversations.metadata_json
(rust/adapters/postgres/src/lib.rs:618). A note that asserts the divergence away
is worse than no note, since it is exactly what someone would rely on before
pointing both at one database.
States the real divergence, that nothing gates it (knownDivergences was retired
in #523, and a test pinning the current table would lock the drift in), and that
moving ONE key would split this store's three across two tables — strictly worse.
Unification tracked as th-52becd.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The gap
spec/conformance/scenarios/is the corpus all five servers replay. It had no cancel scenario, no interaction scenario, no choices scenario.So cancellation and Rich Interactions — the two newest features — were cross-checked only by fixture shape (does an
interaction_requiredevent match its schema), never by cross-language behavior. A port could be fully green on parity while implementing neither correctly. That is exactly what happened: six independent reviewers found the same divergences in four different languages because nothing in CI was positioned to catch them.What this adds
Eight scenarios, plus an expiring xfail marker the five runners honour. Test-only — no server changes, no changeset.
cancel-mid-turncancelled(499, echoing the turn's requestId) in place ofeventual_response; turn slot freedcancel-no-active-turn-noopcancelwith no active turn emits nothinginteraction-park-resumeidentity_intakeparks behindidentity_form; matchingsubmit_interactionresumesinteraction-invalid-retryableinteraction_invalid, turn stays parked (twice), then a corrected submit resumesinteraction-stale-id-rejectedinteractionIdthenerror/INTERACTION_MISMATCH, turn stays parkedinteraction-declineddeclined: trueresolves the park without valuesinteraction-conversational-fallbackinteraction-choices-park-resumechoiceskind rides the same generic envelopeTwo design notes:
cancelframe can race it, and the format has no slow-tool directive (server.toolsentries return a fixed string immediately).cancel-mid-turnopens its in-flight window with a write-confirmation park — the one pause this corpus can express.cancel-no-active-turn-noopdoes it structurally: the cancel step expects zero events, so any stray event is consumed by the next step's first matcher and fails it.knownDivergences— an expiring marker, not a skipThere was no skip/allowlist/xfail mechanism in any runner, so this PR adds one. A scenario names the languages it fails on today, with the reason and pearl id right beside it:
Both halves of the contract are implemented, and the second is the point:
remove <lang> from knownDivergences in <scenario> — it now passes.Without the xpass half the markers rot silently and we recreate the exact "green tests that prove nothing" problem this corpus exists to catch. A marker is a tracked bug with an expiry, never an accepted difference.
Marked strictly to the measured matrix — the five interaction park scenarios for go/typescript/python/dotnet, and
cancel-mid-turnfor go/dotnet.interaction-conversational-fallbackalready passes everywhere and is deliberately unmarked.*testing.Tand panics do not catch alike, so the runners differ: Rust usestokio::spawnso the panic surfaces as aJoinError; Go narrows*testing.Tto a smallparityTinterface and runs a marked scenario against a recorder whoseFatalfpanics with the message (testing.T's ownFatalfis terminal and cannot be un-failed); TypeScript, Python and .NET catch the assertion.The two divergences, exactly
1. Park event vs. the raise tool's
stream_chunk— Rust is 1 of 5, and Rust is rightFailing assertion, identical in four languages:
expected interaction_required, got stream_chunk.Ruled a port bug, not a protocol variant, on three grounds:
hitl-write-confirmation) — so the four are internally inconsistent between their two park paths, while Rust is consistent.request_identity_intake..." before the card appears — framework internals leaking ahead of the semantic event.The four ports change, not these scenarios.
2. Go and .NET keep running a cancelled turn
Failing assertion in
cancel-mid-turn:stream_token: accumulated "" != "The turn slot was free, so this second turn ran."The turn after the cancel produces no reply because the cancelled turn consumed an extra LLM response. In Go the write-confirmation gate returns a deny rather than unwinding (
return core.Deny("connection closed before confirmation"), gateCtx.Err()), the agent loop makes one more model call, and the output is merely suppressed (if turnCtx.Err() != nil { return }). .NET has the same signature.Verified, not inferred: re-running with one extra
mockLlmScriptentry makes both pass, proving the entry is eaten by the cancelled turn.Cancellation is a mute button there, not a stop button — after a visitor hits Stop, the turn keeps burning model calls and keeps executing whatever the agent loop does next, invisibly. Rust, TypeScript and Python abort properly. Corroborates th-8628bf independently; filed against the ports.
Verification
Against
origin/main(f91008e), both directions, per language.Real corpus — all five green:
cargo test --test scenario_paritygo test -count=1 -run TestScenarioParityvitest run test/scenario-parity.test.tsuv run pytest tests/test_scenario_parity.pydotnet test --filter ~ScenarioParityxpass guard — with a passing scenario temporarily marked for all five, every runner fails with its "it now passes" message. All ten pre-existing scenarios still pass on all five.
gofmt,cargo fmt,cargo clippy,tsc --noEmitandruffare clean.🤖 Generated with Claude Code