From c303fa6a1f1e6d00e11d2772294245624f36d291 Mon Sep 17 00:00:00 2001 From: Brent Rager Date: Wed, 19 Aug 2026 18:46:36 -0400 Subject: [PATCH 1/5] th-eae69d: add cancel + Rich Interaction conformance scenarios MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The shared conformance corpus every server replays had no cancel scenario and no interaction scenario. Cancellation and Rich Interactions — the two newest features — were cross-checked only by fixture SHAPE, never by cross-language BEHAVIOR, so a port could be fully green on parity while implementing neither correctly. That is why six reviewers independently found the same divergences in four languages: nothing in CI was positioned to catch them. Eight scenarios, test-only, no runner or server changes: cancel-mid-turn terminal cancelled/499 echoing the TURN's requestId, then the slot is free cancel-no-active-turn-noop a stray cancel emits nothing interaction-park-resume identity_intake parks + resumes interaction-invalid-retryable invalid values stay PARKED, twice interaction-stale-id-rejected INTERACTION_MISMATCH, stays parked interaction-declined declined:true resolves without values interaction-conversational-fallback no capability -> no park interaction-choices-park-resume the choices kind, same envelope A mock turn finishes faster than a cancel can race it and the format has no slow-tool directive, so cancel-mid-turn opens its in-flight window with a write-confirmation park — the one pause the corpus can express. These go in RED on four servers. That is the deliverable: the audit's systemic findings become CI-visible facts. Two real divergences, detailed in the README: * Rust emits interaction_required BEFORE the raise tool's stream_chunk; Go/TS/Python/.NET emit the chunk first. The scenarios assert Rust's order — it is the reference, and it is the order the corpus already pins for the other park type (hitl-write-confirmation), which the other four also honor, making them internally inconsistent between their two park paths. Whether the spec adopts chunk-first instead is a protocol decision, not a test fix. * Go and .NET keep running a cancelled turn: it consumes one more LLM response after the cancel and is merely gagged, not aborted. Cancellation is a mute button there, not a stop button. Confirmed by re-running with an extra mockLlmScript entry — both then pass, so the entry is provably eaten by the cancelled turn. Verified on all five: Rust 18/18. TS, Python 13/18 (the 5 park scenarios). Go, .NET 12/18 (those 5 plus cancel-mid-turn). There is no per-language skip/allowlist/xfail mechanism in any of the five runners and none can be added in JSON, so these land on all five at once. Co-Authored-By: Claude Fable 5 --- spec/conformance/scenarios/README.md | 38 ++++++++++++- .../scenarios/cancel-mid-turn.json | 47 +++++++++++++++ .../scenarios/cancel-no-active-turn-noop.json | 15 +++++ .../interaction-choices-park-resume.json | 42 ++++++++++++++ .../interaction-conversational-fallback.json | 24 ++++++++ .../scenarios/interaction-declined.json | 35 ++++++++++++ .../interaction-invalid-retryable.json | 57 +++++++++++++++++++ .../scenarios/interaction-park-resume.json | 44 ++++++++++++++ .../interaction-stale-id-rejected.json | 38 +++++++++++++ 9 files changed, 339 insertions(+), 1 deletion(-) create mode 100644 spec/conformance/scenarios/cancel-mid-turn.json create mode 100644 spec/conformance/scenarios/cancel-no-active-turn-noop.json create mode 100644 spec/conformance/scenarios/interaction-choices-park-resume.json create mode 100644 spec/conformance/scenarios/interaction-conversational-fallback.json create mode 100644 spec/conformance/scenarios/interaction-declined.json create mode 100644 spec/conformance/scenarios/interaction-invalid-retryable.json create mode 100644 spec/conformance/scenarios/interaction-park-resume.json create mode 100644 spec/conformance/scenarios/interaction-stale-id-rejected.json diff --git a/spec/conformance/scenarios/README.md b/spec/conformance/scenarios/README.md index 9375b3ea..30ce371a 100644 --- a/spec/conformance/scenarios/README.md +++ b/spec/conformance/scenarios/README.md @@ -76,6 +76,42 @@ Each server provides a small test that, for every `*.json` here: The **Python reference runner** is [`python/server/tests/test_scenario_parity.py`](../../../python/server/tests/test_scenario_parity.py) — port its ~80 lines into the TS/Go/C#/Rust server suites. When all five run this corpus green, the servers are at protocol parity. +## Cancellation and Rich Interactions — the newest scenarios, and where the servers disagree + +Until pearl **th-eae69d** this corpus had no `cancel` scenario and no `interaction` scenario. That was the audit's headline finding: cancellation and Rich Interactions — the two newest features — were cross-checked only by fixture *shape* (does an `interaction_required` event match its schema), never by cross-language *behavior*. A port could be fully green on parity while implementing neither correctly, and six independent reviewers found the same divergences in four different languages because nothing in CI was positioned to catch them. + +Eight scenarios now cover them: + +| scenario | what it pins | +|---|---| +| `cancel-mid-turn` | a cancelled turn emits terminal `cancelled` (499, echoing the **turn's** requestId) in place of `eventual_response`, and frees the turn slot | +| `cancel-no-active-turn-noop` | a `cancel` with no active turn emits **nothing** | +| `interaction-park-resume` | `identity_intake` parks behind the `identity_form` capability; a matching `submit_interaction` resumes it | +| `interaction-invalid-retryable` | invalid values → `interaction_invalid`, turn **stays parked**, twice, then a corrected submit resumes | +| `interaction-stale-id-rejected` | a stale `interactionId` → `error`/`INTERACTION_MISMATCH`, turn stays parked | +| `interaction-declined` | `declined: true` resolves the park without values | +| `interaction-conversational-fallback` | a session that did NOT declare the capability gets the text fallback, never a park | +| `interaction-choices-park-resume` | the second kind (`choices`) rides the same generic envelope | + +### Making cancellation deterministic + +A mock turn finishes faster than a `cancel` frame can race it, and the format has no "slow tool" directive (`server.tools` entries return a fixed string immediately). `cancel-mid-turn` therefore opens its in-flight window with a **write-confirmation park** (`server.confirmTools`) — the one pause this corpus can express. `cancel-no-active-turn-noop` asserts "nothing arrives" structurally, since no runner has a drain check: the cancel step expects zero events, so any stray event is consumed by the *next* step's first matcher and fails it. + +### Known divergences these scenarios expose + +Recorded here as facts, not as license to weaken the scenarios. **Do not "fix" a scenario to make a port pass.** + +- **Park event vs. the raise tool's `stream_chunk` — Rust is 1 of 5.** For a Rich Interaction, Rust emits `interaction_required` *before* the raise tool's `toolCall` chunk; Go, TypeScript, Python and .NET all emit the chunk first. The five scenarios that park assert **Rust's** order, because Rust is the reference and because it is the order this corpus already pins for the other park type (`hitl-write-confirmation` defers the gated tool's chunk until after the prompt) — the other four are internally inconsistent between their two park paths. Whether the spec should adopt chunk-first instead is a **protocol decision, not a test fix**; until it is made, four ports are red on these five scenarios. +- **A cancelled turn keeps running in Go and .NET.** In both, the turn after a `cancel` produces no reply because the *cancelled* turn consumed an extra LLM response: the write-confirmation gate returns a deny instead of unwinding, the agent loop makes one more model call, and the output is merely gagged (Go: `if turnCtx.Err() != nil { return }`). Cancellation is a mute button there, not a stop button — a real cost and a real side-effect risk after a visitor hits Stop. Rust, TypeScript and Python abort the turn properly. Verified by re-running with one extra `mockLlmScript` entry: both pass, proving the entry is eaten by the cancelled turn. +- **Ack payloads differ, so only `status` is asserted on a `submit_interaction` ack.** The five servers put different fields in `data` (Go omits `kind`/`values`; Python omits `kind`, and its decline ack omits `interactionId`/`declined`; .NET's decline ack omits `declined`). Asserting more would pin one language's shape rather than the protocol's. + ## Adding a scenario -Drop a `*.json` here; every server's runner picks it up automatically. Cover: multi-turn, tool-call + `confirm_tool_action` (HITL), citations, auth gating, error frames, and graceful-drain (cancel mid-turn → the turn still finishes). +Drop a `*.json` here; every server's runner picks it up automatically — there is **no per-language skip, allowlist or xfail mechanism in any of the five runners**, and no way to add one in JSON (unknown keys are silently ignored everywhere). A new scenario lands on all five simultaneously; gating one would mean editing four runners. + +Two portability rules that bite: + +- **Never assert a fixed number of `stream_token` events** — the mocks chunk text differently per language. Always `repeat` + `accumulate` + `assertAccumulated`, and only on the top-level `token` field. +- **Never assert `null`** to mean "field absent" — .NET's dot-path resolver returns `null` for a missing final segment while the other four fail, so such an assertion passes on exactly one server. + +Still uncovered: auth gating, and graceful-drain (disconnect mid-turn → the turn still finishes). diff --git a/spec/conformance/scenarios/cancel-mid-turn.json b/spec/conformance/scenarios/cancel-mid-turn.json new file mode 100644 index 00000000..e5f81c7e --- /dev/null +++ b/spec/conformance/scenarios/cancel-mid-turn.json @@ -0,0 +1,47 @@ +{ + "name": "cancel-mid-turn", + "description": "A turn cancelled while in flight must emit the terminal `cancelled` event (status 499, echoing the CANCELLED TURN's requestId) IN PLACE OF `eventual_response`, and must free the connection's turn slot so the next `send_message` is accepted. The in-flight window is made deterministic by parking the turn on write-confirmation HITL (`server.confirmTools`) — a mock turn otherwise completes faster than a `cancel` frame can race it, and the scenario format has no way to express a slow tool. Per spec/actions/cancel.schema.json the cancel frame carries the requestId of the `send_message` it aborts; per spec/events/cancelled.schema.json the cancelled turn produces NO answer payload.", + "server": { + "tools": [ + { + "name": "delete_record", + "description": "Delete a record by id (a state-mutating write).", + "parameters": { "type": "object", "properties": { "id": { "type": "string" } }, "required": ["id"] }, + "result": "Record 42 deleted." + } + ], + "confirmTools": ["delete_record"] + }, + "mockLlmScript": [ + { "kind": "toolCall", "name": "delete_record", "arguments": "{\"id\": \"42\"}" }, + { "kind": "text", "text": "The turn slot was free, so this second turn ran." } + ], + "steps": [ + { + "send": { "action": "create_conversation_session", "requestId": "r-create", "agentId": "11111111-1111-1111-1111-111111111111", "userName": "Alice", "userEmail": "alice@example.com" }, + "expect": [ { "type": "immediate_response", "status": 200, "capture": { "sessionId": "data.sessionId" } } ] + }, + { + "send": { "action": "send_message", "requestId": "r-msg-1", "sessionId": "{{sessionId}}", "message": "delete record 42", "stream": true }, + "expect": [ + { "type": "immediate_response", "status": 202 }, + { "type": "write_confirmation_required", "assert": { "requestId": "r-msg-1", "data.data.toolId": "delete_record" } }, + { "type": "stream_chunk", "assert": { "data.state.rawResponse.toolCall.name": "delete_record" } } + ] + }, + { + "send": { "action": "cancel", "requestId": "r-msg-1", "sessionId": "{{sessionId}}" }, + "expect": [ + { "type": "cancelled", "status": 499, "assert": { "requestId": "r-msg-1", "data.requestId": "r-msg-1", "data.status": 499 } } + ] + }, + { + "send": { "action": "send_message", "requestId": "r-msg-2", "sessionId": "{{sessionId}}", "message": "never mind, say something", "stream": true }, + "expect": [ + { "type": "immediate_response", "status": 202 }, + { "type": "stream_token", "repeat": true, "accumulate": "token", "assertAccumulated": "The turn slot was free, so this second turn ran." }, + { "type": "eventual_response", "status": 200, "assert": { "data.data.response.responseParts": ["The turn slot was free, so this second turn ran."] } } + ] + } + ] +} diff --git a/spec/conformance/scenarios/cancel-no-active-turn-noop.json b/spec/conformance/scenarios/cancel-no-active-turn-noop.json new file mode 100644 index 00000000..d1e580c8 --- /dev/null +++ b/spec/conformance/scenarios/cancel-no-active-turn-noop.json @@ -0,0 +1,15 @@ +{ + "name": "cancel-no-active-turn-noop", + "description": "A `cancel` with no active turn is a SILENT no-op: it must emit nothing at all — not a `cancelled` event, not an error, not an ack. The corpus format cannot assert 'nothing arrives', so this asserts it structurally: the cancel step expects zero events, and the very next frame's response must be the FIRST thing on the wire. If a server emitted anything for the stray cancel, the following `create_conversation_session` matcher consumes that stray event instead of the `immediate_response` and fails. Guards the cancel path against phantom terminal events that would reset a client's UI mid-conversation.", + "mockLlmScript": [], + "steps": [ + { + "send": { "action": "cancel", "requestId": "r-stray" }, + "expect": [] + }, + { + "send": { "action": "create_conversation_session", "requestId": "r-create", "agentId": "11111111-1111-1111-1111-111111111111", "userName": "Alice", "userEmail": "alice@example.com" }, + "expect": [ { "type": "immediate_response", "status": 200, "capture": { "sessionId": "data.sessionId" } } ] + } + ] +} diff --git a/spec/conformance/scenarios/interaction-choices-park-resume.json b/spec/conformance/scenarios/interaction-choices-park-resume.json new file mode 100644 index 00000000..97d85614 --- /dev/null +++ b/spec/conformance/scenarios/interaction-choices-park-resume.json @@ -0,0 +1,42 @@ +{ + "name": "interaction-choices-park-resume", + "description": "The second Rich Interaction kind, `choices` (the AskUserQuestion-shaped multiple-choice ask), through the SAME generic envelope: `request_choices` parks behind the `choice_chips` capability, the server emits `interaction_required` with the normalized questions/options spec, and one `submit_interaction` resumes it. Adding a kind must require no new protocol action and no client release — this scenario is what proves that claim holds identically in every server, and that the kind catalog is not identity_intake-shaped by accident.", + "mockLlmScript": [ + { "kind": "toolCall", "name": "request_choices", "arguments": "{\"questions\": [{\"question\": \"Which plan fits best?\", \"header\": \"Plan\", \"options\": [{\"label\": \"Starter\"}, {\"label\": \"Pro\"}]}], \"reason\": \"to route you to the right quote\"}" }, + { "kind": "text", "text": "Pro it is — pulling that quote up." } + ], + "steps": [ + { + "send": { "action": "create_conversation_session", "requestId": "r-create", "agentId": "11111111-1111-1111-1111-111111111111", "userName": "Alice", "userEmail": "alice@example.com", "supports": ["choice_chips"] }, + "expect": [ { "type": "immediate_response", "status": 200, "capture": { "sessionId": "data.sessionId" } } ] + }, + { + "send": { "action": "send_message", "requestId": "r-msg", "sessionId": "{{sessionId}}", "message": "i want a quote", "stream": true }, + "expect": [ + { "type": "immediate_response", "status": 202 }, + { + "type": "interaction_required", + "assert": { + "requestId": "r-msg", + "data.data.kind": "choices", + "data.data.reason": "to route you to the right quote", + "data.data.spec.questions.0.header": "Plan", + "data.data.spec.questions.0.question": "Which plan fits best?", + "data.data.spec.questions.0.options.1.label": "Pro" + }, + "capture": { "interactionId": "data.data.interactionId" } + }, + { "type": "stream_chunk", "assert": { "data.state.rawResponse.toolCall.name": "request_choices" } } + ] + }, + { + "send": { "action": "submit_interaction", "requestId": "r-msg", "sessionId": "{{sessionId}}", "interactionId": "{{interactionId}}", "kind": "choices", "values": { "answers": [{ "header": "Plan", "options": ["Pro"] }] } }, + "expect": [ + { "type": "immediate_response", "status": 200 }, + { "type": "stream_chunk", "assert": { "data.state.rawResponse.toolResult.name": "request_choices", "data.state.rawResponse.toolResult.isError": false } }, + { "type": "stream_token", "repeat": true, "accumulate": "token", "assertAccumulated": "Pro it is — pulling that quote up." }, + { "type": "eventual_response", "status": 200, "assert": { "data.data.response.responseParts": ["Pro it is — pulling that quote up."] } } + ] + } + ] +} diff --git a/spec/conformance/scenarios/interaction-conversational-fallback.json b/spec/conformance/scenarios/interaction-conversational-fallback.json new file mode 100644 index 00000000..b76e4e6b --- /dev/null +++ b/spec/conformance/scenarios/interaction-conversational-fallback.json @@ -0,0 +1,24 @@ +{ + "name": "interaction-conversational-fallback", + "description": "The capability gate. A session that did NOT declare `identity_form` in `supports` (a text-only channel — SMS, voice) must NEVER receive `interaction_required`: the same `request_identity_intake` raise degrades to its conversational fallback, returning immediately to the model so the turn runs straight through to its reply with no park. This is the negative half of interaction-park-resume — same mock script, capability removed — and it pins that the gate is read per turn from the session's declared capabilities rather than the kind being unconditionally rich.", + "mockLlmScript": [ + { "kind": "toolCall", "name": "request_identity_intake", "arguments": "{\"fields\": [{\"key\": \"email\", \"required\": true}], \"reason\": \"to send you the quote\"}" }, + { "kind": "text", "text": "Sure — what's the best email for you?" } + ], + "steps": [ + { + "send": { "action": "create_conversation_session", "requestId": "r-create", "agentId": "11111111-1111-1111-1111-111111111111", "userName": "Alice", "userEmail": "alice@example.com" }, + "expect": [ { "type": "immediate_response", "status": 200, "capture": { "sessionId": "data.sessionId" } } ] + }, + { + "send": { "action": "send_message", "requestId": "r-msg", "sessionId": "{{sessionId}}", "message": "send me a quote", "stream": true }, + "expect": [ + { "type": "immediate_response", "status": 202 }, + { "type": "stream_chunk", "assert": { "data.state.rawResponse.toolCall.name": "request_identity_intake" } }, + { "type": "stream_chunk", "assert": { "data.state.rawResponse.toolResult.name": "request_identity_intake", "data.state.rawResponse.toolResult.isError": false } }, + { "type": "stream_token", "repeat": true, "accumulate": "token", "assertAccumulated": "Sure — what's the best email for you?" }, + { "type": "eventual_response", "status": 200, "assert": { "data.data.response.responseParts": ["Sure — what's the best email for you?"] } } + ] + } + ] +} diff --git a/spec/conformance/scenarios/interaction-declined.json b/spec/conformance/scenarios/interaction-declined.json new file mode 100644 index 00000000..f0a3e5f7 --- /dev/null +++ b/spec/conformance/scenarios/interaction-declined.json @@ -0,0 +1,35 @@ +{ + "name": "interaction-declined", + "description": "`declined: true` resolves the park WITHOUT values. The visitor refusing a Rich Interaction is a first-class outcome, not an error and not a dead turn: the server acks 200, the raise tool returns a declined payload to the model, and the turn resumes and finishes normally. A server that required `values`, or that left the turn parked on a decline, fails here. Only `status` is asserted on the ack — the five servers put different fields in the decline ack's `data` (see the README's divergence note), so asserting more would pin one language's shape rather than the protocol's.", + "mockLlmScript": [ + { "kind": "toolCall", "name": "request_identity_intake", "arguments": "{\"fields\": [{\"key\": \"email\", \"required\": true}], \"reason\": \"to send you the quote\"}" }, + { "kind": "text", "text": "No problem — I'll keep helping without that." } + ], + "steps": [ + { + "send": { "action": "create_conversation_session", "requestId": "r-create", "agentId": "11111111-1111-1111-1111-111111111111", "userName": "Alice", "userEmail": "alice@example.com", "supports": ["identity_form"] }, + "expect": [ { "type": "immediate_response", "status": 200, "capture": { "sessionId": "data.sessionId" } } ] + }, + { + "send": { "action": "send_message", "requestId": "r-msg", "sessionId": "{{sessionId}}", "message": "send me a quote", "stream": true }, + "expect": [ + { "type": "immediate_response", "status": 202 }, + { + "type": "interaction_required", + "assert": { "requestId": "r-msg", "data.data.kind": "identity_intake" }, + "capture": { "interactionId": "data.data.interactionId" } + }, + { "type": "stream_chunk", "assert": { "data.state.rawResponse.toolCall.name": "request_identity_intake" } } + ] + }, + { + "send": { "action": "submit_interaction", "requestId": "r-msg", "sessionId": "{{sessionId}}", "interactionId": "{{interactionId}}", "declined": true }, + "expect": [ + { "type": "immediate_response", "status": 200 }, + { "type": "stream_chunk", "assert": { "data.state.rawResponse.toolResult.name": "request_identity_intake", "data.state.rawResponse.toolResult.isError": false } }, + { "type": "stream_token", "repeat": true, "accumulate": "token", "assertAccumulated": "No problem — I'll keep helping without that." }, + { "type": "eventual_response", "status": 200, "assert": { "data.data.response.responseParts": ["No problem — I'll keep helping without that."] } } + ] + } + ] +} diff --git a/spec/conformance/scenarios/interaction-invalid-retryable.json b/spec/conformance/scenarios/interaction-invalid-retryable.json new file mode 100644 index 00000000..ab0ecf43 --- /dev/null +++ b/spec/conformance/scenarios/interaction-invalid-retryable.json @@ -0,0 +1,57 @@ +{ + "name": "interaction-invalid-retryable", + "description": "Invalid interaction values are RETRYABLE, never terminal. A `submit_interaction` whose values fail the kind's server-side validator must emit `interaction_invalid` (per-field errors + a card-level summary) and leave the turn PARKED — mirroring `otp_invalid`. This scenario proves the park survives the rejection twice over: a second bad submit is rejected the same way, and only then does a corrected submit ack 200 and resume the turn. A server that terminated the turn with an `error` event, or that dropped the park after the first rejection, fails here.", + "mockLlmScript": [ + { "kind": "toolCall", "name": "request_identity_intake", "arguments": "{\"fields\": [{\"key\": \"email\", \"required\": true}], \"reason\": \"to send you the quote\"}" }, + { "kind": "text", "text": "Got it — sending the quote now." } + ], + "steps": [ + { + "send": { "action": "create_conversation_session", "requestId": "r-create", "agentId": "11111111-1111-1111-1111-111111111111", "userName": "Alice", "userEmail": "alice@example.com", "supports": ["identity_form"] }, + "expect": [ { "type": "immediate_response", "status": 200, "capture": { "sessionId": "data.sessionId" } } ] + }, + { + "send": { "action": "send_message", "requestId": "r-msg", "sessionId": "{{sessionId}}", "message": "send me a quote", "stream": true }, + "expect": [ + { "type": "immediate_response", "status": 202 }, + { + "type": "interaction_required", + "assert": { "requestId": "r-msg", "data.data.kind": "identity_intake" }, + "capture": { "interactionId": "data.data.interactionId" } + }, + { "type": "stream_chunk", "assert": { "data.state.rawResponse.toolCall.name": "request_identity_intake" } } + ] + }, + { + "send": { "action": "submit_interaction", "requestId": "r-msg", "sessionId": "{{sessionId}}", "interactionId": "{{interactionId}}", "values": { "email": "not-an-email" } }, + "expect": [ + { + "type": "interaction_invalid", + "assert": { + "requestId": "r-msg", + "data.requestId": "r-msg", + "data.data.kind": "identity_intake", + "data.data.errors.0.field": "email", + "data.data.errors.0.message": "must be a valid email address", + "data.data.message": "Some fields need attention." + } + } + ] + }, + { + "send": { "action": "submit_interaction", "requestId": "r-msg", "sessionId": "{{sessionId}}", "interactionId": "{{interactionId}}", "values": { "email": "still bad@" } }, + "expect": [ + { "type": "interaction_invalid", "assert": { "requestId": "r-msg", "data.data.kind": "identity_intake", "data.data.errors.0.field": "email" } } + ] + }, + { + "send": { "action": "submit_interaction", "requestId": "r-msg", "sessionId": "{{sessionId}}", "interactionId": "{{interactionId}}", "values": { "email": "alice@example.com" } }, + "expect": [ + { "type": "immediate_response", "status": 200 }, + { "type": "stream_chunk", "assert": { "data.state.rawResponse.toolResult.name": "request_identity_intake", "data.state.rawResponse.toolResult.isError": false } }, + { "type": "stream_token", "repeat": true, "accumulate": "token", "assertAccumulated": "Got it — sending the quote now." }, + { "type": "eventual_response", "status": 200, "assert": { "data.data.response.responseParts": ["Got it — sending the quote now."] } } + ] + } + ] +} diff --git a/spec/conformance/scenarios/interaction-park-resume.json b/spec/conformance/scenarios/interaction-park-resume.json new file mode 100644 index 00000000..50f24c84 --- /dev/null +++ b/spec/conformance/scenarios/interaction-park-resume.json @@ -0,0 +1,44 @@ +{ + "name": "interaction-park-resume", + "description": "Rich Interactions, the happy path. A session that DECLARED the `identity_form` render capability in `supports` gets the real card: the agent's server-registered `request_identity_intake` raise tool PARKS the turn and the server emits `interaction_required` (kind `identity_intake`, a server-minted `interactionId`, the canonical `spec.fields`, and the agent's `reason`). A matching `submit_interaction` — same requestId, same interactionId — acks 200 and RESUMES the parked turn, which streams the wrap-up reply. The raise tools are server-registered in every server, so this needs no `server.*` directive; only the mock names the tool. ⚠️ The park event is asserted BEFORE the raise tool's `stream_chunk`, matching the Rust reference and the same ordering the corpus already pins for write-confirmation HITL (`hitl-write-confirmation`). Go / TypeScript / Python / .NET emit the chunk first; see the README's divergence note.", + "mockLlmScript": [ + { "kind": "toolCall", "name": "request_identity_intake", "arguments": "{\"fields\": [{\"key\": \"name\", \"required\": false}, {\"key\": \"email\", \"required\": true}], \"reason\": \"to send you the quote\"}" }, + { "kind": "text", "text": "Thanks — I've got your details." } + ], + "steps": [ + { + "send": { "action": "create_conversation_session", "requestId": "r-create", "agentId": "11111111-1111-1111-1111-111111111111", "userName": "Alice", "userEmail": "alice@example.com", "supports": ["identity_form"] }, + "expect": [ { "type": "immediate_response", "status": 200, "capture": { "sessionId": "data.sessionId" } } ] + }, + { + "send": { "action": "send_message", "requestId": "r-msg", "sessionId": "{{sessionId}}", "message": "send me a quote", "stream": true }, + "expect": [ + { "type": "immediate_response", "status": 202 }, + { + "type": "interaction_required", + "assert": { + "requestId": "r-msg", + "data.requestId": "r-msg", + "data.data.kind": "identity_intake", + "data.data.reason": "to send you the quote", + "data.data.spec.fields.0.key": "name", + "data.data.spec.fields.0.required": false, + "data.data.spec.fields.1.key": "email", + "data.data.spec.fields.1.required": true + }, + "capture": { "interactionId": "data.data.interactionId" } + }, + { "type": "stream_chunk", "assert": { "data.state.rawResponse.toolCall.name": "request_identity_intake" } } + ] + }, + { + "send": { "action": "submit_interaction", "requestId": "r-msg", "sessionId": "{{sessionId}}", "interactionId": "{{interactionId}}", "kind": "identity_intake", "values": { "name": "Alice Example", "email": "alice@example.com" } }, + "expect": [ + { "type": "immediate_response", "status": 200 }, + { "type": "stream_chunk", "assert": { "data.state.rawResponse.toolResult.name": "request_identity_intake", "data.state.rawResponse.toolResult.isError": false } }, + { "type": "stream_token", "repeat": true, "accumulate": "token", "assertAccumulated": "Thanks — I've got your details." }, + { "type": "eventual_response", "status": 200, "assert": { "data.data.response.responseParts": ["Thanks — I've got your details."] } } + ] + } + ] +} diff --git a/spec/conformance/scenarios/interaction-stale-id-rejected.json b/spec/conformance/scenarios/interaction-stale-id-rejected.json new file mode 100644 index 00000000..a9a911ae --- /dev/null +++ b/spec/conformance/scenarios/interaction-stale-id-rejected.json @@ -0,0 +1,38 @@ +{ + "name": "interaction-stale-id-rejected", + "description": "A `submit_interaction` carrying a stale or mismatched `interactionId` must be REJECTED with error/INTERACTION_MISMATCH and must leave the turn parked — the interactionId exists precisely so a stale submit can never resolve a newer park (spec/actions/submit-interaction.schema.json). The park's survival is proved without depending on resume ordering: a follow-up submit with the correct id gets its 200 ack, which a dropped park could not produce.", + "mockLlmScript": [ + { "kind": "toolCall", "name": "request_identity_intake", "arguments": "{\"fields\": [{\"key\": \"email\", \"required\": true}], \"reason\": \"to send you the quote\"}" }, + { "kind": "text", "text": "Got it — sending the quote now." } + ], + "steps": [ + { + "send": { "action": "create_conversation_session", "requestId": "r-create", "agentId": "11111111-1111-1111-1111-111111111111", "userName": "Alice", "userEmail": "alice@example.com", "supports": ["identity_form"] }, + "expect": [ { "type": "immediate_response", "status": 200, "capture": { "sessionId": "data.sessionId" } } ] + }, + { + "send": { "action": "send_message", "requestId": "r-msg", "sessionId": "{{sessionId}}", "message": "send me a quote", "stream": true }, + "expect": [ + { "type": "immediate_response", "status": 202 }, + { + "type": "interaction_required", + "assert": { "requestId": "r-msg", "data.data.kind": "identity_intake" }, + "capture": { "interactionId": "data.data.interactionId" } + }, + { "type": "stream_chunk", "assert": { "data.state.rawResponse.toolCall.name": "request_identity_intake" } } + ] + }, + { + "send": { "action": "submit_interaction", "requestId": "r-msg", "sessionId": "{{sessionId}}", "interactionId": "stale-interaction-id", "values": { "email": "alice@example.com" } }, + "expect": [ + { "type": "error", "assert": { "requestId": "r-msg", "error.code": "INTERACTION_MISMATCH" } } + ] + }, + { + "send": { "action": "submit_interaction", "requestId": "r-msg", "sessionId": "{{sessionId}}", "interactionId": "{{interactionId}}", "values": { "email": "alice@example.com" } }, + "expect": [ + { "type": "immediate_response", "status": 200 } + ] + } + ] +} From 3034c71f4ed432f3dbc01f0967a9af0fd4a6cb33 Mon Sep 17 00:00:00 2001 From: Brent Rager Date: Wed, 19 Aug 2026 18:56:48 -0400 Subject: [PATCH 2/5] th-eae69d: honour knownDivergences in all five parity runners MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Per the ruling on #513: Rust's park ordering is correct (the four ports change, not the spec), and this must not merge red. So the new scenarios now carry an EXPIRING xfail marker that all five runners honour. A scenario names the languages it fails on today, with the reason and the pearl id right beside it: "knownDivergences": ["go", "typescript", "python", "dotnet"], "knownDivergencesReason": "th-eae69d — these four emit the raise tool's toolCall chunk BEFORE interaction_required ..." Both halves of the contract are implemented, and the second is the point: * a listed language that FAILS is reported (reason + the real assertion) and does not fail the build; * a listed language that PASSES FAILS the build — "remove from knownDivergences in — it now passes". Without the xpass half the markers rot silently and we recreate the exact "green tests that prove nothing" problem this corpus exists to catch. A marker is a tracked bug with an expiry, never an accepted difference. Marked strictly to the measured matrix: the five interaction park scenarios for go/typescript/python/dotnet, and cancel-mid-turn for go/dotnet. The sixth interaction scenario (interaction-conversational-fallback) already passes everywhere and is deliberately unmarked. *testing.T and panics do not catch alike, so the five runners differ: rust tokio::spawn, so the scenario's panic surfaces as a JoinError go narrow *testing.T to a small parityT interface, then run a marked scenario against a recorder whose Fatalf panics with the message (testing.T's own Fatalf is terminal and cannot be un-failed) ts/py catch the assertion (vitest / pytest.xfail) dotnet catch XunitException Verified in both directions, per language: with the real corpus all five are green (Rust 18/18, Go/TS/.NET green with divergences logged, Python 13 passed + 5 xfailed), and with a passing scenario temporarily marked for all five, every runner fails with its "it now passes" message. gofmt, cargo fmt, clippy, tsc --noEmit and ruff are clean. Note for anyone iterating locally: `go test` caches results and does NOT invalidate on a scenario-JSON edit — use -count=1. Co-Authored-By: Claude Fable 5 --- .../integration-tests/ScenarioParityTests.cs | 34 ++++++- go/server/scenario_parity_test.go | 95 ++++++++++++++++--- python/server/tests/test_scenario_parity.py | 18 ++++ .../tests/scenario_parity.rs | 48 ++++++++-- spec/conformance/scenarios/README.md | 20 +++- .../scenarios/cancel-mid-turn.json | 2 + .../interaction-choices-park-resume.json | 2 + .../scenarios/interaction-declined.json | 2 + .../interaction-invalid-retryable.json | 2 + .../scenarios/interaction-park-resume.json | 2 + .../interaction-stale-id-rejected.json | 2 + .../server/test/scenario-parity.test.ts | 18 ++++ 12 files changed, 226 insertions(+), 19 deletions(-) diff --git a/dotnet/server/integration-tests/ScenarioParityTests.cs b/dotnet/server/integration-tests/ScenarioParityTests.cs index 6efd4af7..f662a301 100644 --- a/dotnet/server/integration-tests/ScenarioParityTests.cs +++ b/dotnet/server/integration-tests/ScenarioParityTests.cs @@ -10,6 +10,7 @@ using Microsoft.Extensions.DependencyInjection; using SmooAI.SmoothOperator.Core; using SmooAI.SmoothOperator.Server.AspNetCore; +using Xunit.Sdk; namespace SmooAI.SmoothOperator.Server.IntegrationTests; @@ -57,12 +58,43 @@ public void JsonEquals_ComparesNumbersByValue_NotRepresentation() Assert.False(JsonEquals(JsonNode.Parse("""{"a": 1}"""), JsonNode.Parse("""{"a": 1, "b": 2}"""))); } + /// This runner's id in a scenario's knownDivergences list. + private const string Lang = "dotnet"; + [Theory] [MemberData(nameof(Scenarios))] public async Task ScenarioParity(string name, string path) { - _ = name; // surfaced as the test id via MemberData var scenario = JsonNode.Parse(await File.ReadAllTextAsync(path))!.AsObject(); + + // `knownDivergences` lists the languages a scenario is known to fail on today, + // with `knownDivergencesReason` next to it. An EXPIRING marker, not a skip: a + // listed language that fails is reported and tolerated, but one that PASSES fails + // the build, so a marker cannot rot silently into a green test that proves nothing + // (the failure mode this corpus exists to catch). + var divergent = scenario["knownDivergences"]?.AsArray() + .Any(l => l?.GetValue() == Lang) ?? false; + if (divergent) + { + try + { + await RunScenarioAsync(scenario); + } + catch (Exception ex) when (ex is XunitException or IOException or WebSocketException) + { + Console.WriteLine( + $"[known divergence] {name}: {scenario["knownDivergencesReason"]?.GetValue()}\n {ex.Message}"); + return; + } + + Assert.Fail($"remove {Lang} from knownDivergences in {name} — it now passes"); + } + + await RunScenarioAsync(scenario); + } + + private static async Task RunScenarioAsync(JsonObject scenario) + { var chat = BuildMock(scenario["mockLlmScript"]?.AsArray()); var serverDirective = scenario["server"]?.AsObject(); var tools = BuildTools(serverDirective?["tools"]?.AsArray()); diff --git a/go/server/scenario_parity_test.go b/go/server/scenario_parity_test.go index 6c7d5046..8f5f3d43 100644 --- a/go/server/scenario_parity_test.go +++ b/go/server/scenario_parity_test.go @@ -7,6 +7,7 @@ import ( "os" "path/filepath" "reflect" + "slices" "strconv" "strings" "testing" @@ -36,8 +37,19 @@ type scenario struct { Server scenarioServer `json:"server"` MockLlmScript []mockScriptStep `json:"mockLlmScript"` Steps []scenarioStep `json:"steps"` + + // KnownDivergences lists the languages this scenario is known to fail on today, + // with KnownDivergencesReason next to it. An EXPIRING marker, not a skip: a + // listed language that fails is reported and tolerated, but one that PASSES + // fails the build, so a marker cannot rot silently into a green test that proves + // nothing (the failure mode this corpus exists to catch). + KnownDivergences []string `json:"knownDivergences"` + KnownDivergencesReason string `json:"knownDivergencesReason"` } +// lang is this runner's id in a scenario's knownDivergences list. +const lang = "go" + // scenarioServer is the optional `server` directive: deployment-time config the runner // applies when starting the server — the tools the agent may call, and the subset gated // behind write-confirmation HITL. @@ -97,7 +109,48 @@ type matcher struct { // scenariosDir resolves spec/conformance/scenarios relative to the repo root (this // file lives at go/server/, so the root is three parents up). -func scenariosDir(t *testing.T) string { +// parityT is the slice of *testing.T the scenario runner actually uses. Narrowing +// it to an interface lets a known-divergence scenario run against a recorder that +// CAPTURES the first failure instead of failing the build — *testing.T's own +// Fatalf is terminal and cannot be un-failed. +type parityT interface { + Helper() + Fatalf(format string, args ...any) +} + +// divergenceRecorder is a parityT that turns the first Fatalf into a panic +// carrying the message, so the caller can recover it. Fatalf must not return +// (callers rely on it aborting), and panic is the only way to do that while +// leaving the real *testing.T untouched. +type divergenceRecorder struct{ msg string } + +type divergenceFailure struct{ msg string } + +func (r *divergenceRecorder) Helper() {} + +func (r *divergenceRecorder) Fatalf(format string, args ...any) { + r.msg = fmt.Sprintf(format, args...) + panic(divergenceFailure{r.msg}) +} + +// runDivergent runs a scenario against a recorder, reporting whether it passed +// and the failure message if it did not. +func runDivergent(path string) (passed bool, msg string) { + rec := &divergenceRecorder{} + defer func() { + if r := recover(); r != nil { + if f, ok := r.(divergenceFailure); ok { + passed, msg = false, f.msg + return + } + panic(r) + } + }() + runScenario(rec, path) + return true, "" +} + +func scenariosDir(t parityT) string { t.Helper() wd, err := os.Getwd() if err != nil { @@ -110,7 +163,7 @@ func scenariosDir(t *testing.T) string { // ("data.data.response") or, when it parses as a non-negative integer, an array by // position ("citations.0.id") — so a citation field can be asserted by index. Mirrors // the Python reference runner's array-aware dot helper. -func dot(t *testing.T, obj map[string]any, path string) (any, bool) { +func dot(t parityT, obj map[string]any, path string) (any, bool) { t.Helper() var cur any = obj for _, part := range strings.Split(path, ".") { @@ -136,7 +189,7 @@ func dot(t *testing.T, obj map[string]any, path string) (any, bool) { // buildMock loads a scenario's mockLlmScript into the engine's MockLlmProvider — the // deterministic record/replay source that makes the turn identical across languages. -func buildMock(t *testing.T, script []mockScriptStep) *core.MockLlmProvider { +func buildMock(t parityT, script []mockScriptStep) *core.MockLlmProvider { t.Helper() mock := core.NewMockLlmProvider() for _, entry := range script { @@ -275,20 +328,40 @@ func TestScenarioParity(t *testing.T) { path := path name := strings.TrimSuffix(filepath.Base(path), ".json") t.Run(name, func(t *testing.T) { - runScenario(t, path) + sc, err := loadScenario(path) + if err != nil { + t.Fatalf("load scenario: %v", err) + } + if !slices.Contains(sc.KnownDivergences, lang) { + runScenario(t, path) + return + } + passed, msg := runDivergent(path) + if passed { + t.Fatalf("remove %s from knownDivergences in %s.json — it now passes", lang, name) + } + t.Logf("known divergence (%s): %s", sc.KnownDivergencesReason, msg) }) } } -func runScenario(t *testing.T, path string) { - t.Helper() +func loadScenario(path string) (scenario, error) { + var sc scenario raw, err := os.ReadFile(path) if err != nil { - t.Fatalf("read scenario: %v", err) + return sc, fmt.Errorf("read scenario: %w", err) } - var sc scenario if err := json.Unmarshal(raw, &sc); err != nil { - t.Fatalf("parse scenario: %v", err) + return sc, fmt.Errorf("parse scenario: %w", err) + } + return sc, nil +} + +func runScenario(t parityT, path string) { + t.Helper() + sc, err := loadScenario(path) + if err != nil { + t.Fatalf("%v", err) } mock := buildMock(t, sc.MockLlmScript) @@ -334,7 +407,7 @@ func runScenario(t *testing.T, path string) { // nextEvent returns the next protocol event, skipping non-semantic keepalive/pong // frames (as the Python reference does). -func nextEvent(t *testing.T, transport protocol.Transport) map[string]any { +func nextEvent(t parityT, transport protocol.Transport) map[string]any { t.Helper() for { select { @@ -364,7 +437,7 @@ func nextEvent(t *testing.T, transport protocol.Transport) map[string]any { // a faithful port of the Python reference's _match_expected state machine: one-event // lookahead for `repeat` overrun, status / statusGte / assert checks, var capture, and // accumulate + assertAccumulated. -func matchExpected(t *testing.T, transport protocol.Transport, matchers []matcher, vars map[string]any) { +func matchExpected(t parityT, transport protocol.Transport, matchers []matcher, vars map[string]any) { t.Helper() var pending map[string]any // one-event lookahead when a `repeat` matcher overruns for _, m := range matchers { diff --git a/python/server/tests/test_scenario_parity.py b/python/server/tests/test_scenario_parity.py index dd2c656c..8fb99a25 100644 --- a/python/server/tests/test_scenario_parity.py +++ b/python/server/tests/test_scenario_parity.py @@ -25,6 +25,9 @@ SCENARIOS_DIR = Path(__file__).resolve().parents[3] / "spec" / "conformance" / "scenarios" SCENARIOS = sorted(SCENARIOS_DIR.glob("*.json")) +#: This runner's id in a scenario's ``knownDivergences`` list. +LANG = "python" + def _dot(obj, path: str): """Resolve a dotted path (``data.data.response.responseParts``) into a nested @@ -94,6 +97,21 @@ def _subst(value, vars_: dict): @pytest.mark.asyncio async def test_scenario_parity(path: Path) -> None: scenario = json.loads(path.read_text()) + # `knownDivergences` lists the languages a scenario is known to fail on today, + # with `knownDivergencesReason` next to it. An EXPIRING marker, not a skip: a + # listed language that fails is reported and tolerated, but one that PASSES + # fails the build, so a marker cannot rot silently into a green test that + # proves nothing (which is the failure mode this whole corpus exists to catch). + if LANG in scenario.get("knownDivergences", []): + try: + await _run_scenario(scenario) + except AssertionError as exc: + pytest.xfail(f"known divergence ({scenario.get('knownDivergencesReason', '')}): {exc}") + pytest.fail(f"remove {LANG} from knownDivergences in {path.name} — it now passes") + await _run_scenario(scenario) + + +async def _run_scenario(scenario: dict) -> None: mock = _build_mock(scenario.get("mockLlmScript", [])) server_spec = scenario.get("server", {}) tools = _build_tools(server_spec.get("tools", [])) diff --git a/rust/smooth-operator-server/tests/scenario_parity.rs b/rust/smooth-operator-server/tests/scenario_parity.rs index dfca3b8c..428586ff 100644 --- a/rust/smooth-operator-server/tests/scenario_parity.rs +++ b/rust/smooth-operator-server/tests/scenario_parity.rs @@ -496,6 +496,23 @@ async fn match_expected( } } +/// This runner's id in a scenario's `knownDivergences` list. +const LANG: &str = "rust"; + +/// Whether `path` marks this language as a known divergence. +fn is_divergent(path: &Path) -> bool { + let Ok(raw) = std::fs::read_to_string(path) else { + return false; + }; + let Ok(scenario) = serde_json::from_str::(&raw) else { + return false; + }; + scenario + .get("knownDivergences") + .and_then(Value::as_array) + .is_some_and(|langs| langs.iter().any(|l| l.as_str() == Some(LANG))) +} + /// Drive one scenario file end-to-end through the reference server. async fn run_scenario(path: &Path) { let scenario: Value = @@ -554,13 +571,32 @@ async fn scenario_parity_corpus() { scenarios_dir().display() ); for path in &paths { - eprintln!( - "[scenario-parity] {}", - path.file_name().unwrap().to_string_lossy() - ); - run_scenario(path).await; + let name = path.file_name().unwrap().to_string_lossy().to_string(); + eprintln!("[scenario-parity] {name}"); + if !is_divergent(path) { + run_scenario(path).await; + continue; + } + // `knownDivergences` lists the languages a scenario is known to fail on + // today, with `knownDivergencesReason` next to it. An EXPIRING marker, not + // a skip: a listed language that fails is reported and tolerated, but one + // that PASSES fails the build, so a marker cannot rot silently into a green + // test that proves nothing (the failure mode this corpus exists to catch). + // Spawned so the scenario's panic is caught by the JoinHandle rather than + // unwinding the whole corpus. + let owned = path.clone(); + if tokio::spawn(async move { run_scenario(&owned).await }) + .await + .is_ok() + { + panic!("remove {LANG} from knownDivergences in {name} — it now passes"); + } + eprintln!("[scenario-parity] {name}: known divergence for {LANG}, tolerated"); } - eprintln!("[scenario-parity] {} scenario(s) passed", paths.len()); + eprintln!( + "[scenario-parity] {} scenario(s) accounted for", + paths.len() + ); } /// Offline guard for the by-value JSON comparison (pearl th-4f1263). A corpus diff --git a/spec/conformance/scenarios/README.md b/spec/conformance/scenarios/README.md index 30ce371a..0994a86f 100644 --- a/spec/conformance/scenarios/README.md +++ b/spec/conformance/scenarios/README.md @@ -97,11 +97,29 @@ Eight scenarios now cover them: A mock turn finishes faster than a `cancel` frame can race it, and the format has no "slow tool" directive (`server.tools` entries return a fixed string immediately). `cancel-mid-turn` therefore opens its in-flight window with a **write-confirmation park** (`server.confirmTools`) — the one pause this corpus can express. `cancel-no-active-turn-noop` asserts "nothing arrives" structurally, since no runner has a drain check: the cancel step expects zero events, so any stray event is consumed by the *next* step's first matcher and fails it. +### `knownDivergences` — an expiring marker, not a skip + +A scenario may name the languages it is known to fail on today, with the reason and pearl id right next to it: + +```jsonc +"knownDivergences": ["go", "typescript", "python", "dotnet"], +"knownDivergencesReason": "th-eae69d — these four emit the raise tool's toolCall chunk BEFORE interaction_required …", +``` + +All five runners honour it, and the contract has two halves — the second is the one that matters: + +- a **listed** language that FAILS is reported (with the reason and the actual assertion) and does not fail the build; +- a **listed** language that PASSES **fails the build**, with `remove from knownDivergences in — it now passes`. + +Without that second half the markers rot silently and we recreate the exact "green tests that prove nothing" problem this corpus exists to catch. A marker is a tracked bug with an expiry, never an accepted difference — the entry comes out the moment the port is fixed, and the build tells you when that is. + +Implementation note per language, since `*testing.T` and panics do not catch alike: Rust runs a marked scenario on a `tokio::spawn` handle so its panic surfaces as a `JoinError`; Go narrows the runner's `*testing.T` to a small `parityT` interface so a marked scenario can run against a recorder whose `Fatalf` panics with the message instead of failing the build; TypeScript, Python and .NET just catch the assertion. ⚠️ `go test` caches results and does not invalidate on a scenario-JSON edit — use `-count=1` when iterating locally. + ### Known divergences these scenarios expose Recorded here as facts, not as license to weaken the scenarios. **Do not "fix" a scenario to make a port pass.** -- **Park event vs. the raise tool's `stream_chunk` — Rust is 1 of 5.** For a Rich Interaction, Rust emits `interaction_required` *before* the raise tool's `toolCall` chunk; Go, TypeScript, Python and .NET all emit the chunk first. The five scenarios that park assert **Rust's** order, because Rust is the reference and because it is the order this corpus already pins for the other park type (`hitl-write-confirmation` defers the gated tool's chunk until after the prompt) — the other four are internally inconsistent between their two park paths. Whether the spec should adopt chunk-first instead is a **protocol decision, not a test fix**; until it is made, four ports are red on these five scenarios. +- **Park event vs. the raise tool's `stream_chunk` — Rust is 1 of 5, and Rust is right.** For a Rich Interaction, Rust emits `interaction_required` *before* the raise tool's `toolCall` chunk; Go, TypeScript, Python and .NET all emit the chunk first. **Ruled a port bug, not a protocol variant**, on three grounds: all five already defer the gated tool's chunk until after the prompt for the *other* park type (`hitl-write-confirmation`), so the four are internally inconsistent between their two park paths while Rust is consistent; Rust is the designated reference and the ports mirror it; and semantically a client that renders tool calls would otherwise show "calling `request_identity_intake`…" before the card appears, leaking framework internals ahead of the semantic event. The four ports change, not these scenarios. - **A cancelled turn keeps running in Go and .NET.** In both, the turn after a `cancel` produces no reply because the *cancelled* turn consumed an extra LLM response: the write-confirmation gate returns a deny instead of unwinding, the agent loop makes one more model call, and the output is merely gagged (Go: `if turnCtx.Err() != nil { return }`). Cancellation is a mute button there, not a stop button — a real cost and a real side-effect risk after a visitor hits Stop. Rust, TypeScript and Python abort the turn properly. Verified by re-running with one extra `mockLlmScript` entry: both pass, proving the entry is eaten by the cancelled turn. - **Ack payloads differ, so only `status` is asserted on a `submit_interaction` ack.** The five servers put different fields in `data` (Go omits `kind`/`values`; Python omits `kind`, and its decline ack omits `interactionId`/`declined`; .NET's decline ack omits `declined`). Asserting more would pin one language's shape rather than the protocol's. diff --git a/spec/conformance/scenarios/cancel-mid-turn.json b/spec/conformance/scenarios/cancel-mid-turn.json index e5f81c7e..46e99d91 100644 --- a/spec/conformance/scenarios/cancel-mid-turn.json +++ b/spec/conformance/scenarios/cancel-mid-turn.json @@ -1,6 +1,8 @@ { "name": "cancel-mid-turn", "description": "A turn cancelled while in flight must emit the terminal `cancelled` event (status 499, echoing the CANCELLED TURN's requestId) IN PLACE OF `eventual_response`, and must free the connection's turn slot so the next `send_message` is accepted. The in-flight window is made deterministic by parking the turn on write-confirmation HITL (`server.confirmTools`) — a mock turn otherwise completes faster than a `cancel` frame can race it, and the scenario format has no way to express a slow tool. Per spec/actions/cancel.schema.json the cancel frame carries the requestId of the `send_message` it aborts; per spec/events/cancelled.schema.json the cancelled turn produces NO answer payload.", + "knownDivergences": ["go", "dotnet"], + "knownDivergencesReason": "th-eae69d — Go and .NET do not actually abort a cancelled turn. The cancelled turn consumes one MORE LLM response after the cancel (Go's write-confirmation gate returns a deny instead of unwinding, the agent loop makes another model call) and its output is merely gagged, so the NEXT turn finds a drained mock script and streams nothing. Cancellation is a mute button there, not a stop button — after a visitor hits Stop the turn keeps burning model calls and keeps executing whatever comes next, invisibly. Rust, TypeScript and Python abort properly. Tracked as a port bug; do NOT weaken this scenario.", "server": { "tools": [ { diff --git a/spec/conformance/scenarios/interaction-choices-park-resume.json b/spec/conformance/scenarios/interaction-choices-park-resume.json index 97d85614..4751a107 100644 --- a/spec/conformance/scenarios/interaction-choices-park-resume.json +++ b/spec/conformance/scenarios/interaction-choices-park-resume.json @@ -1,6 +1,8 @@ { "name": "interaction-choices-park-resume", "description": "The second Rich Interaction kind, `choices` (the AskUserQuestion-shaped multiple-choice ask), through the SAME generic envelope: `request_choices` parks behind the `choice_chips` capability, the server emits `interaction_required` with the normalized questions/options spec, and one `submit_interaction` resumes it. Adding a kind must require no new protocol action and no client release — this scenario is what proves that claim holds identically in every server, and that the kind catalog is not identity_intake-shaped by accident.", + "knownDivergences": ["go", "typescript", "python", "dotnet"], + "knownDivergencesReason": "th-eae69d — the park-ordering divergence, not the choices kind itself: these four emit the raise tool's `toolCall` stream_chunk BEFORE `interaction_required`, so they fail in step 2 and never reach the choices spec/resume assertions. See interaction-park-resume for the full ruling.", "mockLlmScript": [ { "kind": "toolCall", "name": "request_choices", "arguments": "{\"questions\": [{\"question\": \"Which plan fits best?\", \"header\": \"Plan\", \"options\": [{\"label\": \"Starter\"}, {\"label\": \"Pro\"}]}], \"reason\": \"to route you to the right quote\"}" }, { "kind": "text", "text": "Pro it is — pulling that quote up." } diff --git a/spec/conformance/scenarios/interaction-declined.json b/spec/conformance/scenarios/interaction-declined.json index f0a3e5f7..bcea4d3f 100644 --- a/spec/conformance/scenarios/interaction-declined.json +++ b/spec/conformance/scenarios/interaction-declined.json @@ -1,6 +1,8 @@ { "name": "interaction-declined", "description": "`declined: true` resolves the park WITHOUT values. The visitor refusing a Rich Interaction is a first-class outcome, not an error and not a dead turn: the server acks 200, the raise tool returns a declined payload to the model, and the turn resumes and finishes normally. A server that required `values`, or that left the turn parked on a decline, fails here. Only `status` is asserted on the ack — the five servers put different fields in the decline ack's `data` (see the README's divergence note), so asserting more would pin one language's shape rather than the protocol's.", + "knownDivergences": ["go", "typescript", "python", "dotnet"], + "knownDivergencesReason": "th-eae69d — the park-ordering divergence, not the decline behavior: these four emit the raise tool's `toolCall` stream_chunk BEFORE `interaction_required`, so they fail in step 2 and never reach the decline assertions. See interaction-park-resume for the full ruling.", "mockLlmScript": [ { "kind": "toolCall", "name": "request_identity_intake", "arguments": "{\"fields\": [{\"key\": \"email\", \"required\": true}], \"reason\": \"to send you the quote\"}" }, { "kind": "text", "text": "No problem — I'll keep helping without that." } diff --git a/spec/conformance/scenarios/interaction-invalid-retryable.json b/spec/conformance/scenarios/interaction-invalid-retryable.json index ab0ecf43..20a3c31e 100644 --- a/spec/conformance/scenarios/interaction-invalid-retryable.json +++ b/spec/conformance/scenarios/interaction-invalid-retryable.json @@ -1,6 +1,8 @@ { "name": "interaction-invalid-retryable", "description": "Invalid interaction values are RETRYABLE, never terminal. A `submit_interaction` whose values fail the kind's server-side validator must emit `interaction_invalid` (per-field errors + a card-level summary) and leave the turn PARKED — mirroring `otp_invalid`. This scenario proves the park survives the rejection twice over: a second bad submit is rejected the same way, and only then does a corrected submit ack 200 and resume the turn. A server that terminated the turn with an `error` event, or that dropped the park after the first rejection, fails here.", + "knownDivergences": ["go", "typescript", "python", "dotnet"], + "knownDivergencesReason": "th-eae69d — the park-ordering divergence, not the retry behavior: these four emit the raise tool's `toolCall` stream_chunk BEFORE `interaction_required`, so they fail in step 2 and never reach the retryable-invalid assertions. See interaction-park-resume for the full ruling.", "mockLlmScript": [ { "kind": "toolCall", "name": "request_identity_intake", "arguments": "{\"fields\": [{\"key\": \"email\", \"required\": true}], \"reason\": \"to send you the quote\"}" }, { "kind": "text", "text": "Got it — sending the quote now." } diff --git a/spec/conformance/scenarios/interaction-park-resume.json b/spec/conformance/scenarios/interaction-park-resume.json index 50f24c84..14fb4138 100644 --- a/spec/conformance/scenarios/interaction-park-resume.json +++ b/spec/conformance/scenarios/interaction-park-resume.json @@ -1,6 +1,8 @@ { "name": "interaction-park-resume", "description": "Rich Interactions, the happy path. A session that DECLARED the `identity_form` render capability in `supports` gets the real card: the agent's server-registered `request_identity_intake` raise tool PARKS the turn and the server emits `interaction_required` (kind `identity_intake`, a server-minted `interactionId`, the canonical `spec.fields`, and the agent's `reason`). A matching `submit_interaction` — same requestId, same interactionId — acks 200 and RESUMES the parked turn, which streams the wrap-up reply. The raise tools are server-registered in every server, so this needs no `server.*` directive; only the mock names the tool. ⚠️ The park event is asserted BEFORE the raise tool's `stream_chunk`, matching the Rust reference and the same ordering the corpus already pins for write-confirmation HITL (`hitl-write-confirmation`). Go / TypeScript / Python / .NET emit the chunk first; see the README's divergence note.", + "knownDivergences": ["go", "typescript", "python", "dotnet"], + "knownDivergencesReason": "th-eae69d — these four emit the raise tool's `toolCall` stream_chunk BEFORE `interaction_required`; Rust emits the park event first. Ruled a port bug, not a protocol variant: all five already defer the gated tool's chunk until after the prompt for the OTHER park type (`hitl-write-confirmation`), so the four are internally inconsistent between their two park paths while Rust is consistent — and a client that renders tool calls would show \"calling request_identity_intake…\" before the card appears, leaking framework internals ahead of the semantic event. The ports change, not this scenario.", "mockLlmScript": [ { "kind": "toolCall", "name": "request_identity_intake", "arguments": "{\"fields\": [{\"key\": \"name\", \"required\": false}, {\"key\": \"email\", \"required\": true}], \"reason\": \"to send you the quote\"}" }, { "kind": "text", "text": "Thanks — I've got your details." } diff --git a/spec/conformance/scenarios/interaction-stale-id-rejected.json b/spec/conformance/scenarios/interaction-stale-id-rejected.json index a9a911ae..ce858816 100644 --- a/spec/conformance/scenarios/interaction-stale-id-rejected.json +++ b/spec/conformance/scenarios/interaction-stale-id-rejected.json @@ -1,6 +1,8 @@ { "name": "interaction-stale-id-rejected", "description": "A `submit_interaction` carrying a stale or mismatched `interactionId` must be REJECTED with error/INTERACTION_MISMATCH and must leave the turn parked — the interactionId exists precisely so a stale submit can never resolve a newer park (spec/actions/submit-interaction.schema.json). The park's survival is proved without depending on resume ordering: a follow-up submit with the correct id gets its 200 ack, which a dropped park could not produce.", + "knownDivergences": ["go", "typescript", "python", "dotnet"], + "knownDivergencesReason": "th-eae69d — the park-ordering divergence, not the mismatch behavior: these four emit the raise tool's `toolCall` stream_chunk BEFORE `interaction_required`, so they fail in step 2 and never reach the INTERACTION_MISMATCH assertion. See interaction-park-resume for the full ruling.", "mockLlmScript": [ { "kind": "toolCall", "name": "request_identity_intake", "arguments": "{\"fields\": [{\"key\": \"email\", \"required\": true}], \"reason\": \"to send you the quote\"}" }, { "kind": "text", "text": "Got it — sending the quote now." } diff --git a/typescript/server/test/scenario-parity.test.ts b/typescript/server/test/scenario-parity.test.ts index a0dc6b15..e302eaa8 100644 --- a/typescript/server/test/scenario-parity.test.ts +++ b/typescript/server/test/scenario-parity.test.ts @@ -79,9 +79,14 @@ interface Scenario { description?: string; mockLlmScript?: MockScriptEntry[]; server?: { tools?: ToolSpec[]; confirmTools?: string[]; knowledge?: KnowledgeDoc[] }; + knownDivergences?: string[]; + knownDivergencesReason?: string; steps: Step[]; } +/** This runner's id in a scenario's `knownDivergences` list. */ +const LANG = 'typescript'; + /** * Resolve a dotted path (`data.data.response.responseParts`) into a nested object. * A numeric path segment indexes a list/array (`data.data.citations.0.id`) — JS @@ -211,6 +216,12 @@ describe('scenario parity — TS server runs the shared conformance corpus', () for (const path of SCENARIOS) { const scenario = JSON.parse(readFileSync(path, 'utf8')) as Scenario; + // `knownDivergences` lists the languages a scenario is known to fail on today, + // with `knownDivergencesReason` next to it. An EXPIRING marker, not a skip: a + // listed language that fails is reported and tolerated, but one that PASSES + // fails the build, so a marker cannot rot silently into a green test that + // proves nothing (which is the failure mode this whole corpus exists to catch). + const divergent = (scenario.knownDivergences ?? []).includes(LANG); it(scenario.name, async () => { server = await serve({ chatClient: buildMock(scenario.mockLlmScript ?? []), @@ -231,9 +242,16 @@ describe('scenario parity — TS server runs the shared conformance corpus', () client.sendAction(subst(step.send, vars) as Record); await matchExpected(client, step.expect, vars); } + } catch (err) { + if (!divergent) throw err; + console.warn(`[known divergence] ${scenario.name}: ${scenario.knownDivergencesReason ?? ''}\n ${String(err)}`); + return; } finally { await client.close(); } + if (divergent) { + throw new Error(`remove ${LANG} from knownDivergences in ${scenario.name}.json — it now passes`); + } }); } }); From 9e14c7468d077628d9d572887ba7cee1ba94e8cd Mon Sep 17 00:00:00 2001 From: Brent Rager Date: Wed, 19 Aug 2026 19:00:27 -0400 Subject: [PATCH 3/5] th-eae69d: record the Go cancel-while-parked data race MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CI runs `go test -race`, and cancel-mid-turn trips it: a DATA RACE in core's MockLlmProvider.ChatStream, where the CANCELLED turn's goroutine and the NEXT turn's goroutine pop the same unguarded FIFO script concurrently. That is independent, mechanical proof of the divergence the scenario's assertion already catches — the cancelled turn is genuinely still executing, not merely mis-reporting. Reproduces deterministically, locally and in CI. A knownDivergences marker cannot and must not suppress it: the race detector fails the test outside the runner's assertion path. So vet-test (go/server) stays red until Go actually aborts a cancelled turn. Documented rather than worked around — guarding the mock's FIFO would silence the evidence and leave the bug. Co-Authored-By: Claude Fable 5 --- spec/conformance/scenarios/README.md | 2 ++ spec/conformance/scenarios/cancel-mid-turn.json | 2 +- 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/spec/conformance/scenarios/README.md b/spec/conformance/scenarios/README.md index 0994a86f..b4589250 100644 --- a/spec/conformance/scenarios/README.md +++ b/spec/conformance/scenarios/README.md @@ -120,6 +120,8 @@ Implementation note per language, since `*testing.T` and panics do not catch ali Recorded here as facts, not as license to weaken the scenarios. **Do not "fix" a scenario to make a port pass.** - **Park event vs. the raise tool's `stream_chunk` — Rust is 1 of 5, and Rust is right.** For a Rich Interaction, Rust emits `interaction_required` *before* the raise tool's `toolCall` chunk; Go, TypeScript, Python and .NET all emit the chunk first. **Ruled a port bug, not a protocol variant**, on three grounds: all five already defer the gated tool's chunk until after the prompt for the *other* park type (`hitl-write-confirmation`), so the four are internally inconsistent between their two park paths while Rust is consistent; Rust is the designated reference and the ports mirror it; and semantically a client that renders tool calls would otherwise show "calling `request_identity_intake`…" before the card appears, leaking framework internals ahead of the semantic event. The four ports change, not these scenarios. +- **Go's cancelled turn also trips the race detector, and a marker cannot hide that.** `go test -race` (what CI runs) reports a `DATA RACE` in core's `MockLlmProvider.ChatStream` — the cancelled turn's goroutine and the *next* turn's goroutine pop the same unguarded FIFO script concurrently. It is independent, mechanical proof of the bullet below: the cancelled turn is genuinely still executing. It is also the one failure `knownDivergences` deliberately does not tolerate — the race detector fails the test outside the runner's assertion path, so `vet-test (go/server)` stays red until Go actually aborts a cancelled turn. Do not "fix" it by guarding the mock's FIFO: that would silence the evidence while leaving the bug. + - **A cancelled turn keeps running in Go and .NET.** In both, the turn after a `cancel` produces no reply because the *cancelled* turn consumed an extra LLM response: the write-confirmation gate returns a deny instead of unwinding, the agent loop makes one more model call, and the output is merely gagged (Go: `if turnCtx.Err() != nil { return }`). Cancellation is a mute button there, not a stop button — a real cost and a real side-effect risk after a visitor hits Stop. Rust, TypeScript and Python abort the turn properly. Verified by re-running with one extra `mockLlmScript` entry: both pass, proving the entry is eaten by the cancelled turn. - **Ack payloads differ, so only `status` is asserted on a `submit_interaction` ack.** The five servers put different fields in `data` (Go omits `kind`/`values`; Python omits `kind`, and its decline ack omits `interactionId`/`declined`; .NET's decline ack omits `declined`). Asserting more would pin one language's shape rather than the protocol's. diff --git a/spec/conformance/scenarios/cancel-mid-turn.json b/spec/conformance/scenarios/cancel-mid-turn.json index 46e99d91..d07cc0bb 100644 --- a/spec/conformance/scenarios/cancel-mid-turn.json +++ b/spec/conformance/scenarios/cancel-mid-turn.json @@ -2,7 +2,7 @@ "name": "cancel-mid-turn", "description": "A turn cancelled while in flight must emit the terminal `cancelled` event (status 499, echoing the CANCELLED TURN's requestId) IN PLACE OF `eventual_response`, and must free the connection's turn slot so the next `send_message` is accepted. The in-flight window is made deterministic by parking the turn on write-confirmation HITL (`server.confirmTools`) — a mock turn otherwise completes faster than a `cancel` frame can race it, and the scenario format has no way to express a slow tool. Per spec/actions/cancel.schema.json the cancel frame carries the requestId of the `send_message` it aborts; per spec/events/cancelled.schema.json the cancelled turn produces NO answer payload.", "knownDivergences": ["go", "dotnet"], - "knownDivergencesReason": "th-eae69d — Go and .NET do not actually abort a cancelled turn. The cancelled turn consumes one MORE LLM response after the cancel (Go's write-confirmation gate returns a deny instead of unwinding, the agent loop makes another model call) and its output is merely gagged, so the NEXT turn finds a drained mock script and streams nothing. Cancellation is a mute button there, not a stop button — after a visitor hits Stop the turn keeps burning model calls and keeps executing whatever comes next, invisibly. Rust, TypeScript and Python abort properly. Tracked as a port bug; do NOT weaken this scenario.", + "knownDivergencesReason": "th-eae69d — Go and .NET do not actually abort a cancelled turn. The cancelled turn consumes one MORE LLM response after the cancel (Go's write-confirmation gate returns a deny instead of unwinding, the agent loop makes another model call) and its output is merely gagged, so the NEXT turn finds a drained mock script and streams nothing. Cancellation is a mute button there, not a stop button — after a visitor hits Stop the turn keeps burning model calls and keeps executing whatever comes next, invisibly. Rust, TypeScript and Python abort properly. Tracked as a port bug; do NOT weaken this scenario. ⚠️ In Go this ALSO trips the race detector: `go test -race` reports a DATA RACE in core's MockLlmProvider.ChatStream (smooth-operator-core/go llm_provider.go:142/170), because the cancelled turn's goroutine and the next turn's goroutine pop the same unguarded FIFO concurrently — independent proof that the cancelled turn is still running. A knownDivergences marker CANNOT and MUST NOT suppress that: the race detector fails the test outside the runner's assertion path, so go/server stays red until the port aborts its cancelled turns.", "server": { "tools": [ { From de5d93b305a496485663e96ca9424ea4fc050834 Mon Sep 17 00:00:00 2001 From: Brent Rager Date: Wed, 19 Aug 2026 19:14:12 -0400 Subject: [PATCH 4/5] th-eae69d: unmark Go on cancel-mid-turn so the port fix proves itself MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The go marker on cancel-mid-turn bought nothing and would have backfired. It covers only the assertion; the data race (th-f2ac48) fails the test outside the runner's assertion path, so vet-test (go/server) was red with the marker anyway. And once fix-cancel-unwind lands, Go would start PASSING the assertion — firing the xpass guard and re-reding the build with 'remove go from knownDivergences' instead of going green. Unmarked, cancel-mid-turn turns green on its own the moment Go actually unwinds a cancelled turn. That is the proof the port fix worked. .NET stays marked: it has no race, so the marker is what keeps build-test green today, and its xpass guard is what will tell us when it is fixed. Also spells out in the README that hitl-write-confirmation — the scenario proving the four ports are internally inconsistent with their own other park path — is one ALL FIVE pass today, so the ordering ruling does not get re-litigated by the next reader. Co-Authored-By: Claude Fable 5 --- spec/conformance/scenarios/README.md | 2 +- spec/conformance/scenarios/cancel-mid-turn.json | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/spec/conformance/scenarios/README.md b/spec/conformance/scenarios/README.md index b4589250..b59f647f 100644 --- a/spec/conformance/scenarios/README.md +++ b/spec/conformance/scenarios/README.md @@ -119,7 +119,7 @@ Implementation note per language, since `*testing.T` and panics do not catch ali Recorded here as facts, not as license to weaken the scenarios. **Do not "fix" a scenario to make a port pass.** -- **Park event vs. the raise tool's `stream_chunk` — Rust is 1 of 5, and Rust is right.** For a Rich Interaction, Rust emits `interaction_required` *before* the raise tool's `toolCall` chunk; Go, TypeScript, Python and .NET all emit the chunk first. **Ruled a port bug, not a protocol variant**, on three grounds: all five already defer the gated tool's chunk until after the prompt for the *other* park type (`hitl-write-confirmation`), so the four are internally inconsistent between their two park paths while Rust is consistent; Rust is the designated reference and the ports mirror it; and semantically a client that renders tool calls would otherwise show "calling `request_identity_intake`…" before the card appears, leaking framework internals ahead of the semantic event. The four ports change, not these scenarios. +- **Park event vs. the raise tool's `stream_chunk` — Rust is 1 of 5, and Rust is right.** For a Rich Interaction, Rust emits `interaction_required` *before* the raise tool's `toolCall` chunk; Go, TypeScript, Python and .NET all emit the chunk first. **Ruled a port bug, not a protocol variant**, on three grounds: all five already defer the gated tool's chunk until after the prompt for the *other* park type — `hitl-write-confirmation`, a scenario **all five pass today** — so the four are internally inconsistent between their own two park paths while Rust is consistent; Rust is the designated reference and the ports mirror it; and semantically a client that renders tool calls would otherwise show "calling `request_identity_intake`…" before the card appears, leaking framework internals ahead of the semantic event. The four ports change, not these scenarios. - **Go's cancelled turn also trips the race detector, and a marker cannot hide that.** `go test -race` (what CI runs) reports a `DATA RACE` in core's `MockLlmProvider.ChatStream` — the cancelled turn's goroutine and the *next* turn's goroutine pop the same unguarded FIFO script concurrently. It is independent, mechanical proof of the bullet below: the cancelled turn is genuinely still executing. It is also the one failure `knownDivergences` deliberately does not tolerate — the race detector fails the test outside the runner's assertion path, so `vet-test (go/server)` stays red until Go actually aborts a cancelled turn. Do not "fix" it by guarding the mock's FIFO: that would silence the evidence while leaving the bug. - **A cancelled turn keeps running in Go and .NET.** In both, the turn after a `cancel` produces no reply because the *cancelled* turn consumed an extra LLM response: the write-confirmation gate returns a deny instead of unwinding, the agent loop makes one more model call, and the output is merely gagged (Go: `if turnCtx.Err() != nil { return }`). Cancellation is a mute button there, not a stop button — a real cost and a real side-effect risk after a visitor hits Stop. Rust, TypeScript and Python abort the turn properly. Verified by re-running with one extra `mockLlmScript` entry: both pass, proving the entry is eaten by the cancelled turn. diff --git a/spec/conformance/scenarios/cancel-mid-turn.json b/spec/conformance/scenarios/cancel-mid-turn.json index d07cc0bb..8302aab8 100644 --- a/spec/conformance/scenarios/cancel-mid-turn.json +++ b/spec/conformance/scenarios/cancel-mid-turn.json @@ -1,8 +1,8 @@ { "name": "cancel-mid-turn", "description": "A turn cancelled while in flight must emit the terminal `cancelled` event (status 499, echoing the CANCELLED TURN's requestId) IN PLACE OF `eventual_response`, and must free the connection's turn slot so the next `send_message` is accepted. The in-flight window is made deterministic by parking the turn on write-confirmation HITL (`server.confirmTools`) — a mock turn otherwise completes faster than a `cancel` frame can race it, and the scenario format has no way to express a slow tool. Per spec/actions/cancel.schema.json the cancel frame carries the requestId of the `send_message` it aborts; per spec/events/cancelled.schema.json the cancelled turn produces NO answer payload.", - "knownDivergences": ["go", "dotnet"], - "knownDivergencesReason": "th-eae69d — Go and .NET do not actually abort a cancelled turn. The cancelled turn consumes one MORE LLM response after the cancel (Go's write-confirmation gate returns a deny instead of unwinding, the agent loop makes another model call) and its output is merely gagged, so the NEXT turn finds a drained mock script and streams nothing. Cancellation is a mute button there, not a stop button — after a visitor hits Stop the turn keeps burning model calls and keeps executing whatever comes next, invisibly. Rust, TypeScript and Python abort properly. Tracked as a port bug; do NOT weaken this scenario. ⚠️ In Go this ALSO trips the race detector: `go test -race` reports a DATA RACE in core's MockLlmProvider.ChatStream (smooth-operator-core/go llm_provider.go:142/170), because the cancelled turn's goroutine and the next turn's goroutine pop the same unguarded FIFO concurrently — independent proof that the cancelled turn is still running. A knownDivergences marker CANNOT and MUST NOT suppress that: the race detector fails the test outside the runner's assertion path, so go/server stays red until the port aborts its cancelled turns.", + "knownDivergences": ["dotnet"], + "knownDivergencesReason": "th-eae69d — Go and .NET do not actually abort a cancelled turn. The cancelled turn consumes one MORE LLM response after the cancel (Go's write-confirmation gate returns a deny instead of unwinding, the agent loop makes another model call) and its output is merely gagged, so the NEXT turn finds a drained mock script and streams nothing. Cancellation is a mute button there, not a stop button — after a visitor hits Stop the turn keeps burning model calls and keeps executing whatever comes next, invisibly. Rust, TypeScript and Python abort properly. Tracked as a port bug; do NOT weaken this scenario. ⚠️ In Go this ALSO trips the race detector: `go test -race` reports a DATA RACE in core's MockLlmProvider.ChatStream (smooth-operator-core/go llm_provider.go:142/170), because the cancelled turn's goroutine and the next turn's goroutine pop the same unguarded FIFO concurrently — independent proof that the cancelled turn is still running. A knownDivergences marker CANNOT and MUST NOT suppress that: the race detector fails the test outside the runner's assertion path. So Go is deliberately NOT marked here (th-f2ac48, P0) — a marker would buy nothing while the race keeps go/server red anyway, and it would then FIRE the xpass guard the moment the port fix lands, red-ing the build with 'remove go' instead of letting it go green on its own. Unmarked, the port fix turns this scenario green unaided, which is the proof the fix worked. .NET has no race and stays marked; its xpass guard is what will tell us when it is fixed.", "server": { "tools": [ { From 9d87c3c2946988c548c6aec3923d1b1211441afb Mon Sep 17 00:00:00 2001 From: Brent Rager Date: Wed, 19 Aug 2026 19:39:41 -0400 Subject: [PATCH 5/5] =?UTF-8?q?th-eae69d:=20drop=20cancel-mid-turn's=20mar?= =?UTF-8?q?kers=20=E2=80=94=20the=20port=20fix=20landed?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit #514 made Go and .NET actually unwind a cancelled turn, and the corpus proved it twice, unaided: * Go was UNMARKED and went green on its own, scenario untouched, race detector included. The corpus said Go was broken, the port was fixed, and the corpus now agrees without anyone editing it. * .NET was marked, so the xpass guard FIRED on its first real opportunity: 'remove dotnet from knownDivergences in cancel-mid-turn — it now passes'. Marker removed; .NET is back to 19/19. That is the whole point of an expiring marker over a skip: it told us the moment it became a lie, instead of rotting into a green check that proved nothing. The five interaction-ordering markers (th-ef78d0) are untouched — those still genuinely fail and the ruling stands: Rust's order is correct. Co-Authored-By: Claude Fable 5 --- spec/conformance/scenarios/README.md | 4 +--- spec/conformance/scenarios/cancel-mid-turn.json | 2 -- 2 files changed, 1 insertion(+), 5 deletions(-) diff --git a/spec/conformance/scenarios/README.md b/spec/conformance/scenarios/README.md index b59f647f..35db06f2 100644 --- a/spec/conformance/scenarios/README.md +++ b/spec/conformance/scenarios/README.md @@ -120,9 +120,7 @@ Implementation note per language, since `*testing.T` and panics do not catch ali Recorded here as facts, not as license to weaken the scenarios. **Do not "fix" a scenario to make a port pass.** - **Park event vs. the raise tool's `stream_chunk` — Rust is 1 of 5, and Rust is right.** For a Rich Interaction, Rust emits `interaction_required` *before* the raise tool's `toolCall` chunk; Go, TypeScript, Python and .NET all emit the chunk first. **Ruled a port bug, not a protocol variant**, on three grounds: all five already defer the gated tool's chunk until after the prompt for the *other* park type — `hitl-write-confirmation`, a scenario **all five pass today** — so the four are internally inconsistent between their own two park paths while Rust is consistent; Rust is the designated reference and the ports mirror it; and semantically a client that renders tool calls would otherwise show "calling `request_identity_intake`…" before the card appears, leaking framework internals ahead of the semantic event. The four ports change, not these scenarios. -- **Go's cancelled turn also trips the race detector, and a marker cannot hide that.** `go test -race` (what CI runs) reports a `DATA RACE` in core's `MockLlmProvider.ChatStream` — the cancelled turn's goroutine and the *next* turn's goroutine pop the same unguarded FIFO script concurrently. It is independent, mechanical proof of the bullet below: the cancelled turn is genuinely still executing. It is also the one failure `knownDivergences` deliberately does not tolerate — the race detector fails the test outside the runner's assertion path, so `vet-test (go/server)` stays red until Go actually aborts a cancelled turn. Do not "fix" it by guarding the mock's FIFO: that would silence the evidence while leaving the bug. - -- **A cancelled turn keeps running in Go and .NET.** In both, the turn after a `cancel` produces no reply because the *cancelled* turn consumed an extra LLM response: the write-confirmation gate returns a deny instead of unwinding, the agent loop makes one more model call, and the output is merely gagged (Go: `if turnCtx.Err() != nil { return }`). Cancellation is a mute button there, not a stop button — a real cost and a real side-effect risk after a visitor hits Stop. Rust, TypeScript and Python abort the turn properly. Verified by re-running with one extra `mockLlmScript` entry: both pass, proving the entry is eaten by the cancelled turn. +- **~~A cancelled turn keeps running in Go and .NET~~ — FIXED (th-f2ac48, PR #514).** Recorded because it is what this scenario was built to catch, and because the fix is the corpus's first end-to-end proof of itself. Both ports used to leave the turn running after a `cancel`: the write-confirmation gate returned a deny instead of unwinding, the agent loop made one more model call, and the output was merely gagged (Go: `if turnCtx.Err() != nil { return }`). Cancellation was a mute button, not a stop button — real spend and real side-effect risk after a visitor hits Stop. Two independent proofs: re-running with one extra `mockLlmScript` entry made both pass (the entry was eaten by the cancelled turn), and `go test -race` reported a `DATA RACE` in core's `MockLlmProvider.ChatStream` where the cancelled turn's goroutine and the *next* turn's goroutine popped the same unguarded FIFO concurrently. That race was the one failure `knownDivergences` deliberately did **not** tolerate — it fires outside the runner's assertion path, and suppressing a data race is the opposite of what this corpus is for. The lesson if it recurs: do not "fix" it by guarding the mock's FIFO, which silences the evidence and leaves the bug. - **Ack payloads differ, so only `status` is asserted on a `submit_interaction` ack.** The five servers put different fields in `data` (Go omits `kind`/`values`; Python omits `kind`, and its decline ack omits `interactionId`/`declined`; .NET's decline ack omits `declined`). Asserting more would pin one language's shape rather than the protocol's. ## Adding a scenario diff --git a/spec/conformance/scenarios/cancel-mid-turn.json b/spec/conformance/scenarios/cancel-mid-turn.json index 8302aab8..e5f81c7e 100644 --- a/spec/conformance/scenarios/cancel-mid-turn.json +++ b/spec/conformance/scenarios/cancel-mid-turn.json @@ -1,8 +1,6 @@ { "name": "cancel-mid-turn", "description": "A turn cancelled while in flight must emit the terminal `cancelled` event (status 499, echoing the CANCELLED TURN's requestId) IN PLACE OF `eventual_response`, and must free the connection's turn slot so the next `send_message` is accepted. The in-flight window is made deterministic by parking the turn on write-confirmation HITL (`server.confirmTools`) — a mock turn otherwise completes faster than a `cancel` frame can race it, and the scenario format has no way to express a slow tool. Per spec/actions/cancel.schema.json the cancel frame carries the requestId of the `send_message` it aborts; per spec/events/cancelled.schema.json the cancelled turn produces NO answer payload.", - "knownDivergences": ["dotnet"], - "knownDivergencesReason": "th-eae69d — Go and .NET do not actually abort a cancelled turn. The cancelled turn consumes one MORE LLM response after the cancel (Go's write-confirmation gate returns a deny instead of unwinding, the agent loop makes another model call) and its output is merely gagged, so the NEXT turn finds a drained mock script and streams nothing. Cancellation is a mute button there, not a stop button — after a visitor hits Stop the turn keeps burning model calls and keeps executing whatever comes next, invisibly. Rust, TypeScript and Python abort properly. Tracked as a port bug; do NOT weaken this scenario. ⚠️ In Go this ALSO trips the race detector: `go test -race` reports a DATA RACE in core's MockLlmProvider.ChatStream (smooth-operator-core/go llm_provider.go:142/170), because the cancelled turn's goroutine and the next turn's goroutine pop the same unguarded FIFO concurrently — independent proof that the cancelled turn is still running. A knownDivergences marker CANNOT and MUST NOT suppress that: the race detector fails the test outside the runner's assertion path. So Go is deliberately NOT marked here (th-f2ac48, P0) — a marker would buy nothing while the race keeps go/server red anyway, and it would then FIRE the xpass guard the moment the port fix lands, red-ing the build with 'remove go' instead of letting it go green on its own. Unmarked, the port fix turns this scenario green unaided, which is the proof the fix worked. .NET has no race and stays marked; its xpass guard is what will tell us when it is fixed.", "server": { "tools": [ {