CL-6379: Turn event pipeline — serialize collector events, classify terminal flips - #124
Merged
Merged
Conversation
The event-collector registry test dispatches two full turns fire-and-forget,
the way the session orchestrator's agent.event listener does, and expects
every part persisted in order with both turns finalized — today the
concurrent onEvent processing drops parts ("no active turn") and leaves
the second turn running forever. The decideTerminalRunFlip tests pin the
classification a workflow-run pack's terminal events need: a section
occurrence's repo-local child run (turn__<n>) has no workflow_run row by
design and must not be reported as a foreign-deployment violation.
The event-collector registry dispatches agent events fire-and-forget so it never blocks the websocket loop, but the collector processed each event concurrently across its DB awaits. Under an onTrigger section's rapid end-of-turn burst (inference.done, connector.reply back-to-back) the reply's finalize nulled the current turn while inference.done was still inserting parts — dropped as "no active turn" — and a finalize landing during the next turn's begin-insert marked the new turn finalized at birth, leaving its inference_turn row running forever (stuck busy status, no live output). The collector now chains onEvent/abandon through an internal tail promise so events settle in wire order while dispatch stays fire-and-forget. receiveWorkflowRunPack's terminal-event flip treated every run id without a workflow_run row anchored on the source deployment as an ownership violation. A section occurrence's child run (turn__<n>) is repo-local: it never mints a row, so every completed chat turn logged a spurious ERR and the benign/defect cases were indistinguishable. The flip now classifies through the pure decideTerminalRunFlip: repo-local occurrence ids skip quietly, a minted id with no row still reports loudly, and a row anchored elsewhere is still refused.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem (CL-6379)
On a fresh stack, every chat turn logged:
WRN hub·event-collector: Dropping 'text'/'step-start'/'step-finish' part: no active turn for session 'ses_…'ERR hub·lookups: Ignoring terminal event for run 'turn__1': it does not belong to source deployment 'run_…'and an agent's turn could be left permanently "running" — stuck busy status, no clean termination, live output lost.
Root causes
1. The event collector processed events concurrently. The registry's
dispatchfirescollector.onEvent(event)without awaiting (deliberately, so the websocket loop never blocks), but the collector had no internal ordering. An onTrigger section's end-of-turn burst (inference.done,connector.reply, nextinference.startback-to-back) interleaves across the collector's DB awaits:connector.reply's finalize nullscurrentTurnIdwhileinference.doneis still inserting parts → parts dropped as "no active turn", and persisted ordinals come out scrambled (a real run showedtextat ordinal 0,step-startat 1,step-finishdropped).beginTurninsert is in flight marks the new turnfinalizedat birth → its ownconnector.replyshort-circuits and theinference_turnrow staysrunningforever (permanently busy status, idle-sleep blocked, "typing dots" that never resolve).Fix: the collector chains
onEvent/abandonthrough an internal tail promise, so events settle in wire order while dispatch stays fire-and-forget.2. The terminal-event flip misclassified occurrence child runs.
receiveWorkflowRunPacktreated any newly-terminal run id without aworkflow_runrow anchored on the source deployment as an ownership violation. A section occurrence's child run (turn__<n>, CL-6329) is repo-local by design — it never mints a row — so every completed chat turn logged a spurious ERR, indistinguishable from a real violation. Fix: the flip classifies through the new puredecideTerminalRunFlip: repo-local ids skip quietly, a mintedrun_<hex>id with no row still reports loudly, a row anchored elsewhere is still refused.Verification
vendor/intx/hub-sessions/src/event-collector-registry.test.tsdispatches two full turns fire-and-forget and asserts every part persists in order and both turns finalize (failed before the fix exactly as in production);decideTerminalRunFlipclassification tests inhub-session-lookups.test.ts.turn__NERRs, secondinference_turnstuckrunning, parts out of order; after — zero WRN/ERR, both turnscompleted, partsstep-start/text/step-finishat ordinals 0/1/2.vendor/intx/hub-sessions,packages/chat,packages/folded-runs,packages/insightstest suites green; repo typecheck green; VENDORED.md records the hub-sessions local modifications.