Skip to content

CL-6379: Turn event pipeline — serialize collector events, classify terminal flips - #124

Merged
TheGreatAxios merged 3 commits into
mainfrom
cl-6379-turn-events
Aug 20, 2026
Merged

TheGreatAxios merged 3 commits into
mainfrom
cl-6379-turn-events

Conversation

@TheGreatAxios

Copy link
Copy Markdown
Contributor

Problem (CL-6379)

On a fresh stack, every chat turn logged:

  • WRN hub·event-collector: Dropping 'text'/'step-start'/'step-finish' part: no active turn for session 'ses_…'
  • ERR hub·lookups: Ignoring terminal event for run 'turn__1': it does not belong to source deployment 'run_…'

and an agent's turn could be left permanently "running" — stuck busy status, no clean termination, live output lost.

Root causes

1. The event collector processed events concurrently. The registry's dispatch fires collector.onEvent(event) without awaiting (deliberately, so the websocket loop never blocks), but the collector had no internal ordering. An onTrigger section's end-of-turn burst (inference.done, connector.reply, next inference.start back-to-back) interleaves across the collector's DB awaits:

  • connector.reply's finalize nulls currentTurnId while inference.done is still inserting parts → parts dropped as "no active turn", and persisted ordinals come out scrambled (a real run showed text at ordinal 0, step-start at 1, step-finish dropped).
  • A finalize processed while the NEXT turn's beginTurn insert is in flight marks the new turn finalized at birth → its own connector.reply short-circuits and the inference_turn row stays running forever (permanently busy status, idle-sleep blocked, "typing dots" that never resolve).

Fix: the collector chains onEvent/abandon through an internal tail promise, so events settle in wire order while dispatch stays fire-and-forget.

2. The terminal-event flip misclassified occurrence child runs. receiveWorkflowRunPack treated any newly-terminal run id without a workflow_run row anchored on the source deployment as an ownership violation. A section occurrence's child run (turn__<n>, CL-6329) is repo-local by design — it never mints a row — so every completed chat turn logged a spurious ERR, indistinguishable from a real violation. Fix: the flip classifies through the new pure decideTerminalRunFlip: repo-local ids skip quietly, a minted run_<hex> id with no row still reports loudly, a row anchored elsewhere is still refused.

Verification

  • Red/green: vendor/intx/hub-sessions/src/event-collector-registry.test.ts dispatches two full turns fire-and-forget and asserts every part persists in order and both turns finalize (failed before the fix exactly as in production); decideTerminalRunFlip classification tests in hub-session-lookups.test.ts.
  • Live proof on a scratch stack (real hub + sidecar + Ollama, two messages into a fresh chat): before — both WRN drops and both turn__N ERRs, second inference_turn stuck running, parts out of order; after — zero WRN/ERR, both turns completed, parts step-start/text/step-finish at ordinals 0/1/2.
  • vendor/intx/hub-sessions, packages/chat, packages/folded-runs, packages/insights test suites green; repo typecheck green; VENDORED.md records the hub-sessions local modifications.

The event-collector registry test dispatches two full turns fire-and-forget,
the way the session orchestrator's agent.event listener does, and expects
every part persisted in order with both turns finalized — today the
concurrent onEvent processing drops parts ("no active turn") and leaves
the second turn running forever. The decideTerminalRunFlip tests pin the
classification a workflow-run pack's terminal events need: a section
occurrence's repo-local child run (turn__<n>) has no workflow_run row by
design and must not be reported as a foreign-deployment violation.
The event-collector registry dispatches agent events fire-and-forget so it
never blocks the websocket loop, but the collector processed each event
concurrently across its DB awaits. Under an onTrigger section's rapid
end-of-turn burst (inference.done, connector.reply back-to-back) the reply's
finalize nulled the current turn while inference.done was still inserting
parts — dropped as "no active turn" — and a finalize landing during the
next turn's begin-insert marked the new turn finalized at birth, leaving its
inference_turn row running forever (stuck busy status, no live output). The
collector now chains onEvent/abandon through an internal tail promise so
events settle in wire order while dispatch stays fire-and-forget.

receiveWorkflowRunPack's terminal-event flip treated every run id without a
workflow_run row anchored on the source deployment as an ownership
violation. A section occurrence's child run (turn__<n>) is repo-local: it
never mints a row, so every completed chat turn logged a spurious ERR and
the benign/defect cases were indistinguishable. The flip now classifies
through the pure decideTerminalRunFlip: repo-local occurrence ids skip
quietly, a minted id with no row still reports loudly, and a row anchored
elsewhere is still refused.
@TheGreatAxios
TheGreatAxios merged commit 3b199ab into main Aug 20, 2026
0 of 2 checks passed
@TheGreatAxios
TheGreatAxios deleted the cl-6379-turn-events branch August 25, 2026 15:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant