Chat intake: unconditional tracing from message-persist to turn dispatch (CL-6644) - #315
Merged
Conversation
…tch (CL-6644) Investigating a report of a message posting successfully but no agent turn ever being attempted (no error, no fanout log, no reportError, nothing), every log statement on this path today only fires on failure — a turn that silently resolves zero recipients, or one that stalls before dispatchTurnBatch's own try/catch, is indistinguishable from total silence in the logs. Live repro against the shared stack (build daed6b2) confirmed the room's host participant ("myra") was present in settings and the first attempt did reach a sidecar deploy-pack apply, but a repeat send to the same (by then already-deployed) agent produced zero log output at any layer, ruling out the wake/deploy-hang theory #312-#314 already covered as the sole explanation for every occurrence. Root cause not nailed down within this timebox — no fix is guessed here. - routeToRecipients now logs the resolved recipient list (or its emptiness) for every message, not only on failure. - dispatchTurnBatch logs before starting its per-recipient dispatch. - Hub boot logs an explicit confirmation that turnQueue, chatOrchestrator, and chatPlatform were constructed, so a future composition-root wiring mistake shows up in the boot log instead of only in a missing reply days later. Together these turn the exact gap this investigation hit — routing ran successfully but nothing anywhere said so — into a line every boot carries, for whoever picks this back up.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Context
CL-6644: a message POST is accepted but the agent turn dies silently — no error, no
message-fanoutlog, noreportError, nothing — somewhere between the POST handler persisting the message anddispatchTurnBatchrunning. #312/#313/#314 covered the wake-hang path (bypassed timeouts on directwakeByAddresscalls); this PR picks up the next lead from CL-6644's last comment: the message POST handler's post-persist hook and the composition-root wiring inapps/hub/src/index.ts.What I found
Traced the full path from
POST /workbenches/:id/messagesthroughsendWorkbenchMessage→routeMessage→routeToRecipients→turnQueue.run→dispatchTurnBatch→dispatchTurn→platform.sendMail→sendFoldedMailWithReclaimRetry→sendFoldedMail→ the vendoredsessionService.sendUserMessage. Every step is either synchronous, bounded (the reclaim-retry backoff tops out around 7.75s,wakeByAddressBoundedatDEFAULT_WAKE_TIMEOUT_MSper #314), or already inside atry/catchthat logs viareportError.Live repro against the shared stack (build
daed6b24) confirmed the room's host participant ("myra") is actually present in the workbench's settings (not an empty-recipients case), and the original repro's boot log shows a sidecar deploy-pack apply immediately after the failing POST. But re-sending to the same room, once that agent was already deployed and routable, produced zero log output at any layer — no wake, no deploy, nomessage-fanout, noreportError— which rules out the wake/deploy-hang theory as the sole explanation, since a repeat send shouldn't need to wake anything.I did not find a smoking gun within the timebox, and per the "no guessing" bar #313 set, I'm not shipping a speculative fix. Composition-root wiring in
apps/hub/src/index.tsforturnQueue/chatOrchestrator/chatPlatformreads correctly wired (verified against the #305 tasks-deletion and #294 folded-runs-split diffs — both are import-path/doc-comment changes only, no logic touched).What this PR does instead
The one concrete, verifiable gap this investigation surfaced: every log statement on this path fires only on failure. A turn that resolves zero recipients, or one that stalls before
dispatchTurnBatch's owntry/catch, is indistinguishable from total silence — exactly what happened here, twice, under two different circumstances I could not tell apart from logs alone.routeToRecipientsnow logs the resolved recipient list (or its emptiness) for every message, unconditionally.dispatchTurnBatchlogs before starting its per-recipient dispatch.turnQueue,chatOrchestrator, andchatPlatformwere constructed, so a future composition-root wiring mistake shows up in the boot log rather than only in a missing reply days later.This doesn't fix the bug — it makes the next occurrence diagnosable from logs alone, without needing a live repro session first.
Verification
WORKBENCH_CHECK_SINCE=origin/main bun run typecheck— passWORKBENCH_CHECK_SINCE=origin/main bun run lint— passWORKBENCH_CHECK_SINCE=origin/main bun run test— pass (198 tests,@corbits/chatincluded):3000stack; confirmed 201 + zero downstream log activity, matching the report exactly.Not done
sendFoldedMail/sendUserMessage's vendored call chain and re-drive the live repro with the new tracing from this PR in place — it will now show whether recipients resolved and whether dispatch started, narrowing the search to whatever comes after that line logs.