Bound post-deploy mail delivery so a stalled ack fails loud (CL-6644) - #316
Merged
Conversation
sendFoldedMailWithReclaimRetry's own retry delays only run between attempts that already threw "agent is unreachable" — an attempt that instead stalls forever without ever settling hangs the caller's await with it. Proves the hang by making sessionService.sendUserMessage never resolve and asserting sendMail rejects within a short injected deadline instead of timing out the test.
Fixes the last hop in CL-6644's cold-wake chase: after a wake-deploy completes, sendFoldedMailWithReclaimRetry's reclaim-retry loop only bounds attempts that fail loud with "agent is unreachable" — an attempt that instead stalls (a wedged ack, or anything else in sendFoldedMail's call chain that never settles) hangs the loop's await forever with nothing logged, matching the observed symptom exactly: Deployed agent / Wrote run grants, then silence. Wraps each attempt in the same kind of wall-clock bound wakeByAddressBounded already puts on the wake itself (extracted as a shared withTimeout helper), via a new injectable mailDeliveryTimeoutMs deps field defaulting to DEFAULT_WAKE_TIMEOUT_MS. A stalled attempt now rejects instead of hanging, propagating through sendMail to dispatchTurnBatch's existing reportError catch (#313), which posts the undelivered notice with a refId.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CL-6644: https://linear.app/abklabs/issue/CL-6644/cold-wake-never-completes-and-its-failure-leaves-zero-log-trace
Per the ticket's last comment, #312/#313/#314/#315 traced the whole intake→fanout→wake chain healthy: recipients resolve, dispatch starts, the wake-deploy completes ("Deployed agent", "Wrote run grants") — then nothing. No mail line, no run start, no inference, no server notice.
dispatchTurn's promise stayed pending in the one hop none of the earlier fixes bound: post-deploy mail delivery.Root cause
sendFoldedMailWithReclaimRetry(packages/chat/src/platform-adapter.ts) retries a mail send that throws"agent is unreachable", with a ~7.75s backoff (RECLAIM_RETRY_DELAYS_MS). That backoff only runs between attempts that already failed loud. An attempt that instead stalls — a wedged sidecar ack, or anything else insendFoldedMail's call chain that never settles — hangs the loop'sawaitforever with nothing logged, because nothing ever throws. This matches the observed symptom exactly.wakeByAddressBoundedalready puts a wall-clock bound on the wake step for the same reason (CL-6644 part B / #314); the mail-delivery step had no equivalent bound.Fix
wakeByAddressBounded's timeout logic into a sharedwithTimeouthelper.sendFoldedMailattempt insidesendFoldedMailWithReclaimRetryinwithTimeout, via a new injectablemailDeliveryTimeoutMsdeps field (defaults toDEFAULT_WAKE_TIMEOUT_MS)."agent is unreachable", so it isn't retried — it propagates throughsendMailtodispatchTurnBatch's existingreportErrorcatch (chat: log the undelivered-turn cause and thread a reportError refId (CL-6644, part A) #313), which posts the undelivered notice with arefId.Test
Added a red/green test in
packages/chat/test/platform-adapter.test.ts: makessessionService.sendUserMessagereturn a promise that never settles, and assertssendMailrejects within a short injected deadline instead of hanging. Verified red without the fix (test timed out with the fix reverted) and green with it (~1s).Not run
Live repro against a real Ollama-backed stack (send → typing → real reply) — timeboxed, same as prior sessions on this ticket. The fix is proven by the unit test above plus code reading of the retry loop; a live session would additionally confirm nothing upstream of this hop still stalls silently.
Checks
WORKBENCH_CHECK_SINCE=origin/main bun run typecheck— passWORKBENCH_CHECK_SINCE=origin/main bun run test— pass (652 pass / 0 fail in@corbits/chat)bun run lint— pass (pre-existing warnings only)