Skip to content

agent-lifecycle: bound ensureAwake so a hung wake can't wedge a workbench silently forever - #312

Merged
TheGreatAxios merged 2 commits into
mainfrom
cl-6643-silent-turn
Aug 22, 2026
Merged

agent-lifecycle: bound ensureAwake so a hung wake can't wedge a workbench silently forever#312
TheGreatAxios merged 2 commits into
mainfrom
cl-6643-silent-turn

Conversation

@TheGreatAxios

Copy link
Copy Markdown
Contributor

Summary

CL-6643: a message POST was accepted, then silently swallowed — no fanout log, no wake attempt, no error, no undelivered notice — for a workbench whose agent's run had no sidecar-side deployment record (parked aside, cold wake required).

The prime suspect in the ticket was PR #298's wireHash drift-reconcile logic in packages/chat/src/platform-adapter.ts. That turned out not to be the mechanism: the drift-reconcile path (reconcileDriftedRun) already catches and logs on failure and proceeds ("nothing to reconcile") rather than swallowing, and wakeByAddress's branches all throw or delegate rather than early-returning silently.

The actual mechanism is in packages/agent-lifecycle/src/index.ts's createAgentLifecycle. ensureAwake coalesces concurrent callers for the same address onto one in-flight wake() call via a pendingWakes map, releasing the entry in a .finally() once wake() settles. But nothing bounded how long wake() could take — a cold wake for a run with no sidecar record is a real deploy round-trip to the host, and if that round-trip never acks (confirmed live: reproduced by sending fresh messages into the broken run_d661612… workbench — POST accepted every time, then permanent silence, no fanout/wake/error/undelivered-notice lines ever), the wake() promise never settles. Its .finally never runs, so the address is wedged in pendingWakes forever: every later ensureAwake call for that address — i.e. every future message sent to that workbench — coalesces onto the same dead promise and hangs too, for the rest of the process's life, with nothing ever rejecting for sendMail/dispatchTurnBatch to turn into an undelivered notice.

Fix

ensureAwake now races the injected wake port against a new wakeTimeoutMs option (default 30s) via wakeWithTimeout, and always settles — clearing the address out of pendingWakes on either outcome. A wake that never acks now rejects instead of hanging, which dispatchTurnBatch's existing catch already turns into a real undelivered notice on the timeline.

Test plan

  • Added a red test (packages/agent-lifecycle/test/index.test.ts) reproducing the swallow: a wake that never settles must not permanently wedge the address — a first call rejects on timeout, and a second, later call for the same address gets its own fresh wake attempt rather than coalescing onto the dead promise. Confirmed this test hangs (no output, timeout-killed) against the pre-fix code.
  • WORKBENCH_CHECK_SINCE=origin/main bun run typecheck — pass
  • WORKBENCH_CHECK_SINCE=origin/main bun run test — pass (198 tests across affected packages, including the new one)
  • bun run lint — pass (0 errors)
  • Reproduced live against the running dev stack (wb_verify_0822, workbench run_d661612…): sent a fresh message ("Third wake check...") into the already-broken workbench; POST accepted (201), then confirmed the exact same permanent silence the ticket describes, with no fanout/wake/error lines at all — matching the mechanism above.
  • Live send → typing indicator → real reply on run_d661612… with this fix loaded: not done. The fix lives in this worktree; the shared dev stack (.worktrees/cl-e2e-main, log e2e-final4.log) is running the pre-fix code from its own checkout, and other work depends on it staying up, so I did not restart it to load this branch. Whoever picks this up for review should restart that stack (or a fresh one) on this branch to confirm the end-to-end cold-wake-then-reply path.

CL-6643: a run whose deployed record was parked aside can hang inside
the injected wake port forever with no ack from the host. Without a
bound, that first hung call never leaves pendingWakes, so every later
ensureAwake for the same address coalesces onto the same dead promise
and hangs too, silently, for the rest of the process's life.
A wake for an address with no sidecar-side deployment record is a real
round-trip to the host, not a local check. Previously, if that call
never resolved or rejected, its promise sat forever in pendingWakes:
ensureAwake's own .finally never ran, so every later call for the same
address coalesced onto that dead promise and hung too, with nothing
ever bubbling up to sendMail to report as an undelivered notice.

ensureAwake now races the injected wake port against wakeTimeoutMs
(defaulting to 30s) and always settles, clearing the address out of
pendingWakes on either outcome. A wake that never acks now surfaces as
a rejection callers can turn into the undelivered notice this class of
failure has always needed, instead of leaving the workbench silently
dead for the rest of the process's life.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant