Skip to content

Fix startup crash when a stuck workdir lock survives an interrupt - #596

Merged
TheGreatAxios merged 2 commits into
mainfrom
cl-5753-startup-after-an-interrupt-crashes-with-an-agent-already-open
Aug 24, 2026
Merged

Fix startup crash when a stuck workdir lock survives an interrupt#596
TheGreatAxios merged 2 commits into
mainfrom
cl-5753-startup-after-an-interrupt-crashes-with-an-agent-already-open

Conversation

@TheGreatAxios

@TheGreatAxios TheGreatAxios commented Aug 24, 2026

Copy link
Copy Markdown
Collaborator

Closes CL-5753. Upstream ordering bug in @intx/agent filed as CL-6984, referencing this PR.

Reproduction status

Could not reproduce the operator's exact reported crash on current main (43+ merges landed since the ticket was filed; no test or code currently touches AgentContextLockError at all, so nothing recently could have fixed it either). But reading the actual acquisition path confirmed a real, still-present bug matching the ticket's diagnosis exactly, which this PR fixes with a regression test proving the failure mode and the fix.

Root cause

@intx/agent's close() (node_modules/@intx/agent/dist/agent.js, lines 548-592) sets closed = true at line 551 — before teardown (reactor.abort(), sendQueue.drain(), the shutdown-complete race, lines 552-591) — and only calls lock.release() last, at line 592. If any step in between throws — most likely exactly when an operator interrupts mid-inference, which is when those paths are under the most stress — the lock is never released. Because closed is already true, a retried close() short-circuits at line 549 and also never reaches lock.release(). The lock is now stuck held for the rest of the process.

src/tui/runner.ts has three rebuild sites (interrupt, reloadIfIdle, session rotation) that call currentAgent.close().catch(...) — deliberately swallowing close errors — and then call buildAgent(). When close() fails to reach lock.release(), a second createAgent() for the same workdir throws AgentContextLockError ("an agent is already open for workdir: ...") for a lock nothing will ever free.

The unhandled-rejection path is concrete, not inferred: session-operation-queue.ts's enqueue does tail = tail.then(op, op); return tail;, and reloadIfIdle calls void enqueueOp(async () => { ... }) with no internal try/catch — so a rejecting op has nothing to catch it, and void discards the only promise reference that could. That is the unhandled rejection from the ticket. interrupt() and session rotation already had a try/catch around this sequence, so they failed contained; reloadIfIdle did not.

This is not a stale on-disk lock file (there isn't one — the lock is deliberately in-process only, per its own doc comment) and no lock-file/expiry/force-unlock mechanism has been introduced.

Fix

  • Extracted a shared closeAgentForRebuild(agent, context) helper: closes the current agent, swallows and logs the close error same as before, but now returns whether it succeeded.
  • Extracted a shared agentRebuildFailure(err) helper: turns AgentContextLockError into a clear, plain-language message ("Could not start a new agent: the previous one did not shut down cleanly. Restart Corbits to continue.") and passes any other error through unchanged.
  • interrupt and reloadIfIdle now: on a failed close, skip the second createAgent() call entirely (eliminating the doomed second acquisition) and surface agentRebuildFailure through the existing fatalBuildError/recordRunError path instead of throwing raw. reloadIfIdle now has the try/catch it was missing, so this can no longer escape as an unhandled rejection.
  • Session rotation is deliberately NOT routed through this helper, and that's now documented at its call site and next to closeAgentForRebuild's doc comment: rotation always mints a fresh sessionId/workdir before calling buildAgent(), so even a close() that leaks the old workdir's lock can never be re-acquired there — buildAgent() targets a directory nothing has ever locked. The old lock still leaks for the rest of the process in that case, but nothing tries to re-acquire it, so there's no crash to guard against on that path. Left as-is rather than adding a guard that would do nothing, per review.

What this does and doesn't fix

This removes the crash, not the lock leak. On a failed close, the session becomes a graceful dead end: currentAgent stays the closed agent, the next send() throws the friendly error, and that workdir can never build a new agent for the rest of the process. Restart is the only recovery — same as before, just without the crash and stack trace. The actual ordering bug in @intx/agent (marking closed before lock.release(), making retries no-ops) is the real fix and is out of scope here; it's filed upstream as CL-6984.

Test plan

  • tests/unit/tui/runner.test.ts: closeAgentForRebuild/agentRebuildFailure unit tests, plus an end-to-end test that drives the real session-operation-queue.ts the same way reloadIfIdle actually calls it (void enqueue(...), return value never awaited) with a real process.on("unhandledRejection") listener, proving the rejection is contained and surfaces through fatalBuildError rather than escaping. reloadIfIdle itself can't be exercised in isolation — it's a closure over ~15 of runTUI's local variables (currentAgent, buildAgent, streamPromise, workflowController, pendingReload/inFlight, fatalBuildError, etc.) with no seam short of standing up the full TUI runner (provider config, plugin discovery, MCP wiring, a real OpenTUI host), which is out of scope for this fix.
  • bun run check green in the foreground (lint: 0 errors/1364 pre-existing warnings, typecheck clean, build clean, 5371 tests passing / 0 failed).

@linear-code

linear-code Bot commented Aug 24, 2026

Copy link
Copy Markdown

CL-5753

@TheGreatAxios
TheGreatAxios enabled auto-merge (squash) August 24, 2026 05:06
…rejection

close() on the agent package releases its workdir lock only after
reactor.abort()/sendQueue.drain() and the shutdown-complete race finish. A
throw partway through (most likely right when an operator interrupts
mid-inference, exactly when those paths are stressed) leaves the lock held
forever in-process: the agent is already marked closed, so retrying close()
is a silent no-op that can never release it. The next buildAgent() for that
workdir then throws AgentContextLockError, and reloadIfIdle's rebuild had no
try/catch around it, so the throw escaped as an unhandled rejection and
crashed the process.

Route every rebuild site (interrupt, reload, session rotation) through a
shared close-then-check helper: a failed close now short-circuits the
rebuild instead of attempting a second, doomed acquisition, and the failure
surfaces as a plain-language caught error.
Session rotation was never routed through closeAgentForRebuild: it mints a
fresh sessionId/workdir before rebuilding, so a leaked lock on the old
workdir can never be re-acquired there. Write that reasoning down at the
call site and next to closeAgentForRebuild's doc comment, since the
asymmetry across the three rebuild sites needs an explanation the next
reader can find.

Replace the helper-only regression test with one that drives the real
session-operation-queue the same way reloadIfIdle actually calls it (void
enqueue(...), no awaited return value) and asserts, via a real
process.on("unhandledRejection") listener, that the rejection is contained
and surfaces through fatalBuildError instead of escaping. reloadIfIdle
itself can't be reached in isolation without standing up the full TUI
runner; that's noted at the test.
@TheGreatAxios
TheGreatAxios force-pushed the cl-5753-startup-after-an-interrupt-crashes-with-an-agent-already-open branch from f6709c4 to 5750fd1 Compare August 24, 2026 05:09
@TheGreatAxios
TheGreatAxios merged commit 6c64f65 into main Aug 24, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant