Resuming a task that released its slot (human-parked) re-enters without re-acquiring a slot, pushing in-flight tasks over the cap

[ryanrborn]

Summary

Resuming a task that had released its worker slot (parked for a human, stopped, completed) used to re-enter it without asking about the cap. On 2026-09-23 (cap 1) that let arb worker resume bd-3usjdj start after Autopilot had already admitted bd-d7mfdq into the freed slot, leaving two tasks in flight. The resume gate now checks whether the task holds a slot right now, not who is resuming it.

The rule: Arbiter.Worker.ResumeSlot.admit/2 (new), called by Dispatch.resume/2 and Dispatch.resume_session/2. It runs after the resume's own validity checks and before the prior worker is stopped, so a refusal or a deferral leaves the task exactly as it was.

• Holds its slot means the author row's phase still occupies one. This is the same answer the board counts, via the new SlotGate.slot_holders/2 over Phase.annotate/1. When no worker is registered (after a reboot), the task counts as holding if its latest main run was cut off by the restart (:running, :interrupted, or "server restarted"). A resume of such a task passes through uncapped, whatever the caller, which keeps the Scheduler slots count live agent sessions, not worker records; show the issue's real phase instead of "running" after the main agent exits #1969/Autopilot admission should hold a slot per in-flight TASK until its PR merges, not per live agent session (operator rule; builds on SlotGate/Phase from #1969) #1995 no-deadlock guarantee.
• Released means the resume must re-acquire a slot against the workspace's effective cap (Snapshot.effective_max_concurrent/2). When the cap is full:
  • resume_origin: :human (the default): refused with {:slot_cap_full, %{cap, holders}}. refusal_message/1 names the cap, the tasks holding the slots, and how to force. With force_slot: true the resume goes over the cap, and the override is recorded as a slot_cap_override audit event (Arbiter.Events: actor, cap, holders) and in the worker's meta[:slot_cap_override].
  • resume_origin: :automatic: deferred. The resume is handed to Board.Autopilot.defer_resume/4 and the caller gets {:ok, %{deferred: true, ...}}. If no scheduler can take the deferral, the resume is refused, never let over the cap.

Hand-offs keep the slot. The fix round and the Watchdog auto-resume both fail the author worker only so an automatic round can replace it. Before this change that failed state read as :waiting_on_you, which releases the slot. Those failures now carry meta[:slot_handoff] (Worker.fail/3, Worker.clear_slot_handoff/1), and Phase reads a failed worker with that marker as :handing_off, which holds the slot. Every give-up arm drops the marker before it pages: budget spent, not converging, dispatch failed, deferrals exhausted. This also closes the short window where the board showed a free slot during a fix-round hand-off.

Deferral queue (Board.Autopilot). Deferred resumes wait in an in-memory queue, oldest first; deferring the same task again replaces its options but keeps its place. On each pass:

• The oldest deferred resume takes the first free slot, ahead of any Ready card. While a resume is queued, no Ready card is promoted.
• A pause does not hold deferred resumes back, because pausing stops new dispatches only (Board.Drain).
• The replay is Dispatch.resume/2 (or resume_session/2) with the caller's own options plus slot_admitted: true.
• If the replay loses a race (the task is already active, closed or gone), the resume is dropped quietly. Any other failure is escalated once and dropped.
• Besides the existing triggers, a worker_phase event to waiting_on_you or done asks for a pass while a resume is queued.

Every Dispatch.resume / resume_session caller

Caller	Kind	Handling
MCP worker_resume (mcp/tools/worker.ex)	human	refused at a full cap with the cap and holders. New force arg (in the catalog schema) overrides; actor coordinator
API POST /api/workers/:id/resume (worker_controller.ex, resume_session)	human	409 with the message and details: {cap, holders}. "force": true overrides; actor api
CLI arb worker resume / arb resume (arbiter_cli/cmd/worker.ex)	human	new --force flag sends force: true; the 409 message is surfaced
Board Resume / proceed (board_live.ex, Worker.resume/1 on an :awaiting worker)	human	gated before un-parking: at a full cap the card stays parked and the flash names the cap and holders (no override on the board)
Worker-detail Retry (worker_detail_live.ex)	human	refusal shown inline in the modal, which then offers Resume over the cap (#worker-retry-force-btn), recorded with actor dashboard
Boot reconciler (workers/reconciler.ex)	automatic	resume_origin: :automatic; restart-interrupted tasks (holding) are resumed first, then released tasks compete for what's left and are deferred if nothing is
Watchdog auto-resume (auto_resume_dispatcher.ex, from watchdog.ex)	automatic	the awaiting-review timeout and the unreviewed-head reroute now fail the worker as a slot hand-off, so the resume passes uncapped; if the slot was released it is deferred
MergeQueue revise (revise_dispatcher.ex)	automatic	a finished worker released its slot, so at a full cap the revise is deferred
ReviewGate fix round (review_gate_fix_round_dispatcher.ex)	automatic	the author carries a slot hand-off, so the round spawns at a full cap; deferred only if the task released its slot meanwhile
Autopilot deferred-resume replay (new)	automatic	slot_admitted: true; the scheduler already found the free slot

Other touch points:

• The test env points the new :resume_deferrer setting at Arbiter.Test.StubResumeDeferrer, so no test can queue a real resume on the app's Autopilot.
• Arbiter.Test.ResumeSlotFixture builds the incident on Arbiter.TestSandbox, so every agent CLI is stubbed.
• Guard-policy doc citations are re-anchored for the moved lines in watchdog.ex / worker.ex, and the guard-registry exemption is updated to Worker.fail/3.

Test plan

• Acceptance 1: a resume of a slot-holding task is never cap-checked (resume_slot_test.exs, both origins). A ReviewGate fix round spawns at a full cap (dispatch_resume_slot_test.exs, "a ReviewGate fix round spawns at a full cap"). slot_gate_no_deadlock_test.exs still passes. Hand-off held and released: review_gate_fix_round_test.exs, watchdog_test.exs, watchdog_reviewed_sha_test.exs. The release assertions were mutation-checked: removing the release makes them fail.
• Acceptance 2: refused at a full cap on each human surface, naming the cap and the holders, and allowed with force plus an audit record: MCP (mcp/worker_resume_slot_test.exs), API (api/worker_resume_slot_test.exs, 409 then 201 with force), CLI (resume_test.exs, --force body and 409 surfaced), board (board_live_test.exs), worker detail (worker_detail_resume_slot_test.exs, refusal then override).
• Acceptance 3: automatic resumes are deferred, not failed or bypassed. Revise, Watchdog auto-resume, fix round and boot reconciler each defer (dispatch_resume_slot_test.exs). Queue order, ahead of Ready, draining while paused, and quiet vs escalated drops (autopilot_deferred_resume_test.exs). The real Autopilot replays a real deferred Dispatch.resume once a slot frees ("the scheduler replays it for real once a slot frees"). The reconciler resumes restart-interrupted work first (reconciler_test.exs).
• Acceptance 4: incident regression (cap 1, A parked, B admitted): refused without force, admitted and recorded with force (dispatch_resume_slot_test.exs, "the 2026-09-23 incident").
• Regression suites for the touched code: test/arbiter/board, test/arbiter/worker, test/arbiter/workflows, test/arbiter/tasks, reconciler_test.exs, guard_registry_test.exs, review_coverage_design_test.exs, worker_controller_test.exs, worker_detail_live_test.exs, board_live_test.exs.
• mix precommit: exit 0 (7450 + 1716 + 889 tests, 0 failures), run with the worker's ARB_* env unset.
• mix credo --strict, mix format --check-formatted, mix deps.unlock --check-unused, sobelow: clean.
• mix dialyzer: one unskipped warning, at arbiter_cli/cmd/doctor/checks.ex:177 (dev_install? guard_fail). That file is untouched by this branch, so the warning predates it.
• Not exercised against a live server. The deferral queue is in memory, so a restart drops queued resumes. After a restart the boot reconciler re-resumes mid-flight tasks and the patrols re-watch open PRs.

References

• Task: bd-92mx1m
• Follow-up to bd-45pwo1 (Autopilot admission should hold a slot per in-flight TASK until its PR merges, not per live agent session (operator rule; builds on SlotGate/Phase from #1969) #1995); preserves Scheduler slots count live agent sessions, not worker records; show the issue's real phase instead of "running" after the main agent exits #1969 / Autopilot admission should hold a slot per in-flight TASK until its PR merges, not per live agent session (operator rule; builds on SlotGate/Phase from #1969) #1995's no-deadlock guarantee
• Closes Resuming a task that released its slot (human-parked) re-enters without re-acquiring a slot, pushing in-flight tasks over the cap #2010

🤖 Generated with Claude Code