worker_list reports zero workers while a worker is actively running, leading the coordinator to stop live work

[ryanrborn]

Summary

worker_list (and arb worker list / arb prime) could report zero workers while a worker was genuinely alive and running, which once led the coordinator to worker_stop a live in-flight session. Two independent, compounding bugs caused this:

1. Worker.list_children/0 silently dropped a live worker that missed its :snapshot probe. safe_snapshot/1 gave a worker only 500ms to answer before list_children/0 treated it exactly like a crashed child — a genuinely wedged/busy worker (e.g. draining a burst of mix test output through its mailbox) reads no differently from one that no longer exists. Worker.state/1 — what worker_show / worker_runs use — has no such tight budget, which is why those correctly reported the worker as running at the same instant worker_list reported none.

   Fixed by degrading instead of dropping: a worker that is confirmed alive? but doesn't answer within the (now 5s, matching state/1's effective default) probe window is still emitted, as a status: :unknown / meta.stale_probe: true entry sourced from its latest Arbiter.Workers.Run row — the same fallback worker_show_historical/2 already uses for a worker that has actually exited. This mirrors the pattern already established for the concurrent-start guard (active_sibling/2). The per-child probe also moved from serial to Task.async_stream (bounded concurrency) so raising the timeout doesn't multiply wall-clock cost across every live worker on every LiveView refresh / dispatch check.

   A merge-queue subordinate pass (<task_id>:fixpass / :conflict) registers under a suffixed registry key while its Run row is keyed on the plain task_id; the degrade path strips only the :-suffix (never a review-gate #-id, which genuinely is its own task_id) so a wedged subordinate resolves its real workspace_id instead of degrading to nil and being dropped by finding 2's filter — the same "live worker invisible" bug, just relocated to a narrower class of worker.

2. The MCP worker_list tool silently resolves an unscoped call to a guessed workspace (the caller's bound workspace, else the installation default) when the caller names none. If a worker is genuinely running in a different workspace than the guess, the call returns count: 0 — indistinguishable from "nothing is running anywhere." The response now always echoes the workspace_id it actually scoped to, so a zero count is legible instead of alarming. Confirmed operationally: this is exactly what happened live — worker_list resolved to the default workspace while the live workers were in a different one.

Also fixed along the way: a degraded entry with no matching Run row has a nil started_at, which crashed WorkerIndexLive's Enum.sort_by(&1.started_at, {:asc, DateTime}) (no nil clause) on every /workers refresh; nils now sort last. And list_children/0's reducer/latest_run/1 now handle an {:exit, _} from a probed worker or Ash.read! without turning the whole list into a crash.

Root cause for the reported incident is (1): a resume-specific bug alone couldn't explain the first zero-result, which was against a freshly-dispatched worker that had never been resumed — the mailbox-timeout race applies equally to any live worker, dispatched or resumed. (2) is a related, independently reachable defect the acceptance criteria also called out.

Rebase note (round 3): landing this required rebasing onto #1969 (bd-aw2cyt), which added Arbiter.Worker.Phase — a subordinate pass is classified by its top-level :role, matched against a fixed atom set. A degraded entry didn't carry :role at all, so a wedged subordinate fell through to author_phase/2 and was misclassified as the task's own primary worker instead of e.g. :fixing_ci (not a visibility bug — workspace_id/task_id were already correct post-rebase — but a correctness gap in the same code path). Run.role already durably carries the value record_run_started/1 writes from meta[:role]; round-tripped it back through a fixed allowlist.

Merge note (round 4): the branch fell behind main again (#1980-#1984 landed, none touching this logic) and conflicted only on stale file:line citations in docs/review-coverage-and-guard-policy.md pointing into apps/arbiter/lib/arbiter/worker.ex. Merged origin/main, updated the citations to the post-merge line numbers; no functional code conflicts (worker.ex, tools/worker.ex etc. auto-merged cleanly).

Test plan

• apps/arbiter/test/arbiter/worker_test.exs — a live worker suspended (:sys.suspend/2) past the old 500ms budget is degraded (not dropped) with a non-nil task_id/workspace_id; a subordinate-registry-key worker (<task_id>:fixpass) degrades with the correct unsuffixed task_id, a non-nil workspace_id, and the correct :role; a worker stopped and restarted under the same task_id (mirrors worker_resume at the Worker level) is not dropped while briefly unresponsive.
• apps/arbiter/test/arbiter/mcp/tools_test.exs — the resume scenario above surfaces through Tools.worker_list/2 itself (not just list_children/0); an agnostic coordinator's unscoped worker_list call names the workspace_id it scoped to even when a worker is alive in a different workspace.
• mix precommit (compile --warnings-as-errors, format, full umbrella test suite), mix format --check-formatted, mix credo --strict — all green after rebasing onto current main.
• Re-verified after the round-4 merge: worker_test.exs + mcp/tools_test.exs + worker_shutdown_test.exs (396 tests, 0 failures); full mix precommit (compile clean, format clean, 1645+836 tests). The only failures are the pre-documented ARB_WORKSPACE=default worker-sandbox artifact in arbiter_cli (create_test.exs, release_deploy_test.exs, 2 tests) — confirmed pre-existing and unrelated to this change by re-running with ARB_* unset (66/66 pass).

References

bd-45tkhq. Closes #1931.

🤖 Generated with Claude Code