Make the sub-agent surface tell the truth about a fleet - #429
Conversation
A worker inside one long tool call emits no events until the result lands, so the session store's silence clock cannot tell a wedged reactor from a ten-minute test run. Stamping the start of the outstanding call gives the surfaces above the fact that separates them.
Every lane of a fleet running shell commands flipped to stalled in lockstep while all of them were working, because silence was the only input. A lane is now stalled only when nothing outstanding explains the silence, and the clock beside each state is the clock that justifies it rather than the worker's unrelated lifetime.
At fleet scale the parent is almost always just awaiting children, so the indicator read working permanently, including while every lane was stuck. It now consumes the per-lane state rather than deriving a second one, and falls back to the parent's own clock when nothing is running.
The reactor runs parallel calls concurrently, so one scalar clock could not hold them: a fast grep finishing beside a ten-minute shell command retired the shell command's clock, and thirty seconds later that lane read as stalled while working perfectly. A result carrying an id that was never seen to start now retires nothing, and the lane reports the oldest live call — the one that explains the longest silence.
The zone was sized for the lanes plus the fold-away trailer. Adding the summary on top pushed the trailer past the reservation, so it was clipped at exactly the fan-out where it is the only thing reporting the hidden lanes.
Three ways the state could lie. The tool clock was optional on every type between the store and a surface, so a mapper that dropped it still compiled and silently reclassified a busy lane as stalled — which is how it shipped broken once, caught only by running a fleet; required makes that a compile error. The tool annotation overwrote a live call's name while keeping the previous call's clock, painting one tool beside another's elapsed time; it now fills gaps only. And an outstanding call was believed indefinitely, so a wedged build or a shell blocked on stdin read as busy forever and never reached the fleet stall count.
|
All four addressed. The first was a real bug and I reproduced it before touching anything. 1 — Parallel calls. Confirmed: three failing tests written first, all reproducing on the previous tip. The store now keys outstanding calls by call id (
2 — Required field. Added the end-to-end test: an in-tool lane is asserted to read identically through The annotation mismatch you spotted is real and fixed. It now fills gaps only: when the store has a call outstanding it owns both name and clock, because a progress ping cannot know when the new call started. My previous comment reasoned about not inventing a clock and missed keeping a mismatched one — two tests cover both directions now. 3 — Zone overflow. Correct, and it was clipping. Reservation is 4 — Resolved with a bound, not just documentation. That same bound covers the approval-suspend trap. A suspended call emits no completion, so it would otherwise stay outstanding permanently; it now degrades to a late stall instead of a lane that never stops looking busy. Called out in the constant's comment and in the docs. Live re-verification after all four, 7 agents each in a long shell call:
Not widened: argument previews, tree indent and the per-child transcript stay out of this PR. |
…ty lands Rebasing onto #429's call-id-keyed tool clock left FleetLane's structural type one field short of SubAgentSession, so the fleet report and the agents panel were reading two different lane shapes.
1 participant
Running a fleet, the sub-agent surface could not answer "is this progressing, and is anything stuck". Every lane read
stalledat once while all of them were working, and the top-level indicator saidworkingthroughout.What was actually wrong
The forwarding path was fine. Instrumenting
appendEventand running real fleets showed events flowing continuously — max inter-event gap 7.8s over ~100s runs, zero gaps past the 30s stall window.The failure reproduces only when workers run long tool calls. A second fleet, each agent running one
sleep 90, produced exactly one gap per lane of 90.0s withcurrentToolNamepinned torun_shellthe whole time. Silence was the only input to the stall test, so three lanes flipped tostalledsimultaneously while all three were working correctly. That is the operator's six-lanes-at-once capture.Two further defects sat on top of it:
stalleddescribed silence-since-last-event, so the number an operator watched climbing was unrelated to the word beside it.workingpermanently.Changes
currentToolStartedAton the sub-agent session store — the fact that separates "inside a long tool" from "silent for no reason". Only the store sets it, because only the store observes a call ending.laneState()is now the single definition of what a lane is doing (working/in_tool/stalled); every surface consumes it rather than comparing timestamps. A lane is stalled only when nothing outstanding explains the silence.fleetProgress()rolls the same lane states up.resolveTurnLabel/resolveRampPhasetake it and rank it above the parent's own stall clock;orchestratingjoins the closedACTIVITY_STATESset. A blocked gate and a stopping turn still outrank the fleet.summaryCounts) and the OpenTUI version dropped.With zero running sub-agents the roll-up is empty and both functions behave exactly as before.
Observed live
Four agents each running
sleep 150:Before this change all four read
stalled. Watching it live also caught a bug the tests did not:mapSessionAgentsdropped the new field, so the panel called a lane stalled while its own transcript row correctly showed it in a tool — fixed, and the progress overlay is deliberately not allowed to invent a tool clock, or a finished lane would read busy forever and mask real stalls.On completion the panel cleared and the indicator returned to
working.Pre-cutover comparison
The cutover (
f62211db) deletedsrc/tui/components/agents-strip.tsxand friends. What was lost, beyond the summary row restored here: a per-session scrollable child transcript (subagent-session-view.tsx), tool argument previews rather than bare tool names, and tree indentation for nested dispatches. Worth noting the old surface had no stall detection at all — its per-row duration only appeared once a worker finished. So the operator's read is right in substance but the regression is not that stall detection got worse; it is that the new surface added a stall signal that was wrong, and dropped the aggregate that would have made a fleet legible.Those three remaining gaps are not addressed here.
bun run test: 4289 pass, 0 fail.