Skip to content

Commit f19d041

Browse files
Make the sub-agent surface tell the truth about a fleet (#429)
* Record when a sub-agent's outstanding tool call started A worker inside one long tool call emits no events until the result lands, so the session store's silence clock cannot tell a wedged reactor from a ten-minute test run. Stamping the start of the outstanding call gives the surfaces above the fact that separates them. * Stop reading a long tool call as a stalled sub-agent Every lane of a fleet running shell commands flipped to stalled in lockstep while all of them were working, because silence was the only input. A lane is now stalled only when nothing outstanding explains the silence, and the clock beside each state is the clock that justifies it rather than the worker's unrelated lifetime. * Report the fleet in the activity indicator, not the idle parent At fleet scale the parent is almost always just awaiting children, so the indicator read working permanently, including while every lane was stuck. It now consumes the per-lane state rather than deriving a second one, and falls back to the parent's own clock when nothing is running. * Document lane states and the fleet roll-up * Track a sub-agent's outstanding tool calls by call id The reactor runs parallel calls concurrently, so one scalar clock could not hold them: a fast grep finishing beside a ten-minute shell command retired the shell command's clock, and thirty seconds later that lane read as stalled while working perfectly. A result carrying an id that was never seen to start now retires nothing, and the lane reports the oldest live call — the one that explains the longest silence. * Reserve a row for the fleet summary in the agents zone The zone was sized for the lanes plus the fold-away trailer. Adding the summary on top pushed the trailer past the reservation, so it was clipped at exactly the fan-out where it is the only thing reporting the hidden lanes. * Stop a lane's state being lost or overstated in transit Three ways the state could lie. The tool clock was optional on every type between the store and a surface, so a mapper that dropped it still compiled and silently reclassified a busy lane as stalled — which is how it shipped broken once, caught only by running a fleet; required makes that a compile error. The tool annotation overwrote a live call's name while keeping the previous call's clock, painting one tool beside another's elapsed time; it now fills gaps only. And an outstanding call was believed indefinitely, so a wedged build or a shell blocked on stdin read as busy forever and never reached the fleet stall count. * Document the tool clock's ownership and the bound on in-tool
1 parent f6309be commit f19d041

19 files changed

Lines changed: 828 additions & 68 deletions

docs/TUI.md

Lines changed: 72 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -57,8 +57,8 @@ activity word — never the raw tool, MCP server, or plugin identifier that is
5757
actually executing. `resolveTurnLabel` (`src/tui-opentui/session-chrome.ts`)
5858
maps execution onto the closed set `ACTIVITY_STATES` exported from that
5959
module (`thinking`, `planning`, `researching`, `building`, `working`,
60-
`waiting`, `stalled`, `stopping`); that export is the source of truth for
61-
what the slot can say, not this list. It is led by a single density cell
60+
`waiting`, `orchestrating`, `stalled`, `stopping`); that export is the source
61+
of truth for what the slot can say, not this list. It is led by a single density cell
6262
(`rampPulse`, `src/tui-opentui/ramp.ts`). The cell, not the word,
6363
is what says whether the session is healthy, and it carries four states:
6464

@@ -78,6 +78,18 @@ printed identically, so the only way to tell them apart was to wait.
7878
waiting on something outside itself — and are told apart by motion: `blocked`
7979
holds perfectly still, which is the signal that the session is waiting on *you*.
8080

81+
While sub-agents are running, the slot reports the *fleet*, not the parent.
82+
`resolveTurnLabel` and `resolveRampPhase` take a `FleetProgress` roll-up and
83+
rank it above the parent's own stall clock: with live lanes the parent is
84+
idle by design, so its silence says nothing about whether the session is
85+
progressing, and reporting it was how a session with every lane wedged still
86+
read as `working`. A fleet with no stalled lane reads `orchestrating`; one
87+
stalled lane makes the whole indicator read `stalled`, which is the state that
88+
should pull an operator's eye to the panel. A blocked gate and a stopping turn
89+
still outrank the fleet. With zero running sub-agents the roll-up is empty and
90+
every path through both functions behaves exactly as it does for a plain
91+
single-agent turn.
92+
8193
The stall phase is driven by the watchdog's own silence clock
8294
(`stallLevel`, `src/tui-opentui/stall-watchdog.ts`), so the indicator and the
8395
abort can never disagree about which runs are stuck. It arms at
@@ -158,13 +170,67 @@ fail a test as well as the type checker.
158170
## The live agents panel
159171

160172
The `agents` chrome zone renders a standing panel above the transcript, one
161-
row per currently-running sub-agent — not a count. Each row reads
173+
row per currently-running sub-agent. Each row reads
162174
`agentId: description · elapsed · tool`, sourced from the same
163175
`agentProgress()` clock/tool/stall computation used to trail a task row in the
164176
transcript (`src/tui-opentui/agent-progress.ts`); the panel does not compute
165-
progress a second way. A worker silent past the stall window (`DEFAULT_STALL_MS`)
166-
gets a `· stalled` suffix so it reads distinct from one still working, without
167-
relying on color alone.
177+
progress a second way. Past one running agent the panel is led by a fleet
178+
summary row (`N agents`, plus `· N stalled` or `· in tools`), counted from the
179+
same lane states the rows below render, so header and rows can never disagree.
180+
The zone reserves `AGENTS_PANEL_MAX_VISIBLE + 2` rows to hold that summary, the
181+
lanes, and the `+N more` trailer together — clipping the last of the three
182+
would drop the fold-away count at exactly the fan-out where it is the only
183+
thing reporting the hidden lanes.
184+
185+
`laneState()` is the single definition of what a lane is doing, and every
186+
surface consumes it rather than comparing timestamps itself. It returns one of
187+
three states:
188+
189+
| Lane state | Means | Row reads |
190+
|---|---|---|
191+
| `working` | activity within `DEFAULT_STALL_MS` | `· 2:34 · grep` |
192+
| `in_tool` | silent, but a tool call is outstanding and under `IN_TOOL_STALL_MS` | `· 2:34 · run_shell 1:30` |
193+
| `stalled` | silent with nothing outstanding to explain it | `· 2:34 · quiet 0:45 · stalled` |
194+
195+
`in_tool` is what makes the surface honest. A worker inside one long tool call
196+
emits no events for the entire execution, so silence alone cannot separate a
197+
wedged reactor from a ten-minute test run — and it did not: a fleet whose lanes
198+
were all running shell commands flipped to `stalled` in lockstep while every
199+
one of them was working. `currentToolStartedAt` on the sub-agent session store
200+
(`src/subagent/session-store.ts`) is the fact that separates them; only the
201+
store sets it, because only the store observes a call ending.
202+
203+
The store keys outstanding calls by call id (`outstandingTools`) and reports
204+
the oldest live one — the call that explains the longest silence. It cannot
205+
collapse to a single scalar: the reactor runs parallel calls concurrently, so a
206+
fast grep finishing beside a ten-minute shell command would retire the shell
207+
command's clock and reproduce the original defect on one lane. A result whose
208+
call id was never seen to start retires nothing.
209+
210+
`currentToolStartedAt` is a **required** field on every type between the store
211+
and a surface. There are four hand-written mapping hops on the live path, and
212+
a hop that drops it silently reclassifies a busy lane as stalled — which is how
213+
this shipped broken once, caught only by running a real fleet. Required makes
214+
that a compile error rather than a misclassification; `chrome-state.test.ts`
215+
also asserts the panel and the transcript row agree on a live example.
216+
217+
`in_tool` is bounded, not terminal. A call outstanding longer than
218+
`IN_TOOL_STALL_MS` (10 minutes) escalates to `stalled` regardless, so a wedged
219+
build, a shell blocked on stdin, or a deadlocked child eventually surfaces
220+
instead of reading as busy forever. **Within that window those failures are
221+
genuinely invisible to the stall signal** — the honest trade for not crying
222+
stall over every real test suite. The per-row tool clock climbing is the signal
223+
a human can read in the meantime, which is why the row shows it. The same bound
224+
backstops calls that never report a result at all: the reactor's
225+
approval-suspend path emits no completion, so a before-tool extension returning
226+
suspend would otherwise leave a call outstanding permanently. Nothing registers
227+
such an extension today.
228+
229+
The number beside a lane's state always explains that state. A healthy lane
230+
shows its lifetime; a lane stuck in one tool also shows how long that tool has
231+
run; a stalled lane also shows how long it has been silent. Reading a lifetime
232+
clock next to the word `stalled` was the original defect — the number the
233+
operator watched climbing was unrelated to the word beside it.
168234

169235
The panel is bounded to `AGENTS_PANEL_MAX_VISIBLE` rows
170236
(`src/tui-opentui/geometry/zones.ts`); a larger fan-out degrades to a trailing

src/subagent/session-store.test.ts

Lines changed: 117 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -104,3 +104,120 @@ describe("session-store snapshot caching", () => {
104104
expect(snapshot?.entries).toEqual([]);
105105
});
106106
});
107+
108+
109+
describe("outstanding tool clock", () => {
110+
// A worker inside one long tool call emits nothing until the result lands.
111+
// Without a start clock for that call, silence is indistinguishable from a
112+
// wedged reactor, and a whole fleet running shell commands reads as stalled.
113+
test("tool.start stamps the clock and tool.done clears it", () => {
114+
let clock = 1_000;
115+
const store = createSubAgentSessionStore({ now: () => clock });
116+
const session = store.start({ description: "d", agentId: "a", brief: "b" });
117+
expect(store.get(session.id)?.currentToolStartedAt).toBeNull();
118+
119+
clock = 5_000;
120+
store.appendEvent(session.id, {
121+
type: "tool.start",
122+
seq: 1,
123+
data: { call: { id: "call-1", name: "run_shell", arguments: {} } },
124+
} as unknown as ReactorEmittedEvent);
125+
expect(store.get(session.id)?.currentToolName).toBe("run_shell");
126+
expect(store.get(session.id)?.currentToolStartedAt).toBe(5_000);
127+
128+
clock = 95_000;
129+
store.appendEvent(session.id, {
130+
type: "tool.done",
131+
seq: 2,
132+
data: { result: { callId: "call-1", content: "ok", isError: false } },
133+
} as unknown as ReactorEmittedEvent);
134+
expect(store.get(session.id)?.currentToolName).toBeNull();
135+
expect(store.get(session.id)?.currentToolStartedAt).toBeNull();
136+
});
137+
138+
test("a terminal transition never leaves a tool clock outstanding", () => {
139+
const store = createSubAgentSessionStore();
140+
const session = store.start({ description: "d", agentId: "a", brief: "b" });
141+
store.appendEvent(session.id, startCall(1, "call-1", "grep"));
142+
expect(store.get(session.id)?.currentToolStartedAt).not.toBeNull();
143+
144+
store.complete(session.id, "report");
145+
expect(store.get(session.id)?.currentToolStartedAt).toBeNull();
146+
});
147+
});
148+
149+
150+
describe("parallel tool calls", () => {
151+
const toolStart = (callId: string, name: string) =>
152+
({
153+
type: "tool.start",
154+
seq: 1,
155+
data: { call: { id: callId, name, arguments: {} } },
156+
}) as unknown as ReactorEmittedEvent;
157+
const toolDone = (callId: string) =>
158+
({
159+
type: "tool.done",
160+
seq: 2,
161+
data: { result: { callId, content: "ok", isError: false } },
162+
}) as unknown as ReactorEmittedEvent;
163+
164+
// The reactor runs parallel calls concurrently. A fast sibling finishing must
165+
// not retire the clock of a long call still executing, or the lane reads as
166+
// silent-for-no-reason thirty seconds later while it is working perfectly.
167+
test("a fast sibling completing leaves a long call's clock outstanding", () => {
168+
let clock = 1_000;
169+
const store = createSubAgentSessionStore({ now: () => clock });
170+
const session = store.start({ description: "d", agentId: "a", brief: "b" });
171+
172+
store.appendEvent(session.id, toolStart("slow", "run_shell"));
173+
clock = 2_000;
174+
store.appendEvent(session.id, toolStart("fast", "grep"));
175+
clock = 3_000;
176+
store.appendEvent(session.id, toolDone("fast"));
177+
178+
const stored = store.get(session.id);
179+
expect(stored?.currentToolName).toBe("run_shell");
180+
expect(stored?.currentToolStartedAt).toBe(1_000);
181+
});
182+
183+
test("a completion bearing an unknown call id retires nothing", () => {
184+
let clock = 1_000;
185+
const store = createSubAgentSessionStore({ now: () => clock });
186+
const session = store.start({ description: "d", agentId: "a", brief: "b" });
187+
188+
store.appendEvent(session.id, toolStart("slow", "run_shell"));
189+
clock = 4_000;
190+
store.appendEvent(session.id, toolDone("never-started"));
191+
192+
expect(store.get(session.id)?.currentToolStartedAt).toBe(1_000);
193+
});
194+
195+
// The oldest live call is the one that explains the longest silence, so it is
196+
// the one the lane reports.
197+
test("the reported call is the oldest still outstanding", () => {
198+
let clock = 1_000;
199+
const store = createSubAgentSessionStore({ now: () => clock });
200+
const session = store.start({ description: "d", agentId: "a", brief: "b" });
201+
202+
store.appendEvent(session.id, toolStart("first", "run_shell"));
203+
clock = 2_000;
204+
store.appendEvent(session.id, toolStart("second", "grep"));
205+
expect(store.get(session.id)?.currentToolName).toBe("run_shell");
206+
207+
clock = 3_000;
208+
store.appendEvent(session.id, toolDone("first"));
209+
const stored = store.get(session.id);
210+
expect(stored?.currentToolName).toBe("grep");
211+
expect(stored?.currentToolStartedAt).toBe(2_000);
212+
});
213+
214+
test("the last completion retires the clock entirely", () => {
215+
const store = createSubAgentSessionStore();
216+
const session = store.start({ description: "d", agentId: "a", brief: "b" });
217+
store.appendEvent(session.id, toolStart("only", "run_shell"));
218+
store.appendEvent(session.id, toolDone("only"));
219+
220+
expect(store.get(session.id)?.currentToolName).toBeNull();
221+
expect(store.get(session.id)?.currentToolStartedAt).toBeNull();
222+
});
223+
});

0 commit comments

Comments
 (0)