Skip to content

Add read_agent_trace fleet verb (orchestrator tiers only) - #599

Merged
TheGreatAxios merged 2 commits into
mainfrom
cl-6951-worker-observability-read_agent_trace-over-traces-we-already
Aug 24, 2026
Merged

Add read_agent_trace fleet verb (orchestrator tiers only)#599
TheGreatAxios merged 2 commits into
mainfrom
cl-6951-worker-observability-read_agent_trace-over-traces-we-already

Conversation

@TheGreatAxios

Copy link
Copy Markdown
Collaborator

Summary

Implements the read_agent_trace half of CL-6951. Every sub-agent worker already writes its full turn history to turns.jsonl under its own workdir, but nothing in the runtime reads it back — so a cancelled or interrupted worker's completed work was invisible to the orchestrator even though it was sitting on disk. This gives the orchestrator (and nested orchestrator) tiers a tool to read it directly.

  • src/subagent/trace-reader.ts — locates a worker's trace directory under the caller's subagents/ tree and reads its turns.jsonl segments (reusing listSegmentFiles from incremental-jsonl.ts), flattening turns into typed entries (text / thinking / tool_call / tool_result / error).
  • src/subagent/trace-tool.ts — the read_agent_trace tool definition/handler wrapping the reader.
  • src/subagent/run.ts / src/agent/tools.ts — mount the tool the same way task/search_agents are mounted: gated through assertTierMayMountFleetVerb (leaf directors never get it), unconditionally for the primary (Tier 1) session.
  • src/subagent/authority.ts — comment update; read_agent_trace moves from "future verb" to "implemented."

Bounding strategy

Every response path is capped, with hard maximums the caller cannot exceed regardless of what it asks for:

  • Turn window: default last 40 turns, hard cap 200 (fromTurn/toTurn args).
  • Entry count: default 200, hard cap 500 (limit arg).
  • Per-entry content: capped at 4,000 chars, with truncated: true on the entry.
  • A kinds filter narrows to specific entry types.

Whenever anything is left out, the result carries a machine-readable omitted block (turnsBefore/turnsAfter/reason/hint) telling the caller exactly what was skipped and what fromTurn/toTurn to pass to page for the rest. No argument combination returns an unbounded blob.

Malformed / missing input

  • A torn or malformed JSONL line (the file can be mid-write) is skipped and counted in parseWarnings; it never throws.
  • An unknown target returns a clean AgentTraceNotFoundError message via the tool result, not an exception.

latest symlink double-count

Directory enumeration (listUniqueSubdirs) resolves every entry via realpath and de-duplicates by real path, deriving the reported name from the resolved path's own basename rather than the raw readdir entry name. A latest-style symlink pointing at a sibling directory therefore can never be counted as a second, distinct entry regardless of readdir ordering. Covered by trace-reader.test.ts's listUniqueSubdirs suite, including the exact "latest plus its target enumerates once" case.

Left out (by design)

  • progress_note for leaf workers — separate ticket, not touched here.
  • The spawn_agent/wait_agents/list_agents/send_input/interrupt_agent/close_agent/resume_agent/followup_task verbs already listed in FLEET_VERBS — unimplemented, unaffected.

Test plan

  • bun test src/subagent/trace-reader.test.ts — 17 tests: dedup/enumeration, recursive lookup, bounding (turn window, entry limit, truncation), malformed-line tolerance, kind filtering, missing target.
  • bun test src/subagent/trace-tool.test.ts — tool-level error/format behavior.
  • bun test src/subagent/run-authority.test.ts src/subagent/authority.test.ts — existing tier-gate tests still pass.
  • bun run check — lint, typecheck, build, full test suite (5399 pass, 0 fail).

@linear-code

linear-code Bot commented Aug 24, 2026

Copy link
Copy Markdown

CL-6951

Lets an orchestrator or nested orchestrator read a worker's on-disk
turns.jsonl directly, so a cancelled or interrupted worker's completed
work is no longer invisible once its in-memory session record is gone.
Every response is bounded (turn window, entry count, per-entry chars),
tolerates partially written/malformed lines, and reports a clean error
for an unknown target. Fixes a latest-symlink double-count in directory
enumeration by resolving symlinks and de-duping by real path.

progress_note for leaf workers is a separate follow-up, not included
here.
Review on #599 found two real issues:

- findAgentTraceDir walked the whole (flat, shared) subagents/ tree
  from the root workdirBase, so a Tier-2 nested orchestrator could
  read any worker's trace, not just its own descendants. Wired
  assertCanTargetAgent (authority.ts's first live call site) using the
  worker's own SubAgentSessionStore id and the store's existing
  parentSessionId chain, rather than reshaping the on-disk layout —
  the disk tree is deliberately flat across the whole fleet (shared by
  worktrees and intervention logs too), so a structural per-subtree
  root would be a much larger change. To make that id available,
  run.ts now names a worker's trace directory after its session-store
  id when one is supplied (task-tool.ts passes it), instead of always
  minting a fresh disk-only id.
- The per-entry, entry-count, and turn-window caps multiply (500 *
  4,000 = 2,000,000 chars). Added a total-output character cap that
  stops filling entries once reached and reports the remainder via the
  existing `omitted` block.

Noted but not changed: readAllTurns loads each full segment before
bounds apply. Segments are already bounded to ~256KB by the writer, so
this isn't unbounded, but avoiding the read entirely needs a line-count
index or a streaming reader — left as a follow-up rather than
expanding this fix.
@TheGreatAxios
TheGreatAxios force-pushed the cl-6951-worker-observability-read_agent_trace-over-traces-we-already branch from de35d90 to a438026 Compare August 24, 2026 05:35
@TheGreatAxios

Copy link
Copy Markdown
Collaborator Author

Addressed both issues.

1. Cross-subtree leak — fixed by wiring `assertCanTargetAgent`.
Chose the non-structural fix over reshaping the disk layout: the `subagents/` tree is deliberately flat across the whole fleet at every nesting depth (shared by worktrees and intervention logs too, not just traces), so scoping the search root per-subtree would mean restructuring that shared layout — a much bigger change, and one that collides with #600's own `run.ts` edits.

Instead: `run.ts` now names a worker's on-disk trace directory after its `SubAgentSessionStore` id when the caller supplies one (`task-tool.ts` passes `session.id`), instead of always minting a fresh disk-only uuid. That makes the store's existing `parentSessionId` chain usable as the authority boundary. `trace-tool.ts`'s handler now calls `assertCanTargetAgent({id: actorId, tier}, target, sessions.list())` before touching disk — `authority.ts`'s first live call site, as flagged. Tier 1 (no authority context passed) is unrestricted, matching its actual rule. An actor with no resolvable session id fails closed (denied), consistent with CL-6941's unresolved-tier-denies precedent.

New tests in `trace-tool.test.ts` ("descendant-only scoping" describe block): orchestratorA can read its own descendant workerA1, cannot read sibling-subtree workerY, an unresolvable actor id is denied outright, and Tier 1 can read anyone.

2. Aggregate cap — added `MAX_TRACE_TOTAL_CHARS` (20,000 chars).
The entry-filling loop now also tracks running total content length and stops (marking `entriesTruncated` + populating `omitted` with a "total output cap" reason) once adding the next entry would exceed it, independent of the entry-count/window caps. New test: 600 × ~5,000-char entries at `limit: 500` now returns ≤20,000 total chars instead of ~2MB.

`readAllTurns` full-segment read: left as-is, with a code comment explaining why. Each segment is already bounded to ~256KB by the writer (`createSegmentedJSONLWriter`'s `DEFAULT_MAX_SEGMENT_BYTES`), so this isn't unbounded the way reading one giant `turns.jsonl` would be — but avoiding the read for segments outside the requested window needs either a line-count index or a streaming reader. Please ticket that separately as suggested rather than folding it into this PR.

Rebased onto latest `origin/main` (past #597). `bun run check` (lint, typecheck, build, full suite) is green: 5434 pass, 0 fail. Both `run.ts` and `authority.ts` hunks are additive/minimal (one new field on `RunSubAgentParams`, one line changing how the workdir id is chosen, three lines in `authority.ts`'s comment) to stay out of #600's way.

@TheGreatAxios
TheGreatAxios enabled auto-merge (squash) August 24, 2026 05:35
@TheGreatAxios
TheGreatAxios merged commit 4b1d635 into main Aug 24, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant