feat(harness): turn errors retry the current turn — never drain the queue (plan #759)

AGENTS.md

@@ -295,7 +295,7 @@ invincible/
 | Logical agent cwd + workspace-root↔abs canonicalization (`change_dir` / session / default env; **`sandbox_info`** is the bind/cwd/caps/env introspector — do not `exec env`; `canonicalizePath(R, p)` / `workspaceAbsToRel(R, abs)` / `resolvePathForTool(R, cwd, p)` / `rewriteExecRootToRel(R, text)` in `lib/agent/workPath.ts`) + **`search`** (read-grant-only code-grep via `rg`; `lib/agent/tools.ts`) | `lib/agent/workPath.ts`, `lib/agent/tools.ts`, `lib/agent/runAgent.ts`, `lib/agent/agentBody.ts`, `lib/sandbox/config.ts`, `lib/sessionStore.ts`, `lib/harnessChat.ts`, `lib/agentApi.ts`, `lib/sessionCloudCaps.ts` (shared client-safe `sanitizeSessionCwd` + Redis-safe opaque id predicate), [docs/sandbox.md](docs/sandbox.md), [docs/session-model.md](docs/session-model.md), [docs/agent-stream.md](docs/agent-stream.md). Tool paths accept **in-jail absolute paths** on all FS tools + `change_dir` + `exec` cwd: an absolute under the per-binding jail root R (`resolved.value.workspaceRoot` → `RunAgentParams.workspaceRoot` → `createAgentTools`) is canonicalized to the same workspace-relative freshness key as its relative form (BYO + Vercel parity); out-of-jail absolutes and `..`/symlink escapes fail closed. Absolute paths under `R` that **appear in `exec` stdout/stderr** are likewise rewritten to workspace-relative (`rewriteExecRootToRel` in `lib/agent/workPath.ts`, applied to `result.stdout`/`result.stderr` separately) so `exec pwd` ≡ `pwd`/annotations; when `R` is unresolvable the exec output passes through byte-for-byte (fail-open), and rewrites are capped and never throw. When R is unresolvable (BYO daemon down/pre-v2 — `workspaceRoot === null`) absolute is rejected (“root unavailable — use workspace-relative”) while relative + cwd still work. Initial request/session `cwd` stays relative-only; `.` is the workspace-root default session start (there is no `SANDBOX_DEFAULT_CWD` env knob), `..` walks up toward the workspace root and errors only past it, and an **exact ancestor** of cwd re-roots cleanly (`change_dir invincible` from `cwd=invincible/docs` → `invincible`, not the phantom `invincible/docs/invincible`) while a name-prefix sibling is never re-rooted. P1/GAP-1 (#452/#330): `cwd` + `activeSandboxId` are **session-owned** and ride the Redis record (`meta.{logicalCwd,activeSandboxId}`). `activeSandboxId` is now **server-resolved** (routing override via `lib/tenancy/resolveSandbox.ts` `requestedSandboxId`), not carry-only. A **confirmed successful `change_dir`** is persisted as the session cwd even when the turn later cancels / times out / hard-errors (`lib/harnessChat.ts` host-side `liveCwd`); the success path still prefers the authoritative `agentResult.cwd`, and only a confirmed `change_dir` (never an errored one) is stored on a failed/aborted turn. The **`exec` tool** returns a **compact summary**, not a raw dump: first `EXEC_LOG_HEAD_LINES` (10) + last `EXEC_LOG_TAIL_LINES` (10) lines per stdout/stderr with line/byte counts and `... (N lines truncated)`, each shown line byte-clipped (`EXEC_SUMMARY_LINE_MAX_BYTES`=4096) so a single fat stdio line can't inline the stream or truncate the `log:` pointer off; and when either stream is non-empty writes the full redacted output to `<workspace>/.invincible/logs/exec-<ts>-<seq>.log` via `client.write_file(..., mkdir: true)` (a brand-new hidden workspace dir; backends never auto-create parents; the `-<seq>` monotonic counter keeps same-ms parallel execs from overwriting), reporting two `read_file` pointers — `log: <rel path>` (cwd-relative, from nested cwd `../.invincible/logs/…`) and `log (root): <root path>` (workspace-root-relative, read from the workspace root `cwd .`, so a depth-changing `change_dir` can never strand the full output); the write stays workspace-root, and `.invincible/` is gitignored; both pointers ride immediately after `exit=`/`TIMED_OUT` — empty output (`exec true`) writes no file, and a log-write failure fails soft with a `⚠ log write failed` note whose reason is **sanitized** (a backend/jail path never surfaces) (caps `EXEC_LOG_HEAD_LINES`/`EXEC_LOG_TAIL_LINES`=10 and `EXEC_LOG_MAX_BYTES`=8 MiB in `lib/sandbox/config.ts`) |
 | Cloud multi-device harness session (Redis multi-session, `/api/sessions*`, hybrid local+cloud; **phase 0 #515 envelope + Blob transcript carrier**) | `app/api/sessions/*` (+ `app/api/sessions/[id]/envelope/*`, `[id]/transcript/*`), `lib/sessionRepository.ts`, `lib/sessionCloudCaps.ts`, `lib/sessions/*` (+ `lib/sessions/blobStore.ts`, `blobStores.ts`), `lib/tenancy/harnessSessionsRedis.ts`, `lib/tenancy/harnessSessions.ts` (archive read + shared validator), `lib/di/index.ts` (root), `app/harness/HarnessHost.tsx`, `middleware.ts`, [docs/session-model.md](docs/session-model.md), [docs/bring-your-own.md](docs/bring-your-own.md), [SECURITY.md](SECURITY.md) — one-shot Postgres→Redis backfill: GHA **`sessions-redis-backfill`** (idempotent per-user marker); Postgres `harness_sessions` is a read-only archive. P1/GAP-1 (#452): session-carrier `meta.{logicalCwd,activeSandboxId}` folds into the PUT body and restores on pull/adopt; **plan #616 (source #610)** adds the reserved `meta.selectedModel` session carrier for the selected model pick (restore by id after the model catalog push; server **drops a poisoned value to unset**, never a 400). **Phase 0 (#515):** the transcript lives in **Vercel Blob** (`BLOB_READ_WRITE_TOKEN` / BYO S3-R2 seam) pointed to by `meta.transcriptPointer` on the small Redis envelope (`harness:envelope:*`); server mints short-lived scoped upload URLs for **client→Blob** uploads; legacy full-record GET stays for roll-forward while old blobs stay small. Envelope upsert/read: `PUT`/`GET /api/sessions/:id/envelope`; mint/read: `POST`/`GET /api/sessions/:id/transcript` |
 
-| Harness stream chrome (Thinking collapse/caps, live tools) | `lib/harnessChat.ts`, `native/harness/src/ui.zig` (facade + `frame`; transcript band owner), `native/harness/src/ui/thinking.zig` (Thinking kind), `native/harness/src/transcript_split.zig` (collapsible left rail + session list in the transcript band), `native/harness/src/session_catalog.zig` (v17 catalog + pending switch), protocol **v19** in `lib/harnessBridge.ts` (Stop cancel v9; Thinking kind v8; tool-run kind 6 v10→live paint v11; skill-attach kind 7 v12; **status-slot store v13**; **turn-clock feed v14**; **v14 addendum** `inv_set_busy_tick`; **v16** model persist; **v17** session-rail catalog + pending switch; **v18** `inv_queued_count` submit-queue depth; **v19** `inv_set_queue_promote_allowed` — host arms a one-shot per-terminal scalar so a Stop/Esc/error/timeout Ready **never drains the queue**; only idle ▶ / Ctrl+Enter with an empty composer + non-empty queue promotes, plan #760) |
+| Harness stream chrome (Thinking collapse/caps, live tools) | `lib/harnessChat.ts`, `native/harness/src/ui.zig` (facade + `frame`; transcript band owner), `native/harness/src/ui/thinking.zig` (Thinking kind), `native/harness/src/transcript_split.zig` (collapsible left rail + session list in the transcript band), `native/harness/src/session_catalog.zig` (v17 catalog + pending switch), protocol **v20** in `lib/harnessBridge.ts` (Stop cancel v9; Thinking kind v8; tool-run kind 6 v10→live paint v11; skill-attach kind 7 v12; **status-slot store v13**; **turn-clock feed v14**; **v14 addendum** `inv_set_busy_tick`; **v16** model persist; **v17** session-rail catalog + pending switch; **v18** `inv_queued_count` submit-queue depth; **v19** `inv_set_queue_promote_allowed` — host arms a one-shot per-terminal scalar so a Stop/Esc/error/timeout Ready **never drains the queue**; only idle ▶ / Ctrl+Enter with an empty composer + non-empty queue promotes, plan #760; **v20** `inv_queued_insert_front` — **turn retry that never drains the queue** (plan #759): the host retries a retryable agent-turn error up to `TURN_RETRY_ATTEMPTS`=5 (NEW cap) with bounded backoff via the additive `classify` seam (`lib/sandbox/resilience.ts`), then gives up onto `Lifecycle.Error` (a failed turn is never terminal for the Wasm promote gate — `ui.zig` promotes only on successful Ready), inserting `Continue the current turn` at the queue head (`inv_queued_insert_front`) when non-empty; permanent `PERMANENT_TURN_STATUS` statuses (400/401/403/404/413/422) give up after one attempt (no loop); 408/429/5xx and timeout/empty retry up to 5 attempts — but **1 attempt** once the live stream has painted a ring row past the user line (fail-closed: replaying would re-run tools / duplicate bubbles) |
 | Keyboard shortcuts (keymap, leader, help overlay) | `native/harness/src/keymap.zig` (single chord table + reserved-browser deny-list + leader machine; **NEW caps** `KEYMAP_MAX`=64, `LEADER_WINDOW_MS`=800), `native/harness/src/ui/keymap_dispatch.zig` (one per-frame walk of `dvui.events()`, handled-marking, leader dvui-timer arm/expiry), `native/harness/src/ui/help_overlay.zig` (in-canvas TEAL panel), wired in `native/harness/src/ui.zig` (dispatch before textEntry; overlay paint) + `ui/queue_band.zig` (scan removed; `queue_save`/`cancel_queue_edit` routed via dispatcher). **DOM adds no keyboard UI / `window` keydown / React cheatsheet** |
 | Workspace status bar (protocol v13 status-slot store; bridge overall **v14** — plan #538/#541 + Phase 2 git #540 + Phase 3 context/usage #539, **two-line bottom status bar under the composer — #554/#555/#570**) | `native/harness/src/{bridge,ui,model_picker,model_catalog}.zig` (status-slot store + two-line 64 px bar directly **below the composer**: **line 1** = identity (spinner · `h:{build-id}` · model menu) relocated from the deleted header band, **line 2** = `paintStatusSlots` right-aligned slot pack — header merged by plan #570; each line has explicit 32 px height so the model picker (`PICKER_TRIGGER_H`=32) fits and slots never clip; sandbox/cwd/git + context/usage slots — context painted generically via `STATUS_SLOT_DROP_ORDER`), `lib/harnessBridge.ts` (`StatusSlot`, `setStatusSlot`/`getStatusSlot`/`clearStatusSlot`/`clearStatusSlots`, `STATUS_SLOT_MAX_BYTES` mirror), `lib/harnessChat.ts` (`foldStatusSlots` — folds `activeSandboxId` + `cwd` + **context/usage** (`formatUsageSummary`, re-sanitized on read) after hydrate, after **every** agent turn — success **and** fail (403-clear / committed `change_dir` repaint the pack — PR #543), and **live mid-turn on tool results** (Phase 2 #627 / #625: a confirmed `change_dir` or successful `meta_sandbox_switch` repaints sandbox/cwd immediately, plus the host persists via `onSessionPatch`); context default **hidden** on missing usage, abort/cancel carries the prior honest value forward; host-ellipsized to the byte cap before the wire; `refreshGitStatusSlot` — host polls the read-only `GET /api/harness/status` probe on a ~10 s cadence **and** on-demand after a successful `exec` or `meta_sandbox_switch` mid-turn (not only the cadence), fail-soft keeps the last git value on transient error/429), `app/harness/HarnessHost.tsx` (Clear/New clears the pack; wires the git cadence + `onSessionPatch` → persist), `app/api/harness/status/route.ts` (read-only git probe: envelope-authoritative bind (`meta.activeSandboxId` wins over Redis-safe `?sandboxId=` carry), `resolveSandbox` → bounded argv-only read-only git at the bind workspace root via `lib/agent/statusProbe.ts`, per-instance rate cap `STATUS_PROBE_MIN_INTERVAL_MS`; middleware matcher + in-route `requireSessionUser` dual gate; never mutates a session/envelope — no Production write), `lib/agent/statusProbe.ts` (`STATUS_GIT_PROBE_OUT_MAX_BYTES`=512, fail-soft `{}`), `lib/sessionCloudCaps.ts` (`STATUS_SLOT_MAX_BYTES` = 96 + `STATUS_PROBE_MIN_INTERVAL_MS` = 2000 — client-safe single sources), **context/usage carrier:** `lib/agent/usageSummary.ts` (bounded provider-usage mapper `mapProviderUsage` / read-side `sanitizeUsageSummary` / host `formatUsageSummary`, `USAGE_SUMMARY_MAX_BYTES` = 96 — NEW cap), emitted **live mid-stream** from `finish` parts (aggregate only — never `finish-step` per-step counts) in `lib/agent/agentStream.ts` (SSE `usage` event), reconciled at the final `done.usage` / JSON result / chat result in `lib/agent/runAgent.ts` (+ `app/api/chat/route.ts`), parsed by `lib/agentApi.ts` / `lib/chatApi.ts`, mirrored on `SessionSnapshot.usage` (`lib/sessionStore.ts`; reserved cloud `meta.usage` JSON string, drop-to-unset on poison), docs: [docs/feature-divide.md](docs/feature-divide.md), [docs/harness-limits.md](docs/harness-limits.md), [docs/agent-stream.md](docs/agent-stream.md), [docs/session-model.md](docs/session-model.md) | |
 | Tool-run aggregation + expandable transcript control (#325) | `lib/agent/agentStream.ts` (backend `tool_result.preview` — bounded/redacted L2 detail), `lib/toolRun.ts` (encode/decode, host aggregation, `meaningfulDetail` preview→`detail`, `mergeToolRunPayloads`/`encodeToolRunPayload` hydrate coalesce), `lib/harnessChat.ts` (stream/JSON aggregation → kind 6 `tool_run`, **live-painted**: a tool event opens/grows ONE card immediately via `update_last` — grouping keys off the host's `lastRingRowIsToolRun` flag, the only ring writer: grow iff the last ring row is a tool-run, else a NEW card at `1`; a thinking/assistant/user/error row last is a separator; commit-once is removed; reload coalescing of consecutive `tool_run` rows via `coalesceToolRunMessages` in `pushSessionToBridge`), `lib/sessionStore.ts` role `tool_run`, `native/harness/src/rich/toolrun.zig` (decode), `native/harness/src/ui/toolrun.zig` (`paintToolRun` — **headerless**: no `tools` kind band; 📋 copy on the header row; status glyphs as the single channel from embedded faces, `✓`/`✗` DejaVu symbols + `…` Noto; L2 preview in Vera Sans Mono for command/output tools **or any multi-line detail**, body otherwise; short single-line results → static label, no blank expander), `native/harness/src/bridge.zig` + `lib/harnessBridge.ts` (protocol **v11**; additive test-only ring readback `inv_message_*_at`), protocol **v11**; expand state + stick-to-bottom reuse dvui `reorder_tree.zig` / `scrolling.zig` idioms |

docs/agent-stream.md

@@ -40,7 +40,7 @@ Each SSE block is one `data: <json>\n\n` line:
 | `text_delta` | `text` (chunk) | Grow Assistant bubble(s) |
 | `usage` | `usage` | **Live mid-stream** provider token summary from a `finish` part (aggregate only). `finish-step` never emits this. The host folds the context slot immediately; `done.usage` is the final reconcile. Absent when the part carried no usable counts — never a clear/flicker |
 | `done` | `text`, optional `toolTrace`, optional `cwd`, optional `usage` | Collapse open thinking; finalize session; apply `cwd` on success only; fold bounded provider `usage` (Phase 3 #628 — `done.usage` is the conclusive reconcile); Ready |
-| `error` | `error`, optional `status` | Collapse open thinking; Error message; Ready |
+| `error` | `error`, optional `status` | Collapse open thinking; a host **retryable** failure retries the same turn up to 5 attempts (**1 attempt** if any ring row has already been painted mid-stream — re-painting would duplicate tools/bubbles) before give-up; give-up paints the Error message and the turn lands on **Error** — never consuming the operator queue |
 
 Unknown types are ignored (forward-compatible). String fields are redacted server-side with the same secret list as JSON responses.
 
@@ -105,7 +105,7 @@ Every harness turn paints a final line:
 |---------|------|
 | Model finished | `Turn ended · model finished` (System) |
 | User Stop | `Turn ended · you stopped` (System) |
-| Error / timeout / empty | `Turn ended · error · …` / timed out / empty (Error) |
+| Error / timeout / empty | `Turn ended · error · …` / timed out / empty (Error). A retryable error retries the **same** turn up to **5 attempts** with bounded backoff before give-up (**1 attempt once a ring row has been painted mid-stream**); on give-up the host sets the turn lifecycle to **Error** (so a queued head is never drained) and, if the operator queue is non-empty, inserts `Continue the current turn` as the new head. Permanent failures (the `PERMANENT_TURN_STATUS` whitelist — 400/401/403/404/413/422) give up after a single attempt; **408/429/5xx and timeout/empty stay retryable** (retry the same turn up to 5 attempts) |
 | Standalone chat (`/api/chat`) | `Turn ended · chat finished` (System) — kept helper only; a failed agent turn does **not** fall back here |
 
 These markers are **not** folded as tools into the next prompt.