Remove the in-loop no-progress watchdogs that were failing healthy runs

.changeset/remove-false-positive-stall-watchdogs.md

@@ -0,0 +1,26 @@
+---
+"@agent-native/core": patch
+---
+
+Remove the in-loop no-progress watchdogs, which were failing healthy runs far more often than they caught wedged ones.
+
+Two 90s bounds ran for the whole model stream — one on silence between engine frames (`MODEL_STREAM_NO_PROGRESS_TIMEOUT_MS`), one on a tool input whose byte count stopped growing (`ACTION_PREPARATION_NO_PROGRESS_TIMEOUT_MS`) — plus a zero-byte tool-input restart tripwire. Each inferred a dead stream from the absence of a particular event, and that inference cannot be made on the Anthropic transport: the SDK drops the provider's `ping` keepalives before any consumer sees them (`core/streaming.js`: `if (sse.event === 'ping') continue;`, with no opt-out), so a model composing a large tool argument is indistinguishable from a wedged socket.
+
+That is normal operation, not an edge case. Only a tool declared for eager input streaming emits anything at all while its arguments are generated, so a long file write or a long structured result is a content-silent window whose length is set by the size of the argument. In one production deployment, 2 of 27 one-shot analyst runs completed; the guards added for reliability were the thing taking it away.
+
+- `ACTION_PREPARATION_NO_PROGRESS_TIMEOUT_MS` and its deadline are gone, including the `earliestStartedAt` fallback that anchored the bound to a start time it never advanced past, and the `Math.min` that let it override a demonstrably live stream.
+- `MODEL_STREAM_NO_PROGRESS_TIMEOUT_MS` and its deadline are gone.
+- The zero-byte restart tripwire is gone (`ACTION_PREPARATION_ZERO_BYTE_RESTART_LIMIT`, `noteZeroByteToolInputStart`, `resetZeroByteToolInputRestart`).
+- The two run-lifecycle invariants asserting an ordering between those bounds and the run-manager backstop are gone with them.
+
+One in-loop bound survives: the pre-first-frame cap on the clamped hosted foreground runtime, where the ~57s platform wall arrives before the engine's own 120s abort could. The first real frame releases it, so long first tokens, long thinking, long tool inputs and long outputs are all past it by construction; off that runtime there is no in-loop deadline at all.
+
+Real failures keep the bounds that key off evidence rather than absence: the engine's `FIRST_STREAM_EVENT_TIMEOUT_MS` for a stream that opens and never speaks, the run-manager backstop outside the stream, the per-tool execution timeout, the chunk/run budget, and the stale reaper. The trade is explicit: an in-stream wedge after the first frame is now caught by the run budget rather than at 90s, because no clock in the loop could tell it apart from a model writing a large tool call.
+
+Separately, `runAgentLoop` now takes the caller's real chunk budget instead of re-deriving one. It asked `resolveRunSoftTimeoutMs` for the generic background ceiling (13 min) even when the caller was a background automation, whose budget is its own hard abort minus headroom (10 min − 20s). The per-tool ceiling came out above the run budget, so every per-tool timeout on that path was dead code and the chunk boundary won instead — the exact inversion `RUN_TOOL_TIMEOUT_HEADROOM_MS` exists to prevent, reintroduced by guessing at a number the caller already had.
+
+Also records liveness forensics when a stale reaper flips a run to `errored`. `stale_run` is the largest terminal outcome on the one-shot automation path and the row said nothing about why — `error_detail` is a fixed sentence for every reap, so a correct reap and a false one were indistinguishable afterwards. The reap now records which of the three stale windows applied, whether the row was redispatchable, time since heartbeat and since progress, whether the in-flight grace was in play, and — the discriminator — how far the heartbeat ran AHEAD of the last real progress. A worker that died takes its heartbeat with it and scores ~0 there; a worker still alive while the agent loop stopped producing scores in the thousands of seconds. Those are opposite bugs that look identical in `agent_runs` today. Diagnostics only: nothing reads it to make a decision, it cannot change whether a row is reaped, and it shares the single `diag_stage` write with the existing recovery outcome rather than overwriting it.
+
+Also gives the direct-provider engines the total-request deadline they never had, and the resumed rounds the budget they actually have. `createFirstEventAbortController` is now two-stage: the first real frame releases the 120s first-event bound and arms a 14-minute `STREAM_TOTAL_TIMEOUT_MS` on the whole call, mirroring what builder-engine already applies to its own gateway requests. That matters for the runtimes with no outer budget — local dev and self-hosted resolve the soft timeout to `0`, so the deleted watchdog was the only thing standing between them and a socket that wedges after the first frame. It is a total-request bound, not a no-progress bound, so it cannot fire on healthy content-silent generation. The AI SDK path now also reports a deadline abort as an error rather than letting it fall through as a clean `end_turn`, which is a truncated turn reported as a complete one. Alongside it, `runAgentLoopWithResume` hands each round its own `roundTimeoutMs` rather than the whole invocation's, and the main chat handler passes the chunk budget it already resolved into the loop instead of leaving it to re-derive a generic ceiling — the same inversion as the automation case, two call sites over.
+
+A cancelled request is also no longer classifiable as a timeout. `fireTimeout` recorded its message before checking whether the composed controller had already been aborted, and the parent-abort path left the deadline armed — so a timer firing while the provider settled after a user Stop or a run-budget abort set `didTimeout()`, which is exactly what the engines read to decide a failure was the transport's fault and retryable. The ordering was pre-existing, but harmless while the first frame cleared the timer outright; a deadline that now runs for the whole stream made the window the whole stream. A timeout is recorded only when this controller wins the abort race, parent cancellation clears the deadline, and a frame that lands after a Stop cannot re-arm one.

packages/core/src/agent/engine/ai-sdk-engine.ts

@@ -32,10 +32,7 @@ import {
   classifyProviderError,
   describeErrorWithCauses,
 } from "./error-detail.js";
-import {
-  createFirstEventAbortController,
-  FIRST_STREAM_EVENT_TIMEOUT_MS,
-} from "./first-event-timeout.js";
+import { createFirstEventAbortController } from "./first-event-timeout.js";
 import {
   clampThinkingBudgetTokens,
   resolveMaxOutputTokensForEngine,
@@ -533,11 +530,14 @@ class AISDKEngine implements AgentEngine {
       }
 
       // AI SDK surfaces an aborted stream as a graceful `{type: "abort"}`
-      // part rather than a thrown error, so a first-event timeout would
-      // otherwise fall through to the normal end_turn completion below.
-      if (!sawFirstEvent && firstEventAbort.didTimeout()) {
+      // part rather than a thrown error, so a deadline abort would otherwise
+      // fall through to the normal end_turn completion below. Not gated on
+      // `sawFirstEvent`: the total deadline fires mid-stream by definition, and
+      // reporting that half-delivered turn as a clean end_turn is exactly the
+      // truncated-run-reported-as-complete failure.
+      if (firstEventAbort.didTimeout()) {
         throw new Error(
-          `Model request produced no stream events within ${FIRST_STREAM_EVENT_TIMEOUT_MS / 1000}s; the connection appears wedged.`,
+          `${firstEventAbort.timeoutMessage()}; the connection appears wedged.`,
         );
       }
 

packages/core/src/agent/engine/anthropic-engine.ts

@@ -25,10 +25,7 @@ import {
   LLM_MISSING_CREDENTIALS_MESSAGE,
 } from "./credential-errors.js";
 import { describeErrorWithCauses } from "./error-detail.js";
-import {
-  createFirstEventAbortController,
-  FIRST_STREAM_EVENT_TIMEOUT_MS,
-} from "./first-event-timeout.js";
+import { createFirstEventAbortController } from "./first-event-timeout.js";
 import {
   clampThinkingBudgetTokens,
   resolveMaxOutputTokensForEngine,
@@ -314,12 +311,14 @@ class AnthropicEngine implements AgentEngine {
           : typeof err?.statusCode === "number"
             ? err.statusCode
             : undefined;
-      // A first-event abort surfaces from the SDK as a generic
-      // APIUserAbortError ("Request was aborted.") — replace it with a
-      // message that actually explains what happened.
+      // A deadline abort surfaces from the SDK as a generic APIUserAbortError
+      // ("Request was aborted.") — replace it with a message that actually
+      // explains what happened. Which deadline fired comes from the
+      // controller: a total-deadline abort is a socket that wedged MID-stream,
+      // not a connection that never spoke.
       const rawMessage: string = err?.message ?? String(err);
       const errorMessage = timedOut
-        ? `Model request produced no stream events within ${FIRST_STREAM_EVENT_TIMEOUT_MS / 1000}s; the connection appears wedged.`
+        ? `${firstEventAbort.timeoutMessage()}; the connection appears wedged.`
         : describeErrorWithCauses(err);
       // Anthropic SDK APIConnectionError defaults to "Connection error." with
       // no HTTP status. Tag it so in-run retries and run-level resume treat

packages/core/src/agent/engine/first-event-timeout.spec.ts

@@ -0,0 +1,135 @@
+import { describe, it, expect, vi, beforeEach, afterEach } from "vitest";
+
+import {
+  createFirstEventAbortController,
+  FIRST_STREAM_EVENT_TIMEOUT_MS,
+  STREAM_TOTAL_TIMEOUT_MS,
+} from "./first-event-timeout.js";
+
+describe("createFirstEventAbortController", () => {
+  beforeEach(() => {
+    vi.useFakeTimers();
+  });
+  afterEach(() => {
+    vi.useRealTimers();
+  });
+
+  it("aborts a stream that never produces a first event", () => {
+    const parent = new AbortController();
+    const abort = createFirstEventAbortController(parent.signal);
+
+    vi.advanceTimersByTime(FIRST_STREAM_EVENT_TIMEOUT_MS - 1);
+    expect(abort.signal.aborted).toBe(false);
+
+    vi.advanceTimersByTime(1);
+    expect(abort.signal.aborted).toBe(true);
+    expect(abort.didTimeout()).toBe(true);
+    expect(abort.timeoutMessage()).toContain("no stream events");
+    abort.cleanup();
+  });
+
+  // The whole point of removing the in-loop no-progress watchdogs: a model
+  // composing a large tool input emits nothing a consumer can see, and that is
+  // healthy. The first event must buy the stream the full total budget.
+  it("does not bound silence between events once the stream has spoken", () => {
+    const parent = new AbortController();
+    const abort = createFirstEventAbortController(parent.signal);
+
+    vi.advanceTimersByTime(1_000);
+    abort.markFirstEvent();
+
+    vi.advanceTimersByTime(STREAM_TOTAL_TIMEOUT_MS - 1_001);
+    expect(abort.signal.aborted).toBe(false);
+    expect(abort.didTimeout()).toBe(false);
+    abort.cleanup();
+  });
+
+  // The bound the runtimes with no outer budget (local dev, self-hosted, where
+  // the soft timeout resolves to 0) depend on: a socket that wedges AFTER the
+  // first frame must not hang the run forever.
+  it("aborts a stream that outlives the total deadline, measured from the request start", () => {
+    const parent = new AbortController();
+    const abort = createFirstEventAbortController(parent.signal);
+
+    vi.advanceTimersByTime(30_000);
+    abort.markFirstEvent();
+
+    vi.advanceTimersByTime(STREAM_TOTAL_TIMEOUT_MS - 30_000);
+    expect(abort.signal.aborted).toBe(true);
+    expect(abort.didTimeout()).toBe(true);
+    expect(abort.timeoutMessage()).toContain("total stream deadline");
+    abort.cleanup();
+  });
+
+  it("reports no timeout when the parent aborts", () => {
+    const parent = new AbortController();
+    const abort = createFirstEventAbortController(parent.signal);
+
+    parent.abort("user");
+    expect(abort.signal.aborted).toBe(true);
+    expect(abort.didTimeout()).toBe(false);
+    expect(abort.timeoutMessage()).toBeUndefined();
+    abort.cleanup();
+  });
+
+  // A cancelled request is not a failed one. The engines read `didTimeout()` to
+  // decide a failure was the transport's fault and retryable, so a deadline
+  // left armed across a Stop would turn a user cancellation or a run-budget
+  // abort into a resumable provider error. The provider does not necessarily
+  // settle the moment the signal fires, and `cleanup()` only runs once it does.
+  it("does not classify a cancelled request as a timeout while the provider settles", () => {
+    const parent = new AbortController();
+    const abort = createFirstEventAbortController(parent.signal);
+
+    vi.advanceTimersByTime(5_000);
+    abort.markFirstEvent();
+    vi.advanceTimersByTime(5_000);
+    parent.abort("user");
+
+    // The provider takes its time unwinding, so `cleanup()` has not run yet.
+    vi.advanceTimersByTime(STREAM_TOTAL_TIMEOUT_MS * 2);
+
+    expect(abort.didTimeout()).toBe(false);
+    expect(abort.timeoutMessage()).toBeUndefined();
+    abort.cleanup();
+  });
+
+  it("does not classify a pre-first-event cancellation as a timeout", () => {
+    const parent = new AbortController();
+    const abort = createFirstEventAbortController(parent.signal);
+
+    parent.abort("run_timeout");
+    vi.advanceTimersByTime(FIRST_STREAM_EVENT_TIMEOUT_MS * 2);
+
+    expect(abort.didTimeout()).toBe(false);
+    expect(abort.timeoutMessage()).toBeUndefined();
+    abort.cleanup();
+  });
+
+  // A frame already in flight when the Stop lands must not re-arm a deadline
+  // on a request that is over.
+  it("does not re-arm a deadline for a frame that lands after cancellation", () => {
+    const parent = new AbortController();
+    const abort = createFirstEventAbortController(parent.signal);
+
+    parent.abort("user");
+    abort.markFirstEvent();
+    vi.advanceTimersByTime(STREAM_TOTAL_TIMEOUT_MS * 2);
+
+    expect(abort.didTimeout()).toBe(false);
+    expect(abort.timeoutMessage()).toBeUndefined();
+    abort.cleanup();
+  });
+
+  it("stops both deadlines on cleanup", () => {
+    const parent = new AbortController();
+    const abort = createFirstEventAbortController(parent.signal);
+
+    abort.markFirstEvent();
+    abort.cleanup();
+
+    vi.advanceTimersByTime(STREAM_TOTAL_TIMEOUT_MS * 2);
+    expect(abort.signal.aborted).toBe(false);
+    expect(abort.didTimeout()).toBe(false);
+  });
+});