CL-6365: a deterministic red for the restored-run-comes-back-terminal gap

[TheGreatAxios]

What this is

CL-6365's reproduction and diagnosis, on a real stack. The relaunch itself is not implemented here — see "What is not done" below.

The correction that mattered

Proof 4 killed the sidecar three seconds after asking the agent to "count slowly from one to twenty". On an instance whose model answers that in about a second, the turn had already finished and the run had parked — so the restart being proved was a clean restart, not a crash. Proof 4 passes that way at this branch's tip, which is why the gap can look closed.

Proof 4 now sends a workload no model finishes inside the window and asserts, at the moment of the kill, that nothing has answered yet. With that, the failure is deterministic.

What the mid-turn kill leaves behind

• Both top-level runs (the folded chat run and the section deployment) end up workflow_run.status = 'failed' in the hub's own database. The SIGTERM drain tears the workflow-process child down mid-step and the terminal event is committed to the durable log before exit.
• Boot restore does its half correctly: records found, pins replayed, closure re-materialized, liveness: "ok" ~10s after restart.
• The run inside is over. The section answers 409 workflow_run_terminal; the folded chat run's mail is dequeued and dropped by the supervisor, which reads readWorkflowRunLifecycle off the durable log, sees terminal, and rejects permanently.
• onBodyFailure: "continue" cannot help: the failure is on the top-level run, not on a body occurrence.

Why "wake it again" is not the fix by itself

wakeByAddress returns early for any routable address (CL-6267 handed park-respawn to the sidecar), so a routable-but-dead run is never woken. Restoring the deleted CL-6147 undeploy-then-redeploy branch is necessary but not sufficient:

1. A run's id is derived from its address (deriveWorkflowRunId(deploymentMailAddress)), so redeploying the same address re-creates the same run id.
2. The destructive teardown (teardownDeployment with reclaimDirs) removes the deployment record and the per-step scratch but not the workflow-run repo holding the run's durable event log. The terminal event survives the redeploy and the next message is rejected identically.

So a relaunch must produce a genuinely new run: a fresh run id adopting the room's continuity (the room is data now, nothing is lost), or a destructive teardown that also reclaims the run's durable log. The detection signal is already available to the hub as workflow_run.status.

What is not done

The hub-side detection and the relaunch. This PR lands the red half and the diagnosis; the fix is the next commit on this branch.

Evidence

Scratch database, real signup, real local Ollama, nothing mocked.

Before the harness correction (kill landing after the turn) — all four proofs passed:

TIMING — proof 3: first reply: 12.2s
TIMING — proof 4: sidecar restart + boot restore: 9.2s
TIMING — proof 4: section occurrence after the restart: 1.0s
TRANSCRIPT — >>> Are you still there? One sentence.
TRANSCRIPT — <<< Yes
All four proofs passed.

After, with the kill provably mid-inference:

TIMING — proof 4: sidecar restart + boot restore: 10.2s
error: walking-skeleton hop "PROOF 4 — the section survives the restart and
runs its next occurrence" failed: proof 4 (section): trigger the section:
expected HTTP 202, got 409: {"error":{"code":"workflow_run_terminal",
"message":"Workflow run run_ec8f29f80e15df0a2038e886f557c201 is terminal
and cannot receive more mail"}}

Hub database at that moment: running|2 failed|2 deployed|5.

Checks

• bun run typecheck: clean.
• prettier --check clean on both touched files. (bun run lint reports pre-existing formatting warnings in seven packages/onboarding / packages/cli test files that this branch does not touch.)

Environment note

The proof's model is named by E2E_OLLAMA_MODEL, but ensureSeeded still pins the seeded agent definition at CATALOG_SEEDS.ollama's curated model (qwen3.8:27b) regardless — env trap 1 in the docs is only half closed. An instance without that model pulled dies at proof 3 with "I can't reach a model right now". Workaround used here: ollama cp <a served model> qwen3.8:27b, then E2E_OLLAMA_MODEL=qwen3.8:27b.

[TheGreatAxios]

Landed via the stack tip: PR #108 (merge fbf0852) contains this branch's commits in full.