Skip to content

Seed: redeploy a workflow orphaned by a stack restart, don't skip it - #286

Merged
TheGreatAxios merged 3 commits into
mainfrom
cl-orphaned-deployment
Aug 22, 2026
Merged

Seed: redeploy a workflow orphaned by a stack restart, don't skip it#286
TheGreatAxios merged 3 commits into
mainfrom
cl-orphaned-deployment

Conversation

@TheGreatAxios

Copy link
Copy Markdown
Contributor

Summary

A dev-stack restart permanently strands every previously-deployed
workflow: workbench seed sees the workflow_run row still reading
deployed and skips it, then fails on the one step that could have
fixed it — a 409 not routable from the sidecar-mail trigger, with
advice ("wait for the sidecar to connect") that's already false, since
a sidecar is connected. It's just not the one that row is bound to.

Mechanism (confirmed by reading apps/hub/src/index.ts's
sidecar router construction, vendor/intx/hub-sessions's
sidecar-handler.ts, and a live Postgres read): a shared-placement
workflow_run never populates its sidecar_id column — the only
thing binding an address to a live process is sidecarRouter's
in-memory addressIndex, populated when a sidecar connection proves
ownership over its websocket. That table lives in the hub process
only; a hub/sidecar restart empties it while the DB row keeps reading
deployed forever. Verified against the live fixture
tnt_4799af5615d7fa0eb995c224bc99253a / run
run_6d952fb4a6a85fa6feee7f3af25a3908: status='running',
sidecar_id NULL.

Fix: ensureDeployment (packages/hub-client/src/seed.ts) now
checks GET /api/tenants/:tenantId/workflows/runs/:runId/health (a
live read of the routing table) before skipping an already-"deployed"
row, and pushes a fresh deployment instead when it isn't routable.

Redeploy, not rebind: a shared sidecar carries no durable
per-run state of its own, so handing an old run's identity to a
newly-connected sidecar process would misrepresent what survived the
restart. The stale row is left in place (not deleted/rebound);
POST .../workflows/deployments already mints a brand-new
workflow_run unconditionally, so this needed no new hub-side
endpoint — only a live check before the CLI's own skip decision.

Builds on #283 (seed idempotency on skill-name conflicts) and #284
(package version bumps) — same family of "partial or stale state with
no repair path."

Not in this PR

  • Live restart-and-reseed verification against a real running dev
    stack — not done here since the task explicitly ruled out
    booting/binding the stack on :3000. Verified instead via the live
    Postgres read above, an authenticated-boundary curl (401, not 404)
    confirming the new health route is mounted correctly, and a red/green
    unit test that reproduces the exact trap against pre-fix code.
  • Cleanup of the orphaned row itself — it's left dangling in the DB
    after a fresh deployment replaces it; low-risk since the health check
    now excludes it from future skip decisions, but worth a follow-up if
    stale rows accumulate visibly.

CL-6577

Test plan

  • bun test packages/hub-client — 164 pass
  • tsc --noEmit in packages/hub-client — clean
  • eslint on touched files — clean
  • New test fails against the pre-fix code (verified via temporary revert), passes after
  • Live Postgres read confirms the exact mechanism against the reported fixture
  • Unauthenticated curl against the live dev stack's new health route returns 401 (route exists), not 404

A workflow_run row surviving a restart still reads "deployed" even
though the hub's in-memory sidecar routing table that binds its
address to a live process was wiped, so `workbench seed` skipped it
and then failed on the very step that could have fixed it: the
"not routable" 409 during confirmation. Reproduces that exact trap
against the pre-fix ensureDeployment.
ensureDeployment treated any "deployed"/"pending" workflow_run row as
done, but that status is a DB column, not a live signal: the hub only
routes mail to a deployment through an in-memory table binding its
agent address to a connected sidecar socket, which a hub or sidecar
restart empties while the row keeps reading "deployed". Seed then
skipped the row and failed later with "not routable" advice to wait
for a sidecar that was already connected — a condition that could
never become true because the row was bound to a sidecar process that
no longer existed.

ensureDeployment now checks GET .../runs/:runId/health (a live read of
the routing table) before skipping, and pushes a fresh deployment
instead when the existing one isn't routable. A stale row is left in
place rather than rebound to the new sidecar: a sidecar carries no
durable session state of its own, so reusing an old run's identity on
a new process would misrepresent what survived. Restarting the dev
stack no longer permanently orphans a tenant's seeded workflows.
Documents the address-routing-vs-DB-status gap ensureDeployment now
checks, and why redeploy (not rebind) is the repair.
@TheGreatAxios
TheGreatAxios force-pushed the cl-orphaned-deployment branch from fdb3940 to 6f6774c Compare August 22, 2026 02:29
@TheGreatAxios
TheGreatAxios merged commit 498c8f8 into main Aug 22, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant