Seed: redeploy a workflow orphaned by a stack restart, don't skip it - #286
Merged
Conversation
A workflow_run row surviving a restart still reads "deployed" even though the hub's in-memory sidecar routing table that binds its address to a live process was wiped, so `workbench seed` skipped it and then failed on the very step that could have fixed it: the "not routable" 409 during confirmation. Reproduces that exact trap against the pre-fix ensureDeployment.
ensureDeployment treated any "deployed"/"pending" workflow_run row as done, but that status is a DB column, not a live signal: the hub only routes mail to a deployment through an in-memory table binding its agent address to a connected sidecar socket, which a hub or sidecar restart empties while the row keeps reading "deployed". Seed then skipped the row and failed later with "not routable" advice to wait for a sidecar that was already connected — a condition that could never become true because the row was bound to a sidecar process that no longer existed. ensureDeployment now checks GET .../runs/:runId/health (a live read of the routing table) before skipping, and pushes a fresh deployment instead when the existing one isn't routable. A stale row is left in place rather than rebound to the new sidecar: a sidecar carries no durable session state of its own, so reusing an old run's identity on a new process would misrepresent what survived. Restarting the dev stack no longer permanently orphans a tenant's seeded workflows.
Documents the address-routing-vs-DB-status gap ensureDeployment now checks, and why redeploy (not rebind) is the repair.
TheGreatAxios
force-pushed
the
cl-orphaned-deployment
branch
from
August 22, 2026 02:29
fdb3940 to
6f6774c
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A dev-stack restart permanently strands every previously-deployed
workflow:
workbench seedsees theworkflow_runrow still readingdeployedand skips it, then fails on the one step that could havefixed it — a
409 not routablefrom the sidecar-mail trigger, withadvice ("wait for the sidecar to connect") that's already false, since
a sidecar is connected. It's just not the one that row is bound to.
Mechanism (confirmed by reading
apps/hub/src/index.ts'ssidecar router construction,
vendor/intx/hub-sessions'ssidecar-handler.ts, and a live Postgres read): a shared-placementworkflow_runnever populates itssidecar_idcolumn — the onlything binding an address to a live process is
sidecarRouter'sin-memory
addressIndex, populated when a sidecar connection provesownership over its websocket. That table lives in the hub process
only; a hub/sidecar restart empties it while the DB row keeps reading
deployedforever. Verified against the live fixturetnt_4799af5615d7fa0eb995c224bc99253a/ runrun_6d952fb4a6a85fa6feee7f3af25a3908:status='running',sidecar_idNULL.Fix:
ensureDeployment(packages/hub-client/src/seed.ts) nowchecks
GET /api/tenants/:tenantId/workflows/runs/:runId/health(alive read of the routing table) before skipping an already-"deployed"
row, and pushes a fresh deployment instead when it isn't routable.
Redeploy, not rebind: a shared sidecar carries no durable
per-run state of its own, so handing an old run's identity to a
newly-connected sidecar process would misrepresent what survived the
restart. The stale row is left in place (not deleted/rebound);
POST .../workflows/deploymentsalready mints a brand-newworkflow_rununconditionally, so this needed no new hub-sideendpoint — only a live check before the CLI's own skip decision.
Builds on #283 (seed idempotency on skill-name conflicts) and #284
(package version bumps) — same family of "partial or stale state with
no repair path."
Not in this PR
stack — not done here since the task explicitly ruled out
booting/binding the stack on :3000. Verified instead via the live
Postgres read above, an authenticated-boundary curl (401, not 404)
confirming the new health route is mounted correctly, and a red/green
unit test that reproduces the exact trap against pre-fix code.
after a fresh deployment replaces it; low-risk since the health check
now excludes it from future skip decisions, but worth a follow-up if
stale rows accumulate visibly.
CL-6577
Test plan
bun test packages/hub-client— 164 passtsc --noEmitinpackages/hub-client— cleaneslinton touched files — clean