Sidecar boot restore: skip parked deployments, restore live ones bounded-parallel - #288
Conversation
…oning Covers the pieces the CL-6282 boot-restore cutover needs: a generic bounded-concurrency runner with per-item failure isolation, and a partition of a boot scan into live vs. hibernate-parked records.
…ded-parallel The boot scan used to restore every persisted deployment record, serially, regardless of whether the hub had put it to sleep on purpose. On a workbench with any real usage this meant restoring dozens of hibernated deployments before the sidecar could even connect back to the hub -- work with no payoff, since a hibernated deployment resumes correctly on the next message or routine fire that addresses it (the same wake path that already handles a cold deployment). `parkedAt` (stamped by the hibernate teardown) is a durable, local signal a boot scan can already read without any hub round-trip, so this cuts over to the CL-6282 design the code was already flagging: skip parked records entirely, and restore only the live ones, bounded-parallel (cap of 8) instead of one at a time, with the same per-record failure isolation the serial loop had. Also prunes a deployment record whose stored address derives a different slug than its own directory -- that mismatch is deterministic and permanent, so leaving the record on disk only means warning about it forever.
# Conflicts: # apps/sidecar/src/workflow-deployment-record.test.ts # apps/sidecar/src/workflow-host-wiring/index.ts
|
On serial vs. bounded-parallel restore: the earlier objection was that serial restore was deliberate — deterministic logs, no spawn storm. Bounded concurrency preserves both properties that mattered: Update — a second, worse failure mode surfaced during this merge's verification: a live measurement showed the serial restore loop can wedge completely, not just run slowly — the sidecar sat 4+ minutes at 0% CPU stuck on one record ( Conflict resolution against main (post-#292, post-#293):
Bound kept: 8 concurrent restores ( |
Summary
The boot scan (
apps/sidecar/src/workflow-host-wiring/index.ts) restored every persisted deployment record, serially, before the sidecar even connected to the hub. The target model (owner's call): nothing needs to be running before something addresses it — pre-warming is a latency optimization, never a correctness requirement — so boot should restore as little as possible and letensureAwake-style wake-on-delivery do the rest.This lands the part of that model that's provably safe today:
partitionScannedDeploymentssplits a boot scan into live vs. hibernate-parked (parkedAt) records. A parked record is skipped entirely at boot — it resumes on the next message or routine fire, the same wake path an idle-slept deployment that was never restarted already relies on.RESTORE_CONCURRENCY = 8) via a newrunWithConcurrencyhelper, instead of one at a time, with the same per-record failure isolation the old serial loop had.Verified during this work
wakeByAddress→wakeFoldedRunis a real hub-driven deploy call independent of prior sidecar state, the same call a first-ever launch uses.launchFoldedRunmint-and-deploy, identical to a manual "run now."triggerNativeWorkflowRoutineRun(apps/hub/src/native-workflow-routine-launch.ts) callssidecarRouter.routeMaildirectly with no wake wrapper and throws if the address isn't currently routable — the address has no live WS registration and no reconnect-race queue if it went idle, so the routine fire just fails. This is why native workflow deployments are NOT wired into idle-sleep in this PR and still restore eagerly at boot (the one kind that currently can't safely skip). Filed as CL-6579, including the concrete unblock found:deployAdoptedWorkflowFromSource(vendored, currently unused anywhere in this codebase) is built for exactly this "redeploy onto the same existing anchor" case, but wiring it is real first-integration work, not a same-session fix.Test plan
bun testinapps/sidecar— 192 pass, 0 failtsc --noEmitinapps/sidecareslinton changed files — cleanbunx prettier --writeon changed files — unchanged (already formatted)