Skip to content

Sidecar boot restore: skip parked deployments, restore live ones bounded-parallel - #288

Merged
TheGreatAxios merged 4 commits into
mainfrom
cl-simple-lifecycle
Aug 22, 2026
Merged

Sidecar boot restore: skip parked deployments, restore live ones bounded-parallel#288
TheGreatAxios merged 4 commits into
mainfrom
cl-simple-lifecycle

Conversation

@TheGreatAxios

@TheGreatAxios TheGreatAxios commented Aug 22, 2026

Copy link
Copy Markdown
Contributor

Summary

The boot scan (apps/sidecar/src/workflow-host-wiring/index.ts) restored every persisted deployment record, serially, before the sidecar even connected to the hub. The target model (owner's call): nothing needs to be running before something addresses it — pre-warming is a latency optimization, never a correctness requirement — so boot should restore as little as possible and let ensureAwake-style wake-on-delivery do the rest.

This lands the part of that model that's provably safe today:

  • partitionScannedDeployments splits a boot scan into live vs. hibernate-parked (parkedAt) records. A parked record is skipped entirely at boot — it resumes on the next message or routine fire, the same wake path an idle-slept deployment that was never restarted already relies on.
  • Live records restore bounded-parallel (RESTORE_CONCURRENCY = 8) via a new runWithConcurrency helper, instead of one at a time, with the same per-record failure isolation the old serial loop had.
  • A deployment record whose stored address derives a different slug than its own directory is pruned (deleted) rather than kept around to warn about forever — that mismatch is deterministic and permanent.

Verified during this work

  • Message-addressed wake from full cold (no supervisor, no transport registration at all — not merely "parked") is solid: chat's wakeByAddresswakeFoldedRun is a real hub-driven deploy call independent of prior sidecar state, the same call a first-ever launch uses.
  • Routine fire for a single-step (folded) definition is also cold-safe: every fire is a fresh launchFoldedRun mint-and-deploy, identical to a manual "run now."
  • Routine fire for a multi-step native workflow deployment is NOT cold-safe today. triggerNativeWorkflowRoutineRun (apps/hub/src/native-workflow-routine-launch.ts) calls sidecarRouter.routeMail directly with no wake wrapper and throws if the address isn't currently routable — the address has no live WS registration and no reconnect-race queue if it went idle, so the routine fire just fails. This is why native workflow deployments are NOT wired into idle-sleep in this PR and still restore eagerly at boot (the one kind that currently can't safely skip). Filed as CL-6579, including the concrete unblock found: deployAdoptedWorkflowFromSource (vendored, currently unused anywhere in this codebase) is built for exactly this "redeploy onto the same existing anchor" case, but wiring it is real first-integration work, not a same-session fix.

Test plan

  • bun test in apps/sidecar — 192 pass, 0 fail
  • tsc --noEmit in apps/sidecar
  • eslint on changed files — clean
  • bunx prettier --write on changed files — unchanged (already formatted)

…oning

Covers the pieces the CL-6282 boot-restore cutover needs: a generic
bounded-concurrency runner with per-item failure isolation, and a
partition of a boot scan into live vs. hibernate-parked records.
…ded-parallel

The boot scan used to restore every persisted deployment record, serially,
regardless of whether the hub had put it to sleep on purpose. On a workbench
with any real usage this meant restoring dozens of hibernated deployments
before the sidecar could even connect back to the hub -- work with no payoff,
since a hibernated deployment resumes correctly on the next message or
routine fire that addresses it (the same wake path that already handles a
cold deployment).

`parkedAt` (stamped by the hibernate teardown) is a durable, local signal a
boot scan can already read without any hub round-trip, so this cuts over to
the CL-6282 design the code was already flagging: skip parked records
entirely, and restore only the live ones, bounded-parallel (cap of 8) instead
of one at a time, with the same per-record failure isolation the serial loop
had.

Also prunes a deployment record whose stored address derives a different
slug than its own directory -- that mismatch is deterministic and permanent,
so leaving the record on disk only means warning about it forever.
# Conflicts:
#	apps/sidecar/src/workflow-deployment-record.test.ts
#	apps/sidecar/src/workflow-host-wiring/index.ts
@TheGreatAxios

Copy link
Copy Markdown
Contributor Author

On serial vs. bounded-parallel restore: the earlier objection was that serial restore was deliberate — deterministic logs, no spawn storm. Bounded concurrency preserves both properties that mattered: RESTORE_CONCURRENCY caps how many Bun.spawn calls are in flight at once (8), so the storm a fully-parallel restore would cause never happens; per-record output is already captured and attributed per job (each failure/success logs its own deployment id), so ordering across records was never actually load-bearing for debugging — only within a record's own log lines, which stay ordered. What bounded parallelism buys back is the 12-minute live-measured stall: a boot with 22 records spent 12+ minutes restoring them one at a time before the hub handshake could complete, and every chat message sent in that window failed with "No sidecar available." Serial correctness was not worth that cost once we could bound the parallel case's worst-case fanout.

Update — a second, worse failure mode surfaced during this merge's verification: a live measurement showed the serial restore loop can wedge completely, not just run slowly — the sidecar sat 4+ minutes at 0% CPU stuck on one record (run_cd5f3923…) with no log progress, and the handshake never completed until the stack was killed. Bounded parallelism alone doesn't fix a hang like that — one wedged record still blocks its worker forever, and with a small bound it can eventually pin every worker. To close that gap, this merge adds RESTORE_ATTEMPT_TIMEOUT_MS (30s): a restore attempt that exceeds the deadline is treated as an ordinary transient failure — logged, counted on the record's restoreFailure counter (from #292's quarantine tracking), and skipped. The boot moves on; the record gets another attempt next boot, or quarantines after 3 consecutive permanent failures. That turns "boot hangs forever" into "one record skipped, product up."

Conflict resolution against main (post-#292, post-#293):

  • workflow-deployment-record.ts import list: kept both partitionScannedDeployments (this PR) and recordWorkflowDeploymentRestoreFailure (Quarantine permanently unrestorable workflow deployment records #292).
  • restoreDeploymentFromRecord's address-derivation-mismatch check: took Quarantine permanently unrestorable workflow deployment records #292's throw new WorkflowRestoreFailure("permanent", ...) over this PR's original delete-and-return, so the mismatch flows through Quarantine permanently unrestorable workflow deployment records #292's failure-accounting/quarantine path instead of bypassing it silently.
  • restoreWorkflowDeployments: merged all three behaviors — partitionScannedDeployments splits parked (skipped entirely, left asleep for a later wake) from live; live records restore via runWithConcurrency bounded at RESTORE_CONCURRENCY = 8; each worker does its own quarantine skip-check, restore attempt (now timeout-wrapped), success-clears-failure, and failure-records/quarantines-and-logs bookkeeping from Quarantine permanently unrestorable workflow deployment records #292, inline per record rather than via runWithConcurrency's aggregate failure list, so ordering/isolation semantics match the old serial loop.
  • #293's hibernated-identity vault snapshot/restore in the same file's teardown/spawn paths was untouched by these conflicts and carried through as-is.
  • boot-restore-push-hold.ts's pack-push hold (held until the reconnect challenge passes) is unaffected by parallelism: it keys off onDeploymentRegistered calls made during begin()/end(), which still bracket the whole restoreWorkflowDeployments() call regardless of how many workers run inside it, so parallel-restored children still can't push before the challenge clears.

Bound kept: 8 concurrent restores (RESTORE_CONCURRENCY), plus a 30s per-record timeout (RESTORE_ATTEMPT_TIMEOUT_MS).

@TheGreatAxios
TheGreatAxios merged commit cab7a8f into main Aug 22, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant