Skip to content

sidecar: defer single-step deployment restore to wake (CL-6648) - #327

Merged
TheGreatAxios merged 2 commits into
mainfrom
cl-6648-sources
Aug 22, 2026
Merged

sidecar: defer single-step deployment restore to wake (CL-6648)#327
TheGreatAxios merged 2 commits into
mainfrom
cl-6648-sources

Conversation

@TheGreatAxios

Copy link
Copy Markdown
Contributor

Summary

  • Boot-time restore replayed a deployment's frozen sources[] verbatim, including a chain whose credential died after a provider was reconfigured -- the deployment came back "live" but permanently broken, with nothing to ever refresh it.
  • A single-step ("warm-keep") deployment already has a working lazy-wake port (ensureAwake -> wakeFoldedRun -> deployAtHead) that re-resolves inference sources fresh against the tenant's live catalog on every wake. restoreDeploymentFromRecord now detects that shape from the record alone (its sources map always has exactly one entry per step -- no schema change needed) and defers restore to that path instead of replaying the frozen snapshot.
  • A true multi-step workflow deployment has no such wake port yet, so it keeps restoring eagerly from its frozen sources, unchanged.

Why this shape

The sidecar's boot-time restore runs entirely before the hub WebSocket connection opens, has no HTTP/RPC path to the hub, and the on-disk WorkflowDeploymentRecord carries no tenant id -- there is no way for restore itself to call resolveDefinitionSources (hub-side, DB-backed). Rather than build a new sidecar-to-hub source-resolution channel, this defers to the existing, already-correct wake path: the address stays unroutable until its next message or routine fire, at which point the ordinary wake redeploys it against a current catalog resolution. This is the forward-compat contract for every future update -- a room from any earlier build wakes and works, and a record whose closure or address really is corrupt still surfaces (just at its next real deploy attempt, via the wake path, instead of preemptively at boot).

Test plan

  • WORKBENCH_CHECK_SINCE=origin/main bun run typecheck -- clean
  • WORKBENCH_CHECK_SINCE=origin/main bun run lint -- 0 errors (pre-existing warnings unrelated)
  • WORKBENCH_CHECK_SINCE=origin/main bun run test -- 212/212 sidecar tests pass
  • bun run check:structural -- all checks pass
  • New tests (apps/sidecar/test/workflow-restore-defer-to-wake.test.ts): a single-step record with a dead source is never spawned or gated on buildability at restore and stays untouched on disk; a multi-step record still restores eagerly from frozen sources, unchanged.
  • Live acceptance (dana tenant, :3000) -- NOT done in this pass. Standing up an isolated scratch stack (own port + a copy of the running state) safely within this session's time budget wasn't feasible; recommend a restart-and-verify against :3000 directly: after this lands, restart the sidecar, confirm the boot log's "Deferred N single-step workflow deployment(s) to their lazy-wake path" line for dana's tenant, then message one of her previously-broken rooms and confirm a real Ollama reply.

Refs CL-6648.

Boot-time restore currently replays a deployment's frozen `sources`
snapshot verbatim, including a dead credential frozen before a
provider was reconfigured. These tests pin the fix: a single-step
("warm-keep") deployment's restore must defer to its existing wake
path (which re-resolves inference sources fresh against the live
tenant catalog) rather than ever spawn from the frozen sources, while
a true multi-step deployment keeps restoring eagerly, unchanged.
Boot-time restore reused a deployment's persisted `sources` verbatim
for every deployment, including a single-step ("warm-keep") one whose
chain died after a provider was reconfigured -- the deployment reads
as "already live" to every later check, so nothing ever refreshed it.

A single-step deployment already has a working lazy-wake port
(ensureAwake -> wakeFoldedRun -> deployAtHead) that re-resolves
inference sources fresh against the tenant's current catalog on every
wake. `restoreDeploymentFromRecord` now detects that shape from the
record alone (its `sources` map always carries exactly one entry per
step) and defers to that path instead of restoring eagerly, so a room
from any earlier build wakes and works against the current catalog on
its next message -- without a new sidecar-to-hub source-resolution
channel, which the sidecar has no pre-connect path to build. A true
multi-step deployment has no such wake port yet, so it keeps
restoring eagerly from its frozen sources, unchanged.

Updated the existing quarantine tests' fixture to a two-step
definition, since they exercise the (unchanged) eager-restore
classification path this change now reserves for multi-step
deployments.
@TheGreatAxios
TheGreatAxios merged commit 07c15cd into main Aug 22, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant