Skip to content

CL-6640: quarantine a missing closure staging dir instead of crash-looping - #311

Merged
TheGreatAxios merged 2 commits into
mainfrom
cl-6640-crashloop
Aug 22, 2026
Merged

CL-6640: quarantine a missing closure staging dir instead of crash-looping#311
TheGreatAxios merged 2 commits into
mainfrom
cl-6640-crashloop

Conversation

@TheGreatAxios

Copy link
Copy Markdown
Contributor

Summary

Fixes the P0 sidecar boot crash-loop from CL-6640: an ENOENT reading a
freshly-staged workflow-definition closure's package.json was falling
through the boot restore loop's default "transient" classification in
restoreDeploymentFromRecord (apps/sidecar/src/workflow-host-wiring/index.ts),
so it was retried on every boot without ever quarantining.

The escape path

applyFrozenWorkflowClosure (apps/sidecar/src/workflow-closure-apply.ts)
re-materializes a deployment's closure into a brand-new packages/<uuid>/
directory on every single restore call (a fresh crypto.randomUUID() each
time -- it never persists or trusts a stale directory reference) and reads
the package back in the same await chain. So an ENOENT there is never
this boot's timing -- it is corrupt/incomplete persisted input (the
shared tarball cache under workflow-definition-closure-cache/, or a
durable source-asset checkout resolveDeploymentAssetMounts already
validated) that reproduces byte-for-byte on every future retry. The call
chain is fully awaited inside restoreDeploymentFromRecord's try, which
the boot loop's outer try/catch does classify -- but as a plain Error,
which falls to the default "transient" bucket, so it never crosses
RESTORE_QUARANTINE_THRESHOLD and warns forever while making no progress.

Fix shape

Wrapped the applyClosure(...) call and added isMissingClosureStagingFailure,
which recognizes readPackageJSON's (@intx/workflow-host, published --
not touched) wrapped-ENOENT Error shape and rethrows it as
WorkflowRestoreFailure("permanent", ...). This is the smaller correct
fix: the closure-apply path already re-derives from source fresh on every
call (no persisted staging pointer to make "durably complete"), so nothing
needed to change there -- only the classification of a failure that recurs
identically every time.

Test plan

  • apps/sidecar/test/workflow-restore-quarantine.test.ts: new test
    fabricates the exact wrapped-ENOENT shape via the injectable
    materializeDeploymentClosure seam and proves (a) the record
    quarantines as permanent after RESTORE_QUARANTINE_THRESHOLD
    boots, (b) a sibling deployment's own restore is still attempted
    every boot, unaffected, and (c) restoreWorkflowDeployments() never
    throws across any of those boots.
  • WORKBENCH_CHECK_SINCE=origin/main bun run typecheck -- clean
  • WORKBENCH_CHECK_SINCE=origin/main bun run lint -- clean (0 errors)
  • bun test in apps/sidecar -- 209 pass, 0 fail (no regressions)

Fabricates the exact wrapped-ENOENT shape readPackageJSON throws for a
missing/incomplete closure staging directory and proves the boot
restore loop quarantines it as a permanent failure without stopping,
and that a sibling deployment's own restore is still attempted every
boot.
…e failure

Boot restore was crash-looping: an ENOENT reading a freshly-staged
closure's package.json fell through restoreDeploymentFromRecord's
default "transient" classification, so it was retried every boot
without ever quarantining. applyFrozenWorkflowClosure re-materializes
into a brand-new deploy-id directory on every call and reads back from
it in the same await chain, so a missing file there is never this
boot's timing -- it is corrupt/incomplete persisted input (the tarball
cache, or a durable source-asset checkout) that reproduces identically
on every future retry. Classify it permanent so it quarantines after
RESTORE_QUARANTINE_THRESHOLD attempts instead of warning forever.
@TheGreatAxios
TheGreatAxios merged commit 4693e08 into main Aug 22, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant