Preserve hibernated agent identity across undeploy/redeploy (CL-6581) - #293
Merged
Conversation
4 tasks
CL-6581: the published @intx/hub-agent package destroys an agent's reconnect-challenge keypair on every undeploy, hibernate or not, which is the documented root cause of CL-6203/CL-6044. These tests exercise a snapshot-before-delete/restore-before-redeploy technique (proved viable in PR #291) against the real, unmodified @intx/hub-agent key/repo stores, and end-to-end through createSidecarDeployRouter's actual undeploy/deploy hooks: a hibernate teardown must preserve the identity across a real redeploy, a reclaiming teardown must still destroy it, a broken snapshot must report loudly instead of failing silent, and an orphaned snapshot must be reaped past its retention window. The shared lifecycle fixture gains an optional real-key-store override (needed because its default fake never touches disk) and its default fake now creates a real agentDir so an unrelated hibernate-teardown test does not trip the vault's ordering-broke detector.
CL-6581, compensating for CL-6239 (still open -- the real fix is a non-destructive upstream undeploy). The published @intx/hub-agent package's handleAgentUndeploy unconditionally deletes an agent's reconnect-challenge keypair on every sendAgentUndeploy, hibernate or not, but only AFTER this sidecar's undeploy hook returns. New hibernated-agent-identity-vault.ts snapshots agentDir (hub-agent's only stable public export for this path) into this sidecar's own data dir, hardened to 0600/0700, before that delete runs, and restores it before the next deploy's loadOrGenerateKey call. Wired into createSidecarDeployRouter: teardownDeployment snapshots when reclaimDirs is false (the hibernate flavor); spawnWorkflowDeployment restores before minting/loading the deployment's key, and reports through reportError if a restored snapshot still yields a fresh keypair. A new reapExpiredHibernationSnapshots sweep (run once at boot) reclaims any snapshot whose address hibernated and was never redeployed, so an abandoned hibernate does not leak disk forever. A missing agentDir at snapshot time is reported the same way -- the signal that a future @intx/hub-agent release changed the delete/hook call ordering this technique depends on.
TheGreatAxios
force-pushed
the
cl-suspend-wire
branch
from
August 22, 2026 03:54
a00ea0d to
8c78891
Compare
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
CL-6581, built on the spike in PR #291 -- wires the snapshot/restore
technique into the real
workflow-host-wiringundeploy/deploy hooks,gated on the existing
reclaimDirsflag. This is a compensatingworkaround, not a substitute for CL-6239 (still open: the real fix is
a non-destructive upstream undeploy in
@intx/hub-agent).The bug this closes: the published
@intx/hub-agentpackage'shandleAgentUndeploy(ws/hub-link.js) unconditionally destroys anagent's on-disk identity directory -- including its reconnect-challenge
Ed25519 keypair under
agentDir(dataDir, address)/keys/-- on everysendAgentUndeploy, hibernate or not, and that delete happens AFTERthis sidecar's
undeployhook returns. Losing that keypair is thedocumented root cause of CL-6203/CL-6044: a woken agent mints a fresh
identity and fails the hub's reconnect challenge.
What changed
apps/sidecar/src/hibernated-agent-identity-vault.ts:snapshotAgentIdentity/restoreAgentIdentity/reapExpiredHibernationSnapshots,built on
agentDir(dataDir, address)--@intx/hub-agent's onlystable public export for this path.
workflow-host-wiring/index.ts:teardownDeploymentsnapshots theidentity directory when
reclaimDirsisfalse(the hibernateflavor), before anything else runs, since the destructive delete
happens right after this hook returns.
spawnWorkflowDeploymentrestores it before the
loadOrGenerateKeycall, and reports throughreportErrorif a restored snapshot still yields a fresh keypair(
isNew: true) -- a broken-restore signal that must never failsilent.
reapExpiredHibernationSnapshotsrouter method, run once atboot (
apps/sidecar/src/index.ts), reclaims any snapshot whoseaddress hibernated and was never redeployed within the retention
window, so an abandoned hibernate does not leak disk forever.
Safety
sidecar's own data dir (never a served/public path), hardened to
0700(dirs) /0600(files) -- matching the precedent inworkflow-deployment-record.ts'swriteWorkflowDeploymentRecord--and is never logged or included in an error message.
(
HIBERNATION_SNAPSHOT_RETENTION_MS), generous relative to anyexpected idle-sleep duration (hours to low days), swept once at boot,
independent of and without touching the unrelated deployment-record
boot-restore scan or its concurrency.
@intx/hub-agent's undeploy callordering (delete happens after our hook returns). If a future
package bump reorders that,
agentDirwould already be gone by thetime a hibernate teardown reaches the snapshot step -- that is
detected and reported through
reportErrorimmediately, rather thansilently making every subsequent wake mint a fresh identity (the
exact CL-6203 bug class).
Test plan
apps/sidecar/src/hibernated-agent-identity-vault.test.ts--snapshot/restore/reap against the real, unmodified
@intx/hub-agentkey/repo stores; permission hardening; theordering-broke report path (mocked
reportError); retention-basedreaping.
apps/sidecar/test/workflow-suspend-identity-preservation.test.ts-- end-to-end through the real
createSidecarDeployRouterwiring:a hibernate teardown + real
AgentRepoStore.remove+ redeployloads back the identical keypair; a reclaiming teardown still
destroys it.
bun run check(typecheck, lint, test,check:structural)passes clean from the repo root.
Scope
Does not touch idle-sleep policy,
packages/folded-runs, or the bootrestore loop's concurrency -- those are other lanes' work. The only
shared files touched are
apps/sidecar/test/support/workflow-lifecycle-fixture.ts(added an optional real-key-store override, and made the default fake
key store create a real
agentDirso it doesn't spuriously trip thischange's ordering-broke detector) and
apps/sidecar/src/workflow-host-wiring/index.ts(the
teardownDeployment/spawnWorkflowDeploymenthooks this PR isabout).
Linear: CL-6581. CL-6239 stays open -- commented there noting this
compensating fix and its dependency on upstream call ordering.