Skip to content

Preserve hibernated agent identity across undeploy/redeploy (CL-6581) - #293

Merged
TheGreatAxios merged 2 commits into
mainfrom
cl-suspend-wire
Aug 22, 2026
Merged

Preserve hibernated agent identity across undeploy/redeploy (CL-6581)#293
TheGreatAxios merged 2 commits into
mainfrom
cl-suspend-wire

Conversation

@TheGreatAxios

Copy link
Copy Markdown
Contributor

Summary

CL-6581, built on the spike in PR #291 -- wires the snapshot/restore
technique into the real workflow-host-wiring undeploy/deploy hooks,
gated on the existing reclaimDirs flag. This is a compensating
workaround
, not a substitute for CL-6239 (still open: the real fix is
a non-destructive upstream undeploy in @intx/hub-agent).

The bug this closes: the published @intx/hub-agent package's
handleAgentUndeploy (ws/hub-link.js) unconditionally destroys an
agent's on-disk identity directory -- including its reconnect-challenge
Ed25519 keypair under agentDir(dataDir, address)/keys/ -- on every
sendAgentUndeploy, hibernate or not, and that delete happens AFTER
this sidecar's undeploy hook returns. Losing that keypair is the
documented root cause of CL-6203/CL-6044: a woken agent mints a fresh
identity and fails the hub's reconnect challenge.

What changed

  • New apps/sidecar/src/hibernated-agent-identity-vault.ts:
    snapshotAgentIdentity / restoreAgentIdentity / reapExpiredHibernationSnapshots,
    built on agentDir(dataDir, address) -- @intx/hub-agent's only
    stable public export for this path.
  • workflow-host-wiring/index.ts: teardownDeployment snapshots the
    identity directory when reclaimDirs is false (the hibernate
    flavor), before anything else runs, since the destructive delete
    happens right after this hook returns. spawnWorkflowDeployment
    restores it before the loadOrGenerateKey call, and reports through
    reportError if a restored snapshot still yields a fresh keypair
    (isNew: true) -- a broken-restore signal that must never fail
    silent.
  • A new reapExpiredHibernationSnapshots router method, run once at
    boot (apps/sidecar/src/index.ts), reclaims any snapshot whose
    address hibernated and was never redeployed within the retention
    window, so an abandoned hibernate does not leak disk forever.

Safety

  • The snapshot holds private key material. It lives under this
    sidecar's own data dir (never a served/public path), hardened to
    0700 (dirs) / 0600 (files) -- matching the precedent in
    workflow-deployment-record.ts's writeWorkflowDeploymentRecord --
    and is never logged or included in an error message.
  • Retention: 30 days
    (HIBERNATION_SNAPSHOT_RETENTION_MS), generous relative to any
    expected idle-sleep duration (hours to low days), swept once at boot,
    independent of and without touching the unrelated deployment-record
    boot-restore scan or its concurrency.
  • This technique depends on @intx/hub-agent's undeploy call
    ordering
    (delete happens after our hook returns). If a future
    package bump reorders that, agentDir would already be gone by the
    time a hibernate teardown reaches the snapshot step -- that is
    detected and reported through reportError immediately, rather than
    silently making every subsequent wake mint a fresh identity (the
    exact CL-6203 bug class).

Test plan

  • apps/sidecar/src/hibernated-agent-identity-vault.test.ts --
    snapshot/restore/reap against the real, unmodified
    @intx/hub-agent key/repo stores; permission hardening; the
    ordering-broke report path (mocked reportError); retention-based
    reaping.
  • apps/sidecar/test/workflow-suspend-identity-preservation.test.ts
    -- end-to-end through the real createSidecarDeployRouter wiring:
    a hibernate teardown + real AgentRepoStore.remove + redeploy
    loads back the identical keypair; a reclaiming teardown still
    destroys it.
  • bun run check (typecheck, lint, test, check:structural)
    passes clean from the repo root.

Scope

Does not touch idle-sleep policy, packages/folded-runs, or the boot
restore loop's concurrency -- those are other lanes' work. The only
shared files touched are apps/sidecar/test/support/workflow-lifecycle-fixture.ts
(added an optional real-key-store override, and made the default fake
key store create a real agentDir so it doesn't spuriously trip this
change's ordering-broke detector) and apps/sidecar/src/workflow-host-wiring/index.ts
(the teardownDeployment/spawnWorkflowDeployment hooks this PR is
about).

Linear: CL-6581. CL-6239 stays open -- commented there noting this
compensating fix and its dependency on upstream call ordering.

CL-6581: the published @intx/hub-agent package destroys an agent's
reconnect-challenge keypair on every undeploy, hibernate or not, which is
the documented root cause of CL-6203/CL-6044. These tests exercise a
snapshot-before-delete/restore-before-redeploy technique (proved viable in
PR #291) against the real, unmodified @intx/hub-agent key/repo stores, and
end-to-end through createSidecarDeployRouter's actual undeploy/deploy
hooks: a hibernate teardown must preserve the identity across a real
redeploy, a reclaiming teardown must still destroy it, a broken snapshot
must report loudly instead of failing silent, and an orphaned snapshot
must be reaped past its retention window.

The shared lifecycle fixture gains an optional real-key-store override
(needed because its default fake never touches disk) and its default fake
now creates a real agentDir so an unrelated hibernate-teardown test does
not trip the vault's ordering-broke detector.
CL-6581, compensating for CL-6239 (still open -- the real fix is a
non-destructive upstream undeploy). The published @intx/hub-agent
package's handleAgentUndeploy unconditionally deletes an agent's
reconnect-challenge keypair on every sendAgentUndeploy, hibernate or not,
but only AFTER this sidecar's undeploy hook returns. New
hibernated-agent-identity-vault.ts snapshots agentDir (hub-agent's only
stable public export for this path) into this sidecar's own data dir,
hardened to 0600/0700, before that delete runs, and restores it before
the next deploy's loadOrGenerateKey call.

Wired into createSidecarDeployRouter: teardownDeployment snapshots when
reclaimDirs is false (the hibernate flavor); spawnWorkflowDeployment
restores before minting/loading the deployment's key, and reports through
reportError if a restored snapshot still yields a fresh keypair. A new
reapExpiredHibernationSnapshots sweep (run once at boot) reclaims any
snapshot whose address hibernated and was never redeployed, so an
abandoned hibernate does not leak disk forever. A missing agentDir at
snapshot time is reported the same way -- the signal that a future
@intx/hub-agent release changed the delete/hook call ordering this
technique depends on.
@TheGreatAxios
TheGreatAxios merged commit a74f25f into main Aug 22, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant