diff --git a/.pylon/features.yaml b/.pylon/features.yaml index 13c6904b7a..515d4a78c4 100644 --- a/.pylon/features.yaml +++ b/.pylon/features.yaml @@ -368,6 +368,21 @@ decisions: - Prime upstream exposes an equivalent public-root token, exact caller environment reuse, generation-scoped post-attach proof, and structured bounded cleanup outcomes. - Pylon can remove the fork token without weakening native multi-instance isolation or its pre-create ACP fallback. + recoverable-owned-session-adoption: + area: runtime-reliability + state: shipped + owner: shared + decision: retain + pylon_refs: + - https://github.com/pylon-code/prime-agent/issues/27 + upstream_refs: + - https://github.com/PrimeIntellect-ai/prime-agent/tree/c718bf3c30fd8da206ed551837cbb54f7ad15948 + fork_change: recoverable-owned-session-adoption-v1 + upstream_support: Prime upstream does not expose capability-gated same-supervisor adoption of an existing caller-owned worker with private rotating authority, prepare/commit proof, authoritative snapshot fencing, ordered event reconciliation, and in-place MCP ownership transfer. + revisit_when: + - Prime upstream exposes an equivalent frozen public SDK token, detached-daemon capability, non-enumerating failure contract, same-generation rotating authority, staged proof, and explicit durability confirmation. + - Pylon can remove the fork token without weakening caller-owned worker continuity, event ordering, credential privacy, or owner isolation. + deterministic-pylon-release-artifacts: area: distribution state: candidate diff --git a/packages/coding-agent/.changes/27-recoverable-owned-session-adoption.md b/packages/coding-agent/.changes/27-recoverable-owned-session-adoption.md new file mode 100644 index 0000000000..a66bb702c5 --- /dev/null +++ b/packages/coding-agent/.changes/27-recoverable-owned-session-adoption.md @@ -0,0 +1 @@ +- Added recoverable adoption for caller-owned daemon sessions. ([#27](https://github.com/pylon-code/prime-agent/issues/27)) diff --git a/packages/coding-agent/docs/daemon.md b/packages/coding-agent/docs/daemon.md index 7f1d5b47e9..f31639bbef 100644 --- a/packages/coding-agent/docs/daemon.md +++ b/packages/coding-agent/docs/daemon.md @@ -54,6 +54,14 @@ Headless and ephemeral clients use the same worker runtime as interactive client The full launch environment remains in supervisor memory and is not written to the worker descriptor. Direct SDK calls to print and RPC modes remain in-process so embedders can pass non-serializable extension factories. +### Recoverable ownership adoption + +A full-access client-owned worker can opt into `daemon_recoverable_owned_session_adoption_v1`. Recovery remains limited to the same detached supervisor process and generation. The supervisor keeps worker authority and the HMAC secret in memory, while a separate `OwnedSessionRecoveryStore` persists only opaque keyed verifiers with atomic `0600` writes in a `0700` directory. Recovery handles never enter worker descriptors or the generic command journal. + +Availability requires schema revision 30; the frozen SDK adoption and caller-environment tokens; hello offers for adoption, caller-environment cleanup, and authoritative cleanup; negotiated `event_sequence`, `correlated_prompt_lifecycle_v1`, `client_owned_sessions`, and `caller_owned_session_environment_cleanup_v1`; and exact attached adoption and cleanup proofs. The attached worker replay status must be `complete`. Partial or unavailable replay fails before ownership changes. + +Prepare freezes ownership mutation, permanently retires the original create-request replay authority, obtains an authoritative worker snapshot, and buffers bounded sequenced frames. Commit validates the snapshot proof and caller cursor, transfers both the worker owner and ACP MCP owner in place, attaches the claimant, then publishes only contiguous post-snapshot frames. Exact repeated cursor/frame bytes after reconnect are deduplicated; a conflicting duplicate fails closed, and traffic for other sessions on a shared client is discarded from the staging lane. A disconnect before commit rolls ownership back, but never revives create replay. Confirmation removes the prior-handle retry window only after the caller persists the rotated handle and exact proof. Retention is bounded for no-lifecycle, active, terminal, prepared, and unconfirmed-final states. + ## Session Ownership and Leases Every persisted session is protected by a process-safe lease keyed by canonical JSONL path. diff --git a/packages/coding-agent/docs/sdk.md b/packages/coding-agent/docs/sdk.md index 93422cc2f3..dcbf977199 100644 --- a/packages/coding-agent/docs/sdk.md +++ b/packages/coding-agent/docs/sdk.md @@ -93,6 +93,18 @@ The contract proof contains only the feature/status, protocol identity, schema r Use `disposeOwnedSession({ timeoutMs })` when cleanup must be observable. Concurrent calls join one operation, and `timeoutMs` is one strict total deadline for reconnection, authoritative queries, completion, side-question aborts, and unsupported-peer finalization. Its fixed statuses are `completed`, `already_completed`, `replacement_settled`, `owner_mismatch`, `uncertain`, `transport_failure`, and `unsupported`; `uncertain` also reports whether the last authoritative state was `active` or `stopping`. `replacement_settled` means an authenticated supervisor with a different generation answered the read-only cleanup query with `settled` for the connection's previously proved opaque route. Cleanup never sends completion on a replacement or pending route without a current internal attach proof. Each strict cleanup request is transport-bound and is never replayed after reconnect. Public attachment proof remains absent throughout disposal. Results never return raw errors or environment identity. `dispose()` remains the legacy best-effort `Promise` API. +### Recoverable caller-owned daemon sessions + +Native detached-daemon hosts can opt into same-supervisor recovery with `recoverable_owned_session_adoption_v1`. The complete pre-create gate requires schema revision 30, the frozen package-root SDK tokens `recoverable_owned_session_adoption_v1` and `caller_owned_session_environment_cleanup_v1`, and connected hello offers for `daemon_recoverable_owned_session_adoption_v1`, `caller_owned_session_environment_cleanup_v1`, and `authoritative_owned_session_cleanup_v1`. If any proof is absent, select ACP before creating a native worker. This is a POSIX detached-daemon contract; it does not provide resident promotion, cross-host recovery, or recovery after supervisor process replacement. + +Prepare must then echo the exact requested attachment capabilities `event_sequence`, `correlated_prompt_lifecycle_v1`, `client_owned_sessions`, and `caller_owned_session_environment_cleanup_v1`. The attached snapshot must prove complete replay, the exact adoption proof, and the current caller-owned cleanup contract through `getOwnedSessionContractProof()`. Treat a missing capability, partial or unavailable replay, mismatched adoption proof, or missing cleanup proof as unavailable before ownership commit. A hello offer alone is not attachment proof. + +Call `createRecoverableOwnedSession()` with a fresh request ID that encodes at least 128 bits, one caller-captured launch environment, the recovery config, correlation ID, and MCP owner ID. Durably store the returned 256-bit `recoveryHandle`, `supervisorGeneration`, active session ID, session ID, authoritative event cursor, correlation ID, MCP owner ID, recovery config, and launch environment. Treat the handle as a bearer secret. The daemon persists only opaque keyed verifiers and keeps the authority needed to recover the existing worker private to the current supervisor process. + +After the exact owner transport disconnects, call `adoptRecoverableOwnedSession()` on a new daemon client with that complete authority tuple, the last durable cursor, a new MCP owner ID, and a stable request ID. The SDK performs prepare and commit internally. It does not create or attach to a replacement worker as fallback. It installs an authoritative snapshot before it releases strictly ordered post-snapshot events, and the daemon retags existing MCP servers in place. A live old owner, wrong or stale authority, supervisor replacement, conflicting retry, proof mismatch, race, expiry, or unsupported peer all fail with `Recoverable owned session adoption is unavailable`. + +The result contains the staged connection, the exact post-adoption proof, and a rotated handle. Durably replace the old authority tuple with that exact result before calling `confirmRecoverableOwnedSessionAdoption()`. Use the same request ID for a retry whose response may have been lost. Prepare, commit, and confirmation retries converge on the same rotated receipt. Confirmation closes the old-handle retry window; it does not replace durable caller storage. + `DaemonClient.request()` keeps legacy reconnect replay by default. Pass `{ recoverAcrossReconnect: false }` only when one request must fail on transport close instead of crossing to a new daemon transport. `DaemonClient.close()` is terminal owner disposal. `isClosed` becomes true, later `connect()` calls reject, and a live `DaemonAgentConnection` emits one terminal close. Normal and update recovery stop before any later restart, connect, attach, or restored-session query; already-running recovery callbacks are not cancellable but their results are discarded. diff --git a/packages/coding-agent/src/cli-main.ts b/packages/coding-agent/src/cli-main.ts index c598dc7b7d..7218c4f783 100644 --- a/packages/coding-agent/src/cli-main.ts +++ b/packages/coding-agent/src/cli-main.ts @@ -7,8 +7,9 @@ import { maybeRunOwnedSessionWorkerFrontend, } from "./cli/owned-session-worker.js"; import { APP_NAME } from "./config.js"; +import type { DaemonWorkerBootstrapEnvironment } from "./modes/daemon/daemon-worker-protocol.js"; -export async function runCli(): Promise { +export async function runCli(daemonWorkerBootstrap?: DaemonWorkerBootstrapEnvironment): Promise { try { enableCompileCache?.(); } catch { @@ -38,7 +39,7 @@ export async function runCli(): Promise { setGlobalDispatcher(new EnvHttpProxyAgent({ bodyTimeout: 0, headersTimeout: 0 })); try { - await main(process.argv.slice(2)); + await main(process.argv.slice(2), { daemonWorkerBootstrap }); } finally { closeOwnedSessionWorkerOwnerWatch(); } diff --git a/packages/coding-agent/src/cli.ts b/packages/coding-agent/src/cli.ts index 07d1b9b3d0..ee1c23e629 100644 --- a/packages/coding-agent/src/cli.ts +++ b/packages/coding-agent/src/cli.ts @@ -10,6 +10,20 @@ const supported = assertNodeVersion({ }); if (supported) { + const workerProtocol = await import("./modes/daemon/daemon-worker-protocol.js"); + const authenticatedWorkerLaunch = + workerProtocol.isDaemonWorkerProcess() && + process.env[workerProtocol.DAEMON_WORKER_TOKEN_ENV] !== undefined && + process.env[workerProtocol.DAEMON_WORKER_STARTUP_GATE_FD_ENV] !== undefined; + let daemonWorkerBootstrap: ReturnType | undefined; + try { + if (authenticatedWorkerLaunch) { + workerProtocol.waitForDaemonWorkerStartupGate(); + daemonWorkerBootstrap = workerProtocol.readDaemonWorkerBootstrapEnvironment(); + } + } finally { + workerProtocol.sanitizeDaemonWorkerBootstrapEnvironment(process.env); + } const { runCli } = await import("./cli-main.js"); - await runCli(); + await runCli(daemonWorkerBootstrap); } diff --git a/packages/coding-agent/src/cli/daemon-launch.ts b/packages/coding-agent/src/cli/daemon-launch.ts index 819f0503a4..79062557c5 100644 --- a/packages/coding-agent/src/cli/daemon-launch.ts +++ b/packages/coding-agent/src/cli/daemon-launch.ts @@ -16,13 +16,7 @@ import { DAEMON_PROTOCOL_VERSION, DAEMON_SCHEMA_ID } from "../modes/daemon/daemo import { getDaemonRuntimeIdentity } from "../modes/daemon/daemon-runtime-identity.js"; import { isSessionSummaryBusy, type SessionSummary } from "../modes/daemon/daemon-session-list.js"; import { defaultDaemonSocketPath, normalizeSocketPath } from "../modes/daemon/daemon-socket.js"; -import { - DAEMON_WORKER_ACTIVE_SESSION_ID_ENV, - DAEMON_WORKER_RECOVERY_JOURNAL_ENV, - DAEMON_WORKER_ROLE_ENV, - DAEMON_WORKER_SUPERVISOR_SOCKET_ENV, - DAEMON_WORKER_TOKEN_ENV, -} from "../modes/daemon/daemon-worker-protocol.js"; +import { sanitizeDaemonWorkerBootstrapEnvironment } from "../modes/daemon/daemon-worker-protocol.js"; import { isHelpCommandRequest, PUBLIC_COMMAND_NAMES, REMOVED_COMMAND_NAMES } from "./command-registry.js"; import { createCliSubprocessEnv, formatCurrentCliCommand } from "./subprocess-launch.js"; @@ -357,12 +351,7 @@ async function ensureDaemonRunning(socketPath: string, spawnCwd?: string): Promi // a CLI running inside a daemon worker (e.g. a test spawned by the Prime // Agent daemon) would launch the supervisor in worker mode, which listens // on the socket but never sends the daemon_hello handshake. - const env = createCliSubprocessEnv(); - delete env[DAEMON_WORKER_ROLE_ENV]; - delete env[DAEMON_WORKER_TOKEN_ENV]; - delete env[DAEMON_WORKER_ACTIVE_SESSION_ID_ENV]; - delete env[DAEMON_WORKER_RECOVERY_JOURNAL_ENV]; - delete env[DAEMON_WORKER_SUPERVISOR_SOCKET_ENV]; + const env = sanitizeDaemonWorkerBootstrapEnvironment(createCliSubprocessEnv()); delete env[ORPHAN_PROCESS_JOURNAL_ENV]; delete env[SESSION_LEASES_ENABLED_ENV]; delete env[SESSION_LEASE_OWNER_ID_ENV]; diff --git a/packages/coding-agent/src/cli/daemon-update-restart.ts b/packages/coding-agent/src/cli/daemon-update-restart.ts index 6afcae9fdb..b2edb908f0 100644 --- a/packages/coding-agent/src/cli/daemon-update-restart.ts +++ b/packages/coding-agent/src/cli/daemon-update-restart.ts @@ -9,10 +9,7 @@ import { getProcessStartId, SESSION_LEASE_OWNER_ID_ENV, SESSION_LEASES_ENABLED_E import { defaultDaemonSocketDir, defaultDaemonSocketPath, normalizeSocketPath } from "../modes/daemon/daemon-socket.js"; import { DAEMON_WORKER_ACTIVE_SESSION_ID_ENV, - DAEMON_WORKER_RECOVERY_JOURNAL_ENV, - DAEMON_WORKER_ROLE_ENV, - DAEMON_WORKER_SUPERVISOR_SOCKET_ENV, - DAEMON_WORKER_TOKEN_ENV, + sanitizeDaemonWorkerBootstrapEnvironment, } from "../modes/daemon/daemon-worker-protocol.js"; import { createCliSubprocessLaunchSpec } from "./subprocess-launch.js"; @@ -513,14 +510,9 @@ function createStatusPath(agentDir: string, socketPath: string, requestId: strin } function coordinatorEnvironment(agentDir: string): NodeJS.ProcessEnv { - const environment = { ...process.env }; + const environment = sanitizeDaemonWorkerBootstrapEnvironment({ ...process.env }); environment[ENV_AGENT_DIR] = agentDir; delete environment[SELF_UPDATE_INTERACTIVE_CHILD_ENV]; - delete environment[DAEMON_WORKER_ROLE_ENV]; - delete environment[DAEMON_WORKER_TOKEN_ENV]; - delete environment[DAEMON_WORKER_ACTIVE_SESSION_ID_ENV]; - delete environment[DAEMON_WORKER_RECOVERY_JOURNAL_ENV]; - delete environment[DAEMON_WORKER_SUPERVISOR_SOCKET_ENV]; delete environment[ORPHAN_PROCESS_JOURNAL_ENV]; delete environment[SESSION_LEASES_ENABLED_ENV]; delete environment[SESSION_LEASE_OWNER_ID_ENV]; diff --git a/packages/coding-agent/src/core/agent-session.ts b/packages/coding-agent/src/core/agent-session.ts index 11981f22cf..ce02ec4af6 100644 --- a/packages/coding-agent/src/core/agent-session.ts +++ b/packages/coding-agent/src/core/agent-session.ts @@ -165,7 +165,7 @@ import { import type { HostRequestHandlers, KernelSentAgentMessage } from "./kernel/index.js"; import { type RestoreResult, snapshotPathIn } from "./kernel/state-snapshot.js"; import type { AcpMcpServerConfig } from "./mcp/acp-mcp-types.js"; -import type { McpManager } from "./mcp/mcp-manager.js"; +import type { AcpMcpOwnerTransferProof, McpManager } from "./mcp/mcp-manager.js"; import { type BashExecutionMessage, type CompactionOutcome, @@ -1060,6 +1060,14 @@ function attributeChildUsage(parentUsage: Usage, childUsage: Usage): void { parentUsage.totalTokens = parentContextTokens; } +const MISSING_MCP_OWNER_TRANSFER_RETENTION_MS = 5 * 60_000; +const MISSING_MCP_OWNER_TRANSFER_RECEIPT_LIMIT = 256; + +interface MissingMcpOwnerTransferReceipt { + proof: AcpMcpOwnerTransferProof; + expiresAt: number; +} + export class AgentSession { readonly agent: Agent; readonly sessionManager: SessionManager; @@ -1172,6 +1180,7 @@ export class AgentSession { private _agentMessageController?: AgentSessionMessageController; private _agentObserveController?: AgentObserveController; private _mcpManager?: McpManager; + private readonly _missingMcpOwnerTransfers = new Map(); private _baseToolsOverride?: Record; private _sessionStartEvent: SessionStartEvent; private _extensionUIContext?: ExtensionUIContext; @@ -1389,6 +1398,94 @@ export class AgentSession { this.agent.state.systemPrompt = this._baseSystemPrompt; } + transferAcpMcpServersOwner(previousOwnerId: string, nextOwnerId: string): void { + this._mcpManager?.transferAcpServersOwner(previousOwnerId, nextOwnerId); + } + + transferAcpMcpServersOwnerTransaction( + transactionId: string, + previousOwnerId: string, + nextOwnerId: string, + ): AcpMcpOwnerTransferProof { + if (!this._mcpManager) { + const existing = this.getMissingMcpOwnerTransfer(transactionId); + if (existing) { + if (existing.previousOwnerId !== previousOwnerId || existing.nextOwnerId !== nextOwnerId) { + throw new Error("ACP MCP owner transfer transaction tuple changed"); + } + return { ...existing }; + } + const proof = { transactionId, previousOwnerId, nextOwnerId, changed: false, state: "transferred" as const }; + this.rememberMissingMcpOwnerTransfer(proof); + return { ...proof }; + } + return this._mcpManager.transferAcpServersOwnerTransaction(transactionId, previousOwnerId, nextOwnerId); + } + + queryAcpMcpServersOwnerTransaction( + transactionId: string, + previousOwnerId: string, + nextOwnerId: string, + ): AcpMcpOwnerTransferProof { + if (!this._mcpManager) { + const proof = this.getMissingMcpOwnerTransfer(transactionId); + if (!proof || proof.previousOwnerId !== previousOwnerId || proof.nextOwnerId !== nextOwnerId) { + throw new Error("ACP MCP owner transfer transaction is unknown"); + } + return { ...proof }; + } + return this._mcpManager.queryAcpServersOwnerTransaction(transactionId, previousOwnerId, nextOwnerId); + } + + rollbackAcpMcpServersOwnerTransaction( + transactionId: string, + previousOwnerId: string, + nextOwnerId: string, + ): AcpMcpOwnerTransferProof { + if (!this._mcpManager) { + const proof = this.queryAcpMcpServersOwnerTransaction(transactionId, previousOwnerId, nextOwnerId); + const rolledBack = { ...proof, state: "rolled_back" as const }; + this.rememberMissingMcpOwnerTransfer(rolledBack); + return { ...rolledBack }; + } + return this._mcpManager.rollbackAcpServersOwnerTransaction(transactionId, previousOwnerId, nextOwnerId); + } + + retireAcpMcpServersOwnerTransaction(transactionId: string, previousOwnerId: string, nextOwnerId: string): void { + if (this._mcpManager) { + this._mcpManager.retireAcpServersOwnerTransaction(transactionId, previousOwnerId, nextOwnerId); + return; + } + const proof = this.getMissingMcpOwnerTransfer(transactionId); + if (proof && (proof.previousOwnerId !== previousOwnerId || proof.nextOwnerId !== nextOwnerId)) { + throw new Error("ACP MCP owner transfer transaction tuple changed"); + } + this._missingMcpOwnerTransfers.delete(transactionId); + } + + private getMissingMcpOwnerTransfer(transactionId: string): AcpMcpOwnerTransferProof | undefined { + const receipt = this._missingMcpOwnerTransfers.get(transactionId); + if (!receipt) return undefined; + if (receipt.expiresAt <= Date.now()) { + this._missingMcpOwnerTransfers.delete(transactionId); + return undefined; + } + return receipt.proof; + } + + private rememberMissingMcpOwnerTransfer(proof: AcpMcpOwnerTransferProof): void { + this._missingMcpOwnerTransfers.delete(proof.transactionId); + this._missingMcpOwnerTransfers.set(proof.transactionId, { + proof, + expiresAt: Date.now() + MISSING_MCP_OWNER_TRANSFER_RETENTION_MS, + }); + while (this._missingMcpOwnerTransfers.size > MISSING_MCP_OWNER_TRANSFER_RECEIPT_LIMIT) { + const oldest = this._missingMcpOwnerTransfers.keys().next().value; + if (oldest === undefined) break; + this._missingMcpOwnerTransfers.delete(oldest); + } + } + replaceAcpMcpServers(servers: readonly AcpMcpServerConfig[], ownerId: string): void { if (this.isStreaming) throw new Error("Cannot replace ACP MCP servers while the agent is running"); if (!this._mcpManager) { diff --git a/packages/coding-agent/src/core/mcp/mcp-manager.ts b/packages/coding-agent/src/core/mcp/mcp-manager.ts index 6031651078..6b7ef8088c 100644 --- a/packages/coding-agent/src/core/mcp/mcp-manager.ts +++ b/packages/coding-agent/src/core/mcp/mcp-manager.ts @@ -12,6 +12,14 @@ import type { AuthStorage } from "../auth-storage.js"; import type { McpServerConfig } from "../settings-manager.js"; import type { AcpMcpServerConfig } from "./acp-mcp-types.js"; +export interface AcpMcpOwnerTransferProof { + transactionId: string; + previousOwnerId: string; + nextOwnerId: string; + changed: boolean; + state: "transferred" | "rolled_back"; +} + export interface McpManagerOptions { authStorage: AuthStorage; /** Reads the current Settings.mcpServers (name → config). Re-read on refresh(). */ @@ -22,6 +30,13 @@ export interface McpManagerOptions { /** A resolved integration: a catalog/user entry plus its provider id. */ const GENERIC_SERVER_NAME_PATTERN = /^[A-Za-z0-9][A-Za-z0-9_-]{0,63}$/; +const ACP_MCP_OWNER_TRANSFER_RETENTION_MS = 5 * 60_000; +const ACP_MCP_OWNER_TRANSFER_RECEIPT_LIMIT = 256; + +interface AcpMcpOwnerTransferReceipt { + proof: AcpMcpOwnerTransferProof; + expiresAt: number; +} interface ResolvedIntegration { server: string; @@ -39,6 +54,7 @@ export class McpManager { private integrations = new Map(); private acpServers = new Map(); private acpOwnerId?: string; + private readonly acpOwnerTransfers = new Map(); /** Provider ids we registered for user servers, so refresh can drop removed ones. */ private registeredUserProviderIds = new Set(); @@ -60,6 +76,125 @@ export class McpManager { return this.acpOwnerId === undefined || this.acpOwnerId === ownerId; } + transferAcpServersOwner(previousOwnerId: string, nextOwnerId: string): boolean { + if (!previousOwnerId || !nextOwnerId) throw new Error("ACP MCP owner ids are required"); + if (this.acpOwnerId === undefined) return false; + if (this.acpOwnerId !== previousOwnerId) throw new Error("ACP MCP configuration is owned by another client"); + if (previousOwnerId === nextOwnerId) return false; + this.acpOwnerId = nextOwnerId; + return true; + } + + transferAcpServersOwnerTransaction( + transactionId: string, + previousOwnerId: string, + nextOwnerId: string, + ): AcpMcpOwnerTransferProof { + this.assertAcpOwnerTransferTuple(transactionId, previousOwnerId, nextOwnerId); + const existing = this.getAcpOwnerTransfer(transactionId); + if (existing) { + const matched = this.assertAcpOwnerTransferMatch(existing, previousOwnerId, nextOwnerId); + if (matched.state === "transferred") return { ...matched }; + if (matched.changed) { + if (this.acpOwnerId !== previousOwnerId) { + throw new Error("ACP MCP configuration is owned by another client"); + } + this.acpOwnerId = nextOwnerId; + } + const transferred = { ...matched, state: "transferred" as const }; + this.rememberAcpOwnerTransfer(transferred); + return { ...transferred }; + } + if (this.acpOwnerId !== undefined && this.acpOwnerId !== previousOwnerId) { + throw new Error("ACP MCP configuration is owned by another client"); + } + const changed = this.acpOwnerId !== undefined && previousOwnerId !== nextOwnerId; + if (changed) this.acpOwnerId = nextOwnerId; + const proof: AcpMcpOwnerTransferProof = { + transactionId, + previousOwnerId, + nextOwnerId, + changed, + state: "transferred", + }; + this.rememberAcpOwnerTransfer(proof); + return { ...proof }; + } + + queryAcpServersOwnerTransaction( + transactionId: string, + previousOwnerId: string, + nextOwnerId: string, + ): AcpMcpOwnerTransferProof { + this.assertAcpOwnerTransferTuple(transactionId, previousOwnerId, nextOwnerId); + const proof = this.getAcpOwnerTransfer(transactionId); + if (!proof) throw new Error("ACP MCP owner transfer transaction is unknown"); + return { ...this.assertAcpOwnerTransferMatch(proof, previousOwnerId, nextOwnerId) }; + } + + rollbackAcpServersOwnerTransaction( + transactionId: string, + previousOwnerId: string, + nextOwnerId: string, + ): AcpMcpOwnerTransferProof { + const proof = this.queryAcpServersOwnerTransaction(transactionId, previousOwnerId, nextOwnerId); + if (proof.state === "rolled_back") return proof; + if (proof.changed) { + if (this.acpOwnerId !== nextOwnerId) throw new Error("ACP MCP owner transfer cannot be rolled back"); + this.acpOwnerId = previousOwnerId; + } + const rolledBack = { ...proof, state: "rolled_back" as const }; + this.rememberAcpOwnerTransfer(rolledBack); + return { ...rolledBack }; + } + + retireAcpServersOwnerTransaction(transactionId: string, previousOwnerId: string, nextOwnerId: string): void { + this.assertAcpOwnerTransferTuple(transactionId, previousOwnerId, nextOwnerId); + const proof = this.getAcpOwnerTransfer(transactionId); + if (proof) this.assertAcpOwnerTransferMatch(proof, previousOwnerId, nextOwnerId); + this.acpOwnerTransfers.delete(transactionId); + } + + private getAcpOwnerTransfer(transactionId: string): AcpMcpOwnerTransferProof | undefined { + const receipt = this.acpOwnerTransfers.get(transactionId); + if (!receipt) return undefined; + if (receipt.expiresAt <= Date.now()) { + this.acpOwnerTransfers.delete(transactionId); + return undefined; + } + return receipt.proof; + } + + private rememberAcpOwnerTransfer(proof: AcpMcpOwnerTransferProof): void { + this.acpOwnerTransfers.delete(proof.transactionId); + this.acpOwnerTransfers.set(proof.transactionId, { + proof, + expiresAt: Date.now() + ACP_MCP_OWNER_TRANSFER_RETENTION_MS, + }); + while (this.acpOwnerTransfers.size > ACP_MCP_OWNER_TRANSFER_RECEIPT_LIMIT) { + const oldest = this.acpOwnerTransfers.keys().next().value; + if (oldest === undefined) break; + this.acpOwnerTransfers.delete(oldest); + } + } + + private assertAcpOwnerTransferTuple(transactionId: string, previousOwnerId: string, nextOwnerId: string): void { + if (!transactionId || !previousOwnerId || !nextOwnerId) { + throw new Error("ACP MCP owner transfer transaction and owner ids are required"); + } + } + + private assertAcpOwnerTransferMatch( + proof: AcpMcpOwnerTransferProof, + previousOwnerId: string, + nextOwnerId: string, + ): AcpMcpOwnerTransferProof { + if (proof.previousOwnerId !== previousOwnerId || proof.nextOwnerId !== nextOwnerId) { + throw new Error("ACP MCP owner transfer transaction tuple changed"); + } + return proof; + } + replaceAcpServers(servers: readonly AcpMcpServerConfig[], ownerId: string): boolean { if (!ownerId) throw new Error("ACP MCP owner id is required"); if (servers.length === 0 && this.acpOwnerId !== ownerId) return false; diff --git a/packages/coding-agent/src/index.ts b/packages/coding-agent/src/index.ts index c6e4795bc4..61a2c27e81 100644 --- a/packages/coding-agent/src/index.ts +++ b/packages/coding-agent/src/index.ts @@ -298,6 +298,8 @@ export { type DaemonOwnedSessionDaemonIdentity, type DaemonOwnedSessionDisposeOptions, type DaemonOwnedSessionDisposeResult, + type DaemonRecoverableOwnedSessionAdoptionOptions, + type DaemonRecoverableOwnedSessionAdoptionResult, type ExpiredPromptLifecycle, InProcessAgentConnection, type PromptEventAttribution, @@ -307,6 +309,16 @@ export { type PromptLifecycleSnapshot, type PromptLifecycleStateSnapshot, } from "./modes/agent-connection/index.js"; +export { + adoptRecoverableOwnedSession, + confirmRecoverableOwnedSessionAdoption, + createRecoverableOwnedSession, + type RecoverableOwnedSessionAdoption, + type RecoverableOwnedSessionAdoptionConfirmation, + type RecoverableOwnedSessionAdoptionOptions, + type RecoverableOwnedSessionCreateOptions, + type RecoverableOwnedSessionCreation, +} from "./modes/agent-connection/recoverable-owned-session.js"; // Run modes for programmatic SDK usage export { ClientPromptStashStore, @@ -327,6 +339,7 @@ export { type DaemonCommand, type DaemonCommandEnvelope, type DaemonCommandId, + type DaemonEventCursor, type DaemonEventEnvelope, type DaemonEventId, type DaemonEventMeta, @@ -341,6 +354,10 @@ export { type DaemonProtocolInfo, type DaemonProtocolName, type DaemonProtocolVersion, + type DaemonRecoverableOwnedSessionAdoptionProof, + type DaemonRecoverableOwnedSessionConfirmResult, + type DaemonRecoverableOwnedSessionCreateResult, + type DaemonRecoverableOwnedSessionPrepareResult, type DaemonReplayInfo, type DaemonReplayStatus, type DaemonResponse, @@ -425,6 +442,7 @@ export { CALLER_OWNED_SESSION_ENVIRONMENT_CLEANUP_FEATURE, PRIME_AGENT_SDK_FEATURES, type PrimeAgentSdkFeature, + RECOVERABLE_OWNED_SESSION_ADOPTION_FEATURE, } from "./sdk-features.js"; // Clipboard utilities export { copyToClipboard } from "./utils/clipboard.js"; diff --git a/packages/coding-agent/src/main.ts b/packages/coding-agent/src/main.ts index d5c607f4e1..a98a1e1028 100644 --- a/packages/coding-agent/src/main.ts +++ b/packages/coding-agent/src/main.ts @@ -81,9 +81,10 @@ import { collectDaemonLaunchEnv, } from "./modes/daemon/daemon-protocol.js"; import { - DAEMON_WORKER_ACTIVE_SESSION_ID_ENV, + type DaemonWorkerBootstrapEnvironment, isDaemonWorkerProcess, - requireDaemonWorkerAuthenticationToken, + readDaemonWorkerBootstrapEnvironment, + sanitizeDaemonWorkerBootstrapEnvironment, waitForDaemonWorkerStartupGate, } from "./modes/daemon/daemon-worker-protocol.js"; import { @@ -1053,13 +1054,22 @@ async function findAttachedDaemonSessionSummary( export interface MainOptions { extensionFactories?: ExtensionFactory[]; + /** @internal Captured by the CLI before loading preloads or diagnostics. */ + daemonWorkerBootstrap?: DaemonWorkerBootstrapEnvironment; } export async function main(args: string[], options?: MainOptions) { resetTimings(); - if (isDaemonWorkerProcess()) { - waitForDaemonWorkerStartupGate(); - } + const daemonWorkerBootstrap = + options?.daemonWorkerBootstrap ?? + (isDaemonWorkerProcess() + ? (() => { + waitForDaemonWorkerStartupGate(); + const bootstrap = readDaemonWorkerBootstrapEnvironment(); + sanitizeDaemonWorkerBootstrapEnvironment(process.env); + return bootstrap; + })() + : undefined); installFileLogSink(); if (isDaemonCatalogProcess()) { await runDaemonCatalogProcess(); @@ -1341,14 +1351,17 @@ export async function main(args: string[], options?: MainOptions) { // --list-models still takes the full path to print and exit. if (appMode === "daemon" && parsed.listModels === undefined) { printTimings(); - if (isDaemonWorkerProcess()) { + if (daemonWorkerBootstrap) { await runDaemonMode({ socketPath: parsed.daemonSocket, defaultSessionConfig: daemonDefaultSessionConfig, createRuntime, worker: { - authenticationToken: requireDaemonWorkerAuthenticationToken(), - restoreActiveSessionId: process.env[DAEMON_WORKER_ACTIVE_SESSION_ID_ENV], + authenticationToken: daemonWorkerBootstrap.authenticationToken, + restoreActiveSessionId: daemonWorkerBootstrap.activeSessionId, + supervisorSocketPath: daemonWorkerBootstrap.supervisorSocketPath, + supervisorAgentDir: daemonWorkerBootstrap.supervisorAgentDir, + recoveryJournalPath: daemonWorkerBootstrap.recoveryJournalPath, }, }); } else { diff --git a/packages/coding-agent/src/modes/agent-connection/daemon-agent-connection.ts b/packages/coding-agent/src/modes/agent-connection/daemon-agent-connection.ts index 2a0aeb1cfa..d6e85d9a54 100644 --- a/packages/coding-agent/src/modes/agent-connection/daemon-agent-connection.ts +++ b/packages/coding-agent/src/modes/agent-connection/daemon-agent-connection.ts @@ -32,7 +32,11 @@ import type { RefinementResult } from "../../core/refinement/index.js"; import type { DeleteSessionFileResult } from "../../core/session-file-actions.js"; import { SessionAlreadyActiveError } from "../../core/session-lease.js"; import type { SessionStats } from "../../core/session-stats.js"; -import { CALLER_OWNED_SESSION_ENVIRONMENT_CLEANUP_FEATURE, PRIME_AGENT_SDK_FEATURES } from "../../sdk-features.js"; +import { + CALLER_OWNED_SESSION_ENVIRONMENT_CLEANUP_FEATURE, + PRIME_AGENT_SDK_FEATURES, + RECOVERABLE_OWNED_SESSION_ADOPTION_FEATURE, +} from "../../sdk-features.js"; import { DaemonCapabilityUnavailableError, type DaemonClient, @@ -45,6 +49,7 @@ import { collectDaemonLaunchEnv, DAEMON_PROTOCOL_NAME, DAEMON_PROTOCOL_VERSION, + DAEMON_SCHEMA_REVISION, DAEMON_SNAPSHOT_GENERATION_NONCE_MIN_SCHEMA_REVISION, DAEMON_SUPPORTED_CLIENT_CAPABILITIES, type DaemonAttachResult, @@ -54,6 +59,8 @@ import { type DaemonOutbound, type DaemonOwnedSessionCleanupResult, type DaemonOwnedSessionCompletionResult, + type DaemonRecoverableOwnedSessionAdoptionProof, + type DaemonRecoverableOwnedSessionPrepareResult, type DaemonReplayInfo, type DaemonResponse, type DaemonSessionClosedReason, @@ -62,6 +69,11 @@ import { } from "../daemon/daemon-protocol.js"; import type { SessionSummary } from "../daemon/daemon-session-list.js"; import { listDaemonHeartbeats } from "../daemon/heartbeat-catalog.js"; +import { + appendRecoverableOwnedFrame, + reconcileRecoverableOwnedFrames, + serializeRecoverableOwnedFrame, +} from "../daemon/owned-session-adoption-buffer.js"; import { deleteDaemonSavedSession, listDaemonSavedSessions, @@ -360,6 +372,36 @@ export interface DaemonAgentConnectionOptions { telemetryDisabled?: true; } +export interface DaemonRecoverableOwnedSessionAdoptionOptions { + requestId: string; + recoveryHandle: string; + expectedSupervisorGeneration: string; + activeSessionId: string; + sessionId: string; + correlationId: string; + cursor: DaemonEventCursor; + previousMcpOwnerId: string; + mcpOwnerId: string; + recoveryConfig: AgentSessionRuntimeConfig; + launchEnv: Readonly>; + connectionOptions?: Omit< + DaemonAgentConnectionOptions, + "ownedSession" | "ownedSessionRecoveryConfig" | "ownedSessionLaunchEnv" + >; +} + +export interface DaemonRecoverableOwnedSessionAdoptionResult { + connection: DaemonAgentConnection; + recoveryHandle: string; + proof: DaemonRecoverableOwnedSessionAdoptionProof; +} + +interface RecoverableOwnedClientFrame { + payload: Buffer; + message: DaemonOutbound; + cursor?: DaemonEventCursor; +} + export interface DaemonOwnedSessionDaemonIdentity { readonly protocolName: typeof DAEMON_PROTOCOL_NAME; readonly protocolVersion: number; @@ -568,6 +610,7 @@ export class DaemonAgentConnection implements AgentConnection { private childRosterSequence: number | undefined; private latestSnapshot: AgentConnectionSnapshot | undefined; private latestSnapshotIsFresh = false; + private lastAttachmentReplay: DaemonReplayInfo | undefined; private attachedSessionId: string | undefined; private attachedSessionFile: string | undefined; private daemonLogPath: string | undefined; @@ -588,6 +631,7 @@ export class DaemonAgentConnection implements AgentConnection { private attachmentAdmissionRevision = 0; private attachmentAdmissionInFlight = false; private attachmentInvalidationRevision = 0; + private attachmentSupervisorGenerationFence: string | undefined; private negotiatedCapabilities: ReadonlySet = new Set(); private negotiatedTransportGeneration: number | undefined; private negotiatedRuntimeCapabilities: ReadonlySet = new Set(); @@ -601,8 +645,38 @@ export class DaemonAgentConnection implements AgentConnection { private sessionCloseRetirementInProgress = false; private disposing = false; private disposed = false; + private recoverableAdoptionStaging = false; + private recoverableAdoptionFrames: RecoverableOwnedClientFrame[] = []; + private recoverableAdoptionBufferedBytes = 0; + private recoverableAdoptionError?: Error; private dispatchDaemonMessage(message: DaemonOutbound): void { + if (this.recoverableAdoptionStaging) { + if (this.recoverableAdoptionError) return; + try { + const meta = (message as { meta?: { activeSessionId?: unknown; cursor?: DaemonEventCursor } }).meta; + const envelopeActiveSessionId = (message as { activeSessionId?: unknown }).activeSessionId; + const metaActiveSessionId = meta?.activeSessionId; + const targetsCurrent = + envelopeActiveSessionId === this.activeSessionId || metaActiveSessionId === this.activeSessionId; + if (!targetsCurrent) return; + if ( + (envelopeActiveSessionId !== undefined && envelopeActiveSessionId !== this.activeSessionId) || + (metaActiveSessionId !== undefined && metaActiveSessionId !== this.activeSessionId) || + !meta?.cursor + ) { + throw new Error("Recoverable owned session adoption is unavailable"); + } + this.recoverableAdoptionBufferedBytes = appendRecoverableOwnedFrame( + this.recoverableAdoptionFrames, + this.recoverableAdoptionBufferedBytes, + { payload: serializeRecoverableOwnedFrame(message), message, cursor: meta.cursor }, + ); + } catch { + this.recoverableAdoptionError = new Error("Recoverable owned session adoption is unavailable"); + } + return; + } if (this.replacementReconciliationFailed) return; if (this.deferNegotiatedRuntimeFrame(message)) return; const pendingChunkedReplacement = this.pendingChunkedReplacement; @@ -801,6 +875,201 @@ export class DaemonAgentConnection implements AgentConnection { } } + static async attachRecoverableOwnedSessionCreation( + client: DaemonClient, + activeSessionId: string, + sessionId: string, + expectedSupervisorGeneration: string, + options: DaemonAgentConnectionOptions, + ): Promise { + const connection = new DaemonAgentConnection(client, activeSessionId, options); + try { + connection.attachmentSupervisorGenerationFence = expectedSupervisorGeneration; + await connection.attach(); + connection.assertRecoverableOwnedCreationAttachment(activeSessionId, sessionId, expectedSupervisorGeneration); + connection.attachmentSupervisorGenerationFence = undefined; + return connection; + } catch (error) { + connection.attachmentSupervisorGenerationFence = undefined; + await connection.cleanupFailedRecoverableOwnedCreation(expectedSupervisorGeneration); + throw error; + } + } + + static async adoptRecoverableOwnedSession( + client: DaemonClient, + options: DaemonRecoverableOwnedSessionAdoptionOptions, + ): Promise { + await client.waitForHello(); + if ( + process.platform === "win32" || + !Object.isFrozen(PRIME_AGENT_SDK_FEATURES) || + !PRIME_AGENT_SDK_FEATURES.includes(RECOVERABLE_OWNED_SESSION_ADOPTION_FEATURE) || + !PRIME_AGENT_SDK_FEATURES.includes(CALLER_OWNED_SESSION_ENVIRONMENT_CLEANUP_FEATURE) || + !client.supportsServerCapability("daemon_recoverable_owned_session_adoption_v1") || + !client.supportsServerCapability(CALLER_OWNED_SESSION_ENVIRONMENT_CLEANUP_FEATURE) || + !client.supportsServerCapability("authoritative_owned_session_cleanup_v1") || + (client.hello?.schemaRevision ?? 0) < DAEMON_SCHEMA_REVISION + ) { + throw new Error("Recoverable owned session adoption is unavailable"); + } + let launchEnv: Record; + try { + launchEnv = cloneCallerOwnedSessionLaunchEnv(options.launchEnv) as Record; + } catch { + throw new Error("Recoverable owned session adoption is unavailable"); + } + const connection = new DaemonAgentConnection(client, options.activeSessionId, { + ...options.connectionOptions, + ownedSession: true, + ownedSessionRecoveryConfig: options.recoveryConfig, + ownedSessionLaunchEnv: launchEnv, + }); + connection.recoverableAdoptionStaging = true; + connection.reserveSharedAttachment(options.activeSessionId); + try { + const supportsExtensionUi = options.connectionOptions?.supportsExtensionUi !== false; + const capabilities: DaemonClientCapability[] = [ + "attach_snapshot", + "event_sequence", + "slim_attach", + "correlated_prompt_lifecycle_v1", + "client_owned_sessions", + CALLER_OWNED_SESSION_ENVIRONMENT_CLEANUP_FEATURE, + ...(supportsExtensionUi ? (["extension_ui"] as const) : []), + ]; + const prepared = await connection.requestData( + { + type: "prepare_recoverable_owned_session_adoption", + requestId: options.requestId, + recoveryHandle: options.recoveryHandle, + expectedSupervisorGeneration: options.expectedSupervisorGeneration, + activeSessionId: options.activeSessionId, + sessionId: options.sessionId, + correlationId: options.correlationId, + cursor: options.cursor, + previousMcpOwnerId: options.previousMcpOwnerId, + mcpOwnerId: options.mcpOwnerId, + recoveryConfig: options.recoveryConfig, + launchEnv, + clientId: connection.clientId, + capabilities, + supportsExtensionUi, + telemetryDisabled: options.connectionOptions?.telemetryDisabled, + }, + DAEMON_SNAPSHOT_TIMEOUT_MS, + { recoverAcrossReconnect: true }, + ); + validateDaemonSnapshotIdentity(prepared.snapshot, options.activeSessionId); + if ( + prepared.replay.status !== "complete" || + prepared.snapshot.state.sessionId !== options.sessionId || + prepared.proof.feature !== RECOVERABLE_OWNED_SESSION_ADOPTION_FEATURE || + prepared.proof.status !== "adopted" || + prepared.proof.supervisorGeneration !== options.expectedSupervisorGeneration || + prepared.proof.activeSessionId !== options.activeSessionId || + prepared.proof.sessionId !== options.sessionId || + prepared.proof.correlationId !== options.correlationId || + prepared.proof.mcpOwnerId !== options.mcpOwnerId || + prepared.proof.cursor.generation !== prepared.snapshot.lastEventCursor?.generation || + prepared.proof.cursor.sequence !== prepared.snapshot.lastEventCursor?.sequence + ) { + throw new Error("Recoverable owned session adoption is unavailable"); + } + const snapshotLifecycle = prepared.snapshot.promptLifecycles?.records.find( + (entry) => entry.correlationId === options.correlationId, + ); + const snapshotExpired = prepared.snapshot.promptLifecycles?.expired.find( + (entry) => entry.correlationId === options.correlationId, + ); + if ( + !isDeepStrictEqual( + prepared.proof.lifecycle, + snapshotLifecycle ?? (snapshotExpired ? { ...snapshotExpired, expired: true } : undefined), + ) + ) { + throw new Error("Recoverable owned session adoption is unavailable"); + } + const negotiated = connection.negotiatedCapabilitiesFromAttach(prepared, capabilities); + if ( + !negotiated.has("correlated_prompt_lifecycle_v1") || + !negotiated.has("event_sequence") || + !negotiated.has("client_owned_sessions") || + !negotiated.has(CALLER_OWNED_SESSION_ENVIRONMENT_CLEANUP_FEATURE) + ) { + throw new Error("Recoverable owned session adoption is unavailable"); + } + const staged = connection.stageSnapshotCommit(prepared.snapshot, { + purpose: "attach", + includePromptLifecycles: true, + envelopeActiveSessionId: options.activeSessionId, + expectedSessionId: options.sessionId, + expectedState: prepared.snapshot.state, + replay: prepared.replay, + resetEventProgress: true, + progress: [ + { cursor: prepared.lastEventCursor, sequence: prepared.lastEventSequence }, + { cursor: prepared.snapshot.lastEventCursor, sequence: prepared.snapshot.lastEventSequence }, + ], + }); + if (connection.recoverableAdoptionError) throw connection.recoverableAdoptionError; + const beforeCommit = reconcileRecoverableOwnedFrames( + connection.recoverableAdoptionFrames, + prepared.proof.cursor, + ); + connection.recoverableAdoptionFrames = beforeCommit.frames; + connection.recoverableAdoptionBufferedBytes = beforeCommit.bufferedBytes; + const proof = await connection.requestData( + { + type: "commit_recoverable_owned_session_adoption", + requestId: options.requestId, + expectedSupervisorGeneration: options.expectedSupervisorGeneration, + recoveryHandle: prepared.recoveryHandle, + proof: prepared.proof, + }, + DAEMON_SNAPSHOT_TIMEOUT_MS, + { recoverAcrossReconnect: true }, + ); + if (!isDeepStrictEqual(proof, prepared.proof) || connection.recoverableAdoptionError) { + throw new Error("Recoverable owned session adoption is unavailable"); + } + const afterCommit = reconcileRecoverableOwnedFrames( + connection.recoverableAdoptionFrames, + prepared.proof.cursor, + ); + connection.commitStagedSnapshot(staged); + connection.publishNegotiatedCapabilityProof(negotiated, client.getTransportGeneration()); + const attachedContractProof = connection.getOwnedSessionContractProof(); + if ( + !attachedContractProof || + attachedContractProof.feature !== CALLER_OWNED_SESSION_ENVIRONMENT_CLEANUP_FEATURE || + attachedContractProof.status !== "attached" || + attachedContractProof.daemon.supervisorGeneration !== options.expectedSupervisorGeneration + ) { + throw new Error("Recoverable owned session adoption is unavailable"); + } + connection.recoverableAdoptionStaging = false; + connection.recoverableAdoptionFrames = []; + connection.recoverableAdoptionBufferedBytes = 0; + for (const frame of afterCommit.frames) connection.dispatchDaemonMessage(frame.message); + if (connection.replacementReconciliationFailed) { + throw new Error("Recoverable owned session adoption is unavailable"); + } + return { connection, recoveryHandle: prepared.recoveryHandle, proof }; + } catch { + connection.recoverableAdoptionStaging = false; + connection.recoverableAdoptionFrames = []; + connection.recoverableAdoptionBufferedBytes = 0; + connection.recoverableAdoptionError = undefined; + connection.invalidateNegotiatedCapabilityProof(); + connection.unsubscribeDaemonMessages(); + connection.unsubscribeDaemonClose(); + connection.disposed = true; + if (options.connectionOptions?.closeClientOnDispose) client.close(); + throw new Error("Recoverable owned session adoption is unavailable"); + } + } + async attach(): Promise { if (this.disposing || this.disposed) throw new Error("Daemon connection is disposing"); await this.attachSession(this.activeSessionId, this.lastEventCursor, false); @@ -819,6 +1088,7 @@ export class DaemonAgentConnection implements AgentConnection { this.attachmentAdmissionInFlight = true; this.fenceSnapshotTransfersForAttachmentAdmission(preserveRuntimeSnapshotAttempt); this.retirePendingChunkedReplacement(); + this.lastAttachmentReplay = undefined; this.invalidateNegotiatedCapabilityProof(); if (this.replacementReconciliationFailed) { if (this.attachmentAdmissionRevision === admissionRevision) this.attachmentAdmissionInFlight = false; @@ -892,6 +1162,7 @@ export class DaemonAgentConnection implements AgentConnection { result = await this.requestData({ type: "attach", activeSessionId: requestedActiveSessionId, + expectedSupervisorGeneration: this.attachmentSupervisorGenerationFence, ...(this.supportsSnapshotGenerationNonce() ? { snapshotGenerationNonce: randomUUID() } : {}), supportsExtensionUi, clientId: this.clientId, @@ -982,6 +1253,7 @@ export class DaemonAgentConnection implements AgentConnection { ); this.commitStagedSnapshot(staged); this.publishNegotiatedCapabilityProof(negotiatedCapabilities, transportGeneration); + this.lastAttachmentReplay = result.replay; } else { validateSummaryIdentity(result, nextActiveSessionId); this.assertAttachmentCommit( @@ -998,6 +1270,7 @@ export class DaemonAgentConnection implements AgentConnection { this.latestSnapshot = undefined; this.latestSnapshotIsFresh = false; this.publishNegotiatedCapabilityProof(negotiatedCapabilities, transportGeneration); + this.lastAttachmentReplay = undefined; } this.captureDaemonLogPath(); this.updateReconnectFailed = false; @@ -1143,6 +1416,54 @@ export class DaemonAgentConnection implements AgentConnection { return this.client.supportsServerCapability("acp_mcp_servers"); } + private assertRecoverableOwnedCreationAttachment( + activeSessionId: string, + sessionId: string, + expectedSupervisorGeneration: string, + ): void { + const contract = this.getOwnedSessionContractProof(); + const requiredCapabilities: readonly DaemonClientCapability[] = [ + "attach_snapshot", + "event_sequence", + "correlated_prompt_lifecycle_v1", + "client_owned_sessions", + CALLER_OWNED_SESSION_ENVIRONMENT_CLEANUP_FEATURE, + ]; + if ( + this.client.hello?.supervisorGeneration !== expectedSupervisorGeneration || + this.lastAttachmentReplay?.status !== "complete" || + !this.latestSnapshotIsFresh || + this.activeSessionId !== activeSessionId || + this.attachedSessionId !== sessionId || + this.latestSnapshot?.state.sessionId !== sessionId || + this.sharedAttachmentActiveSessionId !== activeSessionId || + sharedAttachmentOwners.get(this.client)?.get(activeSessionId) !== this || + !contract || + contract.status !== "attached" || + contract.daemon.supervisorGeneration !== expectedSupervisorGeneration || + contract.daemon.transportGeneration !== this.client.getTransportGeneration() || + requiredCapabilities.some((capability) => !this.supportsNegotiatedCapability(capability)) + ) { + throw new Error("Recoverable owned session adoption is unavailable"); + } + } + + private async cleanupFailedRecoverableOwnedCreation(expectedSupervisorGeneration: string): Promise { + const transportGeneration = this.client.getTransportGeneration(); + if ( + this.client.isConnected && + this.client.hello?.supervisorGeneration === expectedSupervisorGeneration && + this.client.getTransportGeneration() === transportGeneration + ) { + await this.client + .request({ type: "complete_owned_session", activeSessionId: this.activeSessionId }, 2_000, { + recoverAcrossReconnect: false, + }) + .catch(() => undefined); + } + await this.finalizeDisposedConnection(this.activeSessionId, "none"); + } + /** Current secret-free proof that this owned attachment uses the caller-owned environment contract. */ getOwnedSessionContractProof(): DaemonOwnedSessionContractProof | undefined { if ( diff --git a/packages/coding-agent/src/modes/agent-connection/index.ts b/packages/coding-agent/src/modes/agent-connection/index.ts index 69d24bff3a..a6d16c0636 100644 --- a/packages/coding-agent/src/modes/agent-connection/index.ts +++ b/packages/coding-agent/src/modes/agent-connection/index.ts @@ -8,15 +8,35 @@ export type { PromptLifecycleSnapshot, PromptLifecycleStateSnapshot, } from "../../core/prompt-lifecycle.js"; +export type { + DaemonEventCursor, + DaemonRecoverableOwnedSessionAdoptionProof, + DaemonRecoverableOwnedSessionConfirmResult, + DaemonRecoverableOwnedSessionCreateResult, + DaemonRecoverableOwnedSessionPrepareResult, +} from "../daemon/daemon-protocol.js"; export type { DaemonAgentConnectionOptions, DaemonOwnedSessionContractProof, DaemonOwnedSessionDaemonIdentity, DaemonOwnedSessionDisposeOptions, DaemonOwnedSessionDisposeResult, + DaemonRecoverableOwnedSessionAdoptionOptions, + DaemonRecoverableOwnedSessionAdoptionResult, } from "./daemon-agent-connection.js"; export { DaemonAgentConnection } from "./daemon-agent-connection.js"; export { InProcessAgentConnection } from "./in-process-agent-connection.js"; +export { + adoptRecoverableOwnedSession, + confirmRecoverableOwnedSessionAdoption, + createRecoverableOwnedSession, + type RecoverableOwnedSessionAdoption, + type RecoverableOwnedSessionAdoptionConfirmation, + type RecoverableOwnedSessionAdoptionOptions, + type RecoverableOwnedSessionCreateOptions, + type RecoverableOwnedSessionCreation, +} from "./recoverable-owned-session.js"; + export { createAgentConnectionCommands, createAgentConnectionState } from "./snapshot.js"; export type { AgentConnection, diff --git a/packages/coding-agent/src/modes/agent-connection/recoverable-owned-session.ts b/packages/coding-agent/src/modes/agent-connection/recoverable-owned-session.ts new file mode 100644 index 0000000000..c07b3734af --- /dev/null +++ b/packages/coding-agent/src/modes/agent-connection/recoverable-owned-session.ts @@ -0,0 +1,252 @@ +import type { AgentSessionRuntimeConfig } from "../../core/agent-session-config.js"; +import type { AgentSessionRuntimeMetadata } from "../../core/agent-session-runtime.js"; +import { + CALLER_OWNED_SESSION_ENVIRONMENT_CLEANUP_FEATURE, + PRIME_AGENT_SDK_FEATURES, + RECOVERABLE_OWNED_SESSION_ADOPTION_FEATURE, +} from "../../sdk-features.js"; +import type { DaemonClient } from "../daemon/daemon-client.js"; +import { deserializeDaemonError } from "../daemon/daemon-errors.js"; +import { + cloneCallerOwnedSessionLaunchEnv, + collectDaemonClientEnv, + DAEMON_SCHEMA_REVISION, + type DaemonEventCursor, + type DaemonRecoverableOwnedSessionAdoptionProof, + type DaemonRecoverableOwnedSessionConfirmResult, + type DaemonRecoverableOwnedSessionCreateResult, + isDaemonRecoveryRequestId, +} from "../daemon/daemon-protocol.js"; +import type { SessionSummary } from "../daemon/daemon-session-list.js"; +import { DaemonAgentConnection, type DaemonAgentConnectionOptions } from "./daemon-agent-connection.js"; + +export interface RecoverableOwnedSessionCreateOptions { + requestId: string; + correlationId: string; + mcpOwnerId: string; + config: AgentSessionRuntimeConfig; + sessionPath?: string; + continueRecent?: boolean; + noSession?: boolean; + name?: string; + runtimeMetadata?: AgentSessionRuntimeMetadata; + launchEnv: Readonly>; + connectionOptions?: Omit< + DaemonAgentConnectionOptions, + "ownedSession" | "ownedSessionRecoveryConfig" | "ownedSessionLaunchEnv" + >; +} + +export interface RecoverableOwnedSessionCreation { + connection: DaemonAgentConnection; + state: SessionSummary; + recoveryHandle: string; + supervisorGeneration: string; + ownershipGeneration: number; +} + +export interface RecoverableOwnedSessionAdoptionOptions { + requestId: string; + recoveryHandle: string; + expectedSupervisorGeneration: string; + activeSessionId: string; + sessionId: string; + correlationId: string; + cursor: DaemonEventCursor; + previousMcpOwnerId: string; + mcpOwnerId: string; + config: AgentSessionRuntimeConfig; + launchEnv: Readonly>; + connectionOptions?: Omit< + DaemonAgentConnectionOptions, + "ownedSession" | "ownedSessionRecoveryConfig" | "ownedSessionLaunchEnv" + >; +} + +export interface RecoverableOwnedSessionAdoption { + connection: DaemonAgentConnection; + recoveryHandle: string; + proof: DaemonRecoverableOwnedSessionAdoptionProof; +} + +export interface RecoverableOwnedSessionAdoptionConfirmation { + requestId: string; + recoveryHandle: string; + proof: DaemonRecoverableOwnedSessionAdoptionProof; +} + +function assertRecoverableOwnedSessionFeature(client: DaemonClient): void { + if ( + process.platform === "win32" || + !Object.isFrozen(PRIME_AGENT_SDK_FEATURES) || + !PRIME_AGENT_SDK_FEATURES.includes(RECOVERABLE_OWNED_SESSION_ADOPTION_FEATURE) || + !PRIME_AGENT_SDK_FEATURES.includes(CALLER_OWNED_SESSION_ENVIRONMENT_CLEANUP_FEATURE) || + !client.supportsServerCapability("daemon_recoverable_owned_session_adoption_v1") || + !client.supportsServerCapability(CALLER_OWNED_SESSION_ENVIRONMENT_CLEANUP_FEATURE) || + !client.supportsServerCapability("authoritative_owned_session_cleanup_v1") || + (client.hello?.schemaRevision ?? 0) < DAEMON_SCHEMA_REVISION + ) { + throw new Error("Recoverable owned session adoption is unavailable"); + } +} + +function assertRequestId(requestId: string): void { + if (!isDaemonRecoveryRequestId(requestId)) { + throw new Error("Recoverable owned session requestId must encode at least 128 bits"); + } +} + +function assertRecoveryHandle(recoveryHandle: string): void { + if (!/^[A-Za-z0-9_-]{43}$/.test(recoveryHandle)) { + throw new Error("Recoverable owned session adoption returned an invalid recovery handle"); + } +} + +async function createRecoverableOwnedSessionInternal( + client: DaemonClient, + options: RecoverableOwnedSessionCreateOptions, +): Promise { + const launchEnv = cloneCallerOwnedSessionLaunchEnv(options.launchEnv) as Record; + await client.waitForHello(); + assertRecoverableOwnedSessionFeature(client); + assertRequestId(options.requestId); + const expectedSupervisorGeneration = client.hello?.supervisorGeneration; + if (!expectedSupervisorGeneration) { + throw new Error("Recoverable owned session adoption is unavailable"); + } + const response = await client.request( + { + type: "create_recoverable_owned_session", + requestId: options.requestId, + expectedSupervisorGeneration, + correlationId: options.correlationId, + mcpOwnerId: options.mcpOwnerId, + recoveryConfig: options.config, + ...(options.sessionPath !== undefined ? { sessionPath: options.sessionPath } : {}), + ...(options.continueRecent !== undefined ? { continueRecent: options.continueRecent } : {}), + ...(options.noSession !== undefined ? { noSession: options.noSession } : {}), + ...(options.name !== undefined ? { name: options.name } : {}), + config: options.config, + ...(options.runtimeMetadata !== undefined ? { runtimeMetadata: options.runtimeMetadata } : {}), + env: collectDaemonClientEnv(), + launchEnv, + launchEnvMode: "replace", + }, + 30_000, + { recoverAcrossReconnect: true }, + ); + if (!response.success) throw deserializeDaemonError(response); + const created = response.data as DaemonRecoverableOwnedSessionCreateResult; + assertRecoveryHandle(created.recoveryHandle); + if ( + created.supervisorGeneration !== expectedSupervisorGeneration || + created.ownershipGeneration !== 0 || + !created.state?.activeSessionId || + !created.state.sessionId + ) { + throw new Error("Recoverable owned session creation returned an invalid authority receipt"); + } + const connection = await DaemonAgentConnection.attachRecoverableOwnedSessionCreation( + client, + created.state.activeSessionId, + created.state.sessionId, + expectedSupervisorGeneration, + { + ...options.connectionOptions, + ownedSession: true, + ownedSessionRecoveryConfig: options.config, + ownedSessionLaunchEnv: launchEnv, + }, + ); + return { connection, ...created }; +} + +async function adoptRecoverableOwnedSessionInternal( + client: DaemonClient, + options: RecoverableOwnedSessionAdoptionOptions, +): Promise { + const launchEnv = cloneCallerOwnedSessionLaunchEnv(options.launchEnv); + await client.waitForHello(); + assertRecoverableOwnedSessionFeature(client); + assertRequestId(options.requestId); + assertRecoveryHandle(options.recoveryHandle); + const adopted = await DaemonAgentConnection.adoptRecoverableOwnedSession(client, { + requestId: options.requestId, + recoveryHandle: options.recoveryHandle, + expectedSupervisorGeneration: options.expectedSupervisorGeneration, + activeSessionId: options.activeSessionId, + sessionId: options.sessionId, + correlationId: options.correlationId, + cursor: options.cursor, + previousMcpOwnerId: options.previousMcpOwnerId, + mcpOwnerId: options.mcpOwnerId, + recoveryConfig: options.config, + launchEnv, + connectionOptions: options.connectionOptions, + }); + assertRecoveryHandle(adopted.recoveryHandle); + return adopted; +} + +/** Confirm only after the caller has durably persisted the rotated handle and exact proof. */ +async function confirmRecoverableOwnedSessionAdoptionInternal( + client: DaemonClient, + confirmation: RecoverableOwnedSessionAdoptionConfirmation, +): Promise { + await client.waitForHello(); + assertRecoverableOwnedSessionFeature(client); + assertRequestId(confirmation.requestId); + assertRecoveryHandle(confirmation.recoveryHandle); + const response = await client.request( + { + type: "confirm_recoverable_owned_session_adoption", + requestId: confirmation.requestId, + expectedSupervisorGeneration: confirmation.proof.supervisorGeneration, + recoveryHandle: confirmation.recoveryHandle, + proof: confirmation.proof, + }, + 30_000, + { recoverAcrossReconnect: true }, + ); + if (!response.success) throw deserializeDaemonError(response); + if ((response.data as Partial | undefined)?.status !== "confirmed") { + throw new Error("Recoverable owned session adoption returned an invalid confirmation"); + } +} + +function unavailable(): Error { + return new Error("Recoverable owned session adoption is unavailable"); +} + +export async function createRecoverableOwnedSession( + client: DaemonClient, + options: RecoverableOwnedSessionCreateOptions, +): Promise { + try { + return await createRecoverableOwnedSessionInternal(client, options); + } catch { + throw unavailable(); + } +} + +export async function adoptRecoverableOwnedSession( + client: DaemonClient, + options: RecoverableOwnedSessionAdoptionOptions, +): Promise { + try { + return await adoptRecoverableOwnedSessionInternal(client, options); + } catch { + throw unavailable(); + } +} + +export async function confirmRecoverableOwnedSessionAdoption( + client: DaemonClient, + confirmation: RecoverableOwnedSessionAdoptionConfirmation, +): Promise { + try { + await confirmRecoverableOwnedSessionAdoptionInternal(client, confirmation); + } catch { + throw unavailable(); + } +} diff --git a/packages/coding-agent/src/modes/daemon/command-recovery-journal.ts b/packages/coding-agent/src/modes/daemon/command-recovery-journal.ts index e4f7bde7a0..51cdeb5fdf 100644 --- a/packages/coding-agent/src/modes/daemon/command-recovery-journal.ts +++ b/packages/coding-agent/src/modes/daemon/command-recovery-journal.ts @@ -1,6 +1,6 @@ import { chmodSync, closeSync, fsyncSync, mkdirSync, openSync, readFileSync, renameSync, writeSync } from "node:fs"; import { dirname } from "node:path"; -import type { DaemonClientId, DaemonCommandId, DaemonResponse } from "./daemon-protocol.js"; +import type { DaemonClientId, DaemonCommand, DaemonCommandId, DaemonResponse } from "./daemon-protocol.js"; interface ReceivedRecord { version: 1; @@ -43,6 +43,17 @@ export type CommandJournalBeginResult = const COMPACT_AFTER_RECORDS = 4096; +const BEARER_RETURNING_COMMANDS: ReadonlySet = new Set([ + "create_recoverable_owned_session", + "prepare_recoverable_owned_session_adoption", + "commit_recoverable_owned_session_adoption", + "confirm_recoverable_owned_session_adoption", +]); + +export function commandRecoveryJournalAllows(commandType: string): boolean { + return !BEARER_RETURNING_COMMANDS.has(commandType as DaemonCommand["type"]); +} + export function createCommandIdempotencyKey(clientId: DaemonClientId, commandId: DaemonCommandId): string { return JSON.stringify([clientId, commandId]); } @@ -84,6 +95,9 @@ export class CommandRecoveryJournal { commandType: string, requestIdentity?: string, ): CommandJournalBeginResult { + if (!commandRecoveryJournalAllows(commandType)) { + throw new Error(`Command recovery journal rejects bearer-returning command: ${commandType}`); + } const key = createCommandIdempotencyKey(clientId, commandId); const existing = this.lookup(clientId, commandId, requestIdentity); if (existing) return existing; diff --git a/packages/coding-agent/src/modes/daemon/daemon-client.ts b/packages/coding-agent/src/modes/daemon/daemon-client.ts index 233289a350..92598be754 100644 --- a/packages/coding-agent/src/modes/daemon/daemon-client.ts +++ b/packages/coding-agent/src/modes/daemon/daemon-client.ts @@ -67,6 +67,8 @@ interface PendingDaemonRequest { recoverAcrossReconnect: boolean; /** Re-checked against the new hello before a reconnect replay. */ compatibilities: readonly DaemonCommandCompatibility[]; + /** Fences generation-bound recovery commands before a reconnect replay. */ + expectedSupervisorGeneration?: string; } function daemonEndpointDetails(socketPath: string): string { @@ -166,6 +168,10 @@ export class DaemonClient { return this.helloMessage; } + get clientId(): string { + return this.protocolClientId; + } + get isConnected(): boolean { return this.socket !== undefined && !this.socket.destroyed; } @@ -380,6 +386,13 @@ export class DaemonClient { ); } const hello = this.helloMessage ?? (await this.waitForHello()); + const expectedSupervisorGeneration = + "expectedSupervisorGeneration" in command && typeof command.expectedSupervisorGeneration === "string" + ? command.expectedSupervisorGeneration + : undefined; + if (expectedSupervisorGeneration !== undefined && hello.supervisorGeneration !== expectedSupervisorGeneration) { + throw new Error("Recoverable owned session adoption is unavailable"); + } const compatibilities = getDaemonCommandCompatibilities(command); const missingCompatibility = compatibilities.find( (compatibility) => !this.meetsCommandCompatibility(hello, compatibility), @@ -461,6 +474,9 @@ export class DaemonClient { acknowledgeResult, recoverAcrossReconnect: options.recoverAcrossReconnect !== false, compatibilities, + ...("expectedSupervisorGeneration" in command && typeof command.expectedSupervisorGeneration === "string" + ? { expectedSupervisorGeneration: command.expectedSupervisorGeneration } + : {}), }; this.pendingRequests.set(id, pending); this.armPendingRequestTimeout(id, pending); @@ -546,6 +562,14 @@ export class DaemonClient { continue; } pending.awaitingReconnect = false; + if ( + pending.expectedSupervisorGeneration !== undefined && + message.supervisorGeneration !== pending.expectedSupervisorGeneration + ) { + this.pendingRequests.delete(id); + pending.reject(new Error("Recoverable owned session adoption is unavailable")); + continue; + } const missingCompatibility = pending.compatibilities.find( (compatibility) => !this.meetsCommandCompatibility(message, compatibility), ); diff --git a/packages/coding-agent/src/modes/daemon/daemon-mode.ts b/packages/coding-agent/src/modes/daemon/daemon-mode.ts index 35688f3641..2c47f42d0f 100644 --- a/packages/coding-agent/src/modes/daemon/daemon-mode.ts +++ b/packages/coding-agent/src/modes/daemon/daemon-mode.ts @@ -7,7 +7,7 @@ */ import { spawn } from "node:child_process"; -import { createHash, randomUUID } from "node:crypto"; +import { createHash, randomBytes, randomUUID } from "node:crypto"; import { existsSync, mkdirSync, readFileSync, renameSync, rmSync, writeFileSync } from "node:fs"; import { stat } from "node:fs/promises"; import { createConnection, createServer, type Server, type Socket } from "node:net"; @@ -18,6 +18,7 @@ import { type Api, getLogger, type Model } from "@earendil-works/pi-ai"; import { createCliSubprocessEnv, createCliSubprocessLaunchSpec } from "../../cli/subprocess-launch.js"; import { appendRotatingLog, + ENV_AGENT_DIR, getCronJobsPath, getDaemonLogPath, getDaemonUpdateRestartManifestPath, @@ -183,16 +184,17 @@ import { } from "./daemon-socket.js"; import { assertDaemonSupervisorOwnerCurrent, isDaemonShutdownAdmissionActive } from "./daemon-supervisor-ownership.js"; import { - DAEMON_WORKER_ACTIVE_SESSION_ID_ENV, DAEMON_WORKER_RECOVERY_JOURNAL_ENV, - DAEMON_WORKER_ROLE_ENV, + DAEMON_WORKER_SUPERVISOR_AGENT_DIR_ENV, DAEMON_WORKER_SUPERVISOR_SOCKET_ENV, - DAEMON_WORKER_TOKEN_ENV, + type DaemonWorkerAcpMcpOwnerTransferProof, + type DaemonWorkerAuthenticationResult, type DaemonWorkerCommand, type DaemonWorkerFrameHeader, isDaemonWorkerFrameHeader, SESSION_LEASE_OWNER_ID_ENV, SESSION_LEASES_ENABLED_ENV, + sanitizeDaemonWorkerBootstrapEnvironment, } from "./daemon-worker-protocol.js"; import { MutationDrainLatch } from "./mutation-drain-latch.js"; import { @@ -226,6 +228,9 @@ export interface DaemonModeOptions { worker?: { authenticationToken: string; restoreActiveSessionId?: string; + supervisorSocketPath?: string; + supervisorAgentDir?: string; + recoveryJournalPath?: string; }; } @@ -484,6 +489,8 @@ export class AgentDaemon { { client: DaemonSocketClient; ownerId: string; serverNames: string[]; release?: Promise } >(); private readonly mutationDrain = new MutationDrainLatch(); + /** Authenticated process incarnation. It is returned only on the private worker channel. */ + private readonly workerIncarnation = randomBytes(32).toString("base64url"); private updateRestart?: { id: symbol; owner?: DaemonSocketClient; @@ -609,8 +616,9 @@ export class AgentDaemon { ? AgentCronJobStore.forSessionArtifacts() : new AgentCronJobStore(getCronJobsPath(this.agentDir)); this.restoreActiveSessionId = options.worker?.restoreActiveSessionId; - const recoveryJournalPath = process.env[DAEMON_WORKER_RECOVERY_JOURNAL_ENV]; - if (options.worker && recoveryJournalPath) { + const recoveryJournalPath = + options.worker?.recoveryJournalPath ?? process.env[DAEMON_WORKER_RECOVERY_JOURNAL_ENV]; + if (recoveryJournalPath) { this.recoveryJournal = new WorkerRecoveryJournal(recoveryJournalPath); } this.cronScheduler = new AgentCronScheduler(this.cronStore, { @@ -702,7 +710,7 @@ export class AgentDaemon { } private supervisorSocketPathFromEnv(): string | undefined { - const raw = process.env[DAEMON_WORKER_SUPERVISOR_SOCKET_ENV]; + const raw = this.options.worker?.supervisorSocketPath ?? process.env[DAEMON_WORKER_SUPERVISOR_SOCKET_ENV]; return raw ? normalizeSocketPath(raw) : undefined; } @@ -887,12 +895,10 @@ export class AgentDaemon { return; } const launch = createCliSubprocessLaunchSpec(["--mode", "daemon", "--daemon-socket", supervisorSocketPath]); - const environment = createCliSubprocessEnv(); - delete environment[DAEMON_WORKER_ROLE_ENV]; - delete environment[DAEMON_WORKER_TOKEN_ENV]; - delete environment[DAEMON_WORKER_ACTIVE_SESSION_ID_ENV]; - delete environment[DAEMON_WORKER_RECOVERY_JOURNAL_ENV]; - delete environment[DAEMON_WORKER_SUPERVISOR_SOCKET_ENV]; + const environment = sanitizeDaemonWorkerBootstrapEnvironment(createCliSubprocessEnv()); + const supervisorAgentDir = + this.options.worker?.supervisorAgentDir ?? process.env[DAEMON_WORKER_SUPERVISOR_AGENT_DIR_ENV]; + if (supervisorAgentDir) environment[ENV_AGENT_DIR] = supervisorAgentDir; delete environment[ORPHAN_PROCESS_JOURNAL_ENV]; delete environment[SESSION_LEASES_ENABLED_ENV]; delete environment[SESSION_LEASE_OWNER_ID_ENV]; @@ -3466,6 +3472,7 @@ export class AgentDaemon { type: "response", command: "worker_auth", success: true, + data: { workerIncarnation: this.workerIncarnation } satisfies DaemonWorkerAuthenticationResult, }); return; } @@ -3626,6 +3633,61 @@ export class AgentDaemon { this.write(client, success(command.id, "detach")); return; } + case "worker_transfer_acp_mcp_owner": { + const state = this.getBoundSessionState(command.activeSessionId); + const current = this.acpMcpOwners.get(state.activeSessionId); + if (current?.release) throw new Error("ACP MCP configuration is being released"); + if (current && current.client !== client) { + throw new Error("ACP MCP configuration is owned by another daemon client"); + } + if (command.action === "retire") { + state.runtime.session.retireAcpMcpServersOwnerTransaction( + command.transactionId, + command.previousOwnerId, + command.ownerId, + ); + this.writeWorkerSuccess(client, command, { + transactionId: command.transactionId, + previousOwnerId: command.previousOwnerId, + nextOwnerId: command.ownerId, + status: "retired" as const, + }); + return; + } + const proof: DaemonWorkerAcpMcpOwnerTransferProof = + command.action === "transfer" + ? state.runtime.session.transferAcpMcpServersOwnerTransaction( + command.transactionId, + command.previousOwnerId, + command.ownerId, + ) + : command.action === "query" + ? state.runtime.session.queryAcpMcpServersOwnerTransaction( + command.transactionId, + command.previousOwnerId, + command.ownerId, + ) + : state.runtime.session.rollbackAcpMcpServersOwnerTransaction( + command.transactionId, + command.previousOwnerId, + command.ownerId, + ); + if (current && proof.changed) { + current.ownerId = proof.state === "transferred" ? command.ownerId : command.previousOwnerId; + } + if ( + proof.transactionId !== command.transactionId || + proof.previousOwnerId !== command.previousOwnerId || + proof.nextOwnerId !== command.ownerId || + (proof.state !== "transferred" && proof.state !== "rolled_back") || + (current && + current.ownerId !== (proof.state === "transferred" ? command.ownerId : command.previousOwnerId)) + ) { + throw new Error("ACP MCP owner transfer proof is invalid"); + } + this.writeWorkerSuccess(client, command, proof); + return; + } case "worker_archive_and_shutdown": { for (const state of [...this.sessions.values()]) { await this.closeSession(state, "killed"); @@ -7470,6 +7532,7 @@ type SequencedDaemonOutbound = Extract< { type: | "session_event" + | "prompt_lifecycle" | "session_status" | "session_replaced" | "session_resynced" diff --git a/packages/coding-agent/src/modes/daemon/daemon-protocol.ts b/packages/coding-agent/src/modes/daemon/daemon-protocol.ts index 6e76cee18a..fbb36792f9 100644 --- a/packages/coding-agent/src/modes/daemon/daemon-protocol.ts +++ b/packages/coding-agent/src/modes/daemon/daemon-protocol.ts @@ -28,7 +28,10 @@ import type { import type { QueuedMessageLane, QueuedMessageMutation } from "../../core/session-action-store.js"; import type { SessionCwdIssue } from "../../core/session-cwd.js"; import type { DeleteSessionFileResult } from "../../core/session-file-actions.js"; -import { CALLER_OWNED_SESSION_ENVIRONMENT_CLEANUP_FEATURE } from "../../sdk-features.js"; +import { + CALLER_OWNED_SESSION_ENVIRONMENT_CLEANUP_FEATURE, + type RECOVERABLE_OWNED_SESSION_ADOPTION_FEATURE, +} from "../../sdk-features.js"; import type { AgentConnectionAgentStatus, AgentConnectionHeartbeat, @@ -81,8 +84,9 @@ export const DAEMON_COMMAND_ENVELOPE_MIN_PROTOCOL_VERSION = 7; // Revision 27 adds a capability-gated authoritative owned-session cleanup query. // Revision 28 negotiates fresh snapshot generations and adds private worker chunk routing metadata. // Revision 29 capability-gates exact caller-owned launch environments and observable cleanup results. -export const DAEMON_SCHEMA_REVISION = 29; -export const DAEMON_SCHEMA_ID = "protocol-7-schema-29-5450eb231171"; +// Revision 30 adds capability-gated same-supervisor recoverable owned-session adoption. +export const DAEMON_SCHEMA_REVISION = 30; +export const DAEMON_SCHEMA_ID = "protocol-7-schema-30-2e0904d3ad1a"; export type DaemonProtocolName = typeof DAEMON_PROTOCOL_NAME; export type DaemonProtocolVersion = number; @@ -141,7 +145,8 @@ export type DaemonServerCapability = | "owned_prompt_cancellation" | "acp_mcp_servers" | "snapshot_generation_nonce_v1" - | "authoritative_owned_session_cleanup_v1"; + | "authoritative_owned_session_cleanup_v1" + | "daemon_recoverable_owned_session_adoption_v1"; export type DaemonReplayStatus = "complete" | "partial" | "unavailable"; @@ -199,8 +204,26 @@ export const DAEMON_SUPERVISOR_SERVER_CAPABILITIES: readonly DaemonServerCapabil ...DAEMON_DEFAULT_SERVER_CAPABILITIES, "authoritative_owned_session_cleanup_v1", CALLER_OWNED_SESSION_ENVIRONMENT_CLEANUP_FEATURE, + "daemon_recoverable_owned_session_adoption_v1", ]; +export function daemonSupervisorServerCapabilities( + platform: NodeJS.Platform = process.platform, +): readonly DaemonServerCapability[] { + return platform === "win32" + ? DAEMON_SUPERVISOR_SERVER_CAPABILITIES.filter( + (capability) => capability !== "daemon_recoverable_owned_session_adoption_v1", + ) + : DAEMON_SUPERVISOR_SERVER_CAPABILITIES; +} + +export function isDaemonRecoveryRequestId(value: unknown): value is string { + if (typeof value !== "string" || value.length > 256) return false; + if (/^[0-9a-f]{32,}$/i.test(value) && value.length % 2 === 0) return true; + if (/^[A-Za-z0-9_-]{22,}$/.test(value)) return true; + return /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i.test(value); +} + export interface DaemonRuntimeIdentity { buildId: string; executablePath: string; @@ -469,6 +492,35 @@ export type DaemonSavedSessionListCommand = scope: AgentConnectionSavedSessionScope; }; +export interface DaemonRecoverableOwnedSessionCreateResult { + state: SessionSummary; + recoveryHandle: string; + supervisorGeneration: string; + ownershipGeneration: number; +} + +export interface DaemonRecoverableOwnedSessionAdoptionProof { + feature: typeof RECOVERABLE_OWNED_SESSION_ADOPTION_FEATURE; + status: "adopted"; + supervisorGeneration: string; + ownershipGeneration: number; + activeSessionId: string; + sessionId: string; + correlationId: string; + lifecycle: PromptLifecycleSnapshot | { correlationId: string; expired: true; deliveryCrossed: boolean }; + cursor: DaemonEventCursor; + mcpOwnerId: string; +} + +export interface DaemonRecoverableOwnedSessionPrepareResult extends DaemonAttachResult { + recoveryHandle: string; + proof: DaemonRecoverableOwnedSessionAdoptionProof; +} + +export interface DaemonRecoverableOwnedSessionConfirmResult { + status: "confirmed"; +} + export type DaemonCommand = | { id?: string; @@ -480,6 +532,57 @@ export type DaemonCommand = } | DaemonSavedSessionListCommand | { id?: string; type: "list_agent_peers"; workerToken: string } + | ({ + id?: string; + type: "create_recoverable_owned_session"; + requestId: string; + expectedSupervisorGeneration: string; + correlationId: string; + mcpOwnerId: string; + recoveryConfig: AgentSessionRuntimeConfig; + sessionPath?: string; + continueRecent?: boolean; + noSession?: boolean; + name?: string; + config?: AgentSessionRuntimeConfig; + runtimeMetadata?: AgentSessionRuntimeMetadata; + } & DaemonClientEnv & + DaemonLaunchEnv & { launchEnv: Record; launchEnvMode: "replace" }) + | { + id?: string; + type: "prepare_recoverable_owned_session_adoption"; + requestId: string; + recoveryHandle: string; + expectedSupervisorGeneration: string; + activeSessionId: string; + sessionId: string; + correlationId: string; + cursor: DaemonEventCursor; + previousMcpOwnerId: string; + mcpOwnerId: string; + recoveryConfig: AgentSessionRuntimeConfig; + launchEnv: Record; + clientId: DaemonClientId; + capabilities: readonly DaemonClientCapability[]; + supportsExtensionUi?: boolean; + telemetryDisabled?: true; + } + | { + id?: string; + type: "commit_recoverable_owned_session_adoption"; + requestId: string; + expectedSupervisorGeneration: string; + recoveryHandle: string; + proof: DaemonRecoverableOwnedSessionAdoptionProof; + } + | { + id?: string; + type: "confirm_recoverable_owned_session_adoption"; + requestId: string; + expectedSupervisorGeneration: string; + recoveryHandle: string; + proof: DaemonRecoverableOwnedSessionAdoptionProof; + } | ({ id?: string; type: "create"; @@ -499,6 +602,7 @@ export type DaemonCommand = id?: string; type: "attach"; activeSessionId: string; + expectedSupervisorGeneration?: string; supportsExtensionUi?: boolean; /** A retry nonce forces a fresh transcript payload generation after a failed transfer. */ snapshotGenerationNonce?: string; @@ -810,6 +914,11 @@ const CLIENT_OWNED_DAEMON_COMMAND = { minProtocol: 7, capability: "client_owned_sessions", } as const; +const RECOVERABLE_OWNED_SESSION_ADOPTION_COMMAND = { + minProtocol: 7, + minSchemaRevision: 30, + capability: "daemon_recoverable_owned_session_adoption_v1", +} as const; const AUTHORITATIVE_OWNED_SESSION_CLEANUP_COMMAND = { minProtocol: 7, minSchemaRevision: 27, @@ -864,6 +973,10 @@ export const DAEMON_COMMAND_COMPATIBILITY = { list: LEGACY_DAEMON_COMMAND, list_saved_sessions: LEGACY_DAEMON_COMMAND, list_agent_peers: AGENT_PEER_LIST_COMMAND, + create_recoverable_owned_session: RECOVERABLE_OWNED_SESSION_ADOPTION_COMMAND, + prepare_recoverable_owned_session_adoption: RECOVERABLE_OWNED_SESSION_ADOPTION_COMMAND, + commit_recoverable_owned_session_adoption: RECOVERABLE_OWNED_SESSION_ADOPTION_COMMAND, + confirm_recoverable_owned_session_adoption: RECOVERABLE_OWNED_SESSION_ADOPTION_COMMAND, create: LEGACY_DAEMON_COMMAND, attach: LEGACY_DAEMON_COMMAND, reattach: LEGACY_DAEMON_COMMAND, @@ -1022,7 +1135,8 @@ export type DaemonErrorInfo = | { code: "session_import_file_not_found"; filePath: string } | { code: "session_already_active"; sessionPath: string; activeSessionId?: string } | { code: "command_result_uncertain"; clientId: DaemonClientId; commandId: DaemonCommandId } - | { code: "owned_session_owner_mismatch" }; + | { code: "owned_session_owner_mismatch" } + | { code: "owned_session_adoption_unavailable" }; export type DaemonSessionClosedReason = "killed" | "shutdown" | "completed" | "replaced" | "update"; export type DaemonClosingReason = "shutdown" | "update"; diff --git a/packages/coding-agent/src/modes/daemon/daemon-supervisor.ts b/packages/coding-agent/src/modes/daemon/daemon-supervisor.ts index c2b2e6583f..36ff901167 100644 --- a/packages/coding-agent/src/modes/daemon/daemon-supervisor.ts +++ b/packages/coding-agent/src/modes/daemon/daemon-supervisor.ts @@ -63,7 +63,11 @@ import type { AgentConnectionHeartbeat } from "../agent-connection/types.js"; import { attachJsonlLineReader, serializeJsonLine } from "../rpc/jsonl.js"; import type { PrivateFrame } from "../session-worker/private-framing.js"; import { createActiveSessionId, type DaemonSocketClient } from "./active-session-state.js"; -import { CommandRecoveryJournal, createCommandIdempotencyKey } from "./command-recovery-journal.js"; +import { + CommandRecoveryJournal, + commandRecoveryJournalAllows, + createCommandIdempotencyKey, +} from "./command-recovery-journal.js"; import { CompactAssistantStreamReconstructor, isCompactAssistantDelta } from "./compact-session-stream.js"; import { DAEMON_CATALOG_ROLE_ENV, DaemonCatalogClient } from "./daemon-catalog-process.js"; import { deserializeDaemonError, serializeDaemonError } from "./daemon-errors.js"; @@ -78,7 +82,6 @@ import { DAEMON_SCHEMA_ID, DAEMON_SCHEMA_REVISION, DAEMON_SNAPSHOT_GENERATION_NONCE_MIN_SCHEMA_REVISION, - DAEMON_SUPERVISOR_SERVER_CAPABILITIES, DAEMON_UPDATE_RESTART_FORMAT_VERSION, type DaemonAttachResult, type DaemonClientCapability, @@ -86,12 +89,18 @@ import { type DaemonCommand, type DaemonOutbound, type DaemonOwnedSessionCleanupResult, + type DaemonRecoverableOwnedSessionAdoptionProof, + type DaemonRecoverableOwnedSessionConfirmResult, + type DaemonRecoverableOwnedSessionCreateResult, + type DaemonRecoverableOwnedSessionPrepareResult, type DaemonResponse, type DaemonUpdateRestartManifest, daemonOutboundForCorrelatedPromptCapability, + daemonSupervisorServerCapabilities, failure, isDaemonCommandEnvelope, isDaemonMutatingCommand, + isDaemonRecoveryRequestId, salvageDaemonCommandId, success, UPDATE_RESTART_DRAIN_COMMANDS, @@ -129,9 +138,12 @@ import { DAEMON_WORKER_ROLE_ENV, DAEMON_WORKER_STARTUP_GATE_COMMIT, DAEMON_WORKER_STARTUP_GATE_FD_ENV, + DAEMON_WORKER_SUPERVISOR_AGENT_DIR_ENV, DAEMON_WORKER_SUPERVISOR_SOCKET_ENV, DAEMON_WORKER_TOKEN_ENV, type DaemonCreateCommand, + type DaemonWorkerAcpMcpOwnerTransferProof, + type DaemonWorkerAcpMcpOwnerTransferRetirement, type DaemonWorkerDescriptor, type DaemonWorkerFrameHeader, type DaemonWorkerLifecycle, @@ -139,8 +151,23 @@ import { durableDaemonWorkerDescriptor, SESSION_LEASE_OWNER_ID_ENV, SESSION_LEASES_ENABLED_ENV, + sanitizeDaemonWorkerBootstrapEnvironment, } from "./daemon-worker-protocol.js"; import { MutationDrainLatch } from "./mutation-drain-latch.js"; +import { appendRecoverableOwnedFrame, reconcileRecoverableOwnedFrames } from "./owned-session-adoption-buffer.js"; +import { + RECOVERABLE_OWNED_CONFIRMATION_RETENTION_MS, + RECOVERABLE_OWNED_DISCONNECTED_RETENTION_MS, + RECOVERABLE_OWNED_PREPARE_TIMEOUT_MS, + RECOVERABLE_OWNED_TERMINAL_RETENTION_MS, + recoverableOwnedRetention, +} from "./owned-session-recovery-retention.js"; +import { + OWNED_SESSION_ADOPTION_UNAVAILABLE, + OwnedSessionAdoptionUnavailableError, + type OwnedSessionRecoveryReceipt, + OwnedSessionRecoveryStore, +} from "./owned-session-recovery-store.js"; import { createRlmLedgerRegistrySeedSource, RlmSpawnLedger } from "./rlm-ledger.js"; import { serializeSavedSessionInfo } from "./saved-session-info.js"; import { @@ -209,6 +236,8 @@ const STALE_RECLAIM_WAIT_MS = 10_000; // supervisor event loop with synchronous subprocess spawns. const LIVENESS_IDENTITY_RECHECK_MS = 500; const OWNED_WORKER_DISCONNECT_GRACE_MS = 30_000; +const OWNED_SESSION_RECOVERY_DIRECTORY = "owned-session-recovery"; +const RECOVERABLE_OWNED_CONNECTED_EXPIRY = Number.MAX_SAFE_INTEGER; const IDLE_EVICTION_MAX_SWEEP_INTERVAL_MS = 5 * 60_000; const IDLE_EVICTION_MIN_SWEEP_INTERVAL_MS = 60_000; const IDLE_EVICTION_DRAIN_TIMEOUT_MS = 5_000; @@ -221,6 +250,10 @@ const DAEMON_COMMAND_TYPES: ReadonlySet = new Set([ "list", "list_agent_peers", "list_saved_sessions", + "create_recoverable_owned_session", + "prepare_recoverable_owned_session_adoption", + "commit_recoverable_owned_session_adoption", + "confirm_recoverable_owned_session_adoption", "create", "attach", "reattach", @@ -326,10 +359,102 @@ const DAEMON_COMMAND_TYPES: ReadonlySet = new Set([ "shutdown", ]); +const RECOVERABLE_OWNED_COMMAND_TYPES: ReadonlySet = new Set([ + "create_recoverable_owned_session", + "prepare_recoverable_owned_session_adoption", + "commit_recoverable_owned_session_adoption", + "confirm_recoverable_owned_session_adoption", +]); + +function isRecoverableOwnedCommandType(command: string): boolean { + return RECOVERABLE_OWNED_COMMAND_TYPES.has(command); +} + +function recoverableOwnedCommandFailure(id: string | undefined, command: string, error: unknown): DaemonResponse { + return isRecoverableOwnedCommandType(command) + ? failure(id, command, OWNED_SESSION_ADOPTION_UNAVAILABLE, { + code: "owned_session_adoption_unavailable", + }) + : failure(id, command, error, serializeDaemonError(error)); +} + +function salvageRecoverableOwnedCommand(line: string): { id?: string; command: string } | undefined { + try { + const parsed = JSON.parse(line) as { id?: unknown; type?: unknown; command?: { type?: unknown } }; + const command = typeof parsed.command?.type === "string" ? parsed.command.type : parsed.type; + if (typeof command !== "string" || !isRecoverableOwnedCommandType(command)) return undefined; + return { ...(typeof parsed.id === "string" ? { id: parsed.id } : {}), command }; + } catch { + return undefined; + } +} + +interface RecoverableOwnedSessionAuthority { + workerId: string; + workerIncarnation: string; + activeSessionId: string; + sessionId: string; + correlationId: string; + mcpOwnerId: string; + recoveryConfig: AgentSessionRuntimeConfig; + launchEnv: Record; + activeDeadline?: number; + proof?: DaemonRecoverableOwnedSessionAdoptionProof; +} + +interface RecoverableOwnedBufferedFrame { + payload: Buffer; + message: DaemonOutbound; + cursor?: { generation: string; sequence: number }; +} + +interface RecoverableOwnedAdoption { + recordId: string; + requestIdDigest: string; + requestDigest: string; + client: DaemonSocketClient; + workerClient: DaemonWorkerClient; + workerIncarnation: string; + workerStopRevision: number; + previousOwnerClientId: string; + activeSessionId: string; + proof: DaemonRecoverableOwnedSessionAdoptionProof; + result?: DaemonRecoverableOwnedSessionPrepareResult; + preparePromise?: Promise; + frames: RecoverableOwnedBufferedFrame[]; + bufferedBytes: number; + error?: Error; + finalRetry?: true; + committing?: true; + timeout: ReturnType; +} + +interface RecoverableOwnedFinalReceipt { + recordId: string; + requestIdDigest: string; + requestDigest: string; + ownerClientId: string; + result: DaemonRecoverableOwnedSessionPrepareResult; + frames: Buffer[]; + mcpTransfer?: { + transactionId: string; + previousOwnerId: string; + nextOwnerId: string; + }; +} + +interface RecoverableOwnedCreateRun { + requestDigest: string; + ownerClientId: string; + promise: Promise; +} + interface ResidentWorker { descriptor: DaemonWorkerDescriptor; descriptorPath: string; client?: DaemonWorkerClient; + /** Private random process incarnation learned only through authenticated worker handshake. */ + workerIncarnation?: string; heartbeatSnapshot?: AgentConnectionHeartbeat[]; heartbeatSnapshotStale?: boolean; summaries: Map; @@ -358,6 +483,10 @@ interface ResidentWorker { promise: Promise; }; ownerCleanupTimer?: ReturnType; + recoveryRecordId?: string; + recoverableAdoption?: RecoverableOwnedAdoption; + recoverableFinal?: RecoverableOwnedFinalReceipt; + recoveryConfirmationTimer?: ReturnType; promotedOwnerClientId?: string; updateRestartPrepareClient?: DaemonWorkerClient; } @@ -388,6 +517,12 @@ interface DaemonSupervisorOptions { socketPath?: string; defaultSessionConfig: AgentSessionRuntimeConfig; descriptorDir?: string; + /** Test seam for platform-specific startup without mutating process.platform. */ + platform?: NodeJS.Platform; + ownedSessionRecoveryStoreFactory?: ( + directory: string, + supervisorGeneration: string, + ) => OwnedSessionRecoveryStore; } interface PersistedSupervisorConfig { @@ -779,6 +914,13 @@ export class DaemonSupervisor { private readonly snapshotCacheParent: string; private snapshotCacheRoot: string; private commandJournal!: CommandRecoveryJournal; + private ownedSessionRecoveryStore!: OwnedSessionRecoveryStore; + private readonly platform: NodeJS.Platform; + private readonly ownedSessionRecoveryStoreFactory: ( + directory: string, + supervisorGeneration: string, + ) => OwnedSessionRecoveryStore; + private readonly recoverableOwnedCreateRuns = new Map(); private readonly streamReconstructor = new CompactAssistantStreamReconstructor(); private readonly compactCatchupInProgress = new Set(); private transcriptPreparations?: WeakMap>; @@ -800,6 +942,10 @@ export class DaemonSupervisor { this.rejectReady = rejectReady; }); void this.ready.catch(() => undefined); + this.platform = options.platform ?? process.platform; + this.ownedSessionRecoveryStoreFactory = + options.ownedSessionRecoveryStoreFactory ?? + ((directory, supervisorGeneration) => new OwnedSessionRecoveryStore(directory, supervisorGeneration)); const agentDir = options.defaultSessionConfig.agentDir; if (!agentDir) { throw new Error("Daemon supervisor config is missing agentDir"); @@ -840,6 +986,12 @@ export class DaemonSupervisor { rmSync(this.snapshotCacheParent, { recursive: true, force: true }); this.snapshotCacheRoot = createSnapshotCacheProcessRoot(this.snapshotCacheParent, this.generation); this.commandJournal = new CommandRecoveryJournal(join(this.descriptorDir, "command-journal.jsonl")); + if (this.platform !== "win32") { + this.ownedSessionRecoveryStore = this.ownedSessionRecoveryStoreFactory( + join(this.descriptorDir, OWNED_SESSION_RECOVERY_DIRECTORY), + this.generation, + ); + } this.loadWorkerDescriptors(); const workersToAdopt = [...this.workers.values()]; @@ -1211,6 +1363,42 @@ export class DaemonSupervisor { if (worker.descriptor.orphanProcessJournalPath) { this.removeWorkerCleanupFile(worker.descriptor.orphanProcessJournalPath); } + if (worker.recoverableAdoption) clearTimeout(worker.recoverableAdoption.timeout); + if (worker.recoveryConfirmationTimer) clearTimeout(worker.recoveryConfirmationTimer); + if (worker.recoveryRecordId) { + const recordId = worker.recoveryRecordId; + let retained = false; + if (!this.shuttingDown) { + try { + const receipt = this.ownedSessionRecoveryStore.get(recordId); + if (receipt?.phase === "final" && receipt.authority.proof) { + const retentionMs = Math.max( + 1, + Math.min( + RECOVERABLE_OWNED_TERMINAL_RETENTION_MS, + receipt.authority.activeDeadline + ? receipt.authority.activeDeadline - Date.now() + : RECOVERABLE_OWNED_TERMINAL_RETENTION_MS, + ), + ); + this.ownedSessionRecoveryStore.markFinal(recordId, Date.now() + retentionMs); + const expiry = setTimeout(() => { + this.retireRecoverableOwnedFinalTransferReceipt(worker); + this.ownedSessionRecoveryStore.remove(recordId); + }, retentionMs); + expiry.unref(); + retained = true; + } + } catch { + // An unreadable stale receipt cannot retain authority. + } + } + if (!retained) { + this.retireRecoverableOwnedFinalTransferReceipt(worker); + this.ownedSessionRecoveryStore.remove(recordId); + } + worker.recoveryRecordId = undefined; + } this.removeWorkerCleanupFile(worker.descriptorPath); } catch (error) { this.log(`Failed to remove worker descriptor ${worker.descriptorPath}: ${String(error)}`); @@ -1292,7 +1480,7 @@ export class DaemonSupervisor { supervisorProcessStartId: this.ownership?.record.processStartId, supervisorSocketPath: this.ownership?.record.socketPath, clientId: client.id, - serverCapabilities: DAEMON_SUPERVISOR_SERVER_CAPABILITIES, + serverCapabilities: daemonSupervisorServerCapabilities(this.platform), }); } }, @@ -1321,6 +1509,11 @@ export class DaemonSupervisor { client.catchupPurposes?.delete(activeSessionId); void this.syncWorkerExtensionUi(activeSessionId); } + for (const worker of this.workers.values()) { + if (worker.recoverableAdoption?.client === client) { + this.rollbackRecoverableOwnedAdoption(worker, worker.recoverableAdoption); + } + } this.scheduleOwnedWorkerCleanupForClient(this.protocolClientId(client)); }; socket.on("close", cleanup); @@ -1415,6 +1608,34 @@ export class DaemonSupervisor { } } + private restoreRecoverableOwnedConnection(client: DaemonSocketClient): void { + const clientId = this.protocolClientId(client); + for (const worker of this.workers.values()) { + if (worker.descriptor.ownerClientId !== clientId || !worker.recoveryRecordId) continue; + const receipt = this.ownedSessionRecoveryStore.get(worker.recoveryRecordId); + if (!receipt || receipt.phase === "final") continue; + const summary = worker.summaries.get(receipt.authority.activeSessionId); + if ( + receipt.authority.workerId !== worker.descriptor.workerId || + worker.workerIncarnation !== receipt.authority.workerIncarnation || + !worker.client?.matchesAuthenticatedIncarnation(receipt.authority.workerIncarnation) || + !summary || + summary.sessionId !== receipt.authority.sessionId || + worker.recoverableAdoption !== undefined + ) { + throw new OwnedSessionAdoptionUnavailableError(); + } + if (receipt.phase !== "connected") { + receipt.authority.activeDeadline = undefined; + this.ownedSessionRecoveryStore.markConnected( + receipt.recordId, + this.recoverableAuthorityDigest(receipt.authority), + RECOVERABLE_OWNED_CONNECTED_EXPIRY, + ); + } + } + } + private scheduleOwnedWorkerCleanupForClient(clientId: string): void { if ([...this.clients].some((client) => this.protocolClientId(client) === clientId)) { return; @@ -1427,6 +1648,10 @@ export class DaemonSupervisor { } private scheduleOwnedWorkerCleanup(worker: ResidentWorker): void { + if (worker.recoveryRecordId) { + this.scheduleRecoverableOwnedWorkerCleanup(worker); + return; + } const ownerClientId = worker.descriptor.ownerClientId; if ( !ownerClientId || @@ -1451,6 +1676,77 @@ export class DaemonSupervisor { worker.ownerCleanupTimer.unref(); } + private scheduleRecoverableOwnedWorkerCleanup(worker: ResidentWorker, force = false): void { + const ownerClientId = worker.descriptor.ownerClientId; + const recordId = worker.recoveryRecordId; + if ( + !ownerClientId || + !recordId || + worker.recoverableAdoption || + [...this.clients].some((client) => this.protocolClientId(client) === ownerClientId) + ) { + return; + } + if (worker.ownerCleanupTimer) { + if (!force) return; + clearTimeout(worker.ownerCleanupTimer); + worker.ownerCleanupTimer = undefined; + } + let receipt: OwnedSessionRecoveryReceipt | undefined; + try { + receipt = this.ownedSessionRecoveryStore.get(recordId); + } catch { + receipt = undefined; + } + if (!receipt) { + void this.stopWorker(worker, true).catch(() => undefined); + return; + } + const summary = worker.summaries.get(receipt.authority.activeSessionId); + const cached = worker.snapshotCache.get(receipt.authority.activeSessionId)?.snapshot.promptLifecycles; + const lifecycle = cached?.records.find((entry) => entry.correlationId === receipt.authority.correlationId); + const expired = cached?.expired.some((entry) => entry.correlationId === receipt.authority.correlationId) === true; + const busy = summary ? isSessionSummaryBusy(summary) : false; + const terminal = expired || (lifecycle ? ["completed", "cancelled", "failed"].includes(lifecycle.phase) : false); + const now = Date.now(); + const retention = recoverableOwnedRetention({ + now, + busy, + hasLifecycle: lifecycle !== undefined || expired, + terminal, + ...(receipt.authority.activeDeadline !== undefined + ? { activeDeadline: receipt.authority.activeDeadline } + : {}), + }); + receipt.authority.activeDeadline = retention.activeDeadline; + const retentionMs = retention.retentionMs; + try { + this.ownedSessionRecoveryStore.markDisconnected(recordId, now + Math.max(1, retentionMs)); + } catch { + void this.stopWorker(worker, true).catch(() => undefined); + return; + } + worker.ownerCleanupTimer = setTimeout(() => { + worker.ownerCleanupTimer = undefined; + if ( + worker.recoveryRecordId !== recordId || + worker.descriptor.ownerClientId !== ownerClientId || + [...this.clients].some((client) => this.protocolClientId(client) === ownerClientId) || + this.workers.get(worker.descriptor.workerId) !== worker + ) { + return; + } + this.ownedSessionRecoveryStore.remove(recordId); + worker.recoveryRecordId = undefined; + void this.stopWorker(worker, true).catch((error) => + this.log( + `Could not clean up recoverable client-owned worker ${worker.descriptor.workerId}: ${String(error)}`, + ), + ); + }, retentionMs); + worker.ownerCleanupTimer.unref(); + } + private promptAdmissionKey(activeSessionId: string, publicAdmissionId: string): string { return `${activeSessionId}\0${publicAdmissionId}`; } @@ -1610,14 +1906,27 @@ export class DaemonSupervisor { try { preParsed = this.parseCommandAndRegisterPromptAdmission(client, line); } catch (error) { - this.write(client, failure(salvageDaemonCommandId(line), "parse", error)); + const recoverable = salvageRecoverableOwnedCommand(line); + this.write( + client, + recoverable + ? recoverableOwnedCommandFailure( + recoverable.id ?? salvageDaemonCommandId(line), + recoverable.command, + error, + ) + : failure(salvageDaemonCommandId(line), "parse", error), + ); return; } const command = preParsed.command; const parsedAdmission = preParsed.admission; const correlatedOrder = preParsed.correlatedOrder; if (command.type === "cancel_prompt_admission" && this.updateRestartPhase !== undefined) { - this.write(client, failure(command.id, command.type, "Daemon is preparing an update restart")); + this.write( + client, + recoverableOwnedCommandFailure(command.id, command.type, "Daemon is preparing an update restart"), + ); return; } const cancellationAdmission = @@ -1633,7 +1942,7 @@ export class DaemonSupervisor { } catch (error) { if (parsedAdmission) this.deletePromptAdmission(parsedAdmission); correlatedOrder?.release(); - this.write(client, failure(command.id, command.type, error)); + this.write(client, recoverableOwnedCommandFailure(command.id, command.type, error)); return; } const envelopeClientId = preParsed.envelopeClientId; @@ -1641,12 +1950,51 @@ export class DaemonSupervisor { this.protocolClientIds.set(client, envelopeClientId); client.id = envelopeClientId; } - if (this.clients?.has(client)) { - this.cancelOwnedWorkerCleanup(client.id); + const duplicatePhysicalClient = [...this.clients].some( + (other) => + other !== client && + this.protocolClientId(other) === this.protocolClientId(client) && + !other.socket.destroyed, + ); + if (duplicatePhysicalClient) { + correlatedOrder?.release(); + this.write( + client, + recoverableOwnedCommandFailure(command.id, command.type, "Client identity is already connected"), + ); + return; + } + const conflictingAdoptions = [...this.workers.values()].filter( + (worker) => + worker.recoverableAdoption && + worker.recoverableAdoption.client !== client && + worker.recoverableAdoption.previousOwnerClientId === this.protocolClientId(client), + ); + if (conflictingAdoptions.some((worker) => worker.recoverableAdoption?.committing)) { + correlatedOrder?.release(); + this.write( + client, + recoverableOwnedCommandFailure(command.id, command.type, OWNED_SESSION_ADOPTION_UNAVAILABLE), + ); + return; + } + for (const worker of conflictingAdoptions) { + if (worker.recoverableAdoption) this.rollbackRecoverableOwnedAdoption(worker, worker.recoverableAdoption); } + try { + this.restoreRecoverableOwnedConnection(client); + } catch (error) { + correlatedOrder?.release(); + this.write(client, recoverableOwnedCommandFailure(command.id, command.type, error)); + return; + } + if (this.clients.has(client)) this.cancelOwnedWorkerCleanup(client.id); if (!DAEMON_COMMAND_TYPES.has(command.type)) { correlatedOrder?.release(); - this.write(client, failure(command.id, command.type, `Unknown daemon command: ${command.type}`)); + this.write( + client, + recoverableOwnedCommandFailure(command.id, command.type, `Unknown daemon command: ${command.type}`), + ); return; } if ( @@ -1670,13 +2018,15 @@ export class DaemonSupervisor { } catch (error) { if (parsedAdmission) this.deletePromptAdmission(parsedAdmission); correlatedOrder?.release(); - this.write(client, failure(command.id, command.type, error)); + this.write(client, recoverableOwnedCommandFailure(command.id, command.type, error)); return; } const mutation = isDaemonMutatingCommand(command); const journalIdentity = - envelopeClientId && command.id && mutation ? { clientId: envelopeClientId, commandId: command.id } : undefined; + envelopeClientId && command.id && mutation && commandRecoveryJournalAllows(command.type) + ? { clientId: envelopeClientId, commandId: command.id } + : undefined; const correlatedRequestIdentity = command.type === "submit_correlated_prompt" || command.type === "cancel_correlated_prompt" ? correlatedCommandRequestIdentity(command) @@ -1689,7 +2039,11 @@ export class DaemonSupervisor { correlatedOrder?.release(); this.write( client, - failure(command.id, command.type, "Correlated command id was reused with a different request"), + recoverableOwnedCommandFailure( + command.id, + command.type, + "Correlated command id was reused with a different request", + ), ); return; } @@ -1720,7 +2074,10 @@ export class DaemonSupervisor { if (restartRejected && mutation) { if (parsedAdmission) this.deletePromptAdmission(parsedAdmission); correlatedOrder?.release(); - this.write(client, failure(command.id, command.type, "Daemon is preparing an update restart")); + this.write( + client, + recoverableOwnedCommandFailure(command.id, command.type, "Daemon is preparing an update restart"), + ); return; } if (journalIdentity) { @@ -1735,7 +2092,11 @@ export class DaemonSupervisor { correlatedOrder?.release(); this.write( client, - failure(command.id, command.type, "Correlated command id was reused with a different request"), + recoverableOwnedCommandFailure( + command.id, + command.type, + "Correlated command id was reused with a different request", + ), ); return; } @@ -1773,7 +2134,11 @@ export class DaemonSupervisor { if (inFlight.requestIdentity !== requestIdentity) { this.write( client, - failure(command.id, command.type, "Correlated command id was reused with a different request"), + recoverableOwnedCommandFailure( + command.id, + command.type, + "Correlated command id was reused with a different request", + ), ); return; } @@ -1795,7 +2160,8 @@ export class DaemonSupervisor { // Attach is intentionally read-only and is not fence-gated. If eviction wins // the race, attach fails cleanly with "Session worker is not connected" and // the client retries through the saved-session path instead of mutating state. - if (mutation) this.mutationDrain.begin(); + const tracksMutation = mutation && command.type !== "prepare_recoverable_owned_session_adoption"; + if (tracksMutation) this.mutationDrain.begin(); try { await correlatedOrder?.previous; const response = await this.handleCommand(client, command, cancellationAdmission, correlatedOrder); @@ -1812,13 +2178,13 @@ export class DaemonSupervisor { } } catch (error) { this.log(`Supervisor command ${command.type} failed: ${error instanceof Error ? error.stack : String(error)}`); - let response = failure(command.id, command.type, error, serializeDaemonError(error)); + let response = recoverableOwnedCommandFailure(command.id, command.type, error); if (journalIdentity && !isSupervisorGenerationStale(error)) { try { await this.assertCurrentOwnership(); this.commandJournal.recordResult(journalIdentity.clientId, journalIdentity.commandId, response); } catch (ownershipError) { - response = failure(command.id, command.type, ownershipError, serializeDaemonError(ownershipError)); + response = recoverableOwnedCommandFailure(command.id, command.type, ownershipError); } } if (correlatedJournalRun) { @@ -1831,14 +2197,14 @@ export class DaemonSupervisor { if (correlatedJournalRun) { if (!correlatedJournalRun.settled) { correlatedJournalRun.resolve( - failure(command.id, command.type, "Correlated command ended without a result"), + recoverableOwnedCommandFailure(command.id, command.type, "Correlated command ended without a result"), ); } if (this.correlatedJournalRuns.get(correlatedJournalRun.key) === correlatedJournalRun) { this.correlatedJournalRuns.delete(correlatedJournalRun.key); } } - if (mutation) this.mutationDrain.end(); + if (tracksMutation) this.mutationDrain.end(); } } @@ -1905,6 +2271,46 @@ export class DaemonSupervisor { } case "list_saved_sessions": return this.handleSavedSessionList(client, command); + case "create_recoverable_owned_session": { + try { + return success(command.id, command.type, await this.createRecoverableOwnedSession(client, command)); + } catch { + throw new OwnedSessionAdoptionUnavailableError(); + } + } + case "prepare_recoverable_owned_session_adoption": { + try { + return success( + command.id, + command.type, + await this.prepareRecoverableOwnedSessionAdoption(client, command), + ); + } catch { + throw new OwnedSessionAdoptionUnavailableError(); + } + } + case "commit_recoverable_owned_session_adoption": { + try { + return success( + command.id, + command.type, + await this.commitRecoverableOwnedSessionAdoption(client, command), + ); + } catch { + throw new OwnedSessionAdoptionUnavailableError(); + } + } + case "confirm_recoverable_owned_session_adoption": { + try { + return success( + command.id, + command.type, + await this.confirmRecoverableOwnedSessionAdoption(client, command), + ); + } catch { + throw new OwnedSessionAdoptionUnavailableError(); + } + } case "create": { const createCommand = prepareCallerOwnedCreateEnvironment(command); const worker = await this.createOrReuseWorker(this.protocolClientId(client), createCommand); @@ -1934,6 +2340,12 @@ export class DaemonSupervisor { return success(command.id, "create", this.publicSummary(worker, summary)); } case "attach": { + if ( + command.expectedSupervisorGeneration !== undefined && + command.expectedSupervisorGeneration !== this.generation + ) { + throw new OwnedSessionAdoptionUnavailableError(); + } const attachmentEpoch = this.advanceAttachmentEpoch(client, command.activeSessionId); const requestedCapabilities = normalizeCapabilities(command.capabilities, command.supportsExtensionUi); let releaseSnapshotReservation = requestedCapabilities.has("chunked_snapshot") @@ -2154,7 +2566,10 @@ export class DaemonSupervisor { return success(command.id, command.type, this.getOwnedSessionCleanup(command.activeSessionId)); case "complete_owned_session": { const match = await this.findWorker(command.activeSessionId); - if (match.worker.descriptor.ownerClientId !== this.protocolClientId(client)) { + if ( + match.worker.recoverableAdoption || + match.worker.descriptor.ownerClientId !== this.protocolClientId(client) + ) { return failure(command.id, command.type, "Session is not owned by this client", { code: "owned_session_owner_mismatch", }); @@ -2200,7 +2615,7 @@ export class DaemonSupervisor { setImmediate(() => void this.shutdown(0, false, true, false, "update")); return success(command.id, command.type); case "shutdown": - setImmediate(() => void this.shutdown(0, true, false, command.force === true, "shutdown")); + void this.shutdown(0, true, false, command.force === true, "shutdown"); return success(command.id, "shutdown"); case "prepare_update_restart": { const manifest = await this.prepareUpdateRestart(); @@ -2244,7 +2659,7 @@ export class DaemonSupervisor { ) .map((worker) => this.forwardToWorker(worker, command, 5000).catch((error: unknown) => - failure(command.id, command.type, error, serializeDaemonError(error)), + recoverableOwnedCommandFailure(command.id, command.type, error), ), ), ); @@ -2273,7 +2688,7 @@ export class DaemonSupervisor { workers.map(async (worker) => { if (worker.client && worker.descriptor.lifecycle === "ready") { const response = await this.forwardToWorker(worker, command, 5000).catch((error: unknown) => - failure(command.id, command.type, error, serializeDaemonError(error)), + recoverableOwnedCommandFailure(command.id, command.type, error), ); if (response.success) { const snapshot = heartbeatsFromResponse(response); @@ -2292,7 +2707,7 @@ export class DaemonSupervisor { const state = worker.descriptor.lifecycle === "ready" ? "disconnected" : worker.descriptor.lifecycle; const error = new Error(`Cannot list heartbeats while session worker is ${state}`); - return { response: failure(command.id, command.type, error, serializeDaemonError(error)) }; + return { response: recoverableOwnedCommandFailure(command.id, command.type, error) }; }), ); const failed = snapshots.find((snapshot) => snapshot.response)?.response; @@ -2517,141 +2932,1333 @@ export class DaemonSupervisor { ); throwIfAdmissionCancelled(admission); if ( - (command.type === "submit_correlated_prompt" || - command.type === "cancel_correlated_prompt" || - command.type === "get_prompt_lifecycles") && - match.summary.sessionId !== command.sessionId + (command.type === "submit_correlated_prompt" || + command.type === "cancel_correlated_prompt" || + command.type === "get_prompt_lifecycles") && + match.summary.sessionId !== command.sessionId + ) { + throw new Error("Correlated prompt targeted a stale session generation"); + } + const resolvedCommand = { + ...command, + activeSessionId: match.summary.activeSessionId ?? match.summary.id, + ...(admission ? { admissionId: admission.workerAdmissionId } : {}), + } as DaemonCommand; + if (admission) { + admission.worker = match.worker; + admission.workerActiveSessionId = match.summary.activeSessionId ?? match.summary.id; + } + const isRootKill = + command.type === "kill" && + (match.summary.activeSessionId ?? match.summary.id) === match.worker.descriptor.rootActiveSessionId; + if (!isRootKill) { + const forward = async () => { + const workerResponse = this.forwardToWorker(match.worker, resolvedCommand); + correlatedOrder?.release(); + const response = await workerResponse; + if (admission && response.success) admission.status = "owned"; + return response; + }; + if (command.type === "rename" || command.type === "set_session_name") { + const reservation = this.summaryNameReservationInput(match.summary, command.name.trim()); + return await this.withSessionNameReservation(reservation, async () => { + await this.assertSupervisorSessionNameAvailable(match.summary, reservation.name); + return forward(); + }); + } + return await forward(); + } + try { + this.persistWorkerStopTombstone(match.worker, true); + } catch (error) { + this.scheduleWorkerStopFinalization(match.worker); + throw error; + } + const releaseStopOwnership = this.acquireWorkerStopOwnership(match.worker); + let response: DaemonResponse; + try { + response = await this.forwardToWorker(match.worker, resolvedCommand); + } finally { + try { + await this.stopWorker(match.worker, true, false, true); + } finally { + releaseStopOwnership(); + } + } + return response; + } finally { + if (admission) this.deletePromptAdmission(admission); + } + } + + private async handleList( + client: DaemonSocketClient, + command: Extract, + ): Promise { + await Promise.all( + [...this.workers.values()] + .filter((worker) => !this.isWorkerStopping(worker)) + .map((worker) => this.refreshWorkerSummaries(worker).catch(() => undefined)), + ); + const clientOwnedWorkers = [...this.workers.values()].filter((worker) => !this.isVisibleWorker(worker)); + // Stopping workers stay listed (with an honest workerState) because this + // list also feeds busy-daemon safety checks in daemon-launch. + const active = [...this.workers.values()] + .filter( + (worker) => + this.isVisibleWorker(worker) || + (command.includeClientOwned === true && this.isWorkerAccessibleToClient(client, worker)), + ) + .flatMap((worker) => [...worker.summaries.values()].map((summary) => this.publicSummary(worker, summary))); + const busyClientOwnedSessionCount = clientOwnedWorkers + .flatMap((worker) => [...worker.summaries.values()]) + .filter(isSessionSummaryBusy).length; + const data = { + sessions: active, + ...(command.includeClientOwned ? { busyClientOwnedSessionCount } : {}), + }; + if (!command.all) { + return success(command.id, "list", data); + } + const sessionDir = command.sessionDir ?? this.defaultSessionConfig.sessionDir; + const saved = await this.catalog.list(command.cwd ? resolve(command.cwd) : undefined, sessionDir); + return success(command.id, "list", { ...data, sessions: mergeSessionLists(active, saved) }); + } + + private async handleSavedSessionList( + client: DaemonSocketClient, + command: Extract, + ): Promise { + let cwd: string; + let sessionDir: string | undefined; + let activeSessionId: string | undefined; + if ("activeSessionId" in command) { + const match = await this.findWorkerForClient(client, command.activeSessionId); + cwd = match.summary.cwd; + sessionDir = this.defaultSessionConfig.sessionDir; + activeSessionId = match.summary.activeSessionId ?? match.summary.id; + } else { + cwd = resolve(command.cwd); + sessionDir = command.sessionDir; + } + const callbacks = command.id + ? { + onProgress: (loaded: number, total: number) => + this.write(client, { + id: command.id, + type: "session_list_progress", + command: "list_saved_sessions", + ...(activeSessionId ? { activeSessionId } : {}), + loaded, + total, + }), + onSession: (session: SessionInfo) => + this.write(client, { + id: command.id, + type: "session_list_item", + command: "list_saved_sessions", + ...(activeSessionId ? { activeSessionId } : {}), + session: serializeSavedSessionInfo(session), + }), + } + : undefined; + const saved = await this.catalog.list(command.scope === "current" ? cwd : undefined, sessionDir, callbacks); + return success(command.id, "list_saved_sessions", { sessions: saved.map(serializeSavedSessionInfo) }); + } + + private recoverableAuthorityDigest( + authority: Pick< + RecoverableOwnedSessionAuthority, + | "workerIncarnation" + | "activeSessionId" + | "sessionId" + | "correlationId" + | "mcpOwnerId" + | "recoveryConfig" + | "launchEnv" + >, + ): string { + return this.ownedSessionRecoveryStore.digestAuthority({ + workerIncarnation: authority.workerIncarnation, + activeSessionId: authority.activeSessionId, + sessionId: authority.sessionId, + correlationId: authority.correlationId, + mcpOwnerId: authority.mcpOwnerId, + recoveryConfig: authority.recoveryConfig, + ...(authority.launchEnv ? { launchEnv: authority.launchEnv } : {}), + }); + } + + private assertRecoverableOwnedSessionPlatform(): void { + if (this.platform === "win32") throw new OwnedSessionAdoptionUnavailableError(); + } + + private async createRecoverableOwnedSession( + client: DaemonSocketClient, + command: Extract, + ): Promise { + this.assertRecoverableOwnedSessionPlatform(); + if ( + command.expectedSupervisorGeneration !== this.generation || + !isDaemonRecoveryRequestId(command.requestId) || + !command.correlationId || + command.correlationId.length > 128 || + !command.mcpOwnerId || + command.mcpOwnerId.length > 256 || + command.launchEnvMode !== "replace" || + command.launchEnv === undefined || + command.config === undefined || + !isDeepStrictEqual(command.config, command.recoveryConfig) + ) { + throw new OwnedSessionAdoptionUnavailableError(); + } + const exactLaunchEnv = cloneCallerOwnedSessionLaunchEnv(command.launchEnv) as Record; + const createRequest = { + type: command.type, + requestId: command.requestId, + expectedSupervisorGeneration: command.expectedSupervisorGeneration, + correlationId: command.correlationId, + mcpOwnerId: command.mcpOwnerId, + recoveryConfig: command.recoveryConfig, + ...(command.sessionPath !== undefined ? { sessionPath: command.sessionPath } : {}), + ...(command.continueRecent !== undefined ? { continueRecent: command.continueRecent } : {}), + ...(command.noSession !== undefined ? { noSession: command.noSession } : {}), + ...(command.name !== undefined ? { name: command.name } : {}), + ...(command.runtimeMetadata !== undefined ? { runtimeMetadata: command.runtimeMetadata } : {}), + ...(command.env !== undefined ? { env: command.env } : {}), + launchEnv: exactLaunchEnv, + launchEnvMode: command.launchEnvMode, + }; + const requestIdDigest = this.ownedSessionRecoveryStore.digestRequest({ + type: command.type, + requestId: command.requestId, + }); + const requestDigest = this.ownedSessionRecoveryStore.digestRequest(createRequest); + const existing = this.ownedSessionRecoveryStore.getByCreateRequest(requestIdDigest, requestDigest); + if (existing) { + const worker = this.workers.get(existing.authority.workerId); + const summary = worker?.summaries.get(existing.authority.activeSessionId); + if ( + !worker || + !summary || + worker.recoveryRecordId !== existing.recordId || + worker.workerIncarnation !== existing.authority.workerIncarnation || + !worker.client?.matchesAuthenticatedIncarnation(existing.authority.workerIncarnation) + ) { + throw new OwnedSessionAdoptionUnavailableError(); + } + const retryOwnerClientId = this.protocolClientId(client); + if (worker.descriptor.ownerClientId !== retryOwnerClientId) { + if ( + worker.descriptor.ownerClientId === undefined || + [...this.clients].some( + (other) => + other !== client && + this.protocolClientId(other) === worker.descriptor.ownerClientId && + !other.socket.destroyed, + ) + ) { + throw new OwnedSessionAdoptionUnavailableError(); + } + worker.descriptor.ownerClientId = retryOwnerClientId; + this.persistWorker(worker); + } + return { + state: this.publicSummary(worker, summary), + recoveryHandle: existing.recoveryHandle, + supervisorGeneration: this.generation, + ownershipGeneration: existing.ownershipGeneration, + }; + } + const running = this.recoverableOwnedCreateRuns.get(requestIdDigest); + if (running) { + if (running.requestDigest !== requestDigest || running.ownerClientId !== this.protocolClientId(client)) + throw new OwnedSessionAdoptionUnavailableError(); + return running.promise; + } + const promise = (async (): Promise => { + const createCommand = prepareCallerOwnedCreateEnvironment({ + type: "create", + id: command.id, + ...(command.sessionPath !== undefined ? { sessionPath: command.sessionPath } : {}), + ...(command.continueRecent !== undefined ? { continueRecent: command.continueRecent } : {}), + ...(command.noSession !== undefined ? { noSession: command.noSession } : {}), + ...(command.name !== undefined ? { name: command.name } : {}), + config: command.recoveryConfig, + ...(command.runtimeMetadata !== undefined ? { runtimeMetadata: command.runtimeMetadata } : {}), + ...(command.env !== undefined ? { env: command.env } : {}), + launchEnv: exactLaunchEnv, + launchEnvMode: "replace", + lifecycle: "client_owned", + }); + let worker: ResidentWorker | undefined; + let launchedWorker: ResidentWorker | undefined; + try { + worker = await this.createOrReuseWorker( + this.protocolClientId(client), + createCommand, + (launched) => (launchedWorker = launched), + ); + if (worker.recoveryRecordId !== undefined) throw new OwnedSessionAdoptionUnavailableError(); + const summary = worker.summaries.get(worker.descriptor.rootActiveSessionId); + if (!summary) throw new OwnedSessionAdoptionUnavailableError(); + const activeSessionId = summary.activeSessionId ?? summary.id; + if ( + !worker.workerIncarnation || + !worker.client?.matchesAuthenticatedIncarnation(worker.workerIncarnation) + ) { + throw new OwnedSessionAdoptionUnavailableError(); + } + const authorityBase = { + workerIncarnation: worker.workerIncarnation, + activeSessionId, + sessionId: summary.sessionId, + correlationId: command.correlationId, + mcpOwnerId: command.mcpOwnerId, + recoveryConfig: command.recoveryConfig, + launchEnv: exactLaunchEnv, + }; + const authority: RecoverableOwnedSessionAuthority = { + workerId: worker.descriptor.workerId, + ...authorityBase, + }; + const receipt = this.ownedSessionRecoveryStore.create({ + requestIdDigest, + requestDigest, + authorityDigest: this.recoverableAuthorityDigest(authorityBase), + authority, + expiresAt: RECOVERABLE_OWNED_CONNECTED_EXPIRY, + }); + worker.recoveryRecordId = receipt.recordId; + if (worker.ownerCleanupTimer) { + clearTimeout(worker.ownerCleanupTimer); + worker.ownerCleanupTimer = undefined; + } + this.scheduleOwnedWorkerCleanup(worker); + return { + state: this.publicSummary(worker, summary), + recoveryHandle: receipt.recoveryHandle, + supervisorGeneration: this.generation, + ownershipGeneration: receipt.ownershipGeneration, + }; + } catch (error) { + if (worker?.recoveryRecordId) { + try { + this.ownedSessionRecoveryStore.remove(worker.recoveryRecordId); + } catch { + // Exact worker containment below remains authoritative when receipt cleanup fails. + } + worker.recoveryRecordId = undefined; + } + if ( + launchedWorker && + worker === launchedWorker && + this.workers.get(worker.descriptor.workerId) === worker + ) { + await this.stopWorker(worker, true, true).catch((cleanupError) => + this.reportCleanupFailure(`recoverable worker create ${worker!.descriptor.workerId}`, cleanupError), + ); + } else if (worker) { + this.scheduleOwnedWorkerCleanup(worker); + } + throw error; + } + })(); + this.recoverableOwnedCreateRuns.set(requestIdDigest, { + requestDigest, + ownerClientId: this.protocolClientId(client), + promise, + }); + try { + return await promise; + } finally { + if (this.recoverableOwnedCreateRuns.get(requestIdDigest)?.promise === promise) { + this.recoverableOwnedCreateRuns.delete(requestIdDigest); + } + } + } + + private async prepareRecoverableOwnedSessionAdoption( + client: DaemonSocketClient, + command: Extract, + ): Promise { + this.assertRecoverableOwnedSessionPlatform(); + if ( + !isDaemonRecoveryRequestId(command.requestId) || + !command.capabilities.includes("event_sequence") || + !command.capabilities.includes("attach_snapshot") || + !command.capabilities.includes("correlated_prompt_lifecycle_v1") || + !command.capabilities.includes(CALLER_OWNED_SESSION_ENVIRONMENT_CLEANUP_FEATURE) || + !command.activeSessionId || + !command.sessionId || + !command.correlationId || + !command.previousMcpOwnerId || + !command.mcpOwnerId || + command.expectedSupervisorGeneration !== this.generation || + command.cursor.generation === "" || + !Number.isSafeInteger(command.cursor.sequence) || + command.cursor.sequence < 0 + ) { + throw new OwnedSessionAdoptionUnavailableError(); + } + const launchEnv = cloneCallerOwnedSessionLaunchEnv(command.launchEnv) as Record; + const candidateReceipt = this.ownedSessionRecoveryStore.getByHandle(command.recoveryHandle); + const candidateWorker = this.workers.get(candidateReceipt.authority.workerId); + if ( + !candidateWorker || + candidateWorker.recoveryRecordId !== candidateReceipt.recordId || + candidateWorker.workerIncarnation !== candidateReceipt.authority.workerIncarnation || + !candidateWorker.client?.matchesAuthenticatedIncarnation(candidateReceipt.authority.workerIncarnation) + ) { + throw new OwnedSessionAdoptionUnavailableError(); + } + const authorityBase = { + workerIncarnation: candidateReceipt.authority.workerIncarnation, + activeSessionId: command.activeSessionId, + sessionId: command.sessionId, + correlationId: command.correlationId, + mcpOwnerId: command.previousMcpOwnerId, + recoveryConfig: command.recoveryConfig, + launchEnv, + }; + const requestIdDigest = this.ownedSessionRecoveryStore.digestRequest({ + type: command.type, + requestId: command.requestId, + }); + const requestDigest = this.ownedSessionRecoveryStore.digestRequest({ + type: command.type, + requestId: command.requestId, + expectedSupervisorGeneration: command.expectedSupervisorGeneration, + activeSessionId: command.activeSessionId, + sessionId: command.sessionId, + correlationId: command.correlationId, + cursor: command.cursor, + previousMcpOwnerId: command.previousMcpOwnerId, + mcpOwnerId: command.mcpOwnerId, + recoveryConfig: command.recoveryConfig, + launchEnv, + }); + const reserved = candidateWorker.recoverableAdoption; + if (reserved) { + if ( + reserved.recordId !== candidateReceipt.recordId || + reserved.requestIdDigest !== requestIdDigest || + reserved.requestDigest !== requestDigest || + reserved.client !== client || + reserved.activeSessionId !== command.activeSessionId || + !reserved.preparePromise + ) { + throw new OwnedSessionAdoptionUnavailableError(); + } + return reserved.preparePromise; + } + const prepareDeadline = Date.now() + RECOVERABLE_OWNED_PREPARE_TIMEOUT_MS; + const receipt = this.ownedSessionRecoveryStore.beginAdoption({ + recoveryHandle: command.recoveryHandle, + requestIdDigest, + requestDigest, + authorityDigest: this.recoverableAuthorityDigest(authorityBase), + expectedSupervisorGeneration: command.expectedSupervisorGeneration, + expiresAt: prepareDeadline, + }); + const worker = this.workers.get(receipt.authority.workerId); + if ( + !worker || + worker !== candidateWorker || + worker.recoveryRecordId !== receipt.recordId || + worker.workerIncarnation !== receipt.authority.workerIncarnation || + !worker.client?.matchesAuthenticatedIncarnation(receipt.authority.workerIncarnation) || + this.isWorkerStopping(worker) + ) { + throw new OwnedSessionAdoptionUnavailableError(); + } + if (receipt.phase === "final") { + return this.prepareRecoverableOwnedFinalRetry( + worker, + client, + command, + receipt, + requestIdDigest, + requestDigest, + prepareDeadline, + worker.recoverableFinal, + ); + } + if ( + worker.descriptor.ownerClientId === undefined || + [...this.clients].some( + (other) => + other !== client && + this.protocolClientId(other) === worker.descriptor.ownerClientId && + !other.socket.destroyed, + ) + ) { + this.ownedSessionRecoveryStore.markDisconnected( + receipt.recordId, + Date.now() + RECOVERABLE_OWNED_DISCONNECTED_RETENTION_MS, + ); + throw new OwnedSessionAdoptionUnavailableError(); + } + if (worker.ownerCleanupTimer) { + clearTimeout(worker.ownerCleanupTimer); + worker.ownerCleanupTimer = undefined; + } + const workerClient = worker.client; + if (!workerClient) throw new OwnedSessionAdoptionUnavailableError(); + const workerStopRevision = worker.stopRevision; + const timeout = setTimeout( + () => this.expireRecoverableOwnedAdoption(worker, receipt.recordId), + Math.max(1, prepareDeadline - Date.now()), + ); + timeout.unref(); + const adoption: RecoverableOwnedAdoption = { + recordId: receipt.recordId, + requestIdDigest, + requestDigest, + client, + workerClient, + workerIncarnation: receipt.authority.workerIncarnation, + workerStopRevision, + previousOwnerClientId: worker.descriptor.ownerClientId, + activeSessionId: command.activeSessionId, + proof: undefined as unknown as DaemonRecoverableOwnedSessionAdoptionProof, + frames: [], + bufferedBytes: 0, + timeout, + }; + worker.recoverableAdoption = adoption; + const preparePromise = (async () => { + this.mutationDrain.begin(); + try { + await this.mutationDrain.waitForDrain( + 1, + AbortSignal.timeout(Math.max(1, prepareDeadline - Date.now())), + "Timed out draining mutations for recoverable owned-session adoption", + ); + if ( + Date.now() >= prepareDeadline || + worker.recoverableAdoption !== adoption || + worker.client !== workerClient || + worker.workerIncarnation !== receipt.authority.workerIncarnation || + !workerClient.matchesAuthenticatedIncarnation(receipt.authority.workerIncarnation) || + worker.stopRevision !== workerStopRevision || + worker.descriptor.ownerClientId === undefined || + [...this.clients].some( + (other) => + other !== client && + this.protocolClientId(other) === worker.descriptor.ownerClientId && + !other.socket.destroyed, + ) + ) { + throw new OwnedSessionAdoptionUnavailableError(); + } + return await this.prepareRecoverableOwnedSnapshot(worker, adoption, receipt, command, prepareDeadline); + } catch (error) { + this.rollbackRecoverableOwnedAdoption(worker, adoption); + throw error; + } finally { + this.mutationDrain.end(); + } + })(); + adoption.preparePromise = preparePromise; + return preparePromise; + } + + private async prepareRecoverableOwnedFinalRetry( + worker: ResidentWorker, + client: DaemonSocketClient, + command: Extract, + receipt: OwnedSessionRecoveryReceipt, + requestIdDigest: string, + requestDigest: string, + prepareDeadline: number, + finalReceipt: RecoverableOwnedFinalReceipt | undefined, + ): Promise { + if ( + !finalReceipt || + finalReceipt.recordId !== receipt.recordId || + finalReceipt.requestIdDigest !== requestIdDigest || + finalReceipt.requestDigest !== requestDigest || + finalReceipt.result.recoveryHandle !== receipt.recoveryHandle || + worker.descriptor.ownerClientId !== finalReceipt.ownerClientId || + receipt.authority.mcpOwnerId !== command.mcpOwnerId || + receipt.authority.proof === undefined || + worker.recoverableAdoption !== undefined || + [...this.clients].some( + (other) => + other !== client && + this.protocolClientId(other) === finalReceipt.ownerClientId && + !other.socket.destroyed, + ) + ) { + throw new OwnedSessionAdoptionUnavailableError(); + } + if (worker.ownerCleanupTimer) { + clearTimeout(worker.ownerCleanupTimer); + worker.ownerCleanupTimer = undefined; + } + const workerClient = worker.client; + const workerIncarnation = worker.workerIncarnation; + if (!workerClient || !workerIncarnation) throw new OwnedSessionAdoptionUnavailableError(); + const workerStopRevision = worker.stopRevision; + const timeout = setTimeout( + () => this.expireRecoverableOwnedAdoption(worker, receipt.recordId), + Math.max(1, prepareDeadline - Date.now()), + ); + timeout.unref(); + const adoption: RecoverableOwnedAdoption = { + recordId: receipt.recordId, + requestIdDigest, + requestDigest, + client, + workerClient, + workerIncarnation, + workerStopRevision, + previousOwnerClientId: finalReceipt.ownerClientId, + activeSessionId: command.activeSessionId, + proof: receipt.authority.proof, + frames: [], + bufferedBytes: 0, + finalRetry: true, + timeout, + }; + worker.recoverableAdoption = adoption; + const preparePromise = (async () => { + this.mutationDrain.begin(); + const previousAuthority = receipt.authority; + const previousAuthorityDigest = this.recoverableAuthorityDigest(previousAuthority); + const previousOwnerClientId = finalReceipt.ownerClientId; + let authorityReplaced = false; + try { + await this.mutationDrain.waitForDrain( + 1, + AbortSignal.timeout(Math.max(1, prepareDeadline - Date.now())), + "Timed out draining mutations for recoverable owned-session adoption", + ); + if ( + Date.now() >= prepareDeadline || + worker.recoverableAdoption !== adoption || + worker.client !== workerClient || + worker.workerIncarnation !== workerIncarnation || + !workerClient.matchesAuthenticatedIncarnation(workerIncarnation) || + worker.stopRevision !== workerStopRevision + ) { + throw new OwnedSessionAdoptionUnavailableError(); + } + const result = await this.prepareRecoverableOwnedSnapshot( + worker, + adoption, + receipt, + command, + prepareDeadline, + ); + await this.assertCurrentOwnership(); + this.assertRecoverableOwnedAdoptionCurrent(worker, adoption, client); + this.validateRecoverableOwnedBuffer(adoption, result.proof.cursor); + const nextAuthority: RecoverableOwnedSessionAuthority = { + ...receipt.authority, + proof: result.proof, + }; + worker.descriptor.ownerClientId = this.protocolClientId(client); + this.persistWorker(worker); + this.ownedSessionRecoveryStore.replaceAuthority( + receipt.recordId, + this.recoverableAuthorityDigest(nextAuthority), + nextAuthority, + ); + authorityReplaced = true; + this.ownedSessionRecoveryStore.markFinal( + receipt.recordId, + Date.now() + RECOVERABLE_OWNED_CONFIRMATION_RETENTION_MS, + ); + // Final-retry prepare has durably rebound ownership to this claimant. + // Keep the reservation validator aligned with that new exact owner. + adoption.previousOwnerClientId = this.protocolClientId(client); + worker.recoverableFinal = { + recordId: receipt.recordId, + requestIdDigest, + requestDigest, + ownerClientId: this.protocolClientId(client), + result, + frames: adoption.frames.map((frame) => Buffer.from(frame.payload)), + ...(finalReceipt.mcpTransfer ? { mcpTransfer: finalReceipt.mcpTransfer } : {}), + }; + // Keep the capture fence reserved until repeated commit queues every retained frame. + // Prepare must not attach live delivery or K+1 can overtake retained S+1..K. + this.scheduleRecoverableOwnedConfirmationExpiry(worker, receipt.recordId); + return result; + } catch (error) { + if (authorityReplaced) { + try { + this.ownedSessionRecoveryStore.replaceAuthority( + receipt.recordId, + previousAuthorityDigest, + previousAuthority, + ); + } catch { + await this.stopWorker(worker, true).catch(() => undefined); + } + } + worker.descriptor.ownerClientId = previousOwnerClientId; + try { + this.persistWorker(worker); + } catch { + await this.stopWorker(worker, true).catch(() => undefined); + } + this.rollbackRecoverableOwnedAdoption(worker, adoption); + throw error; + } finally { + this.mutationDrain.end(); + } + })(); + adoption.preparePromise = preparePromise; + return preparePromise; + } + + private async prepareRecoverableOwnedSnapshot( + worker: ResidentWorker, + adoption: RecoverableOwnedAdoption, + receipt: OwnedSessionRecoveryReceipt, + command: Extract, + prepareDeadline: number, + ): Promise { + const response = await this.requireAvailableWorkerClient(worker).request( + { + type: "attach", + activeSessionId: command.activeSessionId, + capabilities: ["attach_snapshot", "event_sequence", "slim_attach", "correlated_prompt_lifecycle_v1"], + supportsExtensionUi: false, + env: collectDaemonClientEnv(), + }, + Math.max(1, prepareDeadline - Date.now()), + ); + const attached = attachResultFromResponse(response); + const snapshot = attached.snapshot; + const cursor = snapshot.lastEventCursor; + if ( + attached.replay.status !== "complete" || + snapshot.activeSessionId !== command.activeSessionId || + snapshot.summary.sessionId !== command.sessionId || + snapshot.state.sessionId !== command.sessionId || + !cursor || + cursor.generation !== command.cursor.generation || + cursor.sequence < command.cursor.sequence || + cursor.sequence !== snapshot.lastEventSequence + ) { + throw new OwnedSessionAdoptionUnavailableError(); + } + const lifecycle = snapshot.promptLifecycles?.records.find( + (entry) => entry.correlationId === command.correlationId, + ); + const expired = snapshot.promptLifecycles?.expired.find((entry) => entry.correlationId === command.correlationId); + if (!lifecycle && !expired) throw new OwnedSessionAdoptionUnavailableError(); + this.validateRecoverableOwnedBuffer(adoption, cursor); + const proof: DaemonRecoverableOwnedSessionAdoptionProof = { + feature: "recoverable_owned_session_adoption_v1", + status: "adopted", + supervisorGeneration: this.generation, + ownershipGeneration: receipt.ownershipGeneration, + activeSessionId: command.activeSessionId, + sessionId: command.sessionId, + correlationId: command.correlationId, + lifecycle: lifecycle ?? { ...expired!, expired: true as const }, + cursor, + mcpOwnerId: command.mcpOwnerId, + }; + adoption.proof = proof; + adoption.client.capabilities = normalizeCapabilities(command.capabilities, command.supportsExtensionUi); + adoption.client.supportsExtensionUi = adoption.client.capabilities.has("extension_ui"); + const publicSummary = this.publicSummary(worker, snapshot.summary); + const result: DaemonRecoverableOwnedSessionPrepareResult = { + ...attached, + state: attached.state ? publicSummary : undefined, + snapshot: { ...snapshot, summary: publicSummary }, + client: { id: command.clientId, capabilities: [...adoption.client.capabilities] }, + recoveryHandle: receipt.recoveryHandle, + proof, + }; + adoption.result = result; + return result; + } + + private assertRecoverableOwnedAdoptionCurrent( + worker: ResidentWorker, + adoption: RecoverableOwnedAdoption, + client: DaemonSocketClient, + ): void { + if ( + this.workers.get(worker.descriptor.workerId) !== worker || + worker.recoverableAdoption !== adoption || + adoption.client !== client || + worker.descriptor.ownerClientId !== adoption.previousOwnerClientId || + worker.client !== adoption.workerClient || + worker.workerIncarnation !== adoption.workerIncarnation || + !adoption.workerClient.matchesAuthenticatedIncarnation(adoption.workerIncarnation) || + worker.stopRevision !== adoption.workerStopRevision || + this.isWorkerStopping(worker) || + [...this.clients].some( + (other) => + other !== client && + this.protocolClientId(other) === adoption.previousOwnerClientId && + !other.socket.destroyed, + ) + ) { + throw new OwnedSessionAdoptionUnavailableError(); + } + } + + private recoverableOwnedMcpTransferProof( + response: DaemonResponse, + transactionId: string, + previousOwnerId: string, + nextOwnerId: string, + expectedState?: "transferred" | "rolled_back", + ): DaemonWorkerAcpMcpOwnerTransferProof { + if (!response.success || !response.data || typeof response.data !== "object") { + throw new OwnedSessionAdoptionUnavailableError(); + } + const proof = response.data as Partial; + if ( + proof.transactionId !== transactionId || + proof.previousOwnerId !== previousOwnerId || + proof.nextOwnerId !== nextOwnerId || + typeof proof.changed !== "boolean" || + (proof.state !== "transferred" && proof.state !== "rolled_back") || + (expectedState !== undefined && proof.state !== expectedState) + ) { + throw new OwnedSessionAdoptionUnavailableError(); + } + return proof as DaemonWorkerAcpMcpOwnerTransferProof; + } + + private async requestRecoverableOwnedMcpTransfer( + worker: ResidentWorker, + activeSessionId: string, + transactionId: string, + action: "transfer" | "query" | "rollback", + previousOwnerId: string, + nextOwnerId: string, + ): Promise { + const response = await this.requireAvailableWorkerClient(worker).requestWorker( + { + type: "worker_transfer_acp_mcp_owner", + activeSessionId, + transactionId, + action, + previousOwnerId, + ownerId: nextOwnerId, + }, + RECOVERABLE_OWNED_PREPARE_TIMEOUT_MS, + ); + return this.recoverableOwnedMcpTransferProof( + response, + transactionId, + previousOwnerId, + nextOwnerId, + action === "transfer" ? "transferred" : action === "rollback" ? "rolled_back" : undefined, + ); + } + + private async retireRecoverableOwnedMcpTransferReceipt( + worker: ResidentWorker, + activeSessionId: string, + transactionId: string, + previousOwnerId: string, + nextOwnerId: string, + ): Promise { + const response = await this.requireAvailableWorkerClient(worker).requestWorker( + { + type: "worker_transfer_acp_mcp_owner", + activeSessionId, + transactionId, + action: "retire", + previousOwnerId, + ownerId: nextOwnerId, + }, + RECOVERABLE_OWNED_PREPARE_TIMEOUT_MS, + ); + if (!response.success || !response.data || typeof response.data !== "object") { + throw new OwnedSessionAdoptionUnavailableError(); + } + const retired = response.data as Partial; + if ( + retired.transactionId !== transactionId || + retired.previousOwnerId !== previousOwnerId || + retired.nextOwnerId !== nextOwnerId || + retired.status !== "retired" + ) { + throw new OwnedSessionAdoptionUnavailableError(); + } + } + + private async transferRecoverableOwnedMcpOwner( + worker: ResidentWorker, + activeSessionId: string, + transactionId: string, + previousOwnerId: string, + nextOwnerId: string, + ): Promise { + try { + return await this.requestRecoverableOwnedMcpTransfer( + worker, + activeSessionId, + transactionId, + "transfer", + previousOwnerId, + nextOwnerId, + ); + } catch { + const observed = await this.requestRecoverableOwnedMcpTransfer( + worker, + activeSessionId, + transactionId, + "query", + previousOwnerId, + nextOwnerId, + ); + if (observed.state === "transferred") return observed; + return this.requestRecoverableOwnedMcpTransfer( + worker, + activeSessionId, + transactionId, + "transfer", + previousOwnerId, + nextOwnerId, + ); + } + } + + private async rollbackRecoverableOwnedMcpOwner( + worker: ResidentWorker, + activeSessionId: string, + transactionId: string, + previousOwnerId: string, + nextOwnerId: string, + ): Promise { + for (let attempt = 0; attempt < 2; attempt++) { + try { + await this.requestRecoverableOwnedMcpTransfer( + worker, + activeSessionId, + transactionId, + "rollback", + previousOwnerId, + nextOwnerId, + ); + return true; + } catch { + try { + const proof = await this.requestRecoverableOwnedMcpTransfer( + worker, + activeSessionId, + transactionId, + "query", + previousOwnerId, + nextOwnerId, + ); + if (proof.state === "rolled_back") return true; + } catch { + // A failed query leaves rollback unproven and must not authorize a retry. + } + } + } + return false; + } + + private async commitRecoverableOwnedSessionAdoption( + client: DaemonSocketClient, + command: Extract, + ): Promise { + this.assertRecoverableOwnedSessionPlatform(); + if ( + command.expectedSupervisorGeneration !== this.generation || + !isDaemonRecoveryRequestId(command.requestId) || + command.proof.supervisorGeneration !== command.expectedSupervisorGeneration + ) { + throw new OwnedSessionAdoptionUnavailableError(); + } + const requestIdDigest = this.ownedSessionRecoveryStore.digestRequest({ + type: "prepare_recoverable_owned_session_adoption", + requestId: command.requestId, + }); + const adoptionMatches = [...this.workers.values()].filter((candidate) => { + const reserved = candidate.recoverableAdoption; + return ( + reserved?.client === client && + reserved.requestIdDigest === requestIdDigest && + reserved.result?.recoveryHandle === command.recoveryHandle && + isDeepStrictEqual(reserved.proof, command.proof) + ); + }); + const committedMatches = [...this.workers.values()].filter((candidate) => { + const final = candidate.recoverableFinal; + return ( + final?.ownerClientId === this.protocolClientId(client) && + final.requestIdDigest === requestIdDigest && + final.result.recoveryHandle === command.recoveryHandle && + isDeepStrictEqual(final.result.proof, command.proof) + ); + }); + if (adoptionMatches.length > 1 || committedMatches.length > 1) { + throw new OwnedSessionAdoptionUnavailableError(); + } + const selected = adoptionMatches[0]; + const adoption = selected?.recoverableAdoption; + const committedWorker = committedMatches[0]; + const committed = committedWorker?.recoverableFinal; + if (adoption?.finalRetry) { + if (!selected || !adoption.result || committedWorker !== selected || !committed) { + throw new OwnedSessionAdoptionUnavailableError(); + } + const receipt = this.ownedSessionRecoveryStore.getForConfirmation(command.recoveryHandle, requestIdDigest); + if ( + adoption.recordId !== receipt.recordId || + committed.recordId !== receipt.recordId || + selected.recoveryRecordId !== receipt.recordId || + selected.workerIncarnation !== receipt.authority.workerIncarnation || + !selected.client?.matchesAuthenticatedIncarnation(receipt.authority.workerIncarnation) || + selected.descriptor.ownerClientId !== this.protocolClientId(client) || + client.socket.destroyed ) { - throw new Error("Correlated prompt targeted a stale session generation"); + throw new OwnedSessionAdoptionUnavailableError(); + } + this.assertRecoverableOwnedAdoptionCurrent(selected, adoption, client); + this.validateRecoverableOwnedBuffer(adoption, adoption.proof.cursor); + const frames = adoption.frames.map((frame) => Buffer.from(frame.payload)); + committed.frames = frames; + for (const frame of frames) this.writeSerialized(client, frame); + clearTimeout(adoption.timeout); + selected.recoverableAdoption = undefined; + client.attachedActiveSessionIds.add(adoption.proof.activeSessionId); + void this.syncWorkerExtensionUi(adoption.proof.activeSessionId); + return adoption.proof; + } + if (committed && committedWorker && !adoption) { + const receipt = this.ownedSessionRecoveryStore.getForConfirmation(command.recoveryHandle, requestIdDigest); + if ( + committedWorker.recoveryRecordId !== receipt.recordId || + committedWorker.workerIncarnation !== receipt.authority.workerIncarnation || + !committedWorker.client?.matchesAuthenticatedIncarnation(receipt.authority.workerIncarnation) || + committedWorker.descriptor.ownerClientId !== this.protocolClientId(client) || + client.socket.destroyed || + [...this.clients].some( + (other) => + other !== client && + this.protocolClientId(other) === this.protocolClientId(client) && + !other.socket.destroyed, + ) + ) { + throw new OwnedSessionAdoptionUnavailableError(); } - const resolvedCommand = { - ...command, - activeSessionId: match.summary.activeSessionId ?? match.summary.id, - ...(admission ? { admissionId: admission.workerAdmissionId } : {}), - } as DaemonCommand; - if (admission) { - admission.worker = match.worker; - admission.workerActiveSessionId = match.summary.activeSessionId ?? match.summary.id; + for (const frame of committed.frames) this.writeSerialized(client, frame); + client.attachedActiveSessionIds.add(committed.result.proof.activeSessionId); + return committed.result.proof; + } + if (!selected || !adoption || !adoption.result || adoption.finalRetry || committed) { + throw new OwnedSessionAdoptionUnavailableError(); + } + const commitReceipt = this.ownedSessionRecoveryStore.getForCommit( + adoption.recordId, + command.recoveryHandle, + requestIdDigest, + ); + this.assertRecoverableOwnedAdoptionCurrent(selected, adoption, client); + this.validateRecoverableOwnedBuffer(adoption, adoption.proof.cursor); + await this.assertCurrentOwnership(); + this.assertRecoverableOwnedAdoptionCurrent(selected, adoption, client); + this.validateRecoverableOwnedBuffer(adoption, adoption.proof.cursor); + this.ownedSessionRecoveryStore.markCommitting(adoption.recordId); + adoption.committing = true; + const previousOwner = selected.descriptor.ownerClientId; + const previousAuthority = commitReceipt.authority; + const previousAuthorityDigest = this.recoverableAuthorityDigest(previousAuthority); + const transactionId = this.ownedSessionRecoveryStore.digestRequest({ + type: "recoverable_owned_mcp_transfer", + recordId: adoption.recordId, + requestDigest: adoption.requestDigest, + }); + let transferAttempted = false; + let authorityReplaced = false; + try { + transferAttempted = true; + await this.transferRecoverableOwnedMcpOwner( + selected, + adoption.proof.activeSessionId, + transactionId, + commitReceipt.authority.mcpOwnerId, + adoption.proof.mcpOwnerId, + ); + await this.assertCurrentOwnership(); + // This is the ownership cut. No await may occur between the final live-channel/frame + // validation and the synchronous descriptor, authority, and receipt commit below. + this.assertRecoverableOwnedAdoptionCurrent(selected, adoption, client); + this.validateRecoverableOwnedBuffer(adoption, adoption.proof.cursor); + const nextAuthority: RecoverableOwnedSessionAuthority = { + ...commitReceipt.authority, + mcpOwnerId: adoption.proof.mcpOwnerId, + proof: adoption.proof, + }; + selected.descriptor.ownerClientId = this.protocolClientId(client); + this.persistWorker(selected); + this.ownedSessionRecoveryStore.replaceAuthority( + adoption.recordId, + this.recoverableAuthorityDigest(nextAuthority), + nextAuthority, + ); + authorityReplaced = true; + this.ownedSessionRecoveryStore.markFinal( + adoption.recordId, + Date.now() + RECOVERABLE_OWNED_CONFIRMATION_RETENTION_MS, + ); + selected.recoverableFinal = { + recordId: adoption.recordId, + requestIdDigest: adoption.requestIdDigest, + requestDigest: adoption.requestDigest, + ownerClientId: this.protocolClientId(client), + result: adoption.result, + frames: adoption.frames.map((frame) => Buffer.from(frame.payload)), + mcpTransfer: { + transactionId, + previousOwnerId: commitReceipt.authority.mcpOwnerId, + nextOwnerId: adoption.proof.mcpOwnerId, + }, + }; + this.scheduleRecoverableOwnedConfirmationExpiry(selected, adoption.recordId); + } catch (error) { + let rollbackProved = !transferAttempted; + if (transferAttempted) { + rollbackProved = await this.rollbackRecoverableOwnedMcpOwner( + selected, + adoption.proof.activeSessionId, + transactionId, + commitReceipt.authority.mcpOwnerId, + adoption.proof.mcpOwnerId, + ); } - const isRootKill = - command.type === "kill" && - (match.summary.activeSessionId ?? match.summary.id) === match.worker.descriptor.rootActiveSessionId; - if (!isRootKill) { - const forward = async () => { - const workerResponse = this.forwardToWorker(match.worker, resolvedCommand); - correlatedOrder?.release(); - const response = await workerResponse; - if (admission && response.success) admission.status = "owned"; - return response; - }; - if (command.type === "rename" || command.type === "set_session_name") { - const reservation = this.summaryNameReservationInput(match.summary, command.name.trim()); - return await this.withSessionNameReservation(reservation, async () => { - await this.assertSupervisorSessionNameAvailable(match.summary, reservation.name); - return forward(); - }); + if (rollbackProved && transferAttempted) { + await this.retireRecoverableOwnedMcpTransferReceipt( + selected, + adoption.proof.activeSessionId, + transactionId, + commitReceipt.authority.mcpOwnerId, + adoption.proof.mcpOwnerId, + ).catch(() => undefined); + } + let authorityRollbackProved = true; + if (authorityReplaced) { + try { + this.ownedSessionRecoveryStore.replaceAuthority( + adoption.recordId, + previousAuthorityDigest, + previousAuthority, + ); + } catch { + authorityRollbackProved = false; } - return await forward(); } + selected.descriptor.ownerClientId = previousOwner; try { - this.persistWorkerStopTombstone(match.worker, true); - } catch (error) { - this.scheduleWorkerStopFinalization(match.worker); - throw error; + this.persistWorker(selected); + } catch { + authorityRollbackProved = false; } - const releaseStopOwnership = this.acquireWorkerStopOwnership(match.worker); - let response: DaemonResponse; + this.rollbackRecoverableOwnedAdoption(selected, adoption); + if (!rollbackProved || !authorityRollbackProved) { + await this.stopWorker(selected, true).catch(() => undefined); + } + throw error; + } + const frames = selected.recoverableFinal?.frames ?? adoption.frames.map((frame) => frame.payload); + for (const frame of frames) this.writeSerialized(client, frame); + clearTimeout(adoption.timeout); + selected.recoverableAdoption = undefined; + client.attachedActiveSessionIds.add(adoption.proof.activeSessionId); + void this.syncWorkerExtensionUi(adoption.proof.activeSessionId); + return adoption.proof; + } + + private async confirmRecoverableOwnedSessionAdoption( + _client: DaemonSocketClient, + command: Extract, + ): Promise { + this.assertRecoverableOwnedSessionPlatform(); + if ( + !isDaemonRecoveryRequestId(command.requestId) || + command.expectedSupervisorGeneration !== this.generation || + command.proof.supervisorGeneration !== command.expectedSupervisorGeneration + ) { + throw new OwnedSessionAdoptionUnavailableError(); + } + const requestIdDigest = this.ownedSessionRecoveryStore.digestRequest({ + type: "prepare_recoverable_owned_session_adoption", + requestId: command.requestId, + }); + const receipt = this.ownedSessionRecoveryStore.getForConfirmation(command.recoveryHandle, requestIdDigest); + if ( + receipt.ownershipGeneration !== command.proof.ownershipGeneration || + command.proof.supervisorGeneration !== this.generation || + command.proof.sessionId !== receipt.authority.sessionId || + command.proof.correlationId !== receipt.authority.correlationId || + !receipt.authority.proof || + !isDeepStrictEqual(command.proof, receipt.authority.proof) + ) { + throw new OwnedSessionAdoptionUnavailableError(); + } + this.ownedSessionRecoveryStore.confirm({ + recoveryHandle: command.recoveryHandle, + requestIdDigest, + authorityDigest: this.recoverableAuthorityDigest(receipt.authority), + expiresAt: RECOVERABLE_OWNED_CONNECTED_EXPIRY, + }); + const worker = this.workers.get(receipt.authority.workerId); + const finalReceipt = + worker?.recoverableFinal?.recordId === receipt.recordId ? worker.recoverableFinal : undefined; + if (worker && finalReceipt?.mcpTransfer) { + await this.retireRecoverableOwnedMcpTransferReceipt( + worker, + receipt.authority.activeSessionId, + finalReceipt.mcpTransfer.transactionId, + finalReceipt.mcpTransfer.previousOwnerId, + finalReceipt.mcpTransfer.nextOwnerId, + ).catch(() => undefined); + } + if (worker?.recoverableFinal?.recordId === receipt.recordId) worker.recoverableFinal = undefined; + if (worker?.recoveryConfirmationTimer) { + clearTimeout(worker.recoveryConfirmationTimer); + worker.recoveryConfirmationTimer = undefined; + } + return { status: "confirmed" }; + } + + private retireRecoverableOwnedFinalTransferReceipt(worker: ResidentWorker): void { + const finalReceipt = worker.recoverableFinal; + if (!finalReceipt?.mcpTransfer) return; + void this.retireRecoverableOwnedMcpTransferReceipt( + worker, + finalReceipt.result.proof.activeSessionId, + finalReceipt.mcpTransfer.transactionId, + finalReceipt.mcpTransfer.previousOwnerId, + finalReceipt.mcpTransfer.nextOwnerId, + ).catch(() => undefined); + } + + private scheduleRecoverableOwnedConfirmationExpiry(worker: ResidentWorker, recordId: string): void { + if (worker.recoveryConfirmationTimer) clearTimeout(worker.recoveryConfirmationTimer); + worker.recoveryConfirmationTimer = setTimeout(() => { + worker.recoveryConfirmationTimer = undefined; + let expired = false; try { - response = await this.forwardToWorker(match.worker, resolvedCommand); - } finally { - try { - await this.stopWorker(match.worker, true, false, true); - } finally { - releaseStopOwnership(); + const receipt = this.ownedSessionRecoveryStore.get(recordId); + if (!receipt || receipt.phase === "final") { + this.ownedSessionRecoveryStore.remove(recordId); + expired = true; } + } catch { + expired = true; } - return response; - } finally { - if (admission) this.deletePromptAdmission(admission); - } + if (!expired || worker.recoveryRecordId !== recordId) return; + if (worker.ownerCleanupTimer) { + clearTimeout(worker.ownerCleanupTimer); + worker.ownerCleanupTimer = undefined; + } + this.retireRecoverableOwnedFinalTransferReceipt(worker); + worker.recoveryRecordId = undefined; + worker.recoverableFinal = undefined; + if (this.workers.get(worker.descriptor.workerId) !== worker) return; + const ownerClientId = worker.descriptor.ownerClientId; + const ownerConnected = + ownerClientId !== undefined && + [...this.clients].some( + (client) => this.protocolClientId(client) === ownerClientId && !client.socket.destroyed, + ); + if (ownerConnected) { + this.scheduleOwnedWorkerCleanup(worker); + return; + } + void this.stopWorker(worker, true, true).catch((error) => + this.log( + `Could not contain unconfirmed recoverable worker ${worker.descriptor.workerId}: ${String(error)}`, + ), + ); + }, RECOVERABLE_OWNED_CONFIRMATION_RETENTION_MS + 1); + worker.recoveryConfirmationTimer.unref(); } - private async handleList( - client: DaemonSocketClient, - command: Extract, - ): Promise { - await Promise.all( - [...this.workers.values()] - .filter((worker) => !this.isWorkerStopping(worker)) - .map((worker) => this.refreshWorkerSummaries(worker).catch(() => undefined)), - ); - const clientOwnedWorkers = [...this.workers.values()].filter((worker) => !this.isVisibleWorker(worker)); - // Stopping workers stay listed (with an honest workerState) because this - // list also feeds busy-daemon safety checks in daemon-launch. - const active = [...this.workers.values()] - .filter( - (worker) => - this.isVisibleWorker(worker) || - (command.includeClientOwned === true && this.isWorkerAccessibleToClient(client, worker)), - ) - .flatMap((worker) => [...worker.summaries.values()].map((summary) => this.publicSummary(worker, summary))); - const busyClientOwnedSessionCount = clientOwnedWorkers - .flatMap((worker) => [...worker.summaries.values()]) - .filter(isSessionSummaryBusy).length; - const data = { - sessions: active, - ...(command.includeClientOwned ? { busyClientOwnedSessionCount } : {}), - }; - if (!command.all) { - return success(command.id, "list", data); + private validateRecoverableOwnedBuffer( + adoption: RecoverableOwnedAdoption, + cursor: { generation: string; sequence: number }, + ): void { + if (adoption.error) throw adoption.error; + const reconciled = reconcileRecoverableOwnedFrames(adoption.frames, cursor); + adoption.frames = reconciled.frames; + adoption.bufferedBytes = reconciled.bufferedBytes; + } + + private captureRecoverableOwnedFrame( + worker: ResidentWorker, + activeSessionId: string, + message: DaemonOutbound, + payload: Buffer, + ): void { + const adoption = worker.recoverableAdoption; + if (!adoption || adoption.activeSessionId !== activeSessionId) return; + if (adoption.error) return; + const meta = (message as { meta?: { cursor?: { generation?: unknown; sequence?: unknown } } }).meta; + const cursor = meta?.cursor; + if ( + !cursor || + typeof cursor.generation !== "string" || + !Number.isSafeInteger(cursor.sequence) || + (cursor.sequence as number) < 0 + ) { + adoption.error = new OwnedSessionAdoptionUnavailableError(); + return; + } + try { + adoption.bufferedBytes = appendRecoverableOwnedFrame(adoption.frames, adoption.bufferedBytes, { + payload: Buffer.from(payload), + message, + cursor: { generation: cursor.generation, sequence: cursor.sequence as number }, + }); + } catch (error) { + adoption.error = error instanceof Error ? error : new OwnedSessionAdoptionUnavailableError(); } - const sessionDir = command.sessionDir ?? this.defaultSessionConfig.sessionDir; - const saved = await this.catalog.list(command.cwd ? resolve(command.cwd) : undefined, sessionDir); - return success(command.id, "list", { ...data, sessions: mergeSessionLists(active, saved) }); } - private async handleSavedSessionList( - client: DaemonSocketClient, - command: Extract, - ): Promise { - let cwd: string; - let sessionDir: string | undefined; - let activeSessionId: string | undefined; - if ("activeSessionId" in command) { - const match = await this.findWorkerForClient(client, command.activeSessionId); - cwd = match.summary.cwd; - sessionDir = this.defaultSessionConfig.sessionDir; - activeSessionId = match.summary.activeSessionId ?? match.summary.id; - } else { - cwd = resolve(command.cwd); - sessionDir = command.sessionDir; + private expireRecoverableOwnedAdoption(worker: ResidentWorker, recordId: string): void { + const adoption = worker.recoverableAdoption; + if (!adoption || adoption.recordId !== recordId) return; + this.rollbackRecoverableOwnedAdoption(worker, adoption); + } + + private rollbackRecoverableOwnedAdoption(worker: ResidentWorker, adoption: RecoverableOwnedAdoption): void { + clearTimeout(adoption.timeout); + if (worker.recoverableAdoption === adoption) worker.recoverableAdoption = undefined; + try { + const receipt = this.ownedSessionRecoveryStore.get(adoption.recordId); + if (receipt?.phase === "final") { + this.ownedSessionRecoveryStore.markFinal( + adoption.recordId, + Date.now() + RECOVERABLE_OWNED_CONFIRMATION_RETENTION_MS, + ); + } else if (receipt) { + this.ownedSessionRecoveryStore.rollbackAdoption( + adoption.recordId, + this.recoverableAuthorityDigest(receipt.authority), + Date.now() + RECOVERABLE_OWNED_DISCONNECTED_RETENTION_MS, + ); + } + } catch { + // Without a durable disconnected receipt, normal owner cleanup remains authoritative. } - const callbacks = command.id - ? { - onProgress: (loaded: number, total: number) => - this.write(client, { - id: command.id, - type: "session_list_progress", - command: "list_saved_sessions", - ...(activeSessionId ? { activeSessionId } : {}), - loaded, - total, - }), - onSession: (session: SessionInfo) => - this.write(client, { - id: command.id, - type: "session_list_item", - command: "list_saved_sessions", - ...(activeSessionId ? { activeSessionId } : {}), - session: serializeSavedSessionInfo(session), - }), - } - : undefined; - const saved = await this.catalog.list(command.scope === "current" ? cwd : undefined, sessionDir, callbacks); - return success(command.id, "list_saved_sessions", { sessions: saved.map(serializeSavedSessionInfo) }); + this.scheduleOwnedWorkerCleanup(worker); } - private async createOrReuseWorker(clientId: string, command: DaemonCreateCommand): Promise { + private async createOrReuseWorker( + clientId: string, + command: DaemonCreateCommand, + onLaunched?: (worker: ResidentWorker) => void, + ): Promise { let createCommand = command; if (command.name !== undefined) { const normalizedName = command.name.trim(); @@ -2702,8 +4309,13 @@ export class DaemonSupervisor { return this.reuseWorkerForCreate(existing, ownerClientId, createCommand.sessionPath); } } + const launchNewWorker = async (): Promise => { + const worker = await this.launchWorker(createCommand, undefined, ownerClientId); + onLaunched?.(worker); + return worker; + }; const opening = (async () => { - if (!createCommand.name) return this.launchWorker(createCommand, undefined, ownerClientId); + if (!createCommand.name) return launchNewWorker(); const savedSiblings = createCommand.sessionPath ? await this.rlmLedgerSiblings(createCommand.sessionPath) : []; const target = savedSiblings.find( (session) => canonicalSessionPath(session.path) === canonicalSessionPath(createCommand.sessionPath!), @@ -2716,7 +4328,7 @@ export class DaemonSupervisor { } else { await this.assertSupervisorSessionNameAvailable(targetSummary, createCommand.name!); } - return this.launchWorker(createCommand, undefined, ownerClientId); + return launchNewWorker(); }); })(); this.openingWorkers.set(key, opening); @@ -2913,13 +4525,17 @@ export class DaemonSupervisor { const orphanProcessJournalPath = existing?.descriptor.orphanProcessJournalPath ?? join(this.descriptorDir, `${workerId}.orphans.jsonl`); const launch = createCliSubprocessLaunchSpec(["--mode", "daemon", "--daemon-socket", socketPath]); - const workerEnvironmentSource = { + const workerEnvironmentBase = sanitizeDaemonWorkerBootstrapEnvironment({ ...(launchEnvMode === "replace" ? launchEnv : { ...process.env, ...launchEnv }), + }); + const workerEnvironmentSource = { + ...workerEnvironmentBase, // Prime-owned worker authentication, recovery, startup, lease, and orphan-cleanup bootstrap. [DAEMON_WORKER_ROLE_ENV]: "1", [DAEMON_WORKER_TOKEN_ENV]: token, [DAEMON_WORKER_ACTIVE_SESSION_ID_ENV]: rootActiveSessionId, [DAEMON_WORKER_SUPERVISOR_SOCKET_ENV]: this.socketPath, + [DAEMON_WORKER_SUPERVISOR_AGENT_DIR_ENV]: createCommand.config?.agentDir ?? this.defaultSessionConfig.agentDir, [DAEMON_WORKER_RECOVERY_JOURNAL_ENV]: recoveryJournalPath, [DAEMON_WORKER_STARTUP_GATE_FD_ENV]: String(WORKER_STARTUP_GATE_FD), [ORPHAN_PROCESS_JOURNAL_ENV]: orphanProcessJournalPath, @@ -3156,6 +4772,45 @@ export class DaemonSupervisor { } } + private retireRecoverableOwnedAuthorityForWorkerIncarnation( + worker: ResidentWorker, + nextWorkerIncarnation: string, + ): void { + const recordId = worker.recoveryRecordId; + let recordedWorkerIncarnation = worker.workerIncarnation; + if (recordId) { + try { + recordedWorkerIncarnation ??= this.ownedSessionRecoveryStore.get(recordId)?.authority.workerIncarnation; + } catch { + recordedWorkerIncarnation = undefined; + } + } + if ( + recordedWorkerIncarnation === nextWorkerIncarnation && + (!worker.recoverableAdoption || worker.recoverableAdoption.workerIncarnation === nextWorkerIncarnation) + ) { + return; + } + if (!recordId && !worker.recoverableAdoption && !worker.recoverableFinal) return; + this.retireRecoverableOwnedFinalTransferReceipt(worker); + if (recordId) this.ownedSessionRecoveryStore.remove(recordId); + if (worker.recoverableAdoption) { + clearTimeout(worker.recoverableAdoption.timeout); + worker.recoverableAdoption.error = new OwnedSessionAdoptionUnavailableError(); + } + if (worker.recoveryConfirmationTimer) { + clearTimeout(worker.recoveryConfirmationTimer); + worker.recoveryConfirmationTimer = undefined; + } + if (worker.ownerCleanupTimer) { + clearTimeout(worker.ownerCleanupTimer); + worker.ownerCleanupTimer = undefined; + } + worker.recoveryRecordId = undefined; + worker.recoverableAdoption = undefined; + worker.recoverableFinal = undefined; + } + private async connectWorker( worker: ResidentWorker, timeoutMs: number, @@ -3171,19 +4826,23 @@ export class DaemonSupervisor { try { await client.connect(Math.min(500, Math.max(50, deadline - Date.now()))); await client.waitForHello(1000); - await client.authenticateWorker( + const workerIncarnation = await client.authenticateWorker( worker.descriptor.authenticationToken, this.supervisorAuthenticationClaim(), 1000, ); await this.assertRecoveryAllowed(); assertCurrent?.(); + this.retireRecoverableOwnedAuthorityForWorkerIncarnation(worker, workerIncarnation); client.onFrame((frame) => this.handleWorkerFrame(worker, frame, client)); client.onClose((error) => void this.handleWorkerClose(worker, client, error)); - worker.client?.close(); + const previousClient = worker.client; worker.client = client; + worker.workerIncarnation = workerIncarnation; worker.authorityRevision = (worker.authorityRevision ?? 0) + 1; worker.authorizedActiveSessionIds = new Set([worker.descriptor.rootActiveSessionId]); + previousClient?.close(); + this.scheduleOwnedWorkerCleanup(worker); return client; } catch (error) { lastError = error; @@ -3322,8 +4981,12 @@ export class DaemonSupervisor { if (worker.client !== client) { return; } + const interruptedAdoption = + worker.recoverableAdoption?.workerClient === client ? worker.recoverableAdoption : undefined; this.abortTranscriptPreparations(worker, error); worker.client = undefined; + worker.workerIncarnation = undefined; + if (interruptedAdoption) this.rollbackRecoverableOwnedAdoption(worker, interruptedAdoption); worker.authorityRevision = (worker.authorityRevision ?? 0) + 1; worker.authorizedActiveSessionIds = new Set([worker.descriptor.rootActiveSessionId]); this.invalidateWorkerSessionInputPauses(worker, "Session worker disconnected while input was paused"); @@ -3862,12 +5525,11 @@ export class DaemonSupervisor { this.persistWorker(worker); return; } - const safeToKillWorkerProcess = - processAlive && processIdentityMatches && worker.descriptor.processStartId !== undefined; - const promptLifecycleRecovery = await this.recoverUncertainWorkerOperations( - worker, - safeToKillWorkerProcess, - ); + const processToKill = + processAlive && processIdentityMatches && worker.descriptor.processStartId !== undefined + ? { pid: worker.descriptor.pid, processStartId: worker.descriptor.processStartId } + : undefined; + const promptLifecycleRecovery = await this.recoverUncertainWorkerOperations(worker, processToKill); assertCurrent(); await this.launchWorker( recoveryCommand, @@ -3949,12 +5611,10 @@ export class DaemonSupervisor { private async recoverUncertainWorkerOperations( worker: ResidentWorker, - killWorkerProcess = true, + processToKill: { pid: number; processStartId: string } | false = false, ): Promise> { await this.assertRecoveryAllowed(); - if (killWorkerProcess) { - signalProcessGroupOrProcess(worker.descriptor.pid, "SIGKILL"); - } + if (processToKill) this.signalCapturedProcess(processToKill, "SIGKILL"); const orphanProcessJournalPath = worker.descriptor.orphanProcessJournalPath; if (orphanProcessJournalPath) { try { @@ -4409,8 +6069,9 @@ export class DaemonSupervisor { private isWorkerAccessibleToClient(client: DaemonSocketClient, worker: ResidentWorker): boolean { return ( - worker.descriptor.ownerClientId === undefined || - worker.descriptor.ownerClientId === this.protocolClientId(client) + !worker.recoverableAdoption && + (worker.descriptor.ownerClientId === undefined || + worker.descriptor.ownerClientId === this.protocolClientId(client)) ); } @@ -5872,6 +7533,14 @@ export class DaemonSupervisor { return; } this.streamReconstructor.observe(decodedOutbound); + this.captureRecoverableOwnedFrame(worker, activeSessionId, decodedOutbound, publicPayload); + if ( + worker.recoveryRecordId && + decodedOutbound.type === "prompt_lifecycle" && + ["completed", "cancelled", "failed"].includes(decodedOutbound.lifecycle.phase) + ) { + queueMicrotask(() => this.scheduleRecoverableOwnedWorkerCleanup(worker, true)); + } if ( decodedOutbound.type === "session_event" && decodedOutbound.event.type === "rlm_child_update" && @@ -6353,6 +8022,12 @@ export class DaemonSupervisor { renameSync(tempPath, path); } + private signalCapturedProcess(process: { pid: number; processStartId: string }, signal: NodeJS.Signals): boolean { + if (getProcessStartId(process.pid) !== process.processStartId) return false; + signalProcessGroupOrProcess(process.pid, signal); + return true; + } + /** * Verdict on whether a pid is still the process we launched. Callers must * be conservative in both directions: signal a pid only on "current" @@ -6473,19 +8148,36 @@ export class DaemonSupervisor { if (!recoveryCleanup) { worker.stopRevision++; } - // A retry can rescind this stop and relaunch the worker while we await - // below. Bind every liveness check and signal to the process this stop - // entered with, and abort cleanup once the stop no longer applies: the - // pid changed (relaunched) or a removeDescriptor stop lost its tombstone - // (rescinded, even before the successor pid lands). + // Fence recovery synchronously before the first await. Capture the exact + // authenticated channel and process generation at the same boundary so a + // recovery task cannot swap in a successor and inherit this stop. + worker.intentionalStop = true; + if (removeDescriptor) { + worker.descriptor.stopRequestedAt ??= new Date().toISOString(); + worker.descriptor.archiveOnStop ||= archiveSession; + } else { + worker.descriptor.lifecycle = "recovering"; + } + const entryStopRevision = worker.stopRevision; const entryPid = worker.descriptor.pid; const entryStartId = worker.descriptor.processStartId; + const entryWorkerIncarnation = worker.workerIncarnation; + const entryWorkerClient = worker.client; + const stoppingWorkerClient = + entryWorkerClient && + entryWorkerIncarnation && + entryWorkerClient.matchesAuthenticatedIncarnation(entryWorkerIncarnation) + ? entryWorkerClient + : undefined; + const recoveryTasks = + !recoveryCleanup && !directChild + ? [worker.recovery, worker.deferredRecovery].flatMap((task) => (task ? [task] : [])) + : []; const assertStopStillApplies = () => { - if (directChild) { - return; - } + if (directChild) return; if ( this.workers.get(worker.descriptor.workerId) !== worker || + worker.stopRevision !== entryStopRevision || worker.descriptor.pid !== entryPid || worker.descriptor.processStartId !== entryStartId || (removeDescriptor && worker.descriptor.stopRequestedAt === undefined) @@ -6497,22 +8189,36 @@ export class DaemonSupervisor { if (removeDescriptor) { this.persistWorkerStopTombstone(worker, archiveSession); } else { - worker.intentionalStop = true; - worker.descriptor.lifecycle = "recovering"; this.persistWorker(worker); } } catch (error) { - if (!directChild) { - throw error; - } + if (!directChild) throw error; this.reportCleanupFailure(`worker rollback state ${worker.descriptor.workerId}`, error); } - if (!recoveryCleanup && !directChild) { - // The tombstone/revision above closes recovery admission. Join every - // recovery task that could already have passed an earlier async gate - // before proving the process and durable registration absent. - await Promise.all([worker.recovery, worker.deferredRecovery].flatMap((task) => (task ? [task] : []))); - assertStopStillApplies(); + let authenticatedShutdownAccepted = false; + if (stoppingWorkerClient) { + try { + const response = archiveSession + ? await stoppingWorkerClient.requestWorker({ type: "worker_archive_and_shutdown" }, force ? 1000 : 5000) + : await stoppingWorkerClient.request({ type: "shutdown" }, force ? 1000 : 5000); + authenticatedShutdownAccepted = response.success; + } catch { + // The exact channel is gone. The pid fallback below performs a fresh + // start-id check immediately before signalling. + } finally { + stoppingWorkerClient.close(); + if (worker.client === stoppingWorkerClient && worker.workerIncarnation === entryWorkerIncarnation) { + worker.client = undefined; + worker.workerIncarnation = undefined; + } + } + } + if (!authenticatedShutdownAccepted) { + if (directChild) { + directChild.child.kill("SIGTERM"); + } else if (this.processIdentity(entryPid, entryStartId) === "current") { + signalProcessGroupOrProcess(entryPid, "SIGTERM"); + } } const transferError = new Error("Session worker stopped during snapshot transfer"); const generationTranscripts = new Set(); @@ -6536,21 +8242,6 @@ export class DaemonSupervisor { worker.transcriptCaches.clear(); worker.snapshotCache.clear(); worker.snapshotGenerations?.clear(); - if (worker.client) { - if (archiveSession) { - await worker.client - .requestWorker({ type: "worker_archive_and_shutdown" }, force ? 1000 : 5000) - .catch(() => undefined); - } else { - await worker.client.request({ type: "shutdown" }, force ? 1000 : 5000).catch(() => undefined); - } - worker.client.close(); - worker.client = undefined; - } else if (directChild) { - directChild.child.kill("SIGTERM"); - } else if (this.processIdentity(entryPid, entryStartId) === "current") { - signalProcessGroupOrProcess(entryPid, "SIGTERM"); - } // Identity-aware in both directions: a replaced pid counts as gone (never // signal a recycled pid) while an unknown identity counts as alive (never // clean up a possibly-live worker on a transient lookup failure). kill(0) @@ -6602,6 +8293,10 @@ export class DaemonSupervisor { if (directChild) { await directChild.closed; } + // Only join recovery work after the captured process has exited. The + // synchronous stop fence prevents those tasks from recreating durable + // state while descriptor and journal removal is pending. + if (recoveryTasks.length > 0) await Promise.all(recoveryTasks); assertStopStillApplies(); if (removeDescriptor && worker.descriptor.archiveOnStop) { if (force) { @@ -6732,9 +8427,7 @@ export class DaemonSupervisor { // to be recycled by an unrelated process. A transiently // unobservable identity skips this attempt but keeps escalation // armed so a wedged worker is still killed on a later pass. - const observedNow = processStartId === undefined ? undefined : getProcessStartId(pid); - if (processStartId !== undefined && observedNow === processStartId) { - signalProcessGroupOrProcess(pid, "SIGKILL"); + if (processStartId !== undefined && this.signalCapturedProcess({ pid, processStartId }, "SIGKILL")) { killed = true; } } @@ -6970,7 +8663,7 @@ export class DaemonSupervisor { ): Promise { this.shuttingDown = true; this.clearIdleEvictionTimer(); - await this.idleEvictionSweep?.catch(() => undefined); + const idleEvictionSweep = this.idleEvictionSweep; if (closingReason) { for (const client of this.clients) { this.write(client, { type: "daemon_closing", reason: closingReason }); @@ -7022,6 +8715,7 @@ export class DaemonSupervisor { worker.client = undefined; } } + if (idleEvictionSweep) await idleEvictionSweep.catch(() => undefined); await this.catalog.stop(); for (const client of this.clients) { client.detachInput(); @@ -7040,13 +8734,8 @@ export class DaemonSupervisor { await this.runCleanupStep("daemon ownership", async () => ownership?.release()); if (relaunch) { const launch = createCliSubprocessLaunchSpec(["--mode", "daemon", "--daemon-socket", this.socketPath]); - const environment = createCliSubprocessEnv(); + const environment = sanitizeDaemonWorkerBootstrapEnvironment(createCliSubprocessEnv()); delete environment[DAEMON_CATALOG_ROLE_ENV]; - delete environment[DAEMON_WORKER_ROLE_ENV]; - delete environment[DAEMON_WORKER_TOKEN_ENV]; - delete environment[DAEMON_WORKER_ACTIVE_SESSION_ID_ENV]; - delete environment[DAEMON_WORKER_SUPERVISOR_SOCKET_ENV]; - delete environment[DAEMON_WORKER_RECOVERY_JOURNAL_ENV]; delete environment[ORPHAN_PROCESS_JOURNAL_ENV]; delete environment[SESSION_LEASES_ENABLED_ENV]; delete environment[SESSION_LEASE_OWNER_ID_ENV]; diff --git a/packages/coding-agent/src/modes/daemon/daemon-worker-client.ts b/packages/coding-agent/src/modes/daemon/daemon-worker-client.ts index 8b8e9c130e..bc144bf7e1 100644 --- a/packages/coding-agent/src/modes/daemon/daemon-worker-client.ts +++ b/packages/coding-agent/src/modes/daemon/daemon-worker-client.ts @@ -3,6 +3,7 @@ import { serializeJsonLine } from "../rpc/jsonl.js"; import { type PrivateFrame, PrivateFramedChannel } from "../session-worker/private-framing.js"; import type { DaemonCommand, DaemonOutbound, DaemonResponse, DaemonServerCapability } from "./daemon-protocol.js"; import { + type DaemonWorkerAuthenticationResult, type DaemonWorkerCommand, type DaemonWorkerCommandBody, type DaemonWorkerFrameHeader, @@ -34,6 +35,7 @@ export class DaemonWorkerClient { >(); private requestId = 0; private hello?: DaemonHello; + private workerIncarnation?: string; private readonly helloWaiters = new Set<{ resolve: (hello: DaemonHello) => void; reject: (error: Error) => void; @@ -123,17 +125,27 @@ export class DaemonWorkerClient { return this.requestWire(command, timeoutMs); } - async authenticateWorker(token: string, owner: DaemonWorkerAuthentication, timeoutMs = 3000): Promise { + async authenticateWorker(token: string, owner: DaemonWorkerAuthentication, timeoutMs = 3000): Promise { const response = await this.requestWorker({ type: "worker_auth", token, ...owner }, timeoutMs); - if (!response.success) { - throw new Error(response.error); + if (!response.success) throw new Error(response.error); + const workerIncarnation = (response.data as Partial | undefined) + ?.workerIncarnation; + if (typeof workerIncarnation !== "string" || !/^[A-Za-z0-9_-]{43}$/.test(workerIncarnation)) { + throw new Error("Daemon worker returned an invalid authenticated incarnation"); } + this.workerIncarnation = workerIncarnation; + return workerIncarnation; + } + + matchesAuthenticatedIncarnation(workerIncarnation: string): boolean { + return this.workerIncarnation === workerIncarnation; } close(): void { this.rejectAll(new Error("Daemon worker client closed")); this.channel?.close(); this.channel = undefined; + this.workerIncarnation = undefined; this.socket?.destroy(); this.socket = undefined; } @@ -230,6 +242,7 @@ export class DaemonWorkerClient { } this.socket = undefined; this.channel = undefined; + this.workerIncarnation = undefined; this.rejectAll(error); for (const listener of [...this.closeListeners]) { listener(error); diff --git a/packages/coding-agent/src/modes/daemon/daemon-worker-protocol.ts b/packages/coding-agent/src/modes/daemon/daemon-worker-protocol.ts index ade1b370a6..2083a2518f 100644 --- a/packages/coding-agent/src/modes/daemon/daemon-worker-protocol.ts +++ b/packages/coding-agent/src/modes/daemon/daemon-worker-protocol.ts @@ -10,9 +10,33 @@ export const DAEMON_WORKER_ROLE_ENV = "PRIME_AGENT_INTERNAL_DAEMON_WORKER"; export const DAEMON_WORKER_TOKEN_ENV = "PRIME_AGENT_INTERNAL_DAEMON_WORKER_TOKEN"; export const DAEMON_WORKER_ACTIVE_SESSION_ID_ENV = "PRIME_AGENT_INTERNAL_DAEMON_WORKER_ACTIVE_SESSION_ID"; export const DAEMON_WORKER_SUPERVISOR_SOCKET_ENV = "PRIME_AGENT_INTERNAL_DAEMON_SUPERVISOR_SOCKET"; +export const DAEMON_WORKER_SUPERVISOR_AGENT_DIR_ENV = "PRIME_AGENT_INTERNAL_DAEMON_SUPERVISOR_AGENT_DIR"; export const DAEMON_WORKER_RECOVERY_JOURNAL_ENV = "PRIME_AGENT_INTERNAL_DAEMON_WORKER_RECOVERY_JOURNAL"; export const DAEMON_WORKER_STARTUP_GATE_FD_ENV = "PRIME_AGENT_INTERNAL_DAEMON_WORKER_STARTUP_GATE_FD"; export const DAEMON_WORKER_STARTUP_GATE_COMMIT = "start\n"; + +export const DAEMON_WORKER_BOOTSTRAP_ENV_KEYS = [ + DAEMON_WORKER_ROLE_ENV, + DAEMON_WORKER_TOKEN_ENV, + DAEMON_WORKER_ACTIVE_SESSION_ID_ENV, + DAEMON_WORKER_SUPERVISOR_SOCKET_ENV, + DAEMON_WORKER_SUPERVISOR_AGENT_DIR_ENV, + DAEMON_WORKER_RECOVERY_JOURNAL_ENV, + DAEMON_WORKER_STARTUP_GATE_FD_ENV, +] as const; + +/** Remove private worker bootstrap authority before an environment crosses a role or process boundary. */ +export function sanitizeDaemonWorkerBootstrapEnvironment( + environment: T, + platform: NodeJS.Platform = process.platform, +): T { + const reserved = new Set(DAEMON_WORKER_BOOTSTRAP_ENV_KEYS); + for (const key of Object.keys(environment)) { + if (reserved.has(key) || (platform === "win32" && reserved.has(key.toUpperCase()))) delete environment[key]; + } + return environment; +} + export type DaemonWorkerLifecycle = "starting" | "ready" | "recovering" | "stopping" | "failed"; export type DaemonWorkerFrameHeader = @@ -37,6 +61,25 @@ export type DaemonWorkerFrameHeader = export type DaemonCreateCommand = Extract; +export interface DaemonWorkerAuthenticationResult { + workerIncarnation: string; +} + +export interface DaemonWorkerAcpMcpOwnerTransferProof { + transactionId: string; + previousOwnerId: string; + nextOwnerId: string; + changed: boolean; + state: "transferred" | "rolled_back"; +} + +export interface DaemonWorkerAcpMcpOwnerTransferRetirement { + transactionId: string; + previousOwnerId: string; + nextOwnerId: string; + status: "retired"; +} + export interface DurableDaemonCreateCommand { type: "create"; sessionPath?: string; @@ -69,6 +112,15 @@ export type DaemonWorkerCommand = supportsExtensionUi?: boolean; } | { id?: string; type: "worker_unsubscribe"; activeSessionId: string } + | { + id?: string; + type: "worker_transfer_acp_mcp_owner"; + activeSessionId: string; + transactionId: string; + action: "transfer" | "query" | "rollback" | "retire"; + previousOwnerId: string; + ownerId: string; + } | { id?: string; type: "worker_archive_and_shutdown" } | { id?: string; @@ -203,6 +255,35 @@ export function requireDaemonWorkerAuthenticationToken(environment: NodeJS.Proce return token; } +export interface DaemonWorkerBootstrapEnvironment { + authenticationToken: string; + activeSessionId?: string; + supervisorSocketPath?: string; + supervisorAgentDir?: string; + recoveryJournalPath?: string; +} + +/** Capture worker bootstrap authority once, before the private variables are scrubbed from process.env. */ +export function readDaemonWorkerBootstrapEnvironment( + environment: NodeJS.ProcessEnv = process.env, +): DaemonWorkerBootstrapEnvironment { + return { + authenticationToken: requireDaemonWorkerAuthenticationToken(environment), + ...(environment[DAEMON_WORKER_ACTIVE_SESSION_ID_ENV] + ? { activeSessionId: environment[DAEMON_WORKER_ACTIVE_SESSION_ID_ENV] } + : {}), + ...(environment[DAEMON_WORKER_SUPERVISOR_SOCKET_ENV] + ? { supervisorSocketPath: environment[DAEMON_WORKER_SUPERVISOR_SOCKET_ENV] } + : {}), + ...(environment[DAEMON_WORKER_SUPERVISOR_AGENT_DIR_ENV] + ? { supervisorAgentDir: environment[DAEMON_WORKER_SUPERVISOR_AGENT_DIR_ENV] } + : {}), + ...(environment[DAEMON_WORKER_RECOVERY_JOURNAL_ENV] + ? { recoveryJournalPath: environment[DAEMON_WORKER_RECOVERY_JOURNAL_ENV] } + : {}), + }; +} + export function isDaemonWorkerFrameHeader(value: unknown): value is DaemonWorkerFrameHeader { if (!value || typeof value !== "object") { return false; diff --git a/packages/coding-agent/src/modes/daemon/owned-session-adoption-buffer.ts b/packages/coding-agent/src/modes/daemon/owned-session-adoption-buffer.ts new file mode 100644 index 0000000000..ffd2d86fa3 --- /dev/null +++ b/packages/coding-agent/src/modes/daemon/owned-session-adoption-buffer.ts @@ -0,0 +1,82 @@ +import { serializeJsonLine } from "../rpc/jsonl.js"; +import { OwnedSessionAdoptionUnavailableError } from "./owned-session-recovery-store.js"; + +export const RECOVERABLE_OWNED_MAX_BUFFERED_FRAMES = 4096; +export const RECOVERABLE_OWNED_MAX_BUFFERED_BYTES = 8 * 1024 * 1024; + +export interface RecoverableOwnedSequenceCursor { + generation: string; + sequence: number; +} + +export function serializeRecoverableOwnedFrame(value: unknown): Buffer { + try { + return Buffer.from(serializeJsonLine(value)); + } catch { + throw new OwnedSessionAdoptionUnavailableError(); + } +} + +export interface RecoverableOwnedSequenceFrame { + payload: Buffer; + cursor?: RecoverableOwnedSequenceCursor; +} + +export function appendRecoverableOwnedFrame( + frames: TFrame[], + bufferedBytes: number, + frame: TFrame, +): number { + if ( + !frame.cursor || + !frame.cursor.generation || + !Number.isSafeInteger(frame.cursor.sequence) || + frame.cursor.sequence < 0 + ) { + throw new OwnedSessionAdoptionUnavailableError(); + } + const duplicate = frames.find( + (existing) => + existing.cursor?.generation === frame.cursor!.generation && + existing.cursor.sequence === frame.cursor!.sequence, + ); + if (duplicate) { + if (!duplicate.payload.equals(frame.payload)) throw new OwnedSessionAdoptionUnavailableError(); + return bufferedBytes; + } + if ( + frames.length >= RECOVERABLE_OWNED_MAX_BUFFERED_FRAMES || + frame.payload.length > RECOVERABLE_OWNED_MAX_BUFFERED_BYTES - bufferedBytes + ) { + throw new OwnedSessionAdoptionUnavailableError(); + } + frames.push(frame); + return bufferedBytes + frame.payload.length; +} + +export function reconcileRecoverableOwnedFrames( + frames: readonly TFrame[], + snapshotCursor: RecoverableOwnedSequenceCursor, +): { frames: TFrame[]; bufferedBytes: number } { + const retained: TFrame[] = []; + for (const frame of frames) { + if (!frame.cursor || frame.cursor.generation !== snapshotCursor.generation) { + throw new OwnedSessionAdoptionUnavailableError(); + } + if (frame.cursor.sequence > snapshotCursor.sequence) retained.push(frame); + } + let expected = snapshotCursor.sequence + 1; + let bufferedBytes = 0; + for (const frame of retained) { + if ( + frame.cursor!.sequence !== expected || + retained.length > RECOVERABLE_OWNED_MAX_BUFFERED_FRAMES || + frame.payload.length > RECOVERABLE_OWNED_MAX_BUFFERED_BYTES - bufferedBytes + ) { + throw new OwnedSessionAdoptionUnavailableError(); + } + expected++; + bufferedBytes += frame.payload.length; + } + return { frames: retained, bufferedBytes }; +} diff --git a/packages/coding-agent/src/modes/daemon/owned-session-recovery-retention.ts b/packages/coding-agent/src/modes/daemon/owned-session-recovery-retention.ts new file mode 100644 index 0000000000..317bb1554d --- /dev/null +++ b/packages/coding-agent/src/modes/daemon/owned-session-recovery-retention.ts @@ -0,0 +1,38 @@ +export const RECOVERABLE_OWNED_NO_LIFECYCLE_RETENTION_MS = 30_000; +export const RECOVERABLE_OWNED_DISCONNECTED_RETENTION_MS = 15 * 60_000; +export const RECOVERABLE_OWNED_CONFIRMATION_RETENTION_MS = 15 * 60_000; +export const RECOVERABLE_OWNED_TERMINAL_RETENTION_MS = 60_000; +export const RECOVERABLE_OWNED_PREPARE_TIMEOUT_MS = 15_000; + +export interface RecoverableOwnedRetentionInput { + now: number; + busy: boolean; + hasLifecycle: boolean; + terminal: boolean; + activeDeadline?: number; +} + +export interface RecoverableOwnedRetention { + retentionMs: number; + activeDeadline?: number; +} + +export function recoverableOwnedRetention(input: RecoverableOwnedRetentionInput): RecoverableOwnedRetention { + if (input.busy || (input.hasLifecycle && !input.terminal)) { + const activeDeadline = input.activeDeadline ?? input.now + RECOVERABLE_OWNED_DISCONNECTED_RETENTION_MS; + return { retentionMs: Math.max(0, activeDeadline - input.now), activeDeadline }; + } + if (input.terminal) { + return { + retentionMs: Math.max( + 0, + Math.min( + RECOVERABLE_OWNED_TERMINAL_RETENTION_MS, + input.activeDeadline ? input.activeDeadline - input.now : RECOVERABLE_OWNED_TERMINAL_RETENTION_MS, + ), + ), + ...(input.activeDeadline !== undefined ? { activeDeadline: input.activeDeadline } : {}), + }; + } + return { retentionMs: RECOVERABLE_OWNED_NO_LIFECYCLE_RETENTION_MS }; +} diff --git a/packages/coding-agent/src/modes/daemon/owned-session-recovery-store.ts b/packages/coding-agent/src/modes/daemon/owned-session-recovery-store.ts new file mode 100644 index 0000000000..967c83af87 --- /dev/null +++ b/packages/coding-agent/src/modes/daemon/owned-session-recovery-store.ts @@ -0,0 +1,664 @@ +import { createHmac, randomBytes, randomUUID, timingSafeEqual } from "node:crypto"; +import { + chmodSync, + closeSync, + fsyncSync, + lstatSync, + mkdirSync, + openSync, + readdirSync, + readFileSync, + renameSync, + unlinkSync, + writeSync, +} from "node:fs"; +import { join } from "node:path"; + +export const OWNED_SESSION_RECOVERY_HANDLE_BYTES = 32; +export const OWNED_SESSION_ADOPTION_UNAVAILABLE = "Recoverable owned session adoption is unavailable"; + +const UUID_SOURCE = "[0-9a-f]{8}-[0-9a-f]{4}-[1-5][0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}"; +const OWNED_RECORD_NAME_PATTERN = new RegExp(`^(${UUID_SOURCE})\\.json$`, "i"); +const OWNED_TEMP_NAME_PATTERN = new RegExp(`^${UUID_SOURCE}\\.json\\.[1-9][0-9]*\\.${UUID_SOURCE}\\.tmp$`, "i"); + +export type OwnedSessionRecoveryPhase = "connected" | "disconnected" | "prepared" | "committing" | "final"; + +interface PersistedOwnedSessionRecoveryRecord { + version: 1; + recordId: string; + supervisorGeneration: string; + phase: OwnedSessionRecoveryPhase; + ownershipGeneration: number; + createRequestIdDigest: string; + createRequestDigest: string; + createRequestReplayRetired: boolean; + currentVerifier: string; + previousVerifier?: string; + adoptionRequestIdDigest?: string; + adoptionRequestDigest?: string; + previousAuthorityDigest?: string; + authorityDigest: string; + expiresAt: number; +} + +interface MemoryOwnedSessionRecoveryRecord { + persisted: PersistedOwnedSessionRecoveryRecord; + authority: TAuthority; + currentHandle: string; + previousHandle?: string; +} + +export interface OwnedSessionRecoveryReceipt { + readonly recordId: string; + readonly authority: TAuthority; + readonly recoveryHandle: string; + readonly ownershipGeneration: number; + readonly phase: OwnedSessionRecoveryPhase; + readonly repeated: boolean; +} + +export class OwnedSessionAdoptionUnavailableError extends Error { + constructor() { + super(OWNED_SESSION_ADOPTION_UNAVAILABLE); + this.name = "OwnedSessionAdoptionUnavailableError"; + } +} + +function canonicalJson(value: unknown): string { + const seen = new WeakSet(); + const normalize = (current: unknown): unknown => { + if (current === null || typeof current === "string" || typeof current === "boolean") return current; + if (typeof current === "number") { + if (!Number.isFinite(current)) throw new TypeError("Recovery requests must contain finite numbers"); + return current; + } + if (current === undefined) return null; + if (typeof current !== "object") throw new TypeError("Recovery requests must be JSON serializable"); + if (seen.has(current)) throw new TypeError("Recovery requests must not contain cycles"); + seen.add(current); + if (Array.isArray(current)) return current.map(normalize); + const output: Record = {}; + for (const key of Object.keys(current as Record).sort()) { + const entry = (current as Record)[key]; + if (entry !== undefined) output[key] = normalize(entry); + } + return output; + }; + return JSON.stringify(normalize(value)); +} + +function isPersistedRecord(value: unknown): value is PersistedOwnedSessionRecoveryRecord { + if (!value || typeof value !== "object") return false; + const record = value as Partial; + return ( + record.version === 1 && + typeof record.recordId === "string" && + typeof record.supervisorGeneration === "string" && + (record.phase === "connected" || + record.phase === "disconnected" || + record.phase === "prepared" || + record.phase === "committing" || + record.phase === "final") && + Number.isSafeInteger(record.ownershipGeneration) && + (record.ownershipGeneration ?? -1) >= 0 && + typeof record.createRequestIdDigest === "string" && + typeof record.createRequestDigest === "string" && + typeof record.createRequestReplayRetired === "boolean" && + typeof record.currentVerifier === "string" && + (record.previousVerifier === undefined || typeof record.previousVerifier === "string") && + (record.adoptionRequestIdDigest === undefined || typeof record.adoptionRequestIdDigest === "string") && + (record.adoptionRequestDigest === undefined || typeof record.adoptionRequestDigest === "string") && + (record.previousAuthorityDigest === undefined || typeof record.previousAuthorityDigest === "string") && + typeof record.authorityDigest === "string" && + Number.isSafeInteger(record.expiresAt) && + (record.expiresAt ?? 0) > 0 + ); +} + +/** + * Private bearer authority for recoverable owned workers. It deliberately has + * no public descriptor or generic command-journal representation. + */ +export class OwnedSessionRecoveryStore { + private readonly secret: Buffer; + private readonly records = new Map>(); + private readonly createRequests = new Map>(); + private readonly retiredCreateRequests = new Set(); + + constructor( + private readonly directory: string, + private readonly supervisorGeneration: string, + options: { secret?: Buffer; now?: () => number } = {}, + ) { + this.secret = options.secret ? Buffer.from(options.secret) : randomBytes(OWNED_SESSION_RECOVERY_HANDLE_BYTES); + if (this.secret.length !== OWNED_SESSION_RECOVERY_HANDLE_BYTES) { + throw new Error("Owned-session recovery secret must be 256 bits"); + } + this.now = options.now ?? Date.now; + this.prepareDirectory(); + this.removeUnusableRecords(); + } + + private readonly now: () => number; + + private prepareDirectory(): void { + let entry: ReturnType; + try { + entry = lstatSync(this.directory); + } catch (error) { + if ((error as NodeJS.ErrnoException).code !== "ENOENT") throw error; + mkdirSync(this.directory, { recursive: true, mode: 0o700 }); + entry = lstatSync(this.directory); + } + if (entry.isSymbolicLink() || !entry.isDirectory()) { + throw new Error("Owned-session recovery path must be a private directory"); + } + chmodSync(this.directory, 0o700); + const verified = lstatSync(this.directory); + if (verified.isSymbolicLink() || !verified.isDirectory()) { + throw new Error("Owned-session recovery path changed during startup"); + } + } + + digestRequest(value: unknown): string { + return this.mac("request", canonicalJson(value)); + } + + digestAuthority(value: unknown): string { + return this.mac("authority", canonicalJson(value)); + } + + create(input: { + requestIdDigest: string; + requestDigest: string; + authorityDigest: string; + authority: TAuthority; + expiresAt: number; + }): OwnedSessionRecoveryReceipt { + if (this.retiredCreateRequests.has(input.requestIdDigest)) { + throw new OwnedSessionAdoptionUnavailableError(); + } + const previous = this.createRequests.get(input.requestIdDigest); + if (previous) { + this.assertUsable(previous); + if ( + previous.persisted.createRequestDigest !== input.requestDigest || + previous.persisted.authorityDigest !== input.authorityDigest + ) { + throw new OwnedSessionAdoptionUnavailableError(); + } + return this.receipt(previous, false); + } + if (!Number.isSafeInteger(input.expiresAt) || input.expiresAt <= this.now()) { + throw new OwnedSessionAdoptionUnavailableError(); + } + const recordId = randomUUID(); + const currentHandle = randomBytes(OWNED_SESSION_RECOVERY_HANDLE_BYTES).toString("base64url"); + const record: MemoryOwnedSessionRecoveryRecord = { + persisted: { + version: 1, + recordId, + supervisorGeneration: this.supervisorGeneration, + phase: "connected", + ownershipGeneration: 0, + createRequestIdDigest: input.requestIdDigest, + createRequestDigest: input.requestDigest, + createRequestReplayRetired: false, + currentVerifier: this.handleVerifier(currentHandle), + authorityDigest: input.authorityDigest, + expiresAt: input.expiresAt, + }, + authority: input.authority, + currentHandle, + }; + try { + this.persist(record); + } catch (error) { + try { + this.unlinkOwnedRegularFile(this.pathFor(recordId)); + this.fsyncDirectory(); + } catch { + // The supervisor contains the exact new worker; preserve the persistence failure. + } + throw error; + } + this.records.set(recordId, record); + this.createRequests.set(input.requestIdDigest, record); + return this.receipt(record, false); + } + + getByHandle(recoveryHandle: string): OwnedSessionRecoveryReceipt { + const match = this.findHandle(recoveryHandle); + if (!match) throw new OwnedSessionAdoptionUnavailableError(); + this.assertUsable(match.record); + return this.receipt(match.record, true); + } + + getByCreateRequest( + requestIdDigest: string, + requestDigest: string, + ): OwnedSessionRecoveryReceipt | undefined { + const record = this.createRequests.get(requestIdDigest); + if (!record) { + if (this.retiredCreateRequests.has(requestIdDigest)) throw new OwnedSessionAdoptionUnavailableError(); + return undefined; + } + this.assertUsable(record); + if ( + record.persisted.createRequestReplayRetired || + record.persisted.ownershipGeneration !== 0 || + record.persisted.createRequestDigest !== requestDigest + ) { + throw new OwnedSessionAdoptionUnavailableError(); + } + return this.receipt(record, true); + } + + beginAdoption(input: { + recoveryHandle: string; + requestIdDigest: string; + requestDigest: string; + authorityDigest: string; + expectedSupervisorGeneration: string; + expiresAt: number; + }): OwnedSessionRecoveryReceipt { + if (input.expectedSupervisorGeneration !== this.supervisorGeneration) { + throw new OwnedSessionAdoptionUnavailableError(); + } + const match = this.findHandle(input.recoveryHandle); + if (!match) throw new OwnedSessionAdoptionUnavailableError(); + const { record, kind } = match; + this.assertUsable(record); + const repeated = + record.persisted.adoptionRequestIdDigest === input.requestIdDigest && + record.persisted.adoptionRequestDigest === input.requestDigest && + (kind === "previous" || kind === "current"); + const authorityMatches = + record.persisted.authorityDigest === input.authorityDigest || + (repeated && record.persisted.previousAuthorityDigest === input.authorityDigest); + if (!authorityMatches || (kind === "previous" && !repeated)) { + throw new OwnedSessionAdoptionUnavailableError(); + } + if (!repeated) { + record.persisted.createRequestReplayRetired = true; + this.retiredCreateRequests.add(record.persisted.createRequestIdDigest); + const previousHandle = record.currentHandle; + const nextHandle = this.nextHandle(record, input.requestDigest); + record.previousHandle = previousHandle; + record.currentHandle = nextHandle; + record.persisted.previousVerifier = record.persisted.currentVerifier; + record.persisted.currentVerifier = this.handleVerifier(nextHandle); + record.persisted.adoptionRequestIdDigest = input.requestIdDigest; + record.persisted.adoptionRequestDigest = input.requestDigest; + record.persisted.ownershipGeneration++; + } + if (!(repeated && record.persisted.phase === "final")) { + record.persisted.phase = "prepared"; + record.persisted.expiresAt = input.expiresAt; + this.persist(record); + } + if (record.persisted.createRequestReplayRetired) { + this.retiredCreateRequests.add(record.persisted.createRequestIdDigest); + if (this.createRequests.get(record.persisted.createRequestIdDigest) === record) { + this.createRequests.delete(record.persisted.createRequestIdDigest); + } + } + return this.receipt(record, repeated); + } + + replaceAuthority(recordId: string, authorityDigest: string, authority: TAuthority): void { + const record = this.requireRecord(recordId); + record.persisted.previousAuthorityDigest = record.persisted.authorityDigest; + record.persisted.authorityDigest = authorityDigest; + record.authority = authority; + this.persist(record); + } + + markConnected( + recordId: string, + authorityDigest: string, + expiresAt: number, + ): OwnedSessionRecoveryReceipt { + const record = this.requireRecord(recordId); + if (record.persisted.authorityDigest !== authorityDigest || record.persisted.phase === "final") { + throw new OwnedSessionAdoptionUnavailableError(); + } + this.restorePreviousHandle(record); + record.persisted.phase = "connected"; + record.persisted.expiresAt = expiresAt; + this.persist(record); + return this.receipt(record, true); + } + + rollbackAdoption( + recordId: string, + authorityDigest: string, + expiresAt: number, + ): OwnedSessionRecoveryReceipt { + const record = this.requireRecord(recordId); + if ( + record.persisted.authorityDigest !== authorityDigest || + record.persisted.phase === "final" || + record.persisted.phase === "connected" + ) { + throw new OwnedSessionAdoptionUnavailableError(); + } + this.restorePreviousHandle(record); + record.persisted.phase = "disconnected"; + record.persisted.expiresAt = expiresAt; + this.persist(record); + return this.receipt(record, true); + } + + markDisconnected(recordId: string, expiresAt: number): void { + const record = this.requireRecord(recordId); + if (record.persisted.phase !== "final") record.persisted.phase = "disconnected"; + record.persisted.expiresAt = expiresAt; + this.persist(record); + } + + markCommitting(recordId: string): void { + const record = this.requireRecord(recordId); + if (record.persisted.phase !== "prepared") throw new OwnedSessionAdoptionUnavailableError(); + record.persisted.phase = "committing"; + this.persist(record); + } + + markFinal(recordId: string, expiresAt: number): OwnedSessionRecoveryReceipt { + const record = this.requireRecord(recordId); + if (record.persisted.phase !== "committing" && record.persisted.phase !== "final") { + throw new OwnedSessionAdoptionUnavailableError(); + } + record.persisted.phase = "final"; + record.persisted.expiresAt = expiresAt; + this.persist(record); + return this.receipt(record, true); + } + + getForCommit( + recordId: string, + recoveryHandle: string, + requestIdDigest: string, + ): OwnedSessionRecoveryReceipt { + const record = this.requireRecord(recordId); + const match = this.findHandle(recoveryHandle); + if ( + !match || + match.kind !== "current" || + match.record !== record || + record.persisted.phase !== "prepared" || + record.persisted.adoptionRequestIdDigest !== requestIdDigest + ) { + throw new OwnedSessionAdoptionUnavailableError(); + } + return this.receipt(record, true); + } + + getForConfirmation(recoveryHandle: string, requestIdDigest: string): OwnedSessionRecoveryReceipt { + const match = this.findHandle(recoveryHandle); + if ( + !match || + match.kind !== "current" || + (match.record.persisted.phase !== "final" && match.record.persisted.phase !== "connected") || + match.record.persisted.adoptionRequestIdDigest !== requestIdDigest + ) { + throw new OwnedSessionAdoptionUnavailableError(); + } + this.assertUsable(match.record); + return this.receipt(match.record, true); + } + + confirm(input: { + recoveryHandle: string; + requestIdDigest: string; + authorityDigest: string; + expiresAt: number; + }): OwnedSessionRecoveryReceipt { + const match = this.findHandle(input.recoveryHandle); + if (!match || match.kind !== "current") throw new OwnedSessionAdoptionUnavailableError(); + const record = match.record; + this.assertUsable(record); + if ( + (record.persisted.phase !== "final" && record.persisted.phase !== "connected") || + record.persisted.adoptionRequestIdDigest !== input.requestIdDigest || + record.persisted.authorityDigest !== input.authorityDigest + ) { + throw new OwnedSessionAdoptionUnavailableError(); + } + if (!Number.isSafeInteger(input.expiresAt) || input.expiresAt <= this.now()) { + throw new OwnedSessionAdoptionUnavailableError(); + } + record.previousHandle = undefined; + record.persisted.previousVerifier = undefined; + record.persisted.previousAuthorityDigest = undefined; + record.persisted.phase = "connected"; + record.persisted.expiresAt = input.expiresAt; + this.persist(record); + return this.receipt(record, true); + } + + get(recordId: string): OwnedSessionRecoveryReceipt | undefined { + const record = this.records.get(recordId); + if (!record) return undefined; + this.assertUsable(record); + return this.receipt(record, true); + } + + remove(recordId: string): void { + const record = this.records.get(recordId); + if (!record) return; + this.unlinkOwnedRegularFile(this.pathFor(recordId)); + this.fsyncDirectory(); + this.records.delete(recordId); + if (this.createRequests.get(record.persisted.createRequestIdDigest) === record) { + this.createRequests.delete(record.persisted.createRequestIdDigest); + } + } + + sweep(): string[] { + const removed: string[] = []; + for (const [recordId, record] of this.records) { + if (record.persisted.expiresAt > this.now()) continue; + removed.push(recordId); + this.remove(recordId); + } + return removed; + } + + private receipt( + record: MemoryOwnedSessionRecoveryRecord, + repeated: boolean, + ): OwnedSessionRecoveryReceipt { + return { + recordId: record.persisted.recordId, + authority: record.authority, + recoveryHandle: record.currentHandle, + ownershipGeneration: record.persisted.ownershipGeneration, + phase: record.persisted.phase, + repeated, + }; + } + + private restorePreviousHandle(record: MemoryOwnedSessionRecoveryRecord): void { + if (record.previousHandle && record.persisted.previousVerifier) { + record.currentHandle = record.previousHandle; + record.persisted.currentVerifier = record.persisted.previousVerifier; + record.persisted.ownershipGeneration = Math.max(0, record.persisted.ownershipGeneration - 1); + } + record.previousHandle = undefined; + record.persisted.previousVerifier = undefined; + record.persisted.previousAuthorityDigest = undefined; + record.persisted.adoptionRequestIdDigest = undefined; + record.persisted.adoptionRequestDigest = undefined; + } + + private requireRecord(recordId: string): MemoryOwnedSessionRecoveryRecord { + const record = this.records.get(recordId); + if (!record) throw new OwnedSessionAdoptionUnavailableError(); + this.assertUsable(record); + return record; + } + + private assertUsable(record: MemoryOwnedSessionRecoveryRecord): void { + if ( + record.persisted.supervisorGeneration !== this.supervisorGeneration || + record.persisted.expiresAt <= this.now() + ) { + this.remove(record.persisted.recordId); + throw new OwnedSessionAdoptionUnavailableError(); + } + } + + private findHandle( + handle: string, + ): { record: MemoryOwnedSessionRecoveryRecord; kind: "current" | "previous" } | undefined { + if (!/^[A-Za-z0-9_-]{43}$/.test(handle)) return undefined; + const verifier = this.handleVerifier(handle); + let match: { record: MemoryOwnedSessionRecoveryRecord; kind: "current" | "previous" } | undefined; + for (const record of this.records.values()) { + if (this.equalVerifier(verifier, record.persisted.currentVerifier)) match ??= { record, kind: "current" }; + if (record.persisted.previousVerifier && this.equalVerifier(verifier, record.persisted.previousVerifier)) { + match ??= { record, kind: "previous" }; + } + } + return match; + } + + private nextHandle(record: MemoryOwnedSessionRecoveryRecord, requestDigest: string): string { + return createHmac("sha256", this.secret) + .update("next-handle\0") + .update(record.persisted.recordId) + .update("\0") + .update(record.persisted.currentVerifier) + .update("\0") + .update(requestDigest) + .digest("base64url"); + } + + private handleVerifier(handle: string): string { + return this.mac("handle", handle); + } + + private mac(domain: string, value: string): string { + return createHmac("sha256", this.secret).update(domain).update("\0").update(value).digest("base64url"); + } + + private equalVerifier(left: string, right: string): boolean { + const leftBytes = Buffer.from(left); + const rightBytes = Buffer.from(right); + return leftBytes.length === rightBytes.length && timingSafeEqual(leftBytes, rightBytes); + } + + private persist(record: MemoryOwnedSessionRecoveryRecord): void { + const path = this.pathFor(record.persisted.recordId); + const tempPath = `${path}.${process.pid}.${randomUUID()}.tmp`; + let descriptor: number | undefined; + let tempExists = false; + try { + descriptor = openSync(tempPath, "wx", 0o600); + tempExists = true; + writeSync(descriptor, `${JSON.stringify(record.persisted)}\n`); + fsyncSync(descriptor); + closeSync(descriptor); + descriptor = undefined; + chmodSync(tempPath, 0o600); + this.assertReplaceableRecordPath(path); + renameSync(tempPath, path); + tempExists = false; + this.fsyncDirectory(); + } finally { + if (descriptor !== undefined) { + try { + closeSync(descriptor); + } catch { + // Preserve the persistence failure that triggered this best-effort cleanup. + } + } + if (tempExists) { + try { + this.unlinkOwnedRegularFile(tempPath); + } catch { + // Preserve the persistence failure that triggered this best-effort cleanup. + } + } + } + } + + private removeUnusableRecords(): void { + let changed = false; + for (const name of readdirSync(this.directory)) { + const path = join(this.directory, name); + let entry: ReturnType; + try { + entry = lstatSync(path); + } catch (error) { + if ((error as NodeJS.ErrnoException).code === "ENOENT") continue; + throw error; + } + if (entry.isSymbolicLink() || !entry.isFile()) continue; + if (OWNED_TEMP_NAME_PATTERN.test(name)) { + unlinkSync(path); + changed = true; + continue; + } + const recordName = OWNED_RECORD_NAME_PATTERN.exec(name); + if (!recordName) continue; + let parsed: unknown; + try { + parsed = JSON.parse(readFileSync(path, "utf8")); + } catch { + unlinkSync(path); + changed = true; + continue; + } + if ( + !isPersistedRecord(parsed) || + parsed.recordId !== recordName[1] || + parsed.supervisorGeneration !== this.supervisorGeneration + ) { + unlinkSync(path); + changed = true; + } + } + if (changed) this.fsyncDirectory(); + } + + private assertReplaceableRecordPath(path: string): void { + try { + const entry = lstatSync(path); + if (entry.isSymbolicLink() || !entry.isFile()) { + throw new Error("Owned-session recovery record path is not a regular file"); + } + } catch (error) { + if ((error as NodeJS.ErrnoException).code === "ENOENT") return; + throw error; + } + } + + private unlinkOwnedRegularFile(path: string): void { + let entry: ReturnType; + try { + entry = lstatSync(path); + } catch (error) { + if ((error as NodeJS.ErrnoException).code === "ENOENT") return; + throw error; + } + if (entry.isSymbolicLink() || !entry.isFile()) { + throw new Error("Owned-session recovery entry is not a regular file"); + } + unlinkSync(path); + } + + private pathFor(recordId: string): string { + return join(this.directory, `${recordId}.json`); + } + + private fsyncDirectory(): void { + const descriptor = openSync(this.directory, "r"); + try { + fsyncSync(descriptor); + } finally { + closeSync(descriptor); + } + } +} diff --git a/packages/coding-agent/src/modes/index.ts b/packages/coding-agent/src/modes/index.ts index d19a6bf3ea..7cf4034ad5 100644 --- a/packages/coding-agent/src/modes/index.ts +++ b/packages/coding-agent/src/modes/index.ts @@ -6,6 +6,7 @@ export { CALLER_OWNED_SESSION_ENVIRONMENT_CLEANUP_FEATURE, PRIME_AGENT_SDK_FEATURES, type PrimeAgentSdkFeature, + RECOVERABLE_OWNED_SESSION_ADOPTION_FEATURE, } from "../sdk-features.js"; export { type AcpModeOptions, @@ -38,8 +39,21 @@ export type { DaemonOwnedSessionDaemonIdentity, DaemonOwnedSessionDisposeOptions, DaemonOwnedSessionDisposeResult, + DaemonRecoverableOwnedSessionAdoptionOptions, + DaemonRecoverableOwnedSessionAdoptionResult, +} from "./agent-connection/index.js"; +export { + adoptRecoverableOwnedSession, + confirmRecoverableOwnedSessionAdoption, + createRecoverableOwnedSession, + DaemonAgentConnection, + InProcessAgentConnection, + type RecoverableOwnedSessionAdoption, + type RecoverableOwnedSessionAdoptionConfirmation, + type RecoverableOwnedSessionAdoptionOptions, + type RecoverableOwnedSessionCreateOptions, + type RecoverableOwnedSessionCreation, } from "./agent-connection/index.js"; -export { DaemonAgentConnection, InProcessAgentConnection } from "./agent-connection/index.js"; export { type AgentsViewModeOptions, runAgentsViewMode } from "./agents-view/agents-view-mode.js"; export { type AgentsViewRow, @@ -90,6 +104,7 @@ export type { DaemonCommand, DaemonCommandEnvelope, DaemonCommandId, + DaemonEventCursor, DaemonEventEnvelope, DaemonEventId, DaemonEventMeta, @@ -102,6 +117,10 @@ export type { DaemonProtocolInfo, DaemonProtocolName, DaemonProtocolVersion, + DaemonRecoverableOwnedSessionAdoptionProof, + DaemonRecoverableOwnedSessionConfirmResult, + DaemonRecoverableOwnedSessionCreateResult, + DaemonRecoverableOwnedSessionPrepareResult, DaemonReplayInfo, DaemonReplayStatus, DaemonResponse, diff --git a/packages/coding-agent/src/sdk-features.ts b/packages/coding-agent/src/sdk-features.ts index 50b73610e5..423e57db49 100644 --- a/packages/coding-agent/src/sdk-features.ts +++ b/packages/coding-agent/src/sdk-features.ts @@ -5,11 +5,13 @@ * protocol versions, schema revisions, or proof about a remote peer. */ export const CALLER_OWNED_SESSION_ENVIRONMENT_CLEANUP_FEATURE = "caller_owned_session_environment_cleanup_v1" as const; +export const RECOVERABLE_OWNED_SESSION_ADOPTION_FEATURE = "recoverable_owned_session_adoption_v1" as const; export const PRIME_AGENT_SDK_FEATURES = Object.freeze([ "bounded_daemon_ingress_v1", "negotiated_daemon_session_capabilities_v1", CALLER_OWNED_SESSION_ENVIRONMENT_CLEANUP_FEATURE, + RECOVERABLE_OWNED_SESSION_ADOPTION_FEATURE, ] as const); export type PrimeAgentSdkFeature = (typeof PRIME_AGENT_SDK_FEATURES)[number]; diff --git a/packages/coding-agent/test/agent-connection-daemon.test.ts b/packages/coding-agent/test/agent-connection-daemon.test.ts index 85deb0bd30..84b8f93c60 100644 --- a/packages/coding-agent/test/agent-connection-daemon.test.ts +++ b/packages/coding-agent/test/agent-connection-daemon.test.ts @@ -9,6 +9,11 @@ import { DaemonAgentConnection, type DaemonOwnedSessionContractProof, } from "../src/modes/agent-connection/daemon-agent-connection.js"; +import { + adoptRecoverableOwnedSession, + confirmRecoverableOwnedSessionAdoption, + type RecoverableOwnedSessionAdoptionOptions, +} from "../src/modes/agent-connection/recoverable-owned-session.js"; import type { AgentConnectionEvent, AgentConnectionRlmChildAgentSnapshot, @@ -32,8 +37,13 @@ import { type DaemonAttachResult, type DaemonCommand, type DaemonOutbound, + type DaemonRecoverableOwnedSessionPrepareResult, type DaemonResponse, } from "../src/modes/daemon/daemon-protocol.js"; +import { + RECOVERABLE_OWNED_MAX_BUFFERED_BYTES, + RECOVERABLE_OWNED_MAX_BUFFERED_FRAMES, +} from "../src/modes/daemon/owned-session-adoption-buffer.js"; class FakeDaemonClient { readonly requests: DaemonCommand[] = []; @@ -54,6 +64,14 @@ class FakeDaemonClient { reconnectCount = 0; resetTransportCount = 0; reconnectError: Error | undefined; + recoverablePrepareResultFactory: + | (( + command: Extract, + ) => DaemonRecoverableOwnedSessionPrepareResult) + | undefined; + recoverableCommitEvents: DaemonOutbound[] = []; + recoverableTransportError?: Error; + recoverableResponseError?: string; attachFailures = 0; attachError: Error | undefined; connectionStateGate: Promise | undefined; @@ -104,6 +122,12 @@ class FakeDaemonClient { this.requests.push(command); this.requestTimeouts.push(timeoutMs); this.requestOptions.push(options); + if (command.type.includes("recoverable_owned_session")) { + if (this.recoverableTransportError) throw this.recoverableTransportError; + if (this.recoverableResponseError) { + return { type: "response", command: command.type, success: false, error: this.recoverableResponseError }; + } + } switch (command.type) { case "prompt": if (this.promptGate) await this.promptGate; @@ -626,6 +650,19 @@ class FakeDaemonClient { success: true, data: this.ownedSessionCompletionStatus === "invalid" ? { status: "invalid" } : { status: "completed" }, }; + case "prepare_recoverable_owned_session_adoption": + if (!this.recoverablePrepareResultFactory) throw new Error("Missing recoverable prepare fixture"); + return { + type: "response", + command: command.type, + success: true, + data: this.recoverablePrepareResultFactory(command), + }; + case "commit_recoverable_owned_session_adoption": + for (const message of this.recoverableCommitEvents) this.emitMessage(message); + return { type: "response", command: command.type, success: true, data: command.proof }; + case "confirm_recoverable_owned_session_adoption": + return { type: "response", command: command.type, success: true, data: { status: "confirmed" } }; default: throw new Error(`Unexpected command: ${command.type}`); } @@ -667,6 +704,10 @@ class FakeDaemonClient { enableRequestRecovery(): void {} + get clientId(): string { + return "fake-protocol-client"; + } + getTransportGeneration(): number { return this.transportGeneration; } @@ -1026,6 +1067,500 @@ function emitSequencedQueueUpdate(client: FakeDaemonClient, activeSessionId: str } describe("DaemonAgentConnection", () => { + it("stages recoverable adoption across exact frame replay and unrelated shared-client traffic", async () => { + const fakeClient = new FakeDaemonClient(); + fakeClient.hello = { + ...fakeClient.hello!, + schemaRevision: 30, + supervisorGeneration: SUPERVISOR_GENERATION_A, + }; + for (const capability of [ + "daemon_recoverable_owned_session_adoption_v1", + "caller_owned_session_environment_cleanup_v1", + "authoritative_owned_session_cleanup_v1", + "attach_snapshot", + "event_sequence", + "slim_attach", + "correlated_prompt_lifecycle_v1", + "client_owned_sessions", + ]) + fakeClient.serverCapabilities.add(capability); + const activeSessionId = "recover-active"; + const sessionId = "recover-session"; + const correlationId = "recover-correlation"; + const lifecycle: PromptLifecycleSnapshot = { + correlationId, + phase: "queued", + kind: "model_prompt", + revision: 1, + deliveryCrossed: false, + }; + fakeClient.recoverablePrepareResultFactory = (command) => { + const attached = createAttachResult(activeSessionId, command.clientId, command.capabilities, 5, { + state: createConnectionState(activeSessionId, sessionId), + }); + attached.snapshot.promptLifecycles = { records: [lifecycle], expired: [] }; + return { + ...attached, + recoveryHandle: "A".repeat(43), + proof: { + feature: "recoverable_owned_session_adoption_v1", + status: "adopted", + supervisorGeneration: SUPERVISOR_GENERATION_A, + ownershipGeneration: 1, + activeSessionId, + sessionId, + correlationId, + lifecycle, + cursor: { generation: `generation-${activeSessionId}`, sequence: 5 }, + mcpOwnerId: "next-mcp-owner", + }, + }; + }; + const lifecycleEvent = ( + sequence: number, + phase: "delivered" | "completed", + revision: number, + ): DaemonOutbound => ({ + type: "prompt_lifecycle", + activeSessionId, + lifecycle: { ...lifecycle, phase, revision, deliveryCrossed: true }, + meta: { + id: `event-${sequence}`, + protocol: DAEMON_PROTOCOL_INFO, + activeSessionId, + sequence, + emittedAt: "2026-01-01T00:00:00.000Z", + cursor: { generation: `generation-${activeSessionId}`, sequence }, + }, + }); + const delivered = lifecycleEvent(6, "delivered", 2); + const completed = lifecycleEvent(7, "completed", 3); + fakeClient.recoverableCommitEvents = [ + delivered, + { + type: "session_event", + activeSessionId: "concurrent-other-session", + event: { type: "session_action_update", actions: { queuedCount: 0, steering: [], followUps: [] } }, + meta: { + id: "concurrent-other-session:1", + protocol: DAEMON_PROTOCOL_INFO, + activeSessionId: "concurrent-other-session", + sequence: 1, + cursor: { generation: "generation-concurrent-other-session", sequence: 1 }, + emittedAt: "2026-01-01T00:00:00.000Z", + }, + }, + completed, + delivered, + completed, + ]; + const adopted = await adoptRecoverableOwnedSession(fakeClient as unknown as DaemonClient, { + requestId: "00112233445566778899aabbccddeeff", + recoveryHandle: "B".repeat(43), + expectedSupervisorGeneration: SUPERVISOR_GENERATION_A, + activeSessionId, + sessionId, + correlationId, + cursor: { generation: `generation-${activeSessionId}`, sequence: 4 }, + previousMcpOwnerId: "previous-mcp-owner", + mcpOwnerId: "next-mcp-owner", + config: { cwd: "/tmp/project" }, + launchEnv: { PATH: "/caller/bin" }, + connectionOptions: { supportsExtensionUi: false }, + }); + await new Promise((resolveImmediate) => setImmediate(resolveImmediate)); + const snapshot = await adopted.connection.getInitialSnapshot(); + expect(snapshot.lastEventCursor).toEqual({ generation: `generation-${activeSessionId}`, sequence: 7 }); + expect(snapshot.promptLifecycles?.records).toEqual([ + expect.objectContaining({ correlationId, phase: "completed", revision: 3 }), + ]); + const attachedContractProof = adopted.connection.getOwnedSessionContractProof(); + expect(attachedContractProof).toMatchObject({ + feature: "caller_owned_session_environment_cleanup_v1", + status: "attached", + daemon: { supervisorGeneration: SUPERVISOR_GENERATION_A }, + }); + expect(Object.isFrozen(attachedContractProof)).toBe(true); + const prepareCommand = fakeClient.requests[0]; + expect(prepareCommand?.type).toBe("prepare_recoverable_owned_session_adoption"); + if (prepareCommand?.type === "prepare_recoverable_owned_session_adoption") { + expect(prepareCommand.capabilities).toEqual( + expect.arrayContaining([ + "event_sequence", + "correlated_prompt_lifecycle_v1", + "client_owned_sessions", + "caller_owned_session_environment_cleanup_v1", + ]), + ); + } + expect(fakeClient.requests.map((request) => request.type)).toEqual([ + "prepare_recoverable_owned_session_adoption", + "commit_recoverable_owned_session_adoption", + ]); + await confirmRecoverableOwnedSessionAdoption(fakeClient as unknown as DaemonClient, { + requestId: "00112233445566778899aabbccddeeff", + recoveryHandle: adopted.recoveryHandle, + proof: adopted.proof, + }); + expect(fakeClient.requests.at(-1)?.type).toBe("confirm_recoverable_owned_session_adoption"); + const cleanup = await adopted.connection.disposeOwnedSession(); + expect(cleanup).toMatchObject({ + feature: "caller_owned_session_environment_cleanup_v1", + status: "completed", + started: attachedContractProof, + daemonReplaced: false, + }); + expect(Object.isFrozen(cleanup)).toBe(true); + }); + + it("requires exact-generation complete creation attachment proofs and cleans up fenced failures", async () => { + const makeClient = () => { + const client = new FakeDaemonClient(); + client.hello = { + ...client.hello!, + schemaRevision: DAEMON_SCHEMA_REVISION, + supervisorGeneration: SUPERVISOR_GENERATION_A, + }; + for (const capability of [ + "caller_owned_session_environment_cleanup_v1", + "authoritative_owned_session_cleanup_v1", + "attach_snapshot", + "event_sequence", + "slim_attach", + "correlated_prompt_lifecycle_v1", + "client_owned_sessions", + ]) + client.serverCapabilities.add(capability); + return client; + }; + const options = { + ownedSession: true, + ownedSessionRecoveryConfig: { cwd: "/tmp/project" }, + ownedSessionLaunchEnv: { PATH: "/caller/bin" }, + }; + + const partialReplay = makeClient(); + partialReplay.attachResultFactory = (command) => { + const result = createAttachResult(command.activeSessionId, command.clientId, command.capabilities, 2, { + state: createConnectionState(command.activeSessionId, "session-created"), + }); + return { ...result, replay: { ...result.replay, status: "partial" } }; + }; + await expect( + DaemonAgentConnection.attachRecoverableOwnedSessionCreation( + asDaemonClient(partialReplay), + "active-created", + "session-created", + SUPERVISOR_GENERATION_A, + options, + ), + ).rejects.toThrow("Recoverable owned session adoption is unavailable"); + expect(partialReplay.requests.map((request) => request.type)).toContain("complete_owned_session"); + + const missingCapability = makeClient(); + missingCapability.attachResultFactory = (command) => { + const result = createAttachResult(command.activeSessionId, command.clientId, command.capabilities, 2, { + state: createConnectionState(command.activeSessionId, "session-created"), + }); + return { + ...result, + client: { + ...result.client, + capabilities: result.client.capabilities.filter( + (capability) => capability !== "caller_owned_session_environment_cleanup_v1", + ), + }, + }; + }; + await expect( + DaemonAgentConnection.attachRecoverableOwnedSessionCreation( + asDaemonClient(missingCapability), + "active-created", + "session-created", + SUPERVISOR_GENERATION_A, + options, + ), + ).rejects.toThrow("Recoverable owned session adoption is unavailable"); + expect(missingCapability.requests.map((request) => request.type)).toContain("complete_owned_session"); + + const replacedDaemon = makeClient(); + replacedDaemon.attachResultFactory = (command) => { + replacedDaemon.hello = { ...replacedDaemon.hello!, supervisorGeneration: SUPERVISOR_GENERATION_B }; + return createAttachResult(command.activeSessionId, command.clientId, command.capabilities, 2, { + state: createConnectionState(command.activeSessionId, "session-created"), + }); + }; + await expect( + DaemonAgentConnection.attachRecoverableOwnedSessionCreation( + asDaemonClient(replacedDaemon), + "active-created", + "session-created", + SUPERVISOR_GENERATION_A, + options, + ), + ).rejects.toThrow("Recoverable owned session adoption is unavailable"); + expect(replacedDaemon.requests.map((request) => request.type)).toEqual(["attach"]); + }); + + it("refuses recoverable adoption without the hello gate or an exact attached proof", async () => { + const options = { + requestId: "00112233445566778899aabbccddeeff", + recoveryHandle: "B".repeat(43), + expectedSupervisorGeneration: SUPERVISOR_GENERATION_A, + activeSessionId: "recover-active", + sessionId: "recover-session", + correlationId: "recover-correlation", + cursor: { generation: "generation-recover-active", sequence: 4 }, + previousMcpOwnerId: "previous-mcp-owner", + mcpOwnerId: "next-mcp-owner", + config: { cwd: "/tmp/project" }, + launchEnv: { PATH: "/caller/bin" }, + connectionOptions: { supportsExtensionUi: false }, + } as const; + const missingHelloGate = new FakeDaemonClient(); + missingHelloGate.hello = { ...missingHelloGate.hello!, schemaRevision: 30 }; + await expect(adoptRecoverableOwnedSession(missingHelloGate as unknown as DaemonClient, options)).rejects.toThrow( + "Recoverable owned session adoption is unavailable", + ); + expect(missingHelloGate.requests).toEqual([]); + + const invalidProof = new FakeDaemonClient(); + invalidProof.hello = { + ...invalidProof.hello!, + schemaRevision: 30, + supervisorGeneration: SUPERVISOR_GENERATION_A, + }; + for (const capability of [ + "daemon_recoverable_owned_session_adoption_v1", + "caller_owned_session_environment_cleanup_v1", + "authoritative_owned_session_cleanup_v1", + "attach_snapshot", + "event_sequence", + "slim_attach", + "correlated_prompt_lifecycle_v1", + "client_owned_sessions", + ]) + invalidProof.serverCapabilities.add(capability); + invalidProof.recoverablePrepareResultFactory = (command) => { + const attached = createAttachResult(options.activeSessionId, command.clientId, command.capabilities, 5, { + state: createConnectionState(options.activeSessionId, options.sessionId), + }); + const lifecycle: PromptLifecycleSnapshot = { + correlationId: options.correlationId, + phase: "queued", + kind: "model_prompt", + revision: 1, + deliveryCrossed: false, + }; + attached.snapshot.promptLifecycles = { records: [lifecycle], expired: [] }; + return { + ...attached, + recoveryHandle: "A".repeat(43), + proof: { + feature: "recoverable_owned_session_adoption_v1", + status: "adopted", + supervisorGeneration: SUPERVISOR_GENERATION_A, + ownershipGeneration: 1, + activeSessionId: options.activeSessionId, + sessionId: "wrong-session", + correlationId: options.correlationId, + lifecycle, + cursor: { generation: "generation-recover-active", sequence: 5 }, + mcpOwnerId: options.mcpOwnerId, + }, + }; + }; + await expect(adoptRecoverableOwnedSession(invalidProof as unknown as DaemonClient, options)).rejects.toThrow( + "Recoverable owned session adoption is unavailable", + ); + expect(invalidProof.requests.map((request) => request.type)).toEqual([ + "prepare_recoverable_owned_session_adoption", + ]); + }); + + it("refuses malformed, oversized, and over-count adoption frames before commit", async () => { + const activeSessionId = "recover-active"; + const sessionId = "recover-session"; + const correlationId = "recover-correlation"; + const unavailable = "Recoverable owned session adoption is unavailable"; + const options = { + requestId: "00112233445566778899aabbccddeeff", + recoveryHandle: "B".repeat(43), + expectedSupervisorGeneration: SUPERVISOR_GENERATION_A, + activeSessionId, + sessionId, + correlationId, + cursor: { generation: `generation-${activeSessionId}`, sequence: 4 }, + previousMcpOwnerId: "previous-mcp-owner", + mcpOwnerId: "next-mcp-owner", + config: { cwd: "/tmp/project" }, + launchEnv: { PATH: "/caller/bin" }, + connectionOptions: { supportsExtensionUi: false }, + } as const; + const configuredClient = () => { + const client = new FakeDaemonClient(); + client.hello = { + ...client.hello!, + schemaRevision: 30, + supervisorGeneration: SUPERVISOR_GENERATION_A, + }; + for (const capability of [ + "daemon_recoverable_owned_session_adoption_v1", + "caller_owned_session_environment_cleanup_v1", + "authoritative_owned_session_cleanup_v1", + "attach_snapshot", + "event_sequence", + "slim_attach", + "correlated_prompt_lifecycle_v1", + "client_owned_sessions", + ]) + client.serverCapabilities.add(capability); + return client; + }; + const prepareResult = ( + command: Extract, + ) => { + const attached = createAttachResult(activeSessionId, command.clientId, command.capabilities, 5, { + state: createConnectionState(activeSessionId, sessionId), + }); + const lifecycle: PromptLifecycleSnapshot = { + correlationId, + phase: "queued", + kind: "model_prompt", + revision: 1, + deliveryCrossed: false, + }; + attached.snapshot.promptLifecycles = { records: [lifecycle], expired: [] }; + return { + ...attached, + recoveryHandle: "A".repeat(43), + proof: { + feature: "recoverable_owned_session_adoption_v1" as const, + status: "adopted" as const, + supervisorGeneration: SUPERVISOR_GENERATION_A, + ownershipGeneration: 1, + activeSessionId, + sessionId, + correlationId, + lifecycle, + cursor: { generation: `generation-${activeSessionId}`, sequence: 5 }, + mcpOwnerId: options.mcpOwnerId, + }, + }; + }; + const event = (sequence: number, steering = "frame"): DaemonOutbound => ({ + type: "session_event", + activeSessionId, + event: { + type: "session_action_update", + actions: { queuedCount: 0, steering: [steering], followUps: [] }, + }, + meta: { + id: `${activeSessionId}:${sequence}`, + protocol: DAEMON_PROTOCOL_INFO, + activeSessionId, + sequence, + cursor: { generation: `generation-${activeSessionId}`, sequence }, + emittedAt: "2026-01-01T00:00:00.000Z", + }, + }); + + const malformed = configuredClient(); + malformed.recoverablePrepareResultFactory = (command) => { + malformed.emitMessage({ + type: "session_event", + activeSessionId, + event: { type: "session_action_update", actions: { queuedCount: 0, steering: [], followUps: [] } }, + }); + return prepareResult(command); + }; + await expect(adoptRecoverableOwnedSession(malformed as unknown as DaemonClient, options)).rejects.toThrow( + unavailable, + ); + expect(malformed.requests.map((request) => request.type)).toEqual(["prepare_recoverable_owned_session_adoption"]); + + const oversized = configuredClient(); + oversized.recoverablePrepareResultFactory = (command) => { + oversized.emitMessage(event(6, "x".repeat(RECOVERABLE_OWNED_MAX_BUFFERED_BYTES))); + return prepareResult(command); + }; + await expect(adoptRecoverableOwnedSession(oversized as unknown as DaemonClient, options)).rejects.toThrow( + unavailable, + ); + expect(oversized.requests.map((request) => request.type)).toEqual(["prepare_recoverable_owned_session_adoption"]); + + const overCount = configuredClient(); + overCount.recoverablePrepareResultFactory = (command) => { + for (let index = 0; index <= RECOVERABLE_OWNED_MAX_BUFFERED_FRAMES; index++) { + overCount.emitMessage(event(6 + index)); + } + return prepareResult(command); + }; + await expect(adoptRecoverableOwnedSession(overCount as unknown as DaemonClient, options)).rejects.toThrow( + unavailable, + ); + expect(overCount.requests.map((request) => request.type)).toEqual(["prepare_recoverable_owned_session_adoption"]); + }); + + it("normalizes public adoption failures and requires an exact launch environment", async () => { + const client = new FakeDaemonClient(); + client.hello = { + ...client.hello!, + schemaRevision: 30, + supervisorGeneration: SUPERVISOR_GENERATION_A, + }; + client.serverCapabilities.add("daemon_recoverable_owned_session_adoption_v1"); + client.serverCapabilities.add("caller_owned_session_environment_cleanup_v1"); + client.serverCapabilities.add("authoritative_owned_session_cleanup_v1"); + const options: RecoverableOwnedSessionAdoptionOptions = { + requestId: "00112233445566778899aabbccddeeff", + recoveryHandle: "B".repeat(43), + expectedSupervisorGeneration: SUPERVISOR_GENERATION_A, + activeSessionId: "recover-active", + sessionId: "recover-session", + correlationId: "recover-correlation", + cursor: { generation: "generation-recover-active", sequence: 4 }, + previousMcpOwnerId: "previous-mcp-owner", + mcpOwnerId: "next-mcp-owner", + config: { cwd: "/tmp/project" }, + launchEnv: { PRIVATE_CANARY: "launch-environment-canary" }, + }; + const unavailable = "Recoverable owned session adoption is unavailable"; + client.recoverableTransportError = new Error("private-socket-path-canary"); + const transportFailure = adoptRecoverableOwnedSession(client as unknown as DaemonClient, options); + await expect(transportFailure).rejects.toThrow(unavailable); + await expect(transportFailure).rejects.not.toThrow("private-socket-path-canary"); + + client.recoverableTransportError = undefined; + client.recoverableResponseError = "private-registry-stage-canary"; + const responseFailure = adoptRecoverableOwnedSession(client as unknown as DaemonClient, options); + await expect(responseFailure).rejects.toThrow(unavailable); + await expect(responseFailure).rejects.not.toThrow("private-registry-stage-canary"); + + const missingLaunchEnv = { + ...options, + launchEnv: undefined, + } as unknown as RecoverableOwnedSessionAdoptionOptions; + await expect(adoptRecoverableOwnedSession(client as unknown as DaemonClient, missingLaunchEnv)).rejects.toThrow( + unavailable, + ); + // @ts-expect-error launchEnv is a required public authority input. + const compileTimeMissingLaunchEnv: RecoverableOwnedSessionAdoptionOptions = { + requestId: options.requestId, + recoveryHandle: options.recoveryHandle, + expectedSupervisorGeneration: options.expectedSupervisorGeneration, + activeSessionId: options.activeSessionId, + sessionId: options.sessionId, + correlationId: options.correlationId, + cursor: options.cursor, + previousMcpOwnerId: options.previousMcpOwnerId, + mcpOwnerId: options.mcpOwnerId, + config: options.config, + }; + void compileTimeMissingLaunchEnv; + }); + it("carries an opt-out-only telemetry policy on attach", async () => { const fakeClient = new FakeDaemonClient(); const connection = new DaemonAgentConnection(asDaemonClient(fakeClient), "active-1", { diff --git a/packages/coding-agent/test/command-recovery-journal.test.ts b/packages/coding-agent/test/command-recovery-journal.test.ts index 7c42732a03..b7fe426063 100644 --- a/packages/coding-agent/test/command-recovery-journal.test.ts +++ b/packages/coding-agent/test/command-recovery-journal.test.ts @@ -19,6 +19,20 @@ describe("CommandRecoveryJournal", () => { return join(root, "commands.jsonl"); } + it("rejects bearer-returning recovery commands instead of persisting their responses", () => { + const journal = new CommandRecoveryJournal(createPath()); + for (const command of [ + "create_recoverable_owned_session", + "prepare_recoverable_owned_session_adoption", + "commit_recoverable_owned_session_adoption", + "confirm_recoverable_owned_session_adoption", + ]) { + expect(() => journal.begin("client-a", `command-${command}`, command)).toThrow( + "rejects bearer-returning command", + ); + } + }); + it("marks received commands uncertain instead of replaying them", () => { const journal = new CommandRecoveryJournal(createPath()); expect(journal.begin("client-a", "command-a", "prompt")).toEqual({ status: "new" }); diff --git a/packages/coding-agent/test/daemon-client.test.ts b/packages/coding-agent/test/daemon-client.test.ts index 31c38a9b46..c048ce1999 100644 --- a/packages/coding-agent/test/daemon-client.test.ts +++ b/packages/coding-agent/test/daemon-client.test.ts @@ -108,6 +108,7 @@ function emitHello( version = DAEMON_PROTOCOL_VERSION, serverCapabilities: string[] = ["session_input_admission"], schemaRevision?: number, + supervisorGeneration = "supervisor-generation", ): void { socket.emit( "data", @@ -116,6 +117,7 @@ function emitHello( socketPath: "/tmp/prime-agent.sock", protocol: { name: "prime-agent.daemon", version }, schemaRevision, + supervisorGeneration, appVersion: "9.9.9", clientId: "client-1", serverCapabilities, @@ -213,6 +215,7 @@ describe("DaemonClient", () => { "bounded_daemon_ingress_v1", "negotiated_daemon_session_capabilities_v1", "caller_owned_session_environment_cleanup_v1", + "recoverable_owned_session_adoption_v1", ]; const rootConnectionOptions: RootDaemonAgentConnectionOptions = { ownedSession: true, @@ -230,6 +233,10 @@ describe("DaemonClient", () => { expect(publicSdk.CALLER_OWNED_SESSION_ENVIRONMENT_CLEANUP_FEATURE).toBe( "caller_owned_session_environment_cleanup_v1", ); + expect(publicSdk.RECOVERABLE_OWNED_SESSION_ADOPTION_FEATURE).toBe("recoverable_owned_session_adoption_v1"); + expect(publicSdk.createRecoverableOwnedSession).toEqual(expect.any(Function)); + expect(publicSdk.adoptRecoverableOwnedSession).toEqual(expect.any(Function)); + expect(publicSdk.confirmRecoverableOwnedSessionAdoption).toEqual(expect.any(Function)); expect(rootConnectionOptions.ownedSession).toBe(true); expect(rootConnectionOptions.ownedSessionRecoveryConfig).toEqual({}); expect(rootRequestOptions.recoverAcrossReconnect).toBe(false); @@ -651,6 +658,34 @@ describe("DaemonClient", () => { client.close(); }); + it("requires both schema 30 and the recoverable adoption hello capability before writing", async () => { + const command = { + type: "create_recoverable_owned_session" as const, + requestId: "00112233445566778899aabbccddeeff", + expectedSupervisorGeneration: "supervisor-generation", + correlationId: "correlation", + mcpOwnerId: "mcp-owner", + recoveryConfig: { cwd: "/tmp" }, + config: { cwd: "/tmp" }, + launchEnv: {}, + launchEnvMode: "replace" as const, + }; + for (const [capabilities, revision] of [ + [["daemon_recoverable_owned_session_adoption_v1"], 29], + [[], 30], + ] as const) { + const client = new DaemonClient(`/tmp/prime-agent-recover-${revision}-${capabilities.length}.sock`); + const connect = client.connect(); + const socket = netMock.sockets.at(-1)!; + socket.emit("connect"); + await connect; + emitHello(socket, DAEMON_PROTOCOL_VERSION, [...capabilities], revision); + await expect(client.request(command)).rejects.toThrow("daemon_recoverable_owned_session_adoption_v1"); + expect(socket.writes).toEqual([]); + client.close(); + } + }); + it("gates exact caller-owned launch replacement while preserving legacy launch commands", async () => { const oldClient = new DaemonClient("/tmp/prime-agent-old.sock"); const oldConnect = oldClient.connect(); @@ -1313,6 +1348,102 @@ describe("DaemonClient", () => { client.close(); }); + it("fences a generation-bound recoverable create before reconnect replay", async () => { + const client = new DaemonClient("/tmp/prime-agent-recover-generation.sock"); + client.enableRequestRecovery(); + const firstConnect = client.connect(); + const firstSocket = netMock.sockets[0]!; + firstSocket.emit("connect"); + await firstConnect; + emitHello( + firstSocket, + DAEMON_PROTOCOL_VERSION, + ["daemon_recoverable_owned_session_adoption_v1"], + DAEMON_SCHEMA_REVISION, + "supervisor-a", + ); + + const response = client.request({ + type: "create_recoverable_owned_session", + requestId: "00112233445566778899aabbccddeeff", + expectedSupervisorGeneration: "supervisor-a", + correlationId: "correlation", + mcpOwnerId: "mcp-owner", + recoveryConfig: { cwd: "/tmp" }, + config: { cwd: "/tmp" }, + launchEnv: {}, + launchEnvMode: "replace", + }); + expect(firstSocket.writes).toHaveLength(1); + firstSocket.emit("close"); + + const secondConnect = client.connect(); + const secondSocket = netMock.sockets[1]!; + secondSocket.emit("connect"); + await secondConnect; + emitHello( + secondSocket, + DAEMON_PROTOCOL_VERSION, + ["daemon_recoverable_owned_session_adoption_v1"], + DAEMON_SCHEMA_REVISION, + "supervisor-b", + ); + + await expect(response).rejects.toThrow("Recoverable owned session adoption is unavailable"); + expect(secondSocket.writes).toEqual([]); + client.close(); + }); + + it("fences recoverable commit and confirm before their first write", async () => { + const client = new DaemonClient("/tmp/prime-agent-recover-generation.sock"); + const connect = client.connect(); + const socket = netMock.sockets[0]!; + socket.emit("connect"); + await connect; + emitHello( + socket, + DAEMON_PROTOCOL_VERSION, + ["daemon_recoverable_owned_session_adoption_v1"], + DAEMON_SCHEMA_REVISION, + "supervisor-b", + ); + const proof = { + feature: "recoverable_owned_session_adoption_v1" as const, + status: "adopted" as const, + supervisorGeneration: "supervisor-a", + ownershipGeneration: 1, + activeSessionId: "active", + sessionId: "session", + correlationId: "correlation", + lifecycle: { correlationId: "correlation", expired: true as const, deliveryCrossed: true }, + cursor: { generation: "cursor", sequence: 1 }, + mcpOwnerId: "mcp-owner", + }; + const requestId = "00112233445566778899aabbccddeeff"; + const recoveryHandle = "A".repeat(43); + + await expect( + client.request({ + type: "commit_recoverable_owned_session_adoption", + requestId, + expectedSupervisorGeneration: "supervisor-a", + recoveryHandle, + proof, + }), + ).rejects.toThrow("Recoverable owned session adoption is unavailable"); + await expect( + client.request({ + type: "confirm_recoverable_owned_session_adoption", + requestId, + expectedSupervisorGeneration: "supervisor-a", + recoveryHandle, + proof, + }), + ).rejects.toThrow("Recoverable owned session adoption is unavailable"); + expect(socket.writes).toEqual([]); + client.close(); + }); + it("reconnects raw clients and replays pending commands after supervisor replacement", async () => { const client = new DaemonClient("/tmp/prime-agent.sock"); const firstConnect = client.connect(); diff --git a/packages/coding-agent/test/daemon-protocol.test.ts b/packages/coding-agent/test/daemon-protocol.test.ts index ae3ac362e4..c800066624 100644 --- a/packages/coding-agent/test/daemon-protocol.test.ts +++ b/packages/coding-agent/test/daemon-protocol.test.ts @@ -23,16 +23,72 @@ import { type DaemonCommand, type DaemonOutbound, daemonOutboundForCorrelatedPromptCapability, + daemonSupervisorServerCapabilities, getDaemonCommandCompatibilities, isDaemonCommandEnvelope, isDaemonMutatingCommand, salvageDaemonCommandId, } from "../src/modes/daemon/daemon-protocol.js"; import { + DAEMON_WORKER_BOOTSTRAP_ENV_KEYS, type DaemonWorkerDescriptor, durableDaemonWorkerDescriptor, + sanitizeDaemonWorkerBootstrapEnvironment, } from "../src/modes/daemon/daemon-worker-protocol.js"; -import { CALLER_OWNED_SESSION_ENVIRONMENT_CLEANUP_FEATURE } from "../src/sdk-features.js"; +import { + CALLER_OWNED_SESSION_ENVIRONMENT_CLEANUP_FEATURE, + PRIME_AGENT_SDK_FEATURES, + RECOVERABLE_OWNED_SESSION_ADOPTION_FEATURE, +} from "../src/sdk-features.js"; + +function sliceWireSource(source: string, start: string, end: string): string { + const startIndex = source.indexOf(start); + const endIndex = source.indexOf(end, startIndex); + if (startIndex < 0 || endIndex < 0) throw new Error(`Missing wire schema boundary: ${start} -> ${end}`); + return source.slice(startIndex, endIndex); +} + +function daemonSchemaDigest(daemonSource: string, workerSource: string): string { + const sections = [ + sliceWireSource(daemonSource, "export type DaemonClientCapability", "export type DaemonReplayStatus"), + sliceWireSource( + daemonSource, + "export interface DaemonRecoverableOwnedSessionCreateResult", + "export type DaemonCommand =", + ), + sliceWireSource(daemonSource, "export type DaemonCommand =", "type DaemonCommandName"), + sliceWireSource( + daemonSource, + "const RECOVERABLE_OWNED_SESSION_ADOPTION_COMMAND", + "const AUTHORITATIVE_OWNED_SESSION_CLEANUP_COMMAND", + ), + sliceWireSource( + daemonSource, + "export interface DaemonSavedSessionInfo", + "export type DaemonDeleteSavedSessionResult", + ), + sliceWireSource(daemonSource, "export type DaemonOutbound =", "export const DAEMON_OUTBOUND_COMPATIBILITY"), + sliceWireSource( + daemonSource, + "export type DaemonOwnedSessionCleanupStatus", + "export type DaemonServerCapability", + ), + sliceWireSource(daemonSource, "export type DaemonLaunchEnvMode", "/**\n * The allowlist of env vars"), + sliceWireSource(daemonSource, "export type DaemonErrorInfo =", "export type DaemonSessionClosedReason"), + sliceWireSource(workerSource, "export const DAEMON_WORKER_ROLE_ENV", "export type DaemonWorkerLifecycle"), + sliceWireSource( + workerSource, + "export interface DaemonWorkerAuthenticationResult", + "export interface DaemonWorkerDescriptor", + ), + ]; + return createHash("sha256").update(sections.join("\n")).digest("hex").slice(0, 12); +} + +function replaceWireSentinel(source: string, before: string, after: string): string { + if (!source.includes(before)) throw new Error(`Missing wire mutation sentinel: ${before}`); + return source.replace(before, after); +} describe("daemon protocol helpers", () => { it("serializes worker descriptors as identity-only version 2 state", () => { @@ -158,6 +214,20 @@ describe("daemon protocol helpers", () => { ).toEqual({ KEEP_EXACT: "yes" }); }); + it("centrally strips every private worker bootstrap variable across process roles", () => { + const environment: NodeJS.ProcessEnv = { KEEP_PUBLIC: "yes" }; + for (const key of DAEMON_WORKER_BOOTSTRAP_ENV_KEYS) environment[key] = `private-${key}`; + sanitizeDaemonWorkerBootstrapEnvironment(environment); + expect(environment).toEqual({ KEEP_PUBLIC: "yes" }); + + const windowsEnvironment: NodeJS.ProcessEnv = { + keep_public: "yes", + [DAEMON_WORKER_BOOTSTRAP_ENV_KEYS[0]!.toLowerCase()]: "private-role", + }; + sanitizeDaemonWorkerBootstrapEnvironment(windowsEnvironment, "win32"); + expect(windowsEnvironment).toEqual({ keep_public: "yes" }); + }); + it("capability-gates exact launch replacement without changing legacy commands", () => { const exact = getDaemonCommandCompatibilities({ type: "create", @@ -181,41 +251,80 @@ describe("daemon protocol helpers", () => { expect(DAEMON_SUPERVISOR_SERVER_CAPABILITIES).toContain(CALLER_OWNED_SESSION_ENVIRONMENT_CLEANUP_FEATURE); }); - it("keeps the advertised schema identity synchronized with wire type shapes", () => { - const source = readFileSync(resolve(__dirname, "../src/modes/daemon/daemon-protocol.ts"), "utf8"); - const commandSource = source.slice( - source.indexOf("export type DaemonCommand ="), - source.indexOf("type DaemonCommandName"), - ); - const savedSessionSource = source.slice( - source.indexOf("export interface DaemonSavedSessionInfo"), - source.indexOf("export type DaemonDeleteSavedSessionResult"), - ); - const outboundSource = source.slice( - source.indexOf("export type DaemonOutbound ="), - source.indexOf("export const DAEMON_OUTBOUND_COMPATIBILITY"), - ); - const ownedSessionSource = source.slice( - source.indexOf("export type DaemonOwnedSessionCleanupStatus"), - source.indexOf("export type DaemonServerCapability"), - ); - const launchEnvironmentSource = source.slice( - source.indexOf("export type DaemonLaunchEnvMode"), - source.indexOf("/**\n * The allowlist of env vars"), - ); - const errorInfoSource = source.slice( - source.indexOf("export type DaemonErrorInfo ="), - source.indexOf("export type DaemonSessionClosedReason"), - ); - const digest = createHash("sha256") - .update( - `${commandSource}\n${savedSessionSource}\n${outboundSource}\n${ownedSessionSource}\n${launchEnvironmentSource}\n${errorInfoSource}`, - ) - .digest("hex") - .slice(0, 12); + it("independently freezes and schema/capability-gates recoverable owned adoption", () => { + expect(Object.isFrozen(PRIME_AGENT_SDK_FEATURES)).toBe(true); + expect(PRIME_AGENT_SDK_FEATURES).toContain(RECOVERABLE_OWNED_SESSION_ADOPTION_FEATURE); + for (const type of [ + "create_recoverable_owned_session", + "prepare_recoverable_owned_session_adoption", + "commit_recoverable_owned_session_adoption", + "confirm_recoverable_owned_session_adoption", + ] as const) { + expect(DAEMON_COMMAND_COMPATIBILITY[type]).toEqual({ + minProtocol: 7, + minSchemaRevision: 30, + capability: "daemon_recoverable_owned_session_adoption_v1", + }); + } + expect(DAEMON_DEFAULT_SERVER_CAPABILITIES).not.toContain("daemon_recoverable_owned_session_adoption_v1"); + expect(DAEMON_SUPERVISOR_SERVER_CAPABILITIES).toContain("daemon_recoverable_owned_session_adoption_v1"); + expect(daemonSupervisorServerCapabilities("linux")).toContain("daemon_recoverable_owned_session_adoption_v1"); + expect(daemonSupervisorServerCapabilities("win32")).not.toContain("daemon_recoverable_owned_session_adoption_v1"); + expect(DAEMON_SUPPORTED_CLIENT_CAPABILITIES).not.toContain("daemon_recoverable_owned_session_adoption_v1"); + }); + + it("keeps the advertised schema identity synchronized with daemon and worker wire shapes", () => { + const daemonSource = readFileSync(resolve(__dirname, "../src/modes/daemon/daemon-protocol.ts"), "utf8"); + const workerSource = readFileSync(resolve(__dirname, "../src/modes/daemon/daemon-worker-protocol.ts"), "utf8"); + const digest = daemonSchemaDigest(daemonSource, workerSource); expect(DAEMON_SCHEMA_ID).toBe(`protocol-${DAEMON_PROTOCOL_VERSION}-schema-${DAEMON_SCHEMA_REVISION}-${digest}`); }); + it("changes the schema digest for every recoverable adoption wire family", () => { + const daemonSource = readFileSync(resolve(__dirname, "../src/modes/daemon/daemon-protocol.ts"), "utf8"); + const workerSource = readFileSync(resolve(__dirname, "../src/modes/daemon/daemon-worker-protocol.ts"), "utf8"); + const baseline = daemonSchemaDigest(daemonSource, workerSource); + const daemonMutations = [ + ['| "daemon_recoverable_owned_session_adoption_v1";', '| "daemon_recoverable_owned_session_adoption_v2";'], + ['type: "create_recoverable_owned_session";', 'type: "create_recoverable_owned_session_v2";'], + [ + 'type: "commit_recoverable_owned_session_adoption";', + 'type: "commit_recoverable_owned_session_adoption_v2";', + ], + [ + 'type: "confirm_recoverable_owned_session_adoption";', + 'type: "confirm_recoverable_owned_session_adoption_v2";', + ], + ["expectedSupervisorGeneration: string;", "expectedSupervisorGeneration: string & { readonly v2: true };"], + ["recoveryHandle: string;", "recoveryHandle: string & { readonly v2: true };"], + ['status: "adopted";', 'status: "adopted_v2";'], + [ + "export interface DaemonRecoverableOwnedSessionPrepareResult extends DaemonAttachResult", + "export interface DaemonRecoverableOwnedSessionPrepareResultV2 extends DaemonAttachResult", + ], + ['status: "confirmed";', 'status: "confirmed_v2";'], + ["minSchemaRevision: 30,", "minSchemaRevision: 31,"], + ] as const; + for (const [before, after] of daemonMutations) { + expect(daemonSchemaDigest(replaceWireSentinel(daemonSource, before, after), workerSource)).not.toBe(baseline); + } + const workerMutations = [ + [ + 'export const DAEMON_WORKER_SUPERVISOR_AGENT_DIR_ENV = "PRIME_AGENT_INTERNAL_DAEMON_SUPERVISOR_AGENT_DIR";', + 'export const DAEMON_WORKER_SUPERVISOR_AGENT_DIR_ENV = "PRIME_AGENT_INTERNAL_DAEMON_SUPERVISOR_AGENT_DIR_V2";', + ], + ["workerIncarnation: string;", `workerIncarnation: \`\${string}-v2\`;`], + [ + 'action: "transfer" | "query" | "rollback" | "retire";', + 'action: "transfer" | "query" | "rollback" | "retire" | "inspect";', + ], + ['state: "transferred" | "rolled_back";', 'state: "transferred" | "rolled_back" | "unknown";'], + ] as const; + for (const [before, after] of workerMutations) { + expect(daemonSchemaDigest(daemonSource, replaceWireSentinel(workerSource, before, after))).not.toBe(baseline); + } + }); + it("requires compatibility metadata for the heartbeat protocol surface", () => { expect(DAEMON_PROTOCOL_VERSION).toBe(7); expect(DAEMON_SCHEMA_ID).toContain(`protocol-${DAEMON_PROTOCOL_VERSION}`); @@ -289,7 +398,7 @@ describe("daemon protocol helpers", () => { }); it("capability- and schema-gates fresh snapshot generation nonces", () => { - expect(DAEMON_SCHEMA_REVISION).toBe(29); + expect(DAEMON_SCHEMA_REVISION).toBe(30); expect(DAEMON_SNAPSHOT_GENERATION_NONCE_MIN_SCHEMA_REVISION).toBe(28); expect( getDaemonCommandCompatibilities({ diff --git a/packages/coding-agent/test/daemon-supervisor-monitor.test.ts b/packages/coding-agent/test/daemon-supervisor-monitor.test.ts index 3cacd67258..e8c0a65fbd 100644 --- a/packages/coding-agent/test/daemon-supervisor-monitor.test.ts +++ b/packages/coding-agent/test/daemon-supervisor-monitor.test.ts @@ -34,6 +34,7 @@ import { createDeferred } from "./suite/scheduling.js"; const workerLaunchTestState = vi.hoisted(() => ({ capture: false, forceMissingProcessStartId: false, + processStartIdOverride: undefined as ((pid: number) => string | undefined) | undefined, fixtureMode: "worker" as "worker" | "close-gate" | "rollback-gate" | "successful-gate", gateMarkerPath: "", tsxCliPath: "", @@ -109,6 +110,7 @@ vi.mock("../src/core/session-lease.js", async (importOriginal) => { return { ...actual, getProcessStartId(pid: number): string | undefined { + if (workerLaunchTestState.processStartIdOverride) return workerLaunchTestState.processStartIdOverride(pid); return workerLaunchTestState.forceMissingProcessStartId ? undefined : actual.getProcessStartId(pid); }, }; @@ -291,6 +293,7 @@ describe("daemon worker supervisor monitoring", () => { } workerLaunchTestState.capture = false; workerLaunchTestState.forceMissingProcessStartId = false; + workerLaunchTestState.processStartIdOverride = undefined; workerLaunchTestState.fixtureMode = "worker"; workerLaunchTestState.gateMarkerPath = ""; workerLaunchTestState.tsxCliPath = ""; @@ -574,6 +577,7 @@ describe("daemon worker supervisor monitoring", () => { expect(Object.keys(exactEnvironment ?? {}).sort()).toEqual( [ "CALLER_ENV_A", + "PRIME_AGENT_INTERNAL_DAEMON_SUPERVISOR_AGENT_DIR", "PRIME_AGENT_INTERNAL_DAEMON_SUPERVISOR_SOCKET", "PRIME_AGENT_INTERNAL_DAEMON_WORKER", "PRIME_AGENT_INTERNAL_DAEMON_WORKER_ACTIVE_SESSION_ID", @@ -2394,6 +2398,30 @@ describe("daemon worker supervisor monitoring", () => { } }); + it("rechecks captured process identity immediately before SIGKILL", () => { + const processStartId = getProcessStartId(process.pid); + expect(processStartId).toBeDefined(); + const supervisor = Object.create(DaemonSupervisor.prototype) as { + signalCapturedProcess(process: { pid: number; processStartId: string }, signal: NodeJS.Signals): boolean; + }; + const killSpy = vi.spyOn(childProcessModule, "signalProcessGroupOrProcess").mockImplementation(() => {}); + try { + workerLaunchTestState.processStartIdOverride = () => "recycled-process-start"; + expect( + supervisor.signalCapturedProcess({ pid: process.pid, processStartId: processStartId! }, "SIGKILL"), + ).toBe(false); + expect(killSpy).not.toHaveBeenCalled(); + + workerLaunchTestState.processStartIdOverride = () => processStartId; + expect( + supervisor.signalCapturedProcess({ pid: process.pid, processStartId: processStartId! }, "SIGKILL"), + ).toBe(true); + expect(killSpy).toHaveBeenCalledWith(process.pid, "SIGKILL"); + } finally { + workerLaunchTestState.processStartIdOverride = undefined; + killSpy.mockRestore(); + } + }); it("never follows a relaunched worker pid after a retry rescinds the stop", async () => { vi.useFakeTimers(); const worker = { @@ -5297,6 +5325,7 @@ describe("daemon worker supervisor monitoring", () => { const supervisor = Object.assign(Object.create(DaemonSupervisor.prototype), { ready: Promise.resolve(), workers: new Map(), + clients: new Set([client]), protocolClientIds: new WeakMap(), commandJournal, mutationDrain, @@ -5343,6 +5372,7 @@ describe("daemon worker supervisor monitoring", () => { const supervisor = Object.assign(Object.create(DaemonSupervisor.prototype), { ready: Promise.resolve(), workers: new Map(), + clients: new Set([client]), protocolClientIds: new WeakMap(), commandJournal, mutationDrain, diff --git a/packages/coding-agent/test/daemon-supervisor-platform.test.ts b/packages/coding-agent/test/daemon-supervisor-platform.test.ts new file mode 100644 index 0000000000..ff2672890f --- /dev/null +++ b/packages/coding-agent/test/daemon-supervisor-platform.test.ts @@ -0,0 +1,52 @@ +import { mkdtempSync, readdirSync, rmSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { afterEach, describe, expect, it, vi } from "vitest"; +import { DaemonClient } from "../src/modes/daemon/daemon-client.js"; +import { DaemonSupervisor } from "../src/modes/daemon/daemon-supervisor.js"; + +const directories: string[] = []; + +afterEach(() => { + for (const directory of directories.splice(0)) rmSync(directory, { recursive: true, force: true }); +}); + +describe("DaemonSupervisor platform gates", () => { + it("starts and says hello on injected win32 without constructing the POSIX recovery store", async () => { + const root = mkdtempSync(join(tmpdir(), "daemon-supervisor-win32-gate-")); + directories.push(root); + const descriptorDir = join(root, "workers"); + const socketPath = + process.platform === "win32" + ? String.raw`\\.\pipe\prime-win32-gate-${process.pid}` + : join(root, "daemon.sock"); + let recoveryStoreConstructions = 0; + const supervisor = new DaemonSupervisor(socketPath, { + defaultSessionConfig: { agentDir: root, cwd: root }, + descriptorDir, + platform: "win32", + ownedSessionRecoveryStoreFactory: () => { + recoveryStoreConstructions++; + throw new Error("directory fsync is unavailable"); + }, + }); + const internals = supervisor as unknown as { + catalog: { start: ReturnType; stop: ReturnType }; + cleanupSupervisorResources(): Promise; + }; + internals.catalog.start = vi.fn(async () => undefined); + internals.catalog.stop = vi.fn(async () => undefined); + const client = new DaemonClient(socketPath); + try { + await supervisor.start(); + await client.connect(); + const hello = await client.waitForHello(); + expect(hello.serverCapabilities).not.toContain("daemon_recoverable_owned_session_adoption_v1"); + expect(recoveryStoreConstructions).toBe(0); + expect(readdirSync(descriptorDir)).not.toContain("owned-session-recovery"); + } finally { + client.close(); + await internals.cleanupSupervisorResources(); + } + }); +}); diff --git a/packages/coding-agent/test/daemon-supervisor-process.test.ts b/packages/coding-agent/test/daemon-supervisor-process.test.ts index 26d35a0574..d3cce5bad8 100644 --- a/packages/coding-agent/test/daemon-supervisor-process.test.ts +++ b/packages/coding-agent/test/daemon-supervisor-process.test.ts @@ -16,14 +16,24 @@ import { } from "../src/core/session-lease.js"; import { readSessionInfo, SessionManager } from "../src/core/session-manager.js"; import { DaemonAgentConnection } from "../src/modes/agent-connection/daemon-agent-connection.js"; +import { + adoptRecoverableOwnedSession, + confirmRecoverableOwnedSessionAdoption, + createRecoverableOwnedSession, +} from "../src/modes/agent-connection/recoverable-owned-session.js"; import { DaemonClient, getDaemonSocketCloseReason } from "../src/modes/daemon/daemon-client.js"; -import { collectDaemonLaunchEnv, createDaemonCommandEnvelope } from "../src/modes/daemon/daemon-protocol.js"; +import { + collectDaemonClientEnv, + collectDaemonLaunchEnv, + createDaemonCommandEnvelope, +} from "../src/modes/daemon/daemon-protocol.js"; import type { SessionSummary } from "../src/modes/daemon/daemon-session-list.js"; import type { DaemonWorkerDescriptor } from "../src/modes/daemon/daemon-worker-protocol.js"; const cliPath = resolve(__dirname, "../src/cli.ts"); const tsxPath = resolve(__dirname, "../../../node_modules/tsx/dist/cli.mjs"); const blockingProcessPath = resolve(__dirname, "fixtures/blocking-process.mjs"); +const fauxExtensionPath = resolve(__dirname, "fixtures/eng-4600-faux-extension.ts"); const tempDirs: string[] = []; const children = new Set(); const workerPids = new Set(); @@ -259,6 +269,14 @@ async function connectEventually(socketPath: string, child?: ChildProcess): Prom ); } +async function disconnectDaemonClientWithServerCloseBarrier(client: DaemonClient): Promise { + const socket = (client as unknown as { socket?: Socket }).socket; + if (!socket || socket.destroyed) return; + const closed = new Promise((resolveClose) => socket.once("close", () => resolveClose())); + socket.end(); + await closed; +} + async function createSnapshotRetryProxy( proxyPath: string, targetPath: string, @@ -367,6 +385,84 @@ async function createSnapshotRetryProxy( }; } +async function createCommandResponseLossProxy( + proxyPath: string, + targetPath: string, + commandType: string, +): Promise<{ server: Server; responseObserved: Promise; release(): void }> { + let resolveResponseObserved!: () => void; + const responseObserved = new Promise((resolveObserved) => { + resolveResponseObserved = resolveObserved; + }); + let targetCommandId: string | undefined; + let downstreamSocket: Socket | undefined; + let upstreamSocket: Socket | undefined; + const server = createServer((downstream) => { + downstreamSocket = downstream; + const upstream = createConnection(targetPath); + upstreamSocket = upstream; + let requestBuffer = Buffer.alloc(0); + let responseBuffer = Buffer.alloc(0); + downstream.on("data", (chunk: Buffer) => { + requestBuffer = Buffer.concat([requestBuffer, chunk]); + while (true) { + const newline = requestBuffer.indexOf(0x0a); + if (newline < 0) break; + const original = Buffer.from(requestBuffer.subarray(0, newline + 1)); + requestBuffer = requestBuffer.subarray(newline + 1); + try { + const parsed = JSON.parse(original.toString("utf8")) as { + id?: unknown; + command?: { type?: unknown }; + }; + if (parsed.command?.type === commandType && typeof parsed.id === "string") targetCommandId = parsed.id; + } catch {} + upstream.write(original); + } + }); + upstream.on("data", (chunk: Buffer) => { + responseBuffer = Buffer.concat([responseBuffer, chunk]); + while (true) { + const newline = responseBuffer.indexOf(0x0a); + if (newline < 0) break; + const original = Buffer.from(responseBuffer.subarray(0, newline + 1)); + responseBuffer = responseBuffer.subarray(newline + 1); + let dropsResponse = false; + try { + const parsed = JSON.parse(original.toString("utf8")) as { + id?: unknown; + type?: unknown; + command?: unknown; + }; + dropsResponse = + parsed.type === "response" && parsed.command === commandType && parsed.id === targetCommandId; + } catch {} + if (dropsResponse) { + resolveResponseObserved(); + continue; + } + downstream.write(original); + } + }); + upstream.on("error", (error) => downstream.destroy(error)); + downstream.on("error", () => upstream.destroy()); + upstream.on("close", () => downstream.destroy()); + downstream.on("close", () => upstream.destroy()); + }); + await new Promise((resolveListen, reject) => { + server.once("error", reject); + server.listen(proxyPath, () => resolveListen()); + }); + return { + server, + responseObserved, + release() { + downstreamSocket?.destroy(); + upstreamSocket?.destroy(); + }, + }; +} + async function createSnapshotDrainAbortProxy( proxyPath: string, targetPath: string, @@ -1076,6 +1172,497 @@ describe("daemon supervisor resident workers", () => { await waitForSocketGone(socketPath); }, 60_000); + it.skipIf(process.platform === "win32")( + "adopts a recoverable owned worker with an authoritative snapshot and rotated receipt", + async () => { + const root = tempDir(); + const agentDir = join(root, "agent"); + const projectDir = join(root, "project"); + const sessionDir = join(agentDir, "sessions"); + const socketPath = join(tmpdir(), `prime-recover-owned-${process.pid}-${randomUUID().slice(0, 8)}.sock`); + mkdirSync(projectDir, { recursive: true }); + const sessionManager = SessionManager.create(projectDir, sessionDir); + sessionManager.appendMessage({ role: "user", content: "recoverable fixture", timestamp: 1 }); + const sessionPath = sessionManager.getSessionFile(); + if (!sessionPath) throw new Error("Recoverable fixture did not persist"); + const supervisor = spawnSupervisor(agentDir, socketPath, projectDir); + const owner = await connectEventually(socketPath, supervisor); + const correlationId = `recover-correlation-${randomUUID()}`; + const previousMcpOwnerId = `recover-mcp-${randomUUID()}`; + const config = { + cwd: projectDir, + agentDir, + sessionDir, + apiKey: "faux-key", + extensions: [fauxExtensionPath], + provider: "faux", + model: "faux", + noTools: true, + noExtensions: false, + }; + const launchEnv = { + ...collectDaemonLaunchEnv(), + PRIME_AGENT_RECOVERY_TEST: `canary-${randomUUID()}`, + }; + const createRequestId = randomUUID(); + const created = await createRecoverableOwnedSession(owner, { + requestId: createRequestId, + correlationId, + mcpOwnerId: previousMcpOwnerId, + config, + sessionPath, + launchEnv, + connectionOptions: { supportsExtensionUi: false }, + }); + if (!created.state.workerPid || !created.state.activeSessionId) + throw new Error("Recoverable worker was incomplete"); + workerPids.add(created.state.workerPid); + const recoverableMcpServers = [ + { + name: "recoverable-task", + type: "http" as const, + url: "https://recoverable.invalid/mcp", + headers: { Authorization: "Bearer recoverable-mcp-canary" }, + }, + ]; + await created.connection.replaceAcpMcpServers(recoverableMcpServers, previousMcpOwnerId); + await created.connection.submitCorrelatedPrompt("record an adoption lifecycle", { + correlationId, + queueIfBusy: true, + }); + const before = await created.connection.getInitialSnapshot(); + if (!before.lastEventCursor) throw new Error("Recoverable fixture did not expose an event cursor"); + const earlyAdopter = await connectEventually(socketPath, supervisor); + const requestId = randomUUID(); + const nextMcpOwnerId = `recover-mcp-${randomUUID()}`; + const adoptionOptions = { + requestId, + recoveryHandle: created.recoveryHandle, + expectedSupervisorGeneration: created.supervisorGeneration, + activeSessionId: created.state.activeSessionId, + sessionId: created.state.sessionId, + correlationId, + cursor: before.lastEventCursor, + previousMcpOwnerId, + mcpOwnerId: nextMcpOwnerId, + config, + launchEnv, + connectionOptions: { supportsExtensionUi: false }, + } as const; + await expect(adoptRecoverableOwnedSession(earlyAdopter, adoptionOptions)).rejects.toThrow( + "Recoverable owned session adoption is unavailable", + ); + earlyAdopter.close(); + await disconnectDaemonClientWithServerCloseBarrier(owner); + const wrongHandleClient = await connectEventually(socketPath, supervisor); + await expect( + adoptRecoverableOwnedSession(wrongHandleClient, { + ...adoptionOptions, + requestId: randomUUID(), + recoveryHandle: "W".repeat(43), + }), + ).rejects.toThrow("Recoverable owned session adoption is unavailable"); + wrongHandleClient.close(); + + const proxyPath = join(tmpdir(), `prime-recover-loss-${process.pid}-${randomUUID().slice(0, 8)}.sock`); + const responseLoss = await createCommandResponseLossProxy( + proxyPath, + socketPath, + "commit_recoverable_owned_session_adoption", + ); + const lostResponseAdopter = await connectEventually(proxyPath, supervisor); + let lostAdoption: Promise>> | undefined; + for (let attempt = 0; attempt < 20; attempt++) { + const candidate = adoptRecoverableOwnedSession(lostResponseAdopter, adoptionOptions); + const outcome = await Promise.race([ + responseLoss.responseObserved.then(() => "response_lost" as const), + candidate.then( + () => "unexpected_success" as const, + () => "retry" as const, + ), + ]); + if (outcome === "response_lost") { + lostAdoption = candidate; + break; + } + if (outcome === "unexpected_success") throw new Error("Response-loss proxy forwarded the commit receipt"); + } + if (!lostAdoption) throw new Error("Recoverable commit response was not observed by the loss proxy"); + + const retryAdopter = await connectEventually(socketPath, supervisor); + const observedAdoptionMessages: string[] = []; + const stopObservingAdoptionMessages = retryAdopter.onMessage((message) => { + observedAdoptionMessages.push(JSON.stringify(message)); + }); + await expect(adoptRecoverableOwnedSession(retryAdopter, adoptionOptions)).rejects.toThrow( + "Recoverable owned session adoption is unavailable", + ); + responseLoss.release(); + await expect(lostAdoption).rejects.toThrow("Recoverable owned session adoption is unavailable"); + lostResponseAdopter.close(); + await new Promise((resolveClose) => responseLoss.server.close(() => resolveClose())); + + let commitRetry: Awaited> | undefined; + for (let attempt = 0; attempt < 20; attempt++) { + try { + commitRetry = await adoptRecoverableOwnedSession(retryAdopter, adoptionOptions); + break; + } catch (error) { + if (!(error instanceof Error) || error.message !== "Recoverable owned session adoption is unavailable") { + throw error; + } + } + } + if (!commitRetry) throw new Error("Final recoverable adoption receipt did not rebind"); + expect(commitRetry.recoveryHandle).not.toBe(created.recoveryHandle); + expect(commitRetry.proof).toMatchObject({ + feature: "recoverable_owned_session_adoption_v1", + status: "adopted", + activeSessionId: created.state.activeSessionId, + sessionId: created.state.sessionId, + correlationId, + mcpOwnerId: nextMcpOwnerId, + ownershipGeneration: 1, + }); + const after = await commitRetry.connection.getInitialSnapshot(); + expect(after.messages).toEqual(before.messages); + expect(after.lastEventCursor?.sequence).toBeGreaterThanOrEqual(before.lastEventCursor.sequence); + const futureCorrelationId = randomUUID(); + let resolveFutureLifecycle!: () => void; + const futureLifecycle = new Promise((resolve) => { + resolveFutureLifecycle = resolve; + }); + const stopObservingFutureLifecycle = commitRetry.connection.subscribe((event) => { + if ( + event.type === "prompt_lifecycle" && + event.lifecycle.correlationId === futureCorrelationId && + ["completed", "cancelled", "failed"].includes(event.lifecycle.phase) + ) { + resolveFutureLifecycle(); + } + }); + await commitRetry.connection.submitCorrelatedPrompt("future event after adoption retry", { + correlationId: futureCorrelationId, + queueIfBusy: true, + }); + await futureLifecycle; + stopObservingFutureLifecycle(); + await expect( + commitRetry.connection.replaceAcpMcpServers(recoverableMcpServers, previousMcpOwnerId), + ).rejects.toThrow("owned by another daemon client"); + await commitRetry.connection.releaseAcpMcpServers(nextMcpOwnerId, ["recoverable-task"]); + const confirmation = { + requestId, + recoveryHandle: commitRetry.recoveryHandle, + proof: commitRetry.proof, + }; + await confirmRecoverableOwnedSessionAdoption(retryAdopter, confirmation); + await confirmRecoverableOwnedSessionAdoption(retryAdopter, confirmation); + await owner.connect(); + await owner.waitForHello(); + const retiredCreateReplay = await owner.request({ + type: "create_recoverable_owned_session", + requestId: createRequestId, + expectedSupervisorGeneration: created.supervisorGeneration, + correlationId, + mcpOwnerId: previousMcpOwnerId, + recoveryConfig: config, + sessionPath, + config, + env: collectDaemonClientEnv(), + launchEnv, + launchEnvMode: "replace", + }); + expect(retiredCreateReplay).toMatchObject({ + success: false, + error: "Recoverable owned session adoption is unavailable", + }); + expect(JSON.stringify(retiredCreateReplay)).not.toContain(commitRetry.recoveryHandle); + expect(readWorkerDescriptor(agentDir).ownerClientId).toBe(retryAdopter.clientId); + const fencedOldOwnerMutation = await owner.request({ + type: "replace_acp_mcp_servers", + activeSessionId: created.state.activeSessionId, + ownerId: previousMcpOwnerId, + servers: recoverableMcpServers, + }); + expect(fencedOldOwnerMutation).toMatchObject({ + success: false, + error: `Unknown active session: ${created.state.activeSessionId}`, + }); + owner.close(); + const descriptor = readWorkerDescriptor(agentDir); + expect(descriptor.pid).toBe(created.state.workerPid); + expect(descriptor.ownerClientId).toBe(retryAdopter.clientId); + const listed = await retryAdopter.request({ type: "list", includeClientOwned: true }); + const privateSurfaces = [ + JSON.stringify(descriptor), + JSON.stringify(listed), + observedAdoptionMessages.join("\n"), + readDaemonLogs(agentDir), + ]; + for (const canary of [ + created.recoveryHandle, + commitRetry.recoveryHandle, + requestId, + previousMcpOwnerId, + nextMcpOwnerId, + launchEnv.PRIME_AGENT_RECOVERY_TEST, + ]) { + for (const surface of privateSurfaces) expect(surface).not.toContain(canary); + } + if (!descriptor.processStartId) throw new Error("Recoverable worker did not expose a process identity"); + identityTrackedProcesses.set(descriptor.pid, descriptor.processStartId); + signalIdentityVerifiedProcess(descriptor, "SIGKILL"); + await waitForProcessGone(descriptor.pid); + identityTrackedProcesses.delete(descriptor.pid); + workerPids.delete(descriptor.pid); + await waitForCondition( + () => { + const recovered = readWorkerDescriptor(agentDir); + return recovered.pid !== descriptor.pid && recovered.lifecycle === "ready"; + }, + "Adopted owner did not recover through a replacement worker incarnation", + 20_000, + ); + const recoveredDescriptor = readWorkerDescriptor(agentDir); + if (!recoveredDescriptor.processStartId) { + throw new Error("Recovered adopted worker did not expose a process identity"); + } + workerPids.add(recoveredDescriptor.pid); + identityTrackedProcesses.set(recoveredDescriptor.pid, recoveredDescriptor.processStartId); + expect(recoveredDescriptor.ownerClientId).toBe(retryAdopter.clientId); + await expect(confirmRecoverableOwnedSessionAdoption(retryAdopter, confirmation)).rejects.toThrow( + "Recoverable owned session adoption is unavailable", + ); + const recoveredCorrelationId = randomUUID(); + const recoveredLifecycle = new Promise((resolveLifecycle) => { + const unsubscribe = commitRetry.connection.subscribe((event) => { + if (event.type === "prompt_lifecycle" && event.lifecycle.correlationId === recoveredCorrelationId) { + unsubscribe(); + resolveLifecycle(); + } + }); + }); + await commitRetry.connection.submitCorrelatedPrompt("prompt after worker incarnation replacement", { + correlationId: recoveredCorrelationId, + queueIfBusy: true, + }); + await recoveredLifecycle; + stopObservingAdoptionMessages(); + const cleanupProof = await commitRetry.connection.disposeOwnedSession(); + expect(cleanupProof).toMatchObject({ + feature: "caller_owned_session_environment_cleanup_v1", + status: "completed", + started: { status: "attached" }, + observed: { supervisorGeneration: created.supervisorGeneration }, + daemonReplaced: false, + }); + await waitForProcessGone(recoveredDescriptor.pid); + workerPids.delete(recoveredDescriptor.pid); + identityTrackedProcesses.delete(recoveredDescriptor.pid); + await retryAdopter.request({ type: "shutdown" }); + retryAdopter.close(); + await waitForSocketGone(socketPath); + }, + 60_000, + ); + + it.skipIf(process.platform === "win32")( + "grants exactly one authority to simultaneous recoverable claimants", + async () => { + const root = tempDir(); + const agentDir = join(root, "agent"); + const projectDir = join(root, "project"); + const sessionDir = join(agentDir, "sessions"); + const socketPath = join(tmpdir(), `prime-recover-race-${process.pid}-${randomUUID().slice(0, 8)}.sock`); + mkdirSync(projectDir, { recursive: true }); + const sessionManager = SessionManager.create(projectDir, sessionDir); + sessionManager.appendMessage({ role: "user", content: "recoverable race fixture", timestamp: 1 }); + const sessionPath = sessionManager.getSessionFile(); + if (!sessionPath) throw new Error("Recoverable race fixture did not persist"); + const supervisor = spawnSupervisor(agentDir, socketPath, projectDir); + const owner = await connectEventually(socketPath, supervisor); + const correlationId = `recover-race-${randomUUID()}`; + const previousMcpOwnerId = `recover-race-mcp-${randomUUID()}`; + const config = { + cwd: projectDir, + agentDir, + sessionDir, + apiKey: "faux-key", + extensions: [fauxExtensionPath], + provider: "faux", + model: "faux", + noTools: true, + noExtensions: false, + }; + const launchEnv = { + ...collectDaemonLaunchEnv(), + PRIME_AGENT_RECOVERY_RACE: `canary-${randomUUID()}`, + }; + const created = await createRecoverableOwnedSession(owner, { + requestId: randomUUID(), + correlationId, + mcpOwnerId: previousMcpOwnerId, + config, + sessionPath, + launchEnv, + connectionOptions: { supportsExtensionUi: false }, + }); + if (!created.state.workerPid || !created.state.activeSessionId) { + throw new Error("Recoverable race worker was incomplete"); + } + workerPids.add(created.state.workerPid); + await created.connection.submitCorrelatedPrompt("record a recoverable race lifecycle", { + correlationId, + queueIfBusy: true, + }); + const snapshot = await created.connection.getInitialSnapshot(); + if (!snapshot.lastEventCursor) throw new Error("Recoverable race fixture did not expose a cursor"); + await disconnectDaemonClientWithServerCloseBarrier(owner); + const claimantA = await connectEventually(socketPath, supervisor); + const claimantB = await connectEventually(socketPath, supervisor); + const requestId = randomUUID(); + const adoptionOptions = { + requestId, + recoveryHandle: created.recoveryHandle, + expectedSupervisorGeneration: created.supervisorGeneration, + activeSessionId: created.state.activeSessionId, + sessionId: created.state.sessionId, + correlationId, + cursor: snapshot.lastEventCursor, + previousMcpOwnerId, + mcpOwnerId: `recover-race-next-${randomUUID()}`, + config, + launchEnv, + connectionOptions: { supportsExtensionUi: false }, + } as const; + const outcomes = await Promise.allSettled([ + adoptRecoverableOwnedSession(claimantA, adoptionOptions), + adoptRecoverableOwnedSession(claimantB, adoptionOptions), + ]); + const fulfilled = outcomes.flatMap((outcome, index) => + outcome.status === "fulfilled" ? [{ adoption: outcome.value, index }] : [], + ); + const rejected = outcomes.filter((outcome) => outcome.status === "rejected"); + expect(fulfilled).toHaveLength(1); + expect(rejected).toHaveLength(1); + expect(rejected[0]).toMatchObject({ + status: "rejected", + reason: { message: "Recoverable owned session adoption is unavailable" }, + }); + const winner = fulfilled[0]!; + const winnerClient = winner.index === 0 ? claimantA : claimantB; + const loserClient = winner.index === 0 ? claimantB : claimantA; + expect(readWorkerDescriptor(agentDir).ownerClientId).toBe(winnerClient.clientId); + await confirmRecoverableOwnedSessionAdoption(winnerClient, { + requestId, + recoveryHandle: winner.adoption.recoveryHandle, + proof: winner.adoption.proof, + }); + loserClient.close(); + owner.close(); + await winner.adoption.connection.dispose(); + await waitForProcessGone(created.state.workerPid); + workerPids.delete(created.state.workerPid); + await winnerClient.request({ type: "shutdown" }); + winnerClient.close(); + await waitForSocketGone(socketPath); + }, + 60_000, + ); + + it.skipIf(process.platform === "win32")( + "fails closed when recoverable authority crosses a supervisor generation", + async () => { + const root = tempDir(); + const agentDir = join(root, "agent"); + const projectDir = join(root, "project"); + const sessionDir = join(agentDir, "sessions"); + const socketPath = join(tmpdir(), `prime-recover-generation-${process.pid}-${randomUUID().slice(0, 8)}.sock`); + mkdirSync(projectDir, { recursive: true }); + const sessionManager = SessionManager.create(projectDir, sessionDir); + sessionManager.appendMessage({ role: "user", content: "generation fixture", timestamp: 1 }); + const sessionPath = sessionManager.getSessionFile(); + if (!sessionPath) throw new Error("Generation fixture did not persist"); + const supervisor = spawnSupervisor(agentDir, socketPath, projectDir); + const owner = await connectEventually(socketPath, supervisor); + const correlationId = `recover-generation-${randomUUID()}`; + const mcpOwnerId = `recover-generation-mcp-${randomUUID()}`; + const config = { + cwd: projectDir, + agentDir, + sessionDir, + apiKey: "faux-key", + extensions: [fauxExtensionPath], + provider: "faux", + model: "faux", + noTools: true, + noExtensions: false, + }; + const launchEnv = collectDaemonLaunchEnv(); + const created = await createRecoverableOwnedSession(owner, { + requestId: randomUUID(), + correlationId, + mcpOwnerId, + config, + launchEnv, + sessionPath, + connectionOptions: { + supportsExtensionUi: false, + recoverDaemon: () => new Promise(() => undefined), + }, + }); + if (!created.state.workerPid || !created.state.activeSessionId) { + throw new Error("Generation fixture worker was incomplete"); + } + workerPids.add(created.state.workerPid); + await created.connection.submitCorrelatedPrompt("record a generation lifecycle", { + correlationId, + queueIfBusy: true, + }); + const snapshot = await created.connection.getInitialSnapshot(); + if (!snapshot.lastEventCursor) throw new Error("Generation fixture did not expose an event cursor"); + const workerIdentity = readWorkerDescriptor(agentDir); + const supervisorIdentity = { + pid: owner.hello?.supervisorPid, + processStartId: owner.hello?.supervisorProcessStartId, + }; + signalIdentityVerifiedProcess(supervisorIdentity, "SIGKILL"); + await waitForProcessGone(supervisorIdentity.pid!); + owner.close(); + const claimant = await connectEventually(socketPath); + expect(await claimant.request({ type: "list" })).toMatchObject({ success: true, command: "list" }); + expect(readWorkerDescriptor(agentDir)).toMatchObject({ + workerId: workerIdentity.workerId, + pid: workerIdentity.pid, + processStartId: workerIdentity.processStartId, + }); + await expect( + adoptRecoverableOwnedSession(claimant, { + requestId: randomUUID(), + recoveryHandle: created.recoveryHandle, + expectedSupervisorGeneration: created.supervisorGeneration, + activeSessionId: created.state.activeSessionId, + sessionId: created.state.sessionId, + correlationId, + cursor: snapshot.lastEventCursor, + previousMcpOwnerId: mcpOwnerId, + mcpOwnerId: `replacement-mcp-${randomUUID()}`, + config, + launchEnv, + connectionOptions: { supportsExtensionUi: false }, + }), + ).rejects.toThrow("Recoverable owned session adoption is unavailable"); + const shutdownResult = await claimant.request({ type: "shutdown", force: true }); + expect(shutdownResult).toMatchObject({ success: true, command: "shutdown" }); + claimant.close(); + await waitForProcessGone(created.state.workerPid); + workerPids.delete(created.state.workerPid); + await waitForSocketGone(socketPath); + }, + 60_000, + ); + it.skipIf(process.platform === "win32")( "isolates two caller-owned environments through worker and supervisor recovery", async () => { diff --git a/packages/coding-agent/test/daemon-supervisor-recoverable-state.test.ts b/packages/coding-agent/test/daemon-supervisor-recoverable-state.test.ts new file mode 100644 index 0000000000..279ca7dbbf --- /dev/null +++ b/packages/coding-agent/test/daemon-supervisor-recoverable-state.test.ts @@ -0,0 +1,509 @@ +import { mkdtempSync, rmSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { afterEach, describe, expect, it, vi } from "vitest"; +import type { DaemonCommand, DaemonRecoverableOwnedSessionPrepareResult } from "../src/modes/daemon/daemon-protocol.js"; +import { DaemonSupervisor } from "../src/modes/daemon/daemon-supervisor.js"; +import { RECOVERABLE_OWNED_CONFIRMATION_RETENTION_MS } from "../src/modes/daemon/owned-session-recovery-retention.js"; + +const directories: string[] = []; + +afterEach(() => { + vi.useRealTimers(); + for (const directory of directories.splice(0)) rmSync(directory, { recursive: true, force: true }); +}); + +function supervisorFixture() { + const root = mkdtempSync(join(tmpdir(), "daemon-recoverable-state-")); + directories.push(root); + const supervisor = new DaemonSupervisor(join(root, "daemon.sock"), { + defaultSessionConfig: { agentDir: root, cwd: root }, + descriptorDir: join(root, "workers"), + }); + return supervisor as unknown as { + clients: Set; + workers: Map; + generation: string; + ownedSessionRecoveryStore: Record>; + stopWorker: ReturnType; + scheduleOwnedWorkerCleanup: ReturnType; + createRecoverableOwnedSession( + client: unknown, + command: Extract, + ): Promise; + prepareRecoverableOwnedSnapshot( + worker: unknown, + adoption: unknown, + receipt: unknown, + command: Extract, + prepareDeadline: number, + ): Promise; + prepareRecoverableOwnedSessionAdoption( + client: unknown, + command: Extract, + ): Promise; + rollbackRecoverableOwnedAdoption(worker: unknown, adoption: unknown): void; + commitRecoverableOwnedSessionAdoption( + client: unknown, + command: Extract, + ): Promise; + retireRecoverableOwnedAuthorityForWorkerIncarnation(worker: unknown, incarnation: string): void; + scheduleRecoverableOwnedConfirmationExpiry(worker: unknown, recordId: string): void; + }; +} + +describe("DaemonSupervisor recoverable authority lifecycle", () => { + it("revokes every stale recovery surface when a worker incarnation is replaced", () => { + const supervisor = supervisorFixture(); + const oldIncarnation = "A".repeat(43); + const adoptionTimeout = setTimeout(() => {}, 60_000); + const confirmationTimer = setTimeout(() => {}, 60_000); + const cleanupTimer = setTimeout(() => {}, 60_000); + const worker = { + descriptor: { workerId: "worker-1", ownerClientId: "connected-adopted-owner" }, + workerIncarnation: oldIncarnation, + recoveryRecordId: "record-1", + recoverableAdoption: { + workerIncarnation: oldIncarnation, + timeout: adoptionTimeout, + }, + recoverableFinal: { recordId: "record-1" }, + recoveryConfirmationTimer: confirmationTimer, + ownerCleanupTimer: cleanupTimer, + }; + supervisor.ownedSessionRecoveryStore = { + get: vi.fn(() => ({ authority: { workerIncarnation: oldIncarnation } })), + remove: vi.fn(), + }; + + supervisor.retireRecoverableOwnedAuthorityForWorkerIncarnation(worker, "B".repeat(43)); + + expect(supervisor.ownedSessionRecoveryStore.remove).toHaveBeenCalledWith("record-1"); + expect(worker).toMatchObject({ + descriptor: { ownerClientId: "connected-adopted-owner" }, + recoveryRecordId: undefined, + recoverableAdoption: undefined, + recoverableFinal: undefined, + recoveryConfirmationTimer: undefined, + ownerCleanupTimer: undefined, + }); + }); + + it("rejects a stale create generation before recovery lookup or worker writes", async () => { + const supervisor = supervisorFixture(); + const getByCreateRequest = vi.fn(); + const create = vi.fn(); + supervisor.ownedSessionRecoveryStore = { + digestRequest: vi.fn(), + getByCreateRequest, + create, + }; + await expect( + supervisor.createRecoverableOwnedSession( + { id: "creator" }, + { + type: "create_recoverable_owned_session", + requestId: "00112233445566778899aabbccddeeff", + expectedSupervisorGeneration: "replacement-generation", + correlationId: "correlation", + mcpOwnerId: "mcp-owner", + recoveryConfig: { cwd: "/tmp/project" }, + config: { cwd: "/tmp/project" }, + launchEnv: {}, + launchEnvMode: "replace", + }, + ), + ).rejects.toThrow("Recoverable owned session adoption is unavailable"); + expect(getByCreateRequest).not.toHaveBeenCalled(); + expect(create).not.toHaveBeenCalled(); + expect(supervisor.workers.size).toBe(0); + }); + + it("contains the exact launched worker and removes its receipt after post-launch setup fails", async () => { + const supervisor = supervisorFixture(); + const summary = { + id: "active-create", + activeSessionId: "active-create", + sessionId: "session-create", + cwd: "/tmp/project", + }; + const worker = { + descriptor: { workerId: "worker-create", rootActiveSessionId: "active-create", ownerClientId: "creator" }, + workerIncarnation: "I".repeat(43), + client: { matchesAuthenticatedIncarnation: vi.fn(() => true) }, + summaries: new Map([["active-create", summary]]), + recoveryRecordId: undefined as string | undefined, + ownerCleanupTimer: setTimeout(() => {}, 1_000) as NodeJS.Timeout | undefined, + }; + const stopWorker = vi.fn(async () => undefined); + const remove = vi.fn(); + supervisor.ownedSessionRecoveryStore = { + digestRequest: vi.fn((value: unknown) => JSON.stringify(value)), + digestAuthority: vi.fn((value: unknown) => JSON.stringify(value)), + getByCreateRequest: vi.fn(() => undefined), + create: vi.fn(() => ({ + recordId: "record-create", + recoveryHandle: "R".repeat(43), + ownershipGeneration: 1, + })), + remove, + }; + Object.assign(supervisor, { + createOrReuseWorker: vi.fn(async (_owner: string, _command: unknown, onLaunch: (value: unknown) => void) => { + onLaunch(worker); + return worker; + }), + stopWorker, + scheduleOwnedWorkerCleanup: vi.fn(() => { + throw new Error("timer setup failed"); + }), + }); + supervisor.workers.set("worker-create", worker); + const command = { + type: "create_recoverable_owned_session", + requestId: "00112233445566778899aabbccddeeff", + expectedSupervisorGeneration: supervisor.generation, + correlationId: "correlation-create", + mcpOwnerId: "mcp-create", + recoveryConfig: { cwd: "/tmp/project" }, + config: { cwd: "/tmp/project" }, + launchEnv: {}, + launchEnvMode: "replace", + } as const satisfies Extract; + + await expect(supervisor.createRecoverableOwnedSession({ id: "creator" }, command)).rejects.toThrow( + "timer setup failed", + ); + expect(remove).toHaveBeenCalledWith("record-create"); + expect(worker.recoveryRecordId).toBeUndefined(); + expect(worker.ownerCleanupTimer).toBeUndefined(); + expect(stopWorker).toHaveBeenCalledWith(worker, true, true); + }); + + it("single-flights one stable prepare before draining mutations", async () => { + const supervisor = supervisorFixture(); + const workerIncarnation = "A".repeat(43); + const authority = { + workerId: "worker-prepare", + workerIncarnation, + activeSessionId: "active-prepare", + sessionId: "session-prepare", + correlationId: "correlation-prepare", + mcpOwnerId: "mcp-owner-before", + recoveryConfig: { cwd: "/tmp/project" }, + launchEnv: { PATH: "/caller/bin" }, + }; + const receipt = { + recordId: "record-prepare", + authority, + recoveryHandle: "R".repeat(43), + ownershipGeneration: 1, + phase: "prepared", + repeated: false, + }; + const workerClient = { matchesAuthenticatedIncarnation: vi.fn(() => true) }; + const worker = { + descriptor: { workerId: "worker-prepare", ownerClientId: "previous-owner" }, + workerIncarnation, + client: workerClient, + recoveryRecordId: receipt.recordId, + stopRevision: 0, + }; + supervisor.workers.set("worker-prepare", worker); + supervisor.clients = new Set(); + supervisor.scheduleOwnedWorkerCleanup = vi.fn(); + const beginAdoption = vi.fn(() => receipt); + supervisor.ownedSessionRecoveryStore = { + getByHandle: vi.fn(() => receipt), + digestRequest: vi.fn((value: unknown) => JSON.stringify(value)), + digestAuthority: vi.fn((value: unknown) => JSON.stringify(value)), + beginAdoption, + get: vi.fn(() => receipt), + rollbackAdoption: vi.fn(() => receipt), + }; + let releaseSnapshot!: () => void; + const snapshotGate = new Promise((resolve) => { + releaseSnapshot = resolve; + }); + const result = { + recoveryHandle: receipt.recoveryHandle, + proof: { + feature: "recoverable_owned_session_adoption_v1", + status: "adopted", + supervisorGeneration: supervisor.generation, + ownershipGeneration: 1, + activeSessionId: authority.activeSessionId, + sessionId: authority.sessionId, + correlationId: authority.correlationId, + lifecycle: { correlationId: authority.correlationId, expired: true, deliveryCrossed: true }, + cursor: { generation: "cursor-generation", sequence: 4 }, + mcpOwnerId: "mcp-owner-after", + }, + } as unknown as DaemonRecoverableOwnedSessionPrepareResult; + supervisor.prepareRecoverableOwnedSnapshot = vi.fn(async () => { + await snapshotGate; + return result; + }); + const physicalClient = { + id: "claimant", + socket: { destroyed: false }, + attachedActiveSessionIds: new Set(), + capabilities: new Set(), + }; + const otherPhysicalClient = { ...physicalClient, id: "other-claimant", socket: { destroyed: false } }; + const command = { + type: "prepare_recoverable_owned_session_adoption", + requestId: "00112233445566778899aabbccddeeff", + recoveryHandle: receipt.recoveryHandle, + expectedSupervisorGeneration: supervisor.generation, + activeSessionId: authority.activeSessionId, + sessionId: authority.sessionId, + correlationId: authority.correlationId, + cursor: { generation: "cursor-generation", sequence: 3 }, + previousMcpOwnerId: authority.mcpOwnerId, + mcpOwnerId: "mcp-owner-after", + recoveryConfig: authority.recoveryConfig, + launchEnv: authority.launchEnv, + clientId: "attached-claimant", + capabilities: [ + "attach_snapshot", + "event_sequence", + "correlated_prompt_lifecycle_v1", + "caller_owned_session_environment_cleanup_v1", + ], + } as const satisfies Extract; + + const first = supervisor.prepareRecoverableOwnedSessionAdoption(physicalClient, command); + const samePhysicalRetry = supervisor.prepareRecoverableOwnedSessionAdoption(physicalClient, command); + await expect(supervisor.prepareRecoverableOwnedSessionAdoption(otherPhysicalClient, command)).rejects.toThrow( + "Recoverable owned session adoption is unavailable", + ); + expect(beginAdoption).toHaveBeenCalledTimes(1); + expect(supervisor.prepareRecoverableOwnedSnapshot).toHaveBeenCalledTimes(1); + releaseSnapshot(); + await expect(Promise.all([first, samePhysicalRetry])).resolves.toEqual([result, result]); + const adoption = (worker as { recoverableAdoption?: unknown }).recoverableAdoption; + if (adoption) supervisor.rollbackRecoverableOwnedAdoption(worker, adoption); + }); + + it("commits the exact final retry when one daemon client recovers two sessions", async () => { + const supervisor = supervisorFixture(); + const actions: string[] = []; + const physicalClient = { + id: "shared-client", + socket: { destroyed: false }, + attachedActiveSessionIds: { add: (activeSessionId: string) => actions.push(`attach:${activeSessionId}`) }, + }; + const workerClient = { matchesAuthenticatedIncarnation: vi.fn(() => true) }; + const requestDigest = (requestId: string) => + JSON.stringify({ type: "prepare_recoverable_owned_session_adoption", requestId }); + const makeWorker = (suffix: string) => { + const requestId = suffix.toLowerCase().repeat(32); + const proof = { + feature: "recoverable_owned_session_adoption_v1" as const, + status: "adopted" as const, + supervisorGeneration: supervisor.generation, + ownershipGeneration: 1, + activeSessionId: `active-${suffix}`, + sessionId: `session-${suffix}`, + correlationId: `correlation-${suffix}`, + lifecycle: { correlationId: `correlation-${suffix}`, expired: true as const, deliveryCrossed: true }, + cursor: { generation: `cursor-${suffix}`, sequence: 3 }, + mcpOwnerId: `mcp-${suffix}`, + }; + const result = { recoveryHandle: suffix.repeat(43), proof } as DaemonRecoverableOwnedSessionPrepareResult; + const adoption = { + recordId: `record-${suffix}`, + requestIdDigest: requestDigest(requestId), + requestDigest: `digest-${suffix}`, + client: physicalClient, + workerClient, + workerIncarnation: `incarnation-${suffix}`, + workerStopRevision: 0, + previousOwnerClientId: "shared-client", + activeSessionId: proof.activeSessionId, + proof, + result, + frames: [{ payload: Buffer.from(`retained-${suffix}`) }], + bufferedBytes: 10, + finalRetry: true as const, + timeout: setTimeout(() => {}, 60_000), + }; + return { + descriptor: { workerId: `worker-${suffix}`, ownerClientId: "shared-client", lifecycle: "ready" }, + workerIncarnation: `incarnation-${suffix}`, + client: workerClient, + recoveryRecordId: `record-${suffix}`, + stopRevision: 0, + recoverableAdoption: adoption, + recoverableFinal: { + recordId: `record-${suffix}`, + requestIdDigest: adoption.requestIdDigest, + requestDigest: adoption.requestDigest, + ownerClientId: "shared-client", + result, + frames: [], + }, + }; + }; + const first = makeWorker("A"); + const second = makeWorker("B"); + supervisor.workers = new Map([ + ["worker-A", first], + ["worker-B", second], + ]); + supervisor.clients = new Set([physicalClient]); + supervisor.ownedSessionRecoveryStore = { + digestRequest: vi.fn((value: unknown) => JSON.stringify(value)), + getForConfirmation: vi.fn(() => ({ + recordId: "record-B", + authority: { workerIncarnation: "incarnation-B" }, + })), + }; + Object.assign(supervisor, { + validateRecoverableOwnedBuffer: vi.fn(), + writeSerialized: vi.fn((_client: unknown, payload: Buffer) => actions.push(payload.toString())), + syncWorkerExtensionUi: vi.fn(), + }); + const command = { + type: "commit_recoverable_owned_session_adoption", + requestId: "b".repeat(32), + expectedSupervisorGeneration: supervisor.generation, + recoveryHandle: "B".repeat(43), + proof: second.recoverableFinal.result.proof, + } as const satisfies Extract; + + await expect(supervisor.commitRecoverableOwnedSessionAdoption(physicalClient, command)).resolves.toEqual( + command.proof, + ); + expect(first.recoverableAdoption).toBeDefined(); + expect(second.recoverableAdoption).toBeUndefined(); + expect(actions).toEqual(["retained-B", "attach:active-B"]); + }); + + it("rejects incomplete replay, malformed snapshots, and lifecycle mismatches before ownership cut", async () => { + const supervisor = supervisorFixture(); + const activeSessionId = "active-proof"; + const sessionId = "session-proof"; + const correlationId = "correlation-proof"; + const cursor = { generation: "generation-proof", sequence: 5 }; + const command = { + type: "prepare_recoverable_owned_session_adoption", + requestId: "00112233445566778899aabbccddeeff", + recoveryHandle: "R".repeat(43), + expectedSupervisorGeneration: supervisor.generation, + activeSessionId, + sessionId, + correlationId, + cursor: { generation: cursor.generation, sequence: 4 }, + previousMcpOwnerId: "mcp-before", + mcpOwnerId: "mcp-after", + recoveryConfig: { cwd: "/tmp/project" }, + launchEnv: { PATH: "/caller/bin" }, + clientId: "claimant", + capabilities: ["attach_snapshot", "event_sequence", "correlated_prompt_lifecycle_v1"], + } as const satisfies Extract; + const lifecycle = { + correlationId, + phase: "queued" as const, + kind: "model_prompt" as const, + revision: 1, + deliveryCrossed: false, + }; + const attached = { + protocol: { name: "prime-agent.daemon" as const, version: 7 }, + activeSessionId, + snapshot: { + activeSessionId, + summary: { + id: activeSessionId, + activeSessionId, + lifecycle: "live" as const, + activity: "idle" as const, + isSessionActive: false, + sessionId, + cwd: "/tmp/project", + isStreaming: false, + isCompacting: false, + attachedClients: 1, + messageCount: 0, + sessionActions: { queuedCount: 0, steering: [], followUps: [] }, + }, + state: { sessionId }, + messages: [], + lastEventSequence: cursor.sequence, + lastEventCursor: cursor, + promptLifecycles: { records: [lifecycle], expired: [] }, + }, + replay: { status: "complete" as const, toSequence: cursor.sequence, toCursor: cursor }, + lastEventSequence: cursor.sequence, + lastEventCursor: cursor, + client: { id: "worker-client", capabilities: [...command.capabilities] }, + }; + const receipt = { + recordId: "record-proof", + authority: {}, + recoveryHandle: command.recoveryHandle, + ownershipGeneration: 1, + phase: "prepared", + repeated: false, + }; + const adoption = { + frames: [], + bufferedBytes: 0, + client: { capabilities: new Set(), supportsExtensionUi: false }, + }; + for (const mutate of [ + (value: typeof attached) => ({ ...value, replay: { ...value.replay, status: "partial" as const } }), + (value: typeof attached) => ({ + ...value, + snapshot: { ...value.snapshot, state: { sessionId: "wrong-session" } }, + }), + (value: typeof attached) => ({ + ...value, + snapshot: { ...value.snapshot, promptLifecycles: { records: [], expired: [] } }, + }), + ]) { + const worker = { + descriptor: { workerId: "worker-proof", lifecycle: "ready" as const }, + client: { + request: vi.fn(async () => ({ + type: "response", + command: "attach", + success: true, + data: mutate(attached), + })), + }, + }; + await expect( + supervisor.prepareRecoverableOwnedSnapshot(worker, adoption, receipt, command, Date.now() + 1_000), + ).rejects.toThrow("Recoverable owned session adoption is unavailable"); + } + }); + + it("contains a disconnected worker when its unconfirmed final receipt disappears", async () => { + vi.useFakeTimers(); + const supervisor = supervisorFixture(); + const worker = { + descriptor: { workerId: "worker-2", ownerClientId: "disconnected-owner" }, + recoveryRecordId: "record-2", + recoverableFinal: { recordId: "record-2" }, + ownerCleanupTimer: setTimeout(() => {}, 60_000) as NodeJS.Timeout | undefined, + }; + supervisor.workers.set("worker-2", worker); + supervisor.clients = new Set(); + supervisor.ownedSessionRecoveryStore = { get: vi.fn(() => undefined), remove: vi.fn() }; + supervisor.stopWorker = vi.fn(async () => undefined); + + supervisor.scheduleRecoverableOwnedConfirmationExpiry(worker, "record-2"); + await vi.advanceTimersByTimeAsync(RECOVERABLE_OWNED_CONFIRMATION_RETENTION_MS + 1); + + expect(supervisor.ownedSessionRecoveryStore.remove).toHaveBeenCalledWith("record-2"); + expect(supervisor.stopWorker).toHaveBeenCalledWith(worker, true, true); + expect(worker).toMatchObject({ + recoveryRecordId: undefined, + recoverableFinal: undefined, + ownerCleanupTimer: undefined, + }); + }); +}); diff --git a/packages/coding-agent/test/fixtures/eng-4606-update-launcher.ts b/packages/coding-agent/test/fixtures/eng-4606-update-launcher.ts index 1e7b693882..bdc455a3cc 100644 --- a/packages/coding-agent/test/fixtures/eng-4606-update-launcher.ts +++ b/packages/coding-agent/test/fixtures/eng-4606-update-launcher.ts @@ -15,7 +15,11 @@ const completionPath = requireEnvironment("ENG_4606_COMPLETION_PATH"); const pidPath = requireEnvironment("ENG_4606_PID_PATH"); const socketPath = requireEnvironment("ENG_4606_SOCKET_PATH"); const tsxPath = requireEnvironment("ENG_4606_TSX_PATH"); -const originActiveSessionId = requireEnvironment("PRIME_AGENT_INTERNAL_DAEMON_WORKER_ACTIVE_SESSION_ID"); +if (process.env.PRIME_AGENT_INTERNAL_DAEMON_WORKER_ACTIVE_SESSION_ID !== undefined) { + throw new Error("Private worker bootstrap environment leaked into the updater"); +} +const originActiveSessionId = process.argv[2]; +if (!originActiveSessionId) throw new Error("Missing updater origin active session id argument"); process.argv[1] = cliPath; process.execArgv.splice(0, process.execArgv.length, tsxPath); diff --git a/packages/coding-agent/test/mcp-manager.test.ts b/packages/coding-agent/test/mcp-manager.test.ts index fc3589f4fa..8e9d4baac5 100644 --- a/packages/coding-agent/test/mcp-manager.test.ts +++ b/packages/coding-agent/test/mcp-manager.test.ts @@ -281,7 +281,15 @@ describe("McpManager", () => { credentialSource: "acp", }); - expect(manager.replaceAcpServers([], "owner-a")).toBe(true); + expect(manager.transferAcpServersOwner("owner-a", "owner-b")).toBe(true); + expect(await handlers["mcp.config"]({ server: "task" })).toEqual({ + type: "http", + url: "https://task.example/mcp", + headers: { Authorization: "Bearer task-token" }, + credentialSource: "acp", + }); + expect(() => manager.transferAcpServersOwner("owner-a", "owner-c")).toThrow("owned by another client"); + expect(manager.replaceAcpServers([], "owner-b")).toBe(true); expect(await handlers["mcp.config"]({ server: "task" })).toEqual({ type: "http", url: "https://user.example/mcp", @@ -289,4 +297,63 @@ describe("McpManager", () => { }); expect(authStorage.get("mcp:task")).toMatchObject({ access: "stored-oauth-token" }); }); + + it("proves idempotent ACP owner transfer and rollback after response loss", () => { + const manager = new McpManager({ authStorage }); + manager.replaceAcpServers( + [{ name: "task", type: "http", url: "https://task.example/mcp", headers: {} }], + "owner-a", + ); + const transferred = manager.transferAcpServersOwnerTransaction("transaction-a", "owner-a", "owner-b"); + expect(transferred).toEqual({ + transactionId: "transaction-a", + previousOwnerId: "owner-a", + nextOwnerId: "owner-b", + changed: true, + state: "transferred", + }); + expect(manager.transferAcpServersOwnerTransaction("transaction-a", "owner-a", "owner-b")).toEqual(transferred); + expect(manager.queryAcpServersOwnerTransaction("transaction-a", "owner-a", "owner-b")).toEqual(transferred); + expect(() => manager.queryAcpServersOwnerTransaction("transaction-a", "owner-a", "owner-c")).toThrow( + "transaction tuple changed", + ); + + const rolledBack = manager.rollbackAcpServersOwnerTransaction("transaction-a", "owner-a", "owner-b"); + expect(rolledBack).toEqual({ ...transferred, state: "rolled_back" }); + expect(manager.rollbackAcpServersOwnerTransaction("transaction-a", "owner-a", "owner-b")).toEqual(rolledBack); + expect(manager.queryAcpServersOwnerTransaction("transaction-a", "owner-a", "owner-b")).toEqual(rolledBack); + const transferredAgain = manager.transferAcpServersOwnerTransaction("transaction-a", "owner-a", "owner-b"); + expect(transferredAgain).toEqual(transferred); + expect(manager.queryAcpServersOwnerTransaction("transaction-a", "owner-a", "owner-b")).toEqual(transferred); + expect(() => manager.transferAcpServersOwner("owner-a", "owner-c")).toThrow("owned by another client"); + expect(manager.replaceAcpServers([], "owner-b")).toBe(true); + + const empty = new McpManager({ authStorage }); + expect(empty.transferAcpServersOwnerTransaction("transaction-empty", "owner-a", "owner-b")).toMatchObject({ + changed: false, + state: "transferred", + }); + expect(empty.rollbackAcpServersOwnerTransaction("transaction-empty", "owner-a", "owner-b")).toMatchObject({ + changed: false, + state: "rolled_back", + }); + empty.retireAcpServersOwnerTransaction("transaction-empty", "owner-a", "owner-b"); + expect(() => empty.queryAcpServersOwnerTransaction("transaction-empty", "owner-a", "owner-b")).toThrow( + "transaction is unknown", + ); + }); + + it("bounds unconfirmed ACP owner transfer receipts under churn", () => { + const manager = new McpManager({ authStorage }); + for (let index = 0; index < 400; index++) { + manager.transferAcpServersOwnerTransaction(`transaction-${index}`, "owner-a", "owner-b"); + } + const receipts = (manager as unknown as { acpOwnerTransfers: Map }).acpOwnerTransfers; + expect(receipts.size).toBeLessThanOrEqual(256); + expect(() => manager.queryAcpServersOwnerTransaction("transaction-0", "owner-a", "owner-b")).toThrow( + "transaction is unknown", + ); + manager.retireAcpServersOwnerTransaction("transaction-399", "owner-a", "owner-b"); + expect(receipts.has("transaction-399")).toBe(false); + }); }); diff --git a/packages/coding-agent/test/owned-session-adoption-buffer.test.ts b/packages/coding-agent/test/owned-session-adoption-buffer.test.ts new file mode 100644 index 0000000000..a555f71137 --- /dev/null +++ b/packages/coding-agent/test/owned-session-adoption-buffer.test.ts @@ -0,0 +1,60 @@ +import { describe, expect, it } from "vitest"; +import { + appendRecoverableOwnedFrame, + RECOVERABLE_OWNED_MAX_BUFFERED_BYTES, + RECOVERABLE_OWNED_MAX_BUFFERED_FRAMES, + reconcileRecoverableOwnedFrames, +} from "../src/modes/daemon/owned-session-adoption-buffer.js"; +import { OWNED_SESSION_ADOPTION_UNAVAILABLE } from "../src/modes/daemon/owned-session-recovery-store.js"; + +function frame(sequence: number, generation = "generation-a", bytes = 1) { + return { payload: Buffer.alloc(bytes), cursor: { generation, sequence }, message: `event-${sequence}` }; +} + +describe("recoverable owned-session adoption buffers", () => { + it("drops snapshot-covered frames and retains strict post-snapshot order exactly once", () => { + const reconciled = reconcileRecoverableOwnedFrames([frame(4), frame(5), frame(6), frame(7)], { + generation: "generation-a", + sequence: 5, + }); + expect(reconciled.frames.map((entry) => entry.message)).toEqual(["event-6", "event-7"]); + expect(reconciled.bufferedBytes).toBe(2); + }); + + it("deduplicates exact cursor and canonical payload retries but rejects conflicting duplicates", () => { + const frames = [frame(6), frame(7)]; + let bufferedBytes = 2; + bufferedBytes = appendRecoverableOwnedFrame(frames, bufferedBytes, frame(6)); + bufferedBytes = appendRecoverableOwnedFrame(frames, bufferedBytes, frame(7)); + expect(frames.map((entry) => entry.message)).toEqual(["event-6", "event-7"]); + expect(bufferedBytes).toBe(2); + expect(() => + appendRecoverableOwnedFrame(frames, bufferedBytes, { + ...frame(6), + payload: Buffer.from("conflict"), + }), + ).toThrow(OWNED_SESSION_ADOPTION_UNAVAILABLE); + }); + + it("fails closed on malformed, cross-generation, gapped, count-overflow, and byte-overflow frames", () => { + for (const frames of [ + [{ payload: Buffer.alloc(1), message: "missing" }], + [frame(6, "generation-b")], + [frame(7)], + ]) { + expect(() => reconcileRecoverableOwnedFrames(frames, { generation: "generation-a", sequence: 5 })).toThrow( + OWNED_SESSION_ADOPTION_UNAVAILABLE, + ); + } + expect(() => + appendRecoverableOwnedFrame( + Array.from({ length: RECOVERABLE_OWNED_MAX_BUFFERED_FRAMES }, (_, index) => frame(index)), + RECOVERABLE_OWNED_MAX_BUFFERED_FRAMES, + frame(RECOVERABLE_OWNED_MAX_BUFFERED_FRAMES), + ), + ).toThrow(OWNED_SESSION_ADOPTION_UNAVAILABLE); + expect(() => appendRecoverableOwnedFrame([], RECOVERABLE_OWNED_MAX_BUFFERED_BYTES, frame(1))).toThrow( + OWNED_SESSION_ADOPTION_UNAVAILABLE, + ); + }); +}); diff --git a/packages/coding-agent/test/owned-session-recovery-retention.test.ts b/packages/coding-agent/test/owned-session-recovery-retention.test.ts new file mode 100644 index 0000000000..263cfcfed1 --- /dev/null +++ b/packages/coding-agent/test/owned-session-recovery-retention.test.ts @@ -0,0 +1,42 @@ +import { describe, expect, it, vi } from "vitest"; +import { + RECOVERABLE_OWNED_CONFIRMATION_RETENTION_MS, + RECOVERABLE_OWNED_DISCONNECTED_RETENTION_MS, + RECOVERABLE_OWNED_NO_LIFECYCLE_RETENTION_MS, + RECOVERABLE_OWNED_PREPARE_TIMEOUT_MS, + RECOVERABLE_OWNED_TERMINAL_RETENTION_MS, + recoverableOwnedRetention, +} from "../src/modes/daemon/owned-session-recovery-retention.js"; + +describe("recoverable owned-session retention", () => { + it("uses the fixed no-lifecycle, prepared, disconnected, and terminal bounds", () => { + vi.useFakeTimers(); + vi.setSystemTime(1_000); + expect(RECOVERABLE_OWNED_PREPARE_TIMEOUT_MS).toBe(15_000); + expect(RECOVERABLE_OWNED_CONFIRMATION_RETENTION_MS).toBe(15 * 60_000); + expect(recoverableOwnedRetention({ now: Date.now(), busy: false, hasLifecycle: false, terminal: false })).toEqual( + { + retentionMs: RECOVERABLE_OWNED_NO_LIFECYCLE_RETENTION_MS, + }, + ); + const active = recoverableOwnedRetention({ now: Date.now(), busy: true, hasLifecycle: true, terminal: false }); + expect(active).toEqual({ + retentionMs: RECOVERABLE_OWNED_DISCONNECTED_RETENTION_MS, + activeDeadline: Date.now() + RECOVERABLE_OWNED_DISCONNECTED_RETENTION_MS, + }); + vi.advanceTimersByTime(RECOVERABLE_OWNED_DISCONNECTED_RETENTION_MS - 20_000); + expect( + recoverableOwnedRetention({ + now: Date.now(), + busy: false, + hasLifecycle: true, + terminal: true, + activeDeadline: active.activeDeadline, + }), + ).toEqual({ retentionMs: 20_000, activeDeadline: active.activeDeadline }); + expect(recoverableOwnedRetention({ now: Date.now(), busy: false, hasLifecycle: true, terminal: true })).toEqual({ + retentionMs: RECOVERABLE_OWNED_TERMINAL_RETENTION_MS, + }); + vi.useRealTimers(); + }); +}); diff --git a/packages/coding-agent/test/owned-session-recovery-store.test.ts b/packages/coding-agent/test/owned-session-recovery-store.test.ts new file mode 100644 index 0000000000..a047e7c09e --- /dev/null +++ b/packages/coding-agent/test/owned-session-recovery-store.test.ts @@ -0,0 +1,356 @@ +import { randomBytes, randomUUID } from "node:crypto"; +import { + chmodSync, + lstatSync, + mkdirSync, + mkdtempSync, + readdirSync, + readFileSync, + rmSync, + statSync, + symlinkSync, + writeFileSync, +} from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { afterEach, describe, expect, it } from "vitest"; +import { + OWNED_SESSION_ADOPTION_UNAVAILABLE, + OwnedSessionRecoveryStore, +} from "../src/modes/daemon/owned-session-recovery-store.js"; + +const directories: string[] = []; + +afterEach(() => { + for (const directory of directories.splice(0)) rmSync(directory, { recursive: true, force: true }); +}); + +function fixture(now = 1_000) { + const directory = mkdtempSync(join(tmpdir(), "owned-recovery-store-")); + directories.push(directory); + let clock = now; + const store = new OwnedSessionRecoveryStore<{ worker: string }>(directory, randomUUID(), { + secret: randomBytes(32), + now: () => clock, + }); + const createRequest = "create-canary-request"; + const authority = { + activeSessionId: "active-canary", + sessionId: "session-canary", + correlationId: "correlation-canary", + mcpOwnerId: "mcp-canary", + }; + const created = store.create({ + requestIdDigest: store.digestRequest({ requestId: createRequest }), + requestDigest: store.digestRequest({ requestId: createRequest, secret: "prompt-canary" }), + authorityDigest: store.digestAuthority(authority), + authority: { worker: "private-worker" }, + expiresAt: 10_000, + }); + return { + directory, + store, + created, + authority, + setNow(value: number) { + clock = value; + }, + }; +} + +function bytes(directory: string): string { + return readdirSync(directory) + .map((name) => readFileSync(join(directory, name), "utf8")) + .join("\n"); +} + +describe("OwnedSessionRecoveryStore", () => { + it("writes atomic private records without bearer or authority canaries", () => { + const { directory, created } = fixture(); + expect(statSync(directory).mode & 0o777).toBe(0o700); + const names = readdirSync(directory); + expect(names).toHaveLength(1); + expect(statSync(join(directory, names[0]!)).mode & 0o777).toBe(0o600); + const persisted = bytes(directory); + for (const canary of [ + created.recoveryHandle, + "create-canary-request", + "prompt-canary", + "active-canary", + "session-canary", + "correlation-canary", + "mcp-canary", + "private-worker", + ]) { + expect(persisted).not.toContain(canary); + } + expect(readdirSync(directory).some((name) => name.endsWith(".tmp"))).toBe(false); + }); + + it("converges retries on one deterministic next handle until confirmation", () => { + const { store, created, authority } = fixture(); + const requestIdDigest = store.digestRequest({ requestId: "adopt-request-one" }); + const requestDigest = store.digestRequest({ requestId: "adopt-request-one", cursor: 1 }); + const authorityDigest = store.digestAuthority(authority); + const first = store.beginAdoption({ + recoveryHandle: created.recoveryHandle, + requestIdDigest, + requestDigest, + authorityDigest, + expectedSupervisorGeneration: (store as unknown as { supervisorGeneration: string }).supervisorGeneration, + expiresAt: 9_000, + }); + const retry = store.beginAdoption({ + recoveryHandle: created.recoveryHandle, + requestIdDigest, + requestDigest, + authorityDigest, + expectedSupervisorGeneration: (store as unknown as { supervisorGeneration: string }).supervisorGeneration, + expiresAt: 9_000, + }); + expect(retry.recoveryHandle).toBe(first.recoveryHandle); + expect(retry.ownershipGeneration).toBe(first.ownershipGeneration); + store.markCommitting(first.recordId); + store.markFinal(first.recordId, 9_000); + const finalRetry = store.beginAdoption({ + recoveryHandle: created.recoveryHandle, + requestIdDigest, + requestDigest, + authorityDigest, + expectedSupervisorGeneration: (store as unknown as { supervisorGeneration: string }).supervisorGeneration, + expiresAt: 9_000, + }); + expect(finalRetry).toMatchObject({ phase: "final", recoveryHandle: first.recoveryHandle }); + store.confirm({ recoveryHandle: first.recoveryHandle, requestIdDigest, authorityDigest, expiresAt: 9_000 }); + expect(() => + store.beginAdoption({ + recoveryHandle: created.recoveryHandle, + requestIdDigest, + requestDigest, + authorityDigest, + expectedSupervisorGeneration: (store as unknown as { supervisorGeneration: string }).supervisorGeneration, + expiresAt: 9_000, + }), + ).toThrow(OWNED_SESSION_ADOPTION_UNAVAILABLE); + }); + + it("uses one non-enumerating error for wrong, stale, mismatched, and expired authority", () => { + const { store, created, authority, setNow } = fixture(); + const authorityDigest = store.digestAuthority(authority); + const requestIdDigest = store.digestRequest({ requestId: "adopt-request-two" }); + const requestDigest = store.digestRequest({ requestId: "adopt-request-two" }); + const attempts = [ + () => + store.beginAdoption({ + recoveryHandle: randomBytes(32).toString("base64url"), + requestIdDigest, + requestDigest, + authorityDigest, + expectedSupervisorGeneration: "stale", + expiresAt: 9_000, + }), + () => + store.beginAdoption({ + recoveryHandle: created.recoveryHandle, + requestIdDigest, + requestDigest, + authorityDigest: store.digestAuthority({ other: true }), + expectedSupervisorGeneration: "stale", + expiresAt: 9_000, + }), + ]; + for (const attempt of attempts) { + let message = ""; + try { + attempt(); + } catch (error) { + message = error instanceof Error ? error.message : String(error); + } + expect(message).toBe(OWNED_SESSION_ADOPTION_UNAVAILABLE); + } + setNow(10_001); + expect(() => + store.beginAdoption({ + recoveryHandle: created.recoveryHandle, + requestIdDigest, + requestDigest, + authorityDigest, + expectedSupervisorGeneration: "stale", + expiresAt: 11_000, + }), + ).toThrow(OWNED_SESSION_ADOPTION_UNAVAILABLE); + }); + + it("fails closed across replacement secrets and removes corrupt records", () => { + const { directory, created } = fixture(); + writeFileSync(join(directory, `${randomUUID()}.json`), "{not-json", { mode: 0o666 }); + chmodSync(directory, 0o777); + const replacement = new OwnedSessionRecoveryStore<{ worker: string }>(directory, randomUUID()); + expect(statSync(directory).mode & 0o777).toBe(0o700); + expect(readdirSync(directory)).toEqual([]); + expect(() => + replacement.beginAdoption({ + recoveryHandle: created.recoveryHandle, + requestIdDigest: replacement.digestRequest({ requestId: "retry" }), + requestDigest: replacement.digestRequest({ requestId: "retry" }), + authorityDigest: replacement.digestAuthority({}), + expectedSupervisorGeneration: "stale", + expiresAt: 20_000, + }), + ).toThrow(OWNED_SESSION_ADOPTION_UNAVAILABLE); + }); + + it.skipIf(process.platform === "win32")("rejects a recovery-directory symlink without touching its victim", () => { + const root = mkdtempSync(join(tmpdir(), "owned-recovery-symlink-")); + directories.push(root); + const victim = join(root, "victim"); + const recoveryPath = join(root, "recovery"); + mkdirSync(victim, { mode: 0o755 }); + const canaryPath = join(victim, "canary.txt"); + writeFileSync(canaryPath, "victim-canary", { mode: 0o644 }); + symlinkSync(victim, recoveryPath); + + expect(() => new OwnedSessionRecoveryStore(recoveryPath, randomUUID())).toThrow("must be a private directory"); + expect(readFileSync(canaryPath, "utf8")).toBe("victim-canary"); + expect(statSync(victim).mode & 0o777).toBe(0o755); + }); + + it.skipIf(process.platform === "win32")("unlinks its regular temp file when an atomic replace fails", () => { + const root = mkdtempSync(join(tmpdir(), "owned-recovery-persist-error-")); + directories.push(root); + const directory = join(root, "recovery"); + const store = new OwnedSessionRecoveryStore<{ worker: string }>(directory, randomUUID()); + const victim = join(root, "victim.txt"); + writeFileSync(victim, "victim-canary"); + const recordPath = join(directory, `${randomUUID()}.json`); + symlinkSync(victim, recordPath); + (store as unknown as { pathFor(recordId: string): string }).pathFor = () => recordPath; + + expect(() => + store.create({ + requestIdDigest: store.digestRequest({ requestId: "persist-failure" }), + requestDigest: store.digestRequest({ requestId: "persist-failure", cursor: 1 }), + authorityDigest: store.digestAuthority({ worker: "worker" }), + authority: { worker: "worker" }, + expiresAt: Date.now() + 60_000, + }), + ).toThrow("not a regular file"); + expect(readdirSync(directory)).toEqual([recordPath.split("/").at(-1)]); + expect(lstatSync(recordPath).isSymbolicLink()).toBe(true); + expect(readFileSync(victim, "utf8")).toBe("victim-canary"); + }); + + it.skipIf(process.platform === "win32")( + "reclaims only owned regular stale temps and never reads or removes symlink entries", + () => { + const root = mkdtempSync(join(tmpdir(), "owned-recovery-startup-")); + directories.push(root); + const directory = join(root, "recovery"); + mkdirSync(directory, { mode: 0o700 }); + const victim = join(root, "victim.txt"); + writeFileSync(victim, "victim-canary"); + const recordId = randomUUID(); + const staleTemp = `${recordId}.json.${process.pid}.${randomUUID()}.tmp`; + const foreignTemp = "foreign.tmp"; + const recordSymlink = `${randomUUID()}.json`; + const tempSymlink = `${randomUUID()}.json.${process.pid}.${randomUUID()}.tmp`; + writeFileSync(join(directory, staleTemp), "stale"); + writeFileSync(join(directory, foreignTemp), "foreign"); + symlinkSync(victim, join(directory, recordSymlink)); + symlinkSync(victim, join(directory, tempSymlink)); + + new OwnedSessionRecoveryStore(directory, randomUUID()); + + expect(readdirSync(directory).sort()).toEqual([foreignTemp, recordSymlink, tempSymlink].sort()); + expect(lstatSync(join(directory, recordSymlink)).isSymbolicLink()).toBe(true); + expect(lstatSync(join(directory, tempSymlink)).isSymbolicLink()).toBe(true); + expect(readFileSync(victim, "utf8")).toBe("victim-canary"); + expect(readFileSync(join(directory, foreignTemp), "utf8")).toBe("foreign"); + }, + ); + + it("permanently retires create replay when adoption first begins", () => { + const { store, created, authority } = fixture(); + const createRequestIdDigest = store.digestRequest({ requestId: "create-canary-request" }); + const createRequestDigest = store.digestRequest({ + requestId: "create-canary-request", + secret: "prompt-canary", + }); + expect(store.getByCreateRequest(createRequestIdDigest, createRequestDigest)?.recoveryHandle).toBe( + created.recoveryHandle, + ); + const authorityDigest = store.digestAuthority(authority); + store.beginAdoption({ + recoveryHandle: created.recoveryHandle, + requestIdDigest: store.digestRequest({ requestId: "adopt-retire-create" }), + requestDigest: store.digestRequest({ requestId: "adopt-retire-create", cursor: 1 }), + authorityDigest, + expectedSupervisorGeneration: (store as unknown as { supervisorGeneration: string }).supervisorGeneration, + expiresAt: 9_000, + }); + expect(() => store.getByCreateRequest(createRequestIdDigest, createRequestDigest)).toThrow( + OWNED_SESSION_ADOPTION_UNAVAILABLE, + ); + store.rollbackAdoption(created.recordId, authorityDigest, 9_000); + expect(() => store.getByCreateRequest(createRequestIdDigest, createRequestDigest)).toThrow( + OWNED_SESSION_ADOPTION_UNAVAILABLE, + ); + expect(() => + store.create({ + requestIdDigest: createRequestIdDigest, + requestDigest: createRequestDigest, + authorityDigest, + authority: { worker: "private-worker" }, + expiresAt: 9_000, + }), + ).toThrow(OWNED_SESSION_ADOPTION_UNAVAILABLE); + const persisted = JSON.parse(bytes((store as unknown as { directory: string }).directory)) as { + createRequestReplayRetired?: boolean; + }; + expect(persisted.createRequestReplayRetired).toBe(true); + }); + + it("makes removed handles unavailable for adoption and confirmation", () => { + const { store, created } = fixture(); + store.remove(created.recordId); + expect(() => store.getByHandle(created.recoveryHandle)).toThrow(OWNED_SESSION_ADOPTION_UNAVAILABLE); + expect(() => + store.getForConfirmation(created.recoveryHandle, store.digestRequest({ requestId: randomUUID() })), + ).toThrow(OWNED_SESSION_ADOPTION_UNAVAILABLE); + }); + + it("expires records with a fake clock", () => { + const { directory, store, created, setNow } = fixture(); + setNow(10_001); + expect(store.sweep()).toEqual([created.recordId]); + expect(readdirSync(directory)).toEqual([]); + }); + + it("rolls back a prepared claimant and resets the old owner's disconnect deadline on reconnect", () => { + const { store, created, authority, setNow } = fixture(); + const authorityDigest = store.digestAuthority(authority); + const requestIdDigest = store.digestRequest({ requestId: "adopt-reconnect" }); + const requestDigest = store.digestRequest({ requestId: "adopt-reconnect", cursor: 1 }); + const prepared = store.beginAdoption({ + recoveryHandle: created.recoveryHandle, + requestIdDigest, + requestDigest, + authorityDigest, + expectedSupervisorGeneration: (store as unknown as { supervisorGeneration: string }).supervisorGeneration, + expiresAt: 1_500, + }); + expect(prepared.ownershipGeneration).toBe(1); + const rolledBack = store.rollbackAdoption(created.recordId, authorityDigest, 4_000); + expect(rolledBack).toMatchObject({ phase: "disconnected", ownershipGeneration: 0 }); + expect(rolledBack.recoveryHandle).toBe(created.recoveryHandle); + + setNow(2_000); + const connected = store.markConnected(created.recordId, authorityDigest, Number.MAX_SAFE_INTEGER); + expect(connected.phase).toBe("connected"); + setNow(5_000); + expect(store.get(created.recordId)?.phase).toBe("connected"); + store.markDisconnected(created.recordId, 8_000); + setNow(7_000); + expect(store.get(created.recordId)?.phase).toBe("disconnected"); + }); +}); diff --git a/packages/coding-agent/test/recoverable-owned-session-exports.test.ts b/packages/coding-agent/test/recoverable-owned-session-exports.test.ts new file mode 100644 index 0000000000..5f5c6c281b --- /dev/null +++ b/packages/coding-agent/test/recoverable-owned-session-exports.test.ts @@ -0,0 +1,47 @@ +import { describe, expectTypeOf, it } from "vitest"; +import type { + DaemonRecoverableOwnedSessionAdoptionOptions as RootAdoptionOptions, + DaemonRecoverableOwnedSessionAdoptionProof as RootAdoptionProof, + DaemonRecoverableOwnedSessionAdoptionResult as RootAdoptionResult, + DaemonRecoverableOwnedSessionConfirmResult as RootConfirmResult, + DaemonRecoverableOwnedSessionCreateResult as RootCreateResult, + DaemonEventCursor as RootCursor, + DaemonRecoverableOwnedSessionPrepareResult as RootPrepareResult, +} from "../src/index.js"; +import type { + DaemonRecoverableOwnedSessionAdoptionOptions as AgentBarrelAdoptionOptions, + DaemonRecoverableOwnedSessionAdoptionProof as AgentBarrelAdoptionProof, + DaemonRecoverableOwnedSessionAdoptionResult as AgentBarrelAdoptionResult, + DaemonRecoverableOwnedSessionConfirmResult as AgentBarrelConfirmResult, + DaemonRecoverableOwnedSessionCreateResult as AgentBarrelCreateResult, + DaemonEventCursor as AgentBarrelCursor, + DaemonRecoverableOwnedSessionPrepareResult as AgentBarrelPrepareResult, +} from "../src/modes/agent-connection/index.js"; +import type { + DaemonRecoverableOwnedSessionAdoptionOptions as ModesAdoptionOptions, + DaemonRecoverableOwnedSessionAdoptionProof as ModesAdoptionProof, + DaemonRecoverableOwnedSessionAdoptionResult as ModesAdoptionResult, + DaemonRecoverableOwnedSessionConfirmResult as ModesConfirmResult, + DaemonRecoverableOwnedSessionCreateResult as ModesCreateResult, + DaemonEventCursor as ModesCursor, + DaemonRecoverableOwnedSessionPrepareResult as ModesPrepareResult, +} from "../src/modes/index.js"; + +describe("recoverable owned-session public type exports", () => { + it("compile through the root and public barrels", () => { + expectTypeOf().toEqualTypeOf(); + expectTypeOf().toEqualTypeOf(); + expectTypeOf().toEqualTypeOf(); + expectTypeOf().toEqualTypeOf(); + expectTypeOf().toEqualTypeOf(); + expectTypeOf().toEqualTypeOf(); + expectTypeOf().toEqualTypeOf(); + expectTypeOf().toEqualTypeOf(); + expectTypeOf().toEqualTypeOf(); + expectTypeOf().toEqualTypeOf(); + expectTypeOf().toEqualTypeOf(); + expectTypeOf().toEqualTypeOf(); + expectTypeOf().toEqualTypeOf(); + expectTypeOf().toEqualTypeOf(); + }); +}); diff --git a/packages/coding-agent/test/suite/regressions/4602-snapshot-transfer-idempotency.test.ts b/packages/coding-agent/test/suite/regressions/4602-snapshot-transfer-idempotency.test.ts index ab8eabd0ac..b6eea44bde 100644 --- a/packages/coding-agent/test/suite/regressions/4602-snapshot-transfer-idempotency.test.ts +++ b/packages/coding-agent/test/suite/regressions/4602-snapshot-transfer-idempotency.test.ts @@ -28,7 +28,12 @@ const snapshotId = "snapshot-4602"; interface WorkerHarness { descriptor: { workerId: string; rootActiveSessionId: string; lifecycle: "ready" | "recovering"; pid: number }; authorizedActiveSessionIds: Set; - client?: { close: ReturnType; request: ReturnType }; + client?: { + close: ReturnType; + request: ReturnType; + matchesAuthenticatedIncarnation: (incarnation: string) => boolean; + }; + workerIncarnation?: string; summaries: Map; snapshotCache: Map; transcriptCaches: Map; @@ -126,7 +131,8 @@ function workerHarness() { pid: 987_654_321, }, authorizedActiveSessionIds: new Set([activeSessionId]), - client: { close, request }, + client: { close, request, matchesAuthenticatedIncarnation: (incarnation) => incarnation === "incarnation-4602" }, + workerIncarnation: "incarnation-4602", summaries: new Map([[activeSessionId, summary()]]), snapshotCache: new Map(), transcriptCaches: new Map(), diff --git a/packages/coding-agent/test/suite/regressions/4603-worker-recovery.test.ts b/packages/coding-agent/test/suite/regressions/4603-worker-recovery.test.ts index 19ab4a7f43..9a227a3330 100644 --- a/packages/coding-agent/test/suite/regressions/4603-worker-recovery.test.ts +++ b/packages/coding-agent/test/suite/regressions/4603-worker-recovery.test.ts @@ -12,6 +12,7 @@ import { } from "node:fs"; import { createConnection, type Socket } from "node:net"; import { join, resolve } from "node:path"; +import { Writable } from "node:stream"; import { afterEach, describe, expect, it } from "vitest"; import { APP_NAME, ENV_AGENT_DIR } from "../../../src/config.js"; import { getProcessStartId } from "../../../src/core/session-lease.js"; @@ -26,6 +27,8 @@ import { import { DAEMON_WORKER_ACTIVE_SESSION_ID_ENV, DAEMON_WORKER_ROLE_ENV, + DAEMON_WORKER_STARTUP_GATE_COMMIT, + DAEMON_WORKER_STARTUP_GATE_FD_ENV, DAEMON_WORKER_SUPERVISOR_SOCKET_ENV, DAEMON_WORKER_TOKEN_ENV, type DaemonWorkerDescriptor, @@ -94,6 +97,7 @@ const fixturePath = resolve(__dirname, "../../fixtures/eng-4600-supervisor-fixtu const fauxExtensionPath = resolve(__dirname, "../../fixtures/eng-4600-faux-extension.ts"); const cliPath = resolve(__dirname, "../../../src/cli.ts"); const tsxPath = resolve(__dirname, "../../../../../node_modules/tsx/dist/cli.mjs"); +const tsxLoaderPath = resolve(__dirname, "../../../../../node_modules/tsx/dist/loader.mjs"); const tsconfigPath = resolve(__dirname, "../../../../../tsconfig.json"); const supervisorRegistryDirEnv = "PRIME_AGENT_INTERNAL_DAEMON_SUPERVISOR_REGISTRY_DIR"; const handles = new Set(); @@ -174,10 +178,11 @@ function spawnStandaloneWorker( token: string, extraEnv: NodeJS.ProcessEnv = {}, ): ProcessHandle { - return trackProcess( + // The tsx CLI wrapper drops fd 3 when it relaunches Node, so load tsx in-process. + const worker = trackProcess( spawn( paths.executablePath, - [tsxPath, cliPath, "--mode", "daemon", "--daemon-socket", workerSocketPath, "--offline"], + ["--import", tsxLoaderPath, cliPath, "--mode", "daemon", "--daemon-socket", workerSocketPath, "--offline"], { cwd: paths.agentDir, env: { @@ -188,15 +193,22 @@ function spawnStandaloneWorker( [DAEMON_WORKER_ROLE_ENV]: "1", [DAEMON_WORKER_TOKEN_ENV]: token, [DAEMON_WORKER_ACTIVE_SESSION_ID_ENV]: "eng-4603-worker", + [DAEMON_WORKER_STARTUP_GATE_FD_ENV]: "3", [DAEMON_WORKER_SUPERVISOR_SOCKET_ENV]: paths.socketPath, PI_OFFLINE: "1", TSX_TSCONFIG_PATH: tsconfigPath, }, - stdio: ["ignore", "pipe", "pipe"], + stdio: ["ignore", "pipe", "pipe", "pipe"], }, ), "worker", ); + const startupGate = worker.child.stdio[3]; + if (!(startupGate instanceof Writable)) { + throw new Error("Standalone worker startup gate is not writable"); + } + startupGate.end(DAEMON_WORKER_STARTUP_GATE_COMMIT); + return worker; } function trackProcess(child: ChildProcess, role: FixtureProcessIdentity["role"]): ProcessHandle { @@ -342,6 +354,9 @@ function isFixtureDescendant(pid: number, rootPid: number, processes: Map { processStartId?: string; }; expect(record.pid).toBe(process.pid); - expect(record.processStartId).toBe(getProcessStartId(process.pid)); + const processStartId = record.processStartId; + expect(processStartId).toBe(getProcessStartId(process.pid)); + if (!processStartId) throw new Error("Shutdown admission did not capture the Vitest process identity"); + expect(() => signalFixtureProcess({ pid: record.pid, processStartId, role: "supervisor" }, "SIGSTOP")).toThrow( + `Refusing to signal current Vitest process ${process.pid} during fixture cleanup`, + ); const renewal = Reflect.get(first, "renewal") as object | undefined; const refreshTimer = renewal ? (Reflect.get(renewal, "refreshTimer") as ReturnType | undefined) diff --git a/packages/coding-agent/test/suite/regressions/4606-update-restart-coordinator.test.ts b/packages/coding-agent/test/suite/regressions/4606-update-restart-coordinator.test.ts index 26ce76e1b5..88ae62863e 100644 --- a/packages/coding-agent/test/suite/regressions/4606-update-restart-coordinator.test.ts +++ b/packages/coding-agent/test/suite/regressions/4606-update-restart-coordinator.test.ts @@ -385,7 +385,9 @@ describe("ENG-4606 update restart coordinator", () => { ), ); const originalActiveSessionId = created.activeSessionId ?? created.id; - const updateCommand = [process.execPath, tsxPath, launcherFixturePath].map(shellQuote).join(" "); + const updateCommand = [process.execPath, tsxPath, launcherFixturePath, originalActiveSessionId] + .map(shellQuote) + .join(" "); const executeResponse = await client.request( { type: "execute_bash", activeSessionId: originalActiveSessionId, command: updateCommand }, 5000, diff --git a/scripts/pylon-prime-agent-release.test.mjs b/scripts/pylon-prime-agent-release.test.mjs index e6cbabe38b..732ec89106 100644 --- a/scripts/pylon-prime-agent-release.test.mjs +++ b/scripts/pylon-prime-agent-release.test.mjs @@ -264,6 +264,7 @@ test("smokes the exact caller-owned session SDK contract", () => { "bounded_daemon_ingress_v1", "negotiated_daemon_session_capabilities_v1", "caller_owned_session_environment_cleanup_v1", + "recoverable_owned_session_adoption_v1", ]); assert.equal(Object.isFrozen(PYLON_RELEASE_EXPECTED_SDK_FEATURES), true); const smokeSource = readFileSync(join(root, "scripts", "smoke-pylon-prime-agent-release.mjs"), "utf8"); diff --git a/scripts/smoke-pylon-prime-agent-release.mjs b/scripts/smoke-pylon-prime-agent-release.mjs index 5f8506767f..22e2ee1edd 100644 --- a/scripts/smoke-pylon-prime-agent-release.mjs +++ b/scripts/smoke-pylon-prime-agent-release.mjs @@ -21,6 +21,7 @@ export const PYLON_RELEASE_EXPECTED_SDK_FEATURES = Object.freeze([ "bounded_daemon_ingress_v1", "negotiated_daemon_session_capabilities_v1", "caller_owned_session_environment_cleanup_v1", + "recoverable_owned_session_adoption_v1", ]); function parseArgs(args) {