diff --git a/.changeset/restartable-sync-engine.md b/.changeset/restartable-sync-engine.md new file mode 100644 index 00000000..f50b9ada --- /dev/null +++ b/.changeset/restartable-sync-engine.md @@ -0,0 +1,5 @@ +--- +"@cloudflare/computer": minor +--- + +`Workspace.pull()` and `Workspace.push()` are now async iterables that commit one block of changes per iteration. This allows a caller to implement a deferred post-exec sync across several invocations. See [./docs/02_sync_protocol.md](./docs/02_sync_protocol.md) for details. diff --git a/docs/02_sync_protocol.md b/docs/02_sync_protocol.md index 610bd4f6..32f2442f 100644 --- a/docs/02_sync_protocol.md +++ b/docs/02_sync_protocol.md @@ -222,6 +222,148 @@ in-process state. A regression in the post-apply cursor advancement path trips the assertion on the next push or pull rather than corrupting data silently. +## Restartable synchronization + +A large sync cannot assume it finishes in one invocation. A Durable +Object can be evicted between two steps, an alarm has a hard cutoff, +and a JavaScript generator survives neither. `Workspace.pull()` +and `Workspace.push()` therefore treat the iterator as +disposable and SQLite as the durability surface. + +One `next()` performs one complete block: plan it, transfer it, apply +it, persist the cursor, yield. No RPC stream stays open across a yield, +because the caller may never return. Recreating the iterable resumes +from durable state, so these two lines are equivalent to iterating +twice on one iterator: + +```ts +await workspace.pull()[Symbol.asyncIterator]().next(); +// Durable Object may be evicted here. +await workspace.pull()[Symbol.asyncIterator]().next(); +``` + +Durable state lives in two places. `_vfs_sync_operations` holds one row +per `(backend, direction)` carrying the fixed target, the generation +that fences superseded executions, and the block sizing profile. The +committed progress cursor stays in the watermark tables above, because +filesystem application and cursor advancement have to commit together. +The operation row records where a sync is going; the watermark records +how far it got. + +Callers pass no cursor, target, or budget. Block sizing is internal and +shrinks automatically when a block is detected as interrupted, so a +caller cannot ask for a block too large to complete. + +### The library never owns an alarm + +`Workspace` does not call `setAlarm` or `deleteAlarm`. Scheduling +belongs to the application, which knows what else its alarm is for. +A strict one-step handler gives each block a fresh CPU allowance: + +```ts +async alarm() { + // Leave a durable watchdog before entering a block. A hard reset + // cannot erase this future invocation. + await this.ctx.storage.setAlarm(Date.now() + 30_000); + + const iterator = this.workspace.pull()[Symbol.asyncIterator](); + const { value, done } = await iterator.next(); + + if (done || value.complete) { + await this.ctx.storage.deleteAlarm(); + } else { + await this.ctx.storage.setAlarm(Date.now() + 1_000); + } +} +``` + +Completion is exposed on the yielded value rather than only through a +following `done: true`, so a one-step handler can decide whether to +reschedule without paying for a second `next()`. + +An application that would rather drive several blocks per invocation +owns that wall-time budget too: + +```ts +async alarm() { + const stopAt = Date.now() + 12 * 60_000; + await this.ctx.storage.setAlarm(Date.now() + 30_000); + + for await (const progress of this.workspace.pull()) { + if (progress.complete) return; + if (Date.now() >= stopAt) { + await this.ctx.storage.setAlarm(Date.now() + 1_000); + return; + } + } +} +``` + +Breaking out of either loop stops local driving without failing the +operation. The last yielded cursor stays durable and a later iterable +resumes from it. The corollary is that the library cannot guarantee +convergence if the application never calls it again — that is the +deliberate price of leaving scheduling to the implementor. + +### Entry mode and pack mode + +Small windows ship one `ChangeEntry` at a time, with a `hasObjects` +probe and a `fetchObjects` pull for the bytes the receiver lacks. That +is cheap for a handful of changes and hopeless for tens of thousands of +them. + +Above either threshold the operation switches to a **change pack**: one +gzip stream carrying the entries interleaved with the unique objects +they reference, so a block needs no object round trips at all. + +| Threshold | Value | +| --- | --- | +| Entries in the cursor window | 20,000 | +| Estimated payload bytes | 100 MiB | + +Selection happens once per operation and is persisted on the operation +row. The mode cannot change mid-operation: a block's encoding must not +depend on when it was requested, or a replayed block would not match +the original and the receiver could not absorb it as a no-op. Since the +target is fixed at creation, the window cannot grow underneath the +estimate. + +Pack framing is length-prefixed records inside the gzip stream, closed +by a footer carrying the start cursor, block cursor, target, generation, +counts, and a digest. A truncated or tampered pack is rejected as a +protocol error before anything applies, leaving the cursor untouched — +advancing over entries that never arrived would lose them silently. + +`fetchChangePack` and `applyChangePack` are optional on `SyncRPC`. A +peer that predates them keeps working in entry mode, so the two sides +can be deployed in either order. + +### Replay and idempotency + +Because the cursor advances only after a block applies, any block +interrupted before its acknowledgment is re-applied. Replay safety is +per entry kind rather than a blanket property: + +| Entry | Replay behavior | +| --- | --- | +| `file` | `alreadyApplied` compares the manifest hash; identical content is a no-op. | +| `dir` | Compares mode; a match is a no-op. | +| `symlink` | Compares mode and target; a match is a no-op. | +| `delete` | Compares the tombstone's rev against the live inode's. A path recreated **above** the tombstone rev is newer information, so the delete is dropped. | + +The delete case is why replay cannot be assumed idempotent for free. A +tombstone describes a path as of the rev it was stamped with; replaying +it against a newer local recreation would destroy data the tombstone +never described. The recreation is itself a change the next push ships +upstream, so dropping the stale delete loses no work. + +Entries the receiver refuses — most often a container-side write under +a read-only mount — are reported in `SyncProgress.skipped` **and** +written to `_vfs_sync_skips`. The cursor advances past them, because an +entry that can never apply would otherwise stall the operation forever. +Recording them durably keeps the drop auditable rather than visible +only to whoever happened to read that one progress value. + ## Wire shape The wire is symmetric: push and fetch both move `ChangeEntry` diff --git a/docs/05_runtime_interface.md b/docs/05_runtime_interface.md index 39c02144..c6fd643a 100644 --- a/docs/05_runtime_interface.md +++ b/docs/05_runtime_interface.md @@ -64,7 +64,7 @@ interface WorkspaceRuntimeResult { } ``` -Command backends leave `value` unset. `worker-javascript` uses `value` for the module's structured return value and reports a zero-entry completed sync. A command can complete while its post-command pull fails; in that case `sync.status` is `"pending"`, and a configured `SyncRetryScheduler` can durably retry the pull without rerunning the command. +Command backends leave `value` unset. `worker-javascript` uses `value` for the module's structured return value and reports a zero-entry completed sync. A command can complete while its post-command pull fails; in that case `sync.status` is `"pending"`, and the next `pull()` resumes the operation from its durable watermark without rerunning the command. ## Backend routing @@ -98,7 +98,7 @@ push → spawn → events/result → pull A backend with `sync: "none"`, such as `worker-shell`, shares the host store and reports zero push/pull counts. A Container has its own VFS and synchronizes changes before and after command execution. Fully draining either `result()` or the event stream completes the post-command pull before the stream closes. -The pre-command push is a safety gate, not a best-effort optimization. The push and spawn use the same backend handle. If that handle fails before dispatch, the reconnect retry repeats both steps on the replacement container so the command cannot skip its push. If the push still fails, `exec()` rejects before the spawn request is sent; `pushed: 0` means a successful push found no entries, not that synchronization failed. A failed post-command pull does not change the completed command result. The pull is fenced to the runtime UUID that ran the command, so reconnecting to an empty replacement cannot report a clean zero-entry sync. It reports `sync.status: "pending"` and persists that UUID with the retry intent. `retryPendingSync()` resumes against the original runtime when possible; if that runtime was replaced, it clears the unrecoverable intent and reports `status: "lost"` so a later command can schedule sync for the live runtime. +The pre-command push is a safety gate, not a best-effort optimization. The push and spawn use the same backend handle. If that handle fails before dispatch, the reconnect retry repeats both steps on the replacement container so the command cannot skip its push. If the push still fails, `exec()` rejects before the spawn request is sent; `pushed: 0` means a successful push found no entries, not that synchronization failed. A failed post-command pull does not change the completed command result. The pull is fenced to the runtime UUID that ran the command, so reconnecting to an empty replacement cannot report a clean zero-entry sync. It reports `sync.status: "pending"` and persists that UUID with the sync operation. A later `pull()` resumes against the original runtime when possible; if that runtime was replaced, the unrecoverable operation is cleared so a later command can synchronize the live runtime. Container connection failures also get one backend-internal reconnect attempt. Sync calls are safe to repeat. `getExec`, `killExec`, and `disposeExec` are retried only when the new connection reaches the same computerd runtime; a replacement container returns `EEXEC_LOST` instead of applying an old execution id to its new process table. `shell.exec` is different: the backend retries it only when connection setup failed or a locally disposed stub proves that no request was sent. If the transport fails after computerd may have accepted the spawn, the error states that the command may have started and the backend does not replay it. A failure while reading the event stream also invalidates the connection without rerunning the command. diff --git a/packages/computer/README.md b/packages/computer/README.md index 528c9ffb..14833957 100644 --- a/packages/computer/README.md +++ b/packages/computer/README.md @@ -476,17 +476,18 @@ When assigning a workspace to a Think agent's `workspace`, pass `useThink: true` so Think's compatibility methods are added alongside `workspace.fs` and `workspace.runtime`. -### Durable pending-sync retries +### Resuming an incomplete sync A command can change backend files and then have its post-command pull -fail; the result exposes `sync: { status: "pending", ... }`. Configure a -`SyncRetryScheduler` on `Workspace` to persist one coalesced retry per -backend, then call `workspace.retryPendingSync(backend)` from your DO's -alarm. Retries use bounded exponential backoff and return `"exhausted"` -after the configured maximum. A container replacement returns `"lost"` -and clears the unrecoverable intent so new work is not blocked. The library -does not own your DO's alarm. See `SyncRetryScheduler`, `SyncRetryIntent`, -and `SyncRetryOptions` in the package exports. +fail; the result exposes `sync: { status: "pending", ... }`. Nothing +further is required to recover it. The sync operation and its watermark +hold the progress durably, so the next `pull()` resumes from where the +failed one stopped and drains the rest of the fixed target. + +A deferred exec fixes that target when the command finishes, which pins +the command's changes rather than capturing a newer target that could +have raced ahead. Attempt counts, backoff, and exhaustion are not part +of the API: a caller that wants to stop trying stops iterating. ### Observability diff --git a/packages/computer/src/deferred-sync.test.ts b/packages/computer/src/deferred-sync.test.ts new file mode 100644 index 00000000..b618ca5c --- /dev/null +++ b/packages/computer/src/deferred-sync.test.ts @@ -0,0 +1,156 @@ +import { createSyncServer } from "@cloudflare/computer-rpc/server"; +import { + Database, + initializeSchema, + readOperation, + SQLiteWorkspaceProvider, +} from "@cloudflare/dofs"; +import { SQLiteTestStorage } from "@cloudflare/dofs/testing"; +import { describe, expect, it } from "vitest"; + +import type { BackendHandle, WorkspaceBackend } from "./backend.js"; +import { Workspace } from "./workspace.js"; + +// Deferred synchronization used to need a host-supplied retry +// scheduler: the host persisted an intent, set its own alarm, and +// called back into retryPendingSync. The durable operation row makes +// that indirection unnecessary. A deferred exec captures the pull +// target, and any later caller resumes it by iterating pull(). + +function peerBackend(id: string): { + backend: WorkspaceBackend; + remote: Database; + close: () => void; +} { + const storage = new SQLiteTestStorage(); + const remote = new Database(storage); + initializeSchema(remote, () => 1000); + const sync = createSyncServer(remote); + const execId = () => `${id}-exec`; + const shell: import("@cloudflare/computer-rpc").ShellRPC = { + async exec(input) { + const eid = input.id ?? execId(); + return { + id: eid, + events: new ReadableStream({ + start(controller) { + controller.enqueue({ id: eid, seq: 1, name: "exit", code: 0 }); + controller.close(); + }, + }), + }; + }, + getExec: () => Promise.reject(new Error("not used")), + killExec: () => Promise.reject(new Error("not used")), + disposeExec: () => Promise.reject(new Error("not used")), + }; + const backend: WorkspaceBackend = { + id, + type: "fake", + async connect(): Promise { + return { rpc: { sync, shell }, close: async () => {} }; + }, + }; + return { backend, remote, close: () => storage.close() }; +} + +describe("deferred synchronization without a host scheduler", () => { + it("leaves a resumable pull operation after a deferred exec", async () => { + const peer = peerBackend("fake"); + try { + const provider = new SQLiteWorkspaceProvider(peer.remote); + provider.writeFileSync("/produced.txt", "by the command"); + + const ws = new Workspace({ + storage: new SQLiteTestStorage(), + backends: [peer.backend], + }); + await ws.ready(); + + // A deferred exec returns without draining the pull. + const handle = await ws.runtime.exec("true", { sync: "defer" }); + await handle.result(); + + // The application resumes on its own schedule. No retry + // scheduler, no host-persisted intent. + for await (const progress of ws.pull()) { + if (progress.complete) break; + } + + expect(await ws.fs.readFile("/produced.txt", "utf8")).toBe("by the command"); + } finally { + peer.close(); + } + }); + + it("does not require a retryScheduler to defer", async () => { + const peer = peerBackend("fake"); + try { + const ws = new Workspace({ + storage: new SQLiteTestStorage(), + backends: [peer.backend], + }); + await ws.ready(); + + // Previously this threw without a configured retryScheduler. + const handle = await ws.runtime.exec("true", { sync: "defer" }); + await expect(handle.result()).resolves.toBeDefined(); + } finally { + peer.close(); + } + }); + + it("reports a deferred pull as pending rather than complete", async () => { + const peer = peerBackend("fake"); + try { + const provider = new SQLiteWorkspaceProvider(peer.remote); + provider.writeFileSync("/produced.txt", "by the command"); + + const ws = new Workspace({ + storage: new SQLiteTestStorage(), + backends: [peer.backend], + }); + await ws.ready(); + + const handle = await ws.runtime.exec("true", { sync: "defer" }); + const execution = await handle.result(); + + expect(execution.sync.status).toBe("pending"); + // Nothing applied yet: that is the point of deferring. + expect(execution.sync.applied).toBe(0); + } finally { + peer.close(); + } + }); + + it("converges a deferred pull driven one block at a time", async () => { + const peer = peerBackend("fake"); + try { + const provider = new SQLiteWorkspaceProvider(peer.remote); + for (let i = 0; i < 5; i++) { + provider.writeFileSync(`/f${i}.txt`, `content-${i}`); + } + + const ws = new Workspace({ + storage: new SQLiteTestStorage(), + backends: [peer.backend], + }); + await ws.ready(); + const handle = await ws.runtime.exec("true", { sync: "defer" }); + await handle.result(); + + // One block per "alarm", each from a fresh iterable. + for (let i = 0; i < 20; i++) { + const iterator = ws.pull()[Symbol.asyncIterator](); + const { value, done } = await iterator.next(); + if (done || value.complete) break; + } + + expect(await ws.fs.readFile("/f4.txt", "utf8")).toBe("content-4"); + // Completed operations leave no row behind. + expect(readOperation(ws.db, "fake", "pull")).toBeUndefined(); + } finally { + peer.close(); + } + }); +}); diff --git a/packages/computer/src/index.ts b/packages/computer/src/index.ts index 70f2adfd..587c2053 100644 --- a/packages/computer/src/index.ts +++ b/packages/computer/src/index.ts @@ -15,7 +15,7 @@ // TestBackend stays on the main entry because it's a thin // test-only fake with no payload. -export type { SyncBatchBudget, SyncBatchResult } from "@cloudflare/computer-rpc/driver"; +export type { SyncProgress } from "@cloudflare/computer-rpc/driver"; export type { ApplyResult, DurableObjectStorageLike, @@ -44,6 +44,17 @@ export type { MountFactory, MountWriteAPI, } from "./mounts/types.js"; +export { + createSyncLogger, + DEFAULT_CPU_LIMIT_MS, + type SyncBlockTelemetry, + type SyncLogger, + type SyncLoggerOptions, + type SyncOperationTelemetry, + syncBlockLog, + syncOperationLog, + type TelemetryRecord, +} from "./observe/sync-telemetry.js"; export { noopObserver, type WorkspaceAttributes, @@ -97,13 +108,8 @@ export { withWorkspace, } from "./with-workspace.js"; export { - type SyncBatchOptions, - type SyncRetryIntent, - type SyncRetryOptions, - type SyncRetryScheduler, type ThinkWorkspaceCompatibility, Workspace, type WorkspaceGitFactory, type WorkspaceOptions, - type WorkspaceRetryPendingSyncResult, } from "./workspace.js"; diff --git a/packages/computer/src/observe-integration.test.ts b/packages/computer/src/observe-integration.test.ts index 628a889f..f9aa00c4 100644 --- a/packages/computer/src/observe-integration.test.ts +++ b/packages/computer/src/observe-integration.test.ts @@ -152,7 +152,10 @@ describe("Workspace observer — connection", () => { }); describe("Workspace observer — sync", () => { - it("emits workspace.sync.push with the entry count attribute", async () => { + // One span per committed block, not one per call: a sync is now a + // sequence of durably checkpointed blocks, and a caller that drives + // three blocks from three alarms should see three spans. + it("emits workspace.sync.push.block per committed block", async () => { const observer = makeRecorder(); const ws = new Workspace({ storage: makeStorage(), @@ -160,13 +163,15 @@ describe("Workspace observer — sync", () => { observer, }); await ws.ready(); - const pushed = await ws.push(); - const pushSpan = findSpan(observer.spans, "workspace.sync.push"); + for await (const progress of ws.push()) { + if (progress.complete) break; + } + const pushSpan = findSpan(observer.spans, "workspace.sync.push.block"); expect(pushSpan.outcome).toBe("ok"); - expect(pushSpan.attributes["workspace.sync.pushed"]).toBe(pushed); + expect(pushSpan.attributes["workspace.sync.entries"]).toBe(0); }); - it("emits workspace.sync.pull with applied and skipped counts", async () => { + it("emits workspace.sync.pull.block per committed block", async () => { const observer = makeRecorder(); const ws = new Workspace({ storage: makeStorage(), @@ -174,10 +179,12 @@ describe("Workspace observer — sync", () => { observer, }); await ws.ready(); - await ws.pull(); - const pullSpan = findSpan(observer.spans, "workspace.sync.pull"); - expect(pullSpan.attributes["workspace.sync.applied"]).toBe(0); - expect(pullSpan.attributes["workspace.sync.skipped"]).toBe(0); + for await (const progress of ws.pull()) { + if (progress.complete) break; + } + const pullSpan = findSpan(observer.spans, "workspace.sync.pull.block"); + expect(pullSpan.outcome).toBe("ok"); + expect(pullSpan.attributes["workspace.sync.entries"]).toBe(0); }); }); diff --git a/packages/computer/src/observe/sync-telemetry.test.ts b/packages/computer/src/observe/sync-telemetry.test.ts new file mode 100644 index 00000000..4c83d683 --- /dev/null +++ b/packages/computer/src/observe/sync-telemetry.test.ts @@ -0,0 +1,333 @@ +import { describe, expect, it, vi } from "vitest"; + +import { createSyncLogger, syncBlockLog } from "./sync-telemetry.js"; + +// Spans are the right shape for tracing, but Workers Logs is what the +// telemetry query API can aggregate today: structured console.log output +// becomes filterable `$workers.event.*` fields, and percentiles over a +// numeric field need the field to exist in the log record. +// +// So the block emitter's contract is: one flat, single-line JSON record +// per committed block, with every field a caller would want to filter or +// aggregate on already at the top level. Nested objects and arrays would +// force a query to reach through indexed paths, which is fragile. + +describe("syncBlockLog", () => { + it("carries a stable event name so queries can select these records", () => { + const record = syncBlockLog({ + backend: "container", + direction: "pull", + mode: "entries", + operationId: "op-1", + generation: "gen-1", + entries: 12, + bytes: 2048, + skipped: 0, + complete: false, + blockMs: 210, + cursorRev: 42, + targetRev: 99, + }); + + expect(record.event).toBe("sync.block"); + }); + + it("flattens every queryable value to the top level", () => { + const record = syncBlockLog({ + backend: "container", + direction: "pull", + mode: "pack", + operationId: "op-1", + generation: "gen-1", + entries: 2000, + bytes: 1024, + skipped: 3, + complete: true, + blockMs: 15812, + cursorRev: 42, + targetRev: 42, + }); + + // A telemetry filter addresses `$workers.event.`, so nesting + // would turn every query into an indexed path lookup. + for (const value of Object.values(record)) { + expect(typeof value).not.toBe("object"); + } + }); + + it("reports the CPU headroom a block left against its limit", () => { + const record = syncBlockLog({ + backend: "container", + direction: "pull", + mode: "pack", + operationId: "op-1", + generation: "gen-1", + entries: 2000, + bytes: 1024, + skipped: 0, + complete: false, + blockMs: 15000, + cursorRev: 1, + targetRev: 2, + cpuLimitMs: 30_000, + }); + + // The single most important production question: how close did the + // worst block come to the limit. Precomputing it means a query can + // just take min() rather than dividing across two fields. + expect(record.headroom).toBe(2); + }); + + it("omits headroom when the block took no measurable time", () => { + const record = syncBlockLog({ + backend: "container", + direction: "pull", + mode: "entries", + operationId: "op-1", + generation: "gen-1", + entries: 0, + bytes: 0, + skipped: 0, + complete: true, + blockMs: 0, + cursorRev: 1, + targetRev: 1, + }); + + expect(record.headroom).toBeUndefined(); + }); + + it("derives per-entry cost so slow blocks are comparable across sizes", () => { + const record = syncBlockLog({ + backend: "container", + direction: "pull", + mode: "entries", + operationId: "op-1", + generation: "gen-1", + entries: 100, + bytes: 0, + skipped: 0, + complete: false, + blockMs: 250, + cursorRev: 1, + targetRev: 2, + }); + + expect(record.msPerEntry).toBe(2.5); + }); + + it("marks a block that exceeded its CPU budget", () => { + const record = syncBlockLog({ + backend: "container", + direction: "pull", + mode: "pack", + operationId: "op-1", + generation: "gen-1", + entries: 4000, + bytes: 0, + skipped: 0, + complete: false, + blockMs: 31_000, + cursorRev: 1, + targetRev: 2, + cpuLimitMs: 30_000, + }); + + // A boolean is cheaper to filter on than a computed comparison, and + // this is the condition that means the design is failing. + expect(record.overBudget).toBe(true); + }); + + it("does not mark a block inside its budget", () => { + const record = syncBlockLog({ + backend: "container", + direction: "pull", + mode: "pack", + operationId: "op-1", + generation: "gen-1", + entries: 2000, + bytes: 0, + skipped: 0, + complete: false, + blockMs: 4_000, + cursorRev: 1, + targetRev: 2, + cpuLimitMs: 30_000, + }); + + expect(record.overBudget).toBe(false); + }); + + it("reports how far behind the target a block left the cursor", () => { + const record = syncBlockLog({ + backend: "container", + direction: "pull", + mode: "entries", + operationId: "op-1", + generation: "gen-1", + entries: 500, + bytes: 0, + skipped: 0, + complete: false, + blockMs: 100, + cursorRev: 40, + targetRev: 99, + }); + + // Convergence lag. A sync that never closes this gap is the failure + // an operator most wants an alert on. + expect(record.revsBehind).toBe(59); + }); +}); + +describe("createSyncLogger", () => { + it("emits one single-line JSON record per block", () => { + const lines: string[] = []; + const logger = createSyncLogger({ log: (line) => lines.push(line) }); + + logger.block({ + backend: "container", + direction: "pull", + mode: "entries", + operationId: "op-1", + generation: "gen-1", + entries: 10, + bytes: 20, + skipped: 0, + complete: true, + blockMs: 5, + cursorRev: 2, + targetRev: 2, + }); + + expect(lines).toHaveLength(1); + // Workers Logs parses a single JSON argument into queryable fields; + // multi-line output would be split across records. + expect(lines[0]).not.toContain("\n"); + expect(JSON.parse(lines[0]).event).toBe("sync.block"); + }); + + it("defaults to console.log so a Worker needs no wiring", () => { + const spy = vi.spyOn(console, "log").mockImplementation(() => {}); + try { + const logger = createSyncLogger(); + logger.block({ + backend: "container", + direction: "push", + mode: "pack", + operationId: "op-1", + generation: "gen-1", + entries: 1, + bytes: 1, + skipped: 0, + complete: true, + blockMs: 1, + cursorRev: 1, + targetRev: 1, + }); + expect(spy).toHaveBeenCalledTimes(1); + } finally { + spy.mockRestore(); + } + }); + + it("is silent when disabled so the default costs nothing", () => { + const lines: string[] = []; + const logger = createSyncLogger({ enabled: false, log: (line) => lines.push(line) }); + + logger.block({ + backend: "container", + direction: "pull", + mode: "entries", + operationId: "op-1", + generation: "gen-1", + entries: 10, + bytes: 20, + skipped: 0, + complete: true, + blockMs: 5, + cursorRev: 2, + targetRev: 2, + }); + + expect(lines).toEqual([]); + }); + + it("emits an operation summary when a sync completes", () => { + const lines: string[] = []; + const logger = createSyncLogger({ log: (line) => lines.push(line) }); + + logger.operation({ + backend: "container", + direction: "pull", + mode: "pack", + operationId: "op-1", + generation: "gen-1", + blocks: 13, + entries: 49_001, + bytes: 3_597_900, + skipped: 0, + totalMs: 122_325, + worstBlockMs: 15_812, + restarts: 12, + }); + + const record = JSON.parse(lines[0]); + expect(record.event).toBe("sync.operation"); + // The per-block records answer "did any block get close to the + // limit"; this one answers "what did the whole sync cost", which is + // the number the plan's 172-second claim is about. + expect(record.blocks).toBe(13); + expect(record.worstBlockMs).toBe(15_812); + }); + + it("derives effective throughput on the operation summary", () => { + const lines: string[] = []; + const logger = createSyncLogger({ log: (line) => lines.push(line) }); + + logger.operation({ + backend: "container", + direction: "pull", + mode: "pack", + operationId: "op-1", + generation: "gen-1", + blocks: 2, + entries: 100, + bytes: 2_000_000, + skipped: 0, + totalMs: 1_000, + worstBlockMs: 600, + restarts: 0, + }); + + // Bytes per second over the whole operation. This is the field that + // finally answers the open question behind the pack thresholds: + // what bandwidth does the real DO-to-container hop achieve. + expect(JSON.parse(lines[0]).bytesPerSecond).toBe(2_000_000); + }); + + it("survives a logger that throws so telemetry cannot break a sync", () => { + const logger = createSyncLogger({ + log: () => { + throw new Error("log sink exploded"); + }, + }); + + expect(() => + logger.block({ + backend: "container", + direction: "pull", + mode: "entries", + operationId: "op-1", + generation: "gen-1", + entries: 1, + bytes: 1, + skipped: 0, + complete: true, + blockMs: 1, + cursorRev: 1, + targetRev: 1, + }), + ).not.toThrow(); + }); +}); diff --git a/packages/computer/src/observe/sync-telemetry.ts b/packages/computer/src/observe/sync-telemetry.ts new file mode 100644 index 00000000..5d31cd82 --- /dev/null +++ b/packages/computer/src/observe/sync-telemetry.ts @@ -0,0 +1,216 @@ +// Structured sync telemetry for Workers Logs. +// +// The observer hook in `../observe.ts` emits spans, which is the right +// shape for tracing and nesting. It is not a shape the Workers +// Observability query API can aggregate. +// +// Measured against the live query API, not assumed. The `traces` view +// does exist and does return real data: it lists traces, filters them, +// and reports trace-level rollups (traceId, services, span count, +// traceDurationMs). What it will not do is compute. A query carrying +// `calculations` returns an empty trace list, with or without +// `groupBys` — so `min(headroom) by mode`, the question this telemetry +// exists to answer, is not expressible there. Trace-level fields also +// do not surface the per-block attributes the span hook sets via +// setAttribute. +// +// The `events` view, backed by Workers Logs, does aggregate: a single +// JSON argument to `console.log` becomes filterable `$workers.event.*` +// fields, and calculations over them return populated time series. +// Hence a flat log record alongside the span hook rather than instead +// of it. Spans remain useful as an independent cross-check, since +// trace duration is recorded by the runtime rather than self-reported. +// +// So this module exists alongside the span hook rather than replacing +// it. Spans answer "what happened inside this request"; these records +// answer "what did every sync block in production cost", which is the +// question the benchmark harnesses can only answer for a dev container. +// +// Field design follows two constraints of that query API: +// +// * Every value is a scalar at the top level. A filter addresses +// `$workers.event.`, so a nested object turns every query into +// an indexed-path lookup that breaks when shapes change. +// * Derived values are precomputed. The API aggregates a single +// numeric field per calculation, so a ratio a query would otherwise +// have to compute across two fields is computed here instead. +// +// Example queries once a session has run, using the telemetry tool: +// +// Worst block CPU, by mode: +// filters: $workers.event.event = "sync.block" +// calculations: max($workers.event.blockMs) +// groupBys: $workers.event.mode +// +// Any block that came close to the CPU limit: +// filters: $workers.event.event = "sync.block" +// $workers.event.headroom < 3 +// +// Effective transport throughput, which decides whether packs win: +// filters: $workers.event.event = "sync.operation" +// calculations: p50($workers.event.bytesPerSecond) +// p95($workers.event.bytesPerSecond) +// groupBys: $workers.event.mode + +/** Default Durable Object active-CPU allowance, in milliseconds. */ +export const DEFAULT_CPU_LIMIT_MS = 30_000; + +export interface SyncBlockTelemetry { + readonly backend: string; + readonly direction: "pull" | "push"; + readonly mode: "entries" | "pack"; + readonly operationId: string; + readonly generation: string; + readonly entries: number; + readonly bytes: number; + readonly skipped: number; + readonly complete: boolean; + /** Wall time for this block, which for a sync block tracks CPU closely. */ + readonly blockMs: number; + readonly cursorRev: number; + readonly targetRev: number; + readonly cpuLimitMs?: number; +} + +export interface SyncOperationTelemetry { + readonly backend: string; + readonly direction: "pull" | "push"; + readonly mode: "entries" | "pack"; + readonly operationId: string; + readonly generation: string; + readonly blocks: number; + readonly entries: number; + readonly bytes: number; + readonly skipped: number; + readonly totalMs: number; + readonly worstBlockMs: number; + /** How many blocks ran from a recreated iterable rather than a reused one. */ + readonly restarts: number; +} + +export type TelemetryRecord = Record; + +function round(value: number, places: number): number { + const factor = 10 ** places; + return Math.round(value * factor) / factor; +} + +/** + * Build the flat record for one committed block. + * + * `headroom` is the ratio of the CPU allowance to what this block + * actually spent, so `min(headroom)` over a session is the direct + * answer to whether block sizing is safe in production. It is omitted + * for a zero-duration block, where the ratio is meaningless rather than + * infinite. + */ +export function syncBlockLog(input: SyncBlockTelemetry): TelemetryRecord { + const cpuLimitMs = input.cpuLimitMs ?? DEFAULT_CPU_LIMIT_MS; + const record: TelemetryRecord = { + event: "sync.block", + backend: input.backend, + direction: input.direction, + mode: input.mode, + operationId: input.operationId, + generation: input.generation, + entries: input.entries, + bytes: input.bytes, + skipped: input.skipped, + complete: input.complete, + blockMs: round(input.blockMs, 1), + cursorRev: input.cursorRev, + targetRev: input.targetRev, + // Convergence lag. A sync whose blocks never close this gap is the + // failure most worth alerting on. + revsBehind: Math.max(0, input.targetRev - input.cursorRev), + cpuLimitMs, + overBudget: input.blockMs > cpuLimitMs, + }; + if (input.blockMs > 0) { + record.headroom = round(cpuLimitMs / input.blockMs, 2); + } + if (input.entries > 0) { + record.msPerEntry = round(input.blockMs / input.entries, 3); + } + if (input.bytes > 0 && input.blockMs > 0) { + record.bytesPerSecond = Math.round(input.bytes / (input.blockMs / 1000)); + } + return record; +} + +/** + * Build the flat record for a completed operation. + * + * `bytesPerSecond` here is the field that closes the open question + * behind the pack thresholds. The benchmarks could only price the + * bytes/CPU trade against hypothetical link speeds; this measures what + * the real hop achieves. + */ +export function syncOperationLog(input: SyncOperationTelemetry): TelemetryRecord { + const record: TelemetryRecord = { + event: "sync.operation", + backend: input.backend, + direction: input.direction, + mode: input.mode, + operationId: input.operationId, + generation: input.generation, + blocks: input.blocks, + entries: input.entries, + bytes: input.bytes, + skipped: input.skipped, + totalMs: round(input.totalMs, 1), + worstBlockMs: round(input.worstBlockMs, 1), + restarts: input.restarts, + }; + if (input.totalMs > 0) { + record.bytesPerSecond = Math.round(input.bytes / (input.totalMs / 1000)); + record.entriesPerSecond = round(input.entries / (input.totalMs / 1000), 1); + } + if (input.entries > 0) { + record.msPerEntry = round(input.totalMs / input.entries, 3); + } + if (input.blocks > 0) { + record.msPerBlock = round(input.totalMs / input.blocks, 1); + } + return record; +} + +export interface SyncLoggerOptions { + /** + * Set false to silence emission. Present so a caller can keep the + * wiring in place and turn the volume off, rather than branching at + * every call site. + */ + readonly enabled?: boolean; + /** Sink for one serialized record. Defaults to console.log. */ + readonly log?: (line: string) => void; +} + +export interface SyncLogger { + block(input: SyncBlockTelemetry): void; + operation(input: SyncOperationTelemetry): void; +} + +/** + * Emit sync telemetry as single-line JSON. + * + * A throwing sink is swallowed. Telemetry exists to observe sync, and + * an observability failure must not be able to fail the sync it is + * watching. + */ +export function createSyncLogger(options: SyncLoggerOptions = {}): SyncLogger { + const enabled = options.enabled ?? true; + const sink = options.log ?? ((line: string) => console.log(line)); + const emit = (record: TelemetryRecord): void => { + if (!enabled) return; + try { + sink(JSON.stringify(record)); + } catch { + // Deliberately ignored; see the note above. + } + }; + return { + block: (input) => emit(syncBlockLog(input)), + operation: (input) => emit(syncOperationLog(input)), + }; +} diff --git a/packages/computer/src/retry.test.ts b/packages/computer/src/retry.test.ts deleted file mode 100644 index 81043c5c..00000000 --- a/packages/computer/src/retry.test.ts +++ /dev/null @@ -1,615 +0,0 @@ -import type { ChangeEntry } from "@cloudflare/dofs"; -import { SQLiteTestStorage } from "@cloudflare/dofs/testing"; -import { describe, expect, it } from "vitest"; - -import type { BackendHandle, WorkspaceBackend } from "./backend.js"; -import { WorkspaceTransportError } from "./transport-failure.js"; -import type { - SyncRetryIntent, - SyncRetryScheduler, - WorkspaceRetryPendingSyncResult, -} from "./workspace.js"; -import { Workspace } from "./workspace.js"; - -class MemoryRetryScheduler implements SyncRetryScheduler { - readonly intents = new Map(); - readonly scheduled: SyncRetryIntent[] = []; - readonly cleared: string[] = []; - - async get(backend: string): Promise { - return this.intents.get(backend); - } - - async schedule(intent: SyncRetryIntent): Promise { - this.intents.set(intent.backend, intent); - this.scheduled.push(intent); - } - - async clear(backend: string): Promise { - this.intents.delete(backend); - this.cleared.push(backend); - } -} - -function retryBackend(options: { - onExec(): void; - fetchChanges: import("@cloudflare/computer-rpc").SyncRPC["fetchChanges"]; - watermarks?: import("@cloudflare/computer-rpc").SyncRPC["watermarks"]; - close?: () => Promise; -}): WorkspaceBackend { - const sync: import("@cloudflare/computer-rpc").SyncRPC = { - async push(input) { - return { rev: 0, appliedPushCursor: { rev: input.senderRev, path: null } }; - }, - fetchChanges: options.fetchChanges, - async readEntry() { - return null; - }, - async hasObjects(hashes) { - return hashes; - }, - fetchObjects() { - return new ReadableStream({ start: (controller) => controller.close() }); - }, - watermarks: - options.watermarks ?? - (async () => ({ - currentRev: 0, - pushRev: 0, - fetchCursor: { rev: 0, path: null }, - })), - async pushObjects() {}, - }; - return { - id: "sandbox", - type: "fake", - async connect(): Promise { - return { - rpc: { - sync, - shell: { - async exec() { - options.onExec(); - return { - id: "command-1", - events: new ReadableStream({ - start(controller) { - controller.enqueue({ id: "command-1", seq: 1, name: "exit", code: 0 }); - controller.close(); - }, - }), - }; - }, - async getExec() { - throw new Error("not used"); - }, - async killExec() {}, - async disposeExec() {}, - }, - }, - close: options.close ?? (async () => {}), - }; - }, - }; -} - -async function runCommand(ws: Workspace): Promise { - const handle = await ws.runtime.exec("build", { encoding: "utf8" }); - const result = await handle.result(); - expect(result.sync.status).toBe("pending"); - return undefined; -} - -describe("Workspace durable pending-sync retries", () => { - it("clears a lost runtime intent so the live runtime can schedule a new one", async () => { - const scheduler = new MemoryRetryScheduler(); - let connects = 0; - let replacementPulls = 0; - let failLivePull = false; - const backend: WorkspaceBackend = { - id: "sandbox", - type: "fake", - async connect(): Promise { - const connection = connects++; - const runtimeId = connection === 0 ? "runtime-a" : "runtime-b"; - const sync = retryBackend({ - onExec() {}, - async fetchChanges() { - if (connection === 0) { - throw new WorkspaceTransportError("container exited during post-exec pull"); - } - if (failLivePull) throw new Error("live runtime pull failed"); - replacementPulls++; - return { - currentCursor: { rev: 0, path: null }, - appliedPushCursor: { rev: 0, path: null }, - stream: new ReadableStream({ - start(controller) { - controller.close(); - }, - }), - }; - }, - }); - const handle = await sync.connect({} as never); - return { ...handle, runtimeId }; - }, - }; - const ws = new Workspace({ - storage: new SQLiteTestStorage(), - backends: [backend], - retryScheduler: scheduler, - retry: { initialDelayMs: 100, maxDelayMs: 1_000, maxAttempts: 3 }, - now: () => 5_000, - }); - - const handle = await ws.runtime.exec("build", { encoding: "utf8" }); - const result = await handle.result(); - - expect(result.sync).toMatchObject({ - status: "pending", - error: expect.stringContaining("container exited during post-exec pull"), - }); - expect(replacementPulls).toBe(0); - expect(scheduler.intents.get("sandbox")).toEqual({ - backend: "sandbox", - runtimeId: "runtime-a", - attempt: 1, - notBefore: 5_100, - }); - - await expect(ws.retryPendingSync("sandbox")).resolves.toMatchObject({ - status: "lost", - runtimeId: "runtime-a", - }); - expect(replacementPulls).toBe(0); - expect(scheduler.intents.has("sandbox")).toBe(false); - expect(scheduler.cleared).toEqual(["sandbox"]); - - failLivePull = true; - await runCommand(ws); - expect(scheduler.intents.get("sandbox")).toEqual({ - backend: "sandbox", - runtimeId: "runtime-b", - attempt: 1, - notBefore: 5_100, - }); - }); - - it("replaces a stale runtime intent when a live runtime pull becomes pending", async () => { - const scheduler = new MemoryRetryScheduler(); - scheduler.intents.set("sandbox", { - backend: "sandbox", - runtimeId: "runtime-a", - attempt: 3, - notBefore: 0, - }); - const base = retryBackend({ - onExec() {}, - async fetchChanges() { - throw new Error("runtime-b pull failed"); - }, - }); - const backend: WorkspaceBackend = { - ...base, - async connect(host): Promise { - return { ...(await base.connect(host)), runtimeId: "runtime-b" }; - }, - }; - const ws = new Workspace({ - storage: new SQLiteTestStorage(), - backends: [backend], - retryScheduler: scheduler, - retry: { initialDelayMs: 100, maxDelayMs: 1_000, maxAttempts: 3 }, - now: () => 5_000, - }); - - await runCommand(ws); - - expect(scheduler.intents.get("sandbox")).toEqual({ - backend: "sandbox", - runtimeId: "runtime-b", - attempt: 1, - notBefore: 5_100, - }); - }); - - it("schedules the exact durable retry intent after a post-command pull failure", async () => { - const scheduler = new MemoryRetryScheduler(); - let execs = 0; - const backend = retryBackend({ - onExec: () => execs++, - async fetchChanges() { - throw new Error("backend unavailable"); - }, - }); - const ws = new Workspace({ - storage: new SQLiteTestStorage(), - backends: [backend], - retryScheduler: scheduler, - retry: { initialDelayMs: 2_000, maxDelayMs: 30_000, maxAttempts: 4 }, - now: () => 10_000, - }); - - await runCommand(ws); - - expect(execs).toBe(1); - expect(scheduler.scheduled).toEqual([{ backend: "sandbox", attempt: 1, notBefore: 12_000 }]); - }); - - it("resumes a partial batch from the persisted cursor and converges without rerunning the command", async () => { - const scheduler = new MemoryRetryScheduler(); - const after: Array<{ rev: number; path: string | null } | undefined> = []; - let fetches = 0; - let execs = 0; - const entries = Array.from( - { length: 257 }, - (_, index): ChangeEntry => ({ - kind: "delete", - rev: 1, - path: `/generated/${index.toString().padStart(3, "0")}`, - mtime: 1, - }), - ); - const backend = retryBackend({ - onExec: () => execs++, - async fetchChanges(input) { - after.push(input.after); - fetches++; - const remaining = entries.filter((entry) => { - if (!input.after || input.after.rev < entry.rev) return true; - return ( - input.after.rev === entry.rev && - input.after.path !== null && - entry.path > input.after.path - ); - }); - return { - currentCursor: { rev: 1, path: null }, - appliedPushCursor: { rev: 0, path: null }, - stream: - fetches === 1 - ? new ReadableStream({ - pull(controller) { - const entry = remaining.shift(); - if (entry !== undefined) { - controller.enqueue(entry); - return; - } - controller.error(new Error("lost after first batch")); - }, - }) - : new ReadableStream({ - start(controller) { - for (const entry of remaining) controller.enqueue(entry); - controller.close(); - }, - }), - }; - }, - }); - const ws = new Workspace({ - storage: new SQLiteTestStorage(), - backends: [backend], - retryScheduler: scheduler, - retry: { initialDelayMs: 100, maxDelayMs: 1_000, maxAttempts: 3 }, - now: () => 5_000, - }); - - await runCommand(ws); - const retried = await ws.retryPendingSync("sandbox"); - - expect(retried).toMatchObject({ status: "complete", applied: 1 }); - expect(execs).toBe(1); - expect(after).toEqual([ - { rev: 0, path: null }, - { rev: 1, path: "/generated/255" }, - ]); - expect(scheduler.intents.size).toBe(0); - expect(scheduler.cleared).toEqual(["sandbox"]); - }); - - it("does not exhaust retries while bounded batches are making progress", async () => { - const scheduler = new MemoryRetryScheduler(); - const entries = Array.from( - { length: 6 }, - (_, index): ChangeEntry => ({ - kind: "delete", - rev: 1, - path: `/generated/${index}`, - mtime: 1, - }), - ); - const backend = retryBackend({ - onExec() {}, - async fetchChanges(input) { - const remaining = entries.filter((entry) => { - if (!input.after || input.after.rev < entry.rev) return true; - return input.after.path !== null && entry.path > input.after.path; - }); - return { - currentCursor: { rev: 1, path: null }, - appliedPushCursor: { rev: 0, path: null }, - stream: new ReadableStream({ - start(controller) { - for (const entry of remaining) controller.enqueue(entry); - controller.close(); - }, - }), - }; - }, - }); - const ws = new Workspace({ - storage: new SQLiteTestStorage(), - backends: [backend], - retryScheduler: scheduler, - retry: { initialDelayMs: 100, maxDelayMs: 1_000, maxAttempts: 3 }, - now: () => 5_000, - }); - scheduler.intents.set("sandbox", { - backend: "sandbox", - targetCursor: { rev: 1, path: null }, - attempt: 1, - notBefore: 0, - }); - - const outcomes: WorkspaceRetryPendingSyncResult[] = []; - for (let i = 0; i < 7; i++) { - outcomes.push(await ws.retryPendingSync("sandbox", { maxEntries: 1, maxBytes: 1024 })); - } - - expect(outcomes.slice(0, -1).every((result) => result.status === "pending")).toBe(true); - expect(outcomes.at(-1)).toMatchObject({ status: "complete" }); - expect(outcomes.some((result) => result.status === "exhausted")).toBe(false); - expect(scheduler.intents.size).toBe(0); - }); - - it("coalesces repeated command failures into one pending intent per backend", async () => { - const scheduler = new MemoryRetryScheduler(); - const backend = retryBackend({ - onExec() {}, - async fetchChanges() { - throw new Error("still unavailable"); - }, - }); - const ws = new Workspace({ - storage: new SQLiteTestStorage(), - backends: [backend], - retryScheduler: scheduler, - now: () => 1_000, - }); - - await Promise.all([runCommand(ws), runCommand(ws)]); - - expect(scheduler.scheduled).toHaveLength(1); - expect(scheduler.intents.size).toBe(1); - }); - - it("reschedules with bounded exponential backoff and leaves exhaustion visible", async () => { - const scheduler = new MemoryRetryScheduler(); - const backend = retryBackend({ - onExec() {}, - async fetchChanges() { - throw new Error("still unavailable"); - }, - }); - const ws = new Workspace({ - storage: new SQLiteTestStorage(), - backends: [backend], - retryScheduler: scheduler, - retry: { initialDelayMs: 100, maxDelayMs: 150, maxAttempts: 3 }, - now: () => 1_000, - }); - - await runCommand(ws); - expect(await ws.retryPendingSync()).toMatchObject({ status: "pending", attempt: 2 }); - expect(await ws.retryPendingSync()).toMatchObject({ status: "pending", attempt: 3 }); - expect(await ws.retryPendingSync()).toMatchObject({ status: "exhausted", attempt: 3 }); - - expect(scheduler.scheduled).toEqual([ - { backend: "sandbox", attempt: 1, notBefore: 1_100 }, - { backend: "sandbox", attempt: 2, notBefore: 1_150 }, - { backend: "sandbox", attempt: 3, notBefore: 1_150 }, - ]); - expect(scheduler.intents.get("sandbox")).toEqual({ - backend: "sandbox", - attempt: 3, - notBefore: 1_150, - }); - }); - - it("disposes a failed retry envelope and closes its RPC handle", async () => { - const scheduler = new MemoryRetryScheduler(); - let closes = 0; - let disposals = 0; - const backend = retryBackend({ - onExec() {}, - async fetchChanges() { - return { - currentCursor: { rev: 1, path: null }, - appliedPushCursor: { rev: 0, path: null }, - stream: new ReadableStream({ - start(controller) { - controller.error(new Error("cancel retry stream")); - }, - }), - [Symbol.dispose]() { - disposals++; - }, - }; - }, - close: async () => { - closes++; - }, - }); - const ws = new Workspace({ - storage: new SQLiteTestStorage(), - backends: [backend], - retryScheduler: scheduler, - }); - scheduler.intents.set("sandbox", { backend: "sandbox", attempt: 1, notBefore: 0 }); - - await expect(ws.retryPendingSync()).resolves.toMatchObject({ status: "pending" }); - await ws.close(); - - expect(disposals).toBe(1); - expect(closes).toBe(1); - }); -}); - -describe("Workspace deferred synchronization", () => { - it("schedules before returning a deferred result", async () => { - const scheduler = new MemoryRetryScheduler(); - const backend = retryBackend({ - onExec() {}, - async fetchChanges() { - return { - currentCursor: { rev: 0, path: null }, - appliedPushCursor: { rev: 0, path: null }, - stream: new ReadableStream({ - start(controller) { - controller.close(); - }, - }), - }; - }, - }); - const ws = new Workspace({ - storage: new SQLiteTestStorage(), - backends: [backend], - retryScheduler: scheduler, - now: () => 5_000, - }); - - const handle = await ws.runtime.exec("build", { - encoding: "utf8", - sync: "defer", - }); - const result = await handle.result(); - - expect(result.sync).toMatchObject({ - status: "pending", - backend: "sandbox", - targetCursor: { rev: 0, path: null }, - }); - expect(scheduler.intents.get("sandbox")).toMatchObject({ - backend: "sandbox", - targetCursor: { rev: 0, path: null }, - }); - }); - - it("settles the remote filesystem before capturing the deferred target", async () => { - const scheduler = new MemoryRetryScheduler(); - const settleInputs: unknown[] = []; - const backend = retryBackend({ - onExec() {}, - async fetchChanges() { - throw new Error("not used"); - }, - async watermarks(input) { - settleInputs.push(input); - return { - currentRev: input?.settle === true ? 5 : 0, - pushRev: 0, - fetchCursor: { rev: 0, path: null }, - }; - }, - }); - const ws = new Workspace({ - storage: new SQLiteTestStorage(), - backends: [backend], - retryScheduler: scheduler, - }); - - const handle = await ws.runtime.exec("build", { sync: "defer" }); - const result = await handle.result(); - - expect(settleInputs.at(-1)).toEqual({ settle: true }); - expect(result.sync).toMatchObject({ targetCursor: { rev: 5, path: null } }); - expect(scheduler.intents.get("sandbox")).toMatchObject({ - targetCursor: { rev: 5, path: null }, - }); - }); - - it("persists an unfenced intent when target capture fails", async () => { - const scheduler = new MemoryRetryScheduler(); - const backend = retryBackend({ - onExec() {}, - async fetchChanges() { - throw new Error("not used"); - }, - async watermarks(input) { - if (input?.settle === true) throw new Error("settle failed"); - return { currentRev: 0, pushRev: 0, fetchCursor: { rev: 0, path: null } }; - }, - }); - const ws = new Workspace({ - storage: new SQLiteTestStorage(), - backends: [backend], - retryScheduler: scheduler, - }); - - const handle = await ws.runtime.exec("build", { sync: "defer" }); - const result = await handle.result(); - - expect(result.sync).toMatchObject({ - status: "pending", - error: expect.stringContaining("settle failed"), - }); - expect(scheduler.intents.get("sandbox")).toEqual( - expect.objectContaining({ backend: "sandbox", attempt: 1 }), - ); - expect(scheduler.intents.get("sandbox")).not.toHaveProperty("targetCursor"); - }); - - it("widens an existing intent when another deferred command finishes", async () => { - const scheduler = new MemoryRetryScheduler(); - let currentRev = 0; - const backend = retryBackend({ - onExec() { - currentRev++; - }, - async fetchChanges() { - throw new Error("not used"); - }, - async watermarks() { - return { - currentRev, - pushRev: 0, - fetchCursor: { rev: 0, path: null }, - }; - }, - }); - const ws = new Workspace({ - storage: new SQLiteTestStorage(), - backends: [backend], - retryScheduler: scheduler, - }); - - const first = await ws.runtime.exec("first", { sync: "defer" }); - await first.result(); - const second = await ws.runtime.exec("second", { sync: "defer" }); - const result = await second.result(); - - expect(result.sync).toMatchObject({ targetCursor: { rev: 2, path: null } }); - expect(scheduler.intents.get("sandbox")).toMatchObject({ - targetCursor: { rev: 2, path: null }, - }); - }); -}); - -it("rejects deferred execution without a retry scheduler", async () => { - let execs = 0; - const backend = retryBackend({ - onExec: () => { - execs += 1; - }, - async fetchChanges() { - throw new Error("not expected"); - }, - }); - const ws = new Workspace({ storage: new SQLiteTestStorage(), backends: [backend] }); - - await expect(ws.runtime.exec("build", { sync: "defer" })).rejects.toThrow("retryScheduler"); - expect(execs).toBe(0); -}); diff --git a/packages/computer/src/shell.ts b/packages/computer/src/shell.ts index 6d0fd3ad..ce23b6ff 100644 --- a/packages/computer/src/shell.ts +++ b/packages/computer/src/shell.ts @@ -110,7 +110,6 @@ export interface Sync { runtimeId?: string; targetCursor?: ChangeCursor; }>; - assertDeferredReady?(): void | Promise; } type ShellExecInput = Parameters[0]; @@ -155,7 +154,6 @@ export class CommandExecutor { // with stale or incomplete workspace contents is not safe. async exec(source: string, options: ExecOptions = {}): Promise { assertNotTemplate(source); - if (options.sync === "defer") await this.#sync.assertDeferredReady?.(); const input: ShellExecInput = { source, id: options.id, diff --git a/packages/computer/src/workspace-sync.test.ts b/packages/computer/src/workspace-sync.test.ts new file mode 100644 index 00000000..dd983071 --- /dev/null +++ b/packages/computer/src/workspace-sync.test.ts @@ -0,0 +1,313 @@ +import { createSyncServer } from "@cloudflare/computer-rpc/server"; +import { Database, initializeSchema, SQLiteWorkspaceProvider } from "@cloudflare/dofs"; +import { SQLiteTestStorage } from "@cloudflare/dofs/testing"; +import { describe, expect, it, vi } from "vitest"; + +import type { BackendHandle, WorkspaceBackend } from "./backend.js"; +import { createSyncLogger } from "./observe/sync-telemetry.js"; +import { Workspace } from "./workspace.js"; + +// The public surface the plan asks for: two restartable iterables that +// take no cursors and no budgets. These tests drive them the way an +// application would — from something that may not come back — so the +// interesting cases are "consume one block and stop" and "recreate the +// iterable and finish". + +// A backend backed by a real in-process peer, so sync actually moves +// bytes rather than resolving against an empty fake. +function peerBackend(id: string): { + backend: WorkspaceBackend; + remote: Database; + close: () => void; +} { + const storage = new SQLiteTestStorage(); + const remote = new Database(storage); + initializeSchema(remote, () => 1000); + const sync = createSyncServer(remote); + const notWired = () => Promise.reject(new Error("shell not wired in this test")); + const backend: WorkspaceBackend = { + id, + type: "fake", + async connect(): Promise { + return { + rpc: { + sync, + shell: { + exec: notWired, + getExec: notWired, + killExec: notWired, + disposeExec: notWired, + }, + }, + close: async () => {}, + }; + }, + }; + return { backend, remote, close: () => storage.close() }; +} + +function seed(db: Database, count: number): void { + const provider = new SQLiteWorkspaceProvider(db); + for (let i = 0; i < count; i++) { + provider.writeFileSync(`/f${String(i).padStart(3, "0")}.txt`, `content-${i}`); + } +} + +describe("Workspace.pull", () => { + it("takes no cursor and no budget from the caller", async () => { + const peer = peerBackend("fake"); + try { + seed(peer.remote, 3); + const ws = new Workspace({ storage: new SQLiteTestStorage(), backends: [peer.backend] }); + await ws.ready(); + + // The only argument is which backend to talk to. + const seen = []; + for await (const progress of ws.pull()) seen.push(progress); + + expect(seen[seen.length - 1].complete).toBe(true); + expect(await ws.fs.readFile("/f000.txt", "utf8")).toBe("content-0"); + } finally { + peer.close(); + } + }); + + it("resumes from durable state when the iterable is recreated", async () => { + const peer = peerBackend("fake"); + try { + seed(peer.remote, 4); + const ws = new Workspace({ storage: new SQLiteTestStorage(), backends: [peer.backend] }); + await ws.ready(); + + // One block, then abandon the iterator the way an evicted + // Durable Object would. + const iterator = ws.pull()[Symbol.asyncIterator](); + const first = await iterator.next(); + expect(first.done).toBe(false); + + // A brand new iterable finishes the operation. + for await (const progress of ws.pull()) { + if (progress.complete) break; + } + + expect(await ws.fs.readFile("/f003.txt", "utf8")).toBe("content-3"); + } finally { + peer.close(); + } + }); + + it("reports the backend it synced against", async () => { + const peer = peerBackend("container"); + try { + seed(peer.remote, 1); + const ws = new Workspace({ storage: new SQLiteTestStorage(), backends: [peer.backend] }); + await ws.ready(); + + const seen = []; + for await (const progress of ws.pull("container")) seen.push(progress); + + expect(seen[0].backend).toBe("container"); + expect(seen[0].direction).toBe("pull"); + } finally { + peer.close(); + } + }); + + it("completes immediately when there is nothing to pull", async () => { + const peer = peerBackend("fake"); + try { + const ws = new Workspace({ storage: new SQLiteTestStorage(), backends: [peer.backend] }); + await ws.ready(); + + const seen = []; + for await (const progress of ws.pull()) seen.push(progress); + + expect(seen).toHaveLength(1); + expect(seen[0].complete).toBe(true); + expect(seen[0].entries).toBe(0); + } finally { + peer.close(); + } + }); +}); + +describe("Workspace.push", () => { + it("ships local writes to the backend", async () => { + const peer = peerBackend("fake"); + try { + const ws = new Workspace({ storage: new SQLiteTestStorage(), backends: [peer.backend] }); + await ws.ready(); + await ws.fs.writeFile("/local.txt", "from the host"); + + const seen = []; + for await (const progress of ws.push()) seen.push(progress); + + expect(seen[seen.length - 1].complete).toBe(true); + expect(seen[seen.length - 1].direction).toBe("push"); + const names = peer.remote + .all<{ name: string }>("SELECT name FROM vfs_dirents WHERE parent_inode = 1") + .map((r) => r.name); + expect(names).toContain("local.txt"); + } finally { + peer.close(); + } + }); + + it("resumes a push from durable state", async () => { + const peer = peerBackend("fake"); + try { + const ws = new Workspace({ storage: new SQLiteTestStorage(), backends: [peer.backend] }); + await ws.ready(); + for (let i = 0; i < 4; i++) { + await ws.fs.writeFile(`/f${i}.txt`, `content-${i}`); + } + + const iterator = ws.push()[Symbol.asyncIterator](); + await iterator.next(); + + for await (const progress of ws.push()) { + if (progress.complete) break; + } + + const names = peer.remote + .all<{ name: string }>("SELECT name FROM vfs_dirents WHERE parent_inode = 1") + .map((r) => r.name); + expect(names).toHaveLength(4); + } finally { + peer.close(); + } + }); +}); + +describe("sync telemetry", () => { + it("emits a queryable record per block and a summary per operation", async () => { + const peer = peerBackend("container"); + try { + seed(peer.remote, 3); + const lines: string[] = []; + const ws = new Workspace({ + storage: new SQLiteTestStorage(), + backends: [peer.backend], + syncTelemetry: createSyncLogger({ log: (line) => lines.push(line) }), + }); + await ws.ready(); + + for await (const progress of ws.pull()) { + if (progress.complete) break; + } + + const records = lines.map((line) => JSON.parse(line)); + const blocks = records.filter((r) => r.event === "sync.block"); + const operations = records.filter((r) => r.event === "sync.operation"); + + expect(blocks.length).toBeGreaterThan(0); + expect(operations).toHaveLength(1); + // The fields a production query needs to answer the CPU-headroom + // question must be present and numeric. + expect(typeof blocks[0].blockMs).toBe("number"); + expect(typeof blocks[0].headroom === "number" || blocks[0].headroom === undefined).toBe(true); + expect(blocks[0].backend).toBe("container"); + expect(blocks[0].direction).toBe("pull"); + expect(operations[0].entries).toBe(3); + } finally { + peer.close(); + } + }); + + it("is silent by default so a library consumer is not billed uninvited", async () => { + const peer = peerBackend("container"); + const spy = vi.spyOn(console, "log").mockImplementation(() => {}); + try { + seed(peer.remote, 2); + const ws = new Workspace({ + storage: new SQLiteTestStorage(), + backends: [peer.backend], + }); + await ws.ready(); + + for await (const progress of ws.pull()) { + if (progress.complete) break; + } + + const emitted = spy.mock.calls.filter((call) => String(call[0]).includes('"event":"sync.')); + expect(emitted).toEqual([]); + } finally { + spy.mockRestore(); + peer.close(); + } + }); + + it("emits to console.log when opted in without a custom sink", async () => { + const peer = peerBackend("container"); + const spy = vi.spyOn(console, "log").mockImplementation(() => {}); + try { + seed(peer.remote, 2); + const ws = new Workspace({ + storage: new SQLiteTestStorage(), + backends: [peer.backend], + syncTelemetryEnabled: true, + }); + await ws.ready(); + + for await (const progress of ws.pull()) { + if (progress.complete) break; + } + + const emitted = spy.mock.calls.filter((call) => String(call[0]).includes('"event":"sync.')); + expect(emitted.length).toBeGreaterThan(0); + } finally { + spy.mockRestore(); + peer.close(); + } + }); + + it("is silent when telemetry is disabled", async () => { + const peer = peerBackend("container"); + try { + seed(peer.remote, 2); + const lines: string[] = []; + const ws = new Workspace({ + storage: new SQLiteTestStorage(), + backends: [peer.backend], + syncTelemetry: createSyncLogger({ enabled: false, log: (line) => lines.push(line) }), + }); + await ws.ready(); + + for await (const progress of ws.pull()) { + if (progress.complete) break; + } + + expect(lines).toEqual([]); + } finally { + peer.close(); + } + }); +}); + +describe("sync block iterables and module backends", () => { + it("completes without work for a backend that has no sync wire", async () => { + const notWired = () => Promise.reject(new Error("exec not used in this test")); + const module: import("./runtime/types.js").WorkspaceModuleBackend = { + protocol: "module", + id: "module", + type: "module", + async connect() { + return { + exec: notWired, + getExec: notWired, + killExec: notWired, + disposeExec: notWired, + }; + }, + }; + const ws = new Workspace({ storage: new SQLiteTestStorage(), backends: [module] }); + await ws.ready(); + + const seen = []; + for await (const progress of ws.pull()) seen.push(progress); + + expect(seen).toHaveLength(1); + expect(seen[0].complete).toBe(true); + expect(seen[0].entries).toBe(0); + }); +}); diff --git a/packages/computer/src/workspace.test.ts b/packages/computer/src/workspace.test.ts index 9bccce16..de896050 100644 --- a/packages/computer/src/workspace.test.ts +++ b/packages/computer/src/workspace.test.ts @@ -9,6 +9,23 @@ import type { WorkspaceModuleBackend } from "./runtime/types.js"; import { WorkspaceTransportError } from "./transport-failure.js"; import { type ThinkWorkspaceCompatibility, Workspace } from "./workspace.js"; +// Drive a sync iterable to completion. push() and pull() are now +// restartable block iterables, but these tests care about the folded +// outcome — total entries shipped, or entries applied and skipped — so +// each one drains the iterable and sums the progress values. +async function drainSync( + iterable: AsyncIterable<{ entries: number; skipped: number; complete: boolean }>, +): Promise<{ entries: number; skipped: number }> { + let entries = 0; + let skipped = 0; + for await (const progress of iterable) { + entries += progress.entries; + skipped += progress.skipped; + if (progress.complete) break; + } + return { entries, skipped }; +} + function makeStorage(): SQLiteTestStorage { return new SQLiteTestStorage(); } @@ -895,9 +912,9 @@ describe("Workspace backend selection", () => { const ws = new Workspace({ storage: makeStorage() }); await ws.ready(); await ws.fs.writeFile("/a.txt", "hi"); - expect(await ws.push()).toBe(0); - const pulled = await ws.pull(); - expect(pulled).toEqual({ applied: 0, skipped: [] }); + expect((await drainSync(ws.push())).entries).toBe(0); + const pulled = await drainSync(ws.pull()); + expect(pulled).toEqual({ entries: 0, skipped: 0 }); }); it("module-only push and pull are no-op synchronization", async () => { @@ -910,8 +927,8 @@ describe("Workspace backend selection", () => { }, }; const ws = new Workspace({ storage: makeStorage(), backends: [backend] }); - expect(await ws.push()).toBe(0); - expect(await ws.pull()).toEqual({ applied: 0, skipped: [] }); + expect((await drainSync(ws.push())).entries).toBe(0); + expect(await drainSync(ws.pull())).toEqual({ entries: 0, skipped: 0 }); }); it("stub() works and fs methods round-trip through it", async () => { @@ -1014,7 +1031,7 @@ describe("Workspace backend selection", () => { await new Promise((r) => setTimeout(r, 0)); // Should rebuild silently and resolve to a push count, not throw. - const pushed = await ws.push(); + const pushed = (await drainSync(ws.push())).entries; expect(pushed).toBeGreaterThanOrEqual(0); expect(connectCount).toBe(2); }); @@ -1058,6 +1075,8 @@ describe("Workspace backend selection", () => { const sync: import("@cloudflare/computer-rpc").SyncRPC = { push: () => tripwire("push"), fetchChanges: () => tripwire("fetchChanges"), + fetchChangePack: () => tripwire("fetchChangePack"), + applyChangePack: () => tripwire("applyChangePack"), readEntry: () => tripwire("readEntry"), hasObjects: () => tripwire("hasObjects"), fetchObjects: () => { @@ -1089,11 +1108,11 @@ describe("Workspace backend selection", () => { expect(touched).toEqual([]); await ws.fs.writeFile("/local.txt", "hello"); - const pushed = await ws.push(); - const pulled = await ws.pull(); + const pushed = (await drainSync(ws.push())).entries; + const pulled = await drainSync(ws.pull()); expect(pushed).toBe(0); - expect(pulled.applied).toBe(0); - expect(pulled.skipped).toEqual([]); + expect(pulled.entries).toBe(0); + expect(pulled.skipped).toBe(0); expect(touched).toEqual([]); }); }); @@ -1155,30 +1174,36 @@ describe("Workspace.fs against the local store", () => { }); }); -describe("Workspace.pull return shape", () => { - it("resolves to the dofs ApplyResult shape", async () => { - // The fake SyncRPC's fetchChanges returns an empty stream, so - // applied is 0 and skipped is []. The point of the test isn't - // counts but the shape: pull() now returns the structured - // result so callers can read skipped[] without an extra - // round trip. +describe("Workspace.pull progress shape", () => { + it("yields one complete progress value when there is nothing to apply", async () => { + // The fake SyncRPC has no changes to offer, so the operation + // completes in one empty block. The point is the contract: a + // terminating iterable whose last value is marked complete, with + // the cursors the caller needs to decide whether to reschedule. const ws = new Workspace({ storage: makeStorage(), backends: [makeBackend("fake")] }); await ws.ready(); - const result = await ws.pull(); - expect(result).toEqual({ applied: 0, skipped: [] }); + + const seen = []; + for await (const progress of ws.pull()) seen.push(progress); + + expect(seen).toHaveLength(1); + expect(seen[0]).toMatchObject({ + backend: "fake", + direction: "pull", + entries: 0, + skipped: 0, + complete: true, + }); + expect(seen[0].cursor).toBeDefined(); + expect(seen[0].targetCursor).toBeDefined(); }); - it("uses batch options on pull without changing the default overload", async () => { + it("takes no cursor or budget from the caller", async () => { const ws = new Workspace({ storage: makeStorage(), backends: [makeBackend("fake")] }); await ws.ready(); - const result = await ws.pull("fake", { - mode: "batch", - maxEntries: 1, - maxBytes: 1024, - }); - expect(result.status).toBe("complete"); - expect(result.entries).toBe(0); - expect(result.targetCursor).toEqual({ rev: 0, path: null }); + // The only argument is the backend id. + expect(ws.pull.length).toBeLessThanOrEqual(1); + expect(ws.push.length).toBeLessThanOrEqual(1); }); }); @@ -1229,8 +1254,8 @@ describe("Workspace mutation serialization", () => { // Fire two concurrent push() calls. Without the FIFO, both // enter pushOnce simultaneously and peakInFlight.push hits 2. - const a = ws.push(); - const b = ws.push(); + const a = drainSync(ws.push()); + const b = drainSync(ws.push()); // Let the event loop settle so any concurrent entries register. await new Promise((r) => setTimeout(r, 20)); expect(peakInFlight.push).toBe(1); @@ -1268,7 +1293,7 @@ describe("Workspace mutation serialization", () => { const ws = new Workspace({ storage: makeStorage(), backends: [makeBackend("fake", rpc)] }); await ws.ready(); await ws.fs.writeFile("/a.txt", "hello"); - const push = ws.push(); + const push = drainSync(ws.push()); // Wait a beat so push reaches the gated remote.push call. await new Promise((r) => setTimeout(r, 20)); // Read while push is still in flight; must resolve fast. @@ -1310,7 +1335,7 @@ describe("Workspace transport-failure invalidation", () => { const ws = new Workspace({ storage: makeStorage(), backends: [backend] }); await ws.fs.writeFile("/a.txt", "hi"); - await expect(ws.push()).resolves.toBeGreaterThan(0); + await expect(drainSync(ws.push()).then((r) => r.entries)).resolves.toBeGreaterThan(0); expect(connects).toBe(2); expect(closes).toBe(1); await expect(replacement.readEntry("/a.txt")).resolves.toMatchObject({ @@ -1351,7 +1376,7 @@ describe("Workspace transport-failure invalidation", () => { const ws = new Workspace({ storage: makeStorage(), backends: [backend] }); await ws.fs.writeFile("/a.txt", "hi"); - const push = ws.push(); + const push = drainSync(ws.push()); await closeBegan; const ready = ws.ready("only"); await Promise.resolve(); @@ -1367,6 +1392,11 @@ describe("Workspace transport-failure invalidation", () => { let closes = 0; const stale: import("@cloudflare/computer-rpc").SyncRPC = { ...fakeRpc(), + // A pull opens with a settled watermarks() read to fix its + // target, so that is the call a dead session fails first. + async watermarks() { + throw new WorkspaceTransportError("RPC session was shut down"); + }, async fetchChanges() { throw new WorkspaceTransportError("RPC session was shut down"); }, @@ -1387,7 +1417,7 @@ describe("Workspace transport-failure invalidation", () => { }; const ws = new Workspace({ storage: makeStorage(), backends: [backend] }); - await expect(ws.pull()).resolves.toEqual({ applied: 0, skipped: [] }); + await expect(drainSync(ws.pull())).resolves.toEqual({ entries: 0, skipped: 0 }); expect(connects).toBe(2); expect(closes).toBe(1); }); @@ -1410,7 +1440,7 @@ describe("Workspace transport-failure invalidation", () => { const ws = new Workspace({ storage: makeStorage(), backends: [backend] }); await ws.fs.writeFile("/ready.txt", "ready"); - await expect(ws.push()).resolves.toBeGreaterThan(0); + await expect(drainSync(ws.push()).then((r) => r.entries)).resolves.toBeGreaterThan(0); expect(connects).toBe(2); }); @@ -1438,7 +1468,7 @@ describe("Workspace transport-failure invalidation", () => { }; const ws = new Workspace({ storage: makeStorage(), backends: [backend] }); - await expect(ws.pull()).resolves.toEqual({ applied: 0, skipped: [] }); + await expect(drainSync(ws.pull())).resolves.toEqual({ entries: 0, skipped: 0 }); expect(connects).toBe(2); expect(closes).toBe(1); }); @@ -1458,7 +1488,7 @@ describe("Workspace transport-failure invalidation", () => { }; const ws = new Workspace({ storage: makeStorage(), backends: [backend] }); - const error = await ws.pull().catch((caught: unknown) => caught); + const error = await drainSync(ws.pull()).catch((caught: unknown) => caught); expect(error).toBeInstanceOf(WorkspaceTransportError); expect(error).toMatchObject({ cause: errors[1] }); expect(String(error)).toMatch(/pull failed after 1 reconnect retry/); @@ -1491,9 +1521,9 @@ describe("Workspace transport-failure invalidation", () => { await ws.ready("only"); expect(connects).toBe(1); await ws.fs.writeFile("/a.txt", "hi"); - await expect(ws.push()).rejects.toThrow(/EROFS/); + await expect(drainSync(ws.push())).rejects.toThrow(/EROFS/); // Cache survives a non-transport error. - await ws.push().catch(() => undefined); + await drainSync(ws.push()).catch(() => undefined); expect(connects).toBe(1); }); diff --git a/packages/computer/src/workspace.ts b/packages/computer/src/workspace.ts index 818942e4..25f797a2 100644 --- a/packages/computer/src/workspace.ts +++ b/packages/computer/src/workspace.ts @@ -11,18 +11,17 @@ import type { ShellRPC } from "@cloudflare/computer-rpc"; import { - pullBatch, + captureSyncTarget, + pullBlocks, pullOnce, - pushBatch, + pushBlocks, pushOnce, reconcileWatermarks, - type SyncBatchBudget, - type SyncBatchResult, + type SyncProgress, } from "@cloudflare/computer-rpc/driver"; import { type ApplyResult, type ChangeCursor, - compareChangeCursors, Database, type DurableObjectStorageLike, initializeSchema, @@ -43,6 +42,7 @@ import type { GitClient, GitClientFactory, GitIdentity } from "./git/index.js"; import { MountIndex } from "./mounts/index.js"; import { buildMountRegistry, type MountValue } from "./mounts/registry.js"; import type { Mount } from "./mounts/types.js"; +import { createSyncLogger, type SyncLogger } from "./observe/sync-telemetry.js"; import { noopObserver, safeErrorMessage, type WorkspaceObserver, withSpan } from "./observe.js"; import { WorkspaceRuntime } from "./runtime/runtime.js"; import { @@ -61,69 +61,6 @@ import { WorkspaceTransportError, } from "./transport-failure.js"; -export interface SyncRetryIntent { - backend: string; - targetCursor?: ChangeCursor; - // Container process whose post-command changes are pending. Durable - // retries must not report success against an empty replacement. - runtimeId?: string; - attempt: number; - notBefore: number; -} - -/** - * Durable storage boundary for pending post-command pulls. - * - * The host owns persistence and wake-up because the workspace library - * cannot own a Durable Object alarm. Each backend has at most one intent. - */ -export interface SyncRetryScheduler { - get(backend: string): Promise; - schedule(intent: SyncRetryIntent): Promise; - clear(backend: string): Promise; -} - -export interface SyncBatchOptions extends SyncBatchBudget { - mode: "batch"; - targetCursor?: ChangeCursor; -} - -export interface SyncRetryOptions { - initialDelayMs?: number; - maxDelayMs?: number; - maxAttempts?: number; -} - -export type WorkspaceRetryPendingSyncResult = - | { status: "idle"; backend: string } - | { status: "complete"; backend: string; applied: number; skipped: ApplyResult["skipped"] } - | { - status: "pending"; - backend: string; - runtimeId?: string; - attempt: number; - notBefore: number; - cursor?: ChangeCursor; - targetCursor?: ChangeCursor; - error?: string; - } - | { - status: "exhausted"; - backend: string; - runtimeId?: string; - attempt: number; - error: string; - } - | { status: "lost"; backend: string; runtimeId: string; error: string }; - -const DEFAULT_RETRY_INITIAL_DELAY_MS = 1_000; -const DEFAULT_RETRY_MAX_DELAY_MS = 60_000; -const DEFAULT_RETRY_MAX_ATTEMPTS = 5; -const DEFAULT_SYNC_BATCH_BUDGET: SyncBatchBudget = { - maxEntries: 64, - maxBytes: 4 * 1024 * 1024, -}; - // When a backend RPC fails with a transport error, how much replay // the operation tolerates. "always" suits idempotent calls; a // "pre-dispatch" operation is replayed only when the failure proves @@ -167,11 +104,18 @@ export interface WorkspaceOptions { // adapter subpaths for the Cloudflare runtime and OpenTelemetry. observer?: WorkspaceObserver; - // Optional durable retry boundary for failed post-command pulls. - // The host persists one intent per backend and wakes the Durable - // Object at intent.notBefore to call retryPendingSync(backend). - retryScheduler?: SyncRetryScheduler; - retry?: SyncRetryOptions; + // Structured per-block sync telemetry, emitted as single-line JSON so + // the Workers Observability query API can aggregate it into the CPU + // headroom and transport throughput numbers local benchmarks can only + // model. See observe/sync-telemetry.ts for the queries. + // + // Off by default. Workers Logs is billed per event and a large sync + // emits one record per block, so writing to a consumer's log stream + // uninvited would be both surprising and metered. Set + // syncTelemetryEnabled to opt in, or pass syncTelemetry to route the + // records somewhere else. + syncTelemetry?: SyncLogger; + syncTelemetryEnabled?: boolean; // Optional git client factory. Omit it to keep the default // Workspace graph free of isomorphic-git; pass createGitClient() @@ -279,11 +223,8 @@ export class Workspace { readonly #callableBackendIds: Set; readonly #defaultBackendId: string | undefined; readonly #observer: WorkspaceObserver; + readonly #syncLogger: SyncLogger; readonly #now: () => number; - readonly #retryScheduler: SyncRetryScheduler | undefined; - readonly #retryInitialDelayMs: number; - readonly #retryMaxDelayMs: number; - readonly #retryMaxAttempts: number; readonly #sessionId: string; readonly #gitFactory: WorkspaceGitFactory | undefined; readonly #defaultGitIdentity: GitIdentity | undefined; @@ -342,22 +283,6 @@ export class Workspace { constructor(options: WorkspaceOptions) { this.#now = options.now ?? Date.now; - this.#retryScheduler = options.retryScheduler; - this.#retryInitialDelayMs = positiveRetryOption( - options.retry?.initialDelayMs, - DEFAULT_RETRY_INITIAL_DELAY_MS, - "initialDelayMs", - ); - this.#retryMaxDelayMs = positiveRetryOption( - options.retry?.maxDelayMs, - DEFAULT_RETRY_MAX_DELAY_MS, - "maxDelayMs", - ); - this.#retryMaxAttempts = positiveRetryOption( - options.retry?.maxAttempts, - DEFAULT_RETRY_MAX_ATTEMPTS, - "maxAttempts", - ); this.#sessionId = options.sessionId ?? ""; this.#gitFactory = options.git; this.#defaultGitIdentity = options.defaultGitIdentity; @@ -396,6 +321,8 @@ export class Workspace { } this.#defaultBackendId = registered[0]?.id; this.#observer = options.observer ?? noopObserver; + this.#syncLogger = + options.syncTelemetry ?? createSyncLogger({ enabled: options.syncTelemetryEnabled ?? false }); this.#mounts = buildMountRegistry(options.mounts, { sessionId: options.sessionId, vfs: () => this.provider(), @@ -596,239 +523,166 @@ export class Workspace { // Sync the local store with a configured backend. // - // push() ships everything the host has written since the last - // push to that backend; pull() applies everything the backend - // has produced since the last pull. Both are explicit — the - // package doesn't run a background loop. CommandExecutor.exec - // brackets each call automatically against the backend it - // selects; reach for push() / pull() directly only when an - // FS-only flow needs the bracket without an exec. + // push() ships everything the host has written since the last push to + // that backend; pull() applies everything the backend has produced + // since the last pull. Both are restartable async iterables: one + // next() commits one durably checkpointed block, and recreating the + // iterable resumes from the Workspace's durable cursor. // - // `id` selects which backend to push to / pull from. Omitting - // it picks the default (the first backend in the list). + // The caller supplies no cursor, no target, and no budget. Block + // sizing is internal so a caller cannot ask for a block too large to + // finish inside a Durable Object's CPU allowance. // - // push() returns the number of entries shipped to the backend. - // pull() returns the dofs ApplyResult { applied, skipped } — - // `applied` is the number of entries written into the local - // store, `skipped` surfaces remote-side writes the apply path - // rejected because they targeted a read-only mount root. + // `id` selects which backend to sync with. Omitting it picks the + // default (the first backend in the list). // - // Both methods emit a `workspace.sync.push` / `workspace.sync.pull` - // span on the configured observer, tagged with the resolved - // backend id and the entry count. - push(id?: string): Promise; - push(options: SyncBatchOptions): Promise; - push(id: string | undefined, options: SyncBatchOptions): Promise; - push( - idOrOptions?: string | SyncBatchOptions, - options?: SyncBatchOptions, - ): Promise { - const id = - typeof idOrOptions === "string" || idOrOptions === undefined ? idOrOptions : undefined; - const batch = typeof idOrOptions === "object" ? idOrOptions : options; - return this.#serialize(id, (resolvedId) => - withSpan( - this.#observer, - "workspace.sync.push", - { "workspace.sync.backend": resolvedId }, - async () => { - if (batch !== undefined) { - if (resolvedId === undefined || this.#moduleBackendsById.has(resolvedId)) { - return emptyBatchResult(batch.targetCursor); - } - return this.#runWithReconnect(resolvedId, "pushBatch", async (handle) => { - if (handle.sync === "none") return emptyBatchResult(batch.targetCursor); - return pushBatch(this.#db, handle.rpc.sync, { - backend: resolvedId, - targetCursor: batch.targetCursor, - budget: batch, - }); - }); - } - if (resolvedId === undefined || this.#moduleBackendsById.has(resolvedId)) return 0; - return this.#runWithReconnect(resolvedId, "push", async (handle) => { - if (handle.sync === "none") return 0; - return pushOnce(this.#db, handle.rpc.sync, resolvedId); - }); - }, - (span, outcome) => { - if (!outcome.ok) return; - span.setAttribute( - "workspace.sync.pushed", - typeof outcome.value === "number" ? outcome.value : outcome.value.entries, - ); - }, - ), - ); + // Workspace never sets or deletes an alarm. Drive these from a + // request, an alarm, a queue, or a Workflow; see + // docs/02_sync_protocol.md for the one-step and multi-block handler + // shapes. + // + // Both emit a `workspace.sync.push.block` / + // `workspace.sync.pull.block` span per committed block, tagged with + // the resolved backend id and the entry count. + pull(id?: string): AsyncIterable { + return this.#syncBlocks(id, "pull"); } - pull(id?: string): Promise; - pull(options: SyncBatchOptions): Promise; - pull(id: string | undefined, options: SyncBatchOptions): Promise; - pull( - idOrOptions?: string | SyncBatchOptions, - options?: SyncBatchOptions, - ): Promise { - const id = - typeof idOrOptions === "string" || idOrOptions === undefined ? idOrOptions : undefined; - const batch = typeof idOrOptions === "object" ? idOrOptions : options; - if (batch === undefined) - return this.#serialize(id, (resolvedId) => this.#pullResolved(resolvedId)); - return this.#serialize(id, (resolvedId) => - withSpan( - this.#observer, - "workspace.sync.pull", - { "workspace.sync.backend": resolvedId }, - async () => { - if (resolvedId === undefined || this.#moduleBackendsById.has(resolvedId)) { - return emptyBatchResult(batch.targetCursor); - } - return this.#runWithReconnect(resolvedId, "pullBatch", async (handle) => { - if (handle.sync === "none") return emptyBatchResult(batch.targetCursor); - return pullBatch(this.#db, handle.rpc.sync, { - backend: resolvedId, - targetCursor: batch.targetCursor, - budget: batch, - }); - }); - }, - (span, outcome) => { - if (!outcome.ok) return; - span.setAttribute( - "workspace.sync.applied", - typeof outcome.value === "object" ? outcome.value.applied : outcome.value, - ); - }, - ), - ); + push(id?: string): AsyncIterable { + return this.#syncBlocks(id, "push"); } - /** - * Run a host-scheduled pending pull from its persisted cursor. - * - * The call shares the backend's mutation FIFO with push, pull, and - * command brackets. A successful pull clears the host's durable - * intent. A failed pull advances bounded exponential backoff; the - * last failed attempt remains stored and is reported as exhausted. - */ - retryPendingSync( - id?: string, - budget: SyncBatchBudget = DEFAULT_SYNC_BATCH_BUDGET, - ): Promise { - return this.#serialize(id, async (resolvedId) => { - if (resolvedId === undefined) { - throw new Error("Workspace has no backend configured for pending sync retry"); - } - const scheduler = this.#retryScheduler; - if (scheduler === undefined) { - throw new Error("Workspace has no retryScheduler configured"); - } - const intent = await scheduler.get(resolvedId); - if (intent === undefined) return { status: "idle", backend: resolvedId }; - if (intent.attempt > this.#retryMaxAttempts) { - return { - status: "exhausted", - backend: resolvedId, - ...(intent.runtimeId === undefined ? {} : { runtimeId: intent.runtimeId }), - attempt: intent.attempt, - error: "pending sync retry attempts exhausted", - }; - } - try { - const result = await this.#pullBatchResolved( - resolvedId, - intent.runtimeId, - budget, - intent.targetCursor, - ); - if (result.status === "pending") { - // Hitting a batch budget is successful progress, not a failed - // retry. Reset the consecutive-failure count so large trees - // can drain through any number of bounded alarm turns. - const next = this.#retryIntent(resolvedId, 1, intent.runtimeId, result.targetCursor); - await scheduler.schedule(next); - return { - status: "pending", - ...next, - cursor: result.cursor, - targetCursor: result.targetCursor, - }; - } - await scheduler.clear(resolvedId); + // Shared plumbing for both iterables. + // + // Each `next()` is wrapped in the backend's mutation FIFO rather than + // the whole iteration, because holding the FIFO across a yield would + // block every other mutation for as long as the caller took to come + // back — and the caller may never come back. Serializing per block + // keeps the "one mutating sync block per backend and direction" + // guarantee without making an abandoned iterator wedge the workspace. + #syncBlocks(id: string | undefined, direction: "pull" | "push"): AsyncIterable { + const span = direction === "pull" ? "workspace.sync.pull.block" : "workspace.sync.push.block"; + return { + [Symbol.asyncIterator]: (): AsyncIterator => { + let finished = false; + // Per-iterable rollup for the operation summary. Blocks driven + // from separate iterables (the eviction case) each report their + // own partial rollup, which is why the block records carry the + // operation id: a query can regroup them. + let blocks = 0; + let entries = 0; + let bytes = 0; + let skipped = 0; + let worstBlockMs = 0; + const operationStarted = Date.now(); return { - status: "complete", - backend: resolvedId, - applied: result.applied, - skipped: result.skipped, - }; - } catch (error) { - const message = safeErrorMessage(error); - if ( - intent.runtimeId !== undefined && - (error as { code?: unknown } | null)?.code === "EEXEC_LOST" - ) { - await scheduler.clear(resolvedId); - return { - status: "lost", - backend: resolvedId, - runtimeId: intent.runtimeId, - error: message, - }; - } - if (intent.attempt >= this.#retryMaxAttempts) { - return { - status: "exhausted", - backend: resolvedId, - ...(intent.runtimeId === undefined ? {} : { runtimeId: intent.runtimeId }), - attempt: intent.attempt, - error: message, - }; - } - const next = this.#retryIntent( - resolvedId, - intent.attempt + 1, - intent.runtimeId, - intent.targetCursor, - ); - await scheduler.schedule(next); - return { status: "pending", ...next, error: message }; - } - }); - } + next: async (): Promise> => { + if (finished) return { done: true, value: undefined }; + const blockStarted = Date.now(); + const result = await this.#serialize(id, (resolvedId) => + withSpan( + this.#observer, + span, + { "workspace.sync.backend": resolvedId }, + async (): Promise> => { + // A module backend has no sync wire. Report one + // complete, empty block so a caller's loop + // terminates instead of spinning. + if (resolvedId === undefined || this.#moduleBackendsById.has(resolvedId)) { + return { + done: false, + value: emptySyncProgress(resolvedId ?? "none", direction), + }; + } + return this.#runWithReconnect(resolvedId, direction, async (handle) => { + if (handle.sync === "none") { + return { + done: false, + value: emptySyncProgress(resolvedId, direction), + }; + } + // A fresh iterator per attempt, deliberately. + // Caching one across calls would pin the RPC stub + // it was built with, so a reconnect would retry + // against the dead handle. The engine resumes + // from durable state, so building a new iterator + // costs nothing and is what makes the reconnect + // land on the replacement stub. + const iterable = + direction === "pull" + ? pullBlocks(this.#db, handle.rpc.sync, { backend: resolvedId }) + : pushBlocks(this.#db, handle.rpc.sync, { backend: resolvedId }); + return iterable[Symbol.asyncIterator]().next(); + }); + }, + (spanRef, outcome) => { + if (!outcome.ok || outcome.value.done) return; + spanRef.setAttribute("workspace.sync.entries", outcome.value.value.entries); + }, + ), + ); + if (result.done) return result; + + // Structured telemetry alongside the span. The span nests + // for tracing; this record is what the Workers Observability + // query API can aggregate, so production runs can answer the + // CPU-headroom and throughput questions the local benchmarks + // could only model. + const blockMs = Date.now() - blockStarted; + const progress = result.value; + blocks += 1; + entries += progress.entries; + bytes += progress.bytes; + skipped += progress.skipped; + worstBlockMs = Math.max(worstBlockMs, blockMs); + this.#syncLogger.block({ + backend: progress.backend, + direction: progress.direction, + mode: progress.mode, + operationId: progress.operationId, + generation: progress.generation, + entries: progress.entries, + bytes: progress.bytes, + skipped: progress.skipped, + complete: progress.complete, + blockMs, + cursorRev: progress.cursor.rev, + targetRev: progress.targetCursor.rev, + }); - #pullBatchResolved( - resolvedId: string | undefined, - expectedRuntimeId: string | undefined, - budget: SyncBatchBudget, - targetCursor?: ChangeCursor, - ): Promise { - return withSpan( - this.#observer, - "workspace.sync.pull.batch", - { "workspace.sync.backend": resolvedId }, - async () => { - if (resolvedId === undefined || this.#moduleBackendsById.has(resolvedId)) { - return emptyBatchResult(targetCursor); - } - return this.#runWithReconnect(resolvedId, "pullBatch", async (handle) => { - if (expectedRuntimeId !== undefined) { - assertExecutionRuntime("post-command sync", expectedRuntimeId, handle.runtimeId); - } - if (handle.sync === "none") return emptyBatchResult(targetCursor); - return pullBatch(this.#db, handle.rpc.sync, { - backend: resolvedId, - targetCursor, - budget, - }); - }); - }, - (span, outcome) => { - if (!outcome.ok) return; - span.setAttribute("workspace.sync.entries", outcome.value.entries); - span.setAttribute("workspace.sync.bytes", outcome.value.bytes); - span.setAttribute("workspace.sync.applied", outcome.value.applied); + if (progress.complete) { + finished = true; + this.#syncLogger.operation({ + backend: progress.backend, + direction: progress.direction, + mode: progress.mode, + operationId: progress.operationId, + generation: progress.generation, + blocks, + entries, + bytes, + skipped, + totalMs: Date.now() - operationStarted, + worstBlockMs, + // Every block after the first in one iterable reused + // the iterator; a caller that recreates the iterable + // per block reports blocks = 1 each time, so restarts + // is derivable across records by operation id. + restarts: 0, + }); + } + return result; + }, + // Breaking out stops local driving. The operation stays + // pending and its last cursor stays durable, so a later + // iterable resumes it. + return: async (): Promise> => { + finished = true; + return { done: true, value: undefined }; + }, + }; }, - ); + }; } #pullResolved(resolvedId: string | undefined, expectedRuntimeId?: string): Promise { @@ -856,91 +710,38 @@ export class Workspace { ); } - async #schedulePendingSync( + // Capture a durable pull operation for a command whose changes were + // not drained in-band. + // + // This used to persist a host-owned retry intent and rely on the host + // to set an alarm and call back. The operation row makes that + // indirection unnecessary: capturing the target here is enough, and + // any later pull() iteration resumes it from durable state. + async #capturePendingSync( id: string, runtimeId?: string, - captureTarget = false, ): Promise<{ backend?: string; runtimeId?: string; targetCursor?: ChangeCursor }> { - const scheduler = this.#retryScheduler; - if (scheduler === undefined) { - if (captureTarget) { - throw new Error("Workspace requires a retryScheduler for deferred synchronization"); - } - return {}; - } return this.#serialize(id, async (resolvedId) => { - if (resolvedId === undefined) return {}; - const existing = await scheduler.get(resolvedId); - const sameRuntime = - existing !== undefined && (runtimeId === undefined || existing.runtimeId === runtimeId); - if (!captureTarget && sameRuntime) { - return { - backend: resolvedId, - ...(existing.runtimeId === undefined ? {} : { runtimeId: existing.runtimeId }), - ...(existing.targetCursor === undefined ? {} : { targetCursor: existing.targetCursor }), - }; + if (resolvedId === undefined || this.#moduleBackendsById.has(resolvedId)) return {}; + const handle = await this.#handleFor(resolvedId); + if (runtimeId !== undefined) { + assertExecutionRuntime("post-command sync", runtimeId, handle.runtimeId); } + if (handle.sync === "none") return { backend: resolvedId }; - const intentRuntimeId = runtimeId ?? (sameRuntime ? existing.runtimeId : undefined); - let targetCursor: ChangeCursor | undefined; - let captureError: unknown; - if (captureTarget) { - try { - const handle = await this.#handleFor(resolvedId); - if (runtimeId !== undefined) { - assertExecutionRuntime("post-command sync", runtimeId, handle.runtimeId); - } - targetCursor = - handle.sync === "none" - ? { rev: 0, path: null } - : { - rev: (await handle.rpc.sync.watermarks({ settle: true })).currentRev, - path: null, - }; - if ( - sameRuntime && - existing.targetCursor !== undefined && - compareChangeCursors(existing.targetCursor, targetCursor) > 0 - ) { - targetCursor = existing.targetCursor; - } - } catch (error) { - // Preserve a durable, unfenced retry when the settle/capture - // step fails. Its first pull will capture a fresh target. - captureError = error; - targetCursor = undefined; - } - } - - await scheduler.schedule(this.#retryIntent(resolvedId, 1, intentRuntimeId, targetCursor)); - if (captureError !== undefined) throw captureError; + // Open the operation and fix its target now, so the command's + // changes are pinned even though nothing drains them yet. A + // later pull() joins this pending operation rather than + // capturing a newer target that could race ahead. + const target = await captureSyncTarget(this.#db, handle.rpc.sync, resolvedId); return { backend: resolvedId, - ...(intentRuntimeId === undefined ? {} : { runtimeId: intentRuntimeId }), - ...(targetCursor === undefined ? {} : { targetCursor }), + ...(runtimeId === undefined ? {} : { runtimeId }), + ...(target === undefined ? {} : { targetCursor: target }), }; }); } - #retryIntent( - backend: string, - attempt: number, - runtimeId?: string, - targetCursor?: ChangeCursor, - ): SyncRetryIntent { - const delay = Math.min( - this.#retryMaxDelayMs, - this.#retryInitialDelayMs * 2 ** Math.max(0, attempt - 1), - ); - return { - backend, - ...(targetCursor === undefined ? {} : { targetCursor }), - ...(runtimeId === undefined ? {} : { runtimeId }), - attempt, - notBefore: this.#now() + delay, - }; - } - // Drop and close a cached handle after a transport failure. // Matches by identity so a late error from an old operation cannot // tear down a replacement that another caller already installed. @@ -1360,17 +1161,15 @@ export class Workspace { const shell = new CommandExecutor( rpc, { - push: () => this.push(id), + push: () => this.#pushForExecDefault(id), pull: (runtimeId) => this.#pullForExec(id, runtimeId), - onPullPending: async (_error, runtimeId) => { - await this.#schedulePendingSync(id, runtimeId); - }, - onPostExecPending: (runtimeId) => this.#schedulePendingSync(id, runtimeId, true), - assertDeferredReady: () => { - if (this.#retryScheduler === undefined) { - throw new Error("Workspace requires a retryScheduler for deferred synchronization"); - } - }, + // A pull that failed in-band already left its operation + // pending, and its cursor is durable, so the next pull() + // resumes it with no bookkeeping here. Dialing a fresh handle + // just to record that would turn a failed command into a + // second connection attempt. + onPullPending: undefined, + onPostExecPending: (runtimeId) => this.#capturePendingSync(id, runtimeId), }, this.#observer, dispatch, @@ -1384,6 +1183,30 @@ export class Workspace { return this.#serialize(id, (resolvedId) => this.#pullResolved(resolvedId, runtimeId)); } + // The exec bracket's pre-command push. Distinct from the public + // push() iterable: the bracket needs the whole local window shipped + // before the command starts, and an entry count to report on the + // execution, not a block-at-a-time cursor it would have to drive. + #pushForExecDefault(id: string): Promise { + return this.#serialize(id, (resolvedId) => + withSpan( + this.#observer, + "workspace.sync.push", + { "workspace.sync.backend": resolvedId }, + async () => { + if (resolvedId === undefined || this.#moduleBackendsById.has(resolvedId)) return 0; + return this.#runWithReconnect(resolvedId, "push", async (handle) => { + if (handle.sync === "none") return 0; + return pushOnce(this.#db, handle.rpc.sync, resolvedId); + }); + }, + (span, outcome) => { + if (outcome.ok) span.setAttribute("workspace.sync.pushed", outcome.value); + }, + ), + ); + } + #pushForExec(id: string, handle: BackendHandle): Promise { return this.#serialize(id, (resolvedId) => withSpan( @@ -1468,16 +1291,24 @@ export class Workspace { } } -function emptyBatchResult(targetCursor?: ChangeCursor): SyncBatchResult { - const target = targetCursor ?? { rev: 0, path: null }; +// A backend with no sync wire (a module backend, or a handle that +// reports sync: "none") still has to produce a terminating iterable. +// One complete, empty block lets a caller's `for await` finish rather +// than spin waiting for progress that will never come. +function emptySyncProgress(backend: string, direction: "pull" | "push"): SyncProgress { + const cursor = { rev: 0, path: null }; return { - status: "complete", + operationId: "none", + generation: "none", + backend, + direction, + mode: "entries", + cursor, + targetCursor: cursor, entries: 0, bytes: 0, - applied: 0, - skipped: [], - cursor: target, - targetCursor: target, + skipped: 0, + complete: true, }; } @@ -1549,14 +1380,6 @@ function watchStreamForTransportError( }); } -function positiveRetryOption(value: number | undefined, fallback: number, name: string): number { - const resolved = value ?? fallback; - if (!Number.isSafeInteger(resolved) || resolved <= 0) { - throw new Error(`Workspace retry.${name} must be a positive integer`); - } - return resolved; -} - /** * Resolve the session id the artifacts client runs under: the * artifacts-specific one when the caller set it, otherwise the diff --git a/packages/dofs/src/bench/sync-blocks.bench.ts b/packages/dofs/src/bench/sync-blocks.bench.ts new file mode 100644 index 00000000..3eada815 --- /dev/null +++ b/packages/dofs/src/bench/sync-blocks.bench.ts @@ -0,0 +1,397 @@ +// Restartable sync benchmark. +// +// Runs under @cloudflare/vitest-pool-workers so every statement drives a +// REAL Durable Object SqlStorage. That matters more here than in most +// harnesses: the whole point of block-at-a-time sync is to stay inside a +// Durable Object's CPU allowance, and the node SQLiteTestStorage fixture +// caches prepared statements and would understate per-statement cost. +// +// What this measures: +// +// * Block planning cost as a cursor window grows. planBlock is on the +// hot path for both directions and is the step that has to stay +// sublinear-ish in the window it is *not* selecting. +// * Pack encode and decode throughput, plus the compression ratio the +// transport actually achieves on package-shaped trees. +// * Mode selection cost, which runs once per operation and must not +// scan a window it is only sizing. +// * Per-block apply cost at several block sizes, which is the number +// that decides whether a block fits in the 30-second CPU limit. +// +// Output is a set of tables plus one JSON line per group so before and +// after runs are easy to diff. Run with: +// npm run bench --workspace @cloudflare/dofs + +import { env, runInDurableObject } from "cloudflare:test"; +import { it } from "vitest"; +import type { TestBindings } from "../../tests/worker.js"; +import { SQLiteWorkspaceProvider } from "../provider.js"; +import { initializeSchema } from "../schema/index.js"; +import { Database } from "../storage.js"; +import { applyChanges } from "../sync/apply.js"; +import { planBlock, selectMode } from "../sync/blocks.js"; +import { decodeChangePack, encodeChangePack } from "../sync/change-pack.js"; +import { MIN_BLOCK_PROFILE } from "../sync/operations.js"; +import { currentRev } from "../sync/watermarks.js"; +import type { DurableObjectStorageLike } from "../types.js"; + +const NOW = (): number => 1000; +const BIG_PROFILE = { maxEntries: 1_000_000, maxBytes: Number.MAX_SAFE_INTEGER }; + +function freshStub(): DurableObjectStub { + const ns = (env as unknown as TestBindings).TestStorage; + return ns.get(ns.newUniqueId()); +} + +async function withRealDB(fn: (db: Database) => Promise | T): Promise { + return runInDurableObject(freshStub(), async (_i: unknown, state: DurableObjectState) => { + const db = new Database(state.storage as unknown as DurableObjectStorageLike); + initializeSchema(db, NOW); + return await fn(db); + }); +} + +// A package-shaped tree: many small files spread across nested +// directories, with the duplication a real node_modules exhibits (the +// same helper vendored under several packages). Shape matters — the +// planner's cost is driven by directory fan-out and the pack's ratio by +// how compressible and how duplicated the payloads are. +function seedPackageTree( + db: Database, + options: { packages: number; filesPerPackage: number; fileBytes: number }, +): { files: number; logicalBytes: number } { + const provider = new SQLiteWorkspaceProvider(db, { now: NOW }); + let files = 0; + let logicalBytes = 0; + // Payloads are mostly distinct, with a minority genuinely shared, so + // the content-addressed store has real duplicates to collapse without + // the whole tree collapsing to one object. Seeding every file with + // identical bytes would make dedup do all the work and leave the pack + // measuring nothing. + const body = (p: number, f: number): string => { + // Every 8th file is a vendored copy of a shared helper. + const shared = (p + f) % 8 === 0; + const seed = shared ? "shared-helper" : `pkg-${p}-lib-${f}`; + const filler = `${seed} module body with require() calls and comments; `; + return filler.repeat(Math.ceil(options.fileBytes / filler.length)).slice(0, options.fileBytes); + }; + for (let p = 0; p < options.packages; p++) { + const dir = `/node_modules/pkg-${String(p).padStart(4, "0")}`; + provider.mkdirSync(dir, { recursive: true }); + provider.writeFileSync( + `${dir}/package.json`, + `{"name":"pkg-${p}","version":"1.0.0","main":"index.js"}`, + ); + files++; + logicalBytes += 48; + for (let f = 0; f < options.filesPerPackage; f++) { + const content = body(p, f); + provider.writeFileSync(`${dir}/lib-${f}.js`, content); + files++; + logicalBytes += content.length; + } + } + return { files, logicalBytes }; +} + +function ms(start: number): number { + return Number((performance.now() - start).toFixed(1)); +} + +function table(rows: Record[]): void { + if (rows.length === 0) return; + const keys = Object.keys(rows[0]); + const width = keys.map((k) => Math.max(k.length, ...rows.map((r) => String(r[k] ?? "").length))); + const line = (cells: (string | number)[]): string => + cells.map((c, i) => String(c).padEnd(width[i])).join(" "); + console.log(line(keys)); + console.log(width.map((w) => "-".repeat(w)).join(" ")); + for (const row of rows) console.log(line(keys.map((k) => row[k] ?? ""))); +} + +// --------------------------------------------------------------------- +// Block planning as the window grows. +// --------------------------------------------------------------------- + +it("block planning scales with the block, not the window", async () => { + const rows: Record[] = []; + const json: Record[] = []; + + for (const packages of [10, 40, 160]) { + await withRealDB(async (db) => { + const seeded = seedPackageTree(db, { packages, filesPerPackage: 6, fileBytes: 512 }); + const target = { rev: currentRev(db), path: null }; + + // Whole window in one block: the cost floor for "ship everything". + const tFull = performance.now(); + const full = await planBlock(db, { + after: { rev: 0, path: null }, + through: target, + profile: BIG_PROFILE, + }); + const fullMs = ms(tFull); + + // One bounded block out of the same window. This is the number + // that has to stay flat as the window grows, or a big sync pays + // for the whole window on every block. + const tBlock = performance.now(); + const block = await planBlock(db, { + after: { rev: 0, path: null }, + through: target, + profile: { maxEntries: 256, maxBytes: 8 * 1024 * 1024 }, + }); + const blockMs = ms(tBlock); + + rows.push({ + packages, + "window entries": full.entries.length, + "seeded files": seeded.files, + "plan window (ms)": fullMs, + "plan 256-entry block (ms)": blockMs, + "block entries": block.entries.length, + "us/entry (window)": Number(((fullMs * 1000) / full.entries.length).toFixed(1)), + }); + json.push({ + packages, + windowEntries: full.entries.length, + fullMs, + blockMs, + blockEntries: block.entries.length, + }); + }); + } + + console.log("\n=== block planning ==="); + table(rows); + console.log(`BENCH_JSON plan ${JSON.stringify(json)}`); +}); + +// --------------------------------------------------------------------- +// Pack encode / decode throughput and compression ratio. +// --------------------------------------------------------------------- + +it("pack encode and decode throughput", async () => { + const rows: Record[] = []; + const json: Record[] = []; + + for (const shape of [ + { label: "metadata-heavy", packages: 250, filesPerPackage: 8, fileBytes: 512 }, + { label: "byte-heavy", packages: 60, filesPerPackage: 8, fileBytes: 64 * 1024 }, + ]) { + await withRealDB(async (db) => { + const seeded = seedPackageTree(db, shape); + const target = { rev: currentRev(db), path: null }; + const block = await planBlock(db, { + after: { rev: 0, path: null }, + through: target, + profile: BIG_PROFILE, + }); + + const tEncode = performance.now(); + const stream = encodeChangePack(db, { + block, + after: { rev: 0, path: null }, + target, + generation: "bench", + }); + // Drain to bytes so encode time includes gzip, not just planning. + const chunks: Uint8Array[] = []; + let packBytes = 0; + const reader = stream.getReader(); + while (true) { + const { value, done } = await reader.read(); + if (done) break; + chunks.push(value); + packBytes += value.byteLength; + } + reader.releaseLock(); + const encodeMs = ms(tEncode); + + const joined = new Uint8Array(packBytes); + let offset = 0; + for (const c of chunks) { + joined.set(c, offset); + offset += c.byteLength; + } + + const tDecode = performance.now(); + const decoded = await decodeChangePack( + new ReadableStream({ + start(controller) { + controller.enqueue(joined); + controller.close(); + }, + }), + ); + const decodeMs = ms(tDecode); + + // The pack replaces both the entry stream and the object bytes, + // so the ratio has to count both. Comparing against object bytes + // alone reads as inflation on a metadata-heavy tree, where entry + // records dominate the payload. + const entryBytes = block.entries.reduce((total, e) => total + JSON.stringify(e).length, 0); + const wireBytes = entryBytes + block.objectBytes; + const ratio = Number((wireBytes / Math.max(1, packBytes)).toFixed(2)); + rows.push({ + shape: shape.label, + entries: block.entries.length, + objects: block.objects.length, + "logical MB": Number((seeded.logicalBytes / 1024 / 1024).toFixed(2)), + "entry KB": Number((entryBytes / 1024).toFixed(1)), + "object KB": Number((block.objectBytes / 1024).toFixed(1)), + "uncompressed KB": Number((wireBytes / 1024).toFixed(1)), + "pack KB": Number((packBytes / 1024).toFixed(1)), + "gzip ratio": `${ratio}x`, + "encode (ms)": encodeMs, + "decode (ms)": decodeMs, + "encode MB/s": Number((wireBytes / 1024 / 1024 / (encodeMs / 1000)).toFixed(1)), + }); + json.push({ + shape: shape.label, + entries: block.entries.length, + objectBytes: block.objectBytes, + entryBytes, + wireBytes, + packBytes, + ratio, + encodeMs, + decodeMs, + decodedEntries: decoded.entries.length, + }); + }); + } + + console.log("\n=== pack codec ==="); + table(rows); + console.log(`BENCH_JSON pack ${JSON.stringify(json)}`); +}); + +// --------------------------------------------------------------------- +// Mode selection: runs once per operation, must not scan the window. +// --------------------------------------------------------------------- + +it("mode selection cost", async () => { + const rows: Record[] = []; + await withRealDB(async (db) => { + seedPackageTree(db, { packages: 120, filesPerPackage: 6, fileBytes: 512 }); + const target = { rev: currentRev(db), path: null }; + + // Low threshold: the probe fills almost immediately and the + // decision is made without draining the window. + const tLow = performance.now(); + const low = await selectMode(db, { + after: { rev: 0, path: null }, + through: target, + profile: MIN_BLOCK_PROFILE, + thresholdEntries: 64, + }); + const lowMs = ms(tLow); + + // Production threshold: the window is far below 20k, so the probe + // drains it and reports entry mode. + const tHigh = performance.now(); + const high = await selectMode(db, { + after: { rev: 0, path: null }, + through: target, + profile: MIN_BLOCK_PROFILE, + }); + const highMs = ms(tHigh); + + rows.push( + { + case: "threshold crossed early (64)", + mode: low.mode, + "first block entries": low.firstBlock.entries.length, + "select (ms)": lowMs, + }, + { + case: "production threshold (20000)", + mode: high.mode, + "first block entries": high.firstBlock.entries.length, + "select (ms)": highMs, + }, + ); + }); + + console.log("\n=== mode selection ==="); + table(rows); +}); + +// --------------------------------------------------------------------- +// Per-block apply cost at several block sizes. +// --------------------------------------------------------------------- + +it("per-block apply cost by block size", async () => { + const rows: Record[] = []; + const json: Record[] = []; + + // Capture one window's entries and object bytes from a source DO, + // then apply them into a fresh DO in bounded blocks. Cross-DO I/O + // isolation means the snapshot has to be plain values. + const snapshot = await withRealDB(async (db) => { + seedPackageTree(db, { packages: 60, filesPerPackage: 6, fileBytes: 1024 }); + const block = await planBlock(db, { + after: { rev: 0, path: null }, + through: { rev: currentRev(db), path: null }, + profile: BIG_PROFILE, + }); + const objects: [string, Uint8Array][] = []; + for (const object of block.objects) { + const row = db.one<{ bytes: Uint8Array }>( + "SELECT bytes FROM vfs_blob_bytes WHERE hash = ?", + object.hash, + ); + if (row !== undefined) { + let key = ""; + for (const b of object.hash) key += b.toString(16).padStart(2, "0"); + objects.push([key, row.bytes]); + } + } + return { entries: block.entries, objects, objectBytes: block.objectBytes }; + }); + + for (const blockSize of [64, 256, 1024]) { + await withRealDB(async (db) => { + const objectMap = new Map(snapshot.objects); + const started = performance.now(); + let blocks = 0; + let applied = 0; + let worstBlockMs = 0; + for (let i = 0; i < snapshot.entries.length; i += blockSize) { + const slice = snapshot.entries.slice(i, i + blockSize); + const tBlock = performance.now(); + const result = await applyChanges(db, slice, objectMap, { source: "upstream" }); + const blockMs = performance.now() - tBlock; + worstBlockMs = Math.max(worstBlockMs, blockMs); + applied += result.applied; + blocks++; + } + const totalMs = ms(started); + rows.push({ + "block size": blockSize, + blocks, + "entries applied": applied, + "total (ms)": totalMs, + "worst block (ms)": Number(worstBlockMs.toFixed(1)), + "ms/entry": Number((totalMs / Math.max(1, applied)).toFixed(3)), + "blocks per 30s CPU": Math.floor(30_000 / Math.max(1, worstBlockMs)), + }); + json.push({ + blockSize, + blocks, + applied, + totalMs, + worstBlockMs: Number(worstBlockMs.toFixed(1)), + }); + }); + } + + console.log("\n=== per-block apply ==="); + console.log( + `source window: ${snapshot.entries.length} entries, ` + + `${(snapshot.objectBytes / 1024 / 1024).toFixed(2)} MB objects, ` + + `${snapshot.objects.length} unique`, + ); + table(rows); + console.log(`BENCH_JSON apply ${JSON.stringify(json)}`); +}); diff --git a/packages/dofs/src/index.ts b/packages/dofs/src/index.ts index 468ec22a..8f1a830b 100644 --- a/packages/dofs/src/index.ts +++ b/packages/dofs/src/index.ts @@ -41,6 +41,32 @@ export type { ApplyOptions, ApplyResult, SkippedEntry } from "./sync/apply.js"; // to a Database. export { applyChanges, applyChangesSync } from "./sync/apply.js"; export { stageBlob } from "./sync/blobs.js"; +export type { + BlockObject, + ModeDecision, + PlanBlockRequest, + PlannedBlock, + SelectModeRequest, +} from "./sync/blocks.js"; +export { + PACK_THRESHOLD_BYTES, + PACK_THRESHOLD_ENTRIES, + planBlock, + selectMode, +} from "./sync/blocks.js"; +export type { + DecodeChangePackOptions, + DecodedChangePack, + EncodeChangePackInput, + PackFooter, +} from "./sync/change-pack.js"; +export { + decodeChangePack, + encodeChangePack, + PACK_FORMAT_VERSION, + PackProtocolError, + readPackFooter, +} from "./sync/change-pack.js"; export type { ChangeEntry } from "./sync/changes.js"; export { materialiseChange } from "./sync/changes.js"; export type { CoalesceOptions } from "./sync/coalesce.js"; @@ -50,6 +76,32 @@ export { DEFAULT_IGNORE, isIgnored } from "./sync/ignore.js"; export { assertAppliedPushCursor } from "./sync/invariant.js"; export type { ManifestChunk } from "./sync/manifests.js"; export { buildManifest, MANIFEST_VERSION } from "./sync/manifests.js"; +export type { + BlockProfile, + OpenedOperation, + SyncDirection, + SyncMode, + SyncOperation, + SyncOperationStatus, + SyncSkip, + TerminalStatus, +} from "./sync/operations.js"; +export { + beginCapture, + clearBlockMarker, + completeOperation, + DEFAULT_BLOCK_PROFILE, + failOperation, + fixTarget, + MIN_BLOCK_PROFILE, + markBlockStarted, + openOperation, + pruneSkips, + readOperation, + readSkips, + recordSkip, + shrinkBlockProfile, +} from "./sync/operations.js"; export { pushObjects } from "./sync/push.js"; export type { ChangeCursor, WatermarkKey } from "./sync/watermarks.js"; export { diff --git a/packages/dofs/src/schema/core.ts b/packages/dofs/src/schema/core.ts index 88b7daa5..a4f514da 100644 --- a/packages/dofs/src/schema/core.ts +++ b/packages/dofs/src/schema/core.ts @@ -11,9 +11,11 @@ // composite-PK lookups now read straight from the PK b-tree leaf // with no rowid indirection, and `child_inode` lives in the // dirents leaf so the (parent, name) resolve read is covering -// (no separate index needed). See `schema/migrations.ts` for the +// (no separate index needed). Bumped to 7 when `_vfs_sync_operations` +// and `_vfs_sync_skips` landed, carrying the durable half of a +// restartable pull or push. See `schema/migrations.ts` for the // migration list; `sync.ts` carries the fresh-install DDL. -export const SCHEMA_VERSION = 6; +export const SCHEMA_VERSION = 7; export const ROOT_INODE = 1; export const CORE_STATEMENTS = [ diff --git a/packages/dofs/src/schema/index.test.ts b/packages/dofs/src/schema/index.test.ts index f07b597e..3d1ca85e 100644 --- a/packages/dofs/src/schema/index.test.ts +++ b/packages/dofs/src/schema/index.test.ts @@ -444,6 +444,64 @@ describe("initializeSchema", () => { expect(norm(tableSql("vfs_chunks"))).toBe(norm(freshSql("vfs_chunks"))); }); + it("upgrades a v6 database with the sync operation tables", () => { + const storage = new SQLiteTestStorage(); + const db = new Database(storage); + + // A v6 database: everything the current baseline creates except + // the two tables v7 adds. Simulated by initializing at the + // current baseline, dropping the new tables, and winding the + // stamped version back — the migrator must create them rather + // than relying on the IF NOT EXISTS baseline having done it. + initializeSchema(db, () => 0); + db.run("DROP TABLE _vfs_sync_operations"); + db.run("DROP TABLE _vfs_sync_skips"); + db.run("UPDATE vfs_meta SET v = ? WHERE k = ?", 6, "schema_version"); + + // Existing sync progress must survive the upgrade untouched: the + // operation table is additive and backfills nothing. + db.run( + "INSERT INTO _vfs_watermark (k, backend, v) VALUES (?, ?, ?) " + + "ON CONFLICT(k, backend) DO UPDATE SET v = excluded.v", + "fetchRev", + "container", + 77, + ); + + initializeSchema(db, () => 0); + + expect(db.one<{ v: number }>("SELECT v FROM vfs_meta WHERE k = ?", "schema_version")?.v).toBe( + SCHEMA_VERSION, + ); + expect( + db.scalar( + "SELECT v FROM _vfs_watermark WHERE k = ? AND backend = ?", + "fetchRev", + "container", + ), + ).toBe(77); + + // An upgraded database has no in-flight operation, so the first + // pull after the upgrade captures a fresh target. + expect(db.one<{ c: number }>("SELECT COUNT(*) AS c FROM _vfs_sync_operations")?.c).toBe(0); + + // The migrated shape must match a fresh install, or the two + // install paths diverge and only one gets tested. + const tableSql = (source: Database, name: string): string => + source.one<{ sql: string }>( + "SELECT sql FROM sqlite_master WHERE type = 'table' AND name = ?", + name, + )?.sql ?? ""; + const fresh = new Database(new SQLiteTestStorage()); + initializeSchema(fresh, () => 0); + const norm = (sql: string): string => sql.replace(/\s+/g, " ").trim().toUpperCase(); + + expect(norm(tableSql(db, "_vfs_sync_operations"))).toBe( + norm(tableSql(fresh, "_vfs_sync_operations")), + ); + expect(norm(tableSql(db, "_vfs_sync_skips"))).toBe(norm(tableSql(fresh, "_vfs_sync_skips"))); + }); + it("is idempotent across repeat calls", () => { const storage = new SQLiteTestStorage(); const db = new Database(storage); diff --git a/packages/dofs/src/schema/migrations.ts b/packages/dofs/src/schema/migrations.ts index 31b6f374..3a5e14d1 100644 --- a/packages/dofs/src/schema/migrations.ts +++ b/packages/dofs/src/schema/migrations.ts @@ -160,12 +160,59 @@ function v5_to_v6_push_cursor(db: Database): void { ); } +// v6 → v7 — add the restartable sync operation table and its skip +// log. Both are new tables, so the migration is a plain create; no +// existing rows need reshaping. Fresh installs land the same DDL from +// `sync.ts`, and both paths must keep the CHECK constraints aligned. +// +// Nothing is backfilled. An upgraded database has no in-flight +// operation by definition — the old code path had nowhere to record +// one — so the first pull or push after the upgrade captures a fresh +// target from the existing watermark cursor. +function v6_to_v7_sync_operations(db: Database): void { + db.run( + `CREATE TABLE IF NOT EXISTS _vfs_sync_operations ( + backend TEXT NOT NULL, + direction TEXT NOT NULL CHECK (direction IN ('pull', 'push')), + generation TEXT NOT NULL, + status TEXT NOT NULL CHECK ( + status IN ('capturing', 'pending', 'failed', 'lost') + ), + target_rev INTEGER, + target_path TEXT, + runtime_id TEXT, + mode TEXT CHECK (mode IN ('entries', 'pack')), + block_after_rev INTEGER, + block_after_path TEXT, + block_started_at INTEGER, + internal_max_entries INTEGER NOT NULL, + internal_max_bytes INTEGER NOT NULL, + created_at INTEGER NOT NULL, + updated_at INTEGER NOT NULL, + last_error TEXT, + PRIMARY KEY (backend, direction) + )`, + ); + db.run( + `CREATE TABLE IF NOT EXISTS _vfs_sync_skips ( + backend TEXT NOT NULL, + direction TEXT NOT NULL CHECK (direction IN ('pull', 'push')), + generation TEXT NOT NULL, + path TEXT NOT NULL, + reason TEXT NOT NULL, + at INTEGER NOT NULL, + PRIMARY KEY (backend, direction, generation, path) + )`, + ); +} + export const MIGRATIONS: readonly Migration[] = [ { from: 1, to: 2, migrator: v1_to_v2_add_mounts_mode }, { from: 2, to: 3, migrator: v2_to_v3_add_size_column }, { from: 3, to: 4, migrator: v3_to_v4_watermark_backend_column }, { from: 4, to: 5, migrator: v4_to_v5_without_rowid }, { from: 5, to: 6, migrator: v5_to_v6_push_cursor }, + { from: 6, to: 7, migrator: v6_to_v7_sync_operations }, ] as const; // Apply every migration whose `from` matches the current version, diff --git a/packages/dofs/src/schema/sync.ts b/packages/dofs/src/schema/sync.ts index 78f70787..902b4e9e 100644 --- a/packages/dofs/src/schema/sync.ts +++ b/packages/dofs/src/schema/sync.ts @@ -52,6 +52,58 @@ export const SYNC_STATEMENTS = [ path TEXT, PRIMARY KEY (k, backend) )`, + // Durable half of a restartable sync operation. One row per + // (backend, direction): the plan's key. A restarted iterator reads + // this row to recover the fixed target and generation it was working + // against, because a JavaScript generator cannot survive eviction. + // + // `target_rev` / `target_path` are NULL only while status is + // 'capturing' — pull cannot capture its target atomically because + // watermarks({settle:true}) is a remote call, so capture is a + // two-phase insert-then-promote. `block_after_*` records where an + // in-flight block began and is cleared once the cursor advances; a + // later invocation that finds the marker with no cursor progress + // knows the prior execution was interrupted. + // + // The committed progress cursor stays in _vfs_watermark. This table + // never duplicates it. + `CREATE TABLE IF NOT EXISTS _vfs_sync_operations ( + backend TEXT NOT NULL, + direction TEXT NOT NULL CHECK (direction IN ('pull', 'push')), + generation TEXT NOT NULL, + status TEXT NOT NULL CHECK ( + status IN ('capturing', 'pending', 'failed', 'lost') + ), + target_rev INTEGER, + target_path TEXT, + runtime_id TEXT, + mode TEXT CHECK (mode IN ('entries', 'pack')), + block_after_rev INTEGER, + block_after_path TEXT, + block_started_at INTEGER, + internal_max_entries INTEGER NOT NULL, + internal_max_bytes INTEGER NOT NULL, + created_at INTEGER NOT NULL, + updated_at INTEGER NOT NULL, + last_error TEXT, + PRIMARY KEY (backend, direction) + )`, + // Entries the receiver deliberately refused, most often a + // container-side write under a read-only mount. The cursor advances + // past a rejection so the operation cannot stall on it forever, + // which means the only record of the drop would otherwise be a + // yielded progress value the caller may have discarded. These rows + // outlive the operation row and are pruned when a new generation + // starts. See docs/decisions/sync-operations.md. + `CREATE TABLE IF NOT EXISTS _vfs_sync_skips ( + backend TEXT NOT NULL, + direction TEXT NOT NULL CHECK (direction IN ('pull', 'push')), + generation TEXT NOT NULL, + path TEXT NOT NULL, + reason TEXT NOT NULL, + at INTEGER NOT NULL, + PRIMARY KEY (backend, direction, generation, path) + )`, // The `mode` column was added at schema v2; `schema/migrations.ts` // owns the ALTER for existing databases. Keep the CHECK // constraint here aligned with the migration's CHECK so fresh diff --git a/packages/dofs/src/sync/apply.ts b/packages/dofs/src/sync/apply.ts index a4dfdbaf..ccc421a7 100644 --- a/packages/dofs/src/sync/apply.ts +++ b/packages/dofs/src/sync/apply.ts @@ -291,6 +291,11 @@ export async function applyChanges( if (options.source === "upstream" && entry.kind !== "delete") { if (alreadyApplied(db, entry)) continue; } + // A replayed tombstone must not delete a path that was recreated + // above the tombstone's revision. + if (options.source === "upstream" && entry.kind === "delete" && tombstoneIsStale(db, entry)) { + continue; + } // Read-only mount guard. Entries under a registered read-only // mount root are surfaced via the return value and not applied. // The owning workspace's surface (Workspace.pull, exec()) folds @@ -416,6 +421,11 @@ export function applyChangesSync( if (options.source === "upstream" && entry.kind !== "delete") { if (alreadyApplied(db, entry)) continue; } + // See tombstoneIsStale: a replayed delete must not clobber a + // newer local recreation. + if (options.source === "upstream" && entry.kind === "delete" && tombstoneIsStale(db, entry)) { + continue; + } const blockingRoot = readOnlyRootFor(db, entry.path); if (blockingRoot !== undefined) { skipped.push({ @@ -557,6 +567,29 @@ function assertChunkSize(actual: number, declared: number, hash: Uint8Array, pat ); } +// Decide whether an upstream tombstone may delete the live path. +// +// A tombstone describes the path as of the revision it was stamped +// with. Because the sync cursor only advances after a block applies, +// any block interrupted before its acknowledgment is replayed — and a +// replayed tombstone whose path was recreated locally in the meantime +// would destroy content the tombstone never described. +// +// The guard is a revision comparison: apply the delete only when the +// live path is no newer than the tombstone. A path recreated above the +// tombstone's rev is newer information than the delete, so the delete +// is stale and dropped. It is not lost work — the recreation is itself +// a change that the next push ships upstream. +// +// Local deletes are exempt. They are authored here, not replayed, so +// there is no earlier revision to compare against. +function tombstoneIsStale(db: Database, entry: ChangeEntry & { kind: "delete" }): boolean { + const live = resolveInode(db, entry.path, { followSymlinks: false }); + if (live === null) return false; + const row = db.one<{ rev: number }>("SELECT rev FROM vfs_nodes WHERE inode = ?", live.inode); + return row !== undefined && row.rev > entry.rev; +} + // Compare an entry against the local node graph. Returns true when // the entry would be a no-op apply: the manifest hash (files), mode // (dirs), or mode + symlink target (symlinks) already matches. diff --git a/packages/dofs/src/sync/blocks.test.ts b/packages/dofs/src/sync/blocks.test.ts new file mode 100644 index 00000000..e035839d --- /dev/null +++ b/packages/dofs/src/sync/blocks.test.ts @@ -0,0 +1,333 @@ +import { describe, expect, it } from "vitest"; +import { rm } from "../fs/rm.js"; +import { withDB } from "../fs/with-db.js"; +import { writeFile } from "../fs/writeFile.js"; +import type { Database } from "../storage.js"; +import { PACK_THRESHOLD_BYTES, PACK_THRESHOLD_ENTRIES, planBlock, selectMode } from "./blocks.js"; +import { MIN_BLOCK_PROFILE } from "./operations.js"; +import { currentRev } from "./watermarks.js"; + +// The planner turns a cursor window into one block: an ordered prefix +// of the coalesced change stream, bounded by the internal sizing +// profile. Determinism is the property that matters most — the same +// start cursor and target must produce the same block, because an +// unacknowledged block gets re-requested and the receiver has to be +// able to absorb the replay. + +// Distinct content per file. Identical bytes would collapse to one +// content-addressed object and the byte-bound cases would then be +// measuring deduplication rather than the bound. +async function seedFiles(db: Database, count: number, bytes = 4): Promise { + for (let i = 0; i < count; i++) { + const name = `/f${String(i).padStart(4, "0")}.txt`; + const filler = String.fromCharCode(97 + (i % 26)).repeat(Math.max(1, bytes - 1)); + await writeFile(db, name, `${i}${filler}`.slice(0, Math.max(1, bytes)), {}, () => i + 1); + } +} + +describe("block planner", () => { + describe("ordered prefix selection", () => { + it("returns an empty drained block for an empty window", async () => { + await withDB(async (db) => { + const block = await planBlock(db, { + after: { rev: 0, path: null }, + through: { rev: currentRev(db), path: null }, + profile: MIN_BLOCK_PROFILE, + }); + + expect(block.entries).toEqual([]); + expect(block.drained).toBe(true); + }); + }); + + it("selects every entry when the window fits inside the profile", async () => { + await withDB(async (db) => { + await seedFiles(db, 3); + + const block = await planBlock(db, { + after: { rev: 0, path: null }, + through: { rev: currentRev(db), path: null }, + profile: MIN_BLOCK_PROFILE, + }); + + expect(block.entries.map((e) => e.path)).toEqual([ + "/f0000.txt", + "/f0001.txt", + "/f0002.txt", + ]); + expect(block.drained).toBe(true); + }); + }); + + it("sets the block cursor to the target when the window drains", async () => { + await withDB(async (db) => { + await seedFiles(db, 2); + const target = { rev: currentRev(db), path: null }; + + const block = await planBlock(db, { + after: { rev: 0, path: null }, + through: target, + profile: MIN_BLOCK_PROFILE, + }); + + expect(block.cursor).toEqual(target); + }); + }); + + it("stops at the entry limit and reports the window as not drained", async () => { + await withDB(async (db) => { + await seedFiles(db, 5); + + const block = await planBlock(db, { + after: { rev: 0, path: null }, + through: { rev: currentRev(db), path: null }, + profile: { maxEntries: 2, maxBytes: MIN_BLOCK_PROFILE.maxBytes }, + }); + + expect(block.entries).toHaveLength(2); + expect(block.drained).toBe(false); + }); + }); + + it("sets the block cursor to the last selected entry on a partial block", async () => { + await withDB(async (db) => { + await seedFiles(db, 5); + + const block = await planBlock(db, { + after: { rev: 0, path: null }, + through: { rev: currentRev(db), path: null }, + profile: { maxEntries: 2, maxBytes: MIN_BLOCK_PROFILE.maxBytes }, + }); + + const last = block.entries[block.entries.length - 1]; + expect(block.cursor).toEqual({ rev: last.rev, path: last.path }); + }); + }); + + it("resumes exactly where the previous block stopped, with no gap or repeat", async () => { + await withDB(async (db) => { + await seedFiles(db, 5); + const through = { rev: currentRev(db), path: null }; + const profile = { maxEntries: 2, maxBytes: MIN_BLOCK_PROFILE.maxBytes }; + + const collected: string[] = []; + let after = { rev: 0, path: null as string | null }; + for (let i = 0; i < 10; i++) { + const block = await planBlock(db, { after, through, profile }); + collected.push(...block.entries.map((e) => e.path)); + if (block.drained) break; + after = block.cursor; + } + + expect(collected).toEqual([ + "/f0000.txt", + "/f0001.txt", + "/f0002.txt", + "/f0003.txt", + "/f0004.txt", + ]); + }); + }); + + it("produces an identical block when the same request is replayed", async () => { + await withDB(async (db) => { + await seedFiles(db, 5); + const request = { + after: { rev: 0, path: null }, + through: { rev: currentRev(db), path: null }, + profile: { maxEntries: 3, maxBytes: MIN_BLOCK_PROFILE.maxBytes }, + }; + + const first = await planBlock(db, request); + const replay = await planBlock(db, request); + + expect(replay.entries).toEqual(first.entries); + expect(replay.cursor).toEqual(first.cursor); + }); + }); + + it("never selects an entry beyond the fixed target", async () => { + await withDB(async (db) => { + await seedFiles(db, 2); + // The target is fixed here, before the late write lands. + const target = { rev: currentRev(db), path: null }; + // A change created above the target must not join this block. + await writeFile(db, "/late.txt", "late", {}, () => 99); + + const block = await planBlock(db, { + after: { rev: 0, path: null }, + through: target, + profile: MIN_BLOCK_PROFILE, + }); + + expect(block.entries.map((e) => e.path)).not.toContain("/late.txt"); + }); + }); + }); + + describe("byte bound", () => { + it("stops once the unique object bytes exceed the profile", async () => { + await withDB(async (db) => { + await seedFiles(db, 4, 1000); + + const block = await planBlock(db, { + after: { rev: 0, path: null }, + through: { rev: currentRev(db), path: null }, + profile: { maxEntries: 1000, maxBytes: 2500 }, + }); + + expect(block.entries.length).toBeLessThan(4); + expect(block.drained).toBe(false); + }); + }); + + // Otherwise a file larger than the byte budget would be selected + // into an empty block, rejected for being too big, and retried + // forever at the same cursor. + it("allows the first entry to exceed the byte limit so a large file still progresses", async () => { + await withDB(async (db) => { + await writeFile(db, "/big.bin", "x".repeat(5000), {}, () => 1); + + const block = await planBlock(db, { + after: { rev: 0, path: null }, + through: { rev: currentRev(db), path: null }, + profile: { maxEntries: 1000, maxBytes: 100 }, + }); + + expect(block.entries.map((e) => e.path)).toEqual(["/big.bin"]); + }); + }); + + it("counts a duplicated object once within a block", async () => { + await withDB(async (db) => { + // Same content at two paths shares one hash, so the block + // carries the bytes once even though it carries two entries. + await writeFile(db, "/a.txt", "y".repeat(1000), {}, () => 1); + await writeFile(db, "/b.txt", "y".repeat(1000), {}, () => 2); + + const block = await planBlock(db, { + after: { rev: 0, path: null }, + through: { rev: currentRev(db), path: null }, + profile: { maxEntries: 1000, maxBytes: 1500 }, + }); + + expect(block.entries).toHaveLength(2); + expect(block.objectBytes).toBe(1000); + expect(block.objects).toHaveLength(1); + }); + }); + }); + + describe("deletions", () => { + it("carries a tombstone as an entry with no object bytes", async () => { + await withDB(async (db) => { + await writeFile(db, "/gone.txt", "bye", {}, () => 1); + rm(db, "/gone.txt", {}); + + const block = await planBlock(db, { + after: { rev: 0, path: null }, + through: { rev: currentRev(db), path: null }, + profile: MIN_BLOCK_PROFILE, + }); + + expect(block.entries.map((e) => e.kind)).toEqual(["delete"]); + expect(block.objectBytes).toBe(0); + }); + }); + }); + + describe("mode selection", () => { + it("chooses entry mode for a small window", async () => { + await withDB(async (db) => { + await seedFiles(db, 3); + + const decision = await selectMode(db, { + after: { rev: 0, path: null }, + through: { rev: currentRev(db), path: null }, + profile: MIN_BLOCK_PROFILE, + }); + + expect(decision.mode).toBe("entries"); + }); + }); + + it("chooses pack mode once the entry threshold is reached", async () => { + await withDB(async (db) => { + await seedFiles(db, 12); + + const decision = await selectMode(db, { + after: { rev: 0, path: null }, + through: { rev: currentRev(db), path: null }, + profile: MIN_BLOCK_PROFILE, + // Injected so the test does not have to write 20,000 files + // to reach the production threshold. + thresholdEntries: 10, + }); + + expect(decision.mode).toBe("pack"); + }); + }); + + it("chooses pack mode once the payload threshold is reached", async () => { + await withDB(async (db) => { + await seedFiles(db, 4, 1000); + + const decision = await selectMode(db, { + after: { rev: 0, path: null }, + through: { rev: currentRev(db), path: null }, + profile: MIN_BLOCK_PROFILE, + thresholdBytes: 2000, + }); + + expect(decision.mode).toBe("pack"); + }); + }); + + it("keeps the production thresholds at the planned values", () => { + expect(PACK_THRESHOLD_ENTRIES).toBe(20_000); + expect(PACK_THRESHOLD_BYTES).toBe(100 * 1024 * 1024); + }); + + // Q3: one planning pass has to answer the mode question and + // produce the first block, or a large sync pays for two scans of + // the same window before it transfers anything. + it("returns the first block alongside the mode decision", async () => { + await withDB(async (db) => { + await seedFiles(db, 3); + + const decision = await selectMode(db, { + after: { rev: 0, path: null }, + through: { rev: currentRev(db), path: null }, + profile: MIN_BLOCK_PROFILE, + }); + + expect(decision.firstBlock.entries.map((e) => e.path)).toEqual([ + "/f0000.txt", + "/f0001.txt", + "/f0002.txt", + ]); + }); + }); + + it("does not select a mode from entries beyond the target", async () => { + await withDB(async (db) => { + await seedFiles(db, 3); + const target = { rev: currentRev(db), path: null }; + // 12 more changes land above the target; they must not push + // this operation into pack mode. + for (let i = 0; i < 12; i++) { + await writeFile(db, `/late${i}.txt`, "late", {}, () => 100 + i); + } + + const decision = await selectMode(db, { + after: { rev: 0, path: null }, + through: target, + profile: MIN_BLOCK_PROFILE, + thresholdEntries: 10, + }); + + expect(decision.mode).toBe("entries"); + }); + }); + }); +}); diff --git a/packages/dofs/src/sync/blocks.ts b/packages/dofs/src/sync/blocks.ts new file mode 100644 index 00000000..c135b713 --- /dev/null +++ b/packages/dofs/src/sync/blocks.ts @@ -0,0 +1,185 @@ +// Block planning: turn a cursor window into one transferable unit. +// +// A block is an ordered prefix of the coalesced change stream, bounded +// by the internal sizing profile. Determinism is the property the +// whole restart story rests on: the same `after` and `through` with the +// same profile must select the same entries, because an unacknowledged +// block is re-requested and the receiver has to absorb the replay as a +// no-op rather than as new work. +// +// Block sizing is never caller-visible. The public `pull()` and +// `push()` iterables take no budgets; these bounds exist so one block +// fits inside a Durable Object's CPU allowance. + +import type { Database } from "../storage.js"; +import type { ChangeEntry } from "./changes.js"; +import { coalesceChanges } from "./coalesce.js"; +import type { BlockProfile, SyncMode } from "./operations.js"; +import type { ChangeCursor } from "./watermarks.js"; + +// Above either threshold an operation switches from shipping +// individual entries to shipping a compressed content-addressed pack. +// The measured package-install benchmark crossed both by a wide +// margin: 44,264 entries and an estimated 914 MB window. +export const PACK_THRESHOLD_ENTRIES = 20_000; +export const PACK_THRESHOLD_BYTES = 100 * 1024 * 1024; + +export interface BlockObject { + readonly hash: Uint8Array; + readonly size: number; +} + +export interface PlannedBlock { + // Entries in wire order. Already coalesced, so one path appears at + // most once. + readonly entries: ChangeEntry[]; + // Unique objects referenced by this block's entries. Deduplicated + // within the block only; duplicates across blocks are absorbed by + // the receiver's content-addressed store, which is cheaper than + // carrying operation-wide sent-object state. + readonly objects: BlockObject[]; + readonly objectBytes: number; + // Cursor to persist once this block is applied and acknowledged. + // The target when the window drained, otherwise the last fully + // selected entry — never a partially selected one. + readonly cursor: ChangeCursor; + // True when this block reached the fixed target, meaning the + // operation is complete once this block commits. + readonly drained: boolean; +} + +export interface PlanBlockRequest { + readonly after: ChangeCursor; + readonly through: ChangeCursor; + readonly profile: BlockProfile; + readonly ignore?: string[]; +} + +function toHex(bytes: Uint8Array): string { + let out = ""; + for (const b of bytes) out += b.toString(16).padStart(2, "0"); + return out; +} + +// Objects referenced by one entry. Only file entries carry bytes; +// directories, symlinks, and tombstones are metadata only, which is +// why a delete-heavy window stays cheap regardless of entry count. +function objectsOf(entry: ChangeEntry): BlockObject[] { + return entry.kind === "file" ? entry.chunks.map((c) => ({ hash: c.hash, size: c.size })) : []; +} + +// Select an ordered prefix of the window. +// +// The entry limit and the unique-object byte limit both stop +// selection, with one exception: an entry selected into an empty block +// is always kept even if it alone exceeds the byte budget. Without +// that rule a file larger than the budget would be selected, rejected +// for being oversized, and retried at the same cursor forever. +export async function planBlock(db: Database, request: PlanBlockRequest): Promise { + const { after, through, profile } = request; + const entries: ChangeEntry[] = []; + const objects: BlockObject[] = []; + const seen = new Set(); + let objectBytes = 0; + let drained = true; + + const options = request.ignore === undefined ? { through } : { through, ignore: request.ignore }; + + for await (const entry of coalesceChanges(db, after, options)) { + if (entries.length >= profile.maxEntries) { + // The window still has entries the profile cannot carry, so the + // operation is not finished at this cursor. + drained = false; + break; + } + + // Cost this entry against objects the block does not already + // carry. Two paths sharing content cost bytes once. + const candidates = objectsOf(entry).filter((o) => !seen.has(toHex(o.hash))); + const unique = new Map(); + for (const object of candidates) unique.set(toHex(object.hash), object); + let addedBytes = 0; + for (const object of unique.values()) addedBytes += object.size; + + if (entries.length > 0 && objectBytes + addedBytes > profile.maxBytes) { + drained = false; + break; + } + + entries.push(entry); + for (const [key, object] of unique) { + seen.add(key); + objects.push(object); + objectBytes += object.size; + } + } + + // A drained window means every change through the target was + // offered, so the cursor jumps to the target itself. That is what + // lets an operation complete on a cursor whose rev carries no + // entries of its own. + const last = entries[entries.length - 1]; + const cursor: ChangeCursor = drained + ? through + : last === undefined + ? after + : { rev: last.rev, path: last.path }; + + return { entries, objects, objectBytes, cursor, drained }; +} + +export interface SelectModeRequest extends PlanBlockRequest { + // Overridable so tests can cross a threshold without materialising + // 20,000 files. Production callers use the module constants. + readonly thresholdEntries?: number; + readonly thresholdBytes?: number; +} + +export interface ModeDecision { + readonly mode: SyncMode; + // The planning pass that answered the mode question is also the + // first block, so a large operation does not scan its window twice + // before transferring anything. See Q3 in + // docs/decisions/sync-operations.md. + readonly firstBlock: PlannedBlock; +} + +// Decide entry versus pack mode for a whole operation, and return the +// first block from the same pass. +// +// The probe reads up to `max(profile.maxEntries, thresholdEntries)` +// entries. Filling the probe is itself the signal that the window is +// at or past the entry threshold, so the question is answered without +// draining a window that may hold 44,000 entries. Only metadata is +// buffered; object payloads are never held here. +// +// The mode is fixed for the operation's life. A block's encoding must +// not depend on when it was requested or a replayed block would not +// match the original, so a wrong estimate is not corrected mid-flight. +// The target is fixed at creation, so the window cannot grow under the +// estimate. +export async function selectMode(db: Database, request: SelectModeRequest): Promise { + const thresholdEntries = request.thresholdEntries ?? PACK_THRESHOLD_ENTRIES; + const thresholdBytes = request.thresholdBytes ?? PACK_THRESHOLD_BYTES; + + // Probe wide enough to answer the threshold question, but never + // narrower than one block, so the probe's prefix is reusable as the + // first block. + const probeEntries = Math.max(request.profile.maxEntries, thresholdEntries); + const probe = await planBlock(db, { + ...request, + profile: { maxEntries: probeEntries, maxBytes: Number.MAX_SAFE_INTEGER }, + }); + + const crossed = probe.entries.length >= thresholdEntries || probe.objectBytes >= thresholdBytes; + + // Re-derive the first block under the real profile. When the probe + // already fits the profile it *is* the first block and no second + // pass happens. + const fits = + probe.entries.length <= request.profile.maxEntries && + probe.objectBytes <= request.profile.maxBytes; + const firstBlock = fits ? probe : await planBlock(db, request); + + return { mode: crossed ? "pack" : "entries", firstBlock }; +} diff --git a/packages/dofs/src/sync/change-pack.test.ts b/packages/dofs/src/sync/change-pack.test.ts new file mode 100644 index 00000000..9f3f2ba4 --- /dev/null +++ b/packages/dofs/src/sync/change-pack.test.ts @@ -0,0 +1,350 @@ +import { describe, expect, it } from "vitest"; + +import { mkdir } from "../fs/mkdir.js"; +import { rm } from "../fs/rm.js"; +import { symlink } from "../fs/symlink.js"; +import { withDB } from "../fs/with-db.js"; +import { writeFile } from "../fs/writeFile.js"; +import type { Database } from "../storage.js"; +import { planBlock } from "./blocks.js"; +import { + decodeChangePack, + encodeChangePack, + PACK_FORMAT_VERSION, + PackProtocolError, + readPackFooter, +} from "./change-pack.js"; +import { MIN_BLOCK_PROFILE } from "./operations.js"; +import { currentRev } from "./watermarks.js"; + +// The pack is the bulk transport: one gzip stream carrying an ordered +// run of change entries interleaved with the unique objects they +// reference, closed by a footer that identifies the cursor window and +// validates the contents. +// +// Two properties matter most. It must round-trip exactly, because the +// receiver reconstructs a filesystem from it. And a truncated or +// tampered stream must be rejected rather than half-applied, because +// the cursor would otherwise advance over data that never landed. + +async function collect(stream: ReadableStream): Promise { + const chunks: Uint8Array[] = []; + const reader = stream.getReader(); + let total = 0; + while (true) { + const { value, done } = await reader.read(); + if (done) break; + chunks.push(value); + total += value.byteLength; + } + const out = new Uint8Array(total); + let offset = 0; + for (const chunk of chunks) { + out.set(chunk, offset); + offset += chunk.byteLength; + } + return out; +} + +function streamOf(bytes: Uint8Array): ReadableStream { + return new ReadableStream({ + start(controller) { + controller.enqueue(bytes); + controller.close(); + }, + }); +} + +async function seedFiles(db: Database, count: number, bytes = 8): Promise { + for (let i = 0; i < count; i++) { + const filler = String.fromCharCode(97 + (i % 26)).repeat(Math.max(1, bytes - 1)); + await writeFile(db, `/f${String(i).padStart(3, "0")}.txt`, `${i}${filler}`, {}, () => i + 1); + } +} + +// Encode whatever planBlock selected for the whole current window. +async function packWindow( + db: Database, + profile = MIN_BLOCK_PROFILE, +): Promise<{ bytes: Uint8Array; block: Awaited> }> { + const through = { rev: currentRev(db), path: null }; + const block = await planBlock(db, { after: { rev: 0, path: null }, through, profile }); + const bytes = await collect( + encodeChangePack(db, { + block, + after: { rev: 0, path: null }, + target: through, + generation: "gen-1", + }), + ); + return { bytes, block }; +} + +describe("change pack codec", () => { + describe("round trip", () => { + it("carries file entries and their bytes", async () => { + await withDB(async (db) => { + await seedFiles(db, 3); + const { bytes, block } = await packWindow(db); + + const decoded = await decodeChangePack(streamOf(bytes)); + + expect(decoded.entries.map((e) => e.path)).toEqual(block.entries.map((e) => e.path)); + expect(decoded.objects.size).toBe(block.objects.length); + }); + }); + + it("preserves file mode, mtime, and chunk list exactly", async () => { + await withDB(async (db) => { + await writeFile(db, "/only.txt", "payload bytes", {}, () => 5); + const { bytes, block } = await packWindow(db); + + const decoded = await decodeChangePack(streamOf(bytes)); + + expect(decoded.entries[0]).toEqual(block.entries[0]); + }); + }); + + it("round-trips object bytes verbatim", async () => { + await withDB(async (db) => { + await writeFile(db, "/only.txt", "payload bytes", {}, () => 5); + const { bytes, block } = await packWindow(db); + + const decoded = await decodeChangePack(streamOf(bytes)); + + const hash = block.objects[0].hash; + const key = [...hash].map((b) => b.toString(16).padStart(2, "0")).join(""); + expect(new TextDecoder().decode(decoded.objects.get(key))).toBe("payload bytes"); + }); + }); + + it("carries directories, symlinks, and deletions", async () => { + await withDB(async (db) => { + mkdir(db, "/dir", { recursive: true }, () => 1); + await writeFile(db, "/dir/file.txt", "content", {}, () => 2); + symlink(db, "/dir/file.txt", "/link", () => 3); + await writeFile(db, "/gone.txt", "bye", {}, () => 4); + rm(db, "/gone.txt", {}); + const { bytes, block } = await packWindow(db); + + const decoded = await decodeChangePack(streamOf(bytes)); + + expect(decoded.entries.map((e) => e.kind).sort()).toEqual( + block.entries.map((e) => e.kind).sort(), + ); + expect(decoded.entries.some((e) => e.kind === "delete")).toBe(true); + }); + }); + + it("stores an object shared by two paths exactly once", async () => { + await withDB(async (db) => { + await writeFile(db, "/a.txt", "identical", {}, () => 1); + await writeFile(db, "/b.txt", "identical", {}, () => 2); + const { bytes } = await packWindow(db); + + const decoded = await decodeChangePack(streamOf(bytes)); + + expect(decoded.entries).toHaveLength(2); + expect(decoded.objects.size).toBe(1); + }); + }); + + it("handles an empty block", async () => { + await withDB(async (db) => { + const cursor = { rev: 0, path: null }; + const block = await planBlock(db, { + after: cursor, + through: cursor, + profile: MIN_BLOCK_PROFILE, + }); + const bytes = await collect( + encodeChangePack(db, { block, after: cursor, target: cursor, generation: "gen-1" }), + ); + + const decoded = await decodeChangePack(streamOf(bytes)); + + expect(decoded.entries).toEqual([]); + expect(decoded.objects.size).toBe(0); + }); + }); + + it("survives a stream delivered in many small pieces", async () => { + await withDB(async (db) => { + await seedFiles(db, 4, 64); + const { bytes, block } = await packWindow(db); + + // Re-chunk byte by byte: the decoder must not assume record + // boundaries align with transport chunks. + const dribble = new ReadableStream({ + start(controller) { + for (const byte of bytes) controller.enqueue(new Uint8Array([byte])); + controller.close(); + }, + }); + const decoded = await decodeChangePack(dribble); + + expect(decoded.entries).toHaveLength(block.entries.length); + }); + }); + }); + + describe("footer", () => { + it("identifies the cursor window and generation", async () => { + await withDB(async (db) => { + await seedFiles(db, 3); + const { bytes, block } = await packWindow(db); + + const decoded = await decodeChangePack(streamOf(bytes)); + + expect(decoded.footer.version).toBe(PACK_FORMAT_VERSION); + expect(decoded.footer.generation).toBe("gen-1"); + expect(decoded.footer.after).toEqual({ rev: 0, path: null }); + expect(decoded.footer.blockCursor).toEqual(block.cursor); + expect(decoded.footer.target).toEqual({ rev: currentRev(db), path: null }); + }); + }); + + it("counts entries and objects", async () => { + await withDB(async (db) => { + await seedFiles(db, 3); + const { bytes, block } = await packWindow(db); + + const decoded = await decodeChangePack(streamOf(bytes)); + + expect(decoded.footer.entryCount).toBe(block.entries.length); + expect(decoded.footer.objectCount).toBe(block.objects.length); + }); + }); + + it("can be read without decoding the whole pack body", async () => { + await withDB(async (db) => { + await seedFiles(db, 3); + const { bytes, block } = await packWindow(db); + + const footer = await readPackFooter(streamOf(bytes)); + + expect(footer.blockCursor).toEqual(block.cursor); + }); + }); + + it("preserves a same-revision path in the block cursor", async () => { + await withDB(async (db) => { + await seedFiles(db, 5); + const through = { rev: currentRev(db), path: null }; + // A profile of 2 forces a partial block, so the cursor + // carries a path rather than a bare rev. + const block = await planBlock(db, { + after: { rev: 0, path: null }, + through, + profile: { maxEntries: 2, maxBytes: MIN_BLOCK_PROFILE.maxBytes }, + }); + const bytes = await collect( + encodeChangePack(db, { + block, + after: { rev: 0, path: null }, + target: through, + generation: "gen-1", + }), + ); + + const decoded = await decodeChangePack(streamOf(bytes)); + + expect(decoded.footer.blockCursor.path).toBe(block.cursor.path); + expect(typeof decoded.footer.blockCursor.path).toBe("string"); + }); + }); + }); + + describe("validation", () => { + // A truncated pack must not apply partially. The cursor would + // otherwise advance over entries that never arrived. + it("rejects a truncated stream", async () => { + await withDB(async (db) => { + await seedFiles(db, 4, 64); + const { bytes } = await packWindow(db); + + const truncated = bytes.slice(0, Math.floor(bytes.byteLength / 2)); + + await expect(decodeChangePack(streamOf(truncated))).rejects.toThrow(PackProtocolError); + }); + }); + + it("rejects a pack whose object bytes were tampered with", async () => { + await withDB(async (db) => { + await writeFile(db, "/only.txt", "payload bytes", {}, () => 5); + const { bytes } = await packWindow(db); + + // Flip a byte in the middle of the compressed stream. Either + // gzip or the digest must catch it; both are protocol errors. + const tampered = new Uint8Array(bytes); + const at = Math.floor(tampered.byteLength / 2); + tampered[at] = tampered[at] ^ 0xff; + + await expect(decodeChangePack(streamOf(tampered))).rejects.toThrow(PackProtocolError); + }); + }); + + it("rejects an unknown format version", async () => { + await withDB(async (db) => { + await seedFiles(db, 2); + const { bytes } = await packWindow(db); + + await expect( + decodeChangePack(streamOf(bytes), { expectVersion: PACK_FORMAT_VERSION + 1 }), + ).rejects.toThrow(PackProtocolError); + }); + }); + + it("rejects a pack whose declared entry count disagrees with its records", async () => { + await withDB(async (db) => { + await seedFiles(db, 3); + const through = { rev: currentRev(db), path: null }; + const block = await planBlock(db, { + after: { rev: 0, path: null }, + through, + profile: MIN_BLOCK_PROFILE, + }); + + const bytes = await collect( + encodeChangePack(db, { + block, + after: { rev: 0, path: null }, + target: through, + generation: "gen-1", + // Lie about the count the footer will carry. + corruptEntryCount: 99, + }), + ); + + await expect(decodeChangePack(streamOf(bytes))).rejects.toThrow(PackProtocolError); + }); + }); + + it("names the offending pack in the error so a protocol failure is diagnosable", async () => { + await withDB(async (db) => { + await seedFiles(db, 2); + const { bytes } = await packWindow(db); + const truncated = bytes.slice(0, 12); + + await expect(decodeChangePack(streamOf(truncated))).rejects.toThrow(/pack/i); + }); + }); + }); + + describe("compression", () => { + it("compresses repetitive content below its raw size", async () => { + await withDB(async (db) => { + // Highly repetitive distinct files: each is unique so they do + // not dedupe, but gzip should still win big. + for (let i = 0; i < 8; i++) { + await writeFile(db, `/f${i}.txt`, `${i}${"z".repeat(4000)}`, {}, () => i + 1); + } + const raw = 8 * 4001; + + const { bytes } = await packWindow(db, { maxEntries: 100, maxBytes: 64 * 1024 * 1024 }); + + expect(bytes.byteLength).toBeLessThan(raw / 2); + }); + }); + }); +}); diff --git a/packages/dofs/src/sync/change-pack.ts b/packages/dofs/src/sync/change-pack.ts new file mode 100644 index 00000000..c3632197 --- /dev/null +++ b/packages/dofs/src/sync/change-pack.ts @@ -0,0 +1,417 @@ +// Change pack: the bulk transport for large cursor windows. +// +// Entry-at-a-time transfer is fine for a handful of changes and +// hopeless for tens of thousands. A pack carries one ordered run of +// change entries interleaved with the unique objects those entries +// reference, gzip-compressed, closed by a footer that names the cursor +// window and validates the contents. +// +// Interleaving matters. Objects are emitted immediately before the +// entries that first reference them, so a receiver can stage bytes and +// apply metadata incrementally instead of buffering the whole pack. +// The plan calls for Durable Object storage to settle between apply +// groups; interleaved records are what make that possible. +// +// Framing is length-prefixed records rather than a self-describing +// format, because the decoder must never guess where a record ends. A +// truncated stream therefore fails as a protocol error rather than +// silently decoding a prefix — the cursor would otherwise advance over +// entries that never arrived. +// +// The format is generic. Nothing here knows about `node_modules` or any +// other path shape; a pack is selected purely on the size of the cursor +// window (see blocks.ts). + +import type { Database } from "../storage.js"; +import type { PlannedBlock } from "./blocks.js"; +import type { ChangeEntry } from "./changes.js"; +import type { ChangeCursor } from "./watermarks.js"; + +export const PACK_FORMAT_VERSION = 1; + +// Transport values are chunked so no single enqueued buffer is +// unbounded. The plan specifies 1 MiB. +const TRANSPORT_CHUNK_BYTES = 1024 * 1024; + +// Record tags. A tag is one byte; the payload length is a u32 that +// follows it. +const TAG_HEADER = 1; +const TAG_OBJECT = 2; +const TAG_ENTRY = 3; +const TAG_FOOTER = 4; + +// A protocol error means the pack itself is invalid: truncated, +// corrupt, mis-versioned, or internally inconsistent. It is distinct +// from a transport error because it must not be retried blindly — the +// cursor stays put and the failure is surfaced. +export class PackProtocolError extends Error { + readonly code = "EPACK_PROTOCOL"; + constructor(message: string) { + super(`change pack: ${message}`); + this.name = "PackProtocolError"; + } +} + +export interface PackFooter { + readonly version: number; + readonly generation: string; + readonly after: ChangeCursor; + readonly blockCursor: ChangeCursor; + readonly target: ChangeCursor; + readonly entryCount: number; + readonly objectCount: number; + readonly objectBytes: number; + // Digest over every entry and object record in emission order. Cheap + // insurance that the body matches what the footer claims. + readonly digest: string; +} + +export interface EncodeChangePackInput { + readonly block: PlannedBlock; + readonly after: ChangeCursor; + readonly target: ChangeCursor; + readonly generation: string; + // Test-only: force the footer to disagree with the body so the + // decoder's consistency check can be exercised. + readonly corruptEntryCount?: number; +} + +export interface DecodedChangePack { + readonly footer: PackFooter; + readonly entries: ChangeEntry[]; + // Hex-keyed object bytes, ready for stageBlob. + readonly objects: Map; +} + +function toHex(bytes: Uint8Array): string { + let out = ""; + for (const b of bytes) out += b.toString(16).padStart(2, "0"); + return out; +} + +function fromHex(hex: string): Uint8Array { + const out = new Uint8Array(hex.length / 2); + for (let i = 0; i < out.length; i++) { + out[i] = Number.parseInt(hex.slice(i * 2, i * 2 + 2), 16); + } + return out; +} + +// Entries carry Uint8Array chunk hashes, which JSON cannot represent. +// Hashes become hex on the wire and come back as bytes, so a decoded +// entry is === equal to the encoded one. +function encodeEntry(entry: ChangeEntry): string { + if (entry.kind === "file") { + return JSON.stringify({ + ...entry, + chunks: entry.chunks.map((c) => ({ hash: toHex(c.hash), size: c.size })), + }); + } + return JSON.stringify(entry); +} + +function decodeEntry(text: string): ChangeEntry { + const raw = JSON.parse(text) as Record; + if (raw.kind === "file") { + const chunks = (raw.chunks as { hash: string; size: number }[]).map((c) => ({ + hash: fromHex(c.hash), + size: c.size, + })); + return { ...(raw as unknown as ChangeEntry), chunks } as ChangeEntry; + } + return raw as unknown as ChangeEntry; +} + +// FNV-1a over the record stream. Not a security boundary — gzip +// already catches random corruption via its own CRC. This catches the +// case gzip cannot: a well-formed stream whose records disagree with +// the footer that describes them. +function digestUpdate(state: number, bytes: Uint8Array): number { + let hash = state; + for (const byte of bytes) { + hash ^= byte; + hash = Math.imul(hash, 0x01000193) >>> 0; + } + return hash; +} + +function frameRecord(tag: number, payload: Uint8Array): Uint8Array { + const framed = new Uint8Array(5 + payload.byteLength); + framed[0] = tag; + new DataView(framed.buffer).setUint32(1, payload.byteLength, false); + framed.set(payload, 5); + return framed; +} + +// Emit the pack as a gzip byte stream. +// +// Object payloads are read from the blob store one at a time and +// released as they are written, so the encoder's memory stays bounded +// by the largest single object rather than by the whole pack. +export function encodeChangePack( + db: Database, + input: EncodeChangePackInput, +): ReadableStream { + const { block, after, target, generation } = input; + const encoder = new TextEncoder(); + + const raw = new ReadableStream({ + start(controller) { + let digest = 0x811c9dc5; + + controller.enqueue( + frameRecord( + TAG_HEADER, + encoder.encode(JSON.stringify({ version: PACK_FORMAT_VERSION, generation })), + ), + ); + + // Objects precede the entries that reference them. Emitted once + // per pack; duplicates across packs are absorbed by the + // receiver's content-addressed store. + const emitted = new Set(); + let objectBytes = 0; + let objectCount = 0; + + const emitObjectsFor = (entry: ChangeEntry): void => { + if (entry.kind !== "file") return; + for (const chunk of entry.chunks) { + const key = toHex(chunk.hash); + if (emitted.has(key)) continue; + emitted.add(key); + const row = db.one<{ bytes: Uint8Array }>( + "SELECT bytes FROM vfs_blob_bytes WHERE hash = ?", + chunk.hash, + ); + if (row === undefined) { + controller.error(new PackProtocolError(`missing local object ${key}`)); + return; + } + const payload = new Uint8Array(row.bytes.byteLength + 32); + payload.set(chunk.hash, 0); + payload.set(row.bytes, 32); + const record = frameRecord(TAG_OBJECT, payload); + digest = digestUpdate(digest, record); + controller.enqueue(record); + objectBytes += row.bytes.byteLength; + objectCount += 1; + } + }; + + for (const entry of block.entries) { + emitObjectsFor(entry); + const record = frameRecord(TAG_ENTRY, encoder.encode(encodeEntry(entry))); + digest = digestUpdate(digest, record); + controller.enqueue(record); + } + + const footer: PackFooter = { + version: PACK_FORMAT_VERSION, + generation, + after, + blockCursor: block.cursor, + target, + entryCount: input.corruptEntryCount ?? block.entries.length, + objectCount, + objectBytes, + digest: digest.toString(16), + }; + controller.enqueue(frameRecord(TAG_FOOTER, encoder.encode(JSON.stringify(footer)))); + controller.close(); + }, + }); + + // Re-chunk to bounded transport values, then compress. + return raw.pipeThrough(gzipStream("gzip")).pipeThrough(rechunk()); +} + +// CompressionStream / DecompressionStream are typed with a narrower +// Uint8Array view than the ReadableStream the +// sync wire uses elsewhere, so the pair needs a cast at this one +// boundary rather than widening every stream signature in the package. +function gzipStream(format: "gzip"): ReadableWritablePair { + return new CompressionStream(format) as unknown as ReadableWritablePair; +} + +function gunzipStream(format: "gzip"): ReadableWritablePair { + return new DecompressionStream(format) as unknown as ReadableWritablePair; +} + +function rechunk(): TransformStream { + let pending = new Uint8Array(0); + return new TransformStream({ + transform(chunk, controller) { + const merged = new Uint8Array(pending.byteLength + chunk.byteLength); + merged.set(pending, 0); + merged.set(chunk, pending.byteLength); + let offset = 0; + while (merged.byteLength - offset >= TRANSPORT_CHUNK_BYTES) { + controller.enqueue(merged.slice(offset, offset + TRANSPORT_CHUNK_BYTES)); + offset += TRANSPORT_CHUNK_BYTES; + } + pending = merged.slice(offset); + }, + flush(controller) { + if (pending.byteLength > 0) controller.enqueue(pending); + }, + }); +} + +// Pull length-prefixed records out of a decompressed byte stream. +// +// The reader buffers only the record it is currently assembling, so a +// pack larger than memory still decodes as long as no single object +// exceeds it. Transport chunk boundaries are irrelevant: records are +// reassembled across them. +async function* readRecords( + stream: ReadableStream, +): AsyncGenerator<{ tag: number; payload: Uint8Array; framed: Uint8Array }> { + const reader = stream.getReader(); + let buffer = new Uint8Array(0); + let done = false; + + const pull = async (): Promise => { + const { value, done: finished } = await reader.read(); + if (finished) { + done = true; + return false; + } + const merged = new Uint8Array(buffer.byteLength + value.byteLength); + merged.set(buffer, 0); + merged.set(value, buffer.byteLength); + buffer = merged; + return true; + }; + + try { + while (true) { + while (buffer.byteLength < 5 && !done) { + if (!(await pull())) break; + } + if (buffer.byteLength === 0 && done) return; + if (buffer.byteLength < 5) { + throw new PackProtocolError("truncated record header"); + } + const length = new DataView(buffer.buffer, buffer.byteOffset, buffer.byteLength).getUint32( + 1, + false, + ); + while (buffer.byteLength < 5 + length && !done) { + if (!(await pull())) break; + } + if (buffer.byteLength < 5 + length) { + throw new PackProtocolError("truncated record payload"); + } + const framed = buffer.slice(0, 5 + length); + yield { tag: buffer[0], payload: buffer.slice(5, 5 + length), framed }; + buffer = buffer.slice(5 + length); + } + } finally { + reader.releaseLock(); + } +} + +export interface DecodeChangePackOptions { + readonly expectVersion?: number; + readonly expectGeneration?: string; +} + +// Decode and validate a pack. +// +// Validation is deliberately strict and happens before the caller sees +// anything applicable: a pack that fails any check throws, leaving the +// cursor untouched, because a partially applied pack whose cursor +// advanced would lose the entries that never arrived. +export async function decodeChangePack( + stream: ReadableStream, + options: DecodeChangePackOptions = {}, +): Promise { + const expectVersion = options.expectVersion ?? PACK_FORMAT_VERSION; + const decoder = new TextDecoder(); + const entries: ChangeEntry[] = []; + const objects = new Map(); + let footer: PackFooter | undefined; + let digest = 0x811c9dc5; + let sawHeader = false; + + let source: ReadableStream; + try { + source = stream.pipeThrough(gunzipStream("gzip")); + } catch (error) { + throw new PackProtocolError(`could not open gzip stream: ${String(error)}`); + } + + try { + for await (const record of readRecords(source)) { + if (record.tag === TAG_HEADER) { + const header = JSON.parse(decoder.decode(record.payload)) as { + version: number; + generation: string; + }; + if (header.version !== expectVersion) { + throw new PackProtocolError( + `unsupported format version ${header.version}, expected ${expectVersion}`, + ); + } + if ( + options.expectGeneration !== undefined && + header.generation !== options.expectGeneration + ) { + throw new PackProtocolError( + `generation ${header.generation} does not match expected ${options.expectGeneration}`, + ); + } + sawHeader = true; + continue; + } + if (record.tag === TAG_OBJECT) { + digest = digestUpdate(digest, record.framed); + const hash = record.payload.slice(0, 32); + objects.set(toHex(hash), record.payload.slice(32)); + continue; + } + if (record.tag === TAG_ENTRY) { + digest = digestUpdate(digest, record.framed); + entries.push(decodeEntry(decoder.decode(record.payload))); + continue; + } + if (record.tag === TAG_FOOTER) { + footer = JSON.parse(decoder.decode(record.payload)) as PackFooter; + continue; + } + throw new PackProtocolError(`unknown record tag ${record.tag}`); + } + } catch (error) { + if (error instanceof PackProtocolError) throw error; + // A gzip CRC failure or an aborted stream lands here. Both mean the + // pack cannot be trusted. + throw new PackProtocolError(`corrupt or truncated stream: ${String(error)}`); + } + + if (!sawHeader) throw new PackProtocolError("missing header record"); + // Absent footer is the truncation signal: the footer is the last + // record, so a stream that ended without one is incomplete. + if (footer === undefined) throw new PackProtocolError("missing footer record"); + if (footer.entryCount !== entries.length) { + throw new PackProtocolError( + `footer declares ${footer.entryCount} entries but body carried ${entries.length}`, + ); + } + if (footer.objectCount !== objects.size) { + throw new PackProtocolError( + `footer declares ${footer.objectCount} objects but body carried ${objects.size}`, + ); + } + if (footer.digest !== digest.toString(16)) { + throw new PackProtocolError("body digest does not match footer"); + } + + return { footer, entries, objects }; +} + +// Read just the footer. Useful for diagnostics and for a receiver that +// wants the block cursor before committing to a full decode. +export async function readPackFooter(stream: ReadableStream): Promise { + const decoded = await decodeChangePack(stream); + return decoded.footer; +} diff --git a/packages/dofs/src/sync/operations.test.ts b/packages/dofs/src/sync/operations.test.ts new file mode 100644 index 00000000..30700aad --- /dev/null +++ b/packages/dofs/src/sync/operations.test.ts @@ -0,0 +1,429 @@ +import { describe, expect, it } from "vitest"; + +import { withDB } from "../fs/with-db.js"; +import { + beginCapture, + clearBlockMarker, + completeOperation, + DEFAULT_BLOCK_PROFILE, + failOperation, + fixTarget, + MIN_BLOCK_PROFILE, + markBlockStarted, + openOperation, + pruneSkips, + readOperation, + readSkips, + recordSkip, + shrinkBlockProfile, +} from "./operations.js"; + +// The operation row is the durable half of a restartable sync. These +// tests treat it as a state machine: absent -> capturing -> pending -> +// absent, with failed and lost as terminal branches. Every transition +// has to survive being re-read by a fresh caller, because after a +// Durable Object eviction that is the only thing left. + +describe("sync operations", () => { + describe("creation and target capture", () => { + it("reports no operation for a backend that has never synced", async () => { + await withDB(async (db) => { + expect(readOperation(db, "container", "pull")).toBeUndefined(); + }); + }); + + it("beginCapture inserts a capturing row with no target", async () => { + await withDB(async (db) => { + const generation = beginCapture(db, "container", "pull"); + + const operation = readOperation(db, "container", "pull"); + expect(operation?.status).toBe("capturing"); + expect(operation?.generation).toBe(generation); + expect(operation?.target).toBeUndefined(); + }); + }); + + it("fixTarget promotes a capturing row to pending with a fixed target", async () => { + await withDB(async (db) => { + const generation = beginCapture(db, "container", "pull"); + + const promoted = fixTarget(db, "container", "pull", generation, { + target: { rev: 42, path: null }, + mode: "entries", + }); + + expect(promoted).toBe(true); + const operation = readOperation(db, "container", "pull"); + expect(operation?.status).toBe("pending"); + expect(operation?.target).toEqual({ rev: 42, path: null }); + expect(operation?.mode).toBe("entries"); + }); + }); + + it("preserves a same-revision path through target serialization", async () => { + await withDB(async (db) => { + const generation = beginCapture(db, "container", "pull"); + fixTarget(db, "container", "pull", generation, { + target: { rev: 7, path: "/dir/file.txt" }, + mode: "pack", + }); + + expect(readOperation(db, "container", "pull")?.target).toEqual({ + rev: 7, + path: "/dir/file.txt", + }); + }); + }); + + it("treats the empty string as a real target path, not a sentinel", async () => { + await withDB(async (db) => { + const generation = beginCapture(db, "container", "pull"); + fixTarget(db, "container", "pull", generation, { + target: { rev: 9, path: "" }, + mode: "entries", + }); + + expect(readOperation(db, "container", "pull")?.target).toEqual({ rev: 9, path: "" }); + }); + }); + + it("keys operations independently by backend and direction", async () => { + await withDB(async (db) => { + const pull = beginCapture(db, "container", "pull"); + fixTarget(db, "container", "pull", pull, { + target: { rev: 1, path: null }, + mode: "entries", + }); + const push = beginCapture(db, "container", "push"); + fixTarget(db, "container", "push", push, { + target: { rev: 2, path: null }, + mode: "entries", + }); + const other = beginCapture(db, "worker", "pull"); + fixTarget(db, "worker", "pull", other, { + target: { rev: 3, path: null }, + mode: "entries", + }); + + expect(readOperation(db, "container", "pull")?.target?.rev).toBe(1); + expect(readOperation(db, "container", "push")?.target?.rev).toBe(2); + expect(readOperation(db, "worker", "pull")?.target?.rev).toBe(3); + }); + }); + + it("repeats capture when eviction interrupted the first attempt", async () => { + await withDB(async (db) => { + // First iterator inserted a capturing row and was evicted + // before it could fix a target. + beginCapture(db, "container", "pull"); + + // A fresh iterator finds the capturing row and takes it over + // with a new generation, because the abandoned generation may + // still be in flight remotely. + const second = beginCapture(db, "container", "pull"); + const operation = readOperation(db, "container", "pull"); + + expect(operation?.generation).toBe(second); + expect(operation?.status).toBe("capturing"); + }); + }); + + it("rejects a target fixed against a superseded generation", async () => { + await withDB(async (db) => { + const stale = beginCapture(db, "container", "pull"); + beginCapture(db, "container", "pull"); + + const promoted = fixTarget(db, "container", "pull", stale, { + target: { rev: 99, path: null }, + mode: "entries", + }); + + expect(promoted).toBe(false); + expect(readOperation(db, "container", "pull")?.status).toBe("capturing"); + }); + }); + }); + + describe("joining an existing operation", () => { + it("openOperation returns the pending operation instead of a new target", async () => { + await withDB(async (db) => { + const generation = beginCapture(db, "container", "pull"); + fixTarget(db, "container", "pull", generation, { + target: { rev: 42, path: null }, + mode: "entries", + }); + + const joined = openOperation(db, "container", "pull"); + + expect(joined.joined).toBe(true); + expect(joined.operation.generation).toBe(generation); + expect(joined.operation.target).toEqual({ rev: 42, path: null }); + }); + }); + + it("openOperation starts a capture when no operation exists", async () => { + await withDB(async (db) => { + const opened = openOperation(db, "container", "pull"); + + expect(opened.joined).toBe(false); + expect(opened.operation.status).toBe("capturing"); + }); + }); + + it("openOperation replaces a failed operation with a new generation", async () => { + await withDB(async (db) => { + const first = beginCapture(db, "container", "pull"); + fixTarget(db, "container", "pull", first, { + target: { rev: 5, path: null }, + mode: "entries", + }); + failOperation(db, "container", "pull", first, "failed", "transport exploded"); + + const opened = openOperation(db, "container", "pull"); + + expect(opened.joined).toBe(false); + expect(opened.operation.generation).not.toBe(first); + expect(opened.operation.status).toBe("capturing"); + }); + }); + }); + + describe("completion and failure", () => { + it("completeOperation deletes the row so the next call captures a new target", async () => { + await withDB(async (db) => { + const generation = beginCapture(db, "container", "pull"); + fixTarget(db, "container", "pull", generation, { + target: { rev: 5, path: null }, + mode: "entries", + }); + + expect(completeOperation(db, "container", "pull", generation)).toBe(true); + expect(readOperation(db, "container", "pull")).toBeUndefined(); + }); + }); + + it("ignores a completion from a superseded generation", async () => { + await withDB(async (db) => { + const stale = beginCapture(db, "container", "pull"); + fixTarget(db, "container", "pull", stale, { + target: { rev: 5, path: null }, + mode: "entries", + }); + failOperation(db, "container", "pull", stale, "failed", "boom"); + const replacement = openOperation(db, "container", "pull").operation.generation; + + expect(completeOperation(db, "container", "pull", stale)).toBe(false); + expect(readOperation(db, "container", "pull")?.generation).toBe(replacement); + }); + }); + + it("retains a lost operation with its error for diagnostics", async () => { + await withDB(async (db) => { + const generation = beginCapture(db, "container", "pull"); + fixTarget(db, "container", "pull", generation, { + target: { rev: 5, path: null }, + mode: "entries", + }); + + failOperation(db, "container", "pull", generation, "lost", "EEXEC_LOST"); + + const operation = readOperation(db, "container", "pull"); + expect(operation?.status).toBe("lost"); + expect(operation?.lastError).toBe("EEXEC_LOST"); + }); + }); + + it("ignores a failure reported against a superseded generation", async () => { + await withDB(async (db) => { + const stale = beginCapture(db, "container", "pull"); + beginCapture(db, "container", "pull"); + + expect(failOperation(db, "container", "pull", stale, "failed", "late")).toBe(false); + expect(readOperation(db, "container", "pull")?.status).toBe("capturing"); + }); + }); + }); + + describe("block markers and sizing profile", () => { + it("starts an operation at the default block profile", async () => { + await withDB(async (db) => { + const generation = beginCapture(db, "container", "pull"); + fixTarget(db, "container", "pull", generation, { + target: { rev: 5, path: null }, + mode: "pack", + }); + + expect(readOperation(db, "container", "pull")?.profile).toEqual(DEFAULT_BLOCK_PROFILE); + }); + }); + + it("records the cursor a block started from", async () => { + await withDB(async (db) => { + const generation = beginCapture(db, "container", "pull"); + fixTarget(db, "container", "pull", generation, { + target: { rev: 5, path: null }, + mode: "pack", + }); + + markBlockStarted(db, "container", "pull", generation, { rev: 2, path: "/a" }, 1000); + + const operation = readOperation(db, "container", "pull"); + expect(operation?.blockAfter).toEqual({ rev: 2, path: "/a" }); + expect(operation?.blockStartedAt).toBe(1000); + }); + }); + + it("clears the block marker after the cursor advances", async () => { + await withDB(async (db) => { + const generation = beginCapture(db, "container", "pull"); + fixTarget(db, "container", "pull", generation, { + target: { rev: 5, path: null }, + mode: "pack", + }); + markBlockStarted(db, "container", "pull", generation, { rev: 2, path: null }, 1000); + + clearBlockMarker(db, "container", "pull", generation); + + expect(readOperation(db, "container", "pull")?.blockAfter).toBeUndefined(); + }); + }); + + it("halves the profile when a block is detected as interrupted", async () => { + await withDB(async (db) => { + const generation = beginCapture(db, "container", "pull"); + fixTarget(db, "container", "pull", generation, { + target: { rev: 5, path: null }, + mode: "pack", + }); + + shrinkBlockProfile(db, "container", "pull", generation); + + expect(readOperation(db, "container", "pull")?.profile).toEqual({ + maxEntries: DEFAULT_BLOCK_PROFILE.maxEntries / 2, + maxBytes: DEFAULT_BLOCK_PROFILE.maxBytes / 2, + }); + }); + }); + + it("stops shrinking at the minimum profile instead of reaching zero", async () => { + await withDB(async (db) => { + const generation = beginCapture(db, "container", "pull"); + fixTarget(db, "container", "pull", generation, { + target: { rev: 5, path: null }, + mode: "pack", + }); + + for (let i = 0; i < 20; i++) { + shrinkBlockProfile(db, "container", "pull", generation); + } + + expect(readOperation(db, "container", "pull")?.profile).toEqual(MIN_BLOCK_PROFILE); + }); + }); + + // The plan proposed halving on interruption but never restored the + // profile, so one eviction storm would leave a workspace slow + // forever. Completion resets it. See docs/decisions/sync-operations.md. + it("restores the default profile for the operation after a shrunk one completes", async () => { + await withDB(async (db) => { + const first = beginCapture(db, "container", "pull"); + fixTarget(db, "container", "pull", first, { + target: { rev: 5, path: null }, + mode: "pack", + }); + shrinkBlockProfile(db, "container", "pull", first); + completeOperation(db, "container", "pull", first); + + const second = beginCapture(db, "container", "pull"); + fixTarget(db, "container", "pull", second, { + target: { rev: 9, path: null }, + mode: "pack", + }); + + expect(readOperation(db, "container", "pull")?.profile).toEqual(DEFAULT_BLOCK_PROFILE); + }); + }); + }); + + describe("runtime identity", () => { + it("carries the source runtime id so a replaced container can be fenced", async () => { + await withDB(async (db) => { + const generation = beginCapture(db, "container", "pull"); + fixTarget(db, "container", "pull", generation, { + target: { rev: 5, path: null }, + mode: "entries", + runtimeId: "runtime-a", + }); + + expect(readOperation(db, "container", "pull")?.runtimeId).toBe("runtime-a"); + }); + }); + }); + + describe("skipped entries", () => { + // A rejection that lives only in a yielded progress value is + // unauditable once the caller discards it, so rejections are + // durable. See docs/decisions/sync-operations.md. + it("records a rejected read-only entry against the operation generation", async () => { + await withDB(async (db) => { + const generation = beginCapture(db, "container", "pull"); + fixTarget(db, "container", "pull", generation, { + target: { rev: 5, path: null }, + mode: "entries", + }); + + recordSkip(db, "container", "pull", generation, "/ro/file.txt", "EROFS", 2000); + + expect(readSkips(db, "container", "pull")).toEqual([ + { + generation, + path: "/ro/file.txt", + reason: "EROFS", + at: 2000, + }, + ]); + }); + }); + + it("keeps skip rows after the operation row is deleted", async () => { + await withDB(async (db) => { + const generation = beginCapture(db, "container", "pull"); + fixTarget(db, "container", "pull", generation, { + target: { rev: 5, path: null }, + mode: "entries", + }); + recordSkip(db, "container", "pull", generation, "/ro/file.txt", "EROFS", 2000); + + completeOperation(db, "container", "pull", generation); + + expect(readSkips(db, "container", "pull")).toHaveLength(1); + }); + }); + + it("prunes skips from earlier generations when a new operation starts", async () => { + await withDB(async (db) => { + const first = beginCapture(db, "container", "pull"); + fixTarget(db, "container", "pull", first, { + target: { rev: 5, path: null }, + mode: "entries", + }); + recordSkip(db, "container", "pull", first, "/ro/old.txt", "EROFS", 1000); + completeOperation(db, "container", "pull", first); + + const second = beginCapture(db, "container", "pull"); + pruneSkips(db, "container", "pull", second); + + expect(readSkips(db, "container", "pull")).toEqual([]); + }); + }); + + it("does not leak skips across backends", async () => { + await withDB(async (db) => { + const generation = beginCapture(db, "container", "pull"); + recordSkip(db, "container", "pull", generation, "/ro/file.txt", "EROFS", 2000); + + expect(readSkips(db, "worker", "pull")).toEqual([]); + }); + }); + }); +}); diff --git a/packages/dofs/src/sync/operations.ts b/packages/dofs/src/sync/operations.ts new file mode 100644 index 00000000..36ec02f1 --- /dev/null +++ b/packages/dofs/src/sync/operations.ts @@ -0,0 +1,490 @@ +// Durable state for a restartable sync operation. +// +// A `pull()` or `push()` iterable is driven by a JavaScript async +// iterator, and a JavaScript iterator cannot survive Durable Object +// eviction. This module holds everything the iterator would otherwise +// keep in memory: the fixed target it is working toward, the +// generation that fences stale executions, the block sizing profile, +// and a marker for the block currently in flight. +// +// The committed progress cursor is not here. That stays in +// `_vfs_watermark` (see watermarks.ts) because filesystem application +// and cursor advancement must commit in the same storage transaction. +// This table records where an operation is *going*; the watermark +// records how far it has *got*. +// +// Every mutation is conditional on the generation. That is what makes +// a late write from a superseded execution a no-op rather than a +// corruption: the `WHERE generation = ?` clause fails and the caller +// learns it lost the race from the returned boolean. + +import type { Database } from "../storage.js"; +import { type ChangeCursor, DEFAULT_BACKEND_ID } from "./watermarks.js"; + +export type SyncDirection = "pull" | "push"; +export type SyncMode = "entries" | "pack"; +export type SyncOperationStatus = "capturing" | "pending" | "failed" | "lost"; + +// Terminal statuses. A row in one of these states is diagnostic +// history: it no longer describes work in progress, and the next +// caller replaces it with a new generation. +export type TerminalStatus = "failed" | "lost"; + +// Internal block sizing. Not part of any public API — the plan is +// explicit that callers never supply budgets. The profile is persisted +// per operation so an in-flight operation keeps the sizing it started +// with even if the constants change under it. +export interface BlockProfile { + readonly maxEntries: number; + readonly maxBytes: number; +} + +// Measured rather than guessed. The plan proposed 4,000 entries, which +// small trees make look safe: a few hundred entries per block finishes +// in well under a second. At install scale it is not safe. On a +// 44,100-file tree a 4,000-entry block took 15.8s of the default +// 30s allowance — 1.9x headroom, where an unlucky block or a slower +// machine exceeds the limit and the block can never complete. +// +// A sweep at install scale showed the cost is superlinear in block size +// while total time is flat, because the win from fewer round trips runs +// out well before the per-block cost does: +// +// entries blocks total worst block headroom +// 500 49 36.8s 2.8s 10.8x +// 1000 25 29.9s 3.6s 8.4x +// 2000 13 30.5s 3.7s 8.1x +// 4000 7 31.0s 8.1s 3.7x +// +// 2,000 keeps essentially the same total time as 4,000 while more than +// doubling the headroom, so that is the default. Blocks are a +// checkpointing mechanism, not a throughput knob: making them larger +// buys almost nothing and costs exactly the property they exist for. +export const DEFAULT_BLOCK_PROFILE: BlockProfile = { + maxEntries: 2_000, + maxBytes: 64 * 1024 * 1024, +}; + +// Floor for automatic shrinking. Halving without a floor walks to +// zero and an operation that can carry no entries makes no progress. +export const MIN_BLOCK_PROFILE: BlockProfile = { + maxEntries: 500, + maxBytes: 8 * 1024 * 1024, +}; + +export interface SyncOperation { + readonly backend: string; + readonly direction: SyncDirection; + readonly generation: string; + readonly status: SyncOperationStatus; + // Absent exactly while status is 'capturing'. + readonly target?: ChangeCursor; + readonly mode?: SyncMode; + readonly runtimeId?: string; + // Where the in-flight block started. Present means a block was + // opened and has not yet advanced the cursor. + readonly blockAfter?: ChangeCursor; + readonly blockStartedAt?: number; + readonly profile: BlockProfile; + readonly createdAt: number; + readonly updatedAt: number; + readonly lastError?: string; +} + +export interface SyncSkip { + readonly generation: string; + readonly path: string; + readonly reason: string; + readonly at: number; +} + +interface OperationRow { + backend: string; + direction: SyncDirection; + generation: string; + status: SyncOperationStatus; + target_rev: number | null; + target_path: string | null; + runtime_id: string | null; + mode: SyncMode | null; + block_after_rev: number | null; + block_after_path: string | null; + block_started_at: number | null; + internal_max_entries: number; + internal_max_bytes: number; + created_at: number; + updated_at: number; + last_error: string | null; +} + +// A target of `{rev, path: null}` means the rev is fully drained; +// a string path (including the empty string, which is a real path +// value) means resume inside that rev. Both survive the round trip +// because rev and path are stored in separate columns — encoding them +// into one string would make the empty path ambiguous. +function rowToOperation(row: OperationRow): SyncOperation { + const operation: { + -readonly [K in keyof SyncOperation]: SyncOperation[K]; + } = { + backend: row.backend, + direction: row.direction, + generation: row.generation, + status: row.status, + profile: { + maxEntries: row.internal_max_entries, + maxBytes: row.internal_max_bytes, + }, + createdAt: row.created_at, + updatedAt: row.updated_at, + }; + if (row.target_rev !== null) { + operation.target = { rev: row.target_rev, path: row.target_path }; + } + if (row.mode !== null) operation.mode = row.mode; + if (row.runtime_id !== null) operation.runtimeId = row.runtime_id; + if (row.block_after_rev !== null) { + operation.blockAfter = { rev: row.block_after_rev, path: row.block_after_path }; + } + if (row.block_started_at !== null) operation.blockStartedAt = row.block_started_at; + if (row.last_error !== null) operation.lastError = row.last_error; + return operation; +} + +export function readOperation( + db: Database, + backend: string = DEFAULT_BACKEND_ID, + direction: SyncDirection = "pull", +): SyncOperation | undefined { + const row = db.one( + "SELECT * FROM _vfs_sync_operations WHERE backend = ? AND direction = ?", + backend, + direction, + ); + return row === undefined ? undefined : rowToOperation(row); +} + +// Generations only need to be unique per (backend, direction) and +// unguessable enough that a stale execution cannot collide with its +// replacement. crypto.randomUUID is available in workerd and Node. +function newGeneration(): string { + return crypto.randomUUID(); +} + +// Phase one of target capture: claim the slot before talking to the +// remote. Overwrites any existing row, including a 'capturing' row +// left behind by an evicted iterator and a terminal 'failed' or 'lost' +// row, because in every one of those cases the prior execution no +// longer owns the operation. Taking a fresh generation is what +// invalidates it. +export function beginCapture( + db: Database, + backend: string = DEFAULT_BACKEND_ID, + direction: SyncDirection = "pull", + now: number = Date.now(), +): string { + const generation = newGeneration(); + db.run( + `INSERT INTO _vfs_sync_operations ( + backend, direction, generation, status, + internal_max_entries, internal_max_bytes, + created_at, updated_at + ) VALUES (?, ?, ?, 'capturing', ?, ?, ?, ?) + ON CONFLICT(backend, direction) DO UPDATE SET + generation = excluded.generation, + status = 'capturing', + target_rev = NULL, + target_path = NULL, + runtime_id = NULL, + mode = NULL, + block_after_rev = NULL, + block_after_path = NULL, + block_started_at = NULL, + internal_max_entries = excluded.internal_max_entries, + internal_max_bytes = excluded.internal_max_bytes, + updated_at = excluded.updated_at, + last_error = NULL`, + backend, + direction, + generation, + DEFAULT_BLOCK_PROFILE.maxEntries, + DEFAULT_BLOCK_PROFILE.maxBytes, + now, + now, + ); + return generation; +} + +// Phase two: the remote target has been read, so freeze it. Returns +// false when this generation no longer owns the operation, which is +// the case an evicted-and-restarted capture must handle — the target +// it captured is discarded rather than overwriting the replacement's. +export function fixTarget( + db: Database, + backend: string, + direction: SyncDirection, + generation: string, + input: { target: ChangeCursor; mode: SyncMode; runtimeId?: string }, + now: number = Date.now(), +): boolean { + db.run( + `UPDATE _vfs_sync_operations + SET status = 'pending', + target_rev = ?, + target_path = ?, + mode = ?, + runtime_id = ?, + updated_at = ? + WHERE backend = ? AND direction = ? AND generation = ? AND status = 'capturing'`, + input.target.rev, + input.target.path, + input.mode, + input.runtimeId ?? null, + now, + backend, + direction, + generation, + ); + const current = readOperation(db, backend, direction); + return current?.generation === generation && current.status === "pending"; +} + +export interface OpenedOperation { + readonly operation: SyncOperation; + // True when this caller attached to an operation someone else + // created. A joining caller must not capture a new target; the + // target is already fixed and shared. + readonly joined: boolean; +} + +// Read-or-create in one step. A pending operation is joined so +// concurrent callers converge on one target instead of racing to +// create competing ones. A terminal row is replaced. +// +// A 'capturing' row is *not* joined: it has no target, so there is +// nothing to share, and the execution that created it may be gone. +// Taking it over with a new generation is the recovery path for an +// eviction during capture. +export function openOperation( + db: Database, + backend: string = DEFAULT_BACKEND_ID, + direction: SyncDirection = "pull", + now: number = Date.now(), +): OpenedOperation { + return db.transactionSync(() => { + const existing = readOperation(db, backend, direction); + if (existing !== undefined && existing.status === "pending") { + return { operation: existing, joined: true }; + } + const generation = beginCapture(db, backend, direction, now); + const operation = readOperation(db, backend, direction); + if (operation === undefined) { + throw new Error("dofs sync: operation row vanished immediately after capture"); + } + if (operation.generation !== generation) { + throw new Error("dofs sync: operation generation changed inside its own transaction"); + } + return { operation, joined: false }; + }); +} + +// Completion deletes the row. The backend watermark is the durable +// record that the work happened, so keeping a completed operation +// would duplicate it — and a stale 'complete' row is exactly what +// would make the next caller think there is nothing to do. +// +// Deleting also resets the block profile, because the next operation's +// beginCapture writes DEFAULT_BLOCK_PROFILE. That is deliberate: a +// profile shrunk by one bad block must not follow a workspace forever. +export function completeOperation( + db: Database, + backend: string, + direction: SyncDirection, + generation: string, +): boolean { + const existing = readOperation(db, backend, direction); + if (existing?.generation !== generation) return false; + db.run( + "DELETE FROM _vfs_sync_operations WHERE backend = ? AND direction = ? AND generation = ?", + backend, + direction, + generation, + ); + return true; +} + +// Terminal failure. The row is retained with its error so an operator +// can see why a backend stopped converging; the next caller replaces +// it. `lost` specifically means the runtime the operation was fenced +// to no longer exists, which is not retryable against the same target. +export function failOperation( + db: Database, + backend: string, + direction: SyncDirection, + generation: string, + status: TerminalStatus, + error: string, + now: number = Date.now(), +): boolean { + const existing = readOperation(db, backend, direction); + if (existing?.generation !== generation) return false; + db.run( + `UPDATE _vfs_sync_operations + SET status = ?, last_error = ?, updated_at = ? + WHERE backend = ? AND direction = ? AND generation = ?`, + status, + error, + now, + backend, + direction, + generation, + ); + return true; +} + +// Persisted before a block stream opens. If a later invocation finds +// this marker still set and the watermark has not moved, the previous +// execution died mid-block and the profile should shrink. +export function markBlockStarted( + db: Database, + backend: string, + direction: SyncDirection, + generation: string, + after: ChangeCursor, + now: number = Date.now(), +): boolean { + const existing = readOperation(db, backend, direction); + if (existing?.generation !== generation) return false; + db.run( + `UPDATE _vfs_sync_operations + SET block_after_rev = ?, block_after_path = ?, block_started_at = ?, updated_at = ? + WHERE backend = ? AND direction = ? AND generation = ?`, + after.rev, + after.path, + now, + now, + backend, + direction, + generation, + ); + return true; +} + +// Cleared only after the cursor has advanced past the block. Clearing +// earlier would lose the interruption signal. +export function clearBlockMarker( + db: Database, + backend: string, + direction: SyncDirection, + generation: string, + now: number = Date.now(), +): boolean { + const existing = readOperation(db, backend, direction); + if (existing?.generation !== generation) return false; + db.run( + `UPDATE _vfs_sync_operations + SET block_after_rev = NULL, block_after_path = NULL, + block_started_at = NULL, updated_at = ? + WHERE backend = ? AND direction = ? AND generation = ?`, + now, + backend, + direction, + generation, + ); + return true; +} + +// Halve the profile, floored at MIN_BLOCK_PROFILE. Called when a block +// marker survived without cursor progress, meaning the last attempt at +// this size did not fit in the available CPU. +export function shrinkBlockProfile( + db: Database, + backend: string, + direction: SyncDirection, + generation: string, + now: number = Date.now(), +): BlockProfile | undefined { + const existing = readOperation(db, backend, direction); + if (existing?.generation !== generation) return undefined; + const next: BlockProfile = { + maxEntries: Math.max(MIN_BLOCK_PROFILE.maxEntries, Math.floor(existing.profile.maxEntries / 2)), + maxBytes: Math.max(MIN_BLOCK_PROFILE.maxBytes, Math.floor(existing.profile.maxBytes / 2)), + }; + db.run( + `UPDATE _vfs_sync_operations + SET internal_max_entries = ?, internal_max_bytes = ?, updated_at = ? + WHERE backend = ? AND direction = ? AND generation = ?`, + next.maxEntries, + next.maxBytes, + now, + backend, + direction, + generation, + ); + return next; +} + +// Durable record of an entry the receiver refused. Keyed by generation +// so a caller can tell which operation dropped what, and idempotent on +// replay because the same path rejected twice in one generation is one +// fact, not two. +export function recordSkip( + db: Database, + backend: string, + direction: SyncDirection, + generation: string, + path: string, + reason: string, + now: number = Date.now(), +): void { + db.run( + `INSERT INTO _vfs_sync_skips (backend, direction, generation, path, reason, at) + VALUES (?, ?, ?, ?, ?, ?) + ON CONFLICT(backend, direction, generation, path) DO UPDATE SET + reason = excluded.reason, at = excluded.at`, + backend, + direction, + generation, + path, + reason, + now, + ); +} + +export function readSkips( + db: Database, + backend: string = DEFAULT_BACKEND_ID, + direction: SyncDirection = "pull", +): SyncSkip[] { + return db + .all<{ generation: string; path: string; reason: string; at: number }>( + `SELECT generation, path, reason, at + FROM _vfs_sync_skips + WHERE backend = ? AND direction = ? + ORDER BY at, path`, + backend, + direction, + ) + .map((row) => ({ + generation: row.generation, + path: row.path, + reason: row.reason, + at: row.at, + })); +} + +// Drop skips that belong to any generation other than the current one. +// Called when a new operation starts so the log describes the latest +// attempt rather than growing without bound. +export function pruneSkips( + db: Database, + backend: string, + direction: SyncDirection, + generation: string, +): void { + db.run( + "DELETE FROM _vfs_sync_skips WHERE backend = ? AND direction = ? AND generation != ?", + backend, + direction, + generation, + ); +} diff --git a/packages/dofs/src/sync/replay.test.ts b/packages/dofs/src/sync/replay.test.ts new file mode 100644 index 00000000..d821224e --- /dev/null +++ b/packages/dofs/src/sync/replay.test.ts @@ -0,0 +1,144 @@ +import { describe, expect, it } from "vitest"; + +import { readFile } from "../fs/readFile.js"; +import { resolveInode } from "../fs/resolve.js"; +import { withDB } from "../fs/with-db.js"; +import { writeFile } from "../fs/writeFile.js"; +import { applyChanges } from "./apply.js"; +import type { ChangeEntry } from "./changes.js"; + +// The unified sync plan asserts that replaying an unacknowledged block +// is safe because "revision and cursor semantics already make replay +// idempotent". That assertion is load-bearing: the cursor only +// advances after a block applies, so any block interrupted before its +// acknowledgment is re-applied on the next iterator. These tests +// enumerate what idempotent actually means per entry kind instead of +// taking it on faith. +// +// The dangerous case is a tombstone. A file entry replays to identical +// content, but a delete replayed after the path was locally recreated +// would destroy data that the tombstone never described. + +describe("block replay idempotency", () => { + describe("write entries", () => { + it("applies a file entry twice with the same result", async () => { + await withDB(async (db) => { + const entry: ChangeEntry = { + kind: "dir", + rev: 5, + path: "/dir", + mode: 0o755, + mtime: 1000, + }; + + const first = await applyChanges(db, [entry], new Map(), { source: "upstream" }); + const second = await applyChanges(db, [entry], new Map(), { source: "upstream" }); + + expect(first.applied).toBe(1); + // The second pass recognises the entry as already in place and + // does no work, which is what stops a replayed block from + // bumping rev and re-triggering a push. + expect(second.applied).toBe(0); + expect(resolveInode(db, "/dir", { followSymlinks: false })?.type).toBe("dir"); + }); + }); + + it("applies a symlink entry twice with the same result", async () => { + await withDB(async (db) => { + const entry: ChangeEntry = { + kind: "symlink", + rev: 5, + path: "/link", + target: "/target", + mode: 0o777, + mtime: 1000, + }; + + await applyChanges(db, [entry], new Map(), { source: "upstream" }); + const second = await applyChanges(db, [entry], new Map(), { source: "upstream" }); + + expect(second.applied).toBe(0); + expect(resolveInode(db, "/link", { followSymlinks: false })?.linkTarget).toBe("/target"); + }); + }); + }); + + describe("delete entries", () => { + it("applies a tombstone twice without error when the path stays gone", async () => { + await withDB(async (db) => { + await writeFile(db, "/gone.txt", "bye", {}, () => 1); + const entry: ChangeEntry = { kind: "delete", rev: 5, path: "/gone.txt" }; + + await applyChanges(db, [entry], new Map(), { source: "upstream" }); + const second = await applyChanges(db, [entry], new Map(), { source: "upstream" }); + + expect(second.skipped).toEqual([]); + expect(resolveInode(db, "/gone.txt", { followSymlinks: false })).toBeNull(); + }); + }); + + // The plan's idempotency claim breaks here. The tombstone was + // produced at rev 5 and describes the file as it was then. If the + // block carrying it is interrupted before acknowledgment and the + // path is recreated locally in the meantime, replaying the + // tombstone deletes content it never described. + it("does not delete a path recreated at a newer revision than the tombstone", async () => { + await withDB(async (db) => { + await writeFile(db, "/data.txt", "original", {}, () => 1); + // The tombstone's rev is whatever the source stamped it with. + // What matters is that the local recreation lands above it. + const tombstone: ChangeEntry = { kind: "delete", rev: 2, path: "/data.txt" }; + + // The block applied the tombstone but died before it could + // acknowledge its cursor. + await applyChanges(db, [tombstone], new Map(), { source: "upstream" }); + expect(resolveInode(db, "/data.txt", { followSymlinks: false })).toBeNull(); + + // A local write recreates the path at a revision above the + // tombstone's. + await writeFile(db, "/data.txt", "recreated", {}, () => 2); + const liveRev = + db.one<{ rev: number }>( + "SELECT rev FROM vfs_nodes WHERE inode = ?", + resolveInode(db, "/data.txt", { followSymlinks: false })?.inode ?? 0, + )?.rev ?? 0; + expect(liveRev).toBeGreaterThan(tombstone.rev); + + // The interrupted block is replayed from the durable cursor. + const replay = await applyChanges(db, [tombstone], new Map(), { source: "upstream" }); + + expect(resolveInode(db, "/data.txt", { followSymlinks: false })).not.toBeNull(); + expect(await readFile(db, "/data.txt", "utf8")).toBe("recreated"); + expect(replay.applied).toBe(0); + }); + }); + + it("still deletes a path whose live revision predates the tombstone", async () => { + await withDB(async (db) => { + await writeFile(db, "/stale.txt", "stale", {}, () => 1); + // A tombstone from far above the live rev is a genuine delete + // the receiver has not seen yet. + const tombstone: ChangeEntry = { kind: "delete", rev: 9_999, path: "/stale.txt" }; + + const result = await applyChanges(db, [tombstone], new Map(), { source: "upstream" }); + + expect(result.applied).toBe(1); + expect(resolveInode(db, "/stale.txt", { followSymlinks: false })).toBeNull(); + }); + }); + + // A locally-authored delete is not a replay of remote state, so + // the revision guard must not apply to it. + it("applies a local tombstone regardless of the live revision", async () => { + await withDB(async (db) => { + await writeFile(db, "/local.txt", "content", {}, () => 1); + const tombstone: ChangeEntry = { kind: "delete", rev: 1, path: "/local.txt" }; + + const result = await applyChanges(db, [tombstone], new Map(), { source: "local" }); + + expect(result.applied).toBe(1); + expect(resolveInode(db, "/local.txt", { followSymlinks: false })).toBeNull(); + }); + }); + }); +}); diff --git a/packages/rpc/package.json b/packages/rpc/package.json index 51242be8..6ddae704 100644 --- a/packages/rpc/package.json +++ b/packages/rpc/package.json @@ -31,7 +31,8 @@ "build": "tsc -p tsconfig.build.json", "pretypecheck": "npm run build --workspace @cloudflare/dofs", "typecheck": "tsc -p tsconfig.build.json --noEmit", - "test": "vitest run" + "test": "vitest run", + "bench": "vitest run --config vitest.config.bench.ts" }, "dependencies": { "@cloudflare/dofs": "*", diff --git a/packages/rpc/src/bench/sync-engine.bench.ts b/packages/rpc/src/bench/sync-engine.bench.ts new file mode 100644 index 00000000..79eaf883 --- /dev/null +++ b/packages/rpc/src/bench/sync-engine.bench.ts @@ -0,0 +1,411 @@ +// End-to-end restartable sync benchmark. +// +// The dofs harness measures the pieces (planning, pack codec, apply). +// This one measures the thing callers actually experience: driving +// `pullBlocks` to completion against a real peer, in both transport +// modes, and counting what it cost. +// +// Runs under @cloudflare/vitest-pool-workers so both peers are real +// Durable Object SqlStorage instances. Cross-DO I/O isolation means the +// two peers cannot be live in one isolate, so the "remote" is a plain +// in-isolate Database driving createSyncServer while the local side is +// the DO — the transport is a direct stub either way, which is what we +// want: this isolates dofs + engine cost from capnweb and FUSE framing. +// +// The headline numbers are wall time to converge and the worst single +// block, because the worst block is what has to fit inside a Durable +// Object's CPU allowance. +// +// Run with: npm run bench --workspace @cloudflare/computer-rpc + +import { + Database, + initializeSchema, + readFetchCursor, + SQLiteWorkspaceProvider, +} from "@cloudflare/dofs"; +import { SQLiteTestStorage } from "@cloudflare/dofs/testing"; +import { it } from "vitest"; + +import { createSyncServer } from "../server.js"; +import { pullBlocks } from "../sync-engine.js"; + +const NOW = (): number => 1000; + +function makePeer(): { db: Database; close: () => void } { + const storage = new SQLiteTestStorage(); + const db = new Database(storage); + initializeSchema(db, NOW); + return { db, close: () => storage.close() }; +} + +// Package-shaped source tree: nested directories, many small modules, +// a minority of genuinely shared payloads. +function seedPackageTree( + db: Database, + options: { packages: number; filesPerPackage: number; fileBytes: number }, +): { files: number; logicalBytes: number } { + const provider = new SQLiteWorkspaceProvider(db, { now: NOW }); + let files = 0; + let logicalBytes = 0; + const body = (p: number, f: number): string => { + const shared = (p + f) % 8 === 0; + const seed = shared ? "shared-helper" : `pkg-${p}-lib-${f}`; + const filler = `${seed} module body with require() calls and comments; `; + return filler.repeat(Math.ceil(options.fileBytes / filler.length)).slice(0, options.fileBytes); + }; + for (let p = 0; p < options.packages; p++) { + const dir = `/node_modules/pkg-${String(p).padStart(4, "0")}`; + provider.mkdirSync(dir, { recursive: true }); + provider.writeFileSync(`${dir}/package.json`, `{"name":"pkg-${p}","version":"1.0.0"}`); + files++; + logicalBytes += 40; + for (let f = 0; f < options.filesPerPackage; f++) { + const content = body(p, f); + provider.writeFileSync(`${dir}/lib-${f}.js`, content); + files++; + logicalBytes += content.length; + } + } + return { files, logicalBytes }; +} + +function table(rows: Record[]): void { + if (rows.length === 0) return; + const keys = Object.keys(rows[0]); + const width = keys.map((k) => Math.max(k.length, ...rows.map((r) => String(r[k] ?? "").length))); + const line = (cells: (string | number)[]): string => + cells.map((c, i) => String(c).padEnd(width[i])).join(" "); + console.log(line(keys)); + console.log(width.map((w) => "-".repeat(w)).join(" ")); + for (const row of rows) console.log(line(keys.map((k) => row[k] ?? ""))); +} + +interface RunResult { + blocks: number; + entries: number; + totalMs: number; + worstBlockMs: number; + mode: string; +} + +// Drive a pull to completion one block at a time, timing each block. +async function runPull( + local: Database, + remoteDb: Database, + options: { forcePack: boolean; maxEntries: number }, +): Promise { + const rpc = createSyncServer(remoteDb); + const iterableOptions = { + profile: { maxEntries: options.maxEntries, maxBytes: 64 * 1024 * 1024 }, + // thresholdEntries of 1 forces pack; a huge one forces entry mode. + thresholdEntries: options.forcePack ? 1 : Number.MAX_SAFE_INTEGER, + thresholdBytes: options.forcePack ? 1 : Number.MAX_SAFE_INTEGER, + }; + + let blocks = 0; + let entries = 0; + let worstBlockMs = 0; + let mode = "?"; + const started = performance.now(); + + // A fresh iterable per block, which is the eviction-worst-case shape: + // every block re-reads the durable operation and cursor instead of + // reusing an in-memory iterator. + for (let guard = 0; guard < 100_000; guard++) { + const tBlock = performance.now(); + const iterator = pullBlocks(local, rpc, iterableOptions)[Symbol.asyncIterator](); + const { value, done } = await iterator.next(); + const blockMs = performance.now() - tBlock; + if (done) break; + blocks++; + entries += value.entries; + mode = value.mode; + worstBlockMs = Math.max(worstBlockMs, blockMs); + if (value.complete) break; + } + + return { + blocks, + entries, + totalMs: Number((performance.now() - started).toFixed(1)), + worstBlockMs: Number(worstBlockMs.toFixed(1)), + mode, + }; +} + +it("entry mode versus pack mode, converging a package-shaped tree", async () => { + const rows: Record[] = []; + const json: Record[] = []; + + const shapes = [ + { label: "small (incremental)", packages: 5, filesPerPackage: 4, fileBytes: 512 }, + { label: "medium", packages: 60, filesPerPackage: 8, fileBytes: 2048 }, + { label: "large (install-shaped)", packages: 200, filesPerPackage: 8, fileBytes: 4096 }, + ]; + + for (const shape of shapes) { + // Seed one source tree and reuse its bytes for both modes so the + // comparison is apples to apples. + const source = makePeer(); + const seeded = seedPackageTree(source.db, shape); + + for (const forcePack of [false, true]) { + const local = makePeer(); + try { + const result = await runPull(local.db, source.db, { forcePack, maxEntries: 512 }); + const converged = readFetchCursor(local.db).rev > 0; + const localFiles = + local.db.one<{ c: number }>("SELECT COUNT(*) AS c FROM vfs_nodes WHERE type = 'file'") + ?.c ?? 0; + rows.push({ + shape: shape.label, + mode: result.mode, + "source files": seeded.files, + "logical MB": Number((seeded.logicalBytes / 1024 / 1024).toFixed(2)), + entries: result.entries, + blocks: result.blocks, + "total (ms)": result.totalMs, + "worst block (ms)": result.worstBlockMs, + "ms/entry": Number((result.totalMs / Math.max(1, result.entries)).toFixed(3)), + "files applied": localFiles, + converged: converged ? "yes" : "no", + }); + json.push({ shape: shape.label, ...result, sourceFiles: seeded.files }); + } finally { + local.close(); + } + } + source.close(); + } + + console.log("\n=== end-to-end pull: entry vs pack ==="); + table(rows); + console.log(`BENCH_JSON e2e ${JSON.stringify(json)}`); +}); + +it("block size sweep: worst block versus round trips", async () => { + const rows: Record[] = []; + const source = makePeer(); + const seeded = seedPackageTree(source.db, { + packages: 150, + filesPerPackage: 8, + fileBytes: 4096, + }); + + for (const maxEntries of [64, 256, 1024]) { + for (const forcePack of [false, true]) { + const local = makePeer(); + try { + const result = await runPull(local.db, source.db, { forcePack, maxEntries }); + rows.push({ + "block max entries": maxEntries, + mode: result.mode, + blocks: result.blocks, + "total (ms)": result.totalMs, + "worst block (ms)": result.worstBlockMs, + // How many of the worst-case blocks fit in the default + // Durable Object CPU allowance. Under 1 would mean a block + // cannot finish, which is the failure this design exists to + // avoid. + "worst blocks per 30s": Math.floor(30_000 / Math.max(1, result.worstBlockMs)), + }); + } finally { + local.close(); + } + } + } + source.close(); + + console.log(`\n=== block size sweep (${seeded.files} source files) ===`); + table(rows); +}); + +// The in-process stub has no wire, so it charges nothing for bytes and +// everything for CPU. That is the worst possible case for a compressed +// transport and it is not the case the pack exists for. Model the wire +// explicitly: measure bytes moved in each mode, then price them at a +// few plausible link speeds. +it("bytes on the wire, priced against link speed", async () => { + const source = makePeer(); + const seeded = seedPackageTree(source.db, { + packages: 200, + filesPerPackage: 8, + fileBytes: 4096, + }); + + // Count bytes each mode would put on the wire for the same window, + // plus the CPU each spends locally. + const measure = async (forcePack: boolean): Promise<{ wireKB: number; cpuMs: number }> => { + const local = makePeer(); + const rpc = createSyncServer(source.db); + let wireBytes = 0; + + // Wrap the transport to count bytes actually streamed. + const counting = new Proxy(rpc as object, { + get(target, prop, receiver) { + const real = Reflect.get(target, prop, receiver); + if (prop === "fetchChangePack") { + return async (...args: unknown[]) => { + const res = (await ( + real as (...a: unknown[]) => Promise<{ + stream: ReadableStream; + }> + ).call(target, ...args)) as { + stream: ReadableStream; + }; + return { + ...res, + stream: res.stream.pipeThrough( + new TransformStream({ + transform(chunk, controller) { + wireBytes += chunk.byteLength; + controller.enqueue(chunk); + }, + }), + ), + }; + }; + } + if (prop === "fetchObjects") { + return (...args: unknown[]) => { + const stream = ( + real as (...a: unknown[]) => ReadableStream<{ + hash: Uint8Array; + bytes: Uint8Array; + }> + ).call(target, ...args); + return stream.pipeThrough( + new TransformStream< + { hash: Uint8Array; bytes: Uint8Array }, + { hash: Uint8Array; bytes: Uint8Array } + >({ + transform(chunk, controller) { + wireBytes += chunk.bytes.byteLength + 32; + controller.enqueue(chunk); + }, + }), + ); + }; + } + if (prop === "fetchChanges") { + return async (...args: unknown[]) => { + const res = (await ( + real as (...a: unknown[]) => Promise<{ + stream: ReadableStream; + }> + ).call(target, ...args)) as { stream: ReadableStream }; + return { + ...res, + stream: res.stream.pipeThrough( + new TransformStream({ + transform(chunk, controller) { + // Entry records cross the wire as structured + // values; JSON length is a fair proxy. + wireBytes += JSON.stringify(chunk).length; + controller.enqueue(chunk); + }, + }), + ), + }; + }; + } + return real; + }, + }) as typeof rpc; + + const started = performance.now(); + for (let guard = 0; guard < 100_000; guard++) { + const iterator = pullBlocks(local.db, counting, { + profile: { maxEntries: 512, maxBytes: 64 * 1024 * 1024 }, + thresholdEntries: forcePack ? 1 : Number.MAX_SAFE_INTEGER, + thresholdBytes: forcePack ? 1 : Number.MAX_SAFE_INTEGER, + })[Symbol.asyncIterator](); + const { value, done } = await iterator.next(); + if (done || value.complete) break; + } + const cpuMs = performance.now() - started; + local.close(); + return { wireKB: Number((wireBytes / 1024).toFixed(1)), cpuMs: Number(cpuMs.toFixed(1)) }; + }; + + const entry = await measure(false); + const pack = await measure(true); + source.close(); + + const rows: Record[] = [ + { mode: "entries", "wire KB": entry.wireKB, "local CPU (ms)": entry.cpuMs }, + { mode: "pack", "wire KB": pack.wireKB, "local CPU (ms)": pack.cpuMs }, + ]; + console.log(`\n=== bytes on the wire (${seeded.files} files) ===`); + table(rows); + + // Total time = CPU + bytes/bandwidth. The in-process benchmark above + // is the bandwidth = infinity column. + const priced: Record[] = []; + for (const mbps of [10, 50, 100, 500, Number.POSITIVE_INFINITY]) { + const bytesPerMs = (mbps * 1024 * 1024) / 8 / 1000; + const entryTotal = entry.cpuMs + (entry.wireKB * 1024) / bytesPerMs; + const packTotal = pack.cpuMs + (pack.wireKB * 1024) / bytesPerMs; + priced.push({ + "link Mbps": mbps === Number.POSITIVE_INFINITY ? "infinite" : mbps, + "entries total (ms)": Number(entryTotal.toFixed(0)), + "pack total (ms)": Number(packTotal.toFixed(0)), + winner: packTotal < entryTotal ? "pack" : "entries", + margin: `${Math.abs(((entryTotal - packTotal) / Math.max(1, entryTotal)) * 100).toFixed(0)}%`, + }); + } + console.log("\n=== priced against link speed ==="); + table(priced); + console.log(`BENCH_JSON wire ${JSON.stringify({ entry, pack, files: seeded.files })}`); +}); + +it("restart overhead: fresh iterable per block versus one iterator", async () => { + const rows: Record[] = []; + const source = makePeer(); + seedPackageTree(source.db, { packages: 120, filesPerPackage: 8, fileBytes: 2048 }); + + for (const forcePack of [false, true]) { + const options = { + profile: { maxEntries: 128, maxBytes: 64 * 1024 * 1024 }, + thresholdEntries: forcePack ? 1 : Number.MAX_SAFE_INTEGER, + thresholdBytes: forcePack ? 1 : Number.MAX_SAFE_INTEGER, + }; + + // (a) One long-lived iterator: the happy path, no eviction. + const reused = makePeer(); + const rpcA = createSyncServer(source.db); + const tReused = performance.now(); + let reusedBlocks = 0; + for await (const progress of pullBlocks(reused.db, rpcA, options)) { + reusedBlocks++; + if (progress.complete) break; + } + const reusedMs = Number((performance.now() - tReused).toFixed(1)); + const mode = reusedBlocks > 0 ? (forcePack ? "pack" : "entries") : "?"; + reused.close(); + + // (b) A fresh iterable for every block: what an evicted Durable + // Object costs, since each block re-reads the operation row and + // cursor from SQLite. + const restarted = makePeer(); + const result = await runPull(restarted.db, source.db, { + forcePack, + maxEntries: 128, + }); + restarted.close(); + + rows.push({ + mode, + "reused iterator (ms)": reusedMs, + "reused blocks": reusedBlocks, + "fresh per block (ms)": result.totalMs, + "fresh blocks": result.blocks, + "restart overhead": `${(((result.totalMs - reusedMs) / Math.max(1, reusedMs)) * 100).toFixed(0)}%`, + }); + } + source.close(); + + console.log("\n=== restart overhead ==="); + table(rows); +}); diff --git a/packages/rpc/src/bench/sync-scale.bench.ts b/packages/rpc/src/bench/sync-scale.bench.ts new file mode 100644 index 00000000..7462f5fb --- /dev/null +++ b/packages/rpc/src/bench/sync-scale.bench.ts @@ -0,0 +1,269 @@ +// Install-scale sync benchmark. +// +// The engine benchmark next door uses small trees so it can sweep many +// shapes quickly. This one targets the size the plan actually cites as +// its motivating case: a package-install tree of roughly 40,000 entries. +// The plan's reference run was 44,264 entries over an estimated +// 915 MB cursor window, sending 33,683 unique objects in a 257 MB gzip +// stream, and it wants that to stay materially faster than an unbounded +// entry pull. +// +// Seeding this tree costs real time (about 0.4 ms per file, so ~20 s at +// 45,000 files) and the sync itself is measured in seconds, so the +// scenarios here are deliberately few and the whole file is gated behind +// an env var: +// +// SYNC_SCALE=1 npm run bench --workspace @cloudflare/computer-rpc +// +// Without the flag the suite reports skipped, so the default bench run +// stays fast. +// +// Byte totals are scaled down from the plan's 915 MB by default because +// a 1 GB tree in one Durable Object SQLite database is a separate +// question from block sequencing; SYNC_SCALE_BYTES raises the per-file +// payload when that is the thing under test. + +import { + Database, + initializeSchema, + readFetchCursor, + SQLiteWorkspaceProvider, +} from "@cloudflare/dofs"; +import { SQLiteTestStorage } from "@cloudflare/dofs/testing"; +import { describe, it } from "vitest"; + +import { createSyncServer } from "../server.js"; +import { pullBlocks } from "../sync-engine.js"; + +const NOW = (): number => 1000; +const ENABLED = process.env.SYNC_SCALE === "1"; +// Files per package is 9 (one manifest plus eight modules), so 4900 +// packages lands near the plan's 44,264 entries. +const PACKAGES = Number(process.env.SYNC_SCALE_PACKAGES ?? 4900); +const FILE_BYTES = Number(process.env.SYNC_SCALE_BYTES ?? 1024); + +function makePeer(): { db: Database; close: () => void } { + const storage = new SQLiteTestStorage(); + const db = new Database(storage); + initializeSchema(db, NOW); + return { db, close: () => storage.close() }; +} + +function seedInstallTree(db: Database): { files: number; logicalBytes: number; seedMs: number } { + const provider = new SQLiteWorkspaceProvider(db, { now: NOW }); + const started = performance.now(); + let files = 0; + let logicalBytes = 0; + // A real install has a long tail of unique modules over a base of + // widely vendored helpers. One in eight files is a shared payload so + // the content-addressed store has genuine duplicates to collapse, + // matching the plan's 44,264 entries against 33,683 unique objects + // (roughly a quarter deduplicated). + const body = (p: number, f: number): string => { + const shared = (p + f) % 4 === 0; + const seed = shared ? `shared-helper-${(p + f) % 64}` : `pkg-${p}-lib-${f}`; + const filler = `${seed} module body with require() calls and comments; `; + return filler.repeat(Math.ceil(FILE_BYTES / filler.length)).slice(0, FILE_BYTES); + }; + for (let p = 0; p < PACKAGES; p++) { + const dir = `/node_modules/pkg-${String(p).padStart(5, "0")}`; + provider.mkdirSync(dir, { recursive: true }); + provider.writeFileSync(`${dir}/package.json`, `{"name":"pkg-${p}","version":"1.0.0"}`); + files++; + logicalBytes += 40; + for (let f = 0; f < 8; f++) { + const content = body(p, f); + provider.writeFileSync(`${dir}/lib-${f}.js`, content); + files++; + logicalBytes += content.length; + } + } + return { files, logicalBytes, seedMs: performance.now() - started }; +} + +function table(rows: Record[]): void { + if (rows.length === 0) return; + const keys = Object.keys(rows[0]); + const width = keys.map((k) => Math.max(k.length, ...rows.map((r) => String(r[k] ?? "").length))); + const line = (cells: (string | number)[]): string => + cells.map((c, i) => String(c).padEnd(width[i])).join(" "); + console.log(line(keys)); + console.log(width.map((w) => "-".repeat(w)).join(" ")); + for (const row of rows) console.log(line(keys.map((k) => row[k] ?? ""))); +} + +// Count bytes crossing the transport so the comparison is not decided +// by an in-process stub that charges nothing for them. +function countingTransport(rpc: ReturnType): { + rpc: ReturnType; + wireBytes: () => number; +} { + let bytes = 0; + const wrapped = new Proxy(rpc as object, { + get(target, prop, receiver) { + const real = Reflect.get(target, prop, receiver); + if (prop === "fetchChangePack") { + return async (...args: unknown[]) => { + const res = (await ( + real as (...a: unknown[]) => Promise<{ stream: ReadableStream }> + ).call(target, ...args)) as { stream: ReadableStream }; + return { + ...res, + stream: res.stream.pipeThrough( + new TransformStream({ + transform(chunk, controller) { + bytes += chunk.byteLength; + controller.enqueue(chunk); + }, + }), + ), + }; + }; + } + if (prop === "fetchObjects") { + return (...args: unknown[]) => { + const stream = ( + real as (...a: unknown[]) => ReadableStream<{ hash: Uint8Array; bytes: Uint8Array }> + ).call(target, ...args); + return stream.pipeThrough( + new TransformStream< + { hash: Uint8Array; bytes: Uint8Array }, + { hash: Uint8Array; bytes: Uint8Array } + >({ + transform(chunk, controller) { + bytes += chunk.bytes.byteLength + 32; + controller.enqueue(chunk); + }, + }), + ); + }; + } + if (prop === "fetchChanges") { + return async (...args: unknown[]) => { + const res = (await ( + real as (...a: unknown[]) => Promise<{ stream: ReadableStream }> + ).call(target, ...args)) as { stream: ReadableStream }; + return { + ...res, + stream: res.stream.pipeThrough( + new TransformStream({ + transform(chunk, controller) { + bytes += JSON.stringify(chunk).length; + controller.enqueue(chunk); + }, + }), + ), + }; + }; + } + return real; + }, + }) as ReturnType; + return { rpc: wrapped, wireBytes: () => bytes }; +} + +describe.skipIf(!ENABLED)("install-scale sync", () => { + it("converges a package-install tree in bounded blocks", async () => { + const source = makePeer(); + const seeded = seedInstallTree(source.db); + console.log( + `\nseeded ${seeded.files} files, ` + + `${(seeded.logicalBytes / 1024 / 1024).toFixed(1)} MB logical, ` + + `in ${(seeded.seedMs / 1000).toFixed(1)}s`, + ); + + const rows: Record[] = []; + const json: Record[] = []; + + for (const forcePack of [false, true]) { + const local = makePeer(); + const counted = countingTransport(createSyncServer(source.db)); + const options = { + // The shipped default profile, so this measures what production + // would actually do rather than a tuned-for-benchmark size. + profile: { maxEntries: 4000, maxBytes: 64 * 1024 * 1024 }, + thresholdEntries: forcePack ? 1 : Number.MAX_SAFE_INTEGER, + thresholdBytes: forcePack ? 1 : Number.MAX_SAFE_INTEGER, + }; + + let blocks = 0; + let entries = 0; + let worstBlockMs = 0; + let mode = "?"; + const started = performance.now(); + // Fresh iterable per block: the eviction worst case, where every + // block re-reads the operation row and cursor from SQLite. + for (let guard = 0; guard < 100_000; guard++) { + const tBlock = performance.now(); + const iterator = pullBlocks(local.db, counted.rpc, options)[Symbol.asyncIterator](); + const { value, done } = await iterator.next(); + const blockMs = performance.now() - tBlock; + if (done) break; + blocks++; + entries += value.entries; + mode = value.mode; + worstBlockMs = Math.max(worstBlockMs, blockMs); + if (value.complete) break; + } + const totalMs = performance.now() - started; + const appliedFiles = + local.db.one<{ c: number }>("SELECT COUNT(*) AS c FROM vfs_nodes WHERE type = 'file'")?.c ?? + 0; + const uniqueObjects = + local.db.one<{ c: number }>("SELECT COUNT(*) AS c FROM vfs_blobs")?.c ?? 0; + const wireKB = counted.wireBytes() / 1024; + + rows.push({ + mode, + entries, + blocks, + "unique objects": uniqueObjects, + "wire MB": Number((wireKB / 1024).toFixed(1)), + "total (s)": Number((totalMs / 1000).toFixed(1)), + "worst block (ms)": Number(worstBlockMs.toFixed(0)), + "ms/entry": Number((totalMs / Math.max(1, entries)).toFixed(3)), + "files applied": appliedFiles, + converged: + readFetchCursor(local.db).rev > 0 && appliedFiles === seeded.files ? "yes" : "no", + }); + json.push({ + mode, + entries, + blocks, + wireKB: Number(wireKB.toFixed(1)), + totalMs: Number(totalMs.toFixed(0)), + worstBlockMs: Number(worstBlockMs.toFixed(0)), + appliedFiles, + sourceFiles: seeded.files, + }); + local.close(); + } + + console.log("\n=== install-scale pull ==="); + table(rows); + + // Price the byte difference against a link, since an in-process + // peer charges nothing for bytes and that is the one input the + // entries-versus-pack question turns on. + const entryRun = json[0] as { totalMs: number; wireKB: number }; + const packRun = json[1] as { totalMs: number; wireKB: number }; + const priced: Record[] = []; + for (const mbps of [10, 50, 100, 500, Number.POSITIVE_INFINITY]) { + const bytesPerMs = (mbps * 1024 * 1024) / 8 / 1000; + const e = entryRun.totalMs + (entryRun.wireKB * 1024) / bytesPerMs; + const p = packRun.totalMs + (packRun.wireKB * 1024) / bytesPerMs; + priced.push({ + "link Mbps": mbps === Number.POSITIVE_INFINITY ? "infinite" : mbps, + "entries total (s)": Number((e / 1000).toFixed(1)), + "pack total (s)": Number((p / 1000).toFixed(1)), + winner: p < e ? "pack" : "entries", + margin: `${Math.abs(((e - p) / Math.max(1, e)) * 100).toFixed(0)}%`, + }); + } + console.log("\n=== install-scale, priced against link speed ==="); + table(priced); + console.log(`BENCH_JSON scale ${JSON.stringify(json)}`); + + source.close(); + }); +}); diff --git a/packages/rpc/src/interface.ts b/packages/rpc/src/interface.ts index 967986bf..8e46173f 100644 --- a/packages/rpc/src/interface.ts +++ b/packages/rpc/src/interface.ts @@ -96,6 +96,43 @@ export interface SyncRPC { // any hash is unknown — callers must dedupe and probe first. fetchObjects(hashes: Uint8Array[]): ReadableStream<{ hash: Uint8Array; bytes: Uint8Array }>; + // Bulk pull transport. Same cursor semantics as fetchChanges, but the + // response is one gzip change pack carrying entries interleaved with + // the objects they reference, rather than an entry stream the caller + // must follow with hasObjects / fetchObjects round trips. + // + // Selected automatically for large cursor windows; see the pack + // thresholds in @cloudflare/dofs. `maxEntries` and `maxBytes` bound + // the block so one pack stays inside the receiver's CPU budget, and + // `cursor` echoes the block cursor the pack's footer also carries. + // Optional: a peer built before pack transport omits these, and the + // engine falls back to entry mode when it finds them absent. That is + // what lets the two sides be deployed in either order. + fetchChangePack?(input: { + after?: ChangeCursor; + through?: ChangeCursor; + ignore?: string[]; + maxEntries: number; + maxBytes: number; + generation: string; + }): Promise<{ + cursor: ChangeCursor; + drained: boolean; + entryCount: number; + appliedPushCursor: ChangeCursor; + stream: ReadableStream; + }>; + + // Bulk push transport, the mirror of fetchChangePack. The receiver + // decodes the pack, stages its objects, applies its entries, and + // echoes the cursor it applied so the sender can advance only through + // an acknowledgment. + applyChangePack?(input: { generation: string; stream: ReadableStream }): Promise<{ + appliedPushCursor: ChangeCursor; + applied: number; + entryCount: number; + }>; + // DO → container direction of object transfer. The DO streams the // bytes the container reported missing (via hasObjects) during a // push. Pushed objects are addressable immediately by hash. diff --git a/packages/rpc/src/server.ts b/packages/rpc/src/server.ts index 83eacda8..4e675c3d 100644 --- a/packages/rpc/src/server.ts +++ b/packages/rpc/src/server.ts @@ -12,9 +12,12 @@ import { compareChangeCursors, currentRev, type Database, + decodeChangePack, + encodeChangePack, fetchObjects, hasObjects, materialiseChange, + planBlock, readFetchCursor, readWatermark, stageBlob, @@ -144,14 +147,38 @@ class SyncRPCServer extends RpcTarget implements SyncRPC { writeFetchCursor(this.db, senderCursor); } }); + // Settle the receiver's shim, and let the outcome decide what the + // caller is told — which differs by who the caller is. + // + // A peer push is replayable. Its sender holds a durable push + // cursor, so rejecting the acknowledgment leaves that cursor + // where it was and the next push replans the same block. That + // matters because the pre-command bracket pushes, advances on the + // acknowledgment, and only then spawns: acknowledging an + // unflushed block would retire it and let the command read stale + // disk. Replay is safe because these entries carry + // source: "upstream", which routes them through the + // already-applied and stale-tombstone guards. + // + // An external push is not replayable. It has no durable cursor to + // rewind, and its entries are applied as source: "local", which + // deliberately bypasses those guards because local writes are + // authored here rather than replayed. Reporting a failure after + // the commit therefore invites the caller to repeat a mutation + // that is not idempotent: a retried delete would carry the + // original revision and, with the stale-tombstone check skipped, + // remove a file some other writer recreated in between. So the + // failure is logged and the push acknowledged, leaving the shim's + // periodic reconcile to repair disk. if (this.options.afterApply !== undefined && entries.length > 0) { - try { + if (isPeer) { await this.options.afterApply(); - } catch (err) { - // Settle hook failures must not surface as push failures — - // the entries are already committed. Log so the operator - // notices a wedged shim, then return success. - console.warn("[SyncRPCServer] afterApply hook failed:", err); + } else { + try { + await this.options.afterApply(); + } catch (err) { + console.warn("[SyncRPCServer] afterApply hook failed for external push:", err); + } } } return { @@ -197,6 +224,105 @@ class SyncRPCServer extends RpcTarget implements SyncRPC { }; } + // Bulk pull. Plans one block against the same cursor window + // fetchChanges uses, then encodes it as a pack so the caller needs no + // follow-up object round trips. + async fetchChangePack(input: { + after?: ChangeCursor; + through?: ChangeCursor; + ignore?: string[]; + maxEntries: number; + maxBytes: number; + generation: string; + }): Promise<{ + cursor: ChangeCursor; + drained: boolean; + entryCount: number; + appliedPushCursor: ChangeCursor; + stream: ReadableStream; + }> { + if (this.options.beforeFetch !== undefined) { + try { + await this.options.beforeFetch(); + } catch (err) { + console.warn("[SyncRPCServer] beforeFetch hook failed:", err); + } + } + const after = input.after ?? { rev: 0, path: null }; + const ignore = input.ignore ?? this.options.ignore; + const snapshotCursor = { rev: currentRev(this.db), path: null }; + const target = + input.through !== undefined && compareChangeCursors(input.through, snapshotCursor) < 0 + ? input.through + : snapshotCursor; + + const block = await planBlock(this.db, { + after, + through: target, + profile: { maxEntries: input.maxEntries, maxBytes: input.maxBytes }, + ...(ignore === undefined ? {} : { ignore }), + }); + + return { + cursor: block.cursor, + drained: block.drained, + entryCount: block.entries.length, + appliedPushCursor: readFetchCursor(this.db), + stream: encodeChangePack(this.db, { + block, + after, + target, + generation: input.generation, + }), + }; + } + + // Bulk push receiver. Decode fully before applying anything: a pack + // that fails validation must leave no partial state behind, because + // the sender advances its cursor on the acknowledgment this returns. + async applyChangePack(input: { + generation: string; + stream: ReadableStream; + }): Promise<{ + appliedPushCursor: ChangeCursor; + applied: number; + entryCount: number; + }> { + const decoded = await decodeChangePack(input.stream, { + expectGeneration: input.generation, + }); + for (const [key, bytes] of decoded.objects) { + stageBlob(this.db, hexToBytes(key), bytes, Date.now()); + } + const result = applyChangesSync(this.db, decoded.entries, new Map(), { + source: "upstream", + }); + // The footer's block cursor is the sender's checkpoint; echo it so + // the sender advances only through what actually applied here. + writeFetchCursor(this.db, decoded.footer.blockCursor); + // Mirror push(): settle the receiver's shim so a subsequent + // shell.exec sees the just-pushed files on disk. Pack mode is + // selected for exactly the large windows a pre-command push + // carries, so skipping this strands the shim on stale disk state. + // + // A rejection propagates rather than being logged. The caller + // treats an acknowledgment as "the receiver is ready", advances + // its durable push cursor on it, and then spawns the command, so + // acknowledging an unflushed block retires it permanently and + // lets the command read stale disk. Failing the RPC leaves the + // sender cursor where it was and the next push replans the same + // block; the entries are already committed here, and applying + // them again is absorbed by alreadyApplied(). + if (this.options.afterApply !== undefined && decoded.entries.length > 0) { + await this.options.afterApply(); + } + return { + appliedPushCursor: decoded.footer.blockCursor, + applied: result.applied, + entryCount: decoded.entries.length, + }; + } + async readEntry(path: string): Promise { return materialiseChange(this.db, path); } @@ -379,3 +505,12 @@ function iterableToReadableStream(it: AsyncIterable): ReadableStream { }, }); } + +// Pack object keys are hex; stageBlob wants the raw hash bytes. +function hexToBytes(hex: string): Uint8Array { + const out = new Uint8Array(hex.length / 2); + for (let i = 0; i < out.length; i++) { + out[i] = Number.parseInt(hex.slice(i * 2, i * 2 + 2), 16); + } + return out; +} diff --git a/packages/rpc/src/sync-driver.test.ts b/packages/rpc/src/sync-driver.test.ts index 2f1fea3a..850e4af6 100644 --- a/packages/rpc/src/sync-driver.test.ts +++ b/packages/rpc/src/sync-driver.test.ts @@ -16,14 +16,7 @@ import { describe, expect, it } from "vitest"; import type { SyncRPC } from "./interface.js"; import { createSyncServer } from "./server.js"; -import { - pullBatch, - pullOnce, - pushBatch, - pushOnce, - reconcileWatermarks, - tick, -} from "./sync-driver.js"; +import { pullOnce, pushOnce, reconcileWatermarks, tick } from "./sync-driver.js"; // Two peers wired up as direct in-process SyncRPC stubs. No // WebSocket; we already have the real-wire convergence test in @@ -352,23 +345,84 @@ describe("SyncRPC server — afterApply hook", () => { } }); - it("a thrown hook does not fail the push", async () => { + it("a thrown hook fails the push and leaves the sender cursor behind", async () => { const a = makePeer(); const b = makeReceiverWithSpy(); try { const providerA = new SQLiteWorkspaceProvider(a.db, { now: () => 1 }); providerA.writeFileSync("/hi.txt", "hi"); + const before = readWatermark(a.db, "pushRev"); b.setAfterApply(() => { throw new Error("settle blew up"); }); - // The push must still succeed — entries are committed before - // the hook runs, and the server logs+swallows hook errors. - const pushed = await pushOnce(a.db, b.rpc); - expect(pushed).toBe(1); + // The caller spawns a command on the strength of this + // acknowledgment, so an unflushed block must not be reported as + // success. The entries do commit on the receiver — the failure + // is about disk visibility, not the log — but the sender must + // not retire the block. + await expect(pushOnce(a.db, b.rpc)).rejects.toThrow("settle blew up"); expect(b.calls).toBe(1); expect(fileEntries(b.db)).toContain("hi.txt"); + expect(readWatermark(a.db, "pushRev")).toBe(before); + } finally { + a.close(); + b.close(); + } + }); + + it("acknowledges an external push whose settle rejects", async () => { + const b = makeReceiverWithSpy(); + try { + // An external writer (senderRev 0) has no durable cursor to + // rewind, and its entries skip the stale-tombstone guard because + // they apply as local writes. Reporting a failure would invite a + // retry of a delete that could remove a newer recreation, so the + // settle failure is swallowed for this caller only. + const provider = new SQLiteWorkspaceProvider(b.db, { now: () => 1 }); + provider.writeFileSync("/gone.txt", "x"); + + b.setAfterApply(() => { + throw new Error("settle blew up"); + }); + + const changes = new ReadableStream({ + start(controller) { + controller.enqueue({ kind: "delete", rev: 1, path: "/gone.txt" }); + controller.close(); + }, + }); + const ack = await b.rpc.push({ senderRev: 0, changes }); + expect(b.calls).toBe(1); + // Acknowledged despite the failed settle, and the delete stands. + expect(ack.appliedPushCursor).toEqual({ rev: 0, path: null }); + expect(fileEntries(b.db)).not.toContain("gone.txt"); + } finally { + b.close(); + } + }); + + it("replays the same block once the hook recovers", async () => { + const a = makePeer(); + const b = makeReceiverWithSpy(); + try { + const providerA = new SQLiteWorkspaceProvider(a.db, { now: () => 1 }); + providerA.writeFileSync("/hi.txt", "hi"); + + b.setAfterApply(() => { + throw new Error("disk full"); + }); + await expect(pushOnce(a.db, b.rpc)).rejects.toThrow("disk full"); + + // The retry replans the same block. Re-applying entries the + // receiver already holds is absorbed by alreadyApplied(), so the + // second attempt settles cleanly and advances the cursor. + b.setAfterApply(() => {}); + const pushed = await pushOnce(a.db, b.rpc); + expect(pushed).toBeGreaterThan(0); + expect(b.calls).toBe(2); + expect(fileEntries(b.db)).toContain("hi.txt"); } finally { a.close(); b.close(); @@ -1320,145 +1374,3 @@ async function sha256(bytes: Uint8Array): Promise { hash.update(bytes); return new Uint8Array(hash.digest()); } - -describe("bounded synchronization", () => { - it("pulls one entry per batch and resumes at the captured target", async () => { - const upstream = makePeer(); - const downstream = makePeer(); - try { - const provider = new SQLiteWorkspaceProvider(upstream.db, { now: () => 1 }); - await provider.writeFile("/one.txt", "one"); - await provider.writeFile("/two.txt", "two"); - - const first = await pullBatch(downstream.db, upstream.rpc, { - budget: { maxEntries: 1, maxBytes: 1024 }, - }); - expect(first.status).toBe("pending"); - expect(first.entries).toBe(1); - - const second = await pullBatch(downstream.db, upstream.rpc, { - targetCursor: first.targetCursor, - budget: { maxEntries: 1, maxBytes: 1024 }, - }); - expect(second.status).toBe("pending"); - const third = await pullBatch(downstream.db, upstream.rpc, { - targetCursor: first.targetCursor, - budget: { maxEntries: 1, maxBytes: 1024 }, - }); - expect(third.status).toBe("complete"); - expect(second.entries + third.entries).toBe(1); - expect(fileEntries(downstream.db)).toEqual(["one.txt", "two.txt"]); - } finally { - upstream.close(); - downstream.close(); - } - }); - - it("stages a large file across pull batches without redownloading chunks", async () => { - const upstream = makePeer(); - const downstream = makePeer(); - try { - const provider = new SQLiteWorkspaceProvider(upstream.db, { now: () => 1 }); - const large = new Uint8Array(3 * 512 * 1024); - large.fill(1, 0, 512 * 1024); - large.fill(2, 512 * 1024, 2 * 512 * 1024); - large.fill(3, 2 * 512 * 1024); - await provider.writeFile("/large.bin", large); - let fetches = 0; - const rpc = new Proxy(upstream.rpc as object, { - get(target, property, receiver) { - if (property === "fetchObjects") { - return (hashes: Uint8Array[]) => { - fetches += hashes.length; - return Reflect.get(target, property, receiver).call(target, hashes); - }; - } - return Reflect.get(target, property, receiver); - }, - }) as SyncRPC; - - const first = await pullBatch(downstream.db, rpc, { - budget: { maxEntries: 8, maxBytes: 512 * 1024 }, - }); - expect(first.status).toBe("pending"); - expect(first.entries).toBe(0); - expect(first.bytes).toBe(512 * 1024); - - const second = await pullBatch(downstream.db, rpc, { - targetCursor: first.targetCursor, - budget: { maxEntries: 8, maxBytes: 512 * 1024 }, - }); - expect(second.status).toBe("pending"); - expect(second.bytes).toBe(512 * 1024); - - const third = await pullBatch(downstream.db, rpc, { - targetCursor: first.targetCursor, - budget: { maxEntries: 8, maxBytes: 512 * 1024 }, - }); - expect(third.status).toBe("complete"); - expect(fetches).toBe(3); - } finally { - upstream.close(); - downstream.close(); - } - }); - - it("pushes one bounded unit and resumes through a revision", async () => { - const upstream = makePeer(); - const downstream = makePeer(); - try { - const provider = new SQLiteWorkspaceProvider(upstream.db, { now: () => 1 }); - await provider.writeFile("/one.txt", "one"); - await provider.writeFile("/two.txt", "two"); - - const first = await pushBatch(upstream.db, downstream.rpc, { - backend: "container", - budget: { maxEntries: 1, maxBytes: 1024 }, - }); - expect(first.status).toBe("pending"); - expect(first.entries).toBe(1); - - const second = await pushBatch(upstream.db, downstream.rpc, { - backend: "container", - targetCursor: first.targetCursor, - budget: { maxEntries: 1, maxBytes: 1024 }, - }); - expect(second.status).toBe("pending"); - const third = await pushBatch(upstream.db, downstream.rpc, { - backend: "container", - targetCursor: first.targetCursor, - budget: { maxEntries: 1, maxBytes: 1024 }, - }); - expect(third.status).toBe("complete"); - expect(fileEntries(downstream.db)).toEqual(["one.txt", "two.txt"]); - } finally { - upstream.close(); - downstream.close(); - } - }); - - it("leaves an advanced fetch cursor alone when an old target is already satisfied", async () => { - const upstream = makePeer(); - const downstream = makePeer(); - try { - const provider = new SQLiteWorkspaceProvider(upstream.db, { now: () => 1 }); - await provider.writeFile("/one.txt", "one"); - const oldTarget = { rev: currentRev(upstream.db), path: null }; - await provider.writeFile("/two.txt", "two"); - await pullOnce(downstream.db, upstream.rpc); - const advanced = readFetchCursor(downstream.db); - - const result = await pullBatch(downstream.db, upstream.rpc, { - targetCursor: oldTarget, - budget: { maxEntries: 1, maxBytes: 1024 }, - }); - - expect(result.status).toBe("complete"); - expect(result.cursor).toEqual(advanced); - expect(readFetchCursor(downstream.db)).toEqual(advanced); - } finally { - upstream.close(); - downstream.close(); - } - }); -}); diff --git a/packages/rpc/src/sync-driver.ts b/packages/rpc/src/sync-driver.ts index 56e556e2..076e783a 100644 --- a/packages/rpc/src/sync-driver.ts +++ b/packages/rpc/src/sync-driver.ts @@ -15,9 +15,7 @@ import { assertAppliedPushCursor, type ChangeCursor, type ChangeEntry, - coalesceChanges, compareChangeCursors, - currentRev, type Database, hasObjects, readFetchCursor, @@ -32,33 +30,14 @@ import { import type { SyncRPC } from "./interface.js"; -export interface SyncBatchBudget { - maxEntries: number; - maxBytes: number; - maxWallTimeMs?: number; -} - -export interface SyncBatchResult { - status: "complete" | "pending"; - entries: number; - bytes: number; - applied: number; - skipped: SkippedEntry[]; - cursor: ChangeCursor; - targetCursor: ChangeCursor; -} - -export interface PullBatchOptions { - backend?: string; - targetCursor?: ChangeCursor; - budget: SyncBatchBudget; -} +// The restartable engine ships through the same ./driver entry point. +// Explicit-cursor block helpers below remain the low-level surface for +// callers that own their own destination store and checkpoint; the +// engine is what the Workspace iterables are built on. +import { captureSyncTarget, pullBlocks, pushBlocks } from "./sync-engine.js"; -export interface PushBatchOptions { - backend?: string; - targetCursor?: ChangeCursor; - budget: SyncBatchBudget; -} +export type { PullBlocksOptions, PushBlocksOptions, SyncProgress } from "./sync-engine.js"; +export { captureSyncTarget, pullBlocks, pushBlocks }; function hex(bytes: Uint8Array): string { let s = ""; @@ -310,397 +289,19 @@ function writeFetchCursorIfAhead(db: Database, cursor: ChangeCursor, backend?: s } } -function validateBudget(budget: SyncBatchBudget): void { - if (!Number.isSafeInteger(budget.maxEntries) || budget.maxEntries <= 0) { - throw new Error("Sync batch maxEntries must be a positive safe integer"); - } - if (!Number.isSafeInteger(budget.maxBytes) || budget.maxBytes <= 0) { - throw new Error("Sync batch maxBytes must be a positive safe integer"); - } - if ( - budget.maxWallTimeMs !== undefined && - (!Number.isFinite(budget.maxWallTimeMs) || budget.maxWallTimeMs <= 0) - ) { - throw new Error("Sync batch maxWallTimeMs must be positive when provided"); - } -} - -function entryCursor(entry: ChangeEntry): ChangeCursor { - return { rev: entry.rev, path: entry.path }; -} - -function entryHashes(entry: ChangeEntry): { hash: Uint8Array; size: number }[] { - return entry.kind === "file" ? entry.chunks : []; -} - -function minimumCursor(a: ChangeCursor, b: ChangeCursor): ChangeCursor { - return compareChangeCursors(a, b) <= 0 ? a : b; -} - -export function pullBatch( - db: Database, - remote: SyncRPC, - options: PullBatchOptions, -): Promise { - validateBudget(options.budget); - return pullBatchImpl(db, remote, options, false); -} - -async function pullBatchImpl( - db: Database, - remote: SyncRPC, - options: PullBatchOptions, - retried: boolean, -): Promise { - const backend = options.backend; - const after = readFetchCursor(db, backend); - if ( - options.targetCursor !== undefined && - compareChangeCursors(after, options.targetCursor) >= 0 - ) { - return { - status: "complete", - entries: 0, - bytes: 0, - applied: 0, - skipped: [], - cursor: after, - targetCursor: options.targetCursor, - }; - } - const fetchResult = await remote.fetchChanges({ after, through: options.targetCursor }); - const pushCursor = readPushCursor(db, backend); - const pushDiverged = fetchResult.appliedPushCursor.rev < pushCursor.rev; - const fetchDiverged = compareChangeCursors(fetchResult.currentCursor, after) < 0; - if (!retried && (pushDiverged || fetchDiverged)) { - await fetchResult.stream.cancel().catch(() => {}); - maybeDispose(fetchResult); - if (pushDiverged) writePushCursor(db, { rev: 0, path: null }, backend); - if (fetchDiverged) writeFetchCursor(db, { rev: 0, path: null }, backend); - return pullBatchImpl(db, remote, options, true); - } - const targetCursor = minimumCursor( - options.targetCursor ?? fetchResult.currentCursor, - fetchResult.currentCursor, - ); - try { - assertAppliedPushCursor(fetchResult.appliedPushCursor, pushCursor); - } catch (error) { - maybeDispose(fetchResult); - throw error; - } - if (compareChangeCursors(after, targetCursor) >= 0) { - maybeDispose(fetchResult); - return { - status: "complete", - entries: 0, - bytes: 0, - applied: 0, - skipped: [], - cursor: after, - targetCursor, - }; - } - - const reader = fetchResult.stream.getReader(); - let streamDone = false; - let cursor = after; - let entries = 0; - let bytes = 0; - let applied = 0; - const skipped: SkippedEntry[] = []; - const started = Date.now(); - try { - while (entries < options.budget.maxEntries) { - if ( - options.budget.maxWallTimeMs !== undefined && - Date.now() - started >= options.budget.maxWallTimeMs - ) { - break; - } - const next = await reader.read(); - if (next.done) { - streamDone = true; - break; - } - const entry = next.value; - const chunks = entryHashes(entry); - const hashes = chunks.map((chunk) => chunk.hash); - const localHave = new Set(hasObjects(db, hashes).map(hex)); - const remoteHave = new Set( - (hashes.length === 0 ? [] : await remote.hasObjects(hashes)).map(hex), - ); - const missingRemote = chunks.filter((chunk) => !remoteHave.has(hex(chunk.hash))); - if (missingRemote.length > 0) { - throw new Error(`pullBatch: remote is missing object ${hex(missingRemote[0].hash)}`); - } - const missingLocal = chunks.filter((chunk) => !localHave.has(hex(chunk.hash))); - const transferable: { hash: Uint8Array; size: number }[] = []; - let availableBytes = options.budget.maxBytes - bytes; - for (const chunk of missingLocal) { - if (chunk.size <= availableBytes || transferable.length === 0) { - transferable.push(chunk); - availableBytes -= chunk.size; - } else { - break; - } - } - if (transferable.length < missingLocal.length) { - if (transferable.length > 0) { - const objectStream = await remote.fetchObjects(transferable.map((chunk) => chunk.hash)); - const objectReader = objectStream.getReader(); - try { - while (true) { - const object = await objectReader.read(); - if (object.done) break; - stageBlob(db, object.value.hash, object.value.bytes, Date.now()); - bytes += object.value.bytes.byteLength; - } - } finally { - objectReader.releaseLock(); - } - } - return { - status: "pending", - entries, - bytes, - applied, - skipped, - cursor, - targetCursor, - }; - } - if (transferable.length > 0) { - const objectStream = await remote.fetchObjects(transferable.map((chunk) => chunk.hash)); - const objectReader = objectStream.getReader(); - try { - while (true) { - const object = await objectReader.read(); - if (object.done) break; - stageBlob(db, object.value.hash, object.value.bytes, Date.now()); - bytes += object.value.bytes.byteLength; - } - } finally { - objectReader.releaseLock(); - } - } - const result = await applyChanges(db, [entry], new Map(), { - source: "upstream", - backend, - }); - const nextCursor = entryCursor(entry); - writeFetchCursorIfAhead(db, nextCursor, backend); - cursor = nextCursor; - entries += 1; - applied += result.applied; - skipped.push(...result.skipped); - } - if (streamDone) { - writeFetchCursorIfAhead(db, targetCursor, backend); - cursor = targetCursor; - } - return { - status: compareChangeCursors(cursor, targetCursor) >= 0 ? "complete" : "pending", - entries, - bytes, - applied, - skipped, - cursor, - targetCursor, - }; - } finally { - if (!streamDone) await reader.cancel().catch(() => {}); - reader.releaseLock(); - maybeDispose(fetchResult); - } -} - -export async function pushBatch( - db: Database, - remote: SyncRPC, - options: PushBatchOptions, -): Promise { - validateBudget(options.budget); - const backend = options.backend; - const cursor = readPushCursor(db, backend); - const targetCursor = minimumCursor(options.targetCursor ?? { rev: currentRev(db), path: null }, { - rev: currentRev(db), - path: null, - }); - if (compareChangeCursors(cursor, targetCursor) >= 0) { - return { - status: "complete", - entries: 0, - bytes: 0, - applied: 0, - skipped: [], - cursor, - targetCursor, - }; - } - - const candidates: ChangeEntry[] = []; - let exhausted = true; - for await (const entry of coalesceChanges(db, cursor, { through: targetCursor })) { - candidates.push(entry); - if (candidates.length >= options.budget.maxEntries) { - exhausted = false; - break; - } - } - if (candidates.length === 0) { - if (cursor.rev === 0 && cursor.path === null) { - return { - status: "complete", - entries: 0, - bytes: 0, - applied: 0, - skipped: [], - cursor, - targetCursor, - }; - } - const changes = new ReadableStream({ - start(controller) { - controller.close(); - }, - }); - const response = await remote.push({ - senderRev: targetCursor.rev, - senderCursor: targetCursor, - changes, - }); - assertAppliedPushCursor(response.appliedPushCursor, targetCursor); - writePushCursor(db, targetCursor, backend); - return { - status: "complete", - entries: 0, - bytes: 0, - applied: 0, - skipped: [], - cursor: targetCursor, - targetCursor, - }; - } - - const wanted: { hash: Uint8Array; size: number }[] = []; - const seen = new Set(); - for (const entry of candidates) { - for (const chunk of entryHashes(entry)) { - const key = hex(chunk.hash); - if (!seen.has(key)) { - seen.add(key); - wanted.push(chunk); - } - } - } - const have = new Set( - (wanted.length === 0 ? [] : await remote.hasObjects(wanted.map((c) => c.hash))).map(hex), - ); - const missing = wanted.filter((chunk) => !have.has(hex(chunk.hash))); - const transferable: { hash: Uint8Array; size: number }[] = []; - let availableBytes = options.budget.maxBytes; - for (const chunk of missing) { - if (chunk.size <= availableBytes || transferable.length === 0) { - transferable.push(chunk); - availableBytes -= chunk.size; - } else { - break; - } - } - let bytes = 0; - if (transferable.length > 0) { - const local = (function* () { - for (const chunk of transferable) { - const row = db.one<{ bytes: Uint8Array }>( - "SELECT bytes FROM vfs_blob_bytes WHERE hash = ?", - chunk.hash, - ); - if (row === undefined) throw new Error(`pushBatch: missing local blob ${hex(chunk.hash)}`); - bytes += row.bytes.byteLength; - yield { hash: chunk.hash, bytes: row.bytes }; - } - })(); - const objectStream = new ReadableStream<{ hash: Uint8Array; bytes: Uint8Array }>({ - pull(controller) { - const next = local.next(); - if (next.done) controller.close(); - else controller.enqueue(next.value); - }, - }); - await remote.pushObjects(objectStream); - } - if (transferable.length < missing.length) { - return { - status: "pending", - entries: 0, - bytes, - applied: 0, - skipped: [], - cursor, - targetCursor, - }; - } - - const selected = candidates.filter((entry) => { - const entryMissing = entryHashes(entry).filter((chunk) => !have.has(hex(chunk.hash))); - return entryMissing.every((chunk) => - transferable.some((item) => hex(item.hash) === hex(chunk.hash)), - ); - }); - if (selected.length === 0) { - return { - status: "pending", - entries: 0, - bytes, - applied: 0, - skipped: [], - cursor, - targetCursor, - }; - } - const lastCursor = entryCursor(selected[selected.length - 1]); - const acknowledgedCursor = exhausted ? targetCursor : lastCursor; - const entryStream = new ReadableStream({ - start(controller) { - for (const entry of selected) controller.enqueue(entry); - controller.close(); - }, - }); - const response = await remote.push({ - senderRev: targetCursor.rev, - senderCursor: acknowledgedCursor, - changes: entryStream, - }); - assertAppliedPushCursor(response.appliedPushCursor, acknowledgedCursor); - writePushCursor(db, acknowledgedCursor, backend); - return { - status: compareChangeCursors(acknowledgedCursor, targetCursor) >= 0 ? "complete" : "pending", - entries: selected.length, - bytes, - applied: response.applied ?? selected.length, - skipped: [], - cursor: acknowledgedCursor, - targetCursor, - }; -} - // Push every entry the local store has produced since the last // successful push. The wire shape mirrors pullOnce in reverse: // stage bytes the remote lacks, then push the entry stream. +// Drains to completion, so callers that need the whole local window +// shipped before a command starts get one awaitable call. Restartable +// push lives in pushBlocks; this is the in-band bracket. export async function pushOnce(db: Database, remote: SyncRPC, backend?: string): Promise { - let targetCursor: ChangeCursor | undefined; let pushed = 0; - while (true) { - const result = await pushBatch(db, remote, { - backend, - targetCursor, - budget: { maxEntries: PULL_BATCH_SIZE, maxBytes: 4 * 1024 * 1024 }, - }); - targetCursor = result.targetCursor; - pushed += result.entries; - if (result.status === "complete") return pushed; + for await (const progress of pushBlocks(db, remote, backend === undefined ? {} : { backend })) { + pushed += progress.entries; + if (progress.complete) break; } + return pushed; } // One full tick: pull, then push. The order matters \u2014 pulling diff --git a/packages/rpc/src/sync-engine-pack.test.ts b/packages/rpc/src/sync-engine-pack.test.ts new file mode 100644 index 00000000..d987ca65 --- /dev/null +++ b/packages/rpc/src/sync-engine-pack.test.ts @@ -0,0 +1,422 @@ +import { + Database, + initializeSchema, + readOperation, + SQLiteWorkspaceProvider, +} from "@cloudflare/dofs"; +import { SQLiteTestStorage } from "@cloudflare/dofs/testing"; +import { describe, expect, it } from "vitest"; + +import type { SyncRPC } from "./interface.js"; +import { createSyncServer } from "./server.js"; +import { pullBlocks, pushBlocks, type SyncProgress } from "./sync-engine.js"; + +// Pack mode is the bulk path. These tests force the mode thresholds +// down rather than materialising 20,000 files, then assert the two +// things that distinguish pack from entry mode: the transport actually +// used a pack, and convergence is identical either way. + +function makePeer(): { db: Database; rpc: SyncRPC; close: () => void } { + const storage = new SQLiteTestStorage(); + const db = new Database(storage); + initializeSchema(db, () => 1000); + const rpc = createSyncServer(db); + return { db, rpc, close: () => storage.close() }; +} + +function seed(db: Database, count: number, bytes = 16): void { + const provider = new SQLiteWorkspaceProvider(db); + for (let i = 0; i < count; i++) { + const filler = String.fromCharCode(97 + (i % 26)).repeat(Math.max(1, bytes - 1)); + provider.writeFileSync(`/f${String(i).padStart(3, "0")}.txt`, `${i}${filler}`); + } +} + +function names(db: Database): string[] { + return db + .all<{ name: string }>("SELECT name FROM vfs_dirents WHERE parent_inode = 1 ORDER BY name") + .map((r) => r.name); +} + +// Force pack selection at a low entry count. +const PACK_OPTIONS = { thresholdEntries: 3, thresholdBytes: 1024 * 1024 * 1024 }; + +async function drain(iterable: AsyncIterable, limit = 200): Promise { + const seen: SyncProgress[] = []; + for await (const progress of iterable) { + seen.push(progress); + if (seen.length > limit) throw new Error("sync did not terminate"); + } + return seen; +} + +describe("pack mode pull", () => { + it("selects pack mode once the window crosses the entry threshold", async () => { + const local = makePeer(); + const remote = makePeer(); + try { + seed(remote.db, 6); + + const seen = await drain(pullBlocks(local.db, remote.rpc, PACK_OPTIONS)); + + expect(seen[0].mode).toBe("pack"); + expect(seen[seen.length - 1].complete).toBe(true); + } finally { + local.close(); + remote.close(); + } + }); + + it("stays in entry mode below the threshold", async () => { + const local = makePeer(); + const remote = makePeer(); + try { + seed(remote.db, 2); + + const seen = await drain(pullBlocks(local.db, remote.rpc, PACK_OPTIONS)); + + expect(seen[0].mode).toBe("entries"); + } finally { + local.close(); + remote.close(); + } + }); + + it("converges through a pack exactly as entry mode would", async () => { + const local = makePeer(); + const remote = makePeer(); + try { + seed(remote.db, 6); + + await drain(pullBlocks(local.db, remote.rpc, PACK_OPTIONS)); + + expect(names(local.db)).toEqual([ + "f000.txt", + "f001.txt", + "f002.txt", + "f003.txt", + "f004.txt", + "f005.txt", + ]); + } finally { + local.close(); + remote.close(); + } + }); + + it("transfers file content through the pack", async () => { + const local = makePeer(); + const remote = makePeer(); + try { + seed(remote.db, 5); + + await drain(pullBlocks(local.db, remote.rpc, PACK_OPTIONS)); + + const provider = new SQLiteWorkspaceProvider(local.db); + expect(provider.readFileSync("/f000.txt", "utf8")).toBe(`0${"a".repeat(15)}`); + expect(provider.readFileSync("/f004.txt", "utf8")).toBe(`4${"e".repeat(15)}`); + } finally { + local.close(); + remote.close(); + } + }); + + it("splits a pack operation across blocks and still converges", async () => { + const local = makePeer(); + const remote = makePeer(); + try { + seed(remote.db, 8); + + const seen = await drain( + pullBlocks(local.db, remote.rpc, { + ...PACK_OPTIONS, + profile: { maxEntries: 3, maxBytes: 64 * 1024 * 1024 }, + }), + ); + + expect(seen.length).toBeGreaterThanOrEqual(3); + expect(seen.every((p) => p.mode === "pack")).toBe(true); + expect(names(local.db)).toHaveLength(8); + } finally { + local.close(); + remote.close(); + } + }); + + it("resumes a pack operation in a fresh iterable after every block", async () => { + const local = makePeer(); + const remote = makePeer(); + try { + seed(remote.db, 8); + const options = { + ...PACK_OPTIONS, + profile: { maxEntries: 3, maxBytes: 64 * 1024 * 1024 }, + }; + + for (let i = 0; i < 20; i++) { + const iterator = pullBlocks(local.db, remote.rpc, options)[Symbol.asyncIterator](); + const { value, done } = await iterator.next(); + if (done || value.complete) break; + } + + expect(names(local.db)).toHaveLength(8); + } finally { + local.close(); + remote.close(); + } + }); + + it("keeps the mode fixed for the life of an operation", async () => { + const local = makePeer(); + const remote = makePeer(); + try { + seed(remote.db, 8); + const options = { + ...PACK_OPTIONS, + profile: { maxEntries: 3, maxBytes: 64 * 1024 * 1024 }, + }; + + const iterator = pullBlocks(local.db, remote.rpc, options)[Symbol.asyncIterator](); + const first = await iterator.next(); + expect(first.value.mode).toBe("pack"); + // The persisted operation records the mode, so a resumed + // iterator cannot silently switch encodings mid-operation. + expect(readOperation(local.db, "default", "pull")?.mode).toBe("pack"); + } finally { + local.close(); + remote.close(); + } + }); + + it("deduplicates shared content across the pack", async () => { + const local = makePeer(); + const remote = makePeer(); + try { + const provider = new SQLiteWorkspaceProvider(remote.db); + for (let i = 0; i < 6; i++) { + provider.writeFileSync(`/same${i}.txt`, "identical content everywhere"); + } + + await drain(pullBlocks(local.db, remote.rpc, PACK_OPTIONS)); + + expect(names(local.db)).toHaveLength(6); + // One unique object backs all six paths. + const blobs = local.db.one<{ c: number }>("SELECT COUNT(*) AS c FROM vfs_blobs")?.c ?? 0; + expect(blobs).toBe(1); + } finally { + local.close(); + remote.close(); + } + }); + + it("applies deletions carried in a pack", async () => { + const local = makePeer(); + const remote = makePeer(); + try { + seed(remote.db, 6); + await drain(pullBlocks(local.db, remote.rpc, PACK_OPTIONS)); + + const provider = new SQLiteWorkspaceProvider(remote.db); + provider.unlinkSync("/f000.txt"); + provider.unlinkSync("/f001.txt"); + provider.unlinkSync("/f002.txt"); + provider.unlinkSync("/f003.txt"); + await drain(pullBlocks(local.db, remote.rpc, PACK_OPTIONS)); + + expect(names(local.db)).toEqual(["f004.txt", "f005.txt"]); + } finally { + local.close(); + remote.close(); + } + }); + + it("surfaces a corrupt pack as a protocol error without advancing", async () => { + const local = makePeer(); + const remote = makePeer(); + try { + seed(remote.db, 6); + + // Truncate the pack stream mid-flight. + const corrupting = new Proxy(remote.rpc as object, { + get(target, prop, receiver) { + if (prop === "fetchChangePack") { + return async (...args: unknown[]) => { + const real = (await ( + Reflect.get(target, prop, receiver) as (...a: unknown[]) => Promise<{ + stream: ReadableStream; + cursor: unknown; + }> + ).call(target, ...args)) as { stream: ReadableStream; cursor: unknown }; + const reader = real.stream.getReader(); + const truncated = new ReadableStream({ + async pull(controller) { + const { value, done } = await reader.read(); + if (done) { + controller.close(); + return; + } + // Emit a mangled first chunk and stop. + const broken = new Uint8Array(value.slice(0, Math.max(1, value.length >> 1))); + controller.enqueue(broken); + controller.close(); + }, + }); + return { ...real, stream: truncated }; + }; + } + return Reflect.get(target, prop, receiver); + }, + }) as SyncRPC; + + const iterator = pullBlocks(local.db, corrupting, PACK_OPTIONS)[Symbol.asyncIterator](); + await expect(iterator.next()).rejects.toThrow(/pack/i); + + // Nothing applied, cursor untouched. + expect(names(local.db)).toEqual([]); + } finally { + local.close(); + remote.close(); + } + }); +}); + +describe("pack mode push", () => { + it("ships a large local window as a pack and converges", async () => { + const local = makePeer(); + const remote = makePeer(); + try { + seed(local.db, 6); + + const seen = await drain(pushBlocks(local.db, remote.rpc, PACK_OPTIONS)); + + expect(seen[0].mode).toBe("pack"); + expect(names(remote.db)).toHaveLength(6); + } finally { + local.close(); + remote.close(); + } + }); + + it("transfers content correctly through a pushed pack", async () => { + const local = makePeer(); + const remote = makePeer(); + try { + seed(local.db, 5); + + await drain(pushBlocks(local.db, remote.rpc, PACK_OPTIONS)); + + const provider = new SQLiteWorkspaceProvider(remote.db); + expect(provider.readFileSync("/f003.txt", "utf8")).toBe(`3${"d".repeat(15)}`); + } finally { + local.close(); + remote.close(); + } + }); + + it("resumes a pushed pack across blocks", async () => { + const local = makePeer(); + const remote = makePeer(); + try { + seed(local.db, 8); + const options = { + ...PACK_OPTIONS, + profile: { maxEntries: 3, maxBytes: 64 * 1024 * 1024 }, + }; + + for (let i = 0; i < 20; i++) { + const iterator = pushBlocks(local.db, remote.rpc, options)[Symbol.asyncIterator](); + const { value, done } = await iterator.next(); + if (done || value.complete) break; + } + + expect(names(remote.db)).toHaveLength(8); + } finally { + local.close(); + remote.close(); + } + }); + + // A pre-command push carries exactly the large window that selects + // pack mode, and on a shim backend the spawned command reads from + // disk rather than the VFS. If the pack receiver skips the settle + // hook the command sees stale contents, so the hook has to fire on + // both transports. + it("settles the receiver's shim after a pack push", async () => { + const local = makePeer(); + const storage = new SQLiteTestStorage(); + const remoteDb = new Database(storage); + initializeSchema(remoteDb, () => 1000); + let calls = 0; + let namesAtHook: string[] = []; + const remoteRpc = createSyncServer(remoteDb, { + afterApply: () => { + calls += 1; + namesAtHook = names(remoteDb); + }, + }); + try { + seed(local.db, 8); + await drain(pushBlocks(local.db, remoteRpc, PACK_OPTIONS)); + + expect(calls).toBeGreaterThan(0); + // The entries must already be committed when the hook runs, + // mirroring the entry-mode guarantee. + expect(namesAtHook.length).toBeGreaterThan(0); + expect(names(remoteDb)).toHaveLength(8); + } finally { + local.close(); + storage.close(); + } + }); + + it("fails a pack push whose settle rejects, leaving the block unretired", async () => { + const local = makePeer(); + const storage = new SQLiteTestStorage(); + const remoteDb = new Database(storage); + initializeSchema(remoteDb, () => 1000); + let fail = true; + const remoteRpc = createSyncServer(remoteDb, { + afterApply: () => { + if (fail) throw new Error("disk full"); + }, + }); + try { + seed(local.db, 8); + // The sender advances its cursor on the acknowledgment and the + // caller spawns a command on it, so an unflushed pack must not + // acknowledge. + await expect(drain(pushBlocks(local.db, remoteRpc, PACK_OPTIONS))).rejects.toThrow( + "disk full", + ); + + // Once the shim recovers the same block replays and completes. + fail = false; + await drain(pushBlocks(local.db, remoteRpc, PACK_OPTIONS)); + expect(names(remoteDb)).toHaveLength(8); + } finally { + local.close(); + storage.close(); + } + }); + + it("does not settle the shim when a pack push carries no entries", async () => { + const local = makePeer(); + const storage = new SQLiteTestStorage(); + const remoteDb = new Database(storage); + initializeSchema(remoteDb, () => 1000); + let calls = 0; + const remoteRpc = createSyncServer(remoteDb, { + afterApply: () => { + calls += 1; + }, + }); + try { + // Nothing written locally, so no block carries entries. + await drain(pushBlocks(local.db, remoteRpc, PACK_OPTIONS)); + expect(calls).toBe(0); + } finally { + local.close(); + storage.close(); + } + }); +}); diff --git a/packages/rpc/src/sync-engine-push.test.ts b/packages/rpc/src/sync-engine-push.test.ts new file mode 100644 index 00000000..a30a99ef --- /dev/null +++ b/packages/rpc/src/sync-engine-push.test.ts @@ -0,0 +1,292 @@ +import { + Database, + initializeSchema, + readOperation, + readPushCursor, + SQLiteWorkspaceProvider, +} from "@cloudflare/dofs"; +import { SQLiteTestStorage } from "@cloudflare/dofs/testing"; +import { describe, expect, it } from "vitest"; + +import type { SyncRPC } from "./interface.js"; +import { createSyncServer } from "./server.js"; +import { pushBlocks, type SyncProgress } from "./sync-engine.js"; + +// Push runs the same operation and block machinery as pull, in the +// other direction. The property that distinguishes it: the local +// cursor may only advance through a remote acknowledgment. A push that +// advanced its cursor optimistically would silently drop data whenever +// an acknowledgment was lost. + +function makePeer(): { db: Database; rpc: SyncRPC; close: () => void } { + const storage = new SQLiteTestStorage(); + const db = new Database(storage); + initializeSchema(db, () => 1000); + const rpc = createSyncServer(db); + return { db, rpc, close: () => storage.close() }; +} + +function seedLocal(db: Database, count: number): void { + const provider = new SQLiteWorkspaceProvider(db); + for (let i = 0; i < count; i++) { + provider.writeFileSync(`/f${String(i).padStart(3, "0")}.txt`, `content-${i}`); + } +} + +function remoteNames(db: Database): string[] { + return db + .all<{ name: string }>("SELECT name FROM vfs_dirents WHERE parent_inode = 1 ORDER BY name") + .map((r) => r.name); +} + +async function pushOneBlock( + db: Database, + rpc: SyncRPC, + options: Parameters[2] = {}, +): Promise { + const iterator = pushBlocks(db, rpc, options)[Symbol.asyncIterator](); + const { value, done } = await iterator.next(); + return done ? undefined : value; +} + +async function drainPush( + db: Database, + rpc: SyncRPC, + options: Parameters[2] = {}, +): Promise { + const seen: SyncProgress[] = []; + for await (const progress of pushBlocks(db, rpc, options)) { + seen.push(progress); + if (seen.length > 200) throw new Error("push did not terminate"); + } + return seen; +} + +describe("restartable push engine", () => { + describe("completion contract", () => { + it("yields a single complete value for an empty window", async () => { + const local = makePeer(); + const remote = makePeer(); + try { + const seen = await drainPush(local.db, remote.rpc); + + expect(seen).toHaveLength(1); + expect(seen[0].complete).toBe(true); + expect(seen[0].direction).toBe("push"); + } finally { + local.close(); + remote.close(); + } + }); + + it("ships every local entry to the remote", async () => { + const local = makePeer(); + const remote = makePeer(); + try { + seedLocal(local.db, 4); + + await drainPush(local.db, remote.rpc); + + expect(remoteNames(remote.db)).toEqual(["f000.txt", "f001.txt", "f002.txt", "f003.txt"]); + } finally { + local.close(); + remote.close(); + } + }); + + it("splits a large window across blocks and still converges", async () => { + const local = makePeer(); + const remote = makePeer(); + try { + seedLocal(local.db, 6); + + const seen = await drainPush(local.db, remote.rpc, { + profile: { maxEntries: 2, maxBytes: 64 * 1024 * 1024 }, + }); + + expect(seen.length).toBeGreaterThanOrEqual(3); + expect(remoteNames(remote.db)).toHaveLength(6); + } finally { + local.close(); + remote.close(); + } + }); + }); + + describe("acknowledgment is the only cursor authority", () => { + it("advances the push cursor after a successful acknowledgment", async () => { + const local = makePeer(); + const remote = makePeer(); + try { + seedLocal(local.db, 3); + + const seen = await drainPush(local.db, remote.rpc); + + expect(readPushCursor(local.db)).toEqual(seen[seen.length - 1].cursor); + } finally { + local.close(); + remote.close(); + } + }); + + it("does not advance the cursor when the acknowledgment never arrives", async () => { + const local = makePeer(); + const remote = makePeer(); + try { + seedLocal(local.db, 3); + const before = readPushCursor(local.db); + + const failing = new Proxy(remote.rpc as object, { + get(target, prop, receiver) { + if (prop === "push") { + return async () => { + throw new Error("acknowledgment lost"); + }; + } + return Reflect.get(target, prop, receiver); + }, + }) as SyncRPC; + + await expect(pushOneBlock(local.db, failing, {})).rejects.toThrow("acknowledgment lost"); + + expect(readPushCursor(local.db)).toEqual(before); + } finally { + local.close(); + remote.close(); + } + }); + + // A lost acknowledgment after the remote already applied means the + // block is re-sent. The remote must absorb it rather than + // duplicating visible state. + it("does not duplicate remote state when a block is re-sent", async () => { + const local = makePeer(); + const remote = makePeer(); + try { + seedLocal(local.db, 3); + + // First attempt: the remote applies, then the acknowledgment + // is dropped on the way back. + let dropped = false; + const lossy = new Proxy(remote.rpc as object, { + get(target, prop, receiver) { + if (prop === "push") { + return async (...args: unknown[]) => { + const result = await ( + Reflect.get(target, prop, receiver) as (...a: unknown[]) => Promise + ).call(target, ...args); + if (!dropped) { + dropped = true; + throw new Error("acknowledgment lost in transit"); + } + return result; + }; + } + return Reflect.get(target, prop, receiver); + }, + }) as SyncRPC; + + await expect(pushOneBlock(local.db, lossy, {})).rejects.toThrow(); + // The retry re-sends the same block. + await drainPush(local.db, lossy, {}); + + expect(remoteNames(remote.db)).toEqual(["f000.txt", "f001.txt", "f002.txt"]); + } finally { + local.close(); + remote.close(); + } + }); + }); + + describe("restart from durable state", () => { + it("resumes from the durable cursor in a new iterable", async () => { + const local = makePeer(); + const remote = makePeer(); + try { + seedLocal(local.db, 5); + const profile = { maxEntries: 2, maxBytes: 64 * 1024 * 1024 }; + + await pushOneBlock(local.db, remote.rpc, { profile }); + const afterFirst = remoteNames(remote.db).length; + + await drainPush(local.db, remote.rpc, { profile }); + + expect(afterFirst).toBeLessThan(5); + expect(remoteNames(remote.db)).toHaveLength(5); + } finally { + local.close(); + remote.close(); + } + }); + + it("converges when the iterable is recreated after every block", async () => { + const local = makePeer(); + const remote = makePeer(); + try { + seedLocal(local.db, 6); + const profile = { maxEntries: 2, maxBytes: 64 * 1024 * 1024 }; + + for (let i = 0; i < 20; i++) { + const progress = await pushOneBlock(local.db, remote.rpc, { profile }); + if (progress?.complete) break; + } + + expect(remoteNames(remote.db)).toHaveLength(6); + } finally { + local.close(); + remote.close(); + } + }); + + it("deletes the operation row on completion", async () => { + const local = makePeer(); + const remote = makePeer(); + try { + seedLocal(local.db, 2); + + await drainPush(local.db, remote.rpc); + + expect(readOperation(local.db, "default", "push")).toBeUndefined(); + } finally { + local.close(); + remote.close(); + } + }); + }); + + describe("independence", () => { + it("keeps push and pull operations separate for one backend", async () => { + const local = makePeer(); + const remote = makePeer(); + try { + seedLocal(local.db, 4); + const profile = { maxEntries: 2, maxBytes: 64 * 1024 * 1024 }; + + await pushOneBlock(local.db, remote.rpc, { profile }); + + // A pending push must not be visible as a pull operation. + expect(readOperation(local.db, "default", "push")?.direction).toBe("push"); + expect(readOperation(local.db, "default", "pull")).toBeUndefined(); + } finally { + local.close(); + remote.close(); + } + }); + + it("keeps push cursors independent per backend", async () => { + const local = makePeer(); + const remote = makePeer(); + try { + seedLocal(local.db, 2); + + await drainPush(local.db, remote.rpc, { backend: "container" }); + + expect(readPushCursor(local.db, "container").rev).toBeGreaterThan(0); + expect(readPushCursor(local.db, "worker")).toEqual({ rev: 0, path: null }); + } finally { + local.close(); + remote.close(); + } + }); + }); +}); diff --git a/packages/rpc/src/sync-engine.test.ts b/packages/rpc/src/sync-engine.test.ts new file mode 100644 index 00000000..9c5381f9 --- /dev/null +++ b/packages/rpc/src/sync-engine.test.ts @@ -0,0 +1,592 @@ +import { + type ChangeCursor, + Database, + initializeSchema, + readFetchCursor, + readOperation, + readSkips, + SQLiteWorkspaceProvider, +} from "@cloudflare/dofs"; +import { SQLiteTestStorage } from "@cloudflare/dofs/testing"; +import { describe, expect, it } from "vitest"; + +import type { SyncRPC } from "./interface.js"; +import { createSyncServer } from "./server.js"; +import { pullBlocks, type SyncProgress } from "./sync-engine.js"; + +// The engine's contract is that durability lives in SQLite, not in the +// iterator object. Every test here either recreates the iterable or +// abandons one partway, because that is what a Durable Object eviction +// looks like from the caller's side: the generator is gone and only +// the operation row and the watermark survive. + +function makePeer(): { db: Database; rpc: SyncRPC; close: () => void } { + const storage = new SQLiteTestStorage(); + const db = new Database(storage); + initializeSchema(db, () => 1000); + const rpc = createSyncServer(db); + return { db, rpc, close: () => storage.close() }; +} + +function seedRemote(db: Database, count: number): void { + const provider = new SQLiteWorkspaceProvider(db); + for (let i = 0; i < count; i++) { + provider.writeFileSync(`/f${String(i).padStart(3, "0")}.txt`, `content-${i}`); + } +} + +function localNames(db: Database): string[] { + return db + .all<{ name: string }>("SELECT name FROM vfs_dirents WHERE parent_inode = 1 ORDER BY name") + .map((r) => r.name); +} + +// Consume one block and stop, the way a strict one-step alarm handler +// would. +async function pullOneBlock( + db: Database, + rpc: SyncRPC, + options: Parameters[2] = {}, +): Promise { + const iterator = pullBlocks(db, rpc, options)[Symbol.asyncIterator](); + const { value, done } = await iterator.next(); + return done ? undefined : value; +} + +async function drainPull( + db: Database, + rpc: SyncRPC, + options: Parameters[2] = {}, +): Promise { + const seen: SyncProgress[] = []; + for await (const progress of pullBlocks(db, rpc, options)) { + seen.push(progress); + if (seen.length > 200) throw new Error("pull did not terminate"); + } + return seen; +} + +describe("restartable pull engine", () => { + describe("completion contract", () => { + it("yields nothing but a complete value for an empty window", async () => { + const local = makePeer(); + const remote = makePeer(); + try { + const seen = await drainPull(local.db, remote.rpc); + + expect(seen).toHaveLength(1); + expect(seen[0].complete).toBe(true); + expect(seen[0].entries).toBe(0); + } finally { + local.close(); + remote.close(); + } + }); + + it("marks the last yielded value complete and then ends", async () => { + const local = makePeer(); + const remote = makePeer(); + try { + seedRemote(remote.db, 3); + + const seen = await drainPull(local.db, remote.rpc); + + expect(seen[seen.length - 1].complete).toBe(true); + expect(seen.filter((p) => p.complete)).toHaveLength(1); + } finally { + local.close(); + remote.close(); + } + }); + + it("applies every entry in the window", async () => { + const local = makePeer(); + const remote = makePeer(); + try { + seedRemote(remote.db, 4); + + await drainPull(local.db, remote.rpc); + + expect(localNames(local.db)).toEqual(["f000.txt", "f001.txt", "f002.txt", "f003.txt"]); + } finally { + local.close(); + remote.close(); + } + }); + + it("reports one progress value per committed block", async () => { + const local = makePeer(); + const remote = makePeer(); + try { + seedRemote(remote.db, 5); + + const seen = await drainPull(local.db, remote.rpc, { + profile: { maxEntries: 2, maxBytes: 64 * 1024 * 1024 }, + }); + + // 5 entries at 2 per block: three blocks carry entries. + expect(seen.length).toBeGreaterThanOrEqual(3); + expect(seen.reduce((total, p) => total + p.entries, 0)).toBe(5); + } finally { + local.close(); + remote.close(); + } + }); + }); + + describe("restart from durable state", () => { + it("persists a cursor for every yielded block", async () => { + const local = makePeer(); + const remote = makePeer(); + try { + seedRemote(remote.db, 4); + + const first = await pullOneBlock(local.db, remote.rpc, { + profile: { maxEntries: 2, maxBytes: 64 * 1024 * 1024 }, + }); + + expect(first).toBeDefined(); + expect(readFetchCursor(local.db)).toEqual(first?.cursor); + } finally { + local.close(); + remote.close(); + } + }); + + // The plan's central claim: a fresh iterable resumes from SQLite + // and does not depend on the iterator object that came before it. + it("resumes from the durable cursor in a brand new iterable", async () => { + const local = makePeer(); + const remote = makePeer(); + try { + seedRemote(remote.db, 5); + const profile = { maxEntries: 2, maxBytes: 64 * 1024 * 1024 }; + + // One block, then the iterator is abandoned entirely. + await pullOneBlock(local.db, remote.rpc, { profile }); + const afterFirst = localNames(local.db); + + // A completely separate iterable finishes the job. + await drainPull(local.db, remote.rpc, { profile }); + + expect(afterFirst.length).toBeLessThan(5); + expect(localNames(local.db)).toHaveLength(5); + } finally { + local.close(); + remote.close(); + } + }); + + it("converges when the iterable is recreated after every single block", async () => { + const local = makePeer(); + const remote = makePeer(); + try { + seedRemote(remote.db, 6); + const profile = { maxEntries: 2, maxBytes: 64 * 1024 * 1024 }; + + // Simulated eviction between every block: never reuse an + // iterator, always build a new one. + for (let i = 0; i < 20; i++) { + const progress = await pullOneBlock(local.db, remote.rpc, { profile }); + if (progress?.complete) break; + } + + expect(localNames(local.db)).toHaveLength(6); + } finally { + local.close(); + remote.close(); + } + }); + + it("advances the cursor monotonically across restarts", async () => { + const local = makePeer(); + const remote = makePeer(); + try { + seedRemote(remote.db, 6); + const profile = { maxEntries: 2, maxBytes: 64 * 1024 * 1024 }; + + const cursors: ChangeCursor[] = []; + for (let i = 0; i < 20; i++) { + const progress = await pullOneBlock(local.db, remote.rpc, { profile }); + if (progress === undefined) break; + cursors.push(progress.cursor); + if (progress.complete) break; + } + + for (let i = 1; i < cursors.length; i++) { + const previous = cursors[i - 1]; + const next = cursors[i]; + const forward = + next.rev > previous.rev || (next.rev === previous.rev && next.path !== previous.path); + expect(forward || next.rev === previous.rev).toBe(true); + expect(next.rev).toBeGreaterThanOrEqual(previous.rev); + } + } finally { + local.close(); + remote.close(); + } + }); + + it("is a no-op once the window is already drained", async () => { + const local = makePeer(); + const remote = makePeer(); + try { + seedRemote(remote.db, 3); + await drainPull(local.db, remote.rpc); + const cursor = readFetchCursor(local.db); + + const second = await drainPull(local.db, remote.rpc); + + expect(second).toHaveLength(1); + expect(second[0].entries).toBe(0); + expect(readFetchCursor(local.db)).toEqual(cursor); + } finally { + local.close(); + remote.close(); + } + }); + }); + + describe("operation lifecycle", () => { + it("deletes the operation row once the target is reached", async () => { + const local = makePeer(); + const remote = makePeer(); + try { + seedRemote(remote.db, 3); + + await drainPull(local.db, remote.rpc); + + expect(readOperation(local.db, "default", "pull")).toBeUndefined(); + } finally { + local.close(); + remote.close(); + } + }); + + it("keeps a pending operation while blocks remain", async () => { + const local = makePeer(); + const remote = makePeer(); + try { + seedRemote(remote.db, 6); + + await pullOneBlock(local.db, remote.rpc, { + profile: { maxEntries: 2, maxBytes: 64 * 1024 * 1024 }, + }); + + const operation = readOperation(local.db, "default", "pull"); + expect(operation?.status).toBe("pending"); + expect(operation?.target).toBeDefined(); + } finally { + local.close(); + remote.close(); + } + }); + + it("reuses the same generation across restarts of one operation", async () => { + const local = makePeer(); + const remote = makePeer(); + try { + seedRemote(remote.db, 6); + const profile = { maxEntries: 2, maxBytes: 64 * 1024 * 1024 }; + + const first = await pullOneBlock(local.db, remote.rpc, { profile }); + const second = await pullOneBlock(local.db, remote.rpc, { profile }); + + expect(second?.generation).toBe(first?.generation); + } finally { + local.close(); + remote.close(); + } + }); + + // A busy workspace must not produce an endless iterable: changes + // above the fixed target belong to the next operation. + it("does not extend a running operation with changes above its target", async () => { + const local = makePeer(); + const remote = makePeer(); + try { + seedRemote(remote.db, 4); + const profile = { maxEntries: 2, maxBytes: 64 * 1024 * 1024 }; + + await pullOneBlock(local.db, remote.rpc, { profile }); + // New remote work lands mid-operation. + const provider = new SQLiteWorkspaceProvider(remote.db); + provider.writeFileSync("/late.txt", "late"); + + const seen = await drainPull(local.db, remote.rpc, { profile }); + + // The operation that was already running completed without + // waiting for /late.txt. + expect(seen[seen.length - 1].complete).toBe(true); + expect(localNames(local.db)).not.toContain("late.txt"); + } finally { + local.close(); + remote.close(); + } + }); + + it("picks up changes above the previous target on the next operation", async () => { + const local = makePeer(); + const remote = makePeer(); + try { + seedRemote(remote.db, 2); + await drainPull(local.db, remote.rpc); + + const provider = new SQLiteWorkspaceProvider(remote.db); + provider.writeFileSync("/late.txt", "late"); + await drainPull(local.db, remote.rpc); + + expect(localNames(local.db)).toContain("late.txt"); + } finally { + local.close(); + remote.close(); + } + }); + }); + + describe("progress shape", () => { + it("carries the backend, direction, and fixed target", async () => { + const local = makePeer(); + const remote = makePeer(); + try { + seedRemote(remote.db, 2); + + const progress = await pullOneBlock(local.db, remote.rpc, { backend: "container" }); + + expect(progress?.backend).toBe("container"); + expect(progress?.direction).toBe("pull"); + expect(progress?.targetCursor).toBeDefined(); + expect(progress?.operationId).toBeTruthy(); + } finally { + local.close(); + remote.close(); + } + }); + + it("counts entries and bytes for the block it committed", async () => { + const local = makePeer(); + const remote = makePeer(); + try { + seedRemote(remote.db, 3); + + const seen = await drainPull(local.db, remote.rpc); + + expect(seen.reduce((t, p) => t + p.entries, 0)).toBe(3); + expect(seen.reduce((t, p) => t + p.bytes, 0)).toBeGreaterThan(0); + } finally { + local.close(); + remote.close(); + } + }); + + it("keeps cursors independent per backend", async () => { + const local = makePeer(); + const remote = makePeer(); + try { + seedRemote(remote.db, 2); + + await drainPull(local.db, remote.rpc, { backend: "container" }); + + expect(readFetchCursor(local.db, "container").rev).toBeGreaterThan(0); + expect(readFetchCursor(local.db, "worker")).toEqual({ rev: 0, path: null }); + } finally { + local.close(); + remote.close(); + } + }); + }); + + describe("cancellation", () => { + it("leaves the operation resumable after breaking out of iteration", async () => { + const local = makePeer(); + const remote = makePeer(); + try { + seedRemote(remote.db, 6); + const profile = { maxEntries: 2, maxBytes: 64 * 1024 * 1024 }; + + for await (const progress of pullBlocks(local.db, remote.rpc, { profile })) { + expect(progress.complete).toBe(false); + break; + } + + const operation = readOperation(local.db, "default", "pull"); + expect(operation?.status).toBe("pending"); + + await drainPull(local.db, remote.rpc, { profile }); + expect(localNames(local.db)).toHaveLength(6); + } finally { + local.close(); + remote.close(); + } + }); + }); + + describe("interrupted block recovery", () => { + // A block marker that survives with no cursor progress means the + // previous execution died mid-block, so the sizing profile shrinks. + it("shrinks the block profile after an interrupted block", async () => { + const local = makePeer(); + const remote = makePeer(); + try { + seedRemote(remote.db, 6); + const profile = { maxEntries: 4, maxBytes: 64 * 1024 * 1024 }; + + // Fail the first block after the target is fixed but before + // any cursor advances. + let calls = 0; + const flaky = new Proxy(remote.rpc as object, { + get(target, prop, receiver) { + if (prop === "fetchChanges") { + return async (...args: unknown[]) => { + calls += 1; + if (calls === 1) throw new Error("transport interrupted"); + return ( + Reflect.get(target, prop, receiver) as (...a: unknown[]) => Promise + ).call(target, ...args); + }; + } + return Reflect.get(target, prop, receiver); + }, + }) as SyncRPC; + + await expect(pullOneBlock(local.db, flaky, { profile })).rejects.toThrow(); + + const operation = readOperation(local.db, "default", "pull"); + expect(operation?.status).toBe("pending"); + + // The retry detects the stale marker and halves the profile. + await pullOneBlock(local.db, flaky, { profile }); + const after = readOperation(local.db, "default", "pull"); + expect(after === undefined || after.profile.maxEntries <= 4).toBe(true); + } finally { + local.close(); + remote.close(); + } + }); + + it("leaves the operation pending after a recoverable transport error", async () => { + const local = makePeer(); + const remote = makePeer(); + try { + seedRemote(remote.db, 4); + const exploding = new Proxy(remote.rpc as object, { + get(target, prop, receiver) { + if (prop === "fetchChanges") { + return async () => { + throw new Error("transport interrupted"); + }; + } + return Reflect.get(target, prop, receiver); + }, + }) as SyncRPC; + + await expect(pullOneBlock(local.db, exploding, {})).rejects.toThrow( + "transport interrupted", + ); + + // Pending, not failed: recreating the iterable retries. + expect(readOperation(local.db, "default", "pull")?.status).toBe("pending"); + expect(readFetchCursor(local.db)).toEqual({ rev: 0, path: null }); + } finally { + local.close(); + remote.close(); + } + }); + }); + + describe("read-only mount rejections", () => { + it("records a rejected entry durably and still advances", async () => { + const local = makePeer(); + const remote = makePeer(); + try { + seedRemote(remote.db, 2); + // Mount /f000.txt's parent as read-only so the apply refuses + // the incoming entries. + local.db.run( + "INSERT INTO _vfs_mounts (root, kind, indexed, mode) VALUES (?, ?, ?, ?)", + "/", + "test", + 1, + "read-only", + ); + + const seen = await drainPull(local.db, remote.rpc); + + expect(seen[seen.length - 1].complete).toBe(true); + expect(seen.some((p) => p.skipped > 0)).toBe(true); + const skips = readSkips(local.db, "default", "pull"); + expect(skips.length).toBeGreaterThan(0); + expect(skips[0].reason).toBe("read-only"); + } finally { + local.close(); + remote.close(); + } + }); + + it("does not loop forever on a rejected entry", async () => { + const local = makePeer(); + const remote = makePeer(); + try { + seedRemote(remote.db, 2); + local.db.run( + "INSERT INTO _vfs_mounts (root, kind, indexed, mode) VALUES (?, ?, ?, ?)", + "/", + "test", + 1, + "read-only", + ); + + // drainPull throws if it exceeds 200 blocks. + const seen = await drainPull(local.db, remote.rpc); + + expect(seen[seen.length - 1].complete).toBe(true); + } finally { + local.close(); + remote.close(); + } + }); + }); + + describe("same-isolate handoff", () => { + it("joins one generation when two iterators run concurrently", async () => { + const local = makePeer(); + const remote = makePeer(); + try { + seedRemote(remote.db, 6); + const profile = { maxEntries: 2, maxBytes: 64 * 1024 * 1024 }; + + const [a, b] = await Promise.all([ + pullOneBlock(local.db, remote.rpc, { profile }), + pullOneBlock(local.db, remote.rpc, { profile }), + ]); + + // Both callers observe the same operation rather than racing + // to create competing targets. + expect(a?.generation).toBe(b?.generation); + expect(a?.operationId).toBe(b?.operationId); + } finally { + local.close(); + remote.close(); + } + }); + + it("converges when concurrent iterators drive the same backend", async () => { + const local = makePeer(); + const remote = makePeer(); + try { + seedRemote(remote.db, 6); + const profile = { maxEntries: 2, maxBytes: 64 * 1024 * 1024 }; + + await Promise.all([ + drainPull(local.db, remote.rpc, { profile }), + drainPull(local.db, remote.rpc, { profile }), + ]); + + expect(localNames(local.db)).toHaveLength(6); + } finally { + local.close(); + remote.close(); + } + }); + }); +}); diff --git a/packages/rpc/src/sync-engine.ts b/packages/rpc/src/sync-engine.ts new file mode 100644 index 00000000..b26db85d --- /dev/null +++ b/packages/rpc/src/sync-engine.ts @@ -0,0 +1,861 @@ +// Restartable sync engine. +// +// One `next()` performs at most one complete block: plan it, transfer +// it, apply it, persist the cursor, yield. Nothing live crosses a +// yield boundary — every RPC stream is drained and disposed before the +// value is handed to the caller — because the caller may not come back. +// A Durable Object can be evicted between two `next()` calls, and the +// iterator object does not survive that. The operation row and the +// watermark do. +// +// The engine therefore never treats the iterator as state. Each step +// re-reads the durable cursor and the durable operation, which is what +// makes `pullBlocks(...)` and a freshly constructed `pullBlocks(...)` +// interchangeable. +// +// This module owns no timer and no alarm. The application decides when +// to call `next()`, from a request, an alarm, a queue, or a Workflow. + +import { + applyChanges, + assertAppliedPushCursor, + type BlockProfile, + type ChangeCursor, + type ChangeEntry, + clearBlockMarker, + compareChangeCursors, + completeOperation, + currentRev, + type Database, + DEFAULT_BLOCK_PROFILE, + decodeChangePack, + encodeChangePack, + fixTarget, + hasObjects, + markBlockStarted, + openOperation, + PACK_THRESHOLD_BYTES, + PACK_THRESHOLD_ENTRIES, + planBlock, + pruneSkips, + readFetchCursor, + readOperation, + readPushCursor, + recordSkip, + type SyncDirection, + type SyncMode, + selectMode, + shrinkBlockProfile, + stageBlob, + writeFetchCursor, + writePushCursor, +} from "@cloudflare/dofs"; + +import type { SyncRPC } from "./interface.js"; + +export interface SyncProgress { + readonly operationId: string; + readonly generation: string; + readonly backend: string; + readonly direction: SyncDirection; + readonly mode: SyncMode; + readonly cursor: ChangeCursor; + readonly targetCursor: ChangeCursor; + readonly entries: number; + readonly bytes: number; + // Entries the receiver deliberately refused, most often a read-only + // mount conflict. Also written to the durable skip log. + readonly skipped: number; + readonly complete: boolean; +} + +export interface PullBlocksOptions { + readonly backend?: string; + readonly ignore?: string[]; + // Internal sizing override. Not exposed on the public Workspace + // surface; present so tests can force multi-block runs without + // materialising thousands of files. + readonly profile?: BlockProfile; + readonly now?: () => number; + // Mode-selection overrides, internal for the same reason: crossing + // the production thresholds in a test would mean writing 20,000 + // files. + readonly thresholdEntries?: number; + readonly thresholdBytes?: number; +} + +export type PushBlocksOptions = PullBlocksOptions; + +const DEFAULT_BACKEND = "default"; + +function hex(bytes: Uint8Array): string { + let s = ""; + for (let i = 0; i < bytes.byteLength; i++) s += bytes[i].toString(16).padStart(2, "0"); + return s; +} + +function maybeDispose(value: unknown): void { + const d = (value as { [Symbol.dispose]?: () => void } | null | undefined)?.[Symbol.dispose]; + if (typeof d === "function") d.call(value); +} + +function entryCursor(entry: ChangeEntry): ChangeCursor { + return { rev: entry.rev, path: entry.path }; +} + +// In-isolate join table. Two iterators in the same isolate driving the +// same backend and direction share one in-flight block instead of +// duplicating the transfer. This is an optimization only: after an +// eviction the map is empty and the durable operation row is what +// makes the next iterator correct. +const active = new Map }>(); + +function activeKey(backend: string, direction: SyncDirection): string { + return `${backend}:${direction}`; +} + +// Ask the source how big the window is, and pick a transport. +// +// The probe is a pack request bounded at the threshold: the source +// plans the window once and reports the entry count it would carry. +// Filling the probe is itself the signal that the window is at or past +// the threshold, so this answers the mode question without draining a +// window that may hold tens of thousands of entries. The probe's stream +// is cancelled unread — only the counts matter here. +async function selectPullMode( + remote: SyncRPC, + input: { + after: ChangeCursor; + target: ChangeCursor; + profile: BlockProfile; + ignore?: string[]; + thresholdEntries?: number; + thresholdBytes?: number; + generation: string; + }, +): Promise { + const thresholdEntries = input.thresholdEntries ?? PACK_THRESHOLD_ENTRIES; + const thresholdBytes = input.thresholdBytes ?? PACK_THRESHOLD_BYTES; + + // A peer that predates pack transport keeps working in entry mode. + if (typeof remote.fetchChangePack !== "function") return "entries"; + + const probe = await remote.fetchChangePack?.({ + after: input.after, + through: input.target, + ...(input.ignore === undefined ? {} : { ignore: input.ignore }), + maxEntries: thresholdEntries, + maxBytes: thresholdBytes, + generation: input.generation, + }); + if (probe === undefined) return "entries"; + try { + return probe.entryCount >= thresholdEntries || !probe.drained ? "pack" : "entries"; + } finally { + await probe.stream.cancel().catch(() => {}); + maybeDispose(probe); + } +} + +// Transfer and apply one pack block. +// +// The pack is decoded and validated before anything is applied. A pack +// that fails validation leaves no partial state and no cursor movement, +// because the alternative — advancing over entries that never arrived — +// loses data silently. +async function runPullPackBlock( + db: Database, + remote: SyncRPC, + backend: string, + options: PullBlocksOptions, + operation: { generation: string }, + after: ChangeCursor, + target: ChangeCursor, + profile: BlockProfile, +): Promise<{ + entries: number; + bytes: number; + skipped: number; + cursor: ChangeCursor; +}> { + const now = options.now ?? (() => Date.now()); + const response = await remote.fetchChangePack?.({ + after, + through: target, + ...(options.ignore === undefined ? {} : { ignore: options.ignore }), + maxEntries: profile.maxEntries, + maxBytes: profile.maxBytes, + generation: operation.generation, + }); + if (response === undefined) { + throw new Error("sync: peer withdrew pack transport mid-operation"); + } + + let decoded: Awaited>; + try { + decoded = await decodeChangePack(response.stream); + } finally { + maybeDispose(response); + } + + // Stage every object the pack carried before applying metadata, so a + // crash between the two replays as a no-op rather than as a file with + // missing bytes. + let bytes = 0; + for (const [key, payload] of decoded.objects) { + stageBlob(db, hexToBytes(key), payload, now()); + bytes += payload.byteLength; + } + + const result = await applyChanges(db, decoded.entries, new Map(), { + source: "upstream", + backend, + }); + for (const skip of result.skipped) { + recordSkip(db, backend, "pull", operation.generation, skip.path, skip.reason, now()); + } + + // The footer's block cursor is authoritative: it is what the encoder + // committed to, and it already accounts for a drained window jumping + // to the target. + return { + entries: decoded.entries.length, + bytes, + skipped: result.skipped.length, + cursor: decoded.footer.blockCursor, + }; +} + +function hexToBytes(hex: string): Uint8Array { + const out = new Uint8Array(hex.length / 2); + for (let i = 0; i < out.length; i++) { + out[i] = Number.parseInt(hex.slice(i * 2, i * 2 + 2), 16); + } + return out; +} + +// Drive one pull block to completion, or throw. +// +// Ordering is the whole safety argument: metadata is applied before the +// cursor moves, and the cursor never moves past an entry that has not +// been applied or deliberately recorded as skipped. A crash anywhere +// before the cursor write replays the block, which the receiver +// absorbs because staged objects and applied entries are idempotent. +async function runPullBlock( + db: Database, + remote: SyncRPC, + backend: string, + options: PullBlocksOptions, +): Promise { + const now = options.now ?? (() => Date.now()); + const profile = options.profile ?? DEFAULT_BLOCK_PROFILE; + + // 1. Read or create the operation. A pending row is joined, so + // concurrent callers converge on one fixed target. + const opened = openOperation(db, backend, "pull", now()); + let operation = opened.operation; + + if (operation.status === "capturing") { + // Two-phase target capture: the remote settle call cannot join a + // local transaction, so the target is read here and committed + // conditionally. An eviction in between leaves a capturing row + // that the next iterator takes over. + pruneSkips(db, backend, "pull", operation.generation); + // The pull target is the source's change head, not its own inbound + // fetch cursor: we are pulling everything the source has produced. + // A settled read also flushes writes still buffered in the shim, + // so the target covers them instead of stranding them until the + // next operation. + const settled = await remote.watermarks({ settle: true }); + // Decide entries versus pack once, for the whole operation. The + // mode is then fixed: a block's encoding must not depend on when it + // was requested, or a replayed block would not match the original. + const target = { rev: settled.currentRev, path: null }; + const mode = await selectPullMode(remote, { + after: readFetchCursor(db, backend), + target, + profile, + ...(options.ignore === undefined ? {} : { ignore: options.ignore }), + ...(options.thresholdEntries === undefined + ? {} + : { thresholdEntries: options.thresholdEntries }), + ...(options.thresholdBytes === undefined ? {} : { thresholdBytes: options.thresholdBytes }), + generation: operation.generation, + }); + const promoted = fixTarget(db, backend, "pull", operation.generation, { target, mode }, now()); + if (!promoted) { + // Another caller replaced this operation while the target was in + // flight. Its target is authoritative; drop ours and use theirs. + const current = readOperation(db, backend, "pull"); + if (current === undefined) { + throw new Error("sync: operation vanished during target capture"); + } + operation = current; + } else { + const current = readOperation(db, backend, "pull"); + if (current === undefined) { + throw new Error("sync: operation vanished after target capture"); + } + operation = current; + } + } + + const target = operation.target; + if (target === undefined) { + throw new Error("sync: pending operation has no target"); + } + + const after = readFetchCursor(db, backend); + + // Already at the target: the operation is done. Delete the row and + // report completion so a one-step alarm can stop without a second + // next() call. + if (compareChangeCursors(after, target) >= 0) { + completeOperation(db, backend, "pull", operation.generation); + return { + operationId: operation.generation, + generation: operation.generation, + backend, + direction: "pull", + mode: operation.mode ?? "entries", + cursor: after, + targetCursor: target, + entries: 0, + bytes: 0, + skipped: 0, + complete: true, + }; + } + + // A surviving marker with no cursor progress means the previous + // execution died mid-block at this same cursor. Shrink the profile + // before trying again so a block that did not fit is not retried at + // the same size forever. + const effectiveProfile = + operation.blockAfter !== undefined && compareChangeCursors(operation.blockAfter, after) === 0 + ? (shrinkBlockProfile(db, backend, "pull", operation.generation, now()) ?? profile) + : profile; + + markBlockStarted(db, backend, "pull", operation.generation, after, now()); + + // Pack mode: one compressed stream carries entries and their objects + // together, so there are no per-block object round trips. + if (operation.mode === "pack") { + const packed = await runPullPackBlock( + db, + remote, + backend, + options, + operation, + after, + target, + effectiveProfile, + ); + if (compareChangeCursors(packed.cursor, after) > 0) { + writeFetchCursor(db, packed.cursor, backend); + } + clearBlockMarker(db, backend, "pull", operation.generation, now()); + const packComplete = compareChangeCursors(packed.cursor, target) >= 0; + if (packComplete) { + completeOperation(db, backend, "pull", operation.generation); + } + return { + operationId: operation.generation, + generation: operation.generation, + backend, + direction: "pull", + mode: "pack", + cursor: packed.cursor, + targetCursor: target, + entries: packed.entries, + bytes: packed.bytes, + skipped: packed.skipped, + complete: packComplete, + }; + } + + // 2. Ask the source for one deterministic block bounded by the + // fixed target. The remote streams entries; we stop reading at + // the profile bound and let the rest come in the next block. + const fetchResult = await remote.fetchChanges({ + after, + through: target, + ...(options.ignore === undefined ? {} : { ignore: options.ignore }), + }); + + const entries: ChangeEntry[] = []; + let bytes = 0; + let drained = false; + const reader = fetchResult.stream.getReader(); + try { + while (entries.length < effectiveProfile.maxEntries) { + const { value, done } = await reader.read(); + if (done) { + drained = true; + break; + } + entries.push(value); + if (value.kind === "file") { + for (const chunk of value.chunks) bytes += chunk.size; + } + // The first entry is always accepted so one oversized file still + // makes progress; later entries stop the block at the bound. + if (entries.length > 1 && bytes >= effectiveProfile.maxBytes) break; + } + } finally { + reader.releaseLock(); + // No live stream reader crosses the yield boundary. + await fetchResult.stream.cancel().catch(() => {}); + maybeDispose(fetchResult); + } + + // 3. Pull the object bytes this block needs. Deduplicated within + // the block; duplicates across blocks are absorbed by the + // content-addressed store. Bytes are staged into the blob store + // rather than handed to applyChanges in memory, so a block's peak + // memory stays bounded by its metadata. + const wanted: Uint8Array[] = []; + const seen = new Set(); + for (const entry of entries) { + if (entry.kind !== "file") continue; + for (const chunk of entry.chunks) { + const key = hex(chunk.hash); + if (seen.has(key)) continue; + seen.add(key); + wanted.push(chunk.hash); + } + } + if (wanted.length > 0) { + // Ask for only what the source actually holds and we actually + // lack. Requesting a hash the source has dropped would throw + // EUNKNOWN_HASH and fail an otherwise applicable block. + const remoteHas = new Set((await remote.hasObjects(wanted)).map(hex)); + const localHas = new Set(hasObjects(db, wanted).map(hex)); + const missing = wanted.filter((h) => remoteHas.has(hex(h)) && !localHas.has(hex(h))); + if (missing.length > 0) { + const objectStream = await remote.fetchObjects(missing); + const objectReader = objectStream.getReader(); + try { + while (true) { + const { value, done } = await objectReader.read(); + if (done) break; + // Stage before apply so a crash between the two replays as a + // no-op rather than as a missing object. + stageBlob(db, value.hash, value.bytes, now()); + } + } finally { + objectReader.releaseLock(); + maybeDispose(objectStream); + } + } + } + + // 4. Apply metadata before the cursor moves. Chunk bytes come from + // the staged blob store, not from an in-memory map. + const result = await applyChanges(db, entries, new Map(), { + source: "upstream", + backend, + }); + + // 5. Record refusals durably. The cursor advances past them so the + // operation cannot stall on an entry that can never apply, which + // would otherwise loop forever on the same rejection. + for (const skip of result.skipped) { + recordSkip(db, backend, "pull", operation.generation, skip.path, skip.reason, now()); + } + + // 6. Advance the cursor. Drained means every change through the + // target was offered, so the cursor jumps to the target itself. + const lastEntry = entries[entries.length - 1]; + const cursor = drained ? target : lastEntry === undefined ? after : entryCursor(lastEntry); + if (compareChangeCursors(cursor, after) > 0) { + writeFetchCursor(db, cursor, backend); + } + clearBlockMarker(db, backend, "pull", operation.generation, now()); + + // 7. Complete when the cursor reaches the fixed target. + const complete = compareChangeCursors(cursor, target) >= 0; + if (complete) { + completeOperation(db, backend, "pull", operation.generation); + } + + return { + operationId: operation.generation, + generation: operation.generation, + backend, + direction: "pull", + mode: operation.mode ?? "entries", + cursor, + targetCursor: target, + entries: entries.length, + bytes, + skipped: result.skipped.length, + complete, + }; +} + +// Join an in-flight block from the same isolate when one exists for +// this backend and direction, otherwise start one. +async function nextPullBlock( + db: Database, + remote: SyncRPC, + backend: string, + options: PullBlocksOptions, +): Promise { + const key = activeKey(backend, "pull"); + const existing = active.get(key); + if (existing !== undefined) { + // Someone else in this isolate is already driving a block for this + // backend. Wait for it rather than opening a duplicate transfer. + try { + return await existing.settled; + } catch { + // The other driver failed. Fall through and try our own block + // from whatever durable state it left behind. + } + } + + const settled = runPullBlock(db, remote, backend, options); + const operation = readOperation(db, backend, "pull"); + active.set(key, { generation: operation?.generation ?? "", settled }); + try { + return await settled; + } finally { + if (active.get(key)?.settled === settled) active.delete(key); + } +} + +// Drive one push block to completion, or throw. +// +// The mirror of runPullBlock with one asymmetry that matters: the local +// cursor advances only through the remote's acknowledgment. A push that +// advanced optimistically would silently drop data every time an +// acknowledgment was lost, because the next block would start above +// entries the receiver never applied. +// +// Push can capture its target inside the creation transaction — the +// target is the local change head, so there is no remote call to wait +// on and no 'capturing' phase. +async function runPushBlock( + db: Database, + remote: SyncRPC, + backend: string, + options: PushBlocksOptions, +): Promise { + const now = options.now ?? (() => Date.now()); + const profile = options.profile ?? DEFAULT_BLOCK_PROFILE; + + const opened = openOperation(db, backend, "push", now()); + let operation = opened.operation; + + if (operation.status === "capturing") { + pruneSkips(db, backend, "push", operation.generation); + // Push plans locally, so mode selection is a local probe rather + // than a remote round trip. selectMode returns the first block from + // the same pass, which is why this does not scan the window twice. + const target = { rev: currentRev(db), path: null }; + const decision = await selectMode(db, { + after: readPushCursor(db, backend), + through: target, + profile, + ...(options.ignore === undefined ? {} : { ignore: options.ignore }), + ...(options.thresholdEntries === undefined + ? {} + : { thresholdEntries: options.thresholdEntries }), + ...(options.thresholdBytes === undefined ? {} : { thresholdBytes: options.thresholdBytes }), + }); + // A peer without pack transport keeps receiving entry blocks. + const mode = typeof remote.applyChangePack === "function" ? decision.mode : "entries"; + const promoted = fixTarget(db, backend, "push", operation.generation, { target, mode }, now()); + const current = readOperation(db, backend, "push"); + if (current === undefined) { + throw new Error("sync: push operation vanished during target capture"); + } + if (!promoted && current.status === "capturing") { + throw new Error("sync: push target capture lost its generation"); + } + operation = current; + } + + const target = operation.target; + if (target === undefined) { + throw new Error("sync: pending push operation has no target"); + } + + const after = readPushCursor(db, backend); + + if (compareChangeCursors(after, target) >= 0) { + completeOperation(db, backend, "push", operation.generation); + return { + operationId: operation.generation, + generation: operation.generation, + backend, + direction: "push", + mode: operation.mode ?? "entries", + cursor: after, + targetCursor: target, + entries: 0, + bytes: 0, + skipped: 0, + complete: true, + }; + } + + const effectiveProfile = + operation.blockAfter !== undefined && compareChangeCursors(operation.blockAfter, after) === 0 + ? (shrinkBlockProfile(db, backend, "push", operation.generation, now()) ?? profile) + : profile; + + markBlockStarted(db, backend, "push", operation.generation, after, now()); + + // Plan the block locally. planBlock owns the ordered-prefix and + // byte-bound rules, so push and pull select blocks identically. + const planned = await planBlock(db, { + after, + through: target, + profile: effectiveProfile, + ...(options.ignore === undefined ? {} : { ignore: options.ignore }), + }); + + // Pack mode: one compressed stream carries the entries and every + // object they reference, so no hasObjects probe or separate + // pushObjects call is needed. + if (operation.mode === "pack") { + const stream = encodeChangePack(db, { + block: planned, + after, + target, + generation: operation.generation, + }); + const ack = await remote.applyChangePack?.({ + generation: operation.generation, + stream, + }); + if (ack === undefined) { + throw new Error("sync: peer withdrew pack transport mid-operation"); + } + // The receiver's echoed cursor is the only authority for advancing. + assertAppliedPushCursor(ack.appliedPushCursor, planned.cursor); + if (compareChangeCursors(planned.cursor, after) > 0) { + writePushCursor(db, planned.cursor, backend); + } + clearBlockMarker(db, backend, "push", operation.generation, now()); + const packComplete = compareChangeCursors(planned.cursor, target) >= 0; + if (packComplete) { + completeOperation(db, backend, "push", operation.generation); + } + return { + operationId: operation.generation, + generation: operation.generation, + backend, + direction: "push", + mode: "pack", + cursor: planned.cursor, + targetCursor: target, + entries: planned.entries.length, + bytes: planned.objectBytes, + skipped: 0, + complete: packComplete, + }; + } + + // Ship the bytes the receiver lacks before the entries that + // reference them, so the receiver never sees an entry whose content + // it cannot resolve. + let bytes = 0; + if (planned.objects.length > 0) { + const have = new Set((await remote.hasObjects(planned.objects.map((o) => o.hash))).map(hex)); + const missing = planned.objects.filter((o) => !have.has(hex(o.hash))); + if (missing.length > 0) { + const pending = [...missing]; + const objectStream = new ReadableStream<{ hash: Uint8Array; bytes: Uint8Array }>({ + pull(controller) { + const next = pending.shift(); + if (next === undefined) { + controller.close(); + return; + } + const row = db.one<{ bytes: Uint8Array }>( + "SELECT bytes FROM vfs_blob_bytes WHERE hash = ?", + next.hash, + ); + if (row === undefined) { + controller.error(new Error(`sync: missing local blob ${hex(next.hash)}`)); + return; + } + bytes += row.bytes.byteLength; + controller.enqueue({ hash: next.hash, bytes: row.bytes }); + }, + }); + await remote.pushObjects(objectStream); + } + } + + const entryStream = new ReadableStream({ + start(controller) { + for (const entry of planned.entries) controller.enqueue(entry); + controller.close(); + }, + }); + + const response = await remote.push({ + senderRev: target.rev, + senderCursor: planned.cursor, + changes: entryStream, + }); + + // The receiver's echoed cursor is the authority. assertAppliedPushCursor + // refuses an acknowledgment that does not cover what we sent, so a + // confused peer cannot advance us past unapplied entries. + assertAppliedPushCursor(response.appliedPushCursor, planned.cursor); + + if (compareChangeCursors(planned.cursor, after) > 0) { + writePushCursor(db, planned.cursor, backend); + } + clearBlockMarker(db, backend, "push", operation.generation, now()); + + const complete = compareChangeCursors(planned.cursor, target) >= 0; + if (complete) { + completeOperation(db, backend, "push", operation.generation); + } + + return { + operationId: operation.generation, + generation: operation.generation, + backend, + direction: "push", + mode: operation.mode ?? "entries", + cursor: planned.cursor, + targetCursor: target, + entries: planned.entries.length, + bytes, + skipped: 0, + complete, + }; +} + +async function nextPushBlock( + db: Database, + remote: SyncRPC, + backend: string, + options: PushBlocksOptions, +): Promise { + const key = activeKey(backend, "push"); + const existing = active.get(key); + if (existing !== undefined) { + try { + return await existing.settled; + } catch { + // Fall through and drive our own block from durable state. + } + } + + const settled = runPushBlock(db, remote, backend, options); + const operation = readOperation(db, backend, "push"); + active.set(key, { generation: operation?.generation ?? "", settled }); + try { + return await settled; + } finally { + if (active.get(key)?.settled === settled) active.delete(key); + } +} + +// Restartable push. Same contract as pullBlocks in the other +// direction: one block per next(), durable resume, no alarm ownership. +export function pushBlocks( + db: Database, + remote: SyncRPC, + options: PushBlocksOptions = {}, +): AsyncIterable { + const backend = options.backend ?? DEFAULT_BACKEND; + return { + [Symbol.asyncIterator](): AsyncIterator { + let finished = false; + return { + async next(): Promise> { + if (finished) return { done: true, value: undefined }; + const progress = await nextPushBlock(db, remote, backend, options); + if (progress.complete) finished = true; + return { done: false, value: progress }; + }, + async return(): Promise> { + finished = true; + return { done: true, value: undefined }; + }, + }; + }, + }; +} + +// Open a pull operation and fix its target without transferring +// anything. +// +// Deferred synchronization needs the command's changes pinned at the +// moment the command finished, even though nothing will drain them until +// later. Capturing the target here means a later `pullBlocks` iteration +// joins this pending operation instead of capturing a newer target that +// could have raced ahead. +// +// Returns the fixed target, or undefined when another caller already +// owns an operation for this backend — in which case that operation's +// target is authoritative and this one has nothing to add. +export async function captureSyncTarget( + db: Database, + remote: SyncRPC, + backend: string, + options: { now?: () => number } = {}, +): Promise { + const now = options.now ?? (() => Date.now()); + const opened = openOperation(db, backend, "pull", now()); + if (opened.joined) return opened.operation.target; + + const settled = await remote.watermarks({ settle: true }); + const target = { rev: settled.currentRev, path: null }; + // Decide the mode here rather than deferring it. A package install is + // exactly the workload that defers its pull and exactly the workload + // that needs pack transport, so guessing "entries" would strand the + // largest syncs on the slowest path for the life of the operation. + const mode = await selectPullMode(remote, { + after: readFetchCursor(db, backend), + target, + profile: DEFAULT_BLOCK_PROFILE, + generation: opened.operation.generation, + }); + fixTarget(db, backend, "pull", opened.operation.generation, { target, mode }, now()); + return readOperation(db, backend, "pull")?.target; +} + +// Restartable pull. Each `next()` commits at most one block; the last +// value carries `complete: true` and the following `next()` ends the +// iteration. +// +// Recreating the iterable resumes from durable state, so a caller may +// drive one block per alarm, several per request, or abandon iteration +// entirely and pick it up later. +export function pullBlocks( + db: Database, + remote: SyncRPC, + options: PullBlocksOptions = {}, +): AsyncIterable { + const backend = options.backend ?? DEFAULT_BACKEND; + return { + [Symbol.asyncIterator](): AsyncIterator { + let finished = false; + return { + async next(): Promise> { + if (finished) return { done: true, value: undefined }; + const progress = await nextPullBlock(db, remote, backend, options); + if (progress.complete) finished = true; + return { done: false, value: progress }; + }, + // Breaking out of a for-await stops local driving but leaves + // the operation pending. The last yielded cursor is durable and + // a later iterator resumes from it. + async return(): Promise> { + finished = true; + return { done: true, value: undefined }; + }, + }; + }, + }; +} diff --git a/packages/rpc/vitest.config.bench.ts b/packages/rpc/vitest.config.bench.ts new file mode 100644 index 00000000..b71680f6 --- /dev/null +++ b/packages/rpc/vitest.config.bench.ts @@ -0,0 +1,21 @@ +import { defineConfig } from "vitest/config"; + +// Benchmark runner for the sync engine. +// +// Uses the plain node pool, not vitest-pool-workers: these scenarios +// need two peers live in one process, and workerd hard-isolates I/O +// objects between Durable Objects so a two-peer sync cannot run inside +// a single DO context. The dofs bench covers the real-SqlStorage +// per-statement cost; this one isolates the engine's own overhead — +// block sequencing, pack framing, and restart cost — from the storage +// backend underneath it. +// +// Scoped to src/bench/**.bench.ts so it never runs during `npm test`. +export default defineConfig({ + test: { + globals: true, + include: ["src/bench/**/*.bench.ts"], + testTimeout: 600_000, + hookTimeout: 600_000, + }, +});