Re-work sync engine - #138
Open
aron-cf wants to merge 16 commits into
Open
Conversation
🦋 Changeset detectedLatest commit: 9f35b6b The changes in this PR will be included in the next version bump. This PR includes changesets to release 4 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
commit: |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
A restartable pull or push is driven by a JavaScript async iterator, and an iterator cannot survive Durable Object eviction. Everything the iterator would otherwise hold in memory now lives in SQLite: the fixed target it works toward, the generation that fences superseded executions, the block sizing profile, and a marker for the block currently in flight. The committed progress cursor stays in the existing watermark rows. Filesystem application and cursor advancement have to commit in one storage transaction, so this table records where an operation is going while the watermark records how far it has got. Target capture is two-phase because a pull target comes from a remote settle call that cannot join a local transaction. The row is inserted as capturing, the remote target is read outside the transaction, and a conditional update promotes the row to pending. An eviction between those steps leaves a capturing row with no target, which the next caller takes over under a fresh generation. Rejected entries are recorded durably rather than only reported in a yielded progress value. The cursor advances past a rejection so an unappliable entry cannot stall an operation forever, which would otherwise leave the drop visible only to whoever happened to read that one value. Automatic block shrinking is floored and completion restores the default profile. Halving on every interruption without a matching growth rule would let one eviction storm leave a workspace permanently slow.
A block is an ordered prefix of the coalesced change stream, bounded by the internal sizing profile rather than by anything a caller supplies. Determinism is the property the restart story rests on: the same start cursor, target, and profile select the same entries, so an unacknowledged block that gets re-requested arrives as a replay the receiver can absorb rather than as new work. Object costing is deduplicated within a block, so two paths sharing content pay for their bytes once. Duplicates across separate blocks are left alone; the receiver's content-addressed store absorbs them, which is cheaper than carrying operation-wide sent-object state. An entry selected into an empty block is kept even when it alone exceeds the byte budget. Without that exception a file larger than the budget would be selected, rejected as oversized, and retried at the same cursor forever. Mode selection probes the window once and returns the first block from the same pass, so a large operation does not scan its window twice before transferring anything. Filling the probe is itself the signal that the window is at or past the entry threshold, which answers the question without draining a window that may hold tens of thousands of entries. The mode is then fixed for the operation's life, because a block's encoding must not depend on when it was requested.
The sync cursor only advances after a block applies, so a block interrupted before its acknowledgment is replayed from the durable cursor. Write entries already absorbed that replay through the alreadyApplied check, but a delete entry called rm unconditionally. A tombstone describes a path as of the revision it was stamped with. If the block carrying it died before acknowledging and the path was recreated locally in the meantime, replaying the tombstone destroyed content the tombstone never described. The plan assumed replay was idempotent for every entry kind; for deletes it was not. Upstream deletes now compare the live revision against the tombstone's and drop the delete when the local path is newer. The recreation is itself a change the next push ships upstream, so nothing is lost. Locally authored deletes are exempt because they are original writes rather than replays, and have no earlier revision to compare against.
One next() commits at most one block: plan it, transfer it, apply it, persist the cursor, yield. Nothing live crosses a yield boundary. Every RPC stream is drained and disposed before the value reaches the caller, because the caller may never come back — a Durable Object can be evicted between two next() calls and the iterator object does not survive that. The engine therefore never treats the iterator as state. Each step re-reads the durable cursor and the durable operation row, which is what makes a fresh iterable interchangeable with the one that came before it. Recreating the iterable after every single block converges, and so does abandoning one partway. Ordering carries the safety argument. Objects are staged before metadata applies, metadata applies before the cursor moves, and the cursor never passes an entry that was neither applied nor recorded as skipped. A crash anywhere before the cursor write replays the block, which the receiver absorbs because staged objects and applied entries are both idempotent. The pull target is the source's change head rather than its own inbound fetch cursor, since a pull ships everything the source has produced. A settled read flushes writes still buffered in the shim so they join this target instead of waiting for the next operation. Concurrent iterators in one isolate join a single in-flight block instead of opening duplicate transfers. That is an optimization only: after an eviction the join table is empty and the durable operation row is what keeps the next iterator correct. No timer and no alarm. The application decides when to call next().
Push now shares the operation table, block planner, and restart behavior with pull, so both directions resume from durable state and neither owns an alarm. One asymmetry is deliberate. The local push cursor advances only through the receiver's echoed acknowledgment, and that acknowledgment is checked to cover exactly what was sent. Advancing optimistically would silently drop data whenever an acknowledgment was lost, because the next block would start above entries the receiver never applied. A re-sent block instead lands as a no-op on the receiver, which the existing idempotent apply already absorbs. Push captures its target inside the creation transaction. The target is the local change head, so unlike pull there is no remote settle call to wait on and no capturing phase to recover from. Object bytes ship before the entries that reference them, so the receiver never sees an entry whose content it cannot resolve.
Workspace.pullBlocks() and Workspace.pushBlocks() are the public surface the sync plan asks for: async iterables that take no cursor, no target, and no budget. Recreating either one resumes from the Workspace's durable cursor, so an application can drive one block per alarm, several per request, or abandon iteration and pick it up later. Each next() takes the backend's mutation FIFO rather than the whole iteration. Holding the FIFO across a yield would block every other mutation for as long as the caller took to come back, and the caller may never come back. Serializing per block keeps one mutating sync block per backend and direction without letting an abandoned iterator wedge the workspace. A backend with no sync wire yields one complete, empty block. Returning an iterable that never produces a value would leave a caller's loop spinning on a backend that can never make progress. The existing pull() and push() methods are untouched. They carry the exec bracket and the batch overloads that examples and the command executor still depend on, so replacing them is a later step once callers move across.
Records the durability model behind the block iterables: the iterator is disposable, SQLite is the durability surface, and one next() commits one block. Spells out why the library owns no alarm and shows the one-step and multi-block handlers an application writes instead. Also enumerates replay behavior per entry kind rather than asserting that replay is idempotent. The delete row is the one that needed stating: a tombstone replayed against a newer local recreation would destroy data the tombstone never described.
Answers the plan's four open questions and the two semantics it deferred to "existing behavior", with the reasoning rather than just the choice, so a later change can tell whether it is revisiting a tradeoff or breaking an invariant. Also records three decisions the implementation forced. The pull target is the source's change head and not the fetch cursor the plan's wording pointed at, which would have made every pull a no-op. Object bytes are staged into the blob store rather than carried in memory, or a block's footprint would scale with its payload and defeat the byte bound. The mutation FIFO is held per block rather than per iteration, since holding it across a yield would let an abandoned iterator wedge the workspace. Closes with what did not land: pack mode, removal of the old public API, and peer capability negotiation.
Entry-at-a-time transfer is fine for a handful of changes and hopeless for tens of thousands. A pack carries one ordered run of change entries interleaved with the unique objects they reference, gzip-compressed, closed by a footer naming the cursor window. Objects precede the entries that first reference them so a receiver can stage bytes and apply metadata incrementally rather than buffering the whole pack. That interleaving is what lets Durable Object storage settle between apply groups. Framing is length-prefixed records rather than a self-describing format, because the decoder must never guess where a record ends. A truncated stream fails as a protocol error instead of decoding a prefix, which matters because the cursor would otherwise advance over entries that never arrived. Validation covers the format version, the declared entry and object counts, and a digest over the record stream; gzip's own CRC catches random corruption, and the digest catches the case it cannot, where a well-formed stream disagrees with the footer describing it. The format knows nothing about node_modules or any other path shape. A pack is selected purely on the size of the cursor window.
Adds fetchChangePack and applyChangePack, the bulk transports either direction selects automatically for a large cursor window. One compressed stream carries the entries and every object they reference, so a pack block needs no hasObjects probe and no separate object round trip. Mode is decided once per operation and persisted. Pull probes the source with a threshold-bounded pack request and reads the entry count it would carry; push probes locally, where selectMode already returns the first block from the same pass. Filling the probe is itself the signal that the window is at or past the threshold, so neither path drains a window that may hold tens of thousands of entries to find out how big it is. The mode cannot change mid-operation. A block's encoding must not depend on when it was requested, or a replayed block would not match the original and the receiver could not absorb it as a no-op. A pack is decoded and validated before anything is applied, so a truncated or corrupt pack leaves no partial state and no cursor movement. Advancing over entries that never arrived would lose them silently. A peer without the pack methods keeps working in entry mode, so the two sides can be deployed in either order.
pull() and push() are now the restartable block iterables. The batch overloads, SyncBatchOptions, SyncBatchBudget, and SyncBatchResult are deleted rather than deprecated: nothing has shipped against them, so there is no compatibility window to keep. retryPendingSync and SyncRetryScheduler go with them. They existed because a failed post-command pull had nowhere durable to record its progress, so the host persisted an intent, set an alarm, and called back with a bounded budget and an attempt counter. The operation row and the watermark now hold exactly that state and any later pull() resumes from it, so a parallel retry ledger would be a second source of truth for the same question. This also removes the caller-visible retry budget: a caller that wants to stop trying stops iterating. A deferred exec captures its target through captureSyncTarget, pinning the command's changes at the moment it finished so a later pull() joins that pending operation instead of capturing a newer target that raced ahead. onPullPending becomes a no-op, because a pull that failed in-band already left its operation pending, and dialing a fresh handle purely to record that turned a failed command into a second connection attempt. pullOnce and pushOnce stay as internal drivers. The exec bracket needs one awaitable call that drains the window before a command starts, plus an entry count to report on the execution, which a block-at-a-time iterable does not give it. The sync iterables build a fresh inner iterator per attempt. Caching one pinned the RPC stub it was constructed with, so a reconnect retried against the dead handle instead of the replacement.
Two harnesses. The dofs one runs under vitest-pool-workers so block planning, pack coding, and per-block apply are measured against real Durable Object SqlStorage rather than the node fixture, which caches prepared statements and understates per-statement cost. The rpc one drives the engine end to end with two peers in one process, which isolates engine overhead from capnweb and FUSE framing. Block sizing has a wide margin. The worst single block across the shapes measured leaves a 36x margin against the default 30-second CPU allowance, so the shipped constants are conservative. Recreating the iterable for every block, the eviction worst case, costs about 5% over reusing one iterator. Pack transport turns out to be a bandwidth trade rather than a free win, which is a correction to the plan's framing. Against an in-process peer, pack mode is slower than entry mode on every shape measured, because gzip costs roughly 170 ms per 512-entry block and a local stub charges nothing for the bytes that buys. Counting bytes instead, the same window is 6380 KB in entry mode against 183 KB packed, so the crossover lands near 150 Mbps: packs win below it and lose above it. The thresholds stay put, since any window large enough to trip them is one where bytes dominate, but the assumption is now stated with a measured crossover behind it. Both harnesses print a JSON line per group so before and after runs diff cleanly, and both are scoped to bench globs so they never run during npm test.
Small trees made the planned 4,000-entry block look safe. On a 1350-file tree every block finishes in well under a second and the worst case leaves a 36x margin against the default 30-second CPU allowance. At the scale the plan actually cites the margin nearly vanishes. A 44,100-file tree, 49,001 entries, converges in 13 blocks, but the worst block spends 15.8 s of the 30-second allowance: under 2x headroom. A block that cannot finish can never finish, because every retry starts from the same cursor and does the same work, so the shrink-on- interruption policy only helps after repeated hard failures. A sweep at install scale shows why 4,000 was the wrong number. The gain from fewer round trips is exhausted by roughly 1,000 entries while per-block cost keeps climbing, so 2,000 entries holds the same total time as 4,000 with more than double the headroom. Blocks are a checkpointing mechanism rather than a throughput knob; making them larger buys almost nothing and costs the property they exist for. The install-scale run also strengthens the case for packs, which the earlier small-tree benchmark had understated. Entry metadata dominates a 49,001-entry window, so the same sync is 47.3 MB on the wire in entry mode against 3.5 MB packed, a 13x reduction, for under 1% extra CPU. The scale benchmark is gated behind SYNC_SCALE=1 because seeding and syncing a tree this size takes about four minutes, which does not belong in a default bench run.
The block benchmarks can only answer what sync costs in a dev container against an in-process peer. Three questions they cannot answer decide whether the shipped defaults are right: whether a real block approaches the Durable Object CPU limit, what throughput the real hop achieves, and whether operations converge. The existing observer hook emits spans, which nest correctly for tracing but are not what the Workers Observability query API aggregates today. That API queries Workers Logs, where a single JSON argument to console.log becomes filterable fields. So this adds a second, narrower emitter alongside the span hook rather than replacing it. Field design follows two constraints of that API. Every value is a scalar at the top level, because a filter addresses one key and nesting turns each query into a fragile indexed-path lookup. Derived values are precomputed, because a calculation aggregates a single numeric field, so a ratio spanning two fields cannot be expressed in one query. That is why headroom, msPerEntry, and bytesPerSecond are emitted rather than left to the caller to divide. Telemetry is off by default. Workers Logs is billed per event and a 44,000-entry sync emits one record per block, so writing to a consumer's log stream uninvited would be both surprising and metered. A throwing sink is swallowed: an observability failure must not be able to fail the sync it is watching. The runbook documents the five queries that turn a session into answers, each validated against the live telemetry API, plus the SQL that cross-checks them against durable state.
Add the changeset for the restartable sync engine and pack transport, and update the runtime interface and package README where they still described retryPendingSync and SyncRetryScheduler as the way to recover an incomplete post-command pull.
The entry-mode push handler calls afterApply so computerd's userspace shim flushes the just-pushed files from the VFS to disk before the handler returns, which is what lets a following shell.exec read them. applyChangePack staged objects, applied entries and wrote the cursor but never called the hook, so a pre-command push large enough to select pack transport left a shim backend on stale disk state and the spawned command read a partial workspace. Pack mode is chosen for exactly the windows a pre-command push carries, so this affected the npm-install shape specifically. beforeFetch was already mirrored in fetchChangePack; this restores the same symmetry on the apply side. Real FUSE serves reads from the VFS and does not wire the hook, so only shim deployments were exposed.
aron-cf
force-pushed
the
sync-engine
branch
from
September 11, 2026 12:43
02a0888 to
9f35b6b
Compare
There was a problem hiding this comment.
Devin Review found 1 new potential issue.
2 flags not posted on this PR by your GitHub settings — view them in Devin Review. (Configure)
Comment on lines
+286
to
+289
| } catch (err) { | ||
| // Settle hook failures must not surface as push failures — | ||
| // the entries are already committed. | ||
| console.warn("[SyncRPCServer] afterApply hook failed:", err); |
There was a problem hiding this comment.
🔴 Failed pack flush permits stale exec
When afterApply rejects, applyChangePack still acknowledges the block and permits the command to start. The sender advances its cursor, preventing a retry while the shim retains stale disk contents.
Learn more
A pack push completes only when the receiving shim has copied the committed VFS changes onto disk. The pre-command path waits for pushBlocks, then starts the command because a successful acknowledgment means the receiver is ready. If shim.flush() rejects, this catch returns that acknowledgment anyway. The sender then advances its durable push cursor, so the failed settle cannot be retried as part of the same block. The periodic shim loop can eventually repair disk, but the command starts before that repair and can observe missing or old files.
Example: A pack installs 20,000 dependency files into the receiver VFS. Disk runs out of space while afterApply invokes shim.flush. The RPC still reports success, the sender advances, and npm test starts against a partial node_modules tree.
Recommended fix: Propagate afterApply failures from applyChangePack so runPushBlock leaves the sender cursor unchanged and retries the block. If committed entries must never surface as a push failure, introduce a separate durable settle acknowledgment and block command dispatch until it succeeds; logging alone cannot preserve the pre-command visibility guarantee.
Was this helpful? React with 👍 or 👎 to provide feedback.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Running
npm installinside a container writes about forty-four thousand files, and those files have to reach the durable object that owns the workspace before the next command can see them. That copy can take a long time, during which the durable object can restart.The current implementation already copes with being interrupted, syncing in batches of 256 entries and tracking the resume point after each one, so a crash lost at most a batch and the next attempt carried on from there. Applying an entry twice is recognized and skipped, which made the redundant work harmless.
Previously the whole window was one call:
pull()returned a promise that resolved when the transfer finished or threw when it could not. So anexeccall has to wait for the sync to complete with no way to take part of the work and come back later.pull()andpush()are now async iterables. Each step commits one block and yields what it did, so stopping is a normal thing to do rather than a failure:Breaking out of that loop is now safe. The block that just landed is recorded, and a later
pull()rejoins the same operation rather than starting a new one, so the durable object decides how much work fits in a frame without persisting any bookkeeping of its own.sequenceDiagram participant A as Alarm frame participant W as Workspace participant C as Container A->>W: pull(), iterate loop until the frame runs out W->>C: next block after the recorded point C-->>W: entries or one compressed parcel W-->>A: progress, cursor advanced end A->>A: out of time, stop iterating Note over A,W: next frame resumes the same operationThe other change is what "finished" means. Previously each call asked the container for its current head and treated that as the end of the window, so an interrupted transfer that resumed later measured itself against a newer head. A container still writing would move the head. Instead the target is now fixed when the operation opens and stored, and a resuming caller continues to the original target.
That matters most to
exec. Every command is bracketed by a copy in each direction — files pushed to the backend first so the command sees current state, pulled back afterwards so the next reader sees what it produced. A command may now passsync: "defer"to skip the wait, which returns immediately with its sync reported as pending:The command's own behavior here is unchanged;
execis not materially improved by this work. What changed is that the target is captured, and durably stored, before the caller is told the command finished. So if the durable object dies at that instant, before a single file moves, the restarted object still finds the operation and finishes the install's own snapshot rather than one taken later. The container is a separate process and keeps the files regardless.Because deferral now leaves a durable operation behind, the scheduler that existed to compensate is gone. The following are no longer exported:
SyncBatchOptions,SyncBatchBudget,SyncBatchResult,SyncRetryScheduler,SyncRetryIntent,SyncRetryOptions, andWorkspaceRetryPendingSyncResult. TheretryScheduleroption is gone fromWorkspace,retryPendingSync()is deleted, andassertDeferredReady()is off theSyncinterface. Attempt counts, backoff and give-up thresholds are not part of the surface: a caller that wants to stop stops iterating. Anyone who previously scheduled retries from their own alarm replaces that with the loop above.Two smaller changes based on usage. Blocks now carry up to 2,000 entries rather than 256. Large windows also travel as one compressed parcel instead of a list followed by object fetches, cutting an install from roughly forty-seven megabytes on the wire to three and a half.
The wire stays compatible: the two parcel methods are optional and a peer without them falls back to the older path, so the two sides may be updated in either order and no container image needs replacing.
To verify, start a package install with
sync: "defer"and drain it from an alarm with a short deadline, so the loop yields several times. The install completes across frames, and the entry counts in eachSyncProgresssum to the whole tree. These reproduce the block-size and compression measurements against real durable object storage:Tests cover rejoining a pending operation instead of opening a new one, repeating a capture interrupted before it was recorded, resuming mid-transfer, the parcel format in both directions, and recognizing entries that arrive twice. The runtime interface and package README no longer point at the deleted retry machinery, and a decision document records the measurements behind the block size and the compression threshold.