CL-6365: the terminal signal was never delivered — hold boot-restore pushes until the link is routable - #108
Merged
Merged
Conversation
Upstream's 45 commits since 59f5e7b9 retire the on-disk workflow.json: a deployed workflow's definition is evaluated from its own source closure and re-verified in-child against the approved wire hash, source-ref becomes the only deploy lineage, childWorkflow becomes an owned inline import resolved in memory, and run grants derive from a persisted grant-walk snapshot. Re-applies every workbench-local delta on the new trees, none of which upstream subsumed: the inference.usage forward, the terminal-anchor ownsWorkflowRunRepo gate, the hasConversationText mail drop, and the hub-api needs-you approval route carve-out. Their tests stay green. apps/sidecar keeps its old row: the execution host has not been converted off the retired lineage yet.
Upstream deleted the live-config wrap (wrapHarnessAsSingleStepWorkflow) along with the rest of the live-authored deploy chain. The wrap was a thin adapter over buildSingleStepAgentDefinition, which survives, so the launch now passes the resolved fields directly: the folded run's id, its system prompt, and its catalog-resolved inference preferences. Tools stay empty here — a folded launch pins its tools as packages, not factories.
Records the CL-6324 pin delta, the deltas that survived it, and why the app-side conversion is one migration rather than a per-tree bump: workbench has no code-sourced deploy front, and the retired live-authored chain is what every folded run launches through.
Red/green coverage for the onBodyFailure policy on the re-pinned runtime (CL-6326, CL-6324): default policy unchanged, "continue" re-arms past a failed occurrence while a cancelled one stays terminal-is-final, and crash-recovery honors the same policy. Adds projector coverage asserting a projected onTrigger section carries the authored policy through the live->inert projection, and omits the field when no policy was authored. Fails against the unmodified vendored runtime and projector.
…ection Re-applies the CL-6326 vendored delta on top of the re-vendored runtime: onBodyFailure?: "end" | "continue" on OnTriggerPrimitive/OnTriggerOpts (default "end", byte-compatible with prior behavior), read live by the steady-state drive loop and planOnTriggerResume so a "continue" section re-arms past a failed occurrence instead of ending the run. Cancellation is unaffected and always ends the section. Adds what the delta previously lacked: the live->inert projector's InertOnTrigger and projectOnTrigger now carry the field, so a section's policy survives the child->hub projection instead of being silently dropped before deploy. BodyFailurePolicy is exported from the definition barrel for the projector's type reference.
Red/green coverage for a shared-capacity code-sourced deploy that stamps a pre-existing anchor workflow_run instead of inserting one (CL-6324): the adoption succeeds and issues no INSERT, a definition carrying credential bindings fails closed when no cipher is threaded, and an anchor the tenant does not own is refused before any frame reaches the sidecar. Fails against the two upstream fronts, neither of which accepts a pre-existing anchor.
…hor run Neither code-sourced front could deploy onto a run whose anchor row already exists. deployWorkflowFromSource INSERTs its anchor (a primary-key collision against a folded run's row) and threads no credentialCipher; deployPreparedCodeSourcedWorkflow updates a pre-existing row and threads the cipher, but only under the allocation-ownership lock, so it cannot run on shared capacity. Adds a third front composed from the existing halves -- emitSourceRefDeployFrame and buildInertProjectionStepSources -- following the prepared front's semantics minus the allocation lock: ownership is the anchor row's own tenant plus self-anchoring, checked before the frame so a refused adoption leaves no deployed-but-unanchored agent, and re-asserted on the guarded UPDATE that stamps definitionId and publicKey. No deployer read grant is seeded: the anchor predates the call, so its grants belong to whoever created it.
…loy front Records both vendored deltas in VENDORED.md and each package's VENDORED-FROM, and re-records the workflow and hub-sessions tree hashes so check:killdates matches the edited trees.
Covers the deploy-time config contract (arktype-parsed, env-delivered) and the two definition shapes its mode selects: the folded unbounded step and the per-turn onTrigger section.
…deploy The workflow.json retirement makes source-ref the only deploy lineage: a deployment's definition is evaluated from its own pinned code closure and re-verified against the hub-approved wire hash. Workbench had no code-sourced package to deploy, so this adds the one every agent run will share. The bytes are static and versioned; everything per-run — mailbox, system prompt, inference chain, tool package pins, credential bindings — arrives as deploy-time config in the child's environment and is parsed at the entry module's boundary. The config's mode selects the shape, so the deploy front keeps one parameter set and never branches on step-vs-section.
The first shape here read the config from the child's environment. That cannot work: the approval probe and the run child each evaluate the entry module independently, and the hashed projection covers the trigger address, the system prompt, the (provider, model) pairs, the tool package pins, and the credential bindings — every field of the config. A config read from outside the closure diverges between the two evaluations and fails the re-verify barrier closed. There is also nowhere to read one from: no source variant carries an overlay, the deploy frame carries no config bag, and the probe frame carries no environment at all. So the config becomes the bytes. renderAgentRuntimeSourceTree emits a thin per-run package that pins this versioned one and calls the builder with the run's config as a literal, ready to commit into a workflow-kind asset and deploy as source at a commitSha — the only source variant cheap enough to mint per run.
Red/green coverage for the conversion (CL-6324): deployAtHead renders the run's per-run workflow source package, commits it into the run's own definition asset on a per-run ref, and deploys the resulting commitSha through the adopting code-sourced front against the pre-minted anchor. Covers the whole round trip -- the committed tree's shape, the config rendered into the deployed bytes (address, system prompt, model pairs, tool pins, credential bindings, mode), the adopted deploy's frame, the wake path taking the same route, a caller-supplied section mode riding through untouched, and a run whose definition has no workflow-kind asset failing before any deploy. Section mode is proven to author onBodyFailure "continue" and to keep it through the live->inert projection. Fails against the in-memory synthesize-and-deploy path, which neither renders bytes nor touches an asset.
…nition Cuts deployAtHead over to the code-sourced seam. The in-memory single-step definition it used to build and hand to deploySingleStepAtHead is gone -- that front was retired with the on-disk workflow.json, and a deployment's definition is now whatever its own pinned source closure evaluates to. The run's deploy-time config (trigger address, system prompt, resolved inference chain, tool package pins, credential bindings, shape) is rendered into a per-run @corbits/agent-runtime package, committed into the run's OWN definition asset on refs/heads/runs/<runId>, and deployed by pinning that commitSha. The config has to be inside the bytes: the approval probe and the run child evaluate the entry independently and the child refuses a definition whose recomputed wire hash differs, and every one of those fields is in the hashed preimage. The deploy goes through deployAdoptedWorkflowFromSource, the only front a folded run can use -- its anchor workflow_run row is minted before any deployment attaches to it, so the inserting front collides on the primary key and the prepared front needs an exclusive allocation it never has. The credential cipher is threaded instead of a pre-built delivery, since the front resolves the material itself from the deployed definition's own bindings; buildCredentialDelivery stays only for the credential: use grants the run's principal needs in its own grants.json. The step's input selector becomes the config's mode: `step` (with an optional literalInput, the workbench host's CL-6164 pin) or `section` with a per-turn timeout, so the Phase 1.3 swap changes a caller's argument rather than a branch here. Section mode authors onBodyFailure "continue" so one failed turn re-arms the section instead of retiring the run. hubPublicKey leaves FoldedRunsDeps: the adopting front does not take it, and nothing else in the folded-run path read it.
Records the conversion in the CL-6324 inventory: what deployAtHead now
does (render, commit into the run's own definition asset on a per-run
ref, deploy the pinned commit through the adopting front), why the asset
is reused rather than minted per deploy, and how the step/section shape
became config data.
Also records what still blocks EXECUTION -- CLOSURE_PACKAGE_DIR and the
sidecar's WorkflowProbeExecutor, the remaining typecheck failures -- and
a defect the conversion surfaced: apps/hub mints MCP credential handles
("mcp:<slug>") that the platform's ToolCredentialHandle grammar rejects,
which now fails closed at render time because the config is finally
parsed.
The deploy frame no longer carries a definition, so every sidecar test that built one now stages a source-ref pin and registers what the pinned closure evaluates to through an injected materializer. The lifecycle suite asserts the new contract directly: an onTrigger body's sources.json is staged and its definition is not, the child's env carries the materialized closure dir and the hub-approved wire hash, and the durable record carries the pin a restore re-materializes from. The step-coverage gate moves with it -- the frame's arktype can no longer narrow a table it cannot see, so coverage is checked against the closure-derived definition.
The sidecar no longer writes a definition into its deploy tree and reads it back. A deploy materializes the frame's frozen closure, evaluates the pinned code, and runs THAT; the boot-time restore re-materializes the same pin and re-derives the same definition, so both paths reach the runnable definition by one computation. The child gets the closure dir plus the hub-approved wire hash it re-verifies its own projection against, and the durable record carries the pin a restore replays. The pieces that make it executable: - Closure staging (`workflow-host-wiring/closure-staging.ts`) owns the durable per-deployment source stores, the mount resolution both paths derive from the pin alone, and the apply. Injectable so a test can stand in for fetch + SRI-verify + layout + evaluate. - A `WorkflowProbeExecutor` is wired at the boot edge, so a probe answers with a real inert projection and its wire hash instead of the hub-link's rejecting placeholder. Its airlocked child, the closure materializer, the closure apply, and the inline source-asset delivery come from upstream's own sidecar at `4ed8baf4` (see VENDORED.md). - Child spawns are in-memory: a rung lifts its inline children to refs and serves grandchildren from that map, so no rung reads a definition off disk at any depth. - `WORKFLOW_DEFINITION_REPO_ID`/`_REF` are gone. What survives is `WORKFLOW_DEFINITION_ID` — identity for the run-authenticated capabilities route a step tool calls, never a repo to read from.
Closes the three sidecar rows on the conversion table and records what the conversion did not prove: no run has executed end to end, the MCP credential-handle defect still gates a real launch, every test injects the closure materializer, and the pinned tool-package arm was left in place deliberately where upstream went all-source-tools.
…envelope A workflow-kind asset now accepts only a source codebase declaring an interchange.workflow entry; the envelope form is rejected at push time with a path-violation, which failed every seed run outright. The pusher renders the serialized definition into the two-file tree that form takes -- a package.json naming the entry, and the entry module default-exporting the definition -- so a code-sourced deploy evaluates the same definition it used to re-read off disk.
The deployments route now takes the code-sourced pair -- a `source`
naming the asset plus `package: { format: "source", commitSha }`, and
the `entry` the package.json declares -- so the bare `assetId` body the
seed sent is rejected outright. The pusher is the only place that knows
which commit the asset's main now sits at, so it reports the sha and the
deploy pins it.
Covers the inert-projection launch-body readers, the DB-side newest-projected-definition walk, the named pre-cutover error, and the per-run source tree the sidecar can actually resolve.
Under the workflow.json retirement a deployed definition's body is whatever its source closure evaluates to, and a source-format asset carries no envelope to read it back from, so nothing hub-side could answer 'what does this agent launch as'. The projection the approval freeze already hashes is that answer. `workflow_definition_version` gains a `wire_projection` column, stamped in the same transaction as `approved_wire_hash` and `grant_snapshot`, and read back through `loadFrozenWireProjection` validated as a WorkflowProjectionDefinition. Stored beside the hash that addresses it, the two can never disagree. Every folded launch now reads the launch body from there: the chat invite and refresh paths, tasks, webhook triggers, routines, and the one-shot planner run. Grant requirements come from the definition row, because the projector drops them; the projector also flattens the agent's inference chain to `modelSources`, which the reader follows. A row with no stored projection fails as the named DefinitionProjectionMissingError with re-deploy guidance, mapped to a 4xx at every route boundary. The workbench host keeps a live-shape reader: it builds its definition in process and never round-trips through a freeze.
Three things stood between a folded run and a real deploy. The rendered per-run tree pinned `@corbits/agent-runtime` at `workspace:*`. An asset tree is a standalone codebase with no workspace root, so the closure resolver refused it outright. Render the tree the way the seed's default workflows already render theirs: evaluate the builder at render time and write the definition out as a JSON literal, so the whole closure is two files and no dependency. The config is still the bytes; nothing rides beside them. The tree lives on a per-run ref inside the shared definition asset, but the deploy front packed the asset's default ref, shipping a history the pinned commit was not reachable from. `DeployWorkflowFromSourceParams` takes an optional `sourceRef`; omitted, the default ref is packed exactly as upstream. Nothing staged the step's tool-package manifest. Upstream's source-ref front runs no launch phases, but the sidecar's tool loader still reads a step's pins off `deploy/tool-packages-manifest.json`, so a run deployed with its pins in the hash and no tools in the child. `deployAtHead` stages that tree through `stageWorkflowStep` before the deploy frame.
The proofs run on one real stack — scratch database, real signup, real Ollama, nothing mocked — and every step asserts. Ledger the two vendored deltas (the persisted projection, the per-run source ref) and record what the second real boot found, including the two things still open.
Every agent-definition write is asserted as the two-file source tree a workflow-kind asset now accepts — a package.json declaring the interchange.workflow entry plus that entry module — with the definition recovered back out of the entry through the same reader the routes use. Adds red/green coverage for the named retirement error: an asset whose tree still holds a bare workflow.json answers 409 at every route boundary that can reach it, and writes nothing.
A workflow-kind asset accepts only a source codebase — a package.json declaring an interchange.workflow entry plus that entry module — so the agent-directory authoring lineage no longer writes the retired workflow.json envelope. The renderer moves out of @workbench/hub-client (and out of its second copy in @corbits/agent-runtime) into a new dependency-free package, @corbits/workflow-source, which both consume alongside a reader that recovers the definition from the entry module's exact bytes. A missing or non-source entry throws RetiredWorkflowEnvelopeError, which each agent-directory route module answers as a 409 with re-author guidance rather than letting it read as a server fault. @corbits/agent-directory's definition-asset.ts is the lineage's single seam onto that pair; the create core, the read/modify/write routes, apps/hub's planner deploy, and the evals world snapshot all route through it instead of naming an asset path apiece. Both new library packages carry the canonical LGPL-2.1 text that check:licenses requires.
Item 5 of the CL-6324 second-boot list is closed, with a section on where the shared renderer and reader now live and where the named retirement error is mapped.
A folded step-mode run is one unbounded step servicing every inbound mail, so it never publishes a workflow-host RunStarted per message. The harness stopped asserting one and now asserts what that shape actually produces: no per-occurrence child run at all, plus the durable message.run.started/ended bracket, read back as an insights turn-latency sample through the tenant's own latency route. Adds the section shape alongside it. The proof renders a mode: "section" agent-runtime config into its own source package, pushes it as a workflow-kind asset, deploys it by source-ref, and drives it with real mail — so every message becomes an onTrigger occurrence with its own child run id and its own event log, which is where RunStarted really lives. Proof 4 kills the sidecar mid-occurrence in both shapes and asserts the section runs another occurrence after boot restore, which is onBodyFailure: "continue" doing its job. Two environment facts the harness now handles explicitly rather than leaving to chance: the model is named by E2E_OLLAMA_MODEL instead of taken from the curated catalog seed, and the bench catalog is narrowed to that one model so a turn's model is not decided alphabetically among the embedding models a live Ollama connect also seeds.
…nt drop The kill window was three seconds after a prompt a small model answers in one. On a fast instance the turn had already completed and parked by the time the sidecar came down, so proof 4 was proving a clean restart, not crash recovery -- and passed. Proof 4 now sends a workload no model finishes inside the window, asserts nothing has answered at the moment of the kill, and asserts the interrupted turn reaches the reader as a partial answer or the product's own undelivered notice rather than vanishing.
Covers both directions of the mapping a relaunch needs: the room's own address resolving to whichever run is live now, and a live deployment address resolving back to the participant the room has been addressing all along. Also separates a run that is beyond waking (terminal, its durable log already sealed) from a folded run merely parked between messages, which still wakes. The two existing chat fakes gain the launch row every chat run has in production, since that row is now the mapping every lookup goes through.
A run that dies mid-turn commits its terminal event to the durable log before exiting, so waking its address again comes straight back as workflow_run_terminal and the next message is dropped in silence. The fix is a fresh run — and never reclaiming the dead one's log, which is the audit trail this shape exists to keep. The platform fuses a run's id to its address in three independent places, so a fresh run necessarily carries a fresh address. That is only survivable because the room stops being the run: chat.workbench_launch now maps a stable participant id (the address the room uses forever) to the current run id behind it, re-pointed on every relaunch. agent-binding.ts owns both directions — outbound sends resolve the live address, inbound events resolve back to the room address that participant records, mention handles, and posted messages all carry. Detection is workflow_run.status plus folded-runs' isFoldedRunSettled, which had no caller until now: a terminal status that is not a parked folded run means relaunch rather than wake. The relaunch is send-triggered; a boot-time sweep and an in-room notice are still open. See docs/revendor-inventory.md.
Covers the three ways a relaunch is visible from outside: a routable-but-dead participant is swept up and replaced (and a folded run merely parked between messages is not), the room is told in the agent's own voice with wording that names the cause rather than the machinery, and an attachment sent before the crash still opens afterwards even though the fresh run's mail session is a different session. The existing chat fakes gain the prior-run history column every launch row now carries.
The relaunch was send-triggered, so a room whose agent died in a crash stayed silently dead until somebody wrote into it — and the turn the crash interrupted never surfaced at all, because the run that died never sends the message.run.ended the turn-drop notice hangs off. Three closing pieces: - sweepTerminalRuns replaces every participant whose run is beyond waking, bounded and logged per relaunch. The hub runs it at boot and re-arms it on a sidecar disconnect, across a short bounded series of passes: a run that died with its sidecar only reads as terminal once the restarted sidecar has packed its log back to the hub. - Every relaunch, swept or send-triggered, posts a cause-aware notice into each room the replaced participant belongs to, under the stable address the room has always known it by. - workbench_launch keeps the runs it used to be, and fetchBlob walks them: a folded run's mail session hangs off its principal, and a fresh run has a fresh principal, so an attachment sent before the crash is otherwise unreachable forever. Proof 4 gains the hop the fresh-run ruling exists for: the replaced run's durable log is still readable through the ordinary run routes after its replacement is already answering.
Records the three closing pieces (boot sweep, relaunch notice, pre-relaunch attachments) and the audit hop, and corrects an earlier claim with what a real re-run actually showed: only the section deployment's workflow_run row goes 'failed' after a mid-turn kill. The folded chat run's row stays 'running' while its durable log carries the terminal event, so the status-based detection signal never fires for the shape proof 4's third hop measures. The relaunch machinery is built; the signal that should trigger it is the remaining gap.
A restored supervisor that finds a step which died mid-invocation commits that run's terminal events before hubLink.connect() has run, so the pack push fails 'Connection lost' and survives only on the post-challenge re-drive -- which misses a rejection that latches after the challenge fired. The terminal event then stays on sidecar disk for good and the hub's workflow_run row reads 'running' for a run whose supervisor rejects every further message. Hold every address the boot restore registers, the same block the link already applies across a mid-life disconnect, and let the challenge lift it. E2E_LOG_DIR tees spawned-app output so a crash-recovery run stays diagnosable after the fact.
TheGreatAxios
marked this pull request as ready for review
August 20, 2026 11:32
TheGreatAxios
force-pushed
the
cl-6365-signal
branch
from
August 20, 2026 11:32
bec07f1 to
2f0204e
Compare
This was referenced Aug 20, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this fixes
PR #107 recorded that after a mid-turn sidecar kill the folded chat run's
workflow_runrow staysrunningwhile "its terminal event lives only in the durable log", and proposed reconciling the row against the received log.Reading both sides of a real crashed stack shows that diagnosis was wrong, in a way that matters: the hub's copy of the durable log stopped at the same pre-crash event the row did. The sidecar's own workflow-run repo was three commits ahead of the hub's:
So neither
readWorkflowRunLifecyclenor a row-reconciling vendored delta would have helped — both read "still live". The signal was not mis-read; it was never delivered.Why
restoreWorkflowDeployments()runs beforehubLink.connect()— a restored deployment's mailbox has to be live before the hub can route to it. A restored supervisor that finds a step which died mid-invocation commits that run'sStepFailed/RunFailedright then, and the pack-pushing store schedules the push into a link that does not exist yet:Those failures latch on their slot; the only thing that re-ships them is
notifyAddressRoutable, fired when the reconnect challenge passes. That recovers the push when the rejection latches before the challenge, and loses it forever when the rejection lands after — nothing re-arms the slot, and the row readsrunningfor a run whose supervisor rejects every further message.The fix
createBootRestorePushHoldmarks every address the boot restore registers unroutable — the same block the link already applies across a mid-life disconnect — and the challenge lifts it. No workflow-run push is attempted against a link that has not been established, so the race is removed rather than recovered from. No vendored file is touched.E2E_LOG_DIRtees spawned-app output in the e2e harness, which is what made a crash-recovery run diagnosable after the fact.Proof
Proof 4 re-run on a real stack (scratch database, real signup, real Ollama, both shapes):
Connection lostpack push failures (was 9).workflow_run: folded chat run and section deployment bothfailed.workbench_launchshowscurrent_run_id = run_b8930df3…,prior_run_ids = ["run_53c1b05c…"], and the room carries the relaunch notice: "I got cut off partway through that last one and never finished it. I'm back now — send it again and I'll pick it up." That is exactly what proof 4's hop 3 asserts.Honest caveat: the harness itself did not reach hop 3. With the terminal signal now arriving reliably, proof 4's earlier section hop fails deterministically with
409 workflow_run_terminalinstead of intermittently — the section shape's own gap (a plain workflow deployment has no room, so nothing maps a stable id onto a fresh run for it) surfacing on schedule rather than racing the assertion. That stays out of CL-6365's scope and needs its own change before proof 4 can go green end to end.Gates
apps/sidecar148 pass,packages/chat576 pass,apps/hub130 pass,packages/folded-runs52 pass — 0 failcheck:killdatesandlintfail on the base branch for reasons unrelated to this change (two vendored tree hashes, and Prettier inpackages/cli/packages/onboarding); this branch touches no vendored file and its own files are Prettier-clean.