CL-6324: persist the definition's projection, and make a folded run really deploy - #103
Closed
TheGreatAxios wants to merge 27 commits into
Closed
CL-6324: persist the definition's projection, and make a folded run really deploy#103TheGreatAxios wants to merge 27 commits into
TheGreatAxios wants to merge 27 commits into
Conversation
Upstream's 45 commits since 59f5e7b9 retire the on-disk workflow.json: a deployed workflow's definition is evaluated from its own source closure and re-verified in-child against the approved wire hash, source-ref becomes the only deploy lineage, childWorkflow becomes an owned inline import resolved in memory, and run grants derive from a persisted grant-walk snapshot. Re-applies every workbench-local delta on the new trees, none of which upstream subsumed: the inference.usage forward, the terminal-anchor ownsWorkflowRunRepo gate, the hasConversationText mail drop, and the hub-api needs-you approval route carve-out. Their tests stay green. apps/sidecar keeps its old row: the execution host has not been converted off the retired lineage yet.
Upstream deleted the live-config wrap (wrapHarnessAsSingleStepWorkflow) along with the rest of the live-authored deploy chain. The wrap was a thin adapter over buildSingleStepAgentDefinition, which survives, so the launch now passes the resolved fields directly: the folded run's id, its system prompt, and its catalog-resolved inference preferences. Tools stay empty here — a folded launch pins its tools as packages, not factories.
Records the CL-6324 pin delta, the deltas that survived it, and why the app-side conversion is one migration rather than a per-tree bump: workbench has no code-sourced deploy front, and the retired live-authored chain is what every folded run launches through.
Red/green coverage for the onBodyFailure policy on the re-pinned runtime (CL-6326, CL-6324): default policy unchanged, "continue" re-arms past a failed occurrence while a cancelled one stays terminal-is-final, and crash-recovery honors the same policy. Adds projector coverage asserting a projected onTrigger section carries the authored policy through the live->inert projection, and omits the field when no policy was authored. Fails against the unmodified vendored runtime and projector.
…ection Re-applies the CL-6326 vendored delta on top of the re-vendored runtime: onBodyFailure?: "end" | "continue" on OnTriggerPrimitive/OnTriggerOpts (default "end", byte-compatible with prior behavior), read live by the steady-state drive loop and planOnTriggerResume so a "continue" section re-arms past a failed occurrence instead of ending the run. Cancellation is unaffected and always ends the section. Adds what the delta previously lacked: the live->inert projector's InertOnTrigger and projectOnTrigger now carry the field, so a section's policy survives the child->hub projection instead of being silently dropped before deploy. BodyFailurePolicy is exported from the definition barrel for the projector's type reference.
Red/green coverage for a shared-capacity code-sourced deploy that stamps a pre-existing anchor workflow_run instead of inserting one (CL-6324): the adoption succeeds and issues no INSERT, a definition carrying credential bindings fails closed when no cipher is threaded, and an anchor the tenant does not own is refused before any frame reaches the sidecar. Fails against the two upstream fronts, neither of which accepts a pre-existing anchor.
…hor run Neither code-sourced front could deploy onto a run whose anchor row already exists. deployWorkflowFromSource INSERTs its anchor (a primary-key collision against a folded run's row) and threads no credentialCipher; deployPreparedCodeSourcedWorkflow updates a pre-existing row and threads the cipher, but only under the allocation-ownership lock, so it cannot run on shared capacity. Adds a third front composed from the existing halves -- emitSourceRefDeployFrame and buildInertProjectionStepSources -- following the prepared front's semantics minus the allocation lock: ownership is the anchor row's own tenant plus self-anchoring, checked before the frame so a refused adoption leaves no deployed-but-unanchored agent, and re-asserted on the guarded UPDATE that stamps definitionId and publicKey. No deployer read grant is seeded: the anchor predates the call, so its grants belong to whoever created it.
…loy front Records both vendored deltas in VENDORED.md and each package's VENDORED-FROM, and re-records the workflow and hub-sessions tree hashes so check:killdates matches the edited trees.
Covers the deploy-time config contract (arktype-parsed, env-delivered) and the two definition shapes its mode selects: the folded unbounded step and the per-turn onTrigger section.
…deploy The workflow.json retirement makes source-ref the only deploy lineage: a deployment's definition is evaluated from its own pinned code closure and re-verified against the hub-approved wire hash. Workbench had no code-sourced package to deploy, so this adds the one every agent run will share. The bytes are static and versioned; everything per-run — mailbox, system prompt, inference chain, tool package pins, credential bindings — arrives as deploy-time config in the child's environment and is parsed at the entry module's boundary. The config's mode selects the shape, so the deploy front keeps one parameter set and never branches on step-vs-section.
The first shape here read the config from the child's environment. That cannot work: the approval probe and the run child each evaluate the entry module independently, and the hashed projection covers the trigger address, the system prompt, the (provider, model) pairs, the tool package pins, and the credential bindings — every field of the config. A config read from outside the closure diverges between the two evaluations and fails the re-verify barrier closed. There is also nowhere to read one from: no source variant carries an overlay, the deploy frame carries no config bag, and the probe frame carries no environment at all. So the config becomes the bytes. renderAgentRuntimeSourceTree emits a thin per-run package that pins this versioned one and calls the builder with the run's config as a literal, ready to commit into a workflow-kind asset and deploy as source at a commitSha — the only source variant cheap enough to mint per run.
Red/green coverage for the conversion (CL-6324): deployAtHead renders the run's per-run workflow source package, commits it into the run's own definition asset on a per-run ref, and deploys the resulting commitSha through the adopting code-sourced front against the pre-minted anchor. Covers the whole round trip -- the committed tree's shape, the config rendered into the deployed bytes (address, system prompt, model pairs, tool pins, credential bindings, mode), the adopted deploy's frame, the wake path taking the same route, a caller-supplied section mode riding through untouched, and a run whose definition has no workflow-kind asset failing before any deploy. Section mode is proven to author onBodyFailure "continue" and to keep it through the live->inert projection. Fails against the in-memory synthesize-and-deploy path, which neither renders bytes nor touches an asset.
…nition Cuts deployAtHead over to the code-sourced seam. The in-memory single-step definition it used to build and hand to deploySingleStepAtHead is gone -- that front was retired with the on-disk workflow.json, and a deployment's definition is now whatever its own pinned source closure evaluates to. The run's deploy-time config (trigger address, system prompt, resolved inference chain, tool package pins, credential bindings, shape) is rendered into a per-run @corbits/agent-runtime package, committed into the run's OWN definition asset on refs/heads/runs/<runId>, and deployed by pinning that commitSha. The config has to be inside the bytes: the approval probe and the run child evaluate the entry independently and the child refuses a definition whose recomputed wire hash differs, and every one of those fields is in the hashed preimage. The deploy goes through deployAdoptedWorkflowFromSource, the only front a folded run can use -- its anchor workflow_run row is minted before any deployment attaches to it, so the inserting front collides on the primary key and the prepared front needs an exclusive allocation it never has. The credential cipher is threaded instead of a pre-built delivery, since the front resolves the material itself from the deployed definition's own bindings; buildCredentialDelivery stays only for the credential: use grants the run's principal needs in its own grants.json. The step's input selector becomes the config's mode: `step` (with an optional literalInput, the workbench host's CL-6164 pin) or `section` with a per-turn timeout, so the Phase 1.3 swap changes a caller's argument rather than a branch here. Section mode authors onBodyFailure "continue" so one failed turn re-arms the section instead of retiring the run. hubPublicKey leaves FoldedRunsDeps: the adopting front does not take it, and nothing else in the folded-run path read it.
Records the conversion in the CL-6324 inventory: what deployAtHead now
does (render, commit into the run's own definition asset on a per-run
ref, deploy the pinned commit through the adopting front), why the asset
is reused rather than minted per deploy, and how the step/section shape
became config data.
Also records what still blocks EXECUTION -- CLOSURE_PACKAGE_DIR and the
sidecar's WorkflowProbeExecutor, the remaining typecheck failures -- and
a defect the conversion surfaced: apps/hub mints MCP credential handles
("mcp:<slug>") that the platform's ToolCredentialHandle grammar rejects,
which now fails closed at render time because the config is finally
parsed.
The deploy frame no longer carries a definition, so every sidecar test that built one now stages a source-ref pin and registers what the pinned closure evaluates to through an injected materializer. The lifecycle suite asserts the new contract directly: an onTrigger body's sources.json is staged and its definition is not, the child's env carries the materialized closure dir and the hub-approved wire hash, and the durable record carries the pin a restore re-materializes from. The step-coverage gate moves with it -- the frame's arktype can no longer narrow a table it cannot see, so coverage is checked against the closure-derived definition.
The sidecar no longer writes a definition into its deploy tree and reads it back. A deploy materializes the frame's frozen closure, evaluates the pinned code, and runs THAT; the boot-time restore re-materializes the same pin and re-derives the same definition, so both paths reach the runnable definition by one computation. The child gets the closure dir plus the hub-approved wire hash it re-verifies its own projection against, and the durable record carries the pin a restore replays. The pieces that make it executable: - Closure staging (`workflow-host-wiring/closure-staging.ts`) owns the durable per-deployment source stores, the mount resolution both paths derive from the pin alone, and the apply. Injectable so a test can stand in for fetch + SRI-verify + layout + evaluate. - A `WorkflowProbeExecutor` is wired at the boot edge, so a probe answers with a real inert projection and its wire hash instead of the hub-link's rejecting placeholder. Its airlocked child, the closure materializer, the closure apply, and the inline source-asset delivery come from upstream's own sidecar at `4ed8baf4` (see VENDORED.md). - Child spawns are in-memory: a rung lifts its inline children to refs and serves grandchildren from that map, so no rung reads a definition off disk at any depth. - `WORKFLOW_DEFINITION_REPO_ID`/`_REF` are gone. What survives is `WORKFLOW_DEFINITION_ID` — identity for the run-authenticated capabilities route a step tool calls, never a repo to read from.
Closes the three sidecar rows on the conversion table and records what the conversion did not prove: no run has executed end to end, the MCP credential-handle defect still gates a real launch, every test injects the closure materializer, and the pinned tool-package arm was left in place deliberately where upstream went all-source-tools.
…envelope A workflow-kind asset now accepts only a source codebase declaring an interchange.workflow entry; the envelope form is rejected at push time with a path-violation, which failed every seed run outright. The pusher renders the serialized definition into the two-file tree that form takes -- a package.json naming the entry, and the entry module default-exporting the definition -- so a code-sourced deploy evaluates the same definition it used to re-read off disk.
The deployments route now takes the code-sourced pair -- a `source`
naming the asset plus `package: { format: "source", commitSha }`, and
the `entry` the package.json declares -- so the bare `assetId` body the
seed sent is rejected outright. The pusher is the only place that knows
which commit the asset's main now sits at, so it reports the sha and the
deploy pins it.
Covers the inert-projection launch-body readers, the DB-side newest-projected-definition walk, the named pre-cutover error, and the per-run source tree the sidecar can actually resolve.
Under the workflow.json retirement a deployed definition's body is whatever its source closure evaluates to, and a source-format asset carries no envelope to read it back from, so nothing hub-side could answer 'what does this agent launch as'. The projection the approval freeze already hashes is that answer. `workflow_definition_version` gains a `wire_projection` column, stamped in the same transaction as `approved_wire_hash` and `grant_snapshot`, and read back through `loadFrozenWireProjection` validated as a WorkflowProjectionDefinition. Stored beside the hash that addresses it, the two can never disagree. Every folded launch now reads the launch body from there: the chat invite and refresh paths, tasks, webhook triggers, routines, and the one-shot planner run. Grant requirements come from the definition row, because the projector drops them; the projector also flattens the agent's inference chain to `modelSources`, which the reader follows. A row with no stored projection fails as the named DefinitionProjectionMissingError with re-deploy guidance, mapped to a 4xx at every route boundary. The workbench host keeps a live-shape reader: it builds its definition in process and never round-trips through a freeze.
Three things stood between a folded run and a real deploy. The rendered per-run tree pinned `@corbits/agent-runtime` at `workspace:*`. An asset tree is a standalone codebase with no workspace root, so the closure resolver refused it outright. Render the tree the way the seed's default workflows already render theirs: evaluate the builder at render time and write the definition out as a JSON literal, so the whole closure is two files and no dependency. The config is still the bytes; nothing rides beside them. The tree lives on a per-run ref inside the shared definition asset, but the deploy front packed the asset's default ref, shipping a history the pinned commit was not reachable from. `DeployWorkflowFromSourceParams` takes an optional `sourceRef`; omitted, the default ref is packed exactly as upstream. Nothing staged the step's tool-package manifest. Upstream's source-ref front runs no launch phases, but the sidecar's tool loader still reads a step's pins off `deploy/tool-packages-manifest.json`, so a run deployed with its pins in the hash and no tools in the child. `deployAtHead` stages that tree through `stageWorkflowStep` before the deploy frame.
The proofs run on one real stack — scratch database, real signup, real Ollama, nothing mocked — and every step asserts. Ledger the two vendored deltas (the persisted projection, the per-run source ref) and record what the second real boot found, including the two things still open.
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes the milestone's design blocker and takes a folded run all the way to a real, tool-bearing reply on the new rails.
The ruling, and where it landed
The hub persists a definition's evaluated inert projection, stored WITH the definition, keyed to the approved wire hash. It landed at the freeze rather than at the deploy front, because that is where the projection and the hash that addresses it are already written together in one transaction.
workflow_definition_versiongains awire_projectionjsonb column, stamped bycreateDbFrozenApprovalWriteralongsideapproved_wire_hashandgrant_snapshot;loadFrozenWireProjectionreads it back validated as aWorkflowProjectionDefinition. Both code-sourced deploy fronts consume that same value, so the hub's copy and the sidecar's re-verify cannot drift. No new table.Every folded launch now reads its body from there — the chat invite and refresh paths, tasks, webhook triggers, routines, and the one-shot planner run. A pre-cutover row with no stored projection fails as the named
DefinitionProjectionMissingErrorcarrying re-deploy guidance, mapped to a 4xx at every route boundary, never a raw 500.Two shape facts the conversion turned up, both load-bearing: the projector renames and flattens the agent's inference chain (
agent.inference.sources→agent.modelSources), and it dropsgrantRequirementsentirely, so those come from the definition row instead. One live-shape reader survives on purpose, for the workbench host, which builds its definition in process and never round-trips through a freeze.Then three things that stood between a folded run and a real deploy
@corbits/agent-runtimeatworkspace:*. An asset tree is a standalone codebase with no workspace root, so the closure resolver refused every folded deploy. It now renders the way the seed's default workflows already render theirs: evaluate the builder at render time, write the definition out as a JSON literal, whole closure is two files and no dependency. The config is still the bytes.DeployWorkflowFromSourceParamstakes an optionalsourceRef; omitted, upstream behaviour is unchanged.deploy/tool-packages-manifest.json.deployAtHeadnow stages that tree throughstageWorkflowStepbefore the deploy frame — workbench's deliberate divergence.Both vendored deltas are ledgered in VENDORED.md with the existing kill-date hashes.
The proofs
scripts/e2e/cl-6324-launch-proof.tsruns all four on one real stack — scratch database, real signup, real Ollama, nothing mocked.assistantbecomes invitable.materialized workflow-probe closure for folded-run-<runId>), the definition loads from that closure, run grants land in the workflow-run repo, the anchor row flipsdeployed→running. The run's durable event log stays empty, because a foldedstep-mode run is one unbounded step servicing every inbound mail and its per-message bracket is themessage.run.startedAGENT event, not a workflow-hostRunStarted. That assertion is really about CL-6329'ssectionmode. Left as written rather than weakened.Still open
workflow.jsonenvelope into a workflow-kind asset, whichvalidatePushnow refuses. That is authoring, not launching — a projection is read-only and cannot be written back through — so it needs its own cutover. None of the four proofs exercise it.Both are written up in
docs/revendor-inventory.md.Gates
Repo typecheck exit 0, eslint 0 errors. Touched suites green: folded-runs 52, agent-runtime 29, tasks 59 (was 6 red before this branch), webhook-triggers 29, chat 560, hub-client 141, morning-brief 32, hub 130.