CL-6365: close proof 4's remainders — sweep, notice, old attachments, audit - #107
Closed
TheGreatAxios wants to merge 3 commits into
Closed
CL-6365: close proof 4's remainders — sweep, notice, old attachments, audit#107TheGreatAxios wants to merge 3 commits into
TheGreatAxios wants to merge 3 commits into
Conversation
Covers the three ways a relaunch is visible from outside: a routable-but-dead participant is swept up and replaced (and a folded run merely parked between messages is not), the room is told in the agent's own voice with wording that names the cause rather than the machinery, and an attachment sent before the crash still opens afterwards even though the fresh run's mail session is a different session. The existing chat fakes gain the prior-run history column every launch row now carries.
The relaunch was send-triggered, so a room whose agent died in a crash stayed silently dead until somebody wrote into it — and the turn the crash interrupted never surfaced at all, because the run that died never sends the message.run.ended the turn-drop notice hangs off. Three closing pieces: - sweepTerminalRuns replaces every participant whose run is beyond waking, bounded and logged per relaunch. The hub runs it at boot and re-arms it on a sidecar disconnect, across a short bounded series of passes: a run that died with its sidecar only reads as terminal once the restarted sidecar has packed its log back to the hub. - Every relaunch, swept or send-triggered, posts a cause-aware notice into each room the replaced participant belongs to, under the stable address the room has always known it by. - workbench_launch keeps the runs it used to be, and fetchBlob walks them: a folded run's mail session hangs off its principal, and a fresh run has a fresh principal, so an attachment sent before the crash is otherwise unreachable forever. Proof 4 gains the hop the fresh-run ruling exists for: the replaced run's durable log is still readable through the ordinary run routes after its replacement is already answering.
Records the three closing pieces (boot sweep, relaunch notice, pre-relaunch attachments) and the audit hop, and corrects an earlier claim with what a real re-run actually showed: only the section deployment's workflow_run row goes 'failed' after a mid-turn kill. The folded chat run's row stays 'running' while its durable log carries the terminal event, so the status-based detection signal never fires for the shape proof 4's third hop measures. The relaunch machinery is built; the signal that should trigger it is the remaining gap.
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
The four named remainders from #106, plus the honest result of actually re-running proof 4.
Everything downstream of the detection signal is now built and unit-proven. The detection signal itself turns out not to fire for the shape proof 4's third hop measures — see What the re-run found.
The four remainders
1. Boot-time terminal sweep.
createHubChatPlatform.sweepTerminalRunsscansworkbench_launch(bounded at 100 rows), keeps the participantsisBeyondWakecalls terminal-and-not-merely-parked, and relaunches each through the existingwakeByAddressrelaunch path. Every relaunch and every failure is logged underchat·wake; one failure never aborts the pass.The hub runs it at boot and re-arms it on
sidecar.disconnect, as a short bounded series of passes rather than one. That is not belt-and-braces: a run that died with its sidecar only reads as terminal once the restarted sidecar has packed its log back to the hub, seconds after the reconnect.2. The relaunch notice.
createHubChatPlatformtakes a notice port — a ref, since the adapter is built before the room-message store it posts through — armed in the hub besideroomMessages. Every relaunch, swept or send-triggered, posts into each room the replaced participant belongs to, under the stable address the room has always known it by, so it reads as the same teammate rather than a stranger. The wording is cause-aware (crashed / stopped / ended) and stays in the reader's language: the turn didn't finish, it's back now, say it again.3. Pre-relaunch attachments.
fetchBlobread through the live run's session, and a folded run's mail session resolves from its principal — a fresh run has a fresh principal, so every attachment sent before the crash became unreachable.workbench_launch.prior_run_ids(migration0021) records each run as it is retired, capped at 20, andfetchBlobwalks those sessions newest-first after the live one. A retired run whose row or session is gone is skipped, never fatal.4. Audit assertion. Proof 4 gains a hop that reads the replaced run's events back through the ordinary run routes after its replacement is already answering. That is the fresh-run ruling's whole justification, and it is a hop now rather than an argument.
What the re-run found
Proof 4 was re-run on a real stack — scratch database, real signup, real Ollama, both deploy shapes. Proofs 1, 2, 2b and 3 are green; proof 4's restart and boot restore are green at 9.2s. Hop 3 is still red, and the sweep is not why: nothing in its log fired, because it found nothing to sweep.
#106's note that both top-level runs end up
workflow_run.status = 'failed'holds for the section deployment only. The folded chat run's terminal event goes to its durable log — which is why its supervisor rejects the next message — while itsworkflow_runrow staysrunningforever.isBeyondWakereads that row, so for this shape it answers "still alive", and neither the sweep nor the send-triggered relaunch fires. The room keeps the reader's message with nothing after it: the exact silent drop the hop asserts against.So the remaining gap is the detection signal, not the relaunch. The hub needs a folded run's lifecycle from the same durable log the supervisor reads (
readWorkflowRunLifecycle) rather than fromworkflow_run.status— either by reconciling the row when a restarted sidecar packs a terminal log back, or by havingisBeyondWakeconsult the log directly.The section shape stays red for its own separate reason: a plain workflow deployment has no room, so nothing maps a stable id onto a fresh run for it. That is outside CL-6365's scope, which is the room.
Environment note
The proof's own trap #1 bites harder than documented: the onboarding seed pins the definition's model from
CATALOG_SEEDS.ollama(qwen3.8:27b), andE2E_OLLAMA_MODELonly redirects the proof's own pins — so on an instance without that model, proof 3 dies as "I can't reach a model right now" before proof 4 is ever reached. The re-run named that model locally so the two agreed. Worth closing in the harness separately.Checks
bun run typecheck(whole repo): clean.packages/chat: 576 pass, 0 fail (595 across 51 files).apps/hub: 130 pass, 0 fail.prettier --checkclean on every touched file (7 unrelated files underpackages/onboardingandpackages/cliwere already drifting on the base branch).