Skip to content

Finish a delivery the daemon was restarted in the middle of - #1174

Merged
edwin-zvs merged 1 commit into
mainfrom
service-delivery-survives-restart
Aug 2, 2026
Merged

Finish a delivery the daemon was restarted in the middle of#1174
edwin-zvs merged 1 commit into
mainfrom
service-delivery-survives-restart

Conversation

@edwin-zvs

Copy link
Copy Markdown
Contributor

What broke

A Slack thread was left showing _Working on it…_ with a 👀 reaction, permanently — no answer, no failure, no timeout.

The waiter for a service delivery is a tokio::spawn in process_delivery. It lives exactly as long as the daemon process. When the daemon restarts mid-turn, that task disappears, and with it the only thing that was ever going to replace the placeholder. Nothing times out, because the thing that would have timed out is gone.

The daemon restarted 5 times in the two hours around the incident that prompted this. Every restart landing mid-turn strands a delivery this way, and nothing in the system notices or recovers.

What this changes

A delivery is recorded when accepted and cleared only once something has been said about it. The record lives in the service's existing state file and carries what a later daemon needs — session, transcript cursor, submission time — plus a channel-owned context blob. For Slack that's the thread, the message being answered, the placeholder ts, and whether a reaction was placed.

The context is opaque to the ingress on purpose: the ingress routes and waits, it does not render, and a second kind of channel must not force a schema change on state they share.

The affordance records itself as it appears. A placeholder only exists once a turn has run long enough to deserve one, which is well after acceptance — so run_affordance amends the record the moment it posts. Anything placed and not recorded is exactly what a restart would strand.

On channel start, outstanding deliveries are finished before new ones are taken. Ahead of the socket, so a reconnect can't race a resumed turn for the same thread:

  • Resume by default. The harness outlives the daemon and is reattached, so the turn is usually still running and the answer is still coming. resume_outstanding rebuilds the receipt and re-registers the delivery id — which the reply tool authorizes against, and which did not survive the restart either.
  • On what's left of the original allowance. wait_for_final now takes a budget. A resumed delivery passes TTL - elapsed, so restart loops can't extend a turn indefinitely.
  • Report only what can't be resumed — a session that no longer exists, or an allowance already spent.

Records survive being read. A daemon interrupted again mid-recovery finds them a second time. Resolving one twice edits the message already there; losing one strands it forever. That asymmetry decides it.

process_delivery is split into accept → resolve_delivery, with the latter shared by fresh and resumed deliveries — already_waited is zero for the first kind.

Effect

A restart mid-turn goes from "the thread is stuck forever" to, in the common case, nobody in the channel learning a restart happened at all.

Spec

0182-an-accepted-delivery-outlives-the-daemon (new).

Testing

cargo test --workspace green (30 suites, 0 failures).

New coverage: a record persisting across acceptance → amendment → clear; the record actually reaching disk (an in-memory one is lost by the exact event it exists to survive); a resume refused when the session is gone; and per-channel isolation, including two channels of one service using the same request id without colliding.

This PR touches crates/daemon only → the relevant binary is construct.

🤖 Generated with Claude Code

The task waiting on a service turn lives exactly as long as the daemon
process. A restart takes it with it, silently, and whatever the channel
had already placed for that delivery becomes permanent: a placeholder
that will never be replaced, a reaction that will never be settled. No
timeout fires, because the thing that would have timed out is gone.

This is not rare. Restarts happen on upgrade, on config change, and on
operator command, and a channel accepts deliveries throughout. Every
restart landing mid-turn stranded one.

Record a delivery when it is accepted and keep the record until something
has actually been said about how it ended. Alongside it goes a blob the
channel owns — for Slack, where the thread is and what has been posted
into it — kept opaque here so a second kind of channel does not force a
schema change on state they share.

On starting a channel, finish what a previous daemon left outstanding
before taking anything new. The harness outlives the daemon and gets
reattached, so an interrupted turn is usually still running: pick the
wait back up rather than declaring it lost, on what is left of the
original allowance so surviving a restart cannot extend a turn's life.
Only a delivery whose session is gone, or whose allowance is spent, is
reported as over.

Records are left in place while being resolved rather than consumed on
read. A daemon interrupted again mid-recovery then finds them a second
time: resolving one twice edits the message already there, while losing
one strands it forever.
@edwin-zvs
edwin-zvs force-pushed the service-delivery-survives-restart branch from c3d67c3 to f7bfa9d Compare August 2, 2026 19:07
@edwin-zvs
edwin-zvs merged commit e2b8433 into main Aug 2, 2026
1 check passed
@edwin-zvs
edwin-zvs deleted the service-delivery-survives-restart branch August 2, 2026 19:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant