fix(runner): completion persists as its own batch; consumed child errors are not failure evidence

[wolfy-j]

Production evidence

Dataflow a production dataflow instance on a production deployment: dataflows.status flipped to failed while the agent node stayed running mid-run (26 tool calls, 229k tokens). The dataflow row's metadata.runtime_failure shows the killer:

{reason: "runtime_owner_lost", source: "dataflow.overseer",
 message: "Failed to persist workflow completion: Failed to persist commands:
           COMPLETE_WORKFLOW must be the first command in a command batch ..."}

The run was healthy; the status lied to every consumer.

Three defects fixed

1. COMPLETE_WORKFLOW batch ordering loses the owner (src/runner/orchestrator.lua, src/runner/workflow_state.lua)

A failed transaction leaves its command batch queued for retry (workflow_state contract). handle_complete_workflow then appended COMPLETE_WORKFLOW to that retained batch; the persist validator rejected the batch ("COMPLETE_WORKFLOW must be the first command"), the orchestrator exited, the overseer saw owner loss and terminalized the healthy run as failed — leaving the node row a zombie running.

Fix: workflow_state:queue_completion inserts the completion command at the batch head, so retained commands persist behind the generation fence in one transaction — exactly what the persist layer's completion_blocked machinery is built for (fence first; details applied only when the fence wins; dropped when a newer activation owns the workflow). On fence loss the orchestrator rebuilds workflow state from durable rows before rescheduling, so re-evaluation never acts on outcomes that were never persisted. Exit-path persist failures are now logged instead of silently swallowed.

Reproduced end-to-end: driving the real orchestrator + workflow state + scheduler + persistence with one simulated transaction abort yields the exact production error string pre-fix.

2. Consumed child errors counted as workflow failure evidence (src/node/agent/*, src/runner/workflow_state.lua)

get_failed_node_errors counted every COMPLETED_FAILURE node — including tool.call viz children and delegated children whose error the parent agent consumed as an observation and continued past. Any workflow failure was then attributed to the handled tool child (the false trace read off the production run). The parent now declares error_observed on the child in the same durable commit as the delivered observation, and failure evidence derives only from unhandled failures. Aggregate status derivation was verified correct (a failed tool child never flips dataflows.status) and is locked by regression tests.

3. Postgres terminal lifecycle lock order (src/persist/*, pre-existing commit on this branch)

Terminal lifecycle transactions upgraded the workflow row lock after activation/wake writes, deadlocking with concurrent commits holding FK KEY SHARE locks on postgres — the class of transient transaction abort that triggers defect 1. Locks now follow the canonical parent-first order.

Tests

• orchestrator_completion_flush_test (new): completion retries the retained batch as a fenced batch and completes; fence-loss rebuilds from durable state and re-derives the outcome (real orchestrator/state/scheduler/persist, one simulated tx abort).
• agent_tool_failure_test (new): a failing tool consumed by the agent leaves the aggregate completed with the agent node driven to its own terminal status; an unhandled agent failure is attributed to the agent node, never the consumed tool child.
• delegation_handler_test (extended): consumed delegation failures are declared on the child and excluded from failure evidence.
• Full suite: 956 tests passed, 0 failed (sqlite profile); lint clean (one pre-existing unrelated warning).

Reviews

Codex read-only reviews across three iterations; final verdict: APPROVE — no blocking issues, "queue_completion correctly prepends the fence; fence loss reloads durable state; no compensation or weakened assertions found."