fix(triggers/streams): job executions are never published to the durable job stream, so no execution trace can span producer → stream → consumer

[rickylabs]

Summary

A triggered job execution completes successfully and emits a job.execute span, but no record for
that execution is ever published to the durable job stream. The stream contains only the three
startup job-definition snapshots. Any check that expects an execution to appear on the stream — and
therefore any end-to-end trace spanning producer → durable stream → SSE consumer — cannot pass,
because there is nothing to consume.

Found by empirical audit during #1329's runtime gate work. This is a product gap, deliberately not
folded into that PR.

Evidence

Full audit: .llm/runs/release-0.0.5--orchestration/slices/w2-b-1329/join-audit.md, evidence commit
6c6e5980a.

A fresh execution was triggered against a real generated service:

• event/correlation trg_evt_ea436f98-c6b3-4c14-8138-14610897da80
• execution id f27ee619-5c8f-426c-ab34-e78b3da5c877
• job flow-b-callback, completed in 43 ms
• job.execute trace/span 02512162b8ecc8b0c75d4fc532656080 / 98f63759e649fccb

Stream state around it:

• Before the trigger, the complete stream was exactly three startup job snapshots at committed
  offset 0000000000000000_0000000000000926: flow-b-callback, health-check,
  workers-plugin-health-check.
• A live subscription was opened from that committed offset before the trigger and held for 25
  seconds. It emitted no data.
• After the trigger, the complete snapshot still contained the same three records at the same
  committed offset. Only the control cursor advanced (2890128 → 2890130).

Join result — no stream record contains any of:

• the execution id f27ee619-…
• the correlation trg_evt_ea436f98-…
• the producer trace id 02512162…

None of the three snapshots' traceparent trace ids equals the execution trace. The only shared
values are definition-level job id/name, which join job.execute to the job definition, not to
the execution.

This distinguishes "published but unmatchable" from "never published": the subscription was held
open across the trigger and received nothing.

Acceptance criteria

• A completed job execution results in a record published to the durable job stream, or the
  design decision not to publish executions is documented and the surfaces that assume otherwise
  are corrected
• If executions are published, the record carries an identifier that joins it to its job.execute
  span — execution id, correlation, or W3C trace context — so a consumer can correlate without
  guessing
• A live subscription opened before a trigger observes the execution record within a bounded time
• A test asserts the join, so a future change that stops publishing executions fails rather than
  silently emptying the stream

Impact

#1329 acceptance row 6 — "producer → durable stream → SSE consumer in one Aspire trace" — cannot be
proven while this holds. #1329's other seven rows are independently proven and its versioned SSE
envelope ships regardless; row 6 is tracked here.

Note the envelope work is not blocked by this and is not the cause: the consumer, offset commit,
reconnect, heartbeat, malformed-frame handling and the extracted-verbatim EventSource example all
work against real service traffic. There is simply no execution record on the wire to correlate.

Provenance

0.0.5 stable-cut orchestration, W2-B slice, 2026-08-09. Split from #1329 row 6.

Refs #1329.

[rickylabs]

Root cause located — start here, do not re-derive

A separate-session IMPL-EVAL of PR #1395 established the mechanism structurally, beyond the
empirical audit this issue was filed on:

• The publication path exists: createStreamMutationHook in
  packages/plugin-workers-core/src/streams/producer.ts mirrors execution records into
  /workers/executions.
• The only production setMutationHook call in the tree is
  plugins/workers/services/src/main.ts:67 — the workers API service.
• Triggered and scheduled executions run in the workers-combined Aspire resource
  (plugins/workers/bin/combined.ts → startCombinedProcess in plugins/workers/bin/runtime.ts),
  which never installs the hook. The dispatcher mutates a KvExecutionState whose #onMutation
  is undefined, so nothing is published.
• The triggers plugin's parallel stream hook is wired nowhere either, which is why no stream
  anywhere carries a trg_evt_*-correlated record.

The fix surface is the missing setMutationHook wiring in the bin/runtime.ts entrypoints, not
the producer, the envelope, or any consumer.

Also established: had a record existed, it would carry value.correlationId equal to the trigger
correlation and a publish-span traceparent sharing the job.execute trace — which is exactly what
#1395's selector matches and what its TC-14 gate asserts. So TC-14 is already this issue's
acceptance test: it fails today for precisely this reason and will pass once executions publish
with correlatable context. Do not write a new one.

[rickylabs]

This issue now owns re-enabling the gate it made red

PR #1395's CI runtime lanes went terminal red on exactly this gap:

FAILED GATE: behavior.otel.stream-consumer
Flow-B stream selection exhausted after 40/40 batch(es) within 20000ms;
  expectedCorrelationId=trg_evt_117c237d-…
  observedCorrelationIds=[flow-b-callback, health-check, workers-plugin-health-check]

postgres passed=77 failed=1, sqlite passed=71 failed=1 — the same three startup records, in CI,
on two engines.

Splitting row 6 to this issue was correct in principle but incomplete in practice: a gate proving
row 6 could not stay a mandatory member of the 0.0.5 suite while the defect it proves lives in
0.0.6. Left alone it would have blocked every subsequent PR and the release canary itself.

behavior.otel.stream-consumer is therefore deferred out of the critical scaffold.runtime
selection — explicitly, named in a comment at the selection site, reported by the suite as deferred
rather than silently omitted, and pinned by a suite-registry test that names this issue. The gate
implementation, TC-14, and both negative tests remain in the tree unchanged.

Added acceptance criterion

• behavior.otel.stream-consumer is restored to the critical scaffold.runtime selection and
  passes, and the suite-registry deferral entry naming this issue is removed

That row is the reason this issue cannot be closed by a fix that merely publishes execution records:
the gate has to go back and go green. Until then the deferral is visible in the suite output and in
a test, not carried in anyone's memory.

Ownership note: the deferral exists solely because of this gap. It is not a judgement that the gate
is wrong — the gate is correct and is this issue's acceptance test.