Skip to content

fix(workers): publish job executions to the durable stream on the job.execute trace - #1536

Merged
rickylabs merged 5 commits into
mainfrom
fix/1398-publish-job-executions-to-durable-stream
Aug 12, 2026
Merged

fix(workers): publish job executions to the durable stream on the job.execute trace#1536
rickylabs merged 5 commits into
mainfrom
fix/1398-publish-job-executions-to-durable-stream

Conversation

@rickylabs

@rickylabs rickylabs commented Aug 12, 2026

Copy link
Copy Markdown
Owner

Summary

Publishes every workers execution-state mutation from the background worker runtimes and makes each durable-stream publish span inherit the execution's stored W3C context. This closes the wiring gap that left generated workers-combined processes emitting job.execute spans without execution records on the durable stream.

Scope

Slices

Validation

  • scoped workers check — PASS, 101 files / 0 findings
  • scoped plugin-workers-core check — PASS, 111 files / 0 findings
  • scoped workers lint + format — PASS, 101 files / 0 findings each
  • deno task --cwd packages/plugin-workers-core test — PASS, 27 passed / 0 failed
  • focused CLI E2E registry/runner tests — PASS, 27 passed / 0 failed
  • full CLI E2E test tree from repo root — PASS, 152 passed / 0 failed
  • deno task quality:gate — PASS; touched roots report doctrine FAIL=0
  • explicit plugin-workers-core and CLI E2E quality scans — PASS, 0 findings / 0 allowances
  • controlled negative checks — D3 removal failed both trace guards; combined-hook removal failed the installation guard; removing both restored gates failed both suite-registry guards
  • deno task e2e:cli run scaffold.runtime --cleanup --format prettyRED, raw exit 1 at runtime.flow-b-fixture: netscript generate plugins failed: Error: fetch failed; summary passed=33 failed=1 skipped=0; cleanup passed. The suite stopped before either restored OTEL gate ran and was not retried.

Harness

  • Run dir: .llm/runs/release-0.0.6-features--orchestration/
  • Slice worklog: .llm/runs/release-0.0.6-features--orchestration/slices/worklog-1398.md
  • Phase: impl — see phase comments below.

Drift / Debt

Definition of Done

  • Background worker and combined runtimes install the execution stream mutation hook.
  • Created, updated, and deleted execution publications inherit the stored W3C trace id.
  • Unit guards fail when D3 or the combined-runtime installation is removed.
  • Both fix(triggers/streams): job executions are never published to the durable job stream, so no execution trace can span producer → stream → consumer #1398 OTEL deferrals are removed with the runtime-suite tests updated.
  • scaffold.runtime exits 0 with behavior.otel.stream-consumer and behavior.otel.traces passing. Verified in CI on both runtime tiers against head f7d503fee, gates observed by name in the job logs: postgres job 94073971396 (passed=88 failed=0 skipped=0) and sqlite job 94073971501 (passed=83 failed=0 skipped=0). Earlier local runs went red before reaching these gates; CI running the same suite with this change to a clean finish establishes those as environmental to the local host.
  • Separate-session IMPL-EVAL passes — automatic phase dispatcher, openrouter/qwen/qwen3.8-max; see verdict comment.
  • Orchestrator review and merge-authority checks are complete (seven-check pre-merge gate recorded in the run dir).
issue: 1398
entries:
  - box-index: 1
    evidence: "The execution stream mutation hook is installed in startWorkerProcess and startCombinedProcess (plugins/workers/bin/runtime.ts); the new installation guard fails with actual 0 / expected 1 when it is removed. Live on head f7d503fee: behavior.otel.stream-consumer passed in CI jobs 94073971396 and 94073971501."
  - box-index: 2
    evidence: "The hook publishes inside context.with(extractContext({traceparent, tracestate})) so the publish span joins the job.execute trace (packages/plugin-workers-core/src/streams/producer.ts). Live TC-14 asserts the record header traceparent trace id equals the job.execute trace id; behavior.otel.traces passed in CI jobs 94073971396 and 94073971501."
  - box-index: 3
    evidence: "Live-verified on head f7d503fee: behavior.otel.stream-consumer opens a subscription before the trigger and observes the execution record within the gate's own bounded SSE loop (consume-flow-b-stream.ts:203-230). Passed on both runtime tiers, failed=0 skipped=0."
  - box-index: 4
    evidence: "Controlled negatives: removing the trace-context wrapper fails both trace-join guards; removing the combined-runtime hook installation fails the installation guard (actual 0, expected 1); removing the two restored gates from RUNTIME_GATES fails the suite-registry guards. Deferral removal is pinned by suite-registry_test.ts and suite-runner_test.ts."

@rickylabs rickylabs added this to the 0.0.6 milestone Aug 12, 2026 — with ChatGPT Codex Connector

Copy link
Copy Markdown
Owner Author

[PHASE: IMPL] [SLICE: S1]

Landed 3e179ec30: background hook installation plus stored-trace-context publication.

Real gate output

$ deno run --allow-read --allow-run .llm/tools/run-deno-check.ts --root plugins/workers --ext ts,tsx
exit 0
{"selection":{"filesSelected":101,"batches":1,"failedBatches":0},"summary":{"totalOccurrences":0,"uniqueOccurrences":0,"uniqueCodes":0,"uniquePaths":0}}

$ deno run --allow-read --allow-run .llm/tools/run-deno-check.ts --root packages/plugin-workers-core --ext ts,tsx
exit 0
{"selection":{"filesSelected":111,"batches":1,"failedBatches":0},"summary":{"totalOccurrences":0,"uniqueOccurrences":0,"uniqueCodes":0,"uniquePaths":0}}

$ deno run --allow-read --allow-run .llm/tools/run-deno-lint.ts --root plugins/workers --ext ts,tsx
exit 0
{"selection":{"filesSelected":101,"batches":1},"summary":{"totalOccurrences":0,"uniqueOccurrences":0,"uniqueRules":0,"uniquePaths":0}}

$ deno run --allow-read --allow-run .llm/tools/run-deno-fmt.ts --root plugins/workers --ext ts,tsx
exit 0
{"summary":{"filesSelected":101,"batches":1,"failedBatches":0,"findings":0,"ignoredFindings":0}}

$ deno task --cwd packages/plugin-workers-core test
exit 0
ok | 27 passed | 0 failed

$ deno test --allow-all plugins/workers/tests/runtime/background-stream-hook_test.ts
exit 0
ok | 1 passed | 0 failed

$ deno task quality:gate
exit 0
quality:scan ok=true findings=[]
arch:check touched roots: workers FAIL=0; plugin-workers-core FAIL=0

$ deno run --allow-read .llm/tools/quality/scan-code-quality.ts --root packages/plugin-workers-core --pretty
exit 0
{"ok":true,"scanned":["packages/plugin-workers-core"],"findings":[],"allowCount":0,"allowances":[]}

Guard-integrity evidence

D3 removed:
exit 1
createStreamMutationHook publishes mutations on the stored trace context ... FAILED
actual aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
expected 11111111111111111111111111111111
createStreamMutationHook publishes pre-span mutations on the stored trace id ... FAILED
actual 60424cbba9fc12066fcf09428923316a
expected 11111111111111111111111111111111

Combined installation removed:
exit 1
startCombinedProcess installs the execution stream mutation hook ... FAILED
actual 0
expected 1

Reconcile

Issue #1398 remains open at milestone 0.0.6 with type:fix, area:plugins, area:telemetry, priority:p1, and exactly one lifecycle label: status:impl.

Next

  • S2: remove both deferred OTEL gates and update both pinned registry tests in one commit.

Copy link
Copy Markdown
Owner Author

[PHASE: IMPL] [SLICE: S2]

Landed f80339f17: both #1398 OTEL gates now execute in the Postgres and SQLite runtime tiers; all stale deferral assertions are updated.

Real gate output

$ deno test --allow-all packages/cli/e2e/tests/application/runner/suite-runner_test.ts packages/cli/e2e/tests/presentation/suite-registry_test.ts
exit 0
ok | 27 passed | 0 failed

$ deno test --allow-all packages/cli/e2e/tests/
exit 0
ok | 152 passed | 0 failed

$ deno run --allow-read --allow-run .llm/tools/run-deno-check.ts --root packages/cli/e2e --ext ts,tsx
exit 0
{"selection":{"filesSelected":163,"batches":2,"failedBatches":0},"summary":{"totalOccurrences":0,"uniqueOccurrences":0,"uniqueCodes":0,"uniquePaths":0}}

$ deno run --allow-read --allow-run .llm/tools/run-deno-lint.ts --root packages/cli/e2e --ext ts,tsx
exit 0
{"selection":{"filesSelected":163,"batches":1},"summary":{"totalOccurrences":0,"uniqueOccurrences":0,"uniqueRules":0,"uniquePaths":0}}

$ deno run --allow-read --allow-run .llm/tools/run-deno-fmt.ts --root packages/cli/e2e --ext ts,tsx
exit 0
{"summary":{"filesSelected":163,"batches":1,"failedBatches":0,"findings":0,"ignoredFindings":0}}

$ deno run --allow-read .llm/tools/quality/scan-code-quality.ts --root packages/cli/e2e --pretty
exit 0
{"ok":true,"scanned":["packages/cli/e2e"],"findings":[],"allowCount":0,"allowances":[]}

Guard-integrity evidence

Both OTEL gates temporarily removed from RUNTIME_GATES:
exit 1
runtime suite includes full scaffold, database, runtime, and behavior gates ... FAILED
actual false
expected true
runtime suites execute the formerly deferred #1398 OTEL gates ... FAILED
actual false
expected true

Red gates reported and resolved

First full check: TS2339 deferred.issue / deferred.reason on never.
Cause: empty as-const tuple inferred no element type.
Resolution: explicit readonly DeferredGate[] contract; no behavior/assertion relaxed.

Second package-task run:
- in-scope red: suite-runner_test still pinned two #1398 skipped steps; updated in this commit.
- unrelated red: quickstart-command-drift_test read docs/site/quickstart.vto relative to packages/cli/e2e cwd.
Repository-root verification of that test: exit 0, 1 passed / 0 failed.
Repository-root full CLI E2E tests: exit 0, 152 passed / 0 failed.

Drift

PLAN-EVAL F2 named both stale tests in suite-registry_test.ts; the full suite found a third stale #1398 assertion in suite-runner_test.ts. It was necessary D5 fallout and is included in S2, with the factual drift recorded in the slice worklog.

Next

  • Run the final required static/package gates on the two-commit head.
  • Confirm no other serialized scaffold.runtime process is active.
  • Run exactly one deno task e2e:cli run scaffold.runtime --cleanup --format pretty.

@rickylabs

Copy link
Copy Markdown
Owner Author

[PHASE: IMPL] [SLICE: S3]

Commit: f8fa81f2c

The serialized one-pass live runtime gate was run once. No suite lease, competing scaffold.runtime process, or Docker container was present before launch.

$ deno task e2e:cli run scaffold.runtime --cleanup --format pretty
> runtime.flow-b-fixture: Wire real Flow-B callback fixture
  FAILED 10777ms
    Command exited 1; expected 0.
    error: Uncaught (in promise) Error: netscript generate plugins failed: Error: fetch failed
> cleanup.aspire-stop: Stop generated Aspire AppHost
  PASSED 81ms
Summary: passed=33 failed=1 skipped=0
raw exit code: 1

The failure preceded Aspire launch, so behavior.otel.stream-consumer and behavior.otel.traces did not execute and live #1398 acceptance remains unverified. The suite was not retried.

Diagnostics:

plugin doctor: all configured plugins healthy; expected AppHost-not-running warning only
focused preserved-fixture generator: exit 0; 3 registries written
agentic:leak-check: Aspire ok, Docker ok, survivors=[]
docker ps: 0 running

This was not the known Missing plugin reference "streams" residual. The nested Deno failure exposed no URL; the isolated generator's subsequent pass is consistent with a transient dependency/registry fetch but does not change the authoritative red E2E verdict.

@rickylabs

Copy link
Copy Markdown
Owner Author

[PHASE: REVIEW]

Tier-A slice review by the orchestrator (review_codex-tier pairing for the Sol·medium lane). Claims checked against the diff, not read off the report.

Conformance to the locked plan

Decision Implemented Where
D1 hook installed on background runtimes yes, and correctly not on the scheduler plugins/workers/bin/runtime.ts:92,131startWorkerProcess + startCombinedProcess only
D2 publish all mutations yes — created/updated/deleted all routed plugin-workers-core/src/streams/producer.ts:113-128
D3 wrap publish in the stored trace context yes — this was the trap withContext(extractContext({traceparent, tracestate}), …) around both delete and upsert
D4 no WorkerExecutionZodSchema change yes — streams/schema.ts untouched (0 changes in diff) trace fields added to the WorkerExecutionRecord type only
D5 un-defer both gates, both tests, same commit yes, plus a third test the plan missed f80339f17

D3 is the one that would have shipped a silent failure. PLAN-EVAL raised it (F1) precisely because installing the hook without the wrapper looks correct and then fails TC-14 on the pre-span create() record. It is implemented as specified.

A third stale pin the plan and PLAN-EVAL both missed

The plan named suite-registry_test.ts:204-215; PLAN-EVAL F2 found the second pin at :209-234. The slice found a thirdsuite-runner_test.ts, still expecting two deferred skipped steps — and fixed it in the same commit. Neither the plan nor the PLAN-EVAL caught it; the implementer did, by running the full package tests rather than only the tests it had been pointed at. Recorded as factual drift, correctly.

It also hit and fixed a real type trap: an empty as const satisfies readonly DeferredGate[] infers the element type as never, breaking fixture code that reads issue/reason. Declaring against the explicit readonly DeferredGate[] contract fixes it without relaxing any assertion.

Negative guards, per decision

Reported and consistent with the diff:

  • Removing only D3 → both trace-context tests fail (ambient trace id retained for the override case; a fresh random trace id for the pre-span case).
  • Removing only the combined-process hook installation → installation test fails with actual count 0, expected 1. That test (plugins/workers/tests/runtime/background-stream-hook_test.ts) is new, and its absence is why workers-combined shipped without the hook in the first place.

The live gate is RED, and that is the honest result

runtime.flow-b-fixture: FAILED 10777ms
  netscript generate plugins failed: Error: fetch failed
Summary: passed=33 failed=1 skipped=0 · raw exit code: 1

The failure landed before Aspire launched and therefore before either restored gate ran, so behavior.otel.stream-consumer and behavior.otel.traces received no live verdict. The slice states this plainly and does not claim acceptance from it — the correct call, and it did not retry the suite to manufacture a green.

Its diagnosis: not the known Missing plugin reference "streams" residual (that message never appeared); a focused rerun of only the failed generate plugins command then passed in 0.9 s, consistent with a transient registry fetch, though the nested Deno error exposed no URL so the endpoint is unconfirmed. agentic:leak-check reported no survivors and Docker showed zero containers after cleanup.

#1398's acceptance criterion 3 has no live evidence yet, so this PR cannot merge. As orchestrator I have re-run the gate once against the branch updated to current main (so the evidence matches the merge candidate), after confirming serialisation: no lease file, no competing e2e:cli process, zero Docker containers. Result pending.

Next

  • Live scaffold.runtime re-run in progress; both restored gates must pass for real.
  • IMPL-EVAL only after that, then the seven-check pre-merge gate.
  • If the fixture failure reproduces, it is a genuine blocker to diagnose, not a flake to paper over.

@rickylabs
rickylabs marked this pull request as ready for review August 12, 2026 08:53
rickylabs added a commit that referenced this pull request Aug 12, 2026
…spatcher

Owner ruling: once #1524 lands, phase evaluations use the automatic status
workflow unless the owner documents a local route or skip. #1536 stays on its
current head and status; root re-enters status:impl-eval with the Qwen override
after #1524 merges, and this lane only watches the verdict and finishes the
merge gate.

Records that the anticipated local #1536 evaluator was never launched -- prompt
and worktree were prepared, the decision was raised instead, and no evaluator
process or output exists -- so no duplicate spend occurred.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TKxrWGp5uxEHQ2NyZiZSMF
@rickylabs

Copy link
Copy Markdown
Owner Author

[PHASE: IMPL] — live gate evidence

The two gates deferred against #1398 have now run by name and passed live in CI, on both runtime tiers. This is the evidence acceptance criterion 3 was missing.

Verified from the job logs directly rather than inferred from a suite-level green:

scaffold-runtime (aspire + docker + postgres) — job 94062070840

> runtime.flow-b-fixture: Wire real Flow-B callback fixture
> behavior.otel.stream-consumer: Consume real Flow-B stream with fan-in links
> behavior.otel.traces: Validate OTEL trace chain via Dashboard API
Summary: passed=88 failed=0 skipped=0

scaffold-runtime-sqlite (aspire + sqlite + garnet) — job 94062070984

> behavior.otel.stream-consumer: Consume real Flow-B stream with fan-in links
> behavior.otel.traces: Validate OTEL trace chain via Dashboard API
Summary: passed=83 failed=0 skipped=0

failed=0 skipped=0 on both — so neither gate was silently skipped, which is the failure mode the deferral list existed to make visible in the first place.

The SQLite tier genuinely carries these gates: POSTGRES_ONLY_RUNTIME_GATES (capability-suites.ts:146-151) contains only the four DB-specific gates, so RUNTIME_SQLITE_GATES inherits both OTEL gates from RUNTIME_GATES. Both tiers exercising them is stronger evidence than the plan asked for.

This also settles the two red local runs

Local scaffold.runtime went red twice before reaching these gates — run 1 at runtime.flow-b-fixture (generate plugins: fetch failed), run 2 at runtime.wait.triggers-api (unhealthy after 120s). I would not call those flakes without evidence. CI now runs the same suite, with this change, through runtime.flow-b-fixture and past triggers-api to a clean finish on both tiers — so the failures were environmental to the local WSL host, not caused by this change. Recorded rather than assumed.

Remaining before merge

No merge until that verdict lands and the seven-check pre-merge gate is recorded.

rickylabs added a commit that referenced this pull request Aug 12, 2026
Files the quality:gate root-coverage gap (hit independently by three sessions,
and the reason the #1405 merge record rests on a target scan rather than the
repo gate) and the undeclared plugin-streams-core imports, the latter filed as
explicitly unverified with publish:dry-run evidence as its first acceptance box.

Records the waiting state for #1536: #1524 is still open, so no automatic
verdict exists and the merge gate cannot proceed. The staged body transform is
dry-run only and deliberately unapplied.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TKxrWGp5uxEHQ2NyZiZSMF
rickylabs added a commit that referenced this pull request Aug 12, 2026
#1524 merged 7837ef4; the automatic phase dispatcher is live and D-4 routing
is in force. Records #1536 verified unchanged at that moment and the exact
baseline the verdict watch measures against.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TKxrWGp5uxEHQ2NyZiZSMF
rickylabs added a commit that referenced this pull request Aug 12, 2026
Formal PLAN/IMPL eval triggers on labels, never a manual OpenHands dispatch.
This lane already complies: no manual dispatch, and no local evaluator was
launched for #1536.

Records a measured timing finding: #1536's ready_for_review (08:53:43Z) and its
status:impl-eval application (08:53:45Z) both precede the dispatcher merging
(09:24:15Z), so the initial automatic IMPL-EVAL could not have fired and will
not fire on its own -- the label re-entry is required, not merely planned.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TKxrWGp5uxEHQ2NyZiZSMF
@rickylabs rickylabs added eval:model:qwen Override the next OpenHands phase evaluator to Qwen 3.8 Max status:impl status:impl-eval and removed status:impl-eval status:impl labels Aug 12, 2026
rickylabs added a commit that referenced this pull request Aug 12, 2026
Label re-entry executed exactly as instructed and confirmed clean, but the
phase-eval workflow produced no run: openhands-phase-eval.yml is absent from
head e4319c6, whose last main sync (08:45:46Z) predates #1524 merging
(09:24:15Z). pull_request events resolve workflows from the PR merge ref, so no
label cycling can trigger it.

main has also moved to #1547, which fixes the dispatcher's own token step.
Syncing the branch changes the head and forces a full CI re-run, so it is
raised rather than taken unilaterally.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TKxrWGp5uxEHQ2NyZiZSMF
@rickylabs

Copy link
Copy Markdown
Owner Author

@openhands-agent model=openrouter/qwen/qwen3.8-max output=pr-comment iterations=800 phase=impl head=f7d503fee1e312335d432cfee8b9a870ddaccbb4

Trusted base SHA: 281ab76
Evaluated head SHA: f7d503f

use harness

SKILL

  • netscript-harness — apply the formal IMPL-EVAL protocol and verdict vocabulary.
  • openhands-handoff — publish one machine-readable OpenHands verdict.
  • netscript-tools — run the smallest decisive repository-native gates without mutating source.
  • netscript-doctrine — apply package/plugin doctrine when the changed surface requires it.

Act as the formal IMPL-EVAL session for this pull request. Do not edit files, create commits, push,
or repair findings. The trigger metadata supplies the trusted base SHA and immutable head SHA: read
the evaluator protocol, verdict definitions, and selected profiles from that base commit, then
evaluate the PR body, linked issues, run artifacts, final diff, review threads, and architecture
debt at the immutable head. Verify the approved plan or recorded PLAN-EVAL: N/A, design
checkpoint, acceptance criteria, static/runtime/consumer gates, public surface, lock hygiene, and
false-done states. For documentation changes, also read every changed document fully and hand-test
representative executable claims.

Return concise, severity-ranked findings with exact evidence and required action. End with exactly
one supported verdict line using OPENHANDS_VERDICT: PASS, OPENHANDS_VERDICT: FAIL_FIX,
OPENHANDS_VERDICT: FAIL_RESCOPE, OPENHANDS_VERDICT: FAIL_DEBT, or
OPENHANDS_VERDICT: FAIL_PLAN. Write the same verdict to OPENHANDS_SUMMARY_PATH.

@rickylabs rickylabs removed the eval:model:qwen Override the next OpenHands phase evaluator to Qwen 3.8 Max label Aug 12, 2026
@rickylabs

rickylabs commented Aug 12, 2026

Copy link
Copy Markdown
Owner Author

OpenHands Agent — Completed

OPENHANDS_VERDICT: PASS

Model: openrouter/qwen/qwen3.8-max
Provider: OPENROUTER
Job status: success
Verdict source: summary-file
Verdict comment: #1536 (comment)
Summary source: agent
Housekeeping: ack=success trace=success commit-artifacts=skipped replies=skipped draft-pr=skipped
Trace: .llm/tmp/run/openhands/pr-1536/run-31584188459-1 (in the Actions artifact)
Artifact: openhands-agent-31584188459-1

OPENHANDS_VERDICT: PASS

IMPL-EVAL — PR #1536 (fix(workers): publish job executions to the durable stream on the job.execute trace)

Subject: head f7d503fee1e312335d432cfee8b9a870ddaccbb4 vs trusted base 281ab76887, PR #1536 closes #1398.
Protocol: .llm/harness/evaluator/protocol.md + verdict-definitions.md, read-only.

Summary

The implementation satisfies the approved plan (S1/S2/S3, D1–D5), and every static gate and
local runtime guard passes. The decisive live runtime evidence — the two formerly deferred OTEL
gates behavior.otel.stream-consumer and behavior.otel.traces — now passes green in CI at
this exact head
on both runtime tiers (postgres and sqlite), superseding the locally red S3
attempt that died on a transient dependency fetch failed before those gates could run. The
only red check (close-gate) is a merge-time bookkeeping step, not an implementation defect.

Changes (verified against plan D1–D5)

  • packages/plugin-workers-core/src/streams/producer.tscreateStreamMutationHook wraps every
    publish (upsert/delete) in withContext(extractContext({traceparent, tracestate}), …) (D3,
    including the pre-span create() trap); WorkerExecutionRecord type acknowledges the
    already-stored W3C fields (no schema change, D4 honored).
  • plugins/workers/bin/runtime.ts — hook installed in startWorkerProcess and
    startCombinedProcess; scheduler-only correctly untouched (D1).
  • plugins/workers/tests/runtime/background-stream-hook_test.ts — new installation guard (D1).
  • packages/plugin-workers-core/tests/streams/workers-streams_test.ts — stored-context and
    pre-span trace-id join tests (required tests 1 & 3).
  • packages/cli/e2e/** — both OTEL gates restored into RUNTIME_GATES, deferral constant
    emptied, both pinned tests rewritten (D5, F2 honored; third stale pin in suite-runner_test.ts
    recorded as drift).
  • .llm/runs/.../slices/worklog-1398.md — design checkpoint, per-slice gates, negative-guard
    evidence, honest S3 red record.

Validation

  • PLAN-EVAL: PASS (MiniMax M3, separate session) with F1/F2 folded into plan; run-dir artifacts
    confirmed on chore/release-0.0.6-features-orchestration.
  • Re-run at head: deno task --cwd packages/plugin-workers-core test → 27/27 passed;
    background-hook install test → 1/1 passed.
  • CI at head: scaffold-runtime (postgres)Summary: passed=88 failed=0 skipped=0 with both
    restored OTEL gates executing; scaffold-runtime-sqlitepassed=83 failed=0 skipped=0;
    check-test, quality, code-quality, deps-report, surface-diff all PASS.
  • Diff hygiene: no new as unknown as / deno-lint-ignore / @ts-ignore / any (excluding
    run artifacts); no deno.lock churn; no new public exports.
  • Context-join mechanism verified end-to-end: hook → producer.upsertDurableStreamProducer
    #startPublishinstrumentation.startPublish (tracer.startSpan on ambient context), so
    withContext makes the publish span carry the stored trace id; pinned by both new unit tests.

Responses to review/issue context

Remaining risks

OPENHANDS_VERDICT: PASS

Run: https://github.com/rickylabs/netscript/actions/runs/31584188459

@rickylabs

Copy link
Copy Markdown
Owner Author

OPENHANDS_VERDICT: PASS

[PHASE: IMPL-EVAL] head f7d503fee vs trusted base 281ab7688 · PR #1536 closes #1398 · protocol .llm/harness/evaluator/protocol.md

Verdict rationale

The implementation satisfies the approved plan (S1/S2/S3, D1–D5), every static gate and local runtime guard passes, and the decisive live runtime evidence is now green: the two formerly deferred OTEL gates (behavior.otel.stream-consumer, behavior.otel.traces) pass in CI at this exact head on both runtime tiers, superseding the locally red S3 attempt that died on a transient dependency fetch failed before those gates could run.

Verified

  • Plan lineage. PLAN-EVAL = PASS (MiniMax M3, separate session) with F1/F2 folded into the plan; approved plan, research, and this issue's slice brief confirmed on chore/release-0.0.6-features-orchestration. Design checkpoint present in slices/worklog-1398.md with per-slice gate tables and negative-guard evidence.
  • D3 context join (the core claim). createStreamMutationHook wraps every publish in withContext(extractContext({traceparent, tracestate}), …); verified end-to-end: hook → producer.upsertDurableStreamProducer#startPublishinstrumentation.startPublish, whose tracer.startSpan runs on the ambient context, so the publish span carries the stored trace id. Both new unit tests pin this (stored-context override and pre-span create() trap).
  • D1 installation. Hook installed in startWorkerProcess and startCombinedProcess; scheduler-only correctly untouched. Re-run at head: deno task --cwd packages/plugin-workers-core test27/27 passed; background-stream-hook_test.ts1/1 passed.
  • D5 un-deferral. Both OTEL gates restored to RUNTIME_GATES, SCAFFOLD_RUNTIME_DEFERRED_GATES emptied, both pinned suite tests rewritten (F2 honored); the third stale pin in suite-runner_test.ts was recorded as drift, not hidden.
  • Live gates at head (CI run 31584138059). scaffold-runtime (aspire + docker + postgres)Summary: passed=88 failed=0 skipped=0 with both restored gates executing; scaffold-runtime-sqlitepassed=83 failed=0 skipped=0; check-test, quality, code-quality, deps-report, surface-diff all PASS.
  • Hygiene. No new as unknown as / deno-lint-ignore / @ts-ignore / any in the diff (excluding run artifacts); no deno.lock churn; no new public exports; PR body carries Closes #1398, milestone 0.0.6, labels consistent.

Findings (advisory, non-blocking)

  1. close-gate is red for bookkeeping, not code. Issue fix(triggers/streams): job executions are never published to the durable job stream, so no execution trace can span producer → stream → consumer #1398's four acceptance boxes and the PR DoD's final three boxes are unticked because the acceptance-evidence mirror only applies once status:ready-merge is present (gate log: "Mirror skipped because live PR labels do not include status:ready-merge"). The PR body already contains the fenced acceptance-evidence block mapping all four boxes — the same mechanism that ticked merged sibling PR fix(streams): name the closing and refusal states in producer write reasons #1528. Required action (orchestrator, merge time): apply status:ready-merge (triggering the mirror) and tick the one-pass scaffold.runtime DoD box with the CI run evidence.
  2. Stale DoD wording. The PR DoD line about the one-pass run still cites the local red; when ticking it should cite CI run 31584138059 (and sqlite job 94073971501) where both gates executed green.
  3. Pre-existing coverage gap (recorded, not introduced). quality:gate roots omit packages/plugin-workers-core/cli/e2e; the worklog's explicit scans cover it. Out-of-scope undeclared @netscript/plugin-streams-core imports remain tracked separately per PLAN-EVAL F5.

This comment was generated by an AI agent (OpenHands) on behalf of the NetScript agentic runtime.

@rickylabs rickylabs added status:augment-review Advisory augment/review pass status:ready-merge and removed status:impl-eval status:augment-review Advisory augment/review pass labels Aug 12, 2026
@rickylabs
rickylabs merged commit d7e2b67 into main Aug 12, 2026
30 of 31 checks passed
rickylabs added a commit that referenced this pull request Aug 12, 2026
…#1525)

* chore(harness): activate the 0.0.6 runtime lane orchestration run

Bootstraps .llm/runs/release-0.0.6-features--orchestration/ for the topical
runtime/public-surface lane that owns #1405 and #1398.

Records identity/worktree proof, the live re-baseline of both issue bodies,
the lane bindings and their two recorded deviations, the PLAN-EVAL/IMPL-EVAL
decisions, and the #1405 research with line-cited call sites for both
misreported reason strings.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TKxrWGp5uxEHQ2NyZiZSMF

* docs(harness): plan #1398 on a verified trace-context join

Records the early-terminated #1398 research with its unverified list intact,
and plans the fix around two facts checked in-session: job.execute inherits the
stored dispatch traceparent, and the stream publish span starts on the ambient
context. Publishing execution mutations under the record's stored trace context
therefore joins every record -- including the pre-span create() one -- to the
job.execute trace, which is what the deferred TC-14 gate asserts.

Makes the issue's definition of done mechanical: the two Flow-B OTEL gates
deferred against #1398 come out of the deferral list and pass live.

Also records the #1405 Tier-D dispatch identity and corrects an earlier worklog
line that read runtime doctor "sessions: 0" as "nothing is running".

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TKxrWGp5uxEHQ2NyZiZSMF

* docs(harness): record the #1405 slice review and its negative-case proof

Re-verifies the slice independently (33/33) and demonstrates the new guards
actually fire by reverting both fixes and observing 5 failures, per the
gate-integrity rule that a guard enters only with its predicate proven.

Records two advisory findings, and a defect in the orchestrator's own slice
brief that the implementer surfaced rather than hid.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TKxrWGp5uxEHQ2NyZiZSMF

* docs(harness): resolve #1398 S0 and correct a wrong scope inference

workers-combined does receive services__streams__http__0: background
PluginReferences are reconciled from the plugin manifest dependency, asserted
by install-plugin_test.ts, not declared in the Aspire contribution file.

Records that the orchestrator's first reading of the contribution file led to
the opposite conclusion and would have added an unnecessary Aspire slice.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TKxrWGp5uxEHQ2NyZiZSMF

* docs(harness): record #1398 PLAN-EVAL PASS and amend the plan

MiniMax M3 separate-session verdict, verbatim. Folds in two verified findings:
the trace-context join must explicitly wrap producer.upsert in context.with
because StreamsTracerPort.startSpan takes no parent context, and D5 must update
both tests that pin the deferral, not just the one the plan named.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TKxrWGp5uxEHQ2NyZiZSMF

* docs(harness): dispatch the #1398 slice under the amended plan

Records the PLAN-EVAL PASS detail, the two findings verified and folded in, and
the Tier-D dispatch identity for the #1398 implementation thread.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TKxrWGp5uxEHQ2NyZiZSMF

* docs(harness): record the #1405 IMPL-EVAL PASS

DeepSeek V4 Flash 0731 verbatim verdict. Its per-fix revert isolation is what
makes acceptance box 4 real: reverting each fix alone fails only that fix's
tests, so the reasons are pinned individually rather than in aggregate.

Records why the unreachable ?? fallback stays -- the suggested cleanup needs a
non-null assertion the slice brief forbids -- and that the apparently-missing
research file was an artifact of the evaluator worktree's base, not a gap.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TKxrWGp5uxEHQ2NyZiZSMF

* docs(harness): record the #1405 merge in cut-trace

Captures merge 8ff1bcb from origin/main first-parent history after the fact,
with the seven-check pre-merge gate record and the draft-CI skipping trap that
check 4 caught before it could be read as green.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TKxrWGp5uxEHQ2NyZiZSMF

* docs(harness): record the owner ruling on IMPL-EVAL for small taxonomy fixes

Owner ruling: the #1405 class does not get a separate evaluator; focused
negative tests, CI, close-gate, and independent diff review are sufficient.
Records that I had read the waiver as a blocked-transport fallback rather than
the class default, which cost one unnecessary dispatch, and that the ruling
arrived after #1405 had already merged so it does not retract that record.

Formal PLAN/IMPL eval remains mandatory for #1398.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TKxrWGp5uxEHQ2NyZiZSMF

* docs(harness): record two red #1398 live runs and escalate the verdict to CI

Neither local scaffold.runtime run reached the restored OTEL gates -- run 1 died
at fixture generation on a transient fetch, run 2 at a triggers-api health
timeout after getting 17 steps further. Acceptance criterion 3 stays unproven
and the PR cannot merge.

States what the evidence does and does not support: the triggers-api timeout is
consistent with an environmental failure but is not proven to be one, because
the suite was not reproduced on a clean main checkout.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TKxrWGp5uxEHQ2NyZiZSMF

* docs(harness): route future phase evaluation through the automatic dispatcher

Owner ruling: once #1524 lands, phase evaluations use the automatic status
workflow unless the owner documents a local route or skip. #1536 stays on its
current head and status; root re-enters status:impl-eval with the Qwen override
after #1524 merges, and this lane only watches the verdict and finishes the
merge gate.

Records that the anticipated local #1536 evaluator was never launched -- prompt
and worktree were prepared, the decision was raised instead, and no evaluator
process or output exists -- so no duplicate spend occurred.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TKxrWGp5uxEHQ2NyZiZSMF

* docs(harness): record #1398 live gate PASS on both CI runtime tiers

Both formerly-deferred OTEL gates ran by name with failed=0 skipped=0 on the
postgres and sqlite tiers, verified from job logs. Acceptance criterion 3 is
live-verified.

Retires the open question about the two red local runs: CI runs the same suite
with this change to a clean finish, which is the control I lacked, so the local
triggers-api timeout is established as environmental rather than merely
consistent with it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TKxrWGp5uxEHQ2NyZiZSMF

* docs(harness): file the lane's two deferred findings as #1542 and #1543

Files the quality:gate root-coverage gap (hit independently by three sessions,
and the reason the #1405 merge record rests on a target scan rather than the
repo gate) and the undeclared plugin-streams-core imports, the latter filed as
explicitly unverified with publish:dry-run evidence as its first acceptance box.

Records the waiting state for #1536: #1524 is still open, so no automatic
verdict exists and the merge gate cannot proceed. The staged body transform is
dry-run only and deliberately unapplied.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TKxrWGp5uxEHQ2NyZiZSMF

* docs(harness): record #1524 landing and the verdict watch for #1536

#1524 merged 7837ef4; the automatic phase dispatcher is live and D-4 routing
is in force. Records #1536 verified unchanged at that moment and the exact
baseline the verdict watch measures against.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TKxrWGp5uxEHQ2NyZiZSMF

* docs(harness): record the label-driven eval trigger contract as D-5

Formal PLAN/IMPL eval triggers on labels, never a manual OpenHands dispatch.
This lane already complies: no manual dispatch, and no local evaluator was
launched for #1536.

Records a measured timing finding: #1536's ready_for_review (08:53:43Z) and its
status:impl-eval application (08:53:45Z) both precede the dispatcher merging
(09:24:15Z), so the initial automatic IMPL-EVAL could not have fired and will
not fire on its own -- the label re-entry is required, not merely planned.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TKxrWGp5uxEHQ2NyZiZSMF

* docs(harness): dispatcher cannot fire on #1536's head -- workflow absent

Label re-entry executed exactly as instructed and confirmed clean, but the
phase-eval workflow produced no run: openhands-phase-eval.yml is absent from
head e4319c6, whose last main sync (08:45:46Z) predates #1524 merging
(09:24:15Z). pull_request events resolve workflows from the PR merge ref, so no
label cycling can trigger it.

main has also moved to #1547, which fixes the dispatcher's own token step.
Syncing the branch changes the head and forces a full CI re-run, so it is
raised rather than taken unilaterally.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TKxrWGp5uxEHQ2NyZiZSMF

* docs(harness): sync #1398 head, dispatcher fires, Qwen evaluator running

Owner-approved head change to f7d503f brings the phase-eval workflow and
#1547's token fix into the merge ref. Same labels and sequence that produced no
run on the old head now produce a successful dispatch, which is the control for
the diagnosis.

Replaces the comment-count watcher: OpenHands updates its summary comment in
place, so a counting watcher would have polled to timeout while the verdict sat
in an edited comment.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TKxrWGp5uxEHQ2NyZiZSMF

* docs(harness): re-establish #1398 gate evidence on the merging head

Both OTEL gates re-read by name from f7d503f's job logs (94073971396,
94073971501). The differing job ids are direct proof that reusing the pre-sync
verification would have cited a head no longer on the PR; the body transform now
requires head/job ids as arguments and asserts no stale reference survives.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TKxrWGp5uxEHQ2NyZiZSMF

* docs(harness): record the #1398 merge -- lane complete

Both owned issues landed: #1405 at 8ff1bcb, #1398 at d7e2b67. Records the
Qwen IMPL-EVAL PASS, the two problems the merge gate caught (a duplicate status
label from automation, and a stale close-gate result whose job predated the
ready-merge label), and the head-change discipline that kept pre-sync gate
evidence out of the merge record.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TKxrWGp5uxEHQ2NyZiZSMF

* docs(harness): complete cut-trace and write the lane retrospective

Fills the failure-mode table with eight time-costing failures and their
mitigations, and records a factual retrospective: what the lane produced, where
layered review caught what the layer above missed, four mistakes this
orchestrator made, and four candidate rules explicitly not promoted from a
single occurrence.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TKxrWGp5uxEHQ2NyZiZSMF

* docs(harness): close the 0.0.6 runtime lane run

Adds the mandatory context-pack and captures the #1398 Qwen IMPL-EVAL verdict
as a run artifact rather than leaving it only as a PR comment.

Untracks the two raw evaluator JSONL streams: they were 2.4MB of a 2.5MB run
dir against a 96K largest-artifact precedent in the 0.0.5 run, which tracks no
raw streams at all. They move to gitignored .llm/tmp scratch and remain on disk;
their substance is already verbatim in plan-eval.md and evaluate-1405.md with
run id, duration, event count and is_error.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TKxrWGp5uxEHQ2NyZiZSMF

* docs(harness): record the control-PR pre-merge gate

Seven checks at head bdc62b0. Check 6 is the load-bearing one for an
evidence-lane PR: explicit grep confirms no packages/** or plugins/** source
rode along, all 20 changed paths being under .llm/runs.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TKxrWGp5uxEHQ2NyZiZSMF

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
@github-actions github-actions Bot added the canary:0.0.6-canary.2 Published NetScript prerelease 0.0.6-canary.2 label Aug 12, 2026
@rickylabs
rickylabs deleted the fix/1398-publish-job-executions-to-durable-stream branch August 13, 2026 13:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:plugins plugins/* and plugin-core packages area:telemetry canary:0.0.6-canary.2 Published NetScript prerelease 0.0.6-canary.2 status:shipped type:fix

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(triggers/streams): job executions are never published to the durable job stream, so no execution trace can span producer → stream → consumer

1 participant