Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
a4964f9
chore(harness): activate the 0.0.6 runtime lane orchestration run
rickylabs Aug 12, 2026
01aeafb
docs(harness): plan #1398 on a verified trace-context join
rickylabs Aug 12, 2026
7a32f85
docs(harness): record the #1405 slice review and its negative-case proof
rickylabs Aug 12, 2026
2f526ea
docs(harness): resolve #1398 S0 and correct a wrong scope inference
rickylabs Aug 12, 2026
b6e4bab
docs(harness): record #1398 PLAN-EVAL PASS and amend the plan
rickylabs Aug 12, 2026
52c7d0f
docs(harness): dispatch the #1398 slice under the amended plan
rickylabs Aug 12, 2026
fa19c91
docs(harness): record the #1405 IMPL-EVAL PASS
rickylabs Aug 12, 2026
0e6c48f
docs(harness): record the #1405 merge in cut-trace
rickylabs Aug 12, 2026
46501d3
docs(harness): record the owner ruling on IMPL-EVAL for small taxonom…
rickylabs Aug 12, 2026
4df670b
docs(harness): record two red #1398 live runs and escalate the verdic…
rickylabs Aug 12, 2026
70faa01
docs(harness): route future phase evaluation through the automatic di…
rickylabs Aug 12, 2026
542caec
docs(harness): record #1398 live gate PASS on both CI runtime tiers
rickylabs Aug 12, 2026
f56c3f5
docs(harness): file the lane's two deferred findings as #1542 and #1543
rickylabs Aug 12, 2026
103b4df
docs(harness): record #1524 landing and the verdict watch for #1536
rickylabs Aug 12, 2026
8d4790e
docs(harness): record the label-driven eval trigger contract as D-5
rickylabs Aug 12, 2026
1325276
docs(harness): dispatcher cannot fire on #1536's head -- workflow absent
rickylabs Aug 12, 2026
18a067b
docs(harness): sync #1398 head, dispatcher fires, Qwen evaluator running
rickylabs Aug 12, 2026
52606e5
docs(harness): re-establish #1398 gate evidence on the merging head
rickylabs Aug 12, 2026
34a603a
docs(harness): record the #1398 merge -- lane complete
rickylabs Aug 12, 2026
7310b1d
docs(harness): complete cut-trace and write the lane retrospective
rickylabs Aug 12, 2026
bdc62b0
docs(harness): close the 0.0.6 runtime lane run
rickylabs Aug 12, 2026
1bd7e24
docs(harness): record the control-PR pre-merge gate
rickylabs Aug 12, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
76 changes: 76 additions & 0 deletions .llm/runs/release-0.0.6-features--orchestration/context-pack.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
# Context pack — 0.0.6 runtime / public-surface lane

Closing summary. The lane is **complete**: both owned issues landed on `main`. Read this first if
resuming or auditing; everything below is traceable to a named artifact.

| Field | Value |
| --- | --- |
| Run id | `release-0.0.6-features--orchestration` |
| Profile | `.llm/harness/workflow/milestone-run.md` (topical lane) |
| Supervisor | Claude · Opus 5 · high |
| Control branch / PR | `chore/release-0.0.6-features-orchestration` / **#1525** (evidence only, closes nothing) |
| Baseline | `origin/main@01aa12b67` |
| Status | **Complete** — 2/2 issues merged, 2 follow-ups filed |

## Outcome

| Issue | PR | Merge | Gate record |
| --- | --- | --- | --- |
| #1405 durable producer rejection taxonomy | #1528 | `8ff1bcb8f` | `slices/pre-merge-gate-1528.md` |
| #1398 job executions → durable job stream | #1536 | `d7e2b67b2` | `slices/pre-merge-gate-1536.md` |

Both auto-closed `COMPLETED` via closing keywords; `status:shipped` on each issue and PR.

## What changed, in one paragraph each

**#1405** — two settled write **reason strings** misdescribed the state that produced them: a write
rejected during the graceful close-drain reported `producer-failed` (the producer was healthy and
closing), and a non-retryable append on attempt 1 reported `retry-exhausted` (nothing was exhausted).
Fixed by making the closing intent observable to the rejection selector and by branching `#failActive`
on `isRetryable`, adding exactly one public member, `transport-refused`. The façade's duplicate
selector was deleted so the two cannot drift apart again — that drift was the defect's origin. No
change to which writes are accepted, rejected, or delivered.

**#1398** — job executions were never published to the durable job stream because the workers **API
service** installed the execution-state mutation hook while the **background** entrypoints that
generated projects actually run never did. Fixed by installing the hook on the worker and combined
runtimes and publishing inside `context.with(extractContext({traceparent, tracestate}), …)` so the
publish span joins the `job.execute` trace — including the pre-span `create()` record, which is the
trap that would otherwise fail TC-14 silently. The two E2E gates deferred against this issue were
un-deferred and now pass live.

## Evidence chain

- **#1398 acceptance was made mechanical.** Its observational criterion was bound to two gates the
repo had already deferred against this issue. Both now pass **by name** on both CI runtime tiers at
the merging head: postgres `94073971396` (`passed=88 failed=0 skipped=0`), sqlite `94073971501`
(`passed=83 failed=0 skipped=0`).
- **PLAN-EVAL** #1398: PASS, MiniMax M3, separate session — `plan-eval.md`. Findings F1/F2 folded in.
- **IMPL-EVAL** #1405: PASS, DeepSeek V4 Flash 0731 — `slices/evaluate-1405.md`. Superseded as
*policy* by D-3 after the fact, not retracted.
- **IMPL-EVAL** #1398: PASS, Qwen 3.8 Max via the **automatic dispatcher** —
`slices/evaluate-1398.md`.
- Merge order and every time-costing failure: `cut-trace.md`. Lessons and mistakes:
`retrospective.md`.

## Open items leaving this lane

| Item | Where |
| --- | --- |
| `quality:gate` roots omit published packages — **both merge records here rest on explicit target scans, not the repo gate** | **#1542** (0.0.7) |
| Undeclared `@netscript/plugin-streams-core` imports — filed **unverified**, `publish:dry-run` evidence is its first acceptance box | **#1543** (0.0.7) |
| Canary and stable cut | **root's**, not this lane — nothing was published here |

## Drift index

D-1 research sub-agent lane override · D-2 evaluator transport fallback · **D-3** IMPL-EVAL waived
for the small deterministic class · **D-4** phase evaluation moved to the automatic dispatcher ·
**D-5** label-driven eval trigger contract. Full text in `drift.md`.

## Artifact hygiene note

The two raw evaluator JSONL streams (2.4 MB combined) were **untracked** and moved to
`.llm/tmp/` scratch, which `.gitignore` excludes. They were 2.4 MB of a 2.5 MB run dir, against a
96 K largest-artifact precedent in the 0.0.5 run. Their substance is preserved verbatim in
`plan-eval.md` and `slices/evaluate-1405.md`, each carrying run id, duration, event count, and
`is_error`. Nothing was deleted — the files remain on disk locally.
47 changes: 47 additions & 0 deletions .llm/runs/release-0.0.6-features--orchestration/cut-trace.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
# Cut trace — 0.0.6 runtime / public-surface lane

Merge-derived record. Every row is captured **from `git log origin/main` after the merge**, never
from recollection or from the dispatch plan. This lane does not cut or publish; root orchestration
owns the canary and the stable cut.

## Baseline

| Field | Value |
| --- | --- |
| Lane opened | 2026-08-12 |
| `origin/main` at open | `01aa12b67` — `docs(harness): record FILING-LOG -- board migration executed once (#1523)` |
| Owned issues | #1405, #1398 |

## Merges

| # | UTC | Merge commit on `origin/main` | PR | Issues closed | Pre-merge gate record |
| --- | --- | --- | --- | --- | --- |
| 1 | 2026-08-12T08:20:29Z | `8ff1bcb8fc741db17a765a1965861828e0ae6171` (`8ff1bcb8f`) — `fix(streams): distinguish producer refusal reasons (#1528)` | #1528 | **#1405** (auto-closed `COMPLETED` by the body's `Closes #1405`) | `slices/pre-merge-gate-1528.md` — all 7 checks PASS |

| 2 | 2026-08-12T10:27:26Z | `d7e2b67b2be535c9ca13449f97f8f4585344030a` (`d7e2b67b2`) — `fix(workers): publish job executions to the durable stream on the job.execute trace (#1536)` | #1536 | **#1398** (auto-closed `COMPLETED` by the body's `Closes #1398`) | `slices/pre-merge-gate-1536.md` — all 7 checks PASS at head `f7d503fee` |

Both rows captured from `git log origin/main --first-parent -1` **after** each merge, per the
profile's merge-derived rule. Issue states re-read live: both `CLOSED` / `COMPLETED`, with
`status:shipped` applied to each issue and PR.

**Lane complete: both owned issues landed on `main`.**

## Re-planning events

| # | Date | Event | Decision |
| --- | --- | --- | --- |
| 1 | 2026-08-12 | #1536's head changed mid-flight, `e4319c685` → `f7d503fee` | The phase-eval dispatcher (#1524) merged **after** the branch's last `main` sync, so `openhands-phase-eval.yml` was absent from the PR's merge ref and no label cycling could trigger it. Owner approved syncing the branch, accepting a full CI re-run. All gate evidence was then re-read against the new head; no pre-sync evidence was carried forward. |
| 2 | 2026-08-12 | Evaluation route changed twice mid-run | D-3 removed formal IMPL-EVAL for the small deterministic class (after #1405 had already merged); D-4/D-5 moved phase evaluation to the automatic label-driven dispatcher. #1398's IMPL-EVAL consequently ran on the automatic route rather than a manual launch. |

## Failure modes that cost real time

| # | Date | Failure | Cost | Mitigation recorded |
| --- | --- | --- | --- | --- |
| 1 | 2026-08-12 | **Dispatcher absent from the PR's merge ref.** #1524 merged 38 min after #1536's branch last synced with `main`, so `openhands-phase-eval.yml` did not exist for that PR. Labels were cycled correctly and produced **no run at all**; the only runs were the older `openhands-agent.yml`, all `skipped`. | One wasted label re-entry, plus a full CI re-run (~20 min, both `scaffold-runtime` tiers) after the branch sync. | For `pull_request` events GitHub resolves workflows from the **merge ref**. A newly merged workflow cannot fire on a PR whose head predates it. **Check the workflow exists in the PR head before concluding a trigger failed.** Diagnosed by `git cat-file -e <head>:<workflow>` → ABSENT, with the post-sync run succeeding as the control. |
| 2 | 2026-08-12 | **Draft PRs report every check as `skipping`.** #1528 looked "clean" while nothing substantive had run. | None — caught by pre-merge check 4 before merge. | This is the #778/#775 class, alive and current. Absence of red is not green. The blocking tier only runs after draft→ready. |
| 3 | 2026-08-12 | **Two live `scaffold.runtime` runs died before reaching the gates under test** — run 1 on a transient `generate plugins: fetch failed`, run 2 on a `triggers-api` health timeout at 120 s. | ~25 min of local runtime, no verdict produced. | Local WSL was not a usable arbiter for this suite. CI ran the same suite with the same change to a clean finish on both tiers, which is the control that established the local failures as environmental. **Do not label a local red a "flake" without that control.** |
| 4 | 2026-08-12 | **A counting watcher would never have fired.** OpenHands updates its summary comment **in place** (`openhands-agent-summary` marker, `"conclusion"` field), so a watcher keyed on comment count polls to timeout while the verdict sits in an edited comment. | None — caught before arming, on inspecting the posted comment's markers. | Watch the **run status** and the comment's `conclusion` marker, never comment count. |
| 5 | 2026-08-12 | **Stale check summary read as current.** `gh pr checks` reported `close-gate` red from a job that ran 40 min earlier, before the label and body changes it was complaining about. | None — caught by reading the job log, which carried the mirror's own "skipped because labels do not include status:ready-merge" notice. | Compare a check's run time to the change it is judging. The gate-integrity rule already says merge-history audits must take the latest run per check name; the same applies pre-merge. |
| 6 | 2026-08-12 | **Automation added a second `status:` label.** `status:augment-review` was applied 1 s after the orchestrator moved off `status:impl-eval`, breaching the exactly-one-status invariant. | None — caught while verifying labels before merge. | Re-verify the `status:` set after any automated phase transition, not only after manual edits. |
| 7 | 2026-08-12 | **Orchestrator's own slice brief named a broken gate command.** `deno test packages/plugin-streams-core` exits 1 with 19 `NotCapable` errors for want of `--allow-env`. | Minor; the implementer reported the red with its cause rather than hiding or working around it. | Use the package-declared `deno task --cwd <pkg> test`. Corrected in the #1398 brief rather than repeated. |
| 8 | 2026-08-12 | **One unnecessary evaluator dispatch.** The lane brief's IMPL-EVAL waiver for the #1405 class was read as a blocked-transport fallback rather than the class default. | One DeepSeek IMPL-EVAL run (~643 s) that the owner did not want. | Recorded as D-3. "A waiver is available" and "the waiver is the default" are different instructions; resolve the ambiguity before spending. |
124 changes: 124 additions & 0 deletions .llm/runs/release-0.0.6-features--orchestration/drift.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,124 @@
# Drift — 0.0.6 runtime / public-surface lane

Append-only. Severity: `minor` | `significant` | `architectural`.

## D-1 — research/plan sub-agent lane overridden to Opus (minor)

**Date** 2026-08-12. **Canonical route** `deep_analysis` = Claude · Fable 5 · medium. **Used**
Claude · Opus 5 medium/high. **Reason** the owner brief for this lane explicitly directs
"Delegate plan/research to Claude Opus medium/high sub-agents when useful". Owner instruction is
more specific than the default lane binding. Invariants preserved: generator ≠ evaluator, no lane
self-certifies, no paid escalation. Fable 5 · low remains the `review_codex` reviewer for the
#1398 slice, so opposite-family review of Codex work is untouched.

## D-2 — evaluator transport falls back to local fresh sessions (minor)

**Date** 2026-08-12. The brief routes PLAN-EVAL/IMPL-EVAL through OpenHands *after #1524
passes/lands*. Observed: PR #1524 is **OPEN**, `mergedAt: null`, with unticked DoD boxes for the
bounded live DeepSeek smoke and the repository default variable. Per the brief's own condition, this
run uses fresh local Claude/OpenCode OpenRouter evaluator sessions through the toolchain. Re-checked
before each dispatch; state recorded at the point of use.

## D-3 — IMPL-EVAL not required for small deterministic taxonomy fixes (significant)

**Date** 2026-08-12. **Owner ruling.** For the #1405 class — small, deterministic, fully specified
fixes with exact negative tests — a separate formal IMPL-EVAL is **not** to be dispatched. The
sufficient evidence set is: focused negative tests, CI, close-gate, and the orchestrator's own
independent diff review.

**My error.** The lane brief said an IMPL-EVAL owner waiver was "acceptable" for #1405 with exact
negative tests. I read that as a fallback to use only if the evaluator transport was blocked, and
recorded that reading in `supervisor.md`. The owner's intent was that the waiver is the **default**
for this class. One evaluator dispatch (DeepSeek V4 Flash 0731 max, 642,836 ms) was spent that should
not have been.

**Timing, stated plainly.** The ruling arrived after #1405 had already been evaluated **and merged**
(`8ff1bcb8f`, 2026-08-12T08:20:29Z). It therefore does not retract that merge or its evidence — the
IMPL-EVAL returned PASS and its per-fix revert isolation is real evidence that remains in the record.
The ruling governs this class going forward.

**Not weakened for #1398.** Formal PLAN-EVAL and IMPL-EVAL remain **mandatory** for #1398: it changes
public runtime behaviour, publishes a new record to a durable stream, and requires live evidence.
Its PLAN-EVAL has already run (PASS, MiniMax M3) and its IMPL-EVAL is still required before merge.

**Cleanup.** The detached #1405 evaluator worktree `/home/codex/repos/ns006-1405-impleval` was
verified clean (empty `git status --porcelain`) and removed. The #1398 PLAN-EVAL worktree
`/home/codex/repos/ns006-1398-planeval` is retained pending that issue's IMPL-EVAL. Pre-existing
global stashes belonging to other lanes' branches were left untouched.

## D-4 — phase evaluation moves to the automatic status dispatcher (significant)

**Date** 2026-08-12. **Owner ruling.** PR #1524 (automatic phase dispatcher) is about to merge. Once
it lands, **all future phase evaluations use the automatic status workflow** unless the owner selects
a documented local route or an explicit skip. Manual IMPL-EVAL launches and manual
`@openhands-agent` PR comments are not to be used for #1536.

**Standing instruction for #1536:** keep it on its **current head and status** — head `e4319c685`,
`status:impl-eval`, milestone `0.0.6`. Root will **deliberately re-enter** `status:impl-eval` with
the Qwen override *after* #1524 lands, which is what triggers the automatic dispatcher. This
orchestrator must not re-enter the label for the same head, must not trigger OpenHands, and must not
launch a local evaluator. Its remaining job is to **watch the automatic verdict and then finish the
merge gate**.

**Timing, recorded factually.** The steer anticipated that a local #1536 evaluator had already been
launched. It had **not**. For #1398 I wrote the evaluator prompt
(`slices/impl-eval-1398-prompt.md`, 10:57) and pre-created the detached worktree
`/home/codex/repos/ns006-1398-impleval` at `e4319c685`, then **stopped and raised the decision**
rather than dispatching. Verified at the time of this entry: no `openrouter-run`/`claude-openrouter`
process, and no `impl-eval-1398-raw.md` output file — only the prompt. **No duplicate spend occurred
and none was in flight.**

The two evaluator sessions this run did spend are unaffected and both completed before this ruling:
PLAN-EVAL #1398 (MiniMax M3, PASS) and IMPL-EVAL #1405 (DeepSeek V4 Flash 0731, PASS — itself the
subject of D-3).

**Unused artifacts retained, not removed:** `slices/impl-eval-1398-prompt.md` and the
`ns006-1398-impleval` worktree are left in place, clean and unused, in case the owner later selects
the documented local route. They are inert; nothing reads them.

**Interaction with D-3.** D-3 waives formal evaluation for the small deterministic class. D-4 does
not widen that waiver — #1398 still gets a formal IMPL-EVAL; it now arrives through the automatic
dispatcher rather than a manual launch.

## D-5 — automation trigger contract for formal evaluation (significant)

**Date** 2026-08-12. **Owner policy.** Formal PLAN/IMPL evaluation is triggered by labels, never by a
manual OpenHands dispatch:

| Phase | Initial trigger | Rerun |
| --- | --- | --- |
| PLAN-EVAL | the `openhands` + `status:plan-eval` label **pair**, exactly once | move away from `status:plan-eval`, then re-add |
| IMPL-EVAL | automatically on **draft → ready**, unless `impl-eval:skip` | move away from `status:impl-eval`, then re-add |

`eval:model:minimax|deepseek|qwen` is an optional **one-shot** override. A local eval already running
may finish, but must never be duplicated. Manual `@openhands-agent` dispatch for formal PLAN/IMPL
eval is prohibited. Merge continues to go through normal harness authority.

**This lane complies with no change required:** no manual OpenHands dispatch was made for #1536 (or
for anything else this run), and no local evaluator was launched for #1536 — the prompt and worktree
were prepared and the dispatch deliberately withheld (D-4). The two local evaluator sessions this run
did spend — PLAN-EVAL #1398 (MiniMax M3) and IMPL-EVAL #1405 (DeepSeek) — both **completed** well
before this policy and are not duplicated.

### Timing finding: #1536's automatic IMPL-EVAL could not have fired

Measured from the issue timeline rather than assumed:

| Event | UTC |
| --- | --- |
| #1536 `ready_for_review` | **2026-08-12T08:53:43Z** |
| #1536 `status:impl` removed, `status:impl-eval` applied | 2026-08-12T08:53:45Z |
| #1524 (the dispatcher) merged | **2026-08-12T09:24:15Z** |

Both candidate triggers precede the dispatcher's existence by ~30½ minutes. The draft → ready
transition therefore had nothing to fire, and the initial automatic IMPL-EVAL for #1536 **did not
run and will not run on its own**.

Under this policy the rerun path is the only one left: move #1536 away from `status:impl-eval`, then
re-add it. That matches what root already stated it would do with the Qwen override, so the
conclusion is unchanged — but it is now a **requirement** rather than a preference, and a watcher
waiting for a spontaneous verdict would wait forever. Recorded so that is visible rather than
discovered by timeout.

**Not actioned by this lane.** The label re-entry is root's, per D-4. This orchestrator has not
touched #1536's head, labels, or body.
Loading
Loading