diff --git a/.agents/skills/agent-milestone-orchestrator/SKILL.md b/.agents/skills/agent-milestone-orchestrator/SKILL.md new file mode 100644 index 0000000000..234b6ffad5 --- /dev/null +++ b/.agents/skills/agent-milestone-orchestrator/SKILL.md @@ -0,0 +1,175 @@ +--- +name: agent-milestone-orchestrator +description: The milestone-orchestrator role for NetScript releases — reading a milestone into PR-sized clusters of linked issues, sequencing them into dispatch waves, delegating one supervisor per PR, holding merge authority, and deciding when a release canary goes out. Use when orchestrating a release milestone end-to-end, planning waves, delegating per-PR supervisors, or deciding canary points. The run shape lives in .llm/harness/workflow/milestone-run.md; the canary schedule in workflow/canary-cadence.md; publish mechanics in netscript-release. +--- + +# Agent Milestone Orchestrator + +The **role** of running a release milestone: one orchestrator, many delegated supervisors, waves +of PRs, canary points, a cut. This skill is the judgement half of epic #1120 — the decisions the +0.0.3 → 0.0.4 orchestration carried in one agent's head, written down so the next orchestrator +does not rediscover them. The **run** half — artifacts, stage contracts, gate lists, definition of +done — is [`workflow/milestone-run.md`](../../../.llm/harness/workflow/milestone-run.md); the +canary **schedule** is +[`workflow/canary-cadence.md`](../../../.llm/harness/workflow/canary-cadence.md). Nothing in this +skill restates them, publish mechanics (`netscript-release`), or lane routing (`lane-policy.md`). + +## When to Use + +- Driving a release milestone from open issues to a landed cut. +- Clustering milestone issues into PRs and sequencing dispatch waves. +- Deciding when a canary goes out, and what merge order serves it. +- Supervising delegated per-PR supervisors and holding merge authority. + +## When Not to Use + +- A single scoped change — that is an ordinary `run-loop.md` run. +- Planning a board of epics/issues — that is `workflow/seed-run.md`. +- Publishing, verifying, or rolling back a release — `netscript-release`. +- Routing models/providers to lanes — `workflow/lane-policy.md`, never a table here. + +## Evidence discipline + +Rules below are **[observed]** — recorded during the 0.0.4 execution: in the instrumented trace +(`.llm/runs/release-0.0.4--orchestration/cut-trace.md`), in the issues that run filed (#1113, +#1115, …), in the epic's ratified learnings (#1120), or in the ratified design doc's dated +observations — with the source cited at the claim — or **[asserted]** — plausible and unproven. The distinction is load-bearing: 0.0.4 shipped two guards whose predicate could never +fire, and they looked correct while doing nothing. Treat asserted rules as candidates to attack; +when your run falsifies or confirms one, record it in the run's `cut-trace.md` so the marker can +be upgraded with a citation, not by fiat. + +## Reading a milestone into PR clusters + +**One supervisor per PR, each PR closing a group of linked issues [observed — cut-trace merge +table; #1120 learnings].** This avoids +micro-PRs, and avoids a supervisor blocked because closing its issue depends on another lane's +work. Cluster by shared surface and shared acceptance, then check the cluster both ways: + +- **Too big:** a cluster whose issues span the release's most critical code is split even when the + issues share a surface — 0.0.4 deliberately deferred #1013 behind #1075 to avoid a five-issue PR + on the most critical code in the release **[observed]**. +- **Mislabelled:** read each issue's *acceptance*, not its labels — #1020 was labelled `type:docs` + while its acceptance required framework code, which later put framework source on a docs-lane + PR **[observed]**. +- **Unimplementable as scoped:** an issue whose acceptance boxes cannot all be truthfully ticked + by one PR gets split or moved *before* dispatch, not discovered at merge time — #1024/#1061 had + to be split out mid-flight when this was found late **[observed]**. + +## Wave sequencing and dispatch + +**Waves are small [observed — #1120 learnings].** A workflow blocks until every agent in it +completes, so a large fan-out stops the next block from starting — and it is what froze the host +at load 160 in 0.0.4. +Sequence so that each wave's PRs are independent of each other; dependencies run *across* waves, +not inside one. + +The wave is a **dispatch** unit, nothing more. It is not a content contract — see +[`canary-cadence.md`](../../../.llm/harness/workflow/canary-cadence.md) for why membership of a +canary is computed from merge history, never from the dispatch plan. + +Before dispatching a wave, the preconditions in `milestone-run.md` stage B (provider quota, +paid-transport verification) must be green — the orchestrator's judgement call is *sequencing +around* a constrained provider, not dispatching into it. Quota exhaustion is a first-class +failure mode, not bad luck: the 0.0.4 docs lane hit a hard cap mid-delivery and correctly refused +to substitute another model silently **[observed — #1120 learnings; cut-trace failure table]**. +Recovery judgement: redeem the soonest-expiring reset, and treat the provider status panel as +stale afterwards — verify with a real call, not the display **[observed — design record]**. + +## Re-planning is normal + +Three re-planning events happened inside the one observed milestone — a queue-jump folded into an +open PR, a deliberate deferral, a mid-flight split — and none broke the wave structure +**[observed]** (the full stories live in `canary-cadence.md` § Flexibility). The operating rule: +**the plan is a dispatch schedule, merge history is the record.** Absorb priority shifts by +re-clustering the *undispatched* remainder; never rewrite the record of what already landed, and +never hold a blocking fix hostage to wave order — the queue-jump shipped ~1h after being filed +because it was allowed to jump. + +Scope drift is a checkpoint, not a failure: 0.0.4 filed six defects from inside the run, two of +them blockers for the release's own purpose **[observed]**. When the definition of done moves, +move it explicitly (a recorded decision at the wave boundary), not silently. + +## Delegation and effort tiering + +Select every lane from [`lane-policy.md`](../../../.llm/harness/workflow/lane-policy.md) — this +skill holds no routing. The orchestrator's delegation judgement, beyond routing: + +- **Brief the gate as a deliverable.** Supervisors go idle at a red gate rather than escalating — + four occurrences in 0.0.4, plus three slices hard-stopped on an environmental block the brief + could have pre-empted **[observed — design record; cut-trace failure table]**. The brief names + the gates the supervisor must turn green and the known environmental hazards, up front. +- **Launch attached, never one-shot — and through the agentic suite.** An app-server-attached + thread takes further turns via its `threadId`; an ad-hoc `codex exec` is one-shot and + unreachable — an hour was lost to this **[observed — #1120 learnings]**. The launcher, watcher, + and steering surfaces are mapped in `.llm/harness/workflow/tooling.md`, and the handoff + protocol in `workflow/agent-handoff.md` — never ad-hoc shell. +- **Intercept between turns, not on git activity.** `codex-watch --mode turn` fires on + `task_complete`, when the agent is idle between turns — that is the clean point to read, steer, + or stop; git activity is not that signal **[observed — #1120 learnings]**. +- **Evaluate only when necessary [observed — #1120 learnings].** Draft→ready already triggers the + augment review; a label auto-triggers OpenHands (repo CI behaviour). Spawning a local evaluator + per PR is waste. The evaluator invariants and the scoped reviewer-substitution waiver are + defined in `milestone-run.md` § Evaluator protocol. + +## Merge authority + +The orchestrator holds merge authority: a delegated supervisor lands a PR, but only the +orchestrator merges it, and only through the **pre-merge gate defined in `milestone-run.md`** — +run per PR, recorded per PR. Two judgement rules sit with the authority: + +- **Never steer or merge from a truncated log [observed — design record].** A `head -14` excerpt + said two issues were satisfied; the raw log showed zero implementation. Pull the full artifact + before acting. +- **Merge order is a decision the canary consumes.** The orchestrator chooses what lands before + each declared canary point; `canary-cadence.md` turns whatever actually landed into the payload. + Choose merge order to make canary points meaningful; do not expect the cadence to repair an + incoherent order. + +## When a canary goes out + +The orchestrator declares canary points **at wave boundaries** as part of the wave plan — the +boundary, membership, identity, and note rules are all owned by +[`canary-cadence.md`](../../../.llm/harness/workflow/canary-cadence.md). Two cadence questions are +**owner-undecided** (every-boundary vs surface-gated; whether a failed canary blocks the next +dispatch or only the cut — both **[asserted]** there): do not resolve them by habit inside a run; +raise them, or follow the run's recorded decision. + +## Honesty rules + +- **A criterion that cannot be truthfully ticked moves with its issue** to the next milestone — + it is never ticked to clear a gate. Precedents and enforcement live in `milestone-run.md` + § Gate integrity. +- **Observational criteria cannot be closed by a PR.** "A follow-up run shows…" routes to a + verification issue in the next milestone (#1090 pattern) at the moment you notice it, not at + cut time. +- **Record what the run falsifies.** The most valuable output of 0.0.4 was a rule the trace + disproved before it shipped. An orchestrator who patches over a falsified assumption instead of + recording it has destroyed the evidence the next milestone needed. + +## Supervision pitfalls + +- **Liveness is not progress, and artifacts are not always where you launched [observed — run + record, #1115].** A research agent believed idle for 70 minutes was 25/27 complete, writing + into per-sub-agent worktrees, and came within one command of being killed. Verify a growing + artifact, a new commit, or a live session — not an open socket — and look across the repo root + before judging. +- **Never establish ownership by string match [observed — design record].** An "is a turn live + here" check matched worktree paths quoted inside *other agents' brief text*. Match the actual + `--cwd` argument; inverted, this mistake deletes a live agent's worktree. +- **Verify the artefact, never the exit code [observed — design record].** Three agents claimed + to have stopped their AppHost; all three process trees were still running, while the stop + command exited 0. + +## Reference files + +| File | Load when | +| --- | --- | +| `.llm/harness/workflow/milestone-run.md` | every milestone run — the run shape, gates, DoD | +| `.llm/harness/workflow/canary-cadence.md` | declaring canary points; label/note/drift contract | +| `.llm/harness/workflow/lane-policy.md` | lane and evaluator routing | +| `.llm/harness/workflow/tooling.md` | the agentic launch/watch/steer tool surface for stage C | +| `.llm/harness/workflow/agent-handoff.md` | handing work to OpenHands or local agents mid-run | +| `.agents/skills/netscript-release` | any publish, verification, or rollback step | +| `.agents/skills/netscript-pr` | branch/PR/label/milestone mechanics, close-gate | +| `.agents/skills/netscript-harness` | general harness operating model | +| `.llm/runs/release-0.0.4--orchestration/cut-trace.md` | the observed evidence base | diff --git a/.claude/skills/agent-milestone-orchestrator/SKILL.md b/.claude/skills/agent-milestone-orchestrator/SKILL.md new file mode 100644 index 0000000000..234b6ffad5 --- /dev/null +++ b/.claude/skills/agent-milestone-orchestrator/SKILL.md @@ -0,0 +1,175 @@ +--- +name: agent-milestone-orchestrator +description: The milestone-orchestrator role for NetScript releases — reading a milestone into PR-sized clusters of linked issues, sequencing them into dispatch waves, delegating one supervisor per PR, holding merge authority, and deciding when a release canary goes out. Use when orchestrating a release milestone end-to-end, planning waves, delegating per-PR supervisors, or deciding canary points. The run shape lives in .llm/harness/workflow/milestone-run.md; the canary schedule in workflow/canary-cadence.md; publish mechanics in netscript-release. +--- + +# Agent Milestone Orchestrator + +The **role** of running a release milestone: one orchestrator, many delegated supervisors, waves +of PRs, canary points, a cut. This skill is the judgement half of epic #1120 — the decisions the +0.0.3 → 0.0.4 orchestration carried in one agent's head, written down so the next orchestrator +does not rediscover them. The **run** half — artifacts, stage contracts, gate lists, definition of +done — is [`workflow/milestone-run.md`](../../../.llm/harness/workflow/milestone-run.md); the +canary **schedule** is +[`workflow/canary-cadence.md`](../../../.llm/harness/workflow/canary-cadence.md). Nothing in this +skill restates them, publish mechanics (`netscript-release`), or lane routing (`lane-policy.md`). + +## When to Use + +- Driving a release milestone from open issues to a landed cut. +- Clustering milestone issues into PRs and sequencing dispatch waves. +- Deciding when a canary goes out, and what merge order serves it. +- Supervising delegated per-PR supervisors and holding merge authority. + +## When Not to Use + +- A single scoped change — that is an ordinary `run-loop.md` run. +- Planning a board of epics/issues — that is `workflow/seed-run.md`. +- Publishing, verifying, or rolling back a release — `netscript-release`. +- Routing models/providers to lanes — `workflow/lane-policy.md`, never a table here. + +## Evidence discipline + +Rules below are **[observed]** — recorded during the 0.0.4 execution: in the instrumented trace +(`.llm/runs/release-0.0.4--orchestration/cut-trace.md`), in the issues that run filed (#1113, +#1115, …), in the epic's ratified learnings (#1120), or in the ratified design doc's dated +observations — with the source cited at the claim — or **[asserted]** — plausible and unproven. The distinction is load-bearing: 0.0.4 shipped two guards whose predicate could never +fire, and they looked correct while doing nothing. Treat asserted rules as candidates to attack; +when your run falsifies or confirms one, record it in the run's `cut-trace.md` so the marker can +be upgraded with a citation, not by fiat. + +## Reading a milestone into PR clusters + +**One supervisor per PR, each PR closing a group of linked issues [observed — cut-trace merge +table; #1120 learnings].** This avoids +micro-PRs, and avoids a supervisor blocked because closing its issue depends on another lane's +work. Cluster by shared surface and shared acceptance, then check the cluster both ways: + +- **Too big:** a cluster whose issues span the release's most critical code is split even when the + issues share a surface — 0.0.4 deliberately deferred #1013 behind #1075 to avoid a five-issue PR + on the most critical code in the release **[observed]**. +- **Mislabelled:** read each issue's *acceptance*, not its labels — #1020 was labelled `type:docs` + while its acceptance required framework code, which later put framework source on a docs-lane + PR **[observed]**. +- **Unimplementable as scoped:** an issue whose acceptance boxes cannot all be truthfully ticked + by one PR gets split or moved *before* dispatch, not discovered at merge time — #1024/#1061 had + to be split out mid-flight when this was found late **[observed]**. + +## Wave sequencing and dispatch + +**Waves are small [observed — #1120 learnings].** A workflow blocks until every agent in it +completes, so a large fan-out stops the next block from starting — and it is what froze the host +at load 160 in 0.0.4. +Sequence so that each wave's PRs are independent of each other; dependencies run *across* waves, +not inside one. + +The wave is a **dispatch** unit, nothing more. It is not a content contract — see +[`canary-cadence.md`](../../../.llm/harness/workflow/canary-cadence.md) for why membership of a +canary is computed from merge history, never from the dispatch plan. + +Before dispatching a wave, the preconditions in `milestone-run.md` stage B (provider quota, +paid-transport verification) must be green — the orchestrator's judgement call is *sequencing +around* a constrained provider, not dispatching into it. Quota exhaustion is a first-class +failure mode, not bad luck: the 0.0.4 docs lane hit a hard cap mid-delivery and correctly refused +to substitute another model silently **[observed — #1120 learnings; cut-trace failure table]**. +Recovery judgement: redeem the soonest-expiring reset, and treat the provider status panel as +stale afterwards — verify with a real call, not the display **[observed — design record]**. + +## Re-planning is normal + +Three re-planning events happened inside the one observed milestone — a queue-jump folded into an +open PR, a deliberate deferral, a mid-flight split — and none broke the wave structure +**[observed]** (the full stories live in `canary-cadence.md` § Flexibility). The operating rule: +**the plan is a dispatch schedule, merge history is the record.** Absorb priority shifts by +re-clustering the *undispatched* remainder; never rewrite the record of what already landed, and +never hold a blocking fix hostage to wave order — the queue-jump shipped ~1h after being filed +because it was allowed to jump. + +Scope drift is a checkpoint, not a failure: 0.0.4 filed six defects from inside the run, two of +them blockers for the release's own purpose **[observed]**. When the definition of done moves, +move it explicitly (a recorded decision at the wave boundary), not silently. + +## Delegation and effort tiering + +Select every lane from [`lane-policy.md`](../../../.llm/harness/workflow/lane-policy.md) — this +skill holds no routing. The orchestrator's delegation judgement, beyond routing: + +- **Brief the gate as a deliverable.** Supervisors go idle at a red gate rather than escalating — + four occurrences in 0.0.4, plus three slices hard-stopped on an environmental block the brief + could have pre-empted **[observed — design record; cut-trace failure table]**. The brief names + the gates the supervisor must turn green and the known environmental hazards, up front. +- **Launch attached, never one-shot — and through the agentic suite.** An app-server-attached + thread takes further turns via its `threadId`; an ad-hoc `codex exec` is one-shot and + unreachable — an hour was lost to this **[observed — #1120 learnings]**. The launcher, watcher, + and steering surfaces are mapped in `.llm/harness/workflow/tooling.md`, and the handoff + protocol in `workflow/agent-handoff.md` — never ad-hoc shell. +- **Intercept between turns, not on git activity.** `codex-watch --mode turn` fires on + `task_complete`, when the agent is idle between turns — that is the clean point to read, steer, + or stop; git activity is not that signal **[observed — #1120 learnings]**. +- **Evaluate only when necessary [observed — #1120 learnings].** Draft→ready already triggers the + augment review; a label auto-triggers OpenHands (repo CI behaviour). Spawning a local evaluator + per PR is waste. The evaluator invariants and the scoped reviewer-substitution waiver are + defined in `milestone-run.md` § Evaluator protocol. + +## Merge authority + +The orchestrator holds merge authority: a delegated supervisor lands a PR, but only the +orchestrator merges it, and only through the **pre-merge gate defined in `milestone-run.md`** — +run per PR, recorded per PR. Two judgement rules sit with the authority: + +- **Never steer or merge from a truncated log [observed — design record].** A `head -14` excerpt + said two issues were satisfied; the raw log showed zero implementation. Pull the full artifact + before acting. +- **Merge order is a decision the canary consumes.** The orchestrator chooses what lands before + each declared canary point; `canary-cadence.md` turns whatever actually landed into the payload. + Choose merge order to make canary points meaningful; do not expect the cadence to repair an + incoherent order. + +## When a canary goes out + +The orchestrator declares canary points **at wave boundaries** as part of the wave plan — the +boundary, membership, identity, and note rules are all owned by +[`canary-cadence.md`](../../../.llm/harness/workflow/canary-cadence.md). Two cadence questions are +**owner-undecided** (every-boundary vs surface-gated; whether a failed canary blocks the next +dispatch or only the cut — both **[asserted]** there): do not resolve them by habit inside a run; +raise them, or follow the run's recorded decision. + +## Honesty rules + +- **A criterion that cannot be truthfully ticked moves with its issue** to the next milestone — + it is never ticked to clear a gate. Precedents and enforcement live in `milestone-run.md` + § Gate integrity. +- **Observational criteria cannot be closed by a PR.** "A follow-up run shows…" routes to a + verification issue in the next milestone (#1090 pattern) at the moment you notice it, not at + cut time. +- **Record what the run falsifies.** The most valuable output of 0.0.4 was a rule the trace + disproved before it shipped. An orchestrator who patches over a falsified assumption instead of + recording it has destroyed the evidence the next milestone needed. + +## Supervision pitfalls + +- **Liveness is not progress, and artifacts are not always where you launched [observed — run + record, #1115].** A research agent believed idle for 70 minutes was 25/27 complete, writing + into per-sub-agent worktrees, and came within one command of being killed. Verify a growing + artifact, a new commit, or a live session — not an open socket — and look across the repo root + before judging. +- **Never establish ownership by string match [observed — design record].** An "is a turn live + here" check matched worktree paths quoted inside *other agents' brief text*. Match the actual + `--cwd` argument; inverted, this mistake deletes a live agent's worktree. +- **Verify the artefact, never the exit code [observed — design record].** Three agents claimed + to have stopped their AppHost; all three process trees were still running, while the stop + command exited 0. + +## Reference files + +| File | Load when | +| --- | --- | +| `.llm/harness/workflow/milestone-run.md` | every milestone run — the run shape, gates, DoD | +| `.llm/harness/workflow/canary-cadence.md` | declaring canary points; label/note/drift contract | +| `.llm/harness/workflow/lane-policy.md` | lane and evaluator routing | +| `.llm/harness/workflow/tooling.md` | the agentic launch/watch/steer tool surface for stage C | +| `.llm/harness/workflow/agent-handoff.md` | handing work to OpenHands or local agents mid-run | +| `.agents/skills/netscript-release` | any publish, verification, or rollback step | +| `.agents/skills/netscript-pr` | branch/PR/label/milestone mechanics, close-gate | +| `.agents/skills/netscript-harness` | general harness operating model | +| `.llm/runs/release-0.0.4--orchestration/cut-trace.md` | the observed evidence base | diff --git a/.claude/skills/aspire/SKILL.md b/.claude/skills/aspire/SKILL.md index 85fc68b181..7fbd291ecc 100644 --- a/.claude/skills/aspire/SKILL.md +++ b/.claude/skills/aspire/SKILL.md @@ -15,7 +15,7 @@ This repository uses Aspire to orchestrate its distributed application. Resource | Start isolated (worktrees) | `aspire start --isolated` | | Restart the app | `aspire start` (stops previous automatically) | | Wait for resource healthy | `aspire wait ` | -| Stop the app | `aspire stop` | +| Stop the exact AppHost | `aspire stop --apphost ` | | List resources | `aspire describe` or `aspire resources` | | Run resource command | `aspire resource ` | | Start/stop/restart resource | `aspire resource start|stop|restart` | @@ -38,6 +38,11 @@ This repository uses Aspire to orchestrate its distributed application. Resource Most commands support `--format Json` for machine-readable output. Use `--apphost ` to target a specific AppHost. +On a shared host, always take the exact `appHostPath` from `aspire ps` and pass it to the stop +command. Use `aspire resource stop` when only one resource needs cleanup. The host-wide +`aspire stop` mode with `--all` is dangerous to sibling runs and unreliable: it has reported +`No running AppHost found` and exited 0 while processes rooted at the AppHost survived. + ### Detached dashboard discovery In a generated NetScript workspace, prefer `deno task aspire:otel -- [resource]` and @@ -105,6 +110,7 @@ aspire mcp call --input '{"key":"value"}' # invoke a tool - **Always start the app first** (`aspire start`) before making changes to verify the starting state. - **To restart, just run `aspire start` again** — it automatically stops the previous instance. NEVER use `aspire stop` then `aspire run`. NEVER use `aspire run` at all. - **Only restart the AppHost when AppHost code changes.** For .NET project resources, use `aspire resource rebuild` instead. +- **Never kill an `aspire mcp start` process.** It is the session's MCP server, not a stray AppHost. - Use `--isolated` when working in a worktree. - **Avoid persistent containers** early in development to prevent state management issues. - **Never install the Aspire workload** — it is obsolete. diff --git a/.claude/skills/netscript-release/SKILL.md b/.claude/skills/netscript-release/SKILL.md index 77116a598c..7429f6f0d9 100644 --- a/.claude/skills/netscript-release/SKILL.md +++ b/.claude/skills/netscript-release/SKILL.md @@ -117,11 +117,21 @@ After the release content is ready and before creating the stable release: 4. The temporary `release/canary-` branch is deleted best-effort. The canary tag remains as provenance and collision evidence. -Canary versions are immutable. Never delete or reuse a failed canary version. If a canary was -published and then its production E2E failed, preserve the workflow and tag evidence, yank that -version for every package member that published it, fix forward, and dispatch the next `canary.N`. -Yanking removes it from normal resolution without erasing the audit record; yanked versions still -count when choosing the next N. +Canary versions are immutable. Never delete or reuse a failed canary version. + +**A failed canary is not an incident — it is the canary doing its job (owner decision, 2026-08-02).** +If a canary published and its pinned production E2E then failed, preserve the workflow and tag +evidence, fix forward, and dispatch the next `canary.N`. **Do not yank the failed canary.** A canary +is a prerelease: normal semver resolution never selects it, so a failed one is inert for consumers +and remains useful as audit evidence of what was tried. Failed canary versions still count when +choosing the next N. + +Yanking stays available for the case it was written for — a canary that is actively harmful to +resolve, for example one that published corrupt or wrong-identity artifacts. That is a judgement +call for the owner, not the default response to a red gate. + +The stable channel is unaffected either way: `release/canary-pair` is written only on a green pair, +and `release:publish` fails closed without it. ## Merge And Stable Publish Flow diff --git a/.llm/harness/workflow/canary-cadence.md b/.llm/harness/workflow/canary-cadence.md new file mode 100644 index 0000000000..e25236b030 --- /dev/null +++ b/.llm/harness/workflow/canary-cadence.md @@ -0,0 +1,174 @@ +# Rolling Canary Cadence + +The **schedule** contract of epic #1120: *when* a release canary goes out, *what* it contains, and +*where* anyone can see that in GitHub. It is one of three artifacts with a deliberate division of +labour — the `agent-milestone-orchestrator` skill is the **role** (who decides), the +[`milestone-run.md`](./milestone-run.md) profile is the **run** (what a milestone run must prove), +and this document is the **schedule**. A paragraph that would appear in two of them lives in one +and is referenced from the others. + +What this document does **not** own: + +- **Publish mechanics.** Everything about executing, verifying, recovering, or rolling back a + canary publish — and what a failed canary means — is owned by + `.agents/skills/netscript-release`. This document says when a canary point occurs; that skill + owns the rest, and none of it is repeated here. +- **The label/note mechanism.** Label derivation, payload computation, note rendering, and drift + checking are implemented once in `.llm/tools/release/canary-label.ts` + (`deno task release:canary-label`, shipped by #1121/#1122). This document states the contract + that tool enforces; it does not restate the implementation. + +**Disambiguation (#1119):** in this document "canary" always means a **release canary** — a JSR +prerelease of the workspace. The `agentic:provider-canary` / `agentic:rollout-canary` tasks are AI +provider/model rollout canaries and are unrelated; do not route release-cadence work through them, +and do not cite this document for them. The rename that resolves the collision is tracked as #1119 +and is not attempted here. + +## Evidence base + +Every rule below is marked **[observed]** — recorded during the 0.0.4 execution, whether in the +instrumented merge trace +([`.llm/runs/release-0.0.4--orchestration/cut-trace.md`](../../runs/release-0.0.4--orchestration/cut-trace.md); +11 PRs, 42 issues, 2026-08-03), in the issues that run filed, or in the ratified design doc's +dated observations — with the specific source cited at the claim — or **[asserted]** — a reasoned +proposal with no supporting observation yet. Asserted rules are the ones most likely to be wrong; attack them rather than +absorb them. Do not promote an assertion into an earned rule when citing this document. + +## Trigger: the wave boundary **[observed]** + +Merge times in the 0.0.4 trace self-cluster into three groups — 21 min, 42 min, 2h01m — and the +clusters *are* the orchestrator's dispatch waves. No new vocabulary is needed: **the canary +boundary is the wave boundary**, the point at which a dispatched group of supervisors has all +landed. The orchestrator declares that point as part of the wave plan (see the skill); this +document defines what the declaration means. + +## Membership: content-derived, not plan-derived **[observed]** + +This is the rule the trace *falsified* before it could be shipped, and the single most important +constraint on the cadence: + +PR #1086 was not part of wave 2. It was dispatched mid-wave because #1089 blocked the owner's +docs-audit lane, and it merged **between two wave-2 PRs**. A cadence defining a canary as "the PRs +dispatched in wave N" would have mislabelled it. + +Membership is therefore **what actually landed between the previous canary point and this one**, +computed from first-parent merge history. A PR that landed out of plan order is still in the +payload. The wave is a *dispatch* unit; the canary is a *content* unit; they usually coincide and +must never be assumed to. `release:canary-label` computes the payload exactly this way — from +merge history, never from a dispatch plan. + +## Identity: the label is the published version (D3) + +The GitHub label **is** the published prerelease version string: +`canary:-canary.` — e.g. `canary:0.0.5-canary.1`. The label is applied to every PR in +the payload and to the issues those PRs closed, so anyone can see which canary an issue or PR +shipped in without leaving GitHub. + +Identity rules: + +- **The label is derived from what was actually published, never typed by hand.** The canonical + execution is the wiring inside `.github/workflows/release-canary.yml`, which passes the publish + step's **own output version** to `deno task release:canary-label` — derivation, not + transcription. The standalone CLI exists for re-runs and audits; it refuses a version JSR does + not report as published, which bounds a manual invocation but does not make it derived — a + hand-run naming the *wrong already-published* version remains possible, which is exactly why + the workflow integration is the canonical route and a bare CLI run is the exception. + +- **The source of truth is the publish result, not the repo version field** (design finding, + found while scoping #1121 — not a trace observation). At the time this cadence was designed, + `deno.json` read `0.0.3` while the canary train was `0.0.4-canary.N` — `release:canary` takes a + *stable target* and derives the prerelease from it. A generator reading the repo's `version` + field would have emitted `canary:0.0.3-canary.1`, silently wrong in precisely the way D3 exists + to prevent. + +## The canary note + +A canary version string alone answers *that* something shipped, never *what*. Each canary point +therefore publishes a GitHub release note, and **the payload is the note**: the same merge-history +derivation that selects the labelled PRs renders the note's content. One derivation, two views — +this is why the note lives in the same tool invocation as the label, and why a second rendering +path must never be added (two paths to the same fact is how a label and a note drift apart). + +Contract (enforced by the tool): + +- The note is a **GitHub release at tag `v`, marked prerelease**. A canary never takes + `Latest` — that belongs to stable. The tag itself is created by the canary cut, not by this + tool, and the tool does not verify the tag exists before publishing the release — if the tag + were absent, GitHub would create it at the default-branch HEAD, the wrong commit. Publishing a + note therefore assumes the canary cut already created `v`; the published-version + refusal bounds *which* versions can be named, not whether their tag exists. +- Re-running is **idempotent**: an existing canary release is updated, not duplicated or failed. +- An empty payload (nothing merged since the previous canary point) produces an **explicit + empty-payload note**, not a silent success and not a crash. +- Publishing a note for a never-published version **fails**, under the same identity constraint as + the label. + +**Consequence for the cut [asserted]:** the canary notes are the cut's source material — but no +mechanism feeds them into the stable release note today. The stable note is composed by +`release:publish` from a hand-written intro plus independently generated lists (owned by +`netscript-release`); it does not read canary releases. What the canary notes do provide is +continuous visibility: each payload is recorded as it lands, so the #1083 class — a closed issue +whose deliverable *is* a release note, invisible at cut time — can be read off the canary notes +instead of reconstructed at the cut, which is exactly where it was historically dropped. The +cut-time checklist in [`milestone-run.md`](./milestone-run.md) therefore reads the canary notes +when composing the stable intro; today that is a manual read-through, and whether it gains +tooling is an open implementation question. + +## The drift gate + +A non-dry run of `release:canary-label` ends with a drift verdict: a `canary:*` label with no +matching published version, or a published canary with no label, fails the run. A `--dry-run` +deliberately stops before any mutation — its label, note, and drift checks then report as not-run +rather than silently passing, so a dry run can never be mistaken for a real verdict. + +Gate integrity, per the #1120 proof-of-firing bar: + +- **Firing demonstrated, negative case shown.** #1121's acceptance recorded the drift gate failing + on a fabricated mismatch before it was trusted — the negative case is on the record, not + presumed. +- **Pass is distinguishable from did-not-run.** The tool reports an explicit per-check result for + every check it defines, including the ones it did not reach — an early exit leaves visible + not-run records instead of silence. Silence is a failure, not a pass. How that reporting is + implemented is the tool's own concern: consult `canary-label.ts`, not this document. +- **Lineage note:** the drift check originally reported historical canaries from earlier trains + as missing labels; that scoping defect was fixed under #1160 (closed). The doctrine it left + behind stands: when the drift gate reports red, treat it as a finding — never hand-patch labels + to green it, and never widen the gate's tolerance to silence it. + +## Flexibility: what the cadence must absorb **[observed]** + +Three re-planning events happened inside the single observed milestone, none of which broke the +wave structure: + +- **#1089 jumped the entire queue** — folded into an already-open PR (#1086) because it blocked an + external lane; shipped ~1h after being filed. +- **#1013 was deliberately deferred** behind #1075 despite sharing a surface, to avoid a + five-issue PR on the release's most critical code. +- **#1024/#1061 were split out of #1078 mid-flight** when close-gate revealed 15 unchecked boxes + across issues with no implementation. + +A cadence that treats the *plan* as the contract breaks on all three. This one treats **merge +history** as the contract and absorbs all three without special handling — which is the reason the +membership rule above is load-bearing and not a stylistic preference. + +## Open questions — owner-undecided **[asserted]** + +Neither of these is settled. An orchestrator must not treat either answer as a rule: + +1. **Does a canary publish at every wave boundary, or only at boundaries crossing a + public-surface change?** Asserted preference: every boundary — the trace shows only ~three per + milestone, so the cost is low and the observability is worth it. +2. **Does a failed canary block the next wave's dispatch, or only the cut?** Asserted preference: + only the cut — blocking dispatch serialises the whole milestone on a publish step. (What a + failed canary means, and what to do about one, is `netscript-release` doctrine.) + +## Reference + +| Surface | Owns | +| --- | --- | +| `.llm/tools/release/canary-label.ts` (`deno task release:canary-label`) | label derivation, payload computation, note rendering, drift gate | +| `.agents/skills/netscript-release` | all publish mechanics, the green canary pair, failed-canary doctrine, rollback | +| [`milestone-run.md`](./milestone-run.md) | the run shape that consumes this cadence; its cut checklist reads the canary notes manually | +| `.agents/skills/agent-milestone-orchestrator` | the role that declares wave boundaries and decides when a canary goes out | +| [`cut-trace.md`](../../runs/release-0.0.4--orchestration/cut-trace.md) | the observed evidence base | +| #1149 · #1160 · #1119 | live-canary exercise (0.0.5) · drift-scoping defect (fixed) · naming collision | diff --git a/.llm/harness/workflow/milestone-run.md b/.llm/harness/workflow/milestone-run.md new file mode 100644 index 0000000000..1199c4116a --- /dev/null +++ b/.llm/harness/workflow/milestone-run.md @@ -0,0 +1,178 @@ +# Milestone Run — Orchestrating a Release Milestone + +Operating profile for **milestone runs** — supervisor runs that drive an entire release milestone +from open issues to a landed cut: PR-sized clusters delegated to per-PR supervisors, dispatched in +waves, with canary points at wave boundaries and a stable cut at the end. This document is the +**run**: what a milestone run produces and proves. The judgement — how to cluster, sequence, +delegate, and decide — is the `agent-milestone-orchestrator` skill (the **role**); the canary +schedule is [`canary-cadence.md`](./canary-cadence.md). A paragraph lives in exactly one of the +three and is referenced from the others. + +> **Provenance.** Derived from `release-0.0.4--orchestration`, the first real execution of the +> milestone-orchestrator pattern (epic #1120, decision D2), whose instrumented merge record is +> [`cut-trace.md`](../../runs/release-0.0.4--orchestration/cut-trace.md) — 11 PRs, 42 issues, +> ~6h40m on 2026-08-03. Rules below are **[observed]** — recorded during that execution, in the +> trace itself, in the issues the run filed, or in the ratified design doc's dated observations, +> with the source cited at the claim — or **[asserted]** (proposed, unproven). Like `seed-run.md` +> and `supervisor.md`, this freezes the **stage contracts**, not the exemplar's exact folder tree. + +## When to use a milestone run + +Use it when the deliverable is a **landed milestone**: many issues closing through multiple PRs +authored by delegated supervisors, with canaries and (usually) a stable cut. Do not use it for a +single scoped change (`run-loop.md`) or for planning a board (`seed-run.md`). If the deliverable +is *code for one change*, run-loop; if it is *issues*, seed; if it is *a milestone landed*, this. + +General harness mechanics — run activation, artifact discipline, the commit trail, evaluator +separation — are owned by `.agents/skills/netscript-harness` and `run-loop.md` and are not +restated here. This profile adds only what is specific to a milestone run. + +## Run layout + +- **Run dir** `.llm/runs/release---orchestration/` with the standard mandatory artifacts + (`supervisor.md` first, then `plan.md` as the wave plan, `worklog.md`, `context-pack.md`, + `drift.md`; `phase-registry.md` when phase groups apply), plus this profile's signature + artifact: +- **`cut-trace.md`** — the instrumented merge record, captured **during the run from + `git log origin/main`**, never reconstructed from recollection **[observed]**. It records every + merge in order (time, commit, PR, closed issues), the wave clustering, re-planning events, and + the failure modes that cost real time. It is the evidence base future cadence rules are earned + from — the 0.0.4 trace is the reason this profile exists — and it is what canary payload audits + are checked against. +- The milestone's *content* lands through the delegated PRs, not the orchestrator's branch; the + orchestrator's own commits are run artifacts and any doctrine promoted at close. + +## Stage contracts + +Lane assignments are configuration — bindings live only in [`lane-policy.md`](./lane-policy.md). + +| Stage | Contract (produce → proof) | +| --- | --- | +| **A — Bootstrap** | `supervisor.md` + run dir; milestone read (every open issue, its acceptance boxes, its labels) → opening record in `worklog.md` | +| **B — Wave plan** | `plan.md`: PR clusters, wave sequence, declared canary points (per [`canary-cadence.md`](./canary-cadence.md)), and **dispatch preconditions checked: provider quota and paid-transport verification** — both became gates by costing real time in 0.0.4 (a hard stop on an exhausted quota; $7.43 billed to the wrong transport — `cut-trace.md` failure table) **[observed]**. These are procedural gates: the proof is the **recorded check output** (what was queried, when, result) in `worklog.md` before dispatch — a wave dispatched without that record is a did-not-run of the gate, visible as the record's absence | +| **C — Wave dispatch** | one supervisor per PR-cluster, launched attached through the agentic suite ([`tooling.md`](./tooling.md) maps the launcher/watcher/steering surfaces; [`agent-handoff.md`](./agent-handoff.md) owns the handoff protocol; the skill owns the judgement) → per-PR identity recorded (thread id, worktree, PR number) | +| **D — Wave landing** | every PR in the wave merged through the **pre-merge gate** below → per-PR gate record in `worklog.md`, merge appended to `cut-trace.md` | +| **E — Canary point** | at the wave boundary: canary publish per `netscript-release`, then label + note + drift verdict per [`canary-cadence.md`](./canary-cadence.md) → the tool's explicit check records quoted in `worklog.md` | +| **C–E repeat** | per wave, absorbing re-planning (`cut-trace.md` records what actually happened, the plan records what was intended — both are kept, neither is rewritten) | +| **F — Cut** | cut-time checklist below, then the stable cut per `netscript-release` (its completion gate is owned there, not here) → checklist evidence in `worklog.md` | +| **G — Close** | milestone bookkeeping, observational criteria routed to verification issues (#1090 pattern), lessons promoted per `netscript-harness` § Where Lessons Belong → closing `context-pack.md` | + +**PLAN-EVAL of the wave plan [asserted]:** 0.0.4 ran without one — the pattern lived in one +agent's head, which is the gap #1120 closes. Until a trace shows otherwise, apply the standard +harness rule (a committed plan gets a separate-session PLAN-EVAL) rather than treating the wave +plan as exempt; record the choice in `supervisor.md` either way. + +## The pre-merge gate **[observed]** + +The empirical checklist the 0.0.4 orchestrator converged on, run per PR at stage D. Each row +cites what earned it a place — for most, a real firing; where the record shows something +narrower, the row says exactly what is and is not yet demonstrated. Items 6 and 7 were added +*after* being burned: + +| # | Check | Firing evidence (the negative case) | When it does not run | +| --- | --- | --- | --- | +| 1 | `close-gate` result is green | red close-gates idled four supervisors in 0.0.4 | a PR with no close-gate result is **unproven, not clean** — the result must exist | +| 2 | zero unticked `- [ ]` on every issue the PR closes | 15 unchecked boxes across #1078's issues forced the mid-flight split of #1024/#1061 | an unfetched issue body counts as unticked | +| 3 | no new `deno-lint-ignore` / `as unknown as` / `@ts-ignore` in the diff, **excluding `.llm/runs/**`** | two-part record: the exclusion fired in 0.0.4 (false positives from run artifacts and the scanner's own source), and the predicate itself is demonstrated on a synthetic diff — RED on a new ignore in publishable source, GREEN on excluded-path quotes (`release-0.0.4` follow-up demo: `.llm/runs/feat-milestone-orchestrator-artifacts--authoring/gate-demos.md` § Demo 1; #745 is the incident class) | a diff not scanned is a missing verdict, not a pass | +| 4 | named expensive gates report `SUCCESS`, not `SKIPPED`/`CANCELLED` | #778/#775 looked mergeable with every substantive check skipped, on a base dead since 17 July — **"clean" repeatedly meant "nothing ran"** | this check *is* the did-not-run detector; name the gates, don't count greens | +| 5 | the single decisive claim per issue, re-verified independently | a `head -14` truncated log nearly auto-closed two issues with zero implementation; the raw log caught it | an unverified claim stays a claim — record it as unverified | +| 6 | changed-file audit for `packages/**`/`plugins/**` on docs-lane PRs | #1079: a docs slice landed framework source (upstream cause: #1020 labelled `type:docs` with code acceptance) | an unaudited docs PR is unaudited — say so in the gate record | +| 7 | the PR body's own checklist matches what shipped | #1088 merged asserting "implementation hard stop in force" while shipping the change (filed as #1105); `close-gate` validates *issue* boxes, not *PR-body* checklists | not covered by close-gate — skipping this leaves the PR body unverified | + +## Gate integrity rules + +These govern every gate this profile names and every gate a milestone run adds: + +- **Proof of firing.** A gate enters this profile (or a run) only with its negative case + demonstrated. The 0.0.4 run shipped two guards whose predicate could never be true — a watcher + requiring non-draft when every PR was a draft, and an `origin/main..HEAD` ancestry check that is + wrong under squash-merge because merged commits are never ancestors. Both did nothing and looked + correct **[observed]**. That defect class (also #1022, #1012) is the signature failure of this + kind of work. Corollary: **"is it merged" uses PR state, never commit ancestry.** +- **Pass is distinguishable from did-not-run.** Each gate states what it reports when it does not + execute (see the table's last column). Absence of red is not green; silence is a failure. +- **The false-red trap (#1142).** `classify changes` corrupts `$GITHUB_OUTPUT` post-merge, so a + merged PR's `statusCheckRollup` contains superseded red runs. Merge-history audits must compare + check-run timestamps to the merge time and take **only the latest run per check name** — an + audit that sums all runs manufactures false reds, the mirror image of check 4's false greens. + The defect is observed (#1142), and both clauses of the rule are **demonstrated on live data**: + merged PR #1155's rollup carries a post-merge `classify changes` FAILURE (+28s after merge) and + a superseded pre-merge CANCELLED; applying the rule recovers the true pre-merge SUCCESS + (`.llm/runs/feat-milestone-orchestrator-artifacts--authoring/gate-demos.md` § Demo 2). +- **Expensive gates are serialised across slices.** Three concurrent `scaffold.runtime` runs + produced two failures that were contention, not defects **[observed]**. +- **The honesty rule.** A criterion that cannot be truthfully ticked **moves with its issue to + the next milestone**; it is never ticked to clear a gate. Precedents: PR #1092 carried + `Refs #1024` / `Closes #1061` when only #1061 was done; PR #1146 downgraded #1101 to `Refs` and + moved it. Observational criteria ("a follow-up run shows…") cannot be closed by any PR — route + them to a verification issue in the next milestone (#1090 pattern). + +## Cut-time checklist **[observed]** + +Run at stage F, before the stable cut. Each item is something 0.0.4 missed or nearly missed: + +1. **Breaking changes surfaced from closed issues reach the notes.** The #1083 class: a closed + issue whose deliverable *is* a release note is invisible at cut time unless carried forward. + The canary notes record each payload as it lands ([`canary-cadence.md`](./canary-cadence.md)); + at the cut, read them while composing the stable intro and verify nothing of this class was + dropped — today a manual read-through, since no mechanism feeds canary notes into the stable + note. +2. **Issues moved out of the milestone are reflected in the notes**, each with its written + reason — 0.0.4 moved six issues and two PRs, all reasoned on the issue. +3. **Scope drift is an explicit checkpoint, not a discovery.** 0.0.4 began at 31 items and filed + six defects from inside the run, two of them p0/p1 for the release's own purpose. The + definition of done moved during the run — correctly — and the cut records that decision. + +## Evaluator protocol for a milestone run + +Per-PR evaluation composes what already triggers rather than spawning evaluators **[observed — +#1120 learnings]**: draft→ready fires the augment review and the label surface fires OpenHands +(repo CI behaviour); a local evaluator per PR is waste. The invariants that hold regardless +(owned by `netscript-harness`/`lane-policy.md`, only their milestone-run application stated +here): + +- **Generator ≠ evaluator, and the supervisor is not the evaluator either** — #1113 was written + by one family, supervised by a second, evaluated by a third before merge **[observed — run + record, #1113]**. +- **Reviewer substitution is a legitimate waiver [observed — design record, 2026-08-03]**: when + the owner reviews a slice + directly, no-lane-self-certifies is satisfied — a substitution of reviewer, not an absence of + review — recorded in `drift.md`, never silently applied. Scope it by what the review last + caught: **keep opposite-family review for code; it may be dropped for run artifacts and + evidence prose.** The opposite-family pass earned its cost by finding non-failing inline `jq` + substitutions — a check that could not fail — in a tool whose purpose is proving gates fire. + Automated gates are unchanged by any waiver: they are evidence, not sign-off. + +## Definition of done + +A milestone run is done when all of the following hold, with evidence in the run dir: + +- Every milestone issue is **closed with verified acceptance** (pre-merge gate, check 1–2) or + **moved with a written reason** on the issue. +- Every declared canary point has its published version labelled and noted, with the drift gate's + explicit check records quoted — a red or `not run` record is a finding to record, not a nuisance + to hand-patch. +- The stable cut is complete per `netscript-release`'s hard completion gate (owned there). +- `cut-trace.md` records the actual merge order, re-planning events, and time-costing failures — + the next milestone's rules are earned from it. +- Observational criteria are routed to verification issues in the next milestone (#1090 pattern). +- The cut-time checklist and scope-drift checkpoint are recorded in `worklog.md`. + +## What varies per run (do not cargo-cult) + +Wave count and width, canary count (the trace shows ~three boundaries per milestone, not a quota), +whether the run ends in a stable cut or hands one off, lane bindings, and whether `phase-registry.md` +is needed. What does not vary: the stage contracts, the pre-merge gate, the gate integrity rules, +and the honesty rule. + +## Checklist + +- [ ] Deliverable is a landed milestone, not one change (run-loop) or a board (seed-run). +- [ ] `supervisor.md` written at stage A; milestone read complete. +- [ ] Wave plan committed with declared canary points; quota + transport verified before dispatch. +- [ ] PLAN-EVAL decision for the wave plan recorded in `supervisor.md`. +- [ ] Every merge passed the pre-merge gate with a per-PR record; `cut-trace.md` kept live. +- [ ] Every canary point labelled + noted with explicit check records quoted. +- [ ] No criterion ticked that was not truthfully done; movers carry written reasons. +- [ ] Cut-time checklist recorded before the stable cut. +- [ ] Observational criteria routed (#1090); lessons promoted; closing `context-pack.md` written. diff --git a/.llm/runs/feat-milestone-orchestrator-artifacts--authoring/codex-thread-ids.md b/.llm/runs/feat-milestone-orchestrator-artifacts--authoring/codex-thread-ids.md new file mode 100644 index 0000000000..748504f24d --- /dev/null +++ b/.llm/runs/feat-milestone-orchestrator-artifacts--authoring/codex-thread-ids.md @@ -0,0 +1,16 @@ +# sol-eval-1161 — Codex implementation thread +- **Thread / session id:** `019fc874-c9b4-7b43-8af7-8abc6d6dae8d` +- **Rollout:** `/home/codex/.codex/sessions/2026/08/03/rollout-2026-08-03T18-28-44-019fc874-c9b4-7b43-8af7-8abc6d6dae8d.jsonl` +- **Worktree:** `/home/codex/repos/ns-msorch-eval-1161` +- **Branch:** `review/pr-1161-sol` @ `aa11f0b33` (NO upstream by design). +- **Push rule:** explicit refspec only — `git push origin HEAD:refs/heads/review/pr-1161-sol`. +- **Requested route:** provider=openai · model=gpt-5.6-sol · effort=xhigh +- **Observed route:** provider=openai · model=gpt-5.6-sol · effort=xhigh +- **Route verdict:** matched +- **Runtime:** approval=never · sandbox=dangerFullAccess +- **Brief (staged):** `/home/codex/sol-eval-1161-brief.md` +## Steering (same thread — never a second send-message-v2 at this worktree) +```bash +codex exec resume 019fc874-c9b4-7b43-8af7-8abc6d6dae8d -- "" +``` +_Written by `.llm/tools/agentic/codex/launch-codex-slice.ts`._ \ No newline at end of file diff --git a/.llm/runs/feat-milestone-orchestrator-artifacts--authoring/context-pack.md b/.llm/runs/feat-milestone-orchestrator-artifacts--authoring/context-pack.md new file mode 100644 index 0000000000..52fe253ade --- /dev/null +++ b/.llm/runs/feat-milestone-orchestrator-artifacts--authoring/context-pack.md @@ -0,0 +1,25 @@ +# Context Pack — feat-milestone-orchestrator-artifacts--authoring + +**Task**: author the three #1120 artifacts — orchestrator skill (role), milestone-run profile +(run), canary cadence (schedule) — against the ratified design doc +`.llm/harness/design/milestone-orchestrator-and-canary-cadence.md`. One draft PR, `Closes #1120`, +milestone 0.0.5. Effort low: authoring, not design. + +**Hard constraints**: +- Skill = role, profile = run, cadence = schedule; a paragraph appears once, referenced elsewhere. +- `[observed]` (0.0.4 cut-trace) vs `[asserted]` markers carried into the artifacts; never promote + an assertion into an earned rule. +- No publish mechanics (netscript-release owns), no routing tables (lane-policy owns), no label + mechanism internals (`release:canary-label`, #1122, owns). +- Every gate specified: firing evidence (negative case) + explicit did-not-run distinguishability. +- Honesty rule: unticked criteria move with their issue (#1092/#1146 precedents); observational + criteria → verification issue (#1090 pattern; one to file in 0.0.6). +- #1119 collision: do not deepen; "canary" in these docs = release canary. +- `.claude/skills/` stays generated (`deno task agentic:sync-claude`), then `agentic:check-claude`. + +**State**: S0–S4 done — draft PR #1161 complete through acceptance evidence; verification issue +#1163 filed (0.0.6); `status:impl-eval`. **Do not repeat S4's external actions** (issue #1163 +exists; acceptance evidence posted). Eval cycle 1: `review_claude` Sol·xhigh adversarial pass +(thread `019fc874-c9b4-7b43-8af7-8abc6d6dae8d`, verdict `sol-eval-1.md`) returned +CHANGES_REQUESTED (9C/6M); fixes landed in the review-fixes slice; re-review steered on the same +thread. Owner directive: green verdict → merge. diff --git a/.llm/runs/feat-milestone-orchestrator-artifacts--authoring/drift.md b/.llm/runs/feat-milestone-orchestrator-artifacts--authoring/drift.md new file mode 100644 index 0000000000..fe10b4c478 --- /dev/null +++ b/.llm/runs/feat-milestone-orchestrator-artifacts--authoring/drift.md @@ -0,0 +1,52 @@ +# Drift Log — feat-milestone-orchestrator-artifacts--authoring + +Append-only. + +- 2026-08-03 · **lane override (recorded, not drift-severity)** — implementation lane is Claude + (this session) under the CLAUDE.md documentation-authoring exception: the diff touches only + `.agents/skills/`, the generated `.claude/skills/` mirror, `.llm/harness/`, and `.llm/runs/`. + No `packages/`/`plugins/` source. +- 2026-08-03 · **eval substitution (recorded per the reviewer-substitution waiver [observed])** — + formal PLAN-EVAL is substituted by the owner's prior ratification of the design doc (PR #1150, + decisions D1–D3 on #1120), and IMPL-EVAL by owner ratification of this draft PR (D1: the draft + PR *is* the ratification instrument). Scope of the waiver: prose/doctrine artifacts, per the + design doc's rule — "keep opposite-family review for code, drop it for run artifacts and + evidence prose." No code is authored in this run. +- 2026-08-03 · **ratification delegated to the adversarial pass (owner directive, this session)** — + the owner directed: launch a GPT-5.6 Sol high adversarial eval via the agentic toolchain; if + green, merge. Routed per lane policy as `review_claude` (Codex · OpenAI · `gpt-5.6-sol` · + **xhigh** — the canonical binding for reviewing Claude work; the owner's "high" resolved to the + enforced route rather than launching a mismatched identity). Launched app-server-attached in an + isolated worktree (`/home/codex/repos/ns-msorch-eval-1161`, branch `review/pr-1161-sol`, head + `aa11f0b33`); brief at `sol-eval-brief.md` in this run dir; thread id in + `codex-thread-ids.md`. Verdict bar: any C/M finding → CHANGES_REQUESTED (no merge); PASS → mark + ready, `status:ready-merge`, evidence mirror + close-gate, squash-merge. +- 2026-08-03 · **eval cycle 1: CHANGES_REQUESTED (9C/6M), all findings accepted** — verdict in + `sol-eval-1.md` + PR comment. Notable drift the eval surfaced: (a) the artifacts' `[observed]` + definition was narrower than their actual evidence base — widened to the recorded 0.0.4 + execution with per-claim citations rather than promoting or demoting claims silently; (b) the + canary-note→stable-note accumulation was written as mechanism but is design intent — downgraded + to [asserted] with the gap stated; (c) D2's "before any 0.0.5 delivery work" is already + factually strained (#1153/#1155 merged pre-ratification carrying the 0.0.5 milestone) — + surfaced to the owner in the acceptance evidence rather than papered over. Re-review steered on + the same Sol thread (eval loop 1 of 2 before escalation). +- 2026-08-03 · **eval cycle 2: CHANGES_REQUESTED — second failure, ESCALATED to owner** (verdict + `sol-eval-2.md`). Resolved by Sol: C1, C2, C4, C5, C6, C8, M3, M5, M6; no duplicate paragraphs; + mirror byte-identical. Of the 8 cycle-2 findings, 5 fixed in the follow-up slice: C3 residue + (reference-table row), C10 (tag-existence wording corrected against the tool), M7 (#1160 is + closed — lineage note now says so), and C7 + M4 converted from undemonstrated to **demonstrated** + (`gate-demos.md`: check-3 synthetic RED/GREEN; #1142 selection rule recovering PR #1155's true + pre-merge verdict from its live rollup containing a post-merge FAILURE). Escalated, needing an + owner ruling: **C9** — D2 is factually strained by pre-ratification 0.0.5-milestone merges + #1153/#1155; the D2 evidence box is now UNTICKED pending a ruling (orchestrated-delivery reading + recorded on #1120, or criterion moves); **M1/M2** — Sol holds `[observed]` must be cut-trace-only; + the artifacts follow the ratified design doc's practice (trace + filed issues + dated design + observations, cited per claim). Demoting filed-issue observations (#1113, #1115) to + "asserted/unproven" would be less accurate, not more honest — supervisor position, owner to + ratify or overrule. **No merge: the owner's green-gate condition is unmet.** +- 2026-08-03 · **owner ruled on both escalated forks ("approved", in-session)** — (1) D2 reads as + **orchestrated delivery**; #1153/#1155 are outside its scope — recorded as a comment and a box + annotation on #1120; D2 evidence box re-ticked with the ruling cited. (2) The wider + `[observed]` source-of-record definition is **ratified** (trace + filed issues + dated design + observations, cited per claim); no demotion. Cycle 3 steered on the same Sol thread to verify + `bf1f8456f` under the rulings; the standing merge-on-green directive resumes. diff --git a/.llm/runs/feat-milestone-orchestrator-artifacts--authoring/gate-demos.md b/.llm/runs/feat-milestone-orchestrator-artifacts--authoring/gate-demos.md new file mode 100644 index 0000000000..8e64561072 --- /dev/null +++ b/.llm/runs/feat-milestone-orchestrator-artifacts--authoring/gate-demos.md @@ -0,0 +1,49 @@ +# Gate demonstrations — proof-of-firing for the two undemonstrated rules + +Recorded 2026-08-03 during Sol eval cycle 2 (findings C7, M4): both gates the profile carried +without a demonstrated negative case are exercised here — one synthetically, one on live GitHub +data. Referenced from `workflow/milestone-run.md`. + +## Demo 1 — pre-merge check 3: new-ignore scan with `.llm/runs/**` exclusion (C7) + +Synthetic diff with two hunks: a new `// deno-lint-ignore no-explicit-any` in +`packages/example/src/thing.ts`, and a run-artifact line in `.llm/runs/some-run/worklog.md` that +*quotes* the string `deno-lint-ignore`. + +Scan of added lines for `deno-lint-ignore|as unknown as|@ts-ignore`: + +| Variant | Result | +| --- | --- | +| No exclusion | 2 hits — the real ignore **and** the run-artifact quote (the false positive the exclusion exists for) | +| With `.llm/runs/**` exclusion (the gate as specified) | 1 hit — `packages/example/src/thing.ts: +// deno-lint-ignore no-explicit-any` → **RED, exit 1** | +| Same gate on a diff containing only the `.llm/runs/**` hunk | 0 hits → **GREEN, exit 0** | + +**Conclusion:** the predicate fires on a genuinely new ignore in publishable source (negative +case shown), stays silent on excluded-path quotes (the observed 0.0.4 false-positive class), and +a clean diff passes — RED, GREEN, and excluded-GREEN are all distinguishable. + +## Demo 2 — #1142 false-red selection rule on live data (M4) + +PR **#1155** (merged 2026-08-03 **15:49:19Z**) — its `statusCheckRollup` for the check name +`classify changes` contains, verbatim: + +| startedAt | Conclusion | Relation to merge | +| --- | --- | --- | +| 15:33:44Z | SUCCESS | pre-merge | +| 15:34:08Z | SUCCESS | pre-merge | +| 15:34:17Z | SUCCESS | pre-merge | +| 15:34:49Z | SUCCESS | pre-merge | +| 15:42:19Z | CANCELLED | pre-merge, superseded 17s later | +| 15:42:36Z | SUCCESS | pre-merge — **latest pre-merge run** | +| 15:49:34Z | CANCELLED | **post-merge** (+15s) | +| 15:49:47Z | FAILURE | **post-merge** (+28s) — the #1142 corruption | + +A naive rollup read sees a FAILURE and two CANCELLEDs on a merged PR — false red. Applying the +rule exactly as the profile states it — *compare check-run timestamps to the merge time* (drops +the 15:49:34/15:49:47 post-merge runs), *then take only the latest run per check name* (drops the +superseded 15:42:19 CANCELLED in favour of 15:42:36) — yields **SUCCESS**, the true pre-merge +verdict. + +**Conclusion:** both clauses of the rule fired on live data — post-merge exclusion caught a real +#1142-class corrupted run, and latest-per-name selection caught a real superseded run — and the +false-red (naive) and true (selected) verdicts are distinguishable. diff --git a/.llm/runs/feat-milestone-orchestrator-artifacts--authoring/plan.md b/.llm/runs/feat-milestone-orchestrator-artifacts--authoring/plan.md new file mode 100644 index 0000000000..cab294bb1b --- /dev/null +++ b/.llm/runs/feat-milestone-orchestrator-artifacts--authoring/plan.md @@ -0,0 +1,83 @@ +# Plan — the three #1120 artifacts + +Source of record: `.llm/harness/design/milestone-orchestrator-and-canary-cadence.md` (merged, +owner-ratified). Empirical base: `.llm/runs/release-0.0.4--orchestration/cut-trace.md`. This plan +is the outline reported before body-writing, per the brief. + +## The line between the artifacts + +- **Skill** (`.agents/skills/agent-milestone-orchestrator/SKILL.md`) — the **role**: decisions. + How to read a milestone into PR clusters, sequence waves, delegate, hold merge authority, and + decide when a canary goes out. Never contains: run artifacts, gate lists, the label mechanism. +- **Profile** (`.llm/harness/workflow/milestone-run.md`) — the **run**: proof. What a milestone + run must produce and prove — run layout, stage contracts, the pre-merge gate list with per-gate + firing evidence, cut-time checklist, evaluator protocol, definition of done. Never contains: + role judgement, routing tables, the label mechanism. +- **Cadence** (`.llm/harness/workflow/canary-cadence.md`) — the **schedule**: identity. When a + canary happens (trigger), what it contains (membership), what it is called (D3 label identity), + the note, and the drift gate — wired to the shipped `release:canary-label` surface (#1121/#1122), + never reimplementing it. Never contains: publish mechanics (`netscript-release` owns those). + +Shared stories appear once: the #1086 mid-wave falsification lives in the cadence doc; skill and +profile reference it. The pre-merge gate list lives in the profile; the skill references it. Every +rule carries **[observed]** (earned from the 0.0.4 trace) or **[asserted]** (plausible, unproven). + +## Section headings + +### 1. Skill — `agent-milestone-orchestrator` (the role) + +1. When to Use / When Not to Use +2. Evidence discipline — observed vs asserted rules +3. Reading a milestone into PR clusters +4. Wave sequencing and dispatch +5. Re-planning is normal +6. Delegation and effort tiering (defers to `lane-policy.md`) +7. Merge authority +8. When a canary goes out (defers to `canary-cadence.md`) +9. Honesty rules +10. Supervision pitfalls +11. Reference files + +### 2. Profile — `workflow/milestone-run.md` (the run) + +1. When to use a milestone run +2. Run layout (artifacts, incl. the instrumented `cut-trace.md`) +3. Stage contracts (A bootstrap → B wave plan → C dispatch → D wave landing → E canary point → + loop C–E → F cut → G close) +4. The pre-merge gate [observed] — 7 items, each with firing evidence + did-not-run statement +5. Gate integrity rules (proof-of-firing; pass ≠ did-not-run; the #1142 false-red trap; + serialised expensive gates) +6. Cut-time checklist [observed] +7. Evaluator protocol for a milestone run +8. Definition of done +9. What varies per run +10. Checklist + +### 3. Cadence — `workflow/canary-cadence.md` (the schedule) + +1. What this document owns (and the #1119 disambiguation: release canary ≠ model-rollout canary) +2. Evidence base +3. Trigger: the wave boundary [observed] +4. Membership: content-derived, not plan-derived [observed] +5. Identity: the label is the published version (D3) +6. The canary note +7. The drift gate (negative case; did-not-run; known limitation #1160) +8. Flexibility: the re-planning events the cadence absorbs [observed] +9. Open questions — owner-undecided [asserted] +10. Reference + +## Slices + +- **S0** — run-dir bootstrap + this plan; draft PR opened with outline as the opening phase comment +- **S1** — `canary-cadence.md` (written first: skill and profile reference it) +- **S2** — `milestone-run.md` +- **S3** — the skill + `deno task agentic:sync-claude` mirror + `agentic:check-claude` + + fmt check on the new files +- **S4** — verification issue in 0.0.6 for the observational criterion (#1090 pattern), + acceptance-evidence comment, `status:impl-eval` + +## Validation + +`deno task agentic:sync-claude` (generate mirror), `deno task agentic:check-claude` +(validate-claude-surface), `deno fmt --check` on the authored files only. Docs-only diff → +`ci:skip-e2e` + `ci:skip-scaffold` on the draft PR, recorded in the opening comment. diff --git a/.llm/runs/feat-milestone-orchestrator-artifacts--authoring/sol-eval-1.md b/.llm/runs/feat-milestone-orchestrator-artifacts--authoring/sol-eval-1.md new file mode 100644 index 0000000000..5bcaf308b7 --- /dev/null +++ b/.llm/runs/feat-milestone-orchestrator-artifacts--authoring/sol-eval-1.md @@ -0,0 +1,77 @@ +**[PHASE: REVIEW] [VERDICT: CHANGES_REQUESTED]** + +VERDICT: CHANGES_REQUESTED + +## Findings + +- C1 `.llm/harness/workflow/canary-cadence.md:150` — The cadence restates publish doctrine that + `netscript-release` owns: a failed canary is evidence rather than an incident and is not yanked by + default; lines 12–15 additionally enumerate the green-pair, same-semver-republish, and rollback + mechanics — #1120 permits an ownership pointer, not copied mechanics, and + `.agents/skills/netscript-release/SKILL.md:89-134` is already the canonical statement. +- C2 `.llm/harness/workflow/canary-cadence.md:118` — The cadence restates + `canary-label.ts` internals: all five private check-record names, up-front `not run` allocation, + the CLI invocation, bidirectional drift algorithm, and release mutation behavior — those details + are implemented at `.llm/tools/release/canary-label.ts:214-249,360-377,416-441`; the run's own + `context-pack.md:12-14` explicitly forbids label-mechanism internals in these artifacts. +- C3 `.llm/harness/workflow/canary-cadence.md:101` — “Canary notes accumulate into the stable + release note” is a settled but nonexistent mechanism, repeated by `milestone-run.md:108-111` — + `.llm/tools/release/github-release.ts:529-537` composes the stable body from a manual intro, + GitHub-generated changes, and independently fetched closed issues; it never reads or accumulates + canary releases, and neither artifact defines the missing verification. +- C4 `.llm/harness/workflow/canary-cadence.md:109` — The claim that + `release:canary-label` “ends every run with a drift check” is false — the implemented `--dry-run` + path returns at `.llm/tools/release/canary-label.ts:509-512` before label application, note + publication, or drift, leaving those pre-allocated checks `NOT_RUN`. +- C5 `.llm/harness/workflow/canary-cadence.md:67` — The documented standalone command does not make + a hand-typed/wrong-current version “impossible by construction” — `parseArgs` requires the caller + to supply `--published-version` (`canary-label.ts:416-441`) and `assertPublishedCanary` only proves + that the supplied value exists somewhere in registry history (`:187-195`); the machine-derived + identity exists in `.github/workflows/release-canary.yml:121-131`, but the cadence replaces that + integration with a manual placeholder invocation. +- C6 `.llm/harness/workflow/milestone-run.md:51` — Provider quota and paid-transport verification + are explicitly promoted to dispatch gates without a pass record, a did-not-run signature, or a + mechanism that can distinguish either state — `cut-trace.md:61-68` supplies the historical + negative costs only, and the later per-merge table does not cover these two gates. +- C7 `.llm/harness/workflow/milestone-run.md:64` — The assertion that every pre-merge item has fired + is false for check 3 — its cited “negative case” at lines 74–75 and `cut-trace.md:76-78` proves only + that quoted strings in excluded paths cause false positives; it does not demonstrate the stated + predicate firing on a new eligible `deno-lint-ignore`, `as unknown as`, or `@ts-ignore`. +- C8 `.llm/runs/feat-milestone-orchestrator-artifacts--authoring/worklog.md:50` — The acceptance map + checks “a fresh orchestrator can run a milestone” while simultaneously deferring the run-from-it + proof to #1163 — #1163 says it “owns the proof that they work” and all four of its criteria remain + unchecked, so the PR is closing #1120 on an unproven criterion in direct conflict with + `milestone-run.md:98-102`'s honesty rule. +- C9 `.llm/runs/feat-milestone-orchestrator-artifacts--authoring/worklog.md:50` — The D2 acceptance + mapping claims that ratification precedes all 0.0.5 delivery “by construction,” but GitHub shows + unrelated 0.0.5-milestone PRs #1153 and #1155 merged at 15:30 and 15:49, before #1161 was created + at 16:13 and while it remains unratified; the owner amendment allowed only the thin canary + label/publish/observability slice ahead of these artifacts, not general CI delivery. +- M1 `.agents/skills/agent-milestone-orchestrator/SKILL.md:42` — Multiple rules are promoted as + `[observed]` although the declared evidence source does not contain them: supervisor-per-PR + identity (line 42), workflow blocking/host load 160 (58–60), attached-launch behavior (98–100), + evaluator triggers (101–104), truncated-log incident (112–113), worktree ownership matching + (146–148), and AppHost stop claims (149–150) are absent from `cut-trace.md:1-103`. +- M2 `.llm/harness/workflow/milestone-run.md:120` — The `[observed]` evaluator section is not backed + by the cut trace: draft→ready augment behavior, label-triggered OpenHands, #1113's three-family + path, owner-review substitution, and the inline-`jq` finding do not appear in + `cut-trace.md:1-103`; external examples may justify an assertion, but cannot satisfy this + artifact's definition of `[observed]` as trace-derived. +- M3 `.llm/harness/workflow/canary-cadence.md:77` — The repo-version/publish-result trap is marked + `[observed]`, but the 0.0.4 cut trace contains no repo version, canary target, publish result, or + label execution; the ratified design itself calls this “found while scoping,” so it has been + promoted beyond the stated empirical source. +- M4 `.llm/harness/workflow/milestone-run.md:92` — The “latest run per check name before merge” + audit algorithm is presented as a settled gate-integrity rule without proof-of-firing — #1142 + establishes a post-merge false-red defect and proposes fixing/short-circuiting the classifier, + but neither #1142 nor the cut trace establishes this new timestamp-selection algorithm. +- M5 `.agents/skills/agent-milestone-orchestrator/SKILL.md:98` — A fresh orchestrator still cannot + execute or supervise stage C from the three artifacts: they say only “launch attached” and retain + a `threadId`, but omit the repo's required agentic launcher, the `codex-watch --mode turn` + interception rule called out by #1120, the follow-up steering path, and any reference to + `workflow/tooling.md` or `workflow/agent-handoff.md`; this is a load-bearing operational gap, not + routing-table duplication. +- M6 `.llm/runs/feat-milestone-orchestrator-artifacts--authoring/context-pack.md:20` — The resumable + state is knowingly stale: it says S4 is next, while `worklog.md:45-52`, PR #1161's acceptance + comment, and head commit `aa11f0b33` all record S4 complete; a resumed supervisor could duplicate + the verification issue/comment external actions. diff --git a/.llm/runs/feat-milestone-orchestrator-artifacts--authoring/sol-eval-2.md b/.llm/runs/feat-milestone-orchestrator-artifacts--authoring/sol-eval-2.md new file mode 100644 index 0000000000..cbf35aca1b --- /dev/null +++ b/.llm/runs/feat-milestone-orchestrator-artifacts--authoring/sol-eval-2.md @@ -0,0 +1,45 @@ +VERDICT: CHANGES_REQUESTED + +## Findings + +- C3 (survives) `.llm/harness/workflow/canary-cadence.md:166` — The reference table still claims + `milestone-run.md` performs “cut-time verification of note accumulation,” preserving the + nonexistent mechanism C3 identified — the corrected contract at cadence lines 102–111 and + `milestone-run.md:113-118` says the notes are only read manually and no mechanism feeds them into + the stable note. +- C7 (survives) `.llm/harness/workflow/milestone-run.md:76` — Pre-merge check 3 remains an active + gate although its negative predicate has never been observed firing — this directly violates the + profile's own admission rule at lines 86–87 and #1120's acceptance requirement that every gate's + negative case be shown. The PR body's checked acceptance row does not become true by disclosing + the failure; it explicitly admits this predicate is not demonstrated. +- C9 (survives) `PR #1161 body, Acceptance evidence D2` — D2 is still checked despite evidence in + the same sentence that 0.0.5 delivery PRs #1153/#1155 merged before ratification — issue #1120 + says *no 0.0.5 delivery work* begins first, and its only owner amendment narrowly permits the + canary surface. A future merge cannot retroactively ratify an unrecorded “orchestrated delivery” + rewrite, so the conditional mapping is not truthful acceptance evidence. +- C10 `.llm/harness/workflow/canary-cadence.md:94` — The revised observable contract falsely says + the tool enforces publication on an **existing** `v` tag — + `canary-label.ts:360-377` only looks up an existing release and, on 404, POSTs a release with that + tag; it never verifies that the git tag already exists. This is new mechanism drift introduced by + the fix wording. +- M1 (survives) `.agents/skills/agent-milestone-orchestrator/SKILL.md:33-36` — The fix changes the + meaning of `[observed]` to include issue prose and design observations instead of backing every + marker with the cut trace as this review's source-of-record rule requires. The unsupported earned + rules remain, including host load 160 (lines 60–62), attached-launch behavior (101–108), evaluator + triggers (109–112), truncated-log behavior (120–122), and AppHost stop behavior (159–161), none of + which appears in `cut-trace.md`. +- M2 (survives) `.llm/harness/workflow/milestone-run.md:127-142` — The evaluator section remains + promoted as `[observed]` without cut-trace support for draft→ready/OpenHands triggers, #1113's + three-family path, reviewer substitution, or the inline-`jq` incident. Broadening the document's + evidence definition at lines 14–16 does not satisfy the required trace verification. +- M4 (survives) `.llm/harness/workflow/milestone-run.md:94-100` — The latest-run-per-check timestamp + algorithm remains a mandatory “must” rule while simultaneously marked `[asserted]` and + not-yet-exercised — asserted content is still phrased as settled doctrine, contrary to the same + document's proof-before-entry rule at lines 86–87. +- M7 `.llm/harness/workflow/canary-cadence.md:129-131` — The cadence says target-scoped drift is + “not yet” implemented and cites #1160 as a live defect, but #1160 is closed with all four + acceptance boxes checked. The reference is stale and no longer says what the artifact claims. + +Cycle-1 findings C1, C2, C4, C5, C6, C8, M3, M5, and M6 are resolved. Exact-paragraph comparison +found no duplicate paragraph across the three artifacts, and the generated orchestrator mirror is +byte-identical to its source. diff --git a/.llm/runs/feat-milestone-orchestrator-artifacts--authoring/sol-eval-3.md b/.llm/runs/feat-milestone-orchestrator-artifacts--authoring/sol-eval-3.md new file mode 100644 index 0000000000..7b05c300ea --- /dev/null +++ b/.llm/runs/feat-milestone-orchestrator-artifacts--authoring/sol-eval-3.md @@ -0,0 +1,23 @@ +VERDICT: CHANGES_REQUESTED + +## Findings + +- C10 (survives) `.llm/harness/workflow/canary-cadence.md:94-97` — The edit now admits that + `release:canary-label` does not verify prior tag existence, but immediately claims its + published-version guard “in practice means the tag exists before a note can.” That implication + is still false: `assertPublishedCanary` proves only that the JSR version exists, while + `publishCanaryRelease` (`.llm/tools/release/canary-label.ts:360-377`) neither reads nor verifies a + git tag before POSTing the GitHub release. Registry publication is not tag-existence evidence. +- M8 `PR #1161 body, Acceptance evidence — Every gate` — The re-ticked acceptance mapping was not + updated to the evidence it claims: it still says check 3 and the #1142 mitigation are + “not-yet-demonstrated” and still calls #1160 a “known limitation.” At head, check 3 and #1142 are + claimed demonstrated by `gate-demos.md`, and `canary-cadence.md:131-134` says #1160 is fixed. + The checked mapping is therefore internally contradictory and stale even though the underlying + demonstrations pass review. + +Cycle-2 findings C3, C7, C9, M1, M2, M4, and M7 are resolved. C7's predicate independently +returned RED/exit 1 for a new ignore in publishable source and GREEN/exit 0 for an excluded-only +hunk. M4's recorded #1155 rollup exactly matches GitHub: latest pre-merge `classify changes` was +SUCCESS at 15:42:36Z, followed after the 15:49:19Z merge by CANCELLED and FAILURE runs. C9 and +M1/M2 resolve under the owner rulings recorded on #1120. Exact-paragraph comparison remains clean, +and all three regenerated skill mirrors are byte-identical to their sources. diff --git a/.llm/runs/feat-milestone-orchestrator-artifacts--authoring/sol-eval-4.md b/.llm/runs/feat-milestone-orchestrator-artifacts--authoring/sol-eval-4.md new file mode 100644 index 0000000000..bb88478ed4 --- /dev/null +++ b/.llm/runs/feat-milestone-orchestrator-artifacts--authoring/sol-eval-4.md @@ -0,0 +1,14 @@ +VERDICT: PASS + +## Findings + +- None. + +C10 is resolved: `.llm/harness/workflow/canary-cadence.md:94-99` now states that the canary cut +creates the tag, that `release:canary-label` does not verify it, and that an absent tag would make +GitHub create it at default-branch HEAD. This matches `.llm/tools/release/canary.ts:188-190` and +`.llm/tools/release/canary-label.ts:360-377`; the published-version refusal is accurately limited +to version identity rather than tag existence. M8 is resolved: PR #1161's current “Every gate” +acceptance row cites both demonstrations in `gate-demos.md`, records their independent cycle-3 +verification, and identifies #1160 as closed/fixed lineage. The three-file `eb833401d` diff +introduces no new defect in the scoped surfaces. diff --git a/.llm/runs/feat-milestone-orchestrator-artifacts--authoring/sol-eval-brief.md b/.llm/runs/feat-milestone-orchestrator-artifacts--authoring/sol-eval-brief.md new file mode 100644 index 0000000000..90086819f0 --- /dev/null +++ b/.llm/runs/feat-milestone-orchestrator-artifacts--authoring/sol-eval-brief.md @@ -0,0 +1,88 @@ +use harness + +# Adversarial review (eval of record) — PR #1161, the three #1120 artifacts + +You are the opposite-family adversarial reviewer (lane `review_claude`: Codex · GPT-5.6 Sol · +xhigh) for Claude-authored doctrine. You are the merge gate: the owner has directed that a green +verdict from this pass merges the PR. Be adversarial — your value is in what you refute. + +**Read-only session.** Findings only. Do not fix, commit, push, edit labels, or merge. Do not run +expensive gates (no AppHost, no docker, no scaffold/E2E) — the machine is shared with live agents. +Static reads, `git log`/`git show`, targeted greps, and `gh` reads are your instruments. You MAY +post the one PR comment specified at the end, and nothing else. + +## SKILL + +Read `.agents/skills/netscript-harness/SKILL.md` §"Key Concepts" for vocabulary if needed. The +sources of record for this review (read in this order, in this worktree): + +1. `gh issue view 1120 --repo rickylabs/netscript` — the epic: ratified decisions D1–D3, the + non-duplication table, the acceptance criteria. This is the bar. +2. `.llm/harness/design/milestone-orchestrator-and-canary-cadence.md` — the merged, owner-ratified + design doc the PR claims to implement. +3. `.llm/runs/release-0.0.4--orchestration/cut-trace.md` — the empirical trace every `[observed]` + marker must be backed by. + +## Under review + +The diff of PR #1161 (this worktree is checked out at its head, `aa11f0b33`; base is +`origin/main`). The three artifacts: + +- `.agents/skills/agent-milestone-orchestrator/SKILL.md` (+ generated mirror under + `.claude/skills/` — mirror must equal source) +- `.llm/harness/workflow/milestone-run.md` +- `.llm/harness/workflow/canary-cadence.md` + +Also in the diff: run artifacts under `.llm/runs/feat-milestone-orchestrator-artifacts--authoring/` +(hold to honesty, not to doctrine quality), and regenerated mirrors for `aspire`/`netscript-release` +(verify they equal their `.agents/skills/` sources; the sync tool is +`.llm/tools/agentic/claude/sync-claude-skills.ts`). + +## Attack surface — check each, cite evidence for every finding + +1. **Duplication (the epic's stated main failure mode).** Any paragraph appearing in two of the + three artifacts; any restatement of `netscript-release` publish mechanics, `lane-policy.md` + routing, archetype content, or `canary-label.ts` tool internals. Ownership pointers are + allowed; restated content is a finding. +2. **Promoted assertions.** Verify every `[observed]` marker against the cut-trace (and the design + doc's own markers). A rule presented as earned that the trace does not support is a major + finding. Conversely, `[asserted]` content phrased as settled rule is a finding. +3. **Dead-predicate risk.** For every gate the artifacts specify: is its negative case actually + evidenced, and is its did-not-run state truthfully distinguishable from a pass? The 0.0.4 + precedent is two guards whose predicate could never be true. Attack the gates the same way. +4. **Mechanism drift.** Claims about `release:canary-label` behaviour must match + `.llm/tools/release/canary-label.ts` as implemented (label derivation, note rendering, + idempotency, empty payload, refusal on unpublished versions, the five pre-allocated check + records, drift check). Verify against the source, not the PR's claims. +5. **#1119 collision.** Anything that deepens the release-canary vs model-rollout-canary naming + collision. +6. **Reference integrity.** Every file path, task name, issue/PR number, and cross-reference in + the three artifacts must exist and say what the citing text claims. +7. **Fitness for purpose.** Could a fresh orchestrator actually run a milestone from the skill + + profile + cadence alone, without this run's chat history? Name any load-bearing gap. +8. **Acceptance mapping.** The PR's `## Acceptance evidence` comment maps #1120's boxes to + evidence — check each mapping is truthful, including that the observational criterion is + routed to #1163 rather than claimed. + +## Output contract + +1. Write `.llm/runs/feat-milestone-orchestrator-artifacts--authoring/sol-eval.md` **in this + worktree** (do not commit it), formatted: + + ``` + VERDICT: PASS | CHANGES_REQUESTED + + ## Findings + - [C|M|m] : + ``` + + Severity: C critical (false claim, dead gate, restated mechanics), M major (duplication, + promoted assertion, broken reference), m minor (prose/precision). **Bar: any C or M finding → + CHANGES_REQUESTED. Only m findings → PASS with notes. No findings → PASS.** + +2. Post exactly one PR comment: + `gh pr comment 1161 --repo rickylabs/netscript --body-file ` with the first + line `**[PHASE: REVIEW] [VERDICT: ]**` followed by the findings. This comment is the + verdict of record. + +Your final turn message should be the verdict line plus a one-paragraph summary. diff --git a/.llm/runs/feat-milestone-orchestrator-artifacts--authoring/supervisor.md b/.llm/runs/feat-milestone-orchestrator-artifacts--authoring/supervisor.md new file mode 100644 index 0000000000..bdee9040d6 --- /dev/null +++ b/.llm/runs/feat-milestone-orchestrator-artifacts--authoring/supervisor.md @@ -0,0 +1,38 @@ +# Supervisor Identity — feat-milestone-orchestrator-artifacts--authoring + +Written at run start per `workflow/lane-policy.md` § Supervisor identity. A run dir without this +file is not activated. Other supervisors cross-peek a run by reading this file — it is how a run's +operating identity is discoverable without chat memory. + +| Field | Value | +| --- | --- | +| Model | Claude Fable 5 (`claude-fable-5`) | +| Session | https://claude.ai/code/session_01ReZGc3KP8xvEuruz1io7Pq | +| Host | WSL2 Linux, user `codex` | +| Checkout | /home/codex/repos/ns-msorch | +| Worktree | /home/codex/repos/ns-msorch (dedicated worktree, no separate run worktree) | +| Branch | `feat/milestone-orchestrator-artifacts` | +| Baseline | `950a0591a` on `origin/main`, 2026-08-03 | +| Run ID | `feat-milestone-orchestrator-artifacts--authoring` | + +## Routes in force + +| Task lane | Provider / model / effort | Role in this run | +| --- | --- | --- | +| Supervisor + doc author | Claude Fable 5, effort low | authors the three #1120 artifacts (documentation-authoring exception, CLAUDE.md 2026-06-18) | +| Evaluator of record | Owner ratification on the draft PR | D1: the instrument **is** a draft PR for ratification; owner ≠ generator | +| `review_claude` adversarial pass | Codex · OpenAI · `gpt-5.6-sol` · xhigh (canonical binding) | opposite-family eval of the PR; owner directive 2026-08-03: a green verdict merges #1161 | + +Reference `.llm/harness/workflow/lane-policy.md`; do not copy its complete route table here. + +## Recorded lane/eval overrides + +- **Implementation lane = Claude (this session)** under the CLAUDE.md documentation-authoring + exception (recorded 2026-06-18): the diff touches no `packages/`/`plugins/` source — only + `.agents/skills/`, `.claude/skills/` (generated mirror), `.llm/harness/`, `.llm/runs/`. +- **Formal PLAN-EVAL substituted by owner ratification.** The plan of record is the merged, + owner-ratified design doc `.llm/harness/design/milestone-orchestrator-and-canary-cadence.md` + (PR #1150; decisions D1–D3 ratified on #1120, 2026-08-03). This run authors against that source + of record; per D1 the deliverable itself is a draft PR the owner ratifies. Reviewer-substitution + waiver applies (design doc §"Reviewer substitution is a legitimate waiver", [observed]): scope is + prose/doctrine, not code. Mirrored in `drift.md`. diff --git a/.llm/runs/feat-milestone-orchestrator-artifacts--authoring/worklog.md b/.llm/runs/feat-milestone-orchestrator-artifacts--authoring/worklog.md new file mode 100644 index 0000000000..108c750396 --- /dev/null +++ b/.llm/runs/feat-milestone-orchestrator-artifacts--authoring/worklog.md @@ -0,0 +1,141 @@ +# Worklog — feat-milestone-orchestrator-artifacts--authoring + +## S0 — bootstrap + outline + +- Read, in brief order: #1120 (ratified D1–D3 + amendment + acceptance), the merged design doc + (source of record), the 0.0.4 cut-trace, the shipped #1121/#1122 label surface + (`.llm/tools/release/canary-label.ts`, `release:canary-label` task), `netscript-release` and + `netscript-pr` skills, `seed-run.md` (run-shape precedent), lane-policy headings. +- Confirmed the cadence doc's path is already forward-referenced by the cut-trace as + `workflow/canary-cadence.md`; profile path `workflow/milestone-run.md` is fixed by the design + doc; neither file exists yet. +- Confirmed #1149 (canary surface exercise, 0.0.5) and #1160 (drift-check target-scoping defect, + 0.0.5) exist; #1119 (naming collision) exists — not fixed here, disambiguated only. +- Outline locked in `plan.md`. +- Draft PR #1161 opened (labels, milestone 0.0.5, `ci:skip-e2e`+`ci:skip-scaffold` for the + docs-only diff); outline posted as the `[PHASE: PLAN]` comment. + +## S1 — canary-cadence.md (`1774f6c95`) + +- Trigger [observed], membership [observed] (#1086 falsification canonically homed here), D3 + identity + note contract wired to `release:canary-label`, drift gate with firing evidence + (#1121) and did-not-run signature (five pre-allocated `not run` check records), #1160 as known + limitation, #1119 disambiguated, open questions kept [asserted]/owner-undecided. + +## S2 — milestone-run.md (`04efa4b0e`) + +- Stage contracts A–G; `cut-trace.md` as signature artifact; 7-item pre-merge gate as + check → firing evidence → did-not-run table; gate integrity rules (proof-of-firing, #1142 + latest-run-per-check-name, serialised expensive gates, honesty rule w/ #1092+#1146 precedents); + cut-time checklist; evaluator protocol incl. scoped reviewer-substitution waiver. PLAN-EVAL of + the wave plan marked [asserted] — 0.0.4 ran without one; not promoted. + +## S3 — agent-milestone-orchestrator skill + mirror + validation + +- Skill authored: role judgement only (clustering, waves, re-planning, delegation, merge + authority, canary decision, honesty, supervision pitfalls); no gate lists, no routing, no label + mechanism — all referenced. +- `deno task agentic:sync-claude` — SYNCED: 18 skills, 22 mirrored files. Ridealong: regenerated + `aspire` and `netscript-release` mirrors that were stale on main (generated surface must track + its source; +22/−6). +- `deno task agentic:check-claude` — all five checks OK. +- fmt: repo `deno.json` fmt scope is `packages/**`/`plugins/**` TS only — the authored Markdown is + outside fmt jurisdiction; house 100-col style applied by hand. + +## S4 — verification issue + acceptance evidence + status flip + +- Filed #1163 (milestone 0.0.6, `type:test`, #1090 pattern) — owns the observational criterion + "0.0.5 runs on this system", with four checkable criteria incl. upgrading/falsifying each + `[asserted]` rule the run exercises. +- PR #1161 body finalised (all slices + DoD ticked); `## Acceptance evidence` posted mapping every + #1120 acceptance box verbatim to evidence; `status:impl` → `status:impl-eval` in the same + action. +- Non-duplication verified: `grep -n "OIDC\|publish:readiness\|release:preflight"` over the three + artifacts hits only the cadence doc's ownership-disclaimer lines (12–13). +- Awaiting owner ratification (D1). Merge closes #1120; D2 then unblocks 0.0.5 delivery. + +## Eval cycle 1 — Sol·xhigh adversarial pass (`review_claude`), verdict CHANGES_REQUESTED + +Thread `019fc874-c9b4-7b43-8af7-8abc6d6dae8d`, isolated worktree at `aa11f0b33`; verdict of +record: PR comment + `sol-eval-1.md`. 9 critical / 6 major; all triaged **accepted** — none +declined. Fixes: + +- **C1** cadence ownership bullets and the failed-canary parenthetical reduced to pure pointers — + no `netscript-release` doctrine content restated. +- **C2** drift-gate section rewritten to the observable contract; check-name enumeration, + allocation strategy, and algorithm detail removed (consult `canary-label.ts`). +- **C3** "notes accumulate into the stable note" corrected in both cadence and cut checklist: no + mechanism exists; canary notes are cut source material read manually; marked [asserted]. +- **C4** "ends every run with a drift check" corrected: non-dry runs end with the drift verdict; + `--dry-run` stops pre-mutation with visible not-run records. +- **C5** "impossible by construction" overclaim replaced: canonical derivation is the + `release-canary.yml` wiring passing the publish step's own output; the standalone CLI bounds + but does not derive. +- **C6** stage-B quota/transport gates given a proof form: recorded check output in `worklog.md` + before dispatch; absent record = did-not-run. +- **C7** pre-merge check 3 firing status stated honestly: exclusion observed; predicate-on-new- + ignore not yet demonstrated. +- **C8** #1120 acceptance box 1 given a relocation note (run-from-it proof → #1163, #1090/#1121 + precedent); evidence mapping re-scoped accordingly. +- **C9** D2 mapping corrected: #1153/#1155 (milestone 0.0.5) merged 15:30/15:49Z pre-ratification; + surfaced for the owner's D2 interpretation instead of "by construction". +- **M1/M2/M3** `[observed]` definition widened to the recorded 0.0.4 execution (trace + filed + issues + ratified design record) and every flagged claim re-cited to its actual source; the + repo-version trap re-attributed as a scoping finding, not a trace observation. +- **M4** #1142 mitigation split: defect observed, latest-run-per-check selection rule marked + [asserted] until exercised. +- **M5** stage-C operability closed: tooling.md + agent-handoff.md wired into skill and profile; + `codex-watch --mode turn` interception rule added. +- **M6** context-pack state corrected with an explicit do-not-repeat-S4 guard. + +## Eval cycle 2 — verdict CHANGES_REQUESTED; second failure → escalation + +Verdict: `sol-eval-2.md`. Sol confirms C1/C2/C4/C5/C6/C8/M3/M5/M6 resolved, no duplicate +paragraphs, mirror byte-identical. Follow-up slice dispositions: + +- **C3 residue** — cadence reference-table row still said "verification of note accumulation"; + now "reads the canary notes manually". Fixed. +- **C10** — "release on the *existing* tag" implied a tag-existence check the tool does not + perform; wording corrected (tag created by the canary cut; tool creates-or-updates the release; + its guard is the published-version refusal). Fixed. +- **M7** — #1160 is CLOSED; the "known limitation" became a lineage note keeping the + never-hand-patch doctrine. Fixed. +- **C7** — resolved by demonstration rather than argument: `gate-demos.md` § Demo 1 shows the + check-3 predicate RED (exit 1) on a new ignore in publishable source, GREEN on excluded-path + quotes, GREEN on a clean diff. Row updated to cite it. +- **M4** — resolved by demonstration on live data: `gate-demos.md` § Demo 2 applies both clauses + of the #1142 rule to merged PR #1155's real rollup (post-merge FAILURE +28s, superseded + CANCELLED) and recovers the true pre-merge SUCCESS. Row updated; [asserted] marker removed. +- **C9** — escalated: owner ruling required on D2's reading (see drift.md); the D2 evidence box + in the PR body is now UNTICKED pending that ruling. +- **M1/M2** — escalated: definitional dispute over the `[observed]` source of record + (cut-trace-only vs the ratified design record's own practice); supervisor position recorded in + drift.md, owner to ratify or overrule. + +## Eval cycle 3 — CHANGES_REQUESTED on two mechanical residues; substance resolved + +Verdict: `sol-eval-3.md`. C3/C7/C9/M1/M2/M4/M7 resolved — C9 and M1/M2 under the owner rulings +recorded on #1120; Sol independently re-ran the check-3 demo (RED exit 1 / GREEN exit 0) and +re-fetched #1155's rollup, confirming both gate demonstrations against live data. Survivors, +both residues of the cycle-2 fixes, fixed in this slice: + +- **C10** — dropped the false implication that the published-version guard means the tag exists; + the note now states the tool does not check the tag and what GitHub would do if it were absent + (create it at default-branch HEAD — the wrong commit). +- **M8** — the PR body's "every gate" acceptance row now cites the `gate-demos.md` + demonstrations and #1160's fixed status instead of the pre-demo wording. + +## Eval cycle 4 — VERDICT: PASS (zero findings) + +`sol-eval-4.md`: C10 and M8 resolved and verified against `canary.ts`/`canary-label.ts`; the +`eb833401d` diff introduces no new defect. Merge proceeds under the owner's standing +merge-on-green directive: ready → `status:ready-merge` → evidence mirror + close-gate → +squash-merge → terminal labels → eval-worktree teardown. + +## Merge window note — mirror/label event race + +The `ready_for_review`-adjacent CI run at 17:45:36Z carried an event payload snapshotted before +`status:ready-merge` landed, so the label-gated evidence mirror skipped and close-gate correctly +failed on #1120's still-unchecked boxes (the gate distinguishing did-not-run from pass, working +as specified). Re-triggered via synchronize with the label present; mirror dry-run had already +validated all 8 mappings.