fix(agentic): phase-eval status cleanup removes labels from an event snapshot, so a concurrent dispatch fails the run with a 404

[rickylabs]

Summary

openhands-phase-eval.yml's status-cleanup step removes status:* labels read from the event payload
snapshot rather than from live state. When two dispatch runs fire seconds apart for the same head, the
first removes status:impl and the second's removeLabel returns 404 Label does not exist, failing the
whole step and turning the workflow run red — on a PR whose evaluation actually succeeded.

This is a trust defect, not a correctness defect: exactly-once dispatch held. What it produces is a
false red that costs investigation time, and it recurs on any PR that gets two near-simultaneous dispatch
events.

Observed instance — PR #1541, head 0503991ab

31596291515  completed/failure  event=pull_request  created=12:24:36Z   <- raced, red
31596293364  completed/success  event=pull_request  created=12:24:38Z   <- posted the authoritative trigger

Two seconds apart, same head. Failure log, step "Enter IMPL-EVAL status on ready transition":

DELETE /repos/rickylabs/netscript/issues/1541/labels/status%3Aimpl - 404
retry-exempt-status-codes: 400,401,403,404,422

404 is deliberately retry-exempt, which is correct — a retry cannot help — so the step fails hard.

Exactly-once was not violated. Verified: exactly one trigger marker exists for that head —
generation=29339092792 phase=impl head=0503991ab…, model openrouter/deepseek/deepseek-v4-flash-0731.
The generation-dedup mechanism did its job. Only the label cleanup raced.

Root cause

.github/workflows/openhands-phase-eval.yml, "Enter IMPL-EVAL status on ready transition":

const labels = context.payload.pull_request.labels.map((label) => label.name);
for (const label of labels.filter((name) => name.startsWith('status:'))) {
  await github.rest.issues.removeLabel({ owner, repo, issue_number, name: label });
}
await github.rest.issues.addLabels({ owner, repo, issue_number, labels: ['status:impl-eval'] });

context.payload.pull_request.labels is a snapshot taken when the event was created. The loop then
issues unconditional deletes against live state. Any concurrent path that removed a status: label between
event creation and this step makes the delete 404.

This is the same defect class as #1564 — acting on a payload snapshot instead of live state. There it is
pull_request.base.sha feeding a diff range; here it is pull_request.labels feeding a delete loop. Worth
noting because the two fixes rhyme: read live, or compute from something current.

Why it matters more than a stray red

A red run on a PR whose evaluation succeeded is the precise stimulus that teaches operators to discount red
runs. It has already cost real time in this milestone: this defect is what produced the apparent
"duplicate evaluator run" that had to be investigated and then corrected with the finding that extra raw
workflow entries were no-op/skip events and exactly-once was intact. The investigation was necessary
because a red run implied something had gone wrong with dispatch. Nothing had.

Fix

1. Make status cleanup idempotent. Either fetch live labels immediately before removing
   (issues.listLabelsOnIssue) and remove only what is actually present, or tolerate the missing-label
   case — narrowly.
2. Narrowly means narrowly. Tolerate only a 404 whose body is the missing-label error for the label
   being removed. A blanket try {} catch {} around removeLabel would swallow a 403 from a permissions
   regression or a 404 from a wrong issue_number, converting a real failure into a silent pass. Rethrow
   anything else, and keep the existing retry-exempt-status-codes behaviour unchanged.
3. Preserve generation deduplication exactly as it is. It already works — one trigger for one head under
   a genuine race — and it is the property that keeps this defect cosmetic. Do not restructure it.

Acceptance criteria

• A concurrent removal of status:impl between event creation and the cleanup step no longer fails the
  workflow: the step completes and status:impl-eval is applied exactly once.
• Proven RED: a race regression test that fails before the change — the label absent at removal time
  causes the step's logic to throw — and passes after.
• Tolerance is narrow, asserted by test: a non-missing-label error from removeLabel (for example
  403, or a 404 for a different resource) still fails the step. A blanket catch fails this box.
• Generation deduplication is unchanged: under two dispatch events for one head, exactly one trigger
  comment is posted. Asserted by test rather than by inspection.
• The terminal label state after cleanup is exactly one status: label, per the taxonomy's
  single-status rule.
• gate: the phase-eval script's tests pass, and .github/scripts/ type-check, lint and format via the
  scoped wrappers.

Boundaries

• Do not change the workflow's triggers (on: types) or its dispatch conditions. This is the
  cleanup step only.
• Do not change model resolution, the eval:model:* label mapping, or the trusted-base-ref logic that
  fix(agentic): resolve evaluator prompt from current trusted base #1552 landed.
• Do not retrigger PR docs(reference): reference IA completeness, README truth, and CLI verb coverage #1541. Its evaluation succeeded and its trigger is authoritative; re-running it
  would consume evaluator capacity to reproduce a cosmetic failure.
• Do not widen this into fix(ci): pull_request.base.sha is stale on long-lived PRs, so every gate that computes a range from it silently inspects the wrong changeset #1564. That issue owns the stale-base.sha range computations; this one owns
  the label-snapshot race. Same class, different surface, separate fixes.

Provenance

Observed on PR #1541 at head 0503991ab, 2026-08-12, runs 31596291515 (red) and 31596293364 (green).
Run facts, the 404 body, and the exactly-once verification were reproduced from the Actions API and the PR's
trigger markers by the 0.0.6 internals lane before filing. No pre-existing issue covers it (searched).

[github-actions]

Acceptance evidence mirrored from #1567.

• "A concurrent removal of status:impl between event creation and the cleanup step no longer fails the" — Inline live-label transition plus race regression: the exact missing-label 404 completes normally and applies status:impl-eval once.
• "Proven RED: a race regression test that fails before the change — the label absent at removal time" — Pre-fix RED commit 72cf4b7; targeted test exited 1 before the production module, then passed after implementation.
• "Tolerance is narrow, asserted by test: a non-missing-label error from removeLabel (for example" — phase-eval-status.test.ts asserts 403 and unrelated-resource 404 rejection; only exact message Label does not exist is tolerated.
• "Generation deduplication is unchanged: under two dispatch events for one head, exactly one trigger" — Generation marker/existing-claim/early-return ordering regression test passes; dispatch script is unchanged.
• "The terminal label state after cleanup is exactly one status: label, per the taxonomy's" — Pure decision test removes all live status-prefixed labels and adds only status:impl-eval; string parity test guards the inline terminal literal.
• "gate: the phase-eval script's tests pass, and .github/scripts/ type-check, lint and format via the" — S4 IMPL comment records 67/67 script tests and all six gate exit codes on terminal head c4814ff.

Provenance: updatedAt=2026-08-12T13:10:21Z; bodySha256=d4a11b2a83f4965c4b2471a39719107f82858f12aa2b40c75ef8b18f8b421018