Skip to content

fix(runner): do not fail a job on a transient cache-assignment status - #5

Open
msichterman wants to merge 1 commit into
biw:mainfrom
flamel-ai:fix/transient-cache-assignment-status
Open

fix(runner): do not fail a job on a transient cache-assignment status#5
msichterman wants to merge 1 commit into
biw:mainfrom
flamel-ai:fix/transient-cache-assignment-status

Conversation

@msichterman

@msichterman msichterman commented Aug 22, 2026

Copy link
Copy Markdown

What happens today

docker/job-started-hook.sh treats every status except 200 and 202 as a verdict and exits 1 on the first sample:

if [ "$status" != '202' ]; then
  printf '...The Worker returned HTTP $status...'
  exit 1
fi

Two real conditions reach that branch without meaning the assignment is invalid:

  • 401 - the container polls before the Worker's assignment authorization is visible through Cloudflare's edge. The record exists; the reader is early.
  • 000 - curl's output for a connection failure, so a single network blip ends the job.

Both surface as a failed Set up runner step before any repository code runs, and both clear on a re-run.

Evidence

Migrating a private repository's CI to cloudflare-standard-4, we measured every job attempt via the per-attempt jobs API. 30 attempts on Cloudflare runners, 7 failures in this hook (23.3%), split between the two messages:

##[error]The Worker returned HTTP 401 while waiting for GitHub's runner assignment.
##[error]GitHub's runner assignment was not observed within 30 seconds.

Every one succeeded on retry, on the same commit and the same runs-on label.

The failures cluster after a Worker deploy. Steady-state we saw roughly 1 in 8.
Immediately after re-running setup, which redeploys the Worker and rolls the runner
profiles, the next wave failed 4 of 4 with the 401, and a plain re-run of the same
jobs minutes later succeeded at Set up runner on the first sample. That is the shape
of a propagation race, not an invalid assignment: the record is being written and read
back through the edge while the previous value is still cached. It also means a routine
redeploy currently breaks every queued job until someone retries by hand.

The change

Keep polling on 000, 401, 408, 429, and 5xx inside the existing bounded window. No new knob, no longer default wait.

Fail-closed is preserved deliberately, because it is a security boundary rather than an accident: start-runner.sh writes the cache capability to disk for the results proxy regardless of this hook, so the hook's exit status is the only thing preventing a job from running with an assignment the Worker never confirmed. So this still exits 1 on any other status and when the window expires.

The timeout message now reports the last observed status, since "not observed within N seconds" alone does not distinguish a slow assignment from a stuck one.

Tests

pnpm test - 295 passing. The existing "unexpected Worker status" case moves to 403 so it still exercises the fatal path, plus two new cases:

  • a 401 that resolves to 200 starts the job
  • a 401 that never resolves still fails closed and still never prints the capability

Happy to adjust the retryable set or split the timeout-message change out if you would rather keep this minimal.

🤖 Generated with Claude Code

The job-started hook treated every status except 200 and 202 as a verdict and
exited 1 on the first sample. Two real conditions reach that branch without
meaning the assignment is invalid:

- 401, when the container polls before the Worker's assignment authorization
  is visible through Cloudflare's edge. The record exists; the reader is early.
- 000, curl's output for a connection failure, so one network blip ends the job.

Both surface to the user as a failed "Set up runner" step before any repository
code runs, and both clear on a re-run. Observed on a private repository across
about 16 job attempts: 3 failures, split between this branch and the bounded
wait window.

Keep polling on 000, 401, 408, 429, and 5xx inside the existing bounded window.
Continue to fail closed on every other status and when the window expires, so a
job still never runs with a cache capability the Worker did not confirm. Report
the last observed status in the timeout message, because "not observed within N
seconds" alone does not distinguish a slow assignment from a stuck one.

Tests: a 401 that resolves to 200 now starts the job; a 401 that never resolves
still fails closed and still never prints the capability.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant