fix(runner): do not fail a job on a transient cache-assignment status - #5
Open
msichterman wants to merge 1 commit into
Open
fix(runner): do not fail a job on a transient cache-assignment status#5msichterman wants to merge 1 commit into
msichterman wants to merge 1 commit into
Conversation
The job-started hook treated every status except 200 and 202 as a verdict and exited 1 on the first sample. Two real conditions reach that branch without meaning the assignment is invalid: - 401, when the container polls before the Worker's assignment authorization is visible through Cloudflare's edge. The record exists; the reader is early. - 000, curl's output for a connection failure, so one network blip ends the job. Both surface to the user as a failed "Set up runner" step before any repository code runs, and both clear on a re-run. Observed on a private repository across about 16 job attempts: 3 failures, split between this branch and the bounded wait window. Keep polling on 000, 401, 408, 429, and 5xx inside the existing bounded window. Continue to fail closed on every other status and when the window expires, so a job still never runs with a cache capability the Worker did not confirm. Report the last observed status in the timeout message, because "not observed within N seconds" alone does not distinguish a slow assignment from a stuck one. Tests: a 401 that resolves to 200 now starts the job; a 401 that never resolves still fails closed and still never prints the capability. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What happens today
docker/job-started-hook.shtreats every status except200and202as a verdict and exits1on the first sample:Two real conditions reach that branch without meaning the assignment is invalid:
401- the container polls before the Worker's assignment authorization is visible through Cloudflare's edge. The record exists; the reader is early.000- curl's output for a connection failure, so a single network blip ends the job.Both surface as a failed Set up runner step before any repository code runs, and both clear on a re-run.
Evidence
Migrating a private repository's CI to
cloudflare-standard-4, we measured every job attempt via the per-attempt jobs API. 30 attempts on Cloudflare runners, 7 failures in this hook (23.3%), split between the two messages:Every one succeeded on retry, on the same commit and the same
runs-onlabel.The failures cluster after a Worker deploy. Steady-state we saw roughly 1 in 8.
Immediately after re-running setup, which redeploys the Worker and rolls the runner
profiles, the next wave failed 4 of 4 with the 401, and a plain re-run of the same
jobs minutes later succeeded at
Set up runneron the first sample. That is the shapeof a propagation race, not an invalid assignment: the record is being written and read
back through the edge while the previous value is still cached. It also means a routine
redeploy currently breaks every queued job until someone retries by hand.
The change
Keep polling on
000,401,408,429, and5xxinside the existing bounded window. No new knob, no longer default wait.Fail-closed is preserved deliberately, because it is a security boundary rather than an accident:
start-runner.shwrites the cache capability to disk for the results proxy regardless of this hook, so the hook's exit status is the only thing preventing a job from running with an assignment the Worker never confirmed. So this still exits1on any other status and when the window expires.The timeout message now reports the last observed status, since "not observed within N seconds" alone does not distinguish a slow assignment from a stuck one.
Tests
pnpm test- 295 passing. The existing "unexpected Worker status" case moves to403so it still exercises the fatal path, plus two new cases:401that resolves to200starts the job401that never resolves still fails closed and still never prints the capabilityHappy to adjust the retryable set or split the timeout-message change out if you would rather keep this minimal.
🤖 Generated with Claude Code