Skip to content

docs(ci): record why image builds stay on GitHub-hosted runners - #846

Open
funkypenguin wants to merge 1 commit into
mainfrom
docs-record-runner-decision
Open

docs(ci): record why image builds stay on GitHub-hosted runners#846
funkypenguin wants to merge 1 commit into
mainfrom
docs-record-runner-decision

Conversation

@funkypenguin

Copy link
Copy Markdown
Contributor

Comment-only. Closes the "decide on the containers build workflows" item from docs/BRIEF-arc-runners-release-workflows.md (elfhosted/shadowfax#561), which called this the heavyweight case and asked for a deliberate decision rather than a silent deferral.

The decision is: don't move them — and the reason is not the one the brief expected.

  1. This repo is public, so it was never exposed. GitHub-hosted minutes for public repos are free and unmetered, and the org spending-limit failure did not touch them. That held empirically through the incident: while every ubuntu-latest job in the org's private repos died in 4–5s for ~15h from 13:11Z, this repo's jobs ran clean — Release: Manual at 21:03Z and 01:44Z, Image: Rebuild at 01:55Z, and hourly Renovate throughout.
  2. arc couldn't run them anyway. The scale set declares no containerMode, so runner pods have no Docker daemon (buildx is a non-starter) and no sudo/apt (generate-build-matrix installs moreutils and jo).
  3. It would cost resilience. Ten 1-CPU runners with no layer cache, replacing GitHub's free cached fleet — and the image-build path would then depend on the same cluster the images get deployed to.

The comment goes in release-manual.yaml because that's where the next person will ask the question, and because the obvious-looking fix (a blanket runs-on swap) would break every build.

Revisit if this repo ever goes private, or if public-repo minutes start being metered.

Also spotted while looking, not addressed here: Release: Schedule has been failing continuously — before, during and after the incident. Those runs last 300–750s and then fail, so that's real job logic failing, not a 4-second billing rejection. Wanted to flag it rather than fold it into a docs PR.

Full inventory and rationale: docs/DECISION-arc-runners-release-workflows.md in elfhosted/shadowfax.

The 2026-08-23 org billing failure prompted an org-wide move of
release-blocking workflows to the in-cluster arc-runners pool. This repo's
image builds are release-blocking, so the absence of a change here needs to be
a recorded decision rather than an oversight -- particularly since the obvious
next step for a future reader is a blanket runs-on swap that would break every
build.

Comment only -- no behaviour change.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant