You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The first release that includes a new component image fails, four minutes into
the job, with:
ERROR: failed to push ghcr.io/azure/token-refresher:v0.5.0: denied: permission_denied: write_package
run 33655638847, attempt 1, component-images (token-refresher).
This was the failure that started the v0.5.0 cascade: fail-fast cancelled
seven healthy sibling builds, and the pipeline then could not be re-run for
unrelated reasons (fixed in #700). The release was abandoned and v0.6.0 cut
instead.
Cause
release.yaml grants packages: write at the workflow level
(.github/workflows/release.yaml:44-52), which is correct. But GHCR requires
the package to already exist and be linked to the repository before GITHUB_TOKEN can push to it. For a package that has never been pushed, the
first push from Actions is denied.
token-refresher was added in 896cc73 (#646) and is not in images.yaml,
whose only triggers are:
on:
push:
tags:
- "images/*/*"workflow_dispatch:
so nothing had ever pushed ghcr.io/azure/token-refresher before the release
tried to. Someone created the package out of band between attempts 1 and 3, and
the job succeeded from attempt 3 onward, which confirms the diagnosis.
Why it matters
This recurs for every new component image, and it fails in the worst place:
late, during a release, after other images have already been pushed, signed and
attested. The component matrix currently has 11 entries and grows.
Options
Preflight the matrix. A cheap job that checks every component's package
is pushable before any of them build, so a new image fails in seconds with a
message naming the package, instead of four minutes in with a raw registry
error. Does not remove the manual step, but makes it obvious and cheap.
Build component images on merge to main. Extend images.yaml (or add a
path-filtered job) so a component's package is created the first time its
Containerfile lands, long before a release needs it. Removes the failure
entirely; costs CI minutes on every merge that touches images/.
Create the package as part of adding a component. Document it as a step
in whatever checklist covers adding an image, and leave the pipeline alone.
Cheapest, and the one most likely to be forgotten.
Option 2 removes the class of failure; option 1 makes it survivable and is
compatible with either of the others.
Notes
Needs someone with package admin on the org to confirm the current settings,
specifically whether org policy permits GITHUB_TOKEN to create packages, or
whether every package must be created and linked by hand.
release: make the build survive a re-run #700 makes the release re-runnable, so a recurrence is now recoverable rather
than terminal. That reduces the severity but does not fix this.
Symptom
The first release that includes a new component image fails, four minutes into
the job, with:
run 33655638847, attempt 1,
component-images (token-refresher).This was the failure that started the
v0.5.0cascade:fail-fastcancelledseven healthy sibling builds, and the pipeline then could not be re-run for
unrelated reasons (fixed in #700). The release was abandoned and
v0.6.0cutinstead.
Cause
release.yamlgrantspackages: writeat the workflow level(
.github/workflows/release.yaml:44-52), which is correct. But GHCR requiresthe package to already exist and be linked to the repository before
GITHUB_TOKENcan push to it. For a package that has never been pushed, thefirst push from Actions is denied.
token-refresherwas added in 896cc73 (#646) and is not inimages.yaml,whose only triggers are:
so nothing had ever pushed
ghcr.io/azure/token-refresherbefore the releasetried to. Someone created the package out of band between attempts 1 and 3, and
the job succeeded from attempt 3 onward, which confirms the diagnosis.
Why it matters
This recurs for every new component image, and it fails in the worst place:
late, during a release, after other images have already been pushed, signed and
attested. The component matrix currently has 11 entries and grows.
Options
is pushable before any of them build, so a new image fails in seconds with a
message naming the package, instead of four minutes in with a raw registry
error. Does not remove the manual step, but makes it obvious and cheap.
main. Extendimages.yaml(or add apath-filtered job) so a component's package is created the first time its
Containerfile lands, long before a release needs it. Removes the failure
entirely; costs CI minutes on every merge that touches
images/.in whatever checklist covers adding an image, and leave the pipeline alone.
Cheapest, and the one most likely to be forgotten.
Option 2 removes the class of failure; option 1 makes it survivable and is
compatible with either of the others.
Notes
specifically whether org policy permits
GITHUB_TOKENto create packages, orwhether every package must be created and linked by hand.
than terminal. That reduces the severity but does not fix this.