Skip to content

releng: a new component image fails its first release because its GHCR package does not exist #701

Description

Symptom

The first release that includes a new component image fails, four minutes into
the job, with:

ERROR: failed to push ghcr.io/azure/token-refresher:v0.5.0: denied: permission_denied: write_package

run 33655638847, attempt 1, component-images (token-refresher).
This was the failure that started the v0.5.0 cascade: fail-fast cancelled
seven healthy sibling builds, and the pipeline then could not be re-run for
unrelated reasons (fixed in #700). The release was abandoned and v0.6.0 cut
instead.

Cause

release.yaml grants packages: write at the workflow level
(.github/workflows/release.yaml:44-52), which is correct. But GHCR requires
the package to already exist and be linked to the repository before
GITHUB_TOKEN can push to it. For a package that has never been pushed, the
first push from Actions is denied.

token-refresher was added in 896cc73 (#646) and is not in images.yaml,
whose only triggers are:

on:
  push:
    tags:
      - "images/*/*"
  workflow_dispatch:

so nothing had ever pushed ghcr.io/azure/token-refresher before the release
tried to. Someone created the package out of band between attempts 1 and 3, and
the job succeeded from attempt 3 onward, which confirms the diagnosis.

Why it matters

This recurs for every new component image, and it fails in the worst place:
late, during a release, after other images have already been pushed, signed and
attested. The component matrix currently has 11 entries and grows.

Options

  1. Preflight the matrix. A cheap job that checks every component's package
    is pushable before any of them build, so a new image fails in seconds with a
    message naming the package, instead of four minutes in with a raw registry
    error. Does not remove the manual step, but makes it obvious and cheap.
  2. Build component images on merge to main. Extend images.yaml (or add a
    path-filtered job) so a component's package is created the first time its
    Containerfile lands, long before a release needs it. Removes the failure
    entirely; costs CI minutes on every merge that touches images/.
  3. Create the package as part of adding a component. Document it as a step
    in whatever checklist covers adding an image, and leave the pipeline alone.
    Cheapest, and the one most likely to be forgotten.

Option 2 removes the class of failure; option 1 makes it survivable and is
compatible with either of the others.

Notes

  • Needs someone with package admin on the org to confirm the current settings,
    specifically whether org policy permits GITHUB_TOKEN to create packages, or
    whether every package must be created and linked by hand.
  • release: make the build survive a re-run #700 makes the release re-runnable, so a recurrence is now recoverable rather
    than terminal. That reduces the severity but does not fix this.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workinggithub_actionsPull requests that update GitHub Actions codereleng

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions