OCPBUGS-98695: add AWS CPO overrides for 4.22

[csrwng]

Summary

• Add AWS CPO image overrides for 4.22.0-4.22.6 to fix KMS key rotation and related issues
• 4.22.7 does not need an override: both PRs merged before the development cutoff (2026-07-22)
• Image source: Konflux push build (commit fb3b1a3)

branch: 4.22 wants: #8972, #8998

Test plan

• Unit tests pass (go test ./hypershift-operator/controlplaneoperator-overrides/...)
• /validate-pr-override-images passes against this PR
• Override image is publicly pullable from quay.io

🤖 Generated with Claude Code

Summary by CodeRabbit

• Bug Fixes
  • Updated Azure platform overrides for OpenShift 4.22 to use explicit control-plane-operator image digests for versions 4.22.0–4.22.6, with no override required for 4.22.7.
  • Refreshed the Azure testing image references used for control-plane-operator override validation (latest and previous) to align with the 4.22 release stream.

[openshift-merge-bot]

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: LGTM mode

[openshift-ci-robot]

@csrwng: This pull request references Jira Issue OCPBUGS-98627, which is invalid:

• expected the bug to target either version "5.0." or "openshift-5.0.", but it targets "4.22.0" instead
• expected the bug to be in one of the following states: NEW, ASSIGNED, POST, but it is MODIFIED instead

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

This pull request references Jira Issue OCPBUGS-98220, which is invalid:

• expected the bug to target either version "5.0." or "openshift-5.0.", but it targets "4.22.0" instead
• expected the bug to be in one of the following states: NEW, ASSIGNED, POST, but it is ON_QA instead

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

Summary

• Add AWS CPO image overrides for 4.22.0-4.22.6 to fix KMS key rotation and related issues
• 4.22.7 does not need an override: both PRs merged before the development cutoff (2026-07-22)
• Image source: Konflux push build (commit fb3b1a3)

branch: 4.22 wants: #8972, #8998

Test plan

• Unit tests pass (go test ./hypershift-operator/controlplaneoperator-overrides/...)
• /validate-pr-override-images passes against this PR
• Override image is publicly pullable from quay.io

🤖 Generated with Claude Code

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 35ebfe1b-7e04-4dda-b744-94134da29874

📥 Commits

Reviewing files that changed from the base of the PR and between 4ad6e7d and cb65566.

📒 Files selected for processing (1)

• hypershift-operator/controlplaneoperator-overrides/assets/overrides.yaml

🚧 Files skipped from review as they are similar to previous changes (1)

• hypershift-operator/controlplaneoperator-overrides/assets/overrides.yaml

---

📝 Walkthrough

Walkthrough

The Azure platform overrides now include OCP 4.22 entries for control-plane-operator versions 4.22.0 through 4.22.6. Each version uses the control-plane-operator-4-22 image with the specified SHA digest. Azure testing references now use OCP 4.22.5 as latest and OCP 4.21.24 as previous. A comment documents that version 4.22.7 does not require an override.

Suggested reviewers: clebs, nirshal

🚥 Pre-merge checks | ✅ 11

✅ Passed checks (11 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title is concise and accurately reflects the main change: adding CPO overrides for OpenShift 4.22.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names	✅ Passed	Only hypershift-operator/controlplaneoperator-overrides/assets/overrides.yaml changed; no Ginkgo test files or test titles were added or modified.
Test Structure And Quality	✅ Passed	No Ginkgo test code changed; the PR only updates YAML override test-image refs, so the test-structure checklist is not applicable.
Topology-Aware Scheduling Compatibility	✅ Passed	Only overrides.yaml image digests/test refs changed; no deployment manifests, controllers, or topology-sensitive scheduling constraints were added.
Ipv6 And Disconnected Network Test Compatibility	✅ Passed	Only overrides.yaml changed; no new Ginkgo e2e tests or IP/network logic were added, so the IPv6/disconnected test check is not applicable.
No-Weak-Crypto	✅ Passed	The diff only updates ocp-release image refs in YAML; no MD5/SHA1/DES/RC4/3DES/Blowfish, ECB mode, custom crypto, or secret comparisons appear.
Container-Privileges	✅ Passed	The PR only updates override image refs in overrides.yaml; no privileged/host* or allowPrivilegeEscalation settings are present.
No-Sensitive-Data-In-Logs	✅ Passed	The PR only updates YAML override image refs/comments; no logging code or sensitive data exposure is introduced.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: csrwng

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [csrwng]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[openshift-ci-robot]

@csrwng: This pull request references Jira Issue OCPBUGS-98627, which is invalid:

• expected the bug to target either version "5.0." or "openshift-5.0.", but it targets "4.22.0" instead
• expected the bug to be in one of the following states: NEW, ASSIGNED, POST, but it is MODIFIED instead

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

This pull request references Jira Issue OCPBUGS-98220, which is invalid:

• expected the bug to target either version "5.0." or "openshift-5.0.", but it targets "4.22.0" instead
• expected the bug to be in one of the following states: NEW, ASSIGNED, POST, but it is ON_QA instead

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

Summary

• Add AWS CPO image overrides for 4.22.0-4.22.6 to fix KMS key rotation and related issues
• 4.22.7 does not need an override: both PRs merged before the development cutoff (2026-07-22)
• Image source: Konflux push build (commit fb3b1a3)

branch: 4.22 wants: #8972, #8998

Test plan

• Unit tests pass (go test ./hypershift-operator/controlplaneoperator-overrides/...)
• /validate-pr-override-images passes against this PR
• Override image is publicly pullable from quay.io

🤖 Generated with Claude Code

Summary by CodeRabbit

• Bug Fixes
• Updated Azure platform configuration for OpenShift 4.22 control-plane-operator versions 4.22.0–4.22.6.
• Ensured these releases use the required control-plane-operator image digest for improved deployment compatibility.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 44.09%. Comparing base (8fac928) to head (cb65566).

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #9009   +/-   ##
=======================================
  Coverage   44.09%   44.09%           
=======================================
  Files         772      772           
  Lines       96165    96165           
=======================================
  Hits        42402    42402           
  Misses      50819    50819           
  Partials     2944     2944           

Flag	Coverage Δ
cmd-support	38.24% <ø> (ø)
cpo-hostedcontrolplane	46.06% <ø> (ø)
cpo-other	45.13% <ø> (ø)
hypershift-operator	54.09% <ø> (ø)
other	32.12% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[openshift-ci-robot]

@csrwng: This pull request references Jira Issue OCPBUGS-98086, which is invalid:

• expected the bug to be in one of the following states: NEW, ASSIGNED, POST, but it is Verified instead

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

Summary

• Add AWS CPO image overrides for 4.22.0-4.22.6 to fix KMS key rotation and related issues
• 4.22.7 does not need an override: both PRs merged before the development cutoff (2026-07-22)
• Image source: Konflux push build (commit fb3b1a3)

branch: 4.22 wants: #8972, #8998

Test plan

• Unit tests pass (go test ./hypershift-operator/controlplaneoperator-overrides/...)
• /validate-pr-override-images passes against this PR
• Override image is publicly pullable from quay.io

🤖 Generated with Claude Code

Summary by CodeRabbit

• Bug Fixes
• Updated Azure platform configuration for OpenShift 4.22 control-plane-operator versions 4.22.0–4.22.6.
• Ensured these releases use the required control-plane-operator image digest for improved deployment compatibility.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci-robot]

@csrwng: This pull request references Jira Issue OCPBUGS-98695, which is invalid:

• expected the bug to target the "5.0.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

Summary

• Add AWS CPO image overrides for 4.22.0-4.22.6 to fix KMS key rotation and related issues
• 4.22.7 does not need an override: both PRs merged before the development cutoff (2026-07-22)
• Image source: Konflux push build (commit fb3b1a3)

branch: 4.22 wants: #8972, #8998

Test plan

• Unit tests pass (go test ./hypershift-operator/controlplaneoperator-overrides/...)
• /validate-pr-override-images passes against this PR
• Override image is publicly pullable from quay.io

🤖 Generated with Claude Code

Summary by CodeRabbit

• Bug Fixes
• Updated Azure platform configuration for OpenShift 4.22 control-plane-operator versions 4.22.0–4.22.6.
• Ensured these releases use the required control-plane-operator image digest for improved deployment compatibility.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[csrwng]

/jira refresh

[openshift-ci-robot]

@csrwng: This pull request references Jira Issue OCPBUGS-98695, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (5.0.0) matches configured target version for branch (5.0.0)
• bug is in the state New, which is one of the valid states (NEW, ASSIGNED, POST)

Details

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci-robot]

@csrwng: This pull request references Jira Issue OCPBUGS-98695, which is valid.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (5.0.0) matches configured target version for branch (5.0.0)
• bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

Details

In response to this:

Summary

• Add AWS CPO image overrides for 4.22.0-4.22.6 to fix KMS key rotation and related issues
• 4.22.7 does not need an override: both PRs merged before the development cutoff (2026-07-22)
• Image source: Konflux push build (commit fb3b1a3)

branch: 4.22 wants: #8972, #8998

Test plan

• Unit tests pass (go test ./hypershift-operator/controlplaneoperator-overrides/...)
• /validate-pr-override-images passes against this PR
• Override image is publicly pullable from quay.io

🤖 Generated with Claude Code

Summary by CodeRabbit

• Bug Fixes
• Updated Azure platform overrides for OpenShift 4.22 to use explicit control-plane-operator image digests for versions 4.22.0–4.22.6.
• Prevents mismatches by ensuring the correct control-plane-operator image is selected for those releases (no override needed for 4.22.7).

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[csrwng]

/verified by e2e

[openshift-ci-robot]

@csrwng: This PR has been marked as verified by e2e.

Details

In response to this:

/verified by e2e

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci]

@csrwng: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-aks-override	cb65566	link	true	/test e2e-aks-override

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[ahitacat]

Shouldn't these changes be in azure platform section? This are fixes mostly for azure

[hypershift-jira-solve-ci]

Now I have the complete analysis. Let me produce the final report:

Test Failure Analysis Complete

Job Information

• Prow Job: pull-ci-openshift-hypershift-main-e2e-aks-override
• Build ID: 2077222347722461184
• Target: e2e-aks-override
• Platform: Azure AKS
• OCP Version: 4.19.10 (latest), 4.19.9 (previous)
• PR: OCPBUGS-98695: add AWS CPO overrides for 4.22 #9009 — OCPBUGS-98695: add AWS CPO overrides for 4.22
• Result: 214 tests, 59 skipped, 6 failures (2 independent leaf failures + 4 parent-cascade failures)

Test Failure Analysis

Error

Failure 1 — TestCreateCluster/Main/EnsureGlobalPullSecret:
  Timed out after 30.001s.
  DaemonSet is not present
  daemonsets.apps "global-pull-secret-syncer" not found (404 NotFound)

Failure 2 — TestAutoscaling/Main/TestAutoscalerRespectsNodePoolPause:
  MachineDeployment replicas should not change while paused — CAS should not
  decrement replicas on paused MachineDeployments (regression for OCPBUGS-78152)
  Expected <int32>: 1 to equal <int32>: 3

Summary

Both failures are pre-existing flaky tests unrelated to PR #9009. The PR only modifies overrides.yaml to add AWS CPO image overrides for OCP 4.22.0–4.22.6 and update testing image refs from 4.17 to 4.22/4.21. The test suite runs on Azure AKS with OCP 4.19, and the CPO override verification for the pre-existing Azure 4.19 override passed successfully. The two failing tests exercise entirely different subsystems (the HCCO global pull secret reconciler and the Cluster Autoscaler pause behavior) that are not touched by the overrides.yaml change.

Root Cause

Failure 1 — EnsureGlobalPullSecret (TestCreateCluster):

The global-pull-secret-syncer DaemonSet was not found in the guest cluster's kube-system namespace after 30 seconds of polling. This DaemonSet is deployed by the Hosted Cluster Config Operator (HCCO), which always reconciles it on OCP 4.19+ Azure clusters regardless of whether an additional pull secret exists. The failure indicates the HCCO reconciler had not yet created the DaemonSet within the 30-second timeout window. This is a timing/reconciliation race — the HCCO may have been slow to start reconciliation, or the guest cluster's API server was slow to register the resource. The DaemonSet targets only nodes labeled hypershift.openshift.io/nodepool-globalps-enabled: "true" (Replace upgrade strategy nodes), so the HCCO may have deferred creation. This test has no version gate that would skip it.

Failure 2 — TestAutoscalerRespectsNodePoolPause (TestAutoscaling):

The Cluster Autoscaler (CAS) scaled down the MachineDeployment from 3 to 1 replicas despite the NodePool being paused with cluster.x-k8s.io/paused: "true" annotation. The test confirmed the MachineDeployment was paused at 3 replicas, deleted the workload, then polled CAS logs for 3 minutes looking for "discovered a paused node group" — this log was never detected, and the MachineDeployment was subsequently found at 1 replica. This is a regression in CAS pause-awareness behavior (OCPBUGS-78152). The CAS appears to have ignored the pause annotation on the MachineDeployment, possibly due to a timing issue where the scale-down decision was made before CAS processed the pause annotation, or a CAS version/configuration issue in the 4.19 AKS environment. The ingress operator was also not available at teardown (ClusterOperatorNotAvailable(Cluster operator ingress is not available)), suggesting the hosted cluster had degraded conditions.

Neither failure is caused by the PR changes. PR #9009 modifies only overrides.yaml to add AWS-platform overrides for 4.22 and update testing image references. The test runs on Azure 4.19, and the Azure 4.19 CPO override (pre-existing, not changed by this PR) was verified to be correctly applied.

Recommendations

1. Retrigger the job — Both failures appear to be pre-existing flaky tests. A /retest should clear them.

2. For EnsureGlobalPullSecret: Consider increasing the 30-second polling timeout or adding a version/platform-specific skip for environments where the HCCO may take longer to reconcile the DaemonSet. The test lacks retry logic for transient HCCO reconciliation delays.

3. For TestAutoscalerRespectsNodePoolPause: The CAS pause-awareness regression (OCPBUGS-78152) appears to still be manifesting in the 4.19 AKS environment. Investigate whether the CAS version deployed in this environment correctly handles the cluster.x-k8s.io/paused annotation on MachineDeployments. The test correctly verifies the annotation is set before deleting the workload.

4. These failures do not block the PR — The only file changed (overrides.yaml) adds AWS 4.22 entries and updates testing references. The CPO override mechanism itself was validated successfully (the control-plane-operator-4-19 override image was confirmed running).

Evidence

Evidence	Detail
PR scope	Only overrides.yaml changed — adds AWS 4.22 CPO overrides, updates testing refs from 4.17→4.22/4.21
Test platform	Azure AKS with OCP 4.19.10 (not AWS, not 4.22)
CPO override verification	✅ PASSED — control-plane-operator-4-19@sha256:88c55ea... confirmed running at util.go:4441
Failure 1 location	util.go:2014 — 30s timeout polling for global-pull-secret-syncer DaemonSet in kube-system
Failure 1 test gate	skipGlobalPullSecretPreconditions requires OCP ≥4.19 + Azure/AWS platform — test ran, not skipped
Failure 2 location	autoscaling_test.go:635–637 — MachineDeployment replicas 1 (expected 3) after pause
Failure 2 timeline	NodePool paused at 04:45:13, workload deleted at 04:45:16→04:48:16 (3m CAS poll timeout), replicas already 1
Failure 2 CAS log	"CAS log not detected within timeout" — CAS never logged awareness of paused node group
Total test results	214 tests, 59 skipped, 6 failures (2 leaf + 4 parent cascades)
Passing CPO tests	TestCreateClusterHABreakGlassCredentials, TestNodePool (all variants) — all PASSED

---

[celebdor]

This should be for aks, not aws, right?

[celebdor]

This PR places the overrides under the aws platform section, but the referenced bugs (OCPBUGS-98627, OCPBUGS-98220) are both Azure/ARO issues (component: HyperShift / ARO). The overrides should be under the azure section instead.

Superseded by #9010, which adds the overrides under the correct azure platform section and also supersedes the existing CNTRLPLANE-3619/3656 4.22.0-4.22.3 entries with the new image that includes all four PRs (#8721, #8790, #8972, #8998).

[csrwng]

/close

[openshift-ci-robot]

@csrwng: This pull request references Jira Issue OCPBUGS-98695. The bug has been updated to no longer refer to the pull request using the external bug tracker.

Details

In response to this:

Summary

• Add AWS CPO image overrides for 4.22.0-4.22.6 to fix KMS key rotation and related issues
• 4.22.7 does not need an override: both PRs merged before the development cutoff (2026-07-22)
• Image source: Konflux push build (commit fb3b1a3)

branch: 4.22 wants: #8972, #8998

Test plan

• Unit tests pass (go test ./hypershift-operator/controlplaneoperator-overrides/...)
• /validate-pr-override-images passes against this PR
• Override image is publicly pullable from quay.io

🤖 Generated with Claude Code

Summary by CodeRabbit

• Bug Fixes
• Updated Azure platform overrides for OpenShift 4.22 to use explicit control-plane-operator image digests for versions 4.22.0–4.22.6, with no override required for 4.22.7.
• Refreshed the Azure testing image references used for control-plane-operator override validation (latest and previous) to align with the 4.22 release stream.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci]

@csrwng: Closed this PR.

Details

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[openshift-merge-robot]

Fix included in release 4.22.0-0.nightly-2026-07-15-214410