Skip to content

DNM: debug#5898

Draft
damdo wants to merge 11 commits intokubernetes-sigs:mainfrom
damdo:capi-1.12-bump-debug
Draft

DNM: debug#5898
damdo wants to merge 11 commits intokubernetes-sigs:mainfrom
damdo:capi-1.12-bump-debug

Conversation

@damdo
Copy link
Copy Markdown
Member

@damdo damdo commented Mar 10, 2026

No description provided.

@k8s-ci-robot
Copy link
Copy Markdown
Contributor

k8s-ci-robot commented Mar 10, 2026

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@k8s-ci-robot
Copy link
Copy Markdown
Contributor

k8s-ci-robot commented Mar 10, 2026

Adding the "do-not-merge/release-note-label-needed" label because no release-note block was detected, please follow our release note process to remove it.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@k8s-ci-robot k8s-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. labels Mar 10, 2026
@k8s-ci-robot k8s-ci-robot requested a review from fiunchinho March 10, 2026 20:48
@k8s-ci-robot k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Mar 10, 2026
@k8s-ci-robot k8s-ci-robot requested a review from serngawy March 10, 2026 20:48
@k8s-ci-robot
Copy link
Copy Markdown
Contributor

k8s-ci-robot commented Mar 10, 2026

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign nrb for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label Mar 10, 2026
@damdo
Copy link
Copy Markdown
Member Author

damdo commented Mar 10, 2026

/test pull-cluster-api-provider-aws-e2e
/test pull-cluster-api-provider-aws-e2e-eks

@damdo
Copy link
Copy Markdown
Member Author

damdo commented Mar 10, 2026

/test pull-cluster-api-provider-aws-e2e

5 similar comments
@damdo
Copy link
Copy Markdown
Member Author

damdo commented Mar 11, 2026

/test pull-cluster-api-provider-aws-e2e

@damdo
Copy link
Copy Markdown
Member Author

damdo commented Mar 11, 2026

/test pull-cluster-api-provider-aws-e2e

@damdo
Copy link
Copy Markdown
Member Author

damdo commented Mar 12, 2026

/test pull-cluster-api-provider-aws-e2e

@damdo
Copy link
Copy Markdown
Member Author

damdo commented Mar 12, 2026

/test pull-cluster-api-provider-aws-e2e

@damdo
Copy link
Copy Markdown
Member Author

damdo commented Mar 13, 2026

/test pull-cluster-api-provider-aws-e2e

@k8s-ci-robot k8s-ci-robot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Mar 14, 2026
@damdo
Copy link
Copy Markdown
Member Author

damdo commented Mar 16, 2026

/test pull-cluster-api-provider-aws-e2e

1 similar comment
@damdo
Copy link
Copy Markdown
Member Author

damdo commented Mar 16, 2026

/test pull-cluster-api-provider-aws-e2e

@damdo damdo mentioned this pull request Mar 17, 2026
Merged
@damdo
Copy link
Copy Markdown
Member Author

damdo commented Mar 19, 2026

/test pull-cluster-api-provider-aws-e2e

@damdo damdo force-pushed the capi-1.12-bump-debug branch from ff78ddf to 2db02c8 Compare March 30, 2026 19:58
@damdo
Copy link
Copy Markdown
Member Author

damdo commented Mar 30, 2026

/test pull-cluster-api-provider-aws-e2e

1 similar comment
@damdo
Copy link
Copy Markdown
Member Author

damdo commented Mar 31, 2026

/test pull-cluster-api-provider-aws-e2e

@damdo damdo force-pushed the capi-1.12-bump-debug branch from 2db02c8 to b51760f Compare March 31, 2026 13:13
@damdo
Copy link
Copy Markdown
Member Author

damdo commented Mar 31, 2026

/test pull-cluster-api-provider-aws-e2e

@damdo damdo force-pushed the capi-1.12-bump-debug branch from b51760f to aab2eb4 Compare March 31, 2026 15:33
@damdo
Copy link
Copy Markdown
Member Author

damdo commented Mar 31, 2026

/test pull-cluster-api-provider-aws-e2e

@damdo
Copy link
Copy Markdown
Member Author

damdo commented Mar 31, 2026

/test pull-cluster-api-provider-aws-e2e

3 similar comments
@damdo
Copy link
Copy Markdown
Member Author

damdo commented Mar 31, 2026

/test pull-cluster-api-provider-aws-e2e

@damdo
Copy link
Copy Markdown
Member Author

damdo commented Apr 1, 2026

/test pull-cluster-api-provider-aws-e2e

@damdo
Copy link
Copy Markdown
Member Author

damdo commented Apr 1, 2026

/test pull-cluster-api-provider-aws-e2e

@k8s-ci-robot k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Apr 1, 2026
@damdo damdo force-pushed the capi-1.12-bump-debug branch from aab2eb4 to 978d143 Compare April 2, 2026 18:34
clebs and others added 11 commits April 2, 2026 21:08
@clebs @damdo
deps: update cluster-api, k8s and controller-runtime deps
97304f3 
- Update cluster-api to v1.12.2
- Update kubernetes depencencies to v0.34.3:
  - k8s.io/api
  - k8s.io/apiextensions-apiserver
  - k8s.io/apimachinery
  - k8s.io/client-go
  - k8s.io/component-base
- Update controller-runtime to v0.22.5
- Update ginkgo to v2.27.2
- Update gomega to v1.38.2
- Regenerate base CRDs

Signed-off-by: Borja Clemente <bclement@redhat.com>
@clebs @damdo
deps: update e2e testing versions and envtest
f625a85 
Signed-off-by: Borja Clemente <bclement@redhat.com>
@clebs @damdo
fix(e2e): use correct AMI k8s version
ba0265f 
Signed-off-by: Borja Clemente <bclement@redhat.com>
@clebs @damdo
test(e2e): Add v1.12 to CAPI version to contract map
f4989ac 
Signed-off-by: Borja Clemente <bclement@redhat.com>
@damdo
fix: flatcar log collection
8e139e5
@damdo
e2e: use goroutine-aware fail handler to prevent diagnostic dump panics
2ef6cbc 
Background diagnostic goroutines (resource dump every 5s, machine dump
every 60s) call upstream CAPI framework functions (DumpAllResources,
DumpMachines) that use Gomega Expect() assertions internally. When a
cluster is being deleted, these assertions can fail with "not found"
errors. Since these dumps are purely informational, their failures
should not mark the test spec as failed or cause panics.

The previous fix used InterceptGomegaFailure() to catch these failures,
but that function is not goroutine-safe: it temporarily replaces the
global Gomega fail handler, which can race with assertions in the test's
main goroutine. This caused [PANICKED] failures when Eventually()
timeouts in the test goroutine hit the intercepted handler, which panics
with "stop execution" instead of calling ginkgo.Fail().

Replace InterceptGomegaFailure with a goroutine-aware custom fail
handler registered via RegisterFailHandler. The handler uses a sync.Map
to track diagnostic goroutine IDs. When a Gomega assertion fails:
- In a diagnostic goroutine: panic with a sentinel value (without ever
  calling ginkgo.Fail), caught by a per-call recover() which logs a
  warning.
- In any other goroutine: delegate to ginkgo.Fail normally.

This ensures diagnostic dump failures are silently absorbed without
affecting global state or racing with test assertions.

Ref: https://kubernetes.slack.com/archives/CD6U2V71N/p1758795545213209
@damdo
fix SpotMarketOptions comparison
2dd81cc
@damdo
fix(e2e): use ECR image ref for CAPA controller to survive pod resche…
4276152 
…duling

During self-hosted e2e tests, the CAPA controller pod can be evicted
from a control plane node during upgrade drain and rescheduled to a
worker node. The previous gcr.io/k8s-staging-cluster-api/capa-manager:e2e
image reference is a local-only tag that doesn't exist on any registry,
so if the kubelet garbage-collects the pre-loaded image, the pod enters
ImagePullBackOff permanently.

Fix this by dynamically rewriting the CAPA provider component image
replacement to use the ECR Public URL (where the image is already
pushed by ensureTestImageUploaded). With imagePullPolicy: IfNotPresent,
the kubelet uses the local cache when available and falls back to
pulling from ECR Public if needed. The ECR-tagged image is also added
to the Kind bootstrap images list for consistency.
@damdo
chore: bump calico to v3.31.4 (to support k8s 1.34)
1cbe838
@damdo
fix!: tune API server ELB health check defaults to prevent kubeadm in…
e5feea8 
…it race

Lower the healthy threshold from 5 to 2 and the health check interval
from 10s to 5s, reducing the time for the target group to mark the API
server as healthy from 50s to 10s. This leaves a comfortable 50s margin
within kubeadm's default 60s kubernetesAPICallTimeout, preventing the
upload-config/kubeadm phase from hitting a context deadline when the ELB
has not yet started forwarding traffic.

Also increase the unhealthy threshold from 3 to 6 to compensate for the
shorter interval and avoid flapping during transient hiccups.
@damdo
feat/fix: re-add configurable DNS resolution check for API server LB
0512411 
Add a new `DNSResolutionCheck` field to AWSLoadBalancerSpec that allows
users to control whether the provider verifies DNS resolution of the API
server LB before marking the load balancer as ready and setting the
control plane endpoint.

By default (when unset), the DNS lookup is performed
So this check is now made by default as it is necessary to have a fully ready
and routable load balancer before proceeding further with the cluster
bootstrapping.
This has now become a strict prerequisite for CAPI clusters being
installed and bootrapped via the CAPI KubeAdm bootstrap provider,
given the recent changes to make this a stricter requirement that have
been introduced by kubeadm (see:
kubernetes/kubeadm#3294)

Setting the field to "None" skips the check entirely, which is useful in
environments with no kubeadm requirements, slow DNS propagation, custom resolvers, or private hosted
zones where the controller node may not be able to resolve the ELB's
FQDN despite it being valid.
@damdo damdo force-pushed the capi-1.12-bump-debug branch from 978d143 to 0512411 Compare April 2, 2026 19:15
@damdo
Copy link
Copy Markdown
Member Author

damdo commented Apr 2, 2026

/test pull-cluster-api-provider-aws-e2e

@k8s-ci-robot k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Apr 2, 2026
@damdo
Copy link
Copy Markdown
Member Author

damdo commented Apr 3, 2026

Flakes

/test pull-cluster-api-provider-aws-e2e

@k8s-ci-robot
Copy link
Copy Markdown
Contributor

k8s-ci-robot commented Apr 3, 2026

@damdo: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
pull-cluster-api-provider-aws-e2e-eks 507499a link false /test pull-cluster-api-provider-aws-e2e-eks
pull-cluster-api-provider-aws-e2e 0512411 link false /test pull-cluster-api-provider-aws-e2e

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@fiunchinho fiunchinho Awaiting requested review from fiunchinho

@serngawy serngawy Awaiting requested review from serngawy

Assignees

No one assigned

Labels

cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. needs-priority size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@damdo @k8s-ci-robot @clebs