Old ACM certificate is not removed from ALB

[MartinEmrich]

Bug Description

Old certificate stays in use by the ALB.

When replacing the TLS certificate, by updating the alb.ingress.kubernetes.io/certificate-arn: annotation, the new certificate is added as an SNI certificate, and replaced as the default certificate. But the previous certificate ARN, despite being never mentioned on the EKS cluster, is still bound as an additional SNI certificate.

AWS support also confirmed to me that adding the "new" certificate both as default certificate and again as SNI certificate is unnecessary and redundant.

Steps to Reproduce

• Create an ACM certificicate, and an Ingress with TLS, configure that ACM certificate.
• Create another ACM certificate (same hostname(s)).
• Replace the ARN in the Ingress resource, let ALB controller reconcile.

Expected Behavior

Only the new certificate should be the default certificate, no need for any additional SNI certificate entries.

Actual Behavior

See above: both are still bound to the ALB.
Removing the wrong entries from the Listener is useless, they reappear after next reconcile.

Current Workarounds

Disable ALB controller.
Manually edit the ALB listener settings after the change.
Delete the old certificate before ALB controller has a chance to discover them
Re-Enable ALB controller.

Environment

• AWS Load Balancer controller version: 3.0.0
• Kubernetes version: 1.35
• Using EKS (yes/no), if so version?: yes, 1.35
• Using Service or Ingress: Ingess
• AWS region: eu-central-1
• How was the aws-load-balancer-controller installed: Helm chart (via Flux, thus no helm command line)

values:
  clusterName: {{ .Values.clusterName }}
  serviceAccount:
    name: aws-load-balancer-controller
    create: false
  ingressClassConfig:
    default: true

Possible Solution (Optional)

This is most likely a bug in the TLS ACM certificate autodiscovery. First, it seems to be enabled, while we both have a certificate explicitly configured, and no configuration or annotation explicitly enabling it.

Even if, if it discovers multiple certificates for the same domain name(s), it should deterministically one of them (The "best" crypto-wise, or even better, just the newest based on creation time stamp).

The documentation rather uses wordings like "can be discovered" suggesting it is optional, not mandatory. Indeed the examples all omit an explicit certificate-arn annotation, or use the tls: object in the Ingress spec.

Another cause could be that

if the spec.certificateArn in IngressClassParams or alb.ingress.kubernetes.io/certificate-arn annotation is not specified.

Is evaluated for each Ingress in the same ingress group before merge, instead of after the annotation merge.

[shraddhabang]

Hey @MartinEmrich , Can you clarify if you are using cert-discovery or using annotationalb.ingress.kubernetes.io/certificate-arn: to attach your certs ? You have mentioned both which is confusing.

If you are using cert-discovery, the LBC does not support to choose the best cert as you mentioned. It will attach all the matching certs. That is expected behavior.

But if you are using annotation alb.ingress.kubernetes.io/certificate-arn:, the LBC will only attach the certs specified here. You should not see any other certs apart from what you have given.

[MartinEmrich]

I only use the annotation (on one ingress in the group bearing all global annotations). It looks like the discovery is active, too, but I don't want it to be.

[shraddhabang]

If you use annotation only, LBC will not discover certs. It will only use certs specified by the annotation. Can you ensure if you are using any IngressClassParam which has cert arns defined in it?

[MartinEmrich]

On one cluster, I noticed a stray Ingress that still had another certificate configured explicitly, this certainly has led my analysis off-track. Apologies!

But I experienced the phenomenon on two other clusters with only that single entry, so I am still on to it.

I have only one IngressClassParam resource on the whole cluster, named "alb", apparently installed by the AWS Load Balancer Controller. It does not even have a spec: object.

I will try to tear down and rebuild a test environment from scratch, and report back.

[MartinEmrich]

I tried it on a test platform and could reproduce it:

• Start with one TLS certficate, configure it with annotation on one ingress in the group. Works as expected. Verified, kubectl get ingress -A -oyaml |grep certificate-arn lists exactly that one line at that ingress, no other certificates.
• Create a new certificate, replace the ARN in the Ingress.
• ALB controller reconciles, now both certificates are bound to the Listener.

In the logs, both old and new certificate ID are present in the long "Successfully built model" log line, too. So I guess it is not just "forgotten to be removed".