Bug Report: AWS Security Group Leakage when manage-backend-security-group-rules is false

[sss-ng]

Bug Description
The AWS Load Balancer Controller incorrectly triggers ec2:AuthorizeSecurityGroupIngress on Node/Pod security groups even when the Ingress annotation alb.ingress.kubernetes.io/manage-backend-security-group-rules is set to "false".

I believe this occurs because the TargetGroupBinding (TGB) reconciler unconditionally invokes the NetworkingManager's reconciliation logic, regardless of whether a networking specification is present in the TGB spec. When the annotation is set to "false", the generated TGB has a nil Spec.Networking, but the controller proceeds to track it in its internal security group reconciliation lifecycle. In environments with strict IAM Permissions Boundaries (denying ec2:AuthorizeSecurityGroupIngress), this leads to recurring 403 UnauthorizedOperation errors and prevents successful reconciliation.

Steps to Reproduce

1. Deploy an Ingress with the following annotations:

       alb.ingress.kubernetes.io/scheme: internet-facing
       alb.ingress.kubernetes.io/target-type: ip
       alb.ingress.kubernetes.io/security-groups: sg-xxxxxx # Custom SG
       alb.ingress.kubernetes.io/manage-backend-security-group-rules: "false"

2. Ensure the controller's IAM role has a Permissions Boundary that denies ec2:AuthorizeSecurityGroupIngress (or simply lack the permission).
3. Observe the controller logs.

• Manifests applied while reproducing the issue:

   apiVersion: networking.k8s.io/v1
   kind: Ingress
   metadata:
     name: repro-ingress
     annotations:
       alb.ingress.kubernetes.io/scheme: internet-facing
       alb.ingress.kubernetes.io/target-type: ip
       alb.ingress.kubernetes.io/security-groups: sg-xxxxxx
       alb.ingress.kubernetes.io/manage-backend-security-group-rules: "false"
   spec:
     ingressClassName: alb
     rules:
     - http:
         paths:
         - path: /
           pathType: Prefix
           backend:
             service:
               name: repro-svc
               port:
                 number: 80

• Controller logs/error messages while reproducing the issue:

{"level": "error",
"ts": "2026-03-27T14:56:07Z",
"msg": "Requesting network requeue due to error from ReconcileForPodEndpoints",
"tgb": {"name":"k8s-REDACTED", "namespace": "REDACTED"},
"error": "operation error EC2: AuthorizeSecurityGroupIngress, https response error StatusCode: 403, RequestID: ..."

Expected Behavior
When manage-backend-security-group-rules is set to "false", the controller should skip all security group mutations for the associated TargetGroupBinding resources. Specifically, it should not call into the NetworkingManager for resources that have opted out.

Actual Behavior
The controller attempts to reconcile security group rules for the TGB despite the opt-out.

• The bug causes reconciliation failures and 403 errors in logs, though target group registration might still succeed if permissions for that are present.
• This happens always when the annotation is set to "false" but the global flag --enable-backend-security-group is true (default).

Regression
Was the functionality working correctly in a previous version ? [No / Unknown]
Verified in v2.17.1.

Current Workarounds
Setting the global flag --enable-backend-security-group=false resolves the issue by disabling backend SG management cluster-wide. However, this is not a viable solution if other Ingresses in the cluster do require managed security groups.

Environment

• AWS Load Balancer controller version: v2.17.1
• Kubernetes version: v1.30+
• Using EKS (yes/no), if so version?: Yes, 1.33
• Using Service or Ingress: Both
• AWS region: us-east-1
• How was the aws-load-balancer-controller installed: Helm

Possible Solution (Optional)
Add a nil guard in pkg/targetgroupbinding/resource_manager.go to check if tgb.Spec.Networking is non-nil before invoking the NetworkingManager.

In reconcileWithIPTargetType around line 209:

	if tgb.Spec.Networking != nil {
		if err := m.networkingManager.ReconcileForPodEndpoints(ctx, tgb, endpoints); err != nil {
			// ...
		}
	}

And similarly in reconcileWithInstanceTargetType.

Contribution Intention (Optional)

• Yes, I'm willing to submit a PR to fix this issue

Additional Context
Testing with a reproduction unit test confirmed that ReconcileForPodEndpoints is called even when Spec.Networking is nil. Implementing the guard locally fixed the issue and resolved the unauthorized API calls.

[shraddhabang]

@sss-ng Thank you for reaching out and providing the detailed analysis. I think it makes sense. I will try to reproduce this and will work on fixing this.

[shraddhabang]

I have tried to reproduce this configuration and confirmed that the resulting TargetGroupBinding has spec.networking: null. No AuthorizeSecurityGroupIngress calls were observed for this.
Could you confirm:

1. Are there other TargetGroupBindings in your cluster that have spec.networking populated?
2. Are the 403 errors referencing the same SGs that your opted-out pods are running on?

If your IAM permissions boundary denies ec2:AuthorizeSecurityGroupIngress entirely, then any TGB in the cluster with networking enabled will trigger those errors — not just the ones associated with your specific Ingress. The manage-backend-security-group-rules: "false" annotation only controls whether a given Ingress's TGBs get networking rules; it can't prevent other TGBs from managing rules on shared SGs.

[sss-ng]

I was afraid that I ran into some edge case that wasn't easily replicable.

Just to clarify a few things, we also saw that spec.networking was null, and yet the AWS LBC pod was continually logging the AuthorizeSecurityGroupIngress ... error.

I took a look at cloudtrail and saw that, it was trying to edit a security group attached to our EKS nodes, this I believe is the "backend" security group. However, this SG did not match the SG in our annotation alb.ingress.kubernetes.io/security-groups: sg-xxxxxx.

Indeed, we do have other Ingress objects which we are expecting to have this issue, as they were still "managing" their own SGs/Rules prior to us disabling all SG/Rule management.

However, we are seeing this issue for NEW Ingress objects being deployed that explicitly have manage-backend-security-group-rules: "false". The log explicitly shows that this new ingress is the reason.

It's as if the older Ingress objects are somehow tainting the new objects that I'd expect to not have this issue - since after all, the new ones are configured correctly.

In any case, we are planning to migrate the old ones away from trying to manage their own security groups, as this would likely fix the issue, but we thought we'd report this issue we saw during the transition period.

After some more research, I'm pretty convinced that adding that nil guard to the initial ReconcileForPodEndpoints function call will solve this, but not totally sure. What are your thoughts?

[zac-nixon]

I think the misunderstanding is how the various security groups interact:

Internet
   |
   v
+=======================+
|   Frontend SG (top)   |
+=======================+
          ||
          \/ 
   +-----------------+
   |  Load Balancer  |
   +-----------------+
          ||
          \/
+=======================+
| Backend SG (bottom)   |
+=======================+
          ||
          \/
        [Targets]

(Generated w/ Chat GPT)

The annotation alb.ingress.kubernetes.io/security-groups influences the Frontend SG. The SG that is being influenced, like you said, is the backend sg that controls traffic from your LB to inside the cluster.

The Frontend SG can is usually dedicated to a single load balancer (you can share it -- not recommended). The backend SG is always shared, this is what protects ingress to your worker nodes. So it doesn't make sense to revoke AuthorizeSecurityGroupIngress to the backend SG, if not all of your workloads disable backend sg management.

After some more research, I'm pretty convinced that adding that nil guard to the initial ReconcileForPodEndpoints function call will solve this, but not totally sure. What are your thoughts?

This doesn't work because we need to generate a full picture of the Backend SG rules to ensure we can remove rules safely.

[sss-ng]

Hi @zac-nixon, thanks for the clarification on that. I see the distinction between frontend/backend. However, I still have one thing I'm hung up on. Here's a scenario to illustrate:

Lets say I have a VPC that's allowed to create and manage security groups (young startup!)

We deploy App-A which has the following annotations:

       alb.ingress.kubernetes.io/scheme: internet-facing
       alb.ingress.kubernetes.io/target-type: ip

Then, a year later, we get acquired by an enterprise (mega corp!) who says, "we are disabling your ability to manage security groups via IAM." They perform the disabling, but App-A keeps working (for now) because the rules already exist.

Then, we get a new feature to develop App-B. We do that with the following annotations to comply with the new rules:

       alb.ingress.kubernetes.io/scheme: internet-facing
       alb.ingress.kubernetes.io/target-type: ip
       alb.ingress.kubernetes.io/security-groups: sg-xxxxxx # Custom SG provided by Mega Corp
       alb.ingress.kubernetes.io/manage-backend-security-group-rules: "false"

Now, we expected this to work, and the traffic does actually flow. However, what we saw in the logs is a constant stream of errors specifically for App-B:

{"level": "error",
"ts": "2026-03-27T14:56:07Z",
"msg": "Requesting network requeue due to error from ReconcileForPodEndpoints",
"tgb": {"name":"k8s-AppB", "namespace": "AppB"},
"error": "operation error EC2: AuthorizeSecurityGroupIngress, https response error StatusCode: 403..."

This is the core of my confusion. Since App-B has manage-backend-security-group-rules: "false", why is the controller attempting a write operation (AuthorizeSecurityGroupIngress) for this specific TGB?

If a specific Ingress has explicitly opted out of management, shouldn't the controller treat its associated TGB as a "no-op" for security group mutations?

[zac-nixon]

The backend SG is shared by all TGBs. On each reconcile run, the code is generating what it believes the state of the ingress rules should be, based off what is put into each TGB networking block.

Ref: https://github.com/kubernetes-sigs/aws-load-balancer-controller/blob/main/pkg/networking/networking_manager.go#L229-L241

So, just because App-B has no rules, the code is still attempting to correct the state of the backend SG because it is missing rules for another TGB.

[sss-ng]

Thanks, that makes sense to me. It sounds like this is expected behavior? It kind of seems like an issue worth fixing to me, but as this is only an issue due to the intermediate state that we are in, we hope to have it resolved by correcting our environment - so we wouldn't likely benefit from a code fix even if someone were to develop a fix for it

[maksym-iv]

I wonder if the behaviour is still same for the Gateway API.
Using manageBackendSecurityGroupRules: false in the LoadBalancerConfig, default (unchanged) enableManageBackendSecurityGroupRules in the helm values.
Despite that being set the controller always adds the security group rule to the Node (have not tested Pod) security group allowing traffic from the LB "shared" security group ([k8s] Shared Backend SecurityGroup for LoadBalancer)

For the test I've removed the ec2: AuthorizeSecurityGroupIngress permission from the role:

The Gateway with the LoadBalancerConfig

YAML manifest

---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  labels:
    cloud: aws
    env: none
    plane: none
    project: ops
    region: us-east-2
  name: test-priv
  namespace: aws-lb-controller
spec:
  gatewayClassName: aws-alb
  infrastructure:
    parametersRef:
      group: gateway.k8s.aws
      kind: LoadBalancerConfiguration
      name: test-priv
  listeners:
    - allowedRoutes:
        namespaces:
          from: All
      name: https
      port: 443
      protocol: HTTPS
---
apiVersion: gateway.k8s.aws/v1beta1

kind: LoadBalancerConfiguration
metadata:
  labels:
    cloud: aws
    env: none
    plane: none
    project: ops
    region: us-east-2
  name: test-priv
  namespace: aws-lb-controller
spec:
  ipAddressType: ipv4
  listenerConfigurations:
    - alpnPolicy: None
      defaultCertificate: arn:aws:acm:us-east-2:000000000000:certificate/64c9115-663f-4133-a039-01344acb0d58
      listenerAttributes:
        - key: routing.http.response.strict_transport_security.header_value
          value: max-age=31536000; includeSubDomains; preload
      protocolPort: HTTPS:443
      sslPolicy: ELBSecurityPolicy-TLS13-1-2-Res-PQ-2025-09
  loadBalancerName: test-priv
  loadBalancerSubnetsSelector:
    tier:
      - private
  manageBackendSecurityGroupRules: false
  scheme: internal
  sourceRanges:
    - 10.0.0.0/8

And getting the error

{
  "level": "error",
  "ts": "2026-04-09T10:42:57Z",
  "msg": "Requesting network requeue due to error from ReconcileForPodEndpoints",
  "tgb": {
    "name": "k8s-default-backendp-ns37s3acc8",
    "namespace": "default"
  },
  "error": "operation error EC2: AuthorizeSecurityGroupIngress, https response error StatusCode: 403, RequestID: 73ff0bce-cfd0-4086-b637-ef189c2875d0, api error UnauthorizedOperation: You are not authorized to perform this operation. User: arn:aws:sts::000000000000:assumed-role/aws-lb-controller/aws-lb-con-e0055419-58d2-490c-8618-b57d020342f2 is not authorized to perform: ec2:AuthorizeSecurityGroupIngress on resource: arn:aws:ec2:us-east-2:000000000000:security-group/sg-0ek342vc32j45rff0 because no identity-based policy allows the ec2:AuthorizeSecurityGroupIngress action...."
}