[Bug]: EKS Detector relies on existence of & access to `aws-auth` ConfigMap

[milas]

Component

Detector: AWS EKS

Describe the issue you're facing

The EKS detector determines that it's running in EKS by fetching the aws-auth ConfigMap in the kube-system namespace:

opentelemetry-go-contrib/detectors/aws/eks/detector.go

Lines 121 to 125 in db30b79

	// Make HTTP GET request
	awsAuth, err := utils.getConfigMap(ctx, authConfigmapNS, authConfigmapName)
	if err != nil {
	return false, fmt.Errorf("isEks() error retrieving auth configmap: %w", err)
	}

This is unfortunate for a couple reasons:

1. This ConfigMap is actually not required in EKS clusters and is in fact deprecated, so is not a reliable signal.
2. It requires workloads to have read access to an object in privileged namespace (despite not using any of the data).

Expected behavior

• EKS detector uses a reliable/non-deprecated approach to detecting an EKS cluster
• EKS detector does not require special in-cluster RBAC

I'm not sure if IMDS is a good fallback since that's often blocked within Pods.

It's also worth noting that I'm not sure what the expected error behavior is here. If the detector determines it's not running in an EKS cluster, it returns gracefully. However, if the detector runs into an RBAC limitation here (which means the configmap might not even exist because it might not actually be running in an EKS cluster), it does return an error.

Steps to Reproduce

1. Add the EKS detector
2. Launch in an K8s cluster (of any kind!) but do not allow the Pod to read that ConfigMap.

error detecting resource: isEks() error retrieving auth configmap: failed to retrieve ConfigMap kube-system/aws-auth: configmaps "aws-auth" is forbidden: User "system:serviceaccount:my-namespace:my-service-account" cannot get resource "configmaps" in API group "" in the namespace "kube-system"

Operating System

Linux

Device Architecture

ARM64

Go Version

1.25

Component Version

detectors/aws/eks v1.41.0

[dmathieu]

cc @akats7 as code owner.

[alimx07]

Hey @dmathieu @milas

I ran into this same issue and while looking at the code and noticed that isK8s() already checks if the SA token exists at /var/run/secrets/kubernetes.io/serviceaccount/token. Could we just go one step further and actually read that token, decode the JWT payload, and check the iss claim?

opentelemetry-go-contrib/detectors/aws/eks/detector.go

Lines 23 to 25 in db30b79

	const (
	k8sTokenPath = "/var/run/secrets/kubernetes.io/serviceaccount/token" //nolint:gosec // False positive G101: Potential hardcoded credentials. The detector only check if the token exists.
	k8sCertPath = "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt"

opentelemetry-go-contrib/detectors/aws/eks/detector.go

Lines 148 to 151 in db30b79

	func isK8s(utils detectorUtils) bool {
	return utils.fileExists(k8sTokenPath) && utils.fileExists(k8sCertPath)
	}

On EKS the service account issuer is always https://oidc.eks.<region>.amazonaws.com/id/<cluster-id> , this is set by the control plane and not configurable by users. So checking for oidc.eks. in the issuer should be a pretty reliable EKS signal.

One edge case is automountServiceAccountToken: false where the token file won't be there , but isK8s() already relies on that same file existing, so if it's not mounted the detector already bails out before we even get to the EKS check. So I think we're fine there (for now).

Is this a sufficient approach or am I missing something? If this sounds reasonable I'm happy to open a PR.

[Aminkbi]

I took a closer look at the current main branch and this still seems unfixed.

Right now the detector still uses kube-system/aws-auth as the EKS signal in isEKS (https://github.com/open-telemetry/opentelemetry-go-contrib/blob/main/detectors/aws/eks/detector.go), and still returns an error if that lookup is
forbidden or missing. So the core problem from this issue is still present for normal in-cluster workloads without extra RBAC. Related reports also point to the same broader problem in #1856
(#1856), #6309 (#6309), and #7502
(#7502).

I also looked at the suggestion of decoding the service account JWT and checking the iss claim from the token mounted at /var/run/secrets/kubernetes.io/serviceaccount/token (discussion here
(#8637 (comment))). I think that is directionally better than relying on aws-auth, because it removes the privileged ConfigMap dependency.

That said, I do not think a simple string check like strings.Contains(iss, "oidc.eks.") is robust enough long term. It hardcodes one URL shape into the detector, and AWS OIDC issuer patterns have already evolved over time. If t
he detector goes down that route, I think it should parse the issuer structurally and validate against known EKS issuer forms rather than rely on a substring match. For context, AWS documents OIDC issuer usage for service accou
nts in the IRSA docs here: Associate IAM roles with Kubernetes service accounts (https://docs.aws.amazon.com/eks/latest/userguide/associate-service-account-role.html). AWS also documents the legacy aws-auth ConfigMap separately
here: Grant IAM users access to Kubernetes with a ConfigMap (https://docs.aws.amazon.com/eks/latest/userguide/auth-configmap.html), which reinforces that this is cluster auth/config state, not a very strong detector signal.

Separately, even if the EKS detection step is changed, the detector still has the cluster-name problem from #6309 (#6309): it currently relies on amazon-cloudwatc
h/cluster-info, which is not generally present, and that lookup is still treated as fatal in the current code path.

So from a user perspective there are really two independent brittle dependencies today:

• aws-auth for “am I on EKS?”
• amazon-cloudwatch/cluster-info for k8s.cluster.name

My bias would be:

1. Make EKS detection RBAC-free.
2. Make cluster name best-effort instead of fatal.
3. Keep container ID out of scope for this change since there is already an open PR for that in Fix EKS detector container ID parsing for Kubernetes/containerd #7786 (Fix EKS detector container ID parsing for Kubernetes/containerd #7786).

Concretely, a good first step seems to be:

• stop using aws-auth as the detection source
• switch detection to a non-privileged signal
• if cluster name cannot be resolved, still return the EKS resource with cloud.provider=aws / cloud.platform=aws_eks instead of failing the detector

If maintainers think that direction is reasonable, I’d be happy to put together a focused PR with tests covering:

• aws-auth forbidden / not found
• missing cluster-info
• valid and invalid issuer-based EKS detection paths

[alimx07]

Hey, thanks for the detailed write-up, but I want to push back on a few things here because I'm not sure what concrete value this adds over what was already discussed.

On the issuer format changing:

You say strings.Contains(iss, "oidc.eks.") isn't robust enough because "AWS OIDC issuer patterns have already evolved over time." When exactly did they change? The EKS service account issuer has been https://oidc.eks.<region>.amazonaws.com/id/<cluster-id> since IRSA was introduced. This is set by the control plane and is not user-configurable on standard EKS. If you have a concrete example of the format changing I'd genuinely like to see it, but I can't find one. Also, Maybe we could add more signals to detect EKS if there are cases JWT will fail, but again you did not mention any signal we should add.

On the cluster name:

You call out amazon-cloudwatch/cluster-info as a second brittle dependency and say cluster name should be best-effort instead of fatal. I agree with that, but what's the proposed alternative for actually resolving the cluster name? As far as I can tell there isn't one that doesn't require either additional RBAC or calling the AWS API (DescribeCluster, IMDS, etc.). If you have one I'm all ears, but listing it as a problem without a path forward doesn't move things along.

I don't mean this harshly, I think we're aligned on the direction (drop aws-auth, use a non-privileged signal, don't fail hard on missing cluster name). I just want to make sure we're moving toward a PR rather than restating the problem.

[Aminkbi]

Fair pushback, and I think you’re right to ask for a more concrete PR direction.

On the issuer format: there is now an AWS-documented second EKS OIDC issuer family. In the EKS document history, AWS says on August 1, 2024:

“New clusters have a new dual-stack OIDC issuer URL (oidc-eks.region.api.aws).”
Source: Amazon EKS document history (https://docs.aws.amazon.com/eks/latest/userguide/doc-history.html)

AWS’s general reference also lists:

“IRSA OIDC Issuer URLs”
oidc-eks.region.api.aws
Source: Amazon EKS endpoints and quotas (https://docs.aws.amazon.com/general/latest/gr/eks.html)

So I think there is a concrete reason not to use something like strings.Contains(iss, "oidc.eks."): that would be too narrow and would miss at least some newer AWS-documented EKS issuers.

To be clear, I’m not claiming the service-account token issuer is a perfect, formally proven EKS detector on its own. What I’m saying is that it seems like a much better non-privileged signal than kube-system/aws-auth, and it is
grounded in how Kubernetes projected service-account tokens work plus AWS’s documented EKS issuer families:

• Kubernetes service account token issuer claim (iss): ServiceAccount token projection (https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#serviceaccount-token-volume-projection)
• AWS EKS OIDC / IRSA background: IAM roles for service accounts (https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html)

So my concrete suggestion for detection would be:

1. stop using kube-system/aws-auth as the EKS detection signal
2. read the projected service-account token issuer (iss) instead
3. parse it as a URL and accept both AWS-documented EKS issuer families:
   • oidc.eks..amazonaws.com
   • oidc-eks..api.aws
4. validate the expected /id/ path shape instead of doing a substring match

On cluster name: I agree with you, I dont currently know of a generic zero-extra-permission replacement for k8s.cluster.name. So I’m not proposing to solve that in the same PR.

What I am proposing is to separate detection from enrichment.

Today the detector still:

• uses kube-system/aws-auth for EKS detection in isEKS (https://github.com/open-telemetry/opentelemetry-go-contrib/blob/main/detectors/aws/eks/detector.go)
• and then hard-fails cluster-name lookup via amazon-cloudwatch/cluster-info in the same detector path

That second dependency is also brittle, because amazon-cloudwatch/cluster-info is part of CloudWatch / Container Insights setup, not a universal EKS primitive.

So the bugfix value of the first PR would be:

• normal pods no longer need access to aws-auth
• clusters using access entries / no aws-auth still detect as EKS
• missing cluster-info no longer fails the whole detector
• partial EKS resource attrs can still be returned when cluster name isn’t available

Tests I had in mind:

• non-EKS issuer => empty resource
• legacy EKS issuer => detect EKS
• new dual-stack EKS issuer => detect EKS
• aws-auth forbidden / missing => no longer breaks detection
• cluster-info missing / forbidden => still returns partial EKS resource instead of an error