fix(test): OTel v0.150 integ compatibility (SDKv2, kubeletstats, journald)#719
Merged
Conversation
The journald units subtest asserts on entries a provisioner-run oneshot service emits ~9 min before the agent starts. Contrib v0.149.0 made the stanza journald input pass --lines=0 for the default start_at: end, dropping the backlog that v0.124's implicit --lines=10 incidentally surfaced -- so on al2023 (newer journalctl) the journald-units stream is never created and GetLogEvents fails with ResourceNotFoundException. Re-trigger cwagent-unit-test.service after the agent initializes so fresh entries land while the receiver is following, matching the priority/regex subtests. Keeps start_at: end (the correct default; avoids backfilling the whole journal on install).
…-sdk-go-v2 Two integ checks asserted on aws-sdk-go v1 artifacts that changed under the SDK v2 migration. Credentials resolve correctly in both cases (metrics publish, the access-key-id matches), so these are test-side, not agent regressions. - assume_role confused_deputy_headers: the log scan keyed on v1 wire-debug markers (---[ REQUEST POST-SIGN ]---). Under v2 the STS client logs via smithy-go's RequestResponseLogger, which emits no such markers. Parse both formats so the X-Amz-Source-Account / X-Amz-Source-Arn headers are still detected (with a unit test covering both). - credential_chain CommonConfigTest: expected provider name SharedCredentialsProvider (v1). Under v2 shared credentials resolve via config.LoadSharedConfigProfile and report SharedConfigCredentials. Update the expectation to match, consistent with HomeEnvTest and the fork's provider.
sky333999
force-pushed
the
otel-bump-v0.150.0
branch
from
July 8, 2026 22:04
df4be61 to
50f14c6
Compare
sky333999
force-pushed
the
otel-bump-v0.150.0
branch
from
July 8, 2026 22:06
50f14c6 to
1687aa6
Compare
…g format Two app_signals_service_events assertions broke on brittle log-string matching, not agent behavior (export + delivery are verified by the sibling subtests): - service_events_sent_to_otlp_monitoring_endpoint: collector-core v0.125.0 injects otelcol.component.id/kind fields between the exporter's "Preparing to make HTTP request" msg and its "url", breaking the contiguous-substring match. Use a same-line regexp instead. - TestAppSignalsOnPremCredentialsStartup: drop the assertion on "no EC2 IMDS role found" -- an aws-sdk-go-v2-specific string emitted by entitystore's benign best-effort IMDS probe (v1 wording differed). sigv4auth resolving from the credentials file is still covered by the surviving assertion + E2E delivery with IMDS disabled.
…0.150
kubeletstatsreceiver deprecated the *.cpu.utilization metrics (default-off from
contrib v0.125.0) and removed k8s.node.cpu.utilization, k8s.pod.cpu.utilization
and container.cpu.utilization by v0.136.0, replaced by *.cpu.usage (unit {cpu}).
The bump to v0.150 crosses both. The deployed container-insights kubeletstats
config already emits *.cpu.usage, so update the EKS otel test expectations
(standard, attr_limit) to match, and refresh the stale entries in the
gpu/neuron/efa metric tables (those suites don't assert them today but
shouldn't carry deleted metric names).
…ent warmup The kube_node_* host attributes (host.id / host.image.id / cloud.availability_zone) are added by the agent's Lease-based enrichment, which is eventually consistent (cold for the first few minutes after start, then warm). The suite's QueryCache queries each metric once and caches it, so a single early cold read poisoned every TestKSM_NodeBucket_* assertion. Add a TestMain readiness gate that polls kube_node_info (non-cached) until all nodes are enriched before any cached query is issued. Best-effort (logs + proceeds on timeout); assertions unchanged.
liscsi validated with a single one-shot metric fetch and no retry, so a slow node_diskio_instance_store_* propagation window (e.g. ec2_instance_performance_exceeded_*) caused hard failures (353 vs 1076 metrics on the same commit). Wrap go test in the same 3x/60s retry loop efa uses; required metrics unchanged.
movence
approved these changes
Jul 15, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Test-side fixes for the integration tests that regressed with the CloudWatch Agent OpenTelemetry
contrib v0.124.1 → v0.150.0 bump (which also folds in the AWS SDK Go v1 → v2 migration and advances
the amazon-contributing fork pin). In every case the agent behaves correctly — the tests asserted on
v1-era log formats / provider names, on metric names upstream renamed, or on timing/backlog behavior that
upstream changes intentionally altered. No agent-code change is required; these are test-only fixes
(plus test-timing hardening for two pre-existing flakes surfaced by the full matrix run).
Changes
fix(journald): re-trigger unit-test service after agent startTestJournaldUnitsLogsvalidated capture of entries emitted by a provisioner-run oneshot service ~9 minbefore the agent starts. Upstream contrib v0.149.0 (PR #46556, "Fix emitting of historical entries on
startup") made the stanza journald input pass
--lines=0for the defaultstart_at: end, removing theincidental
--lines=10backlog that v0.124's--followimplied. On al2023 (newer journalctl) thejournald-unitsstream is therefore never created and the check fails withResourceNotFoundException(al2 passed only via a journalctl off-by-one). Fix: re-trigger
cwagent-unit-test.serviceafter agentinit so fresh entries land while the receiver is following — consistent with the priority/regex subtests.
start_at: endis kept (correct production default; avoids backfilling the whole journal to CloudWatch oninstall).
fix(test): align assume_role and credential_chain assertions with aws-sdk-go-v2confused_deputy_headers— the log scan keyed on aws-sdk-go v1 wire-debug markers(
---[ REQUEST POST-SIGN ]---). Under v2 the STS client logs via smithy-go'sRequestResponseLogger,which emits no such markers, so the scan always failed even though the
X-Amz-Source-Account/X-Amz-Source-Arnheaders are sent and enforced (proven by the positive tests publishing metrics and thenegative tests getting STS
AccessDenied). Fix: parse both v1 and v2 log formats (with a new unit testcovering both).
CommonConfigTest— expected provider nameSharedCredentialsProvider(v1). Underv2, shared credentials resolve via
config.LoadSharedConfigProfileand reportSharedConfigCredentials(the accurate source name, matching
HomeEnvTestand the fork's reference provider). Credentials stillresolve correctly (metric delivery + access-key-id match unchanged). Fix: update the expected provider
name.
fix(test): align app_signals service-events assertions with v0.150 log formatTwo
app_signals_service_eventsassertions broke on brittle log-string matching, not agent behavior(export + delivery are verified by the sibling subtests):
service_events_sent_to_otlp_monitoring_endpoint: collector-core v0.125.0 injectsotelcol.component.id/kindfields between the exporter's "Preparing to make HTTP request" msg and its"url", breaking the contiguous-substring match. Use a same-line regexp instead.
TestAppSignalsOnPremCredentialsStartup: drop the assertion on "no EC2 IMDS role found" — anaws-sdk-go-v2-specific string emitted by entitystore's benign best-effort IMDS probe (v1 wording
differed). sigv4auth resolving from the credentials file is still covered by the surviving assertion +
E2E delivery with IMDS disabled.
fix(test): rename kubeletstats *.cpu.utilization to *.cpu.usage for v0.150kubeletstatsreceiver deprecated the
*.cpu.utilizationmetrics (default-off from contrib v0.125.0) andremoved
k8s.node.cpu.utilization,k8s.pod.cpu.utilizationandcontainer.cpu.utilizationbyv0.136.0, replaced by
*.cpu.usage(unit{cpu}). The bump to v0.150 crosses both. The deployedcontainer-insights kubeletstats config already emits
*.cpu.usage, so update the EKS otel test expectations(standard, attr_limit) to match, and refresh the stale entries in the gpu/neuron/efa metric tables (those
suites don't assert them today but shouldn't carry deleted metric names).
fix(test): gate otel/standard KSM assertions on node-metadata enrichment warmupThe
kube_node_*host attributes (host.id/host.image.id/cloud.availability_zone) are added by theagent's Lease-based enrichment, which is eventually consistent (cold for the first few minutes after start,
then warm). The suite's QueryCache queries each metric once and caches it, so a single early cold read
poisoned every
TestKSM_NodeBucket_*assertion. Add aTestMainreadiness gate that pollskube_node_info(non-cached) until all nodes are enriched before any cached query is issued. Best-effort (logs + proceeds on
timeout); assertions unchanged.
fix(test): add validator retry loop to liscsi (match efa)liscsivalidated with a single one-shot metric fetch and no retry, so a slownode_diskio_instance_store_*propagation window caused hard failures (353 vs 1076 metrics on the samecommit). Wrap
go testin the same 3×/60s retry loopefauses; required metrics unchanged.Testing
go vet(with-tags=integration) +gofmtclean across all changed packages; newassume_rolescannerunit test passes for both v1 and v2 log formats.
ab169d5a(OTel v0.150.0, fork pin with the cadvisorfs-plugin fix) with this test branch:
https://github.com/aws/amazon-cloudwatch-agent/actions/runs/29217023388
All fixes above validated green; the only remaining failures were unrelated to this PR (an ITAR
ssm_documentcommand-channel infra issue and a pre-existingecs_service_discoverypropagation flake).License
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this
contribution, under the terms of your choice.