[amazon-cloudwatch-agent-test] Add scraper routing integration tests#724
Open
wenegiemepraise wants to merge 5 commits into
Open
[amazon-cloudwatch-agent-test] Add scraper routing integration tests#724wenegiemepraise wants to merge 5 commits into
wenegiemepraise wants to merge 5 commits into
Conversation
Add an integration suite (test/otel/pernode) and Terraform harness (terraform/eks/daemon/otel-pernode) that validate the Target Allocator per-node allocation strategy end to end, plus the zero-step ServiceMonitor/PodMonitor CRD bundling (G1) and TA resilience to missing CRDs (G2). Suite: - per_node_test.go asserts every scraped series is collected by the agent on the scraped pod's own node (target_node == @resource.k8s.node.name) and that the workload spans >= 2 nodes. - crd_bundling_test.go asserts the SM/PM CRDs are served via discovery after a plain chart install (no prometheus-operator prerequisite). - ta_resilience_test.go asserts the TA Deployment is Available with zero container restarts and that it discovers the monitors once the CRDs exist. - resources/workload.yaml: sm-app/pm-app behind a ServiceMonitor and PodMonitor with a target_node relabel, plus a load generator. Harness installs a helm-charts checkout that bundles the CRDs (no separate CRD install step, by design), deploys custom operator and Target Allocator images carrying the per-node + CRD-watch code, forces the per-node strategy on the CR, applies the workload, and runs the suite.
Validated the pernode suite against a live per-node cluster: - Select Target Allocator pods by app.kubernetes.io/name=cloudwatch-agent- target-allocator (the operator labels component as amazon-cloudwatch-agent-target-allocator, so the previous selector matched nothing). - Assert the TA is currently Ready/Running and not in CrashLoopBackOff instead of requiring a lifetime restart count of 0. The readiness/crashloop check still catches a TA that died on a missing CRD, but is portable across reruns on long-lived clusters; the lifetime restart count is now logged informationally (expected 0 only on a freshly provisioned harness cluster).
- Add local_chart_path var to install the chart from a local checkout (skipping the git clone) so unpushed working-tree changes can be exercised. - Raise the helm_release timeout to 900s for the multi-deployment fresh install (operator, agent DaemonSet, Target Allocator, webhook).
Add scraper_routing_test.go to the pernode E2E suite (integration build tag): - TestScraperRoleWiring: the cluster-scraper AmazonCloudWatchAgent CR carries spec.targetAllocator.prometheusCR.scraperRole=cluster-scraper and the per-node agent carries none (default role) -- the annotation-routing partition wiring. - TestClusterScraperTargetAllocatorHealthy: the operator built the cluster-scraper Target Allocator Deployment and it is Available and not crashlooping.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add annotation-based scraper-routing integration tests to the
test/otel/pernodesuite (integrationbuild tag).What
TestScraperRoleWiring: the cluster-scraperAmazonCloudWatchAgentCR carriesspec.targetAllocator.prometheusCR.scraperRole=cluster-scraperand the per-node agent carries none (default role) — the annotation-routing partition wiring.TestClusterScraperTargetAllocatorHealthy: the operator built the cluster-scraper Target Allocator Deployment and it is Available and not crashlooping.Testing
go vet -tags integration ./test/otel/pernode/clean. Run against a cluster with the SR operator + helm PRs deployed:KUBECONFIG=... CLUSTER_NAME=<c> AWS_REGION=<r> go test -tags integration ./test/otel/pernode/ -run 'TestScraperRoleWiring|TestClusterScraperTargetAllocatorHealthy' -vDependencies
Exercises the SR operator (aws/amazon-cloudwatch-agent-operator#399) + helm PRs. Stacks on the E2E suite (#720,
pernode-e2e).