Retry AWS SSI across zones

[nccatoni]

Motivation

AWS SSI runs can fail intermittently when EC2 has no capacity for the requested instance type in the randomly selected availability zone. This was observed for AlmaLinux 8 arm64 with t4g.medium capacity in us-east-1c, forcing an expensive full CI retry even when capacity is available in another configured zone.

Changes

• Normalize comma-separated subnet and security-group configuration so multiple network IDs are handled reliably.
• Shuffle configured subnets once per run and, on InsufficientInstanceCapacity, retry the Pulumi update with each remaining subnet at most once.
• Preserve the existing same-subnet retry behavior for IdempotentParameterMismatch and keep exhausted capacity errors classified for the existing CI retry fallback.
• Document that AWS SSI should be configured with subnet IDs from multiple availability zones.

Testing

• Added focused tests covering multi-subnet normalization, successful zone rotation after repeated capacity failures, exhaustion after all zones, and unchanged idempotency retry behavior.
• DD_SITE=datadoghq.com venv/bin/python -m pytest --confcutdir=tests/test_the_test tests/test_the_test/test_aws_provider.py -q: 4 passed.
• DD_SITE=datadoghq.com ./format.sh --check passed mypy, Ruff, whitespace, yamlfmt, yamllint, manifest validation, AI Guard fixture validation, and shellcheck. The final Node.js linter image pull could not run because the sandbox proxy returned Bad Gateway for Docker Hub.

Workflow

1. ⚠️ Create your PR as draft ⚠️
2. Work on you PR until the CI passes
3. Mark it as ready for review
   • Test logic is modified? -> Get a review from RFC owner.
   • Framework is modified, or non obvious usage of it -> get a review from R&P team

🚀 Once your PR is reviewed and the CI green, you can merge it!

🛟 #apm-shared-testing 🛟

Reviewer checklist

• Anything but tests/ or manifests/ is modified ? I have the approval from R&P team
• A docker base image is modified?
  • the relevant build-XXX-image label is present
• A scenario is added, removed or renamed?
  • Get a review from R&P team

---

PR by Bits - View session in Datadog

Comment @DataDog to request changes

[datadog-datadog-prod-us1]

View session in Datadog

Bits Code status: ✅ Done

CI Auto-fix: Disabled | Enable

Comment @DataDog to request changes

[datadog-datadog-prod-us1]

I can only run on private repositories.

[github-actions]

CODEOWNERS have been resolved as:

tests/test_the_test/test_aws_provider.py                                @DataDog/system-tests-reviewers
docs/understand/scenarios/onboarding.md                                 @DataDog/system-tests-core
utils/virtual_machine/aws_infra_exceptions.json                         @DataDog/system-tests-core
utils/virtual_machine/aws_provider.py                                   @DataDog/system-tests-core
utils/virtual_machine/virtual_machines.py                               @DataDog/system-tests-core