APMSP-3752 Automate SSI Agent updates

[nccatoni]

Motivation

SSI tests intentionally pin the Datadog Agent to protect tracer pipelines from incompatible or faulty releases, but the pin currently requires manual maintenance. A daily, checks-gated update keeps the tested Agent current without reintroducing the risk of an unvalidated latest-version dependency.

Changes

• Add the Agent update automation to the existing nightly GitLab schedule.
• Keep the GitLab job declarative with a single Python invocation.
• Move dd-octo-sts authentication and cleanup, latest stable Agent 7 lookup, pin updates, Git branch refresh, pull request creation/reuse, and squash auto-merge into the Python entry point.
• Use GitHub's API directly so the automation works with the existing CI runner image and does not require publishing a new bootstrap image.
• Update both host-installer and Docker SSI pins through strict, idempotent replacement that rejects prereleases and unexpected file layouts.

Testing

• Added focused coverage for both SSI pin formats, idempotent reruns, release validation, publish/no-op paths, existing and missing pull requests, token cleanup after failure, and empty-token rejection.
• Ran 22 focused updater, scenario-coherence, and CI-orchestrator tests successfully.
• Verified the scenario collector succeeds and assigns the new updater tests to TEST_THE_TEST.
• Mypy, Ruff, yamllint, manifest validation, shellcheck, CI image tag validation, and diff checks passed. The full TEST_THE_TEST run reached 191 passing tests before this environment's unavailable Docker daemon stopped unrelated Docker-dependent tests; the full formatter likewise only stopped when Docker Hub was unavailable for Node.js lint.

Workflow

1. ⚠️ Create your PR as draft ⚠️
2. Work on you PR until the CI passes
3. Mark it as ready for review
   • Test logic is modified? -> Get a review from RFC owner.
   • Framework is modified, or non obvious usage of it -> get a review from R&P team

🚀 Once your PR is reviewed and the CI green, you can merge it!

🛟 #apm-shared-testing 🛟

Reviewer checklist

• Anything but tests/ or manifests/ is modified ? I have the approval from R&P team
• A docker base image is modified?
  • the relevant build-XXX-image label is present
• A scenario is added, removed or renamed?
  • Get a review from R&P team

---

PR by Bits - View session in Datadog

Comment @DataDog to request changes

[datadog-datadog-prod-us1-2]

View session in Datadog

Bits Code status: ✅ Done

CI Auto-fix: Disabled | Enable

Comment @DataDog to request changes

[datadog-datadog-prod-us1]

I can only run on private repositories.

[github-actions]

CODEOWNERS have been resolved as:

.github/chainguard/self.gitlab-update-agent-version.sts.yaml            @DataDog/system-tests-core
tests/test_the_test/test_update_agent_version.py                        @DataDog/system-tests-reviewers
utils/scripts/update_agent_version.py                                   @DataDog/system-tests-core
.gitlab-ci.yml                                                          @DataDog/system-tests-core
utils/scripts/libraries_and_scenarios_rules.yml                         @DataDog/system-tests-core

[datadog-datadog-prod-us1-2]

✅ All CI checks and tests passed.

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 35af397 | Docs | View more details | Give us feedback!}

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-14T16:03:35.987179Z	e1e37ed	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e1e37ed33d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Assign updater tests to TEST_THE_TEST

This module has no @scenarios.test_the_test marker on this or any other test. The collection hook assigns unmarked items to DEFAULT, while the repository-internals workflow runs ./run.sh TEST_THE_TEST, so all of this new updater coverage is deselected from its normal CI check. Add the TEST_THE_TEST scenario marker to these tests. .cursor/rules/test_the_test.mdcL7-L9

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Filter release discovery to Agent 7

The repository-wide /releases/latest endpoint is not major-version aware, so this does not reliably implement a latest-Agent-7 lookup. Once DataDog/datadog-agent marks an 8.x stable release as latest while 7.x is still maintained, normalize_version will raise before inspecting any 7.x release and the nightly job will stop updating these Agent 7 pins. Query the release list and select the newest stable tag matching 7.x instead.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Prevent revocation failures from logging the token

If dd-octo-sts revoke exits nonzero, run_command raises an uncaught CalledProcessError whose message includes the complete argument list, including the write-scoped GitHub token passed after -t. This exposes the token in the GitLab job log precisely when revocation failed and the credential may remain usable; handle revocation errors without propagating the secret-bearing command or otherwise redact the token before logging.

Useful? React with 👍 / 👎.