Skip to content

chore(ci): Disable testcontainers ryuk reaper in integration tests - #19604

Merged
skartikey merged 1 commit into
influxdata:masterfrom
skartikey:chore/ci-disable-ryuk-reaper
Sep 1, 2026
Merged

chore(ci): Disable testcontainers ryuk reaper in integration tests#19604
skartikey merged 1 commit into
influxdata:masterfrom
skartikey:chore/ci-disable-ryuk-reaper

Conversation

@skartikey

@skartikey skartikey commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Summary

The test-integration job fails a few times a week with a single test dying on reaper: from container "xxxx": wait for reaper xxxx: context deadline exceeded (recently always TestIntegrationArtemis in jolokia2_agent, e.g. jobs 551595, 551618, 551879, 551886, 551994, 552040). All test package processes of one go test ./... invocation share a single ryuk container, and ryuk exits 10 seconds after the last test process disconnects. In a full ./... run there are natural gaps with no active container tests, so ryuk legitimately dies mid-run; the next package that needs a container then finds the dead ryuk container (testcontainers-go looks it up with All: true and no running-state filter) and spends 60 seconds waiting for it to become ready before failing. The same package keeps getting hit because package scheduling on the fixed runner is roughly deterministic.

Ryuk only exists to clean up leaked containers, and the job runs on an ephemeral CircleCI machine VM that is destroyed after the job, so it provides no value here. Disabling it removes the whole reaper code path and with it this failure mode.

Checklist

Related issues

resolves #19605

@telegraf-tiger telegraf-tiger Bot added the chore label Sep 1, 2026

@srebhan srebhan left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome finding @skartikey! 🚀

@telegraf-tiger

telegraf-tiger Bot commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Download PR build artifacts for linux_amd64.tar.gz, darwin_arm64.tar.gz, and windows_amd64.zip.
Downloads for additional architectures and packages are available below.

⚠️ This pull request increases the Telegraf binary size by 3.07 % for linux amd64 (new size: 317.7 MB, nightly size 308.2 MB)

📦 Click here to get additional PR build artifacts

Artifact URLs

. DEB . RPM . TAR . GZ . ZIP
amd64.deb aarch64.rpm darwin_amd64.tar.gz windows_amd64.zip
arm64.deb armel.rpm darwin_arm64.tar.gz windows_arm64.zip
armel.deb armv6hl.rpm freebsd_amd64.tar.gz windows_i386.zip
armhf.deb i386.rpm freebsd_armv7.tar.gz
i386.deb ppc64le.rpm freebsd_i386.tar.gz
mips.deb riscv64.rpm linux_amd64.tar.gz
mipsel.deb s390x.rpm linux_arm64.tar.gz
ppc64el.deb x86_64.rpm linux_armel.tar.gz
riscv64.deb linux_armhf.tar.gz
s390x.deb linux_i386.tar.gz
linux_mips.tar.gz
linux_mipsel.tar.gz
linux_ppc64le.tar.gz
linux_riscv64.tar.gz
linux_s390x.tar.gz

@srebhan srebhan added the ci CI pipeline fixes, optimizations, and infrastructure label Sep 1, 2026
@skartikey
skartikey merged commit 7f6daa5 into influxdata:master Sep 1, 2026
31 checks passed
@github-actions github-actions Bot added this to the v1.40.0 milestone Sep 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

chore ci CI pipeline fixes, optimizations, and infrastructure

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Flaky test-integration: reaper "context deadline exceeded" kills TestIntegrationArtemis

2 participants