Fix keep-alive deadlock: wait for a signal instead of bare select {}

[gsi85]

📝 Description

When -exit-after-warmup is not set (the default), block() kept the process alive by parking the main goroutine on a bare select {}. This is not a run-forever primitive: after warm-up completes, every other goroutine winds down, leaving the empty select as the only remaining goroutine. Go's runtime then aborts the process with fatal error: all goroutines are asleep - deadlock! the moment it goes idle — crash-looping the container even though it has already written its probe files. This reproduces on recent Go toolchains and matches the stack trace in #366 (cmd/root.go, block()); older toolchains only lengthened the fuse.

This replaces the bare select with a wait on SIGINT/SIGTERM. A signal receive is externally wakeable, so the deadlock detector never fires, and the process now shuts down gracefully when the orchestrator sends a termination signal. The -exit-after-warmup behaviour is unchanged.

Reproduce (before this change): build with a recent Go toolchain and run so that no warm-up requests are sent, e.g.

CGO_ENABLED=0 go build -o mittens . && ./mittens -max-readiness-wait-seconds 1

→ fatal error: all goroutines are asleep - deadlock!. After this change the process stays up and exits cleanly on Ctrl-C / SIGTERM.

New unit tests cover the -exit-after-warmup short-circuit and that the keep-alive wait unblocks on a signal. go vet ./..., go build, and go test ./... all pass locally.

⚠️ Why this surfaces now

The deadlock is only armed when Mittens is built with Go 1.22 or newer: a runtime scheduler change in 1.22 lets the deadlock detector fire once the process is genuinely idle. Earlier Go versions kept a background thread alive that masked it — which is why published release images (built with older Go) have not shown it, and why #366 could sit un-triaged for so long.

Crucially, main already declares go 1.24.0 in go.mod (bumped via #389), and that has not yet been in a tagged release. So a build from current main — i.e. the next official release — will ship this crash by default. Landing the fix before that release avoids regressing every consumer the moment they pick up the new version.

🔗 Related Issues

Fixes #366

[sumitanvekar]

Could you test this in a kubernetes environment and confirm the behaviour is as expected?

[gsi85]

Added TestKeepAliveExitsGracefullyOnSigterm (1ebb539) that reproduces the pod scenario at process level: it runs the built binary with the target never becoming ready (so warm-up sends nothing and the process goes idle — the exact condition that used to deadlock), asserts the process stays up, then sends SIGTERM (what kubelet delivers on pod termination) and asserts a clean exit (status 0). Before the fix that idle state aborts with all goroutines are asleep - deadlock!; after it, the container stays up and exits 0 on SIGTERM. Runs in ~3.5s locally and in CI.

We've also already validated the behaviour on live pods in our own Kubernetes environment. Happy to go into more detail over internal channels if that would help.

[sumitanvekar]

could you move these tests into test/root_test.go. I know there other test files in the same package and we will clean them up too.

[gsi85]

Done in 1ebb539 — moved into test/root_test.go as TestKeepAliveExitsGracefullyOnSigterm. The old cmd tests reached unexported helpers (block/opts), so rather than move them verbatim I reworked them into a black-box test that builds and runs the binary and drives the real signal path, and removed cmd/root_test.go. I also simplified block() back to the inline signal wait since the test seam is no longer needed.