Fix flaky TestTCPCollectingProcess_ConcurrentClient - #486
Merged
Conversation
The test stopped the collector from inside one of the client goroutines, as soon as that goroutine observed at least 2 sessions. Nothing guaranteed that the other client had connected by then, so the collector's listener could be closed before the other client dialed, and the test failed with "Cannot establish connection". The session count is not a reliable signal for this, as it also includes the connection that waitForCollectorReady opens to probe the collector. Instead, stop the collector when the test returns, once all the clients are known to be connected. The clients still connect concurrently, which is the point of the test. The connections are also kept open and referenced until the end of the test: they used to be discarded right after being established, which allowed the garbage collector to close them at any point. Signed-off-by: Antonin Bas <antonin.bas@gmail.com>
waitForCollectorReady used to dial the collector to determine whether it was ready. For TCP, such a connection is registered as a session by the accept loop, and is only unregistered asynchronously once it has been closed, so it can be observed by tests which assert on the collector's sessions. For UDP, it did not prove anything at all, as dialing a UDP socket is a purely local operation. Both the TCP and the UDP code paths bind the socket before publishing the collector's address, so waiting for that address is enough, and has no side effect on the collector. Because it no longer needs to wait for a connection to be accepted, the helper can also poll more frequently and return as soon as the collector is ready. With this source of interference gone, the concurrent client test can assert that the collector has a session for each client, and no other session, instead of asserting on the number of sessions only. Signed-off-by: Antonin Bas <antonin.bas@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TestTCPCollectingProcess_ConcurrentClientfails intermittently in CI on main, e.g.:Cause
The test stopped the collector from inside one of the client goroutines, as soon as that goroutine observed at least 2 sessions. Nothing guaranteed that the other client had connected by then, so the listener could be closed before the other client dialed.
The session count is not a reliable signal for this:
waitForCollectorReadydials the collector to probe for readiness, and for TCP that probe connection is itself registered as a session by the accept loop, and is only unregistered asynchronously after it is closed. Both goroutines run a probe, so the>= 2condition could be satisfied by the two probe connections alone, before either client had connected.Changes
Split into two commits:
Fix the flake. Stop the collector when the test returns, once all clients are known to be connected, rather than from a client goroutine. The clients still connect concurrently, which is the point of the test. The connections are also kept open and referenced until the end of the test — they used to be discarded immediately, allowing the garbage collector to close them at any point.
Stop dialing the collector in
waitForCollectorReady. Both the TCP and UDP code paths bind the socket before publishing the collector's address, so waiting for that address is sufficient and has no side effect on the collector. For UDP the old probe never proved anything anyway, since dialing a UDP socket is a purely local operation. With the interference gone, the test now asserts that the collector has a session for each client and no other session, instead of asserting on the count only.Testing
-race -cpu=1, parallel runs).go test -race ./...andgo vet -tags integration ./...are clean.As a side effect, the tests get faster, since
waitForCollectorReadyno longer imposes a 100ms minimum delay at each of its 17 call sites. Locally, over 5 runs ofgo test -race ./pkg/collector/, the reported test time goes from 3.78-4.27s on main to 2.16-2.27s. The full unit suite (go test -race ./...) goes from 6.08-6.28s to 5.77-5.88s wall time over 3 runs.Note:
pkg/test/collector_intermediate_test.gohas its own copy ofwaitForCollectorReadywith the same probe-dial pattern. It is left untouched here, as no test in that package asserts on the collector's sessions.