Skip to content

Fix flaky TestTCPCollectingProcess_ConcurrentClient - #486

Merged
antoninbas merged 2 commits into
vmware:mainfrom
antoninbas:fix-concurrent-client-flake
Sep 2, 2026
Merged

Fix flaky TestTCPCollectingProcess_ConcurrentClient#486
antoninbas merged 2 commits into
vmware:mainfrom
antoninbas:fix-concurrent-client-flake

Conversation

@antoninbas

@antoninbas antoninbas commented Sep 2, 2026

Copy link
Copy Markdown
Member

TestTCPCollectingProcess_ConcurrentClient fails intermittently in CI on main, e.g.:

--- FAIL: TestTCPCollectingProcess_ConcurrentClient (0.10s)
    process_test.go:352: Cannot establish connection to 127.0.0.1:36861

Cause

The test stopped the collector from inside one of the client goroutines, as soon as that goroutine observed at least 2 sessions. Nothing guaranteed that the other client had connected by then, so the listener could be closed before the other client dialed.

The session count is not a reliable signal for this: waitForCollectorReady dials the collector to probe for readiness, and for TCP that probe connection is itself registered as a session by the accept loop, and is only unregistered asynchronously after it is closed. Both goroutines run a probe, so the >= 2 condition could be satisfied by the two probe connections alone, before either client had connected.

Changes

Split into two commits:

  1. Fix the flake. Stop the collector when the test returns, once all clients are known to be connected, rather than from a client goroutine. The clients still connect concurrently, which is the point of the test. The connections are also kept open and referenced until the end of the test — they used to be discarded immediately, allowing the garbage collector to close them at any point.

  2. Stop dialing the collector in waitForCollectorReady. Both the TCP and UDP code paths bind the socket before publishing the collector's address, so waiting for that address is sufficient and has no side effect on the collector. For UDP the old probe never proved anything anyway, since dialing a UDP socket is a purely local operation. With the interference gone, the test now asserts that the collector has a session for each client and no other session, instead of asserting on the count only.

Testing

  • Reproduced the original failure with the exact CI message by stressing the pre-fix test under scheduling pressure (-race -cpu=1, parallel runs).
  • The fixed test passes 900 runs under the same conditions; the whole package passes 120 runs.
  • Verified the new assertion is falsifiable: with a deliberately introduced regression that registers sessions under the wrong key, the new assertion fails while the previous count-based one passes.
  • go test -race ./... and go vet -tags integration ./... are clean.

As a side effect, the tests get faster, since waitForCollectorReady no longer imposes a 100ms minimum delay at each of its 17 call sites. Locally, over 5 runs of go test -race ./pkg/collector/, the reported test time goes from 3.78-4.27s on main to 2.16-2.27s. The full unit suite (go test -race ./...) goes from 6.08-6.28s to 5.77-5.88s wall time over 3 runs.

Note: pkg/test/collector_intermediate_test.go has its own copy of waitForCollectorReady with the same probe-dial pattern. It is left untouched here, as no test in that package asserts on the collector's sessions.

The test stopped the collector from inside one of the client
goroutines, as soon as that goroutine observed at least 2 sessions.
Nothing guaranteed that the other client had connected by then, so the
collector's listener could be closed before the other client dialed,
and the test failed with "Cannot establish connection".

The session count is not a reliable signal for this, as it also
includes the connection that waitForCollectorReady opens to probe the
collector. Instead, stop the collector when the test returns, once all
the clients are known to be connected. The clients still connect
concurrently, which is the point of the test.

The connections are also kept open and referenced until the end of the
test: they used to be discarded right after being established, which
allowed the garbage collector to close them at any point.

Signed-off-by: Antonin Bas <antonin.bas@gmail.com>
waitForCollectorReady used to dial the collector to determine whether
it was ready. For TCP, such a connection is registered as a session by
the accept loop, and is only unregistered asynchronously once it has
been closed, so it can be observed by tests which assert on the
collector's sessions. For UDP, it did not prove anything at all, as
dialing a UDP socket is a purely local operation.

Both the TCP and the UDP code paths bind the socket before publishing
the collector's address, so waiting for that address is enough, and
has no side effect on the collector. Because it no longer needs to
wait for a connection to be accepted, the helper can also poll more
frequently and return as soon as the collector is ready.

With this source of interference gone, the concurrent client test can
assert that the collector has a session for each client, and no other
session, instead of asserting on the number of sessions only.

Signed-off-by: Antonin Bas <antonin.bas@gmail.com>
@antoninbas
antoninbas merged commit 38f3978 into vmware:main Sep 2, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant