fix(scheduler): Fix flaky TestQueueConcurrency deadlock#7496
Merged
yeya24 merged 1 commit intocortexproject:masterfrom May 11, 2026
Merged
fix(scheduler): Fix flaky TestQueueConcurrency deadlock#7496yeya24 merged 1 commit intocortexproject:masterfrom
yeya24 merged 1 commit intocortexproject:masterfrom
Conversation
The test could deadlock because goroutines calling dequeueRequest on a FIFORequestQueue would block indefinitely on an empty channel. This happened when odd multiples of 5 (cnt=5,15,25) raced ahead of the enqueue goroutines, causing the WaitGroup to never complete. Fix by checking queue.length() > 0 before attempting to dequeue, preventing the blocking channel receive on an empty queue. Signed-off-by: Ben Ye <benye@amazon.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does
Fix flaky
TestQueueConcurrencythat could deadlock due to goroutines blocking ondequeueRequestwith an empty channel.Root Cause
FIFORequestQueue.dequeueRequestperforms a blocking channel receive (r := <-f.queue). In the test, goroutines withcnt=5,15,25(odd multiples of 5) calldequeueRequestunconditionally. If they execute before any enqueue goroutines populate the channel, they block forever, and theWaitGroup.Wait()never completes — causing a 30-minute timeout.Fix
Guard the
dequeueRequestcall withqueue.length() > 0to prevent blocking on an empty channel.How it was tested
go test -run TestQueueConcurrency -count=10— 10/10 passgo test -race -run TestQueueConcurrency -count=10— clean