fix(rabbitmq): resolve consumer auto-recovery race condition and stale callbacks on reconnect - #1
Merged
Merged
Conversation
…ecovery race condition - Replace list with collections.deque for self.callbacks with popleft() for atomic O(1) FIFO consumption - Fix start_rpc_publisher returning rpc_consumer_future instead of rpc_publisher_future when rpc_publisher_starting is True - Ensure auto-recovery triggers on connection open whenever backup subscriptions exist - Reset self._closing flag when connection open is invoked - Add unit and integration tests covering RPC provider and subscriber auto-recovery on connection drops, network flapping, and concurrent handlers
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary of Changes
This PR fixes a critical bug where RabbitMQ consumers (
basic_consume) were not auto-registering after a broker disconnection or restart, leaving services in a zombie state (TCP connection established, but 0 consumers listening on queues).Root Causes Addressed
start_rpc_publisher(): Line 308 ofasync_channel.pyreturnedawait self.rpc_consumer_futureinstead ofawait self.rpc_publisher_futurewhenrpc_publisher_startingwasTrue, causing unhandledAttributeErrorexceptions in the event loop during concurrent handler reconnection.retry_connection(): Recovery relied on a 1-secondcall_latertimer that ran afteron_channel_openhad already completed, causing recovery callbacks to remain unexecuted incallbacks.collections.deque): Replacedlistwithcollections.dequeusingpopleft()forself._closingis reset toFalseonopen()and auto-recovery is triggered onon_connection_openwhenever backup subscriptions exist.Test Coverage Added
tests/unit/eventbus/test_auto_recovery.py):test_start_rpc_publisher_concurrency_fixtest_process_callbacks_clears_listtest_auto_recovery_on_connection_opentests/integration/async_eventbus/auto_recovery_test.py):test_rpc_provider_auto_recovery_on_connection_droptest_subscribe_auto_recovery_on_connection_droptest_multiple_consecutive_drops_auto_recovery(network flapping edge case)test_multiple_handlers_simultaneous_recovery(concurrent recovery edge case)All 46 unit tests and 14 integration tests pass successfully.