Skip to content

fix(rabbitmq): resolve consumer auto-recovery race condition and stale callbacks on reconnect - #1

Merged
berrytern merged 2 commits into
developfrom
fix/auto-recovery-reconnect
Aug 11, 2026
Merged

berrytern merged 2 commits into
developfrom
fix/auto-recovery-reconnect

Conversation

@berrytern

Copy link
Copy Markdown
Member

Summary of Changes

This PR fixes a critical bug where RabbitMQ consumers (basic_consume) were not auto-registering after a broker disconnection or restart, leaving services in a zombie state (TCP connection established, but 0 consumers listening on queues).

Root Causes Addressed

  1. Concurrency Bug in start_rpc_publisher(): Line 308 of async_channel.py returned await self.rpc_consumer_future instead of await self.rpc_publisher_future when rpc_publisher_starting was True, causing unhandled AttributeError exceptions in the event loop during concurrent handler reconnection.
  2. Timing Disconnect in retry_connection(): Recovery relied on a 1-second call_later timer that ran after on_channel_open had already completed, causing recovery callbacks to remain unexecuted in callbacks.
  3. Queue Datastructure Refactoring (collections.deque): Replaced list with collections.deque using popleft() for $O(1)$ atomic FIFO consumption of pending callbacks, preventing duplicate callback execution and guaranteeing order.
  4. State Management: Ensured self._closing is reset to False on open() and auto-recovery is triggered on on_connection_open whenever backup subscriptions exist.

Test Coverage Added

  • Unit Tests (tests/unit/eventbus/test_auto_recovery.py):
    • test_start_rpc_publisher_concurrency_fix
    • test_process_callbacks_clears_list
    • test_auto_recovery_on_connection_open
  • Integration Tests (tests/integration/async_eventbus/auto_recovery_test.py):
    • test_rpc_provider_auto_recovery_on_connection_drop
    • test_subscribe_auto_recovery_on_connection_drop
    • test_multiple_consecutive_drops_auto_recovery (network flapping edge case)
    • test_multiple_handlers_simultaneous_recovery (concurrent recovery edge case)

All 46 unit tests and 14 integration tests pass successfully.

…ecovery race condition

- Replace list with collections.deque for self.callbacks with popleft() for atomic O(1) FIFO consumption
- Fix start_rpc_publisher returning rpc_consumer_future instead of rpc_publisher_future when rpc_publisher_starting is True
- Ensure auto-recovery triggers on connection open whenever backup subscriptions exist
- Reset self._closing flag when connection open is invoked
- Add unit and integration tests covering RPC provider and subscriber auto-recovery on connection drops, network flapping, and concurrent handlers
@berrytern
berrytern merged commit f280335 into develop Aug 11, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant