Skip to content

[broadcast] Use non-blocking send in broadcast() so one stalled subscriber cannot freeze the bus - #1094

Open
Atishyy27 wants to merge 1 commit into
meshery:masterfrom
Atishyy27:fix/broadcaster-blocking-send
Open

[broadcast] Use non-blocking send in broadcast() so one stalled subscriber cannot freeze the bus#1094
Atishyy27 wants to merge 1 commit into
meshery:masterfrom
Atishyy27:fix/broadcaster-blocking-send

Conversation

@Atishyy27

@Atishyy27 Atishyy27 commented Aug 15, 2026

Copy link
Copy Markdown

Fixes #1092

What

broadcaster.broadcast() sent to each registered channel with a plain blocking ch <- m. Register, Unregister and Submit all funnel through the same single run() goroutine, so if any one subscriber stops draining its channel, that blocking send wedges run() - freezing delivery to every other subscriber and every subsequent Register/Unregister call, indefinitely, not just for the stalled one.

Fix wraps the send in select with a default case, so an unready subscriber has its message dropped instead of blocking the shared goroutine.

Why it isn't a meshkit-only issue, and why fix it anyway

This exact pattern is inherited from the reference implementation this file's own header cites (github.com/dustin/go-broadcast) - upstream's broadcast() has the identical unprotected ch <- m as of today. Not a regression introduced here. But it's a real gap in an exported, shared-library Broadcaster type, and meshery/meshery's server already instantiates one (server/cmd/main.go) - any current or future caller that registers a slow consumer hits this. I checked: the server's GraphQL subscriptions currently use their own separate channels rather than Broadcaster.Register, so this specific deadlock isn't live in that codepath today, but the exported type still ships the bug.

Test

Added TestBroadcasterSlowSubscriberDoesNotBlockOthers: registers a slow (never-drained) subscriber and a fast one, submits a message, and asserts (a) the fast subscriber still receives it and (b) a follow-up Register and Unregister each return within 2s. Confirmed locally: fails on master (fast subscriber times out waiting for the message), passes with the fix.

Metrics

  • 2 files changed: utils/broadcast/broadcaster.go, utils/broadcast/broadcaster_test.go
  • Before: one stalled subscriber blocks the entire broadcaster forever. After: it only drops its own message.
  • go build ./... and go test ./utils/broadcast/... both pass.

Summary by CodeRabbit

  • Bug Fixes

    • Improved broadcast responsiveness by preventing slow or inactive subscribers from blocking message delivery.
    • Ensured subscriber registration and removal remain responsive while broadcasts are in progress.
  • Tests

    • Added coverage for delivery to responsive subscribers and concurrent subscriber management.

…riber cannot freeze the bus

broadcast() sent to each registered channel with a plain, blocking
ch <- m. If any one subscriber stops draining its channel - a dead
consumer, a panicked goroutine on the other end, anything that just
falls behind - the single run() goroutine wedges on that send. Since
Register, Unregister and every future Submit all go through that same
goroutine (the select in run()), one stuck subscriber freezes delivery
to every other subscriber and blocks all Register/Unregister calls,
indefinitely.

This pattern is inherited as-is from the reference implementation this
file's own header cites (github.com/dustin/go-broadcast, broadcaster.go
on master as of this writing) - upstream's broadcast() has the same
unprotected ch <- m, so this isn't a meshkit-introduced regression, but
it is a real gap in an exported, shared-library type.

Fix: wrap the send in select with a default case, so a subscriber that
isn't ready has its message dropped rather than blocking the shared
goroutine for everyone else.

Regression test registers a slow (never-drained) and a fast subscriber,
submits one message, and asserts the fast subscriber still receives it
and that a subsequent Register/Unregister each return within 2s. Verified
the test fails on the pre-fix code (fast subscriber times out) and passes
after the fix.

Signed-off-by: Atishyy27 <sethatishayjain@gmail.com>
@coderabbitai

coderabbitai Bot commented Aug 15, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 0819d77c-f454-4649-9e4a-6837861fff31

📥 Commits

Reviewing files that changed from the base of the PR and between cf39c57 and 5047741.

📒 Files selected for processing (2)
  • utils/broadcast/broadcaster.go
  • utils/broadcast/broadcaster_test.go

📝 Walkthrough

Walkthrough

The broadcaster now skips messages for subscribers whose channels are not ready. A regression test verifies that a stalled subscriber does not block delivery to other subscribers or concurrent Register and Unregister operations.

Changes

Broadcast delivery

Layer / File(s) Summary
Non-blocking delivery and regression coverage
utils/broadcast/broadcaster.go, utils/broadcast/broadcaster_test.go
broadcast uses a non-blocking channel send. The test verifies continued delivery and timeout-bounded subscriber management with a non-draining subscriber.

Estimated code review effort: 2 (Simple) | ~10 minutes

Merge Risk: ⚪ Minimal · up to 50477

The change prevents a stalled subscriber from freezing shared broadcaster operations and adds focused regression coverage; no actionable merge-blocking risk remains after normal final checks.

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely describes the non-blocking broadcast change and its purpose.
Linked Issues check ✅ Passed The implementation prevents stalled subscribers from blocking the broadcaster, and the regression test covers delivery and management operations [#1092].
Out of Scope Changes check ✅ Passed All changes are limited to the broadcaster fix and its targeted regression test, which directly support issue #1092.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

broadcaster.broadcast() blocking send lets one stalled subscriber freeze Register/Unregister for everyone

1 participant