[broadcast] Use non-blocking send in broadcast() so one stalled subscriber cannot freeze the bus - #1094
Conversation
…riber cannot freeze the bus broadcast() sent to each registered channel with a plain, blocking ch <- m. If any one subscriber stops draining its channel - a dead consumer, a panicked goroutine on the other end, anything that just falls behind - the single run() goroutine wedges on that send. Since Register, Unregister and every future Submit all go through that same goroutine (the select in run()), one stuck subscriber freezes delivery to every other subscriber and blocks all Register/Unregister calls, indefinitely. This pattern is inherited as-is from the reference implementation this file's own header cites (github.com/dustin/go-broadcast, broadcaster.go on master as of this writing) - upstream's broadcast() has the same unprotected ch <- m, so this isn't a meshkit-introduced regression, but it is a real gap in an exported, shared-library type. Fix: wrap the send in select with a default case, so a subscriber that isn't ready has its message dropped rather than blocking the shared goroutine for everyone else. Regression test registers a slow (never-drained) and a fast subscriber, submits one message, and asserts the fast subscriber still receives it and that a subsequent Register/Unregister each return within 2s. Verified the test fails on the pre-fix code (fast subscriber times out) and passes after the fix. Signed-off-by: Atishyy27 <sethatishayjain@gmail.com>
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (2)
📝 WalkthroughWalkthroughThe broadcaster now skips messages for subscribers whose channels are not ready. A regression test verifies that a stalled subscriber does not block delivery to other subscribers or concurrent ChangesBroadcast delivery
Estimated code review effort: 2 (Simple) | ~10 minutes Merge Risk: ⚪ Minimal · up to The change prevents a stalled subscriber from freezing shared broadcaster operations and adds focused regression coverage; no actionable merge-blocking risk remains after normal final checks. 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Fixes #1092
What
broadcaster.broadcast()sent to each registered channel with a plain blockingch <- m.Register,UnregisterandSubmitall funnel through the same singlerun()goroutine, so if any one subscriber stops draining its channel, that blocking send wedgesrun()- freezing delivery to every other subscriber and every subsequentRegister/Unregistercall, indefinitely, not just for the stalled one.Fix wraps the send in
selectwith adefaultcase, so an unready subscriber has its message dropped instead of blocking the shared goroutine.Why it isn't a meshkit-only issue, and why fix it anyway
This exact pattern is inherited from the reference implementation this file's own header cites (
github.com/dustin/go-broadcast) - upstream'sbroadcast()has the identical unprotectedch <- mas of today. Not a regression introduced here. But it's a real gap in an exported, shared-libraryBroadcastertype, andmeshery/meshery's server already instantiates one (server/cmd/main.go) - any current or future caller that registers a slow consumer hits this. I checked: the server's GraphQL subscriptions currently use their own separate channels rather thanBroadcaster.Register, so this specific deadlock isn't live in that codepath today, but the exported type still ships the bug.Test
Added
TestBroadcasterSlowSubscriberDoesNotBlockOthers: registers a slow (never-drained) subscriber and a fast one, submits a message, and asserts (a) the fast subscriber still receives it and (b) a follow-upRegisterandUnregistereach return within 2s. Confirmed locally: fails on master (fast subscriber times out waiting for the message), passes with the fix.Metrics
utils/broadcast/broadcaster.go,utils/broadcast/broadcaster_test.gogo build ./...andgo test ./utils/broadcast/...both pass.Summary by CodeRabbit
Bug Fixes
Tests