Built-in relay fan-out multiplies publisher frames by player count enabling CPU exhaustion DoS

[cursor]

Summary

Severity: Medium

Location: src/server/mod.rs

Attacker

Any unauthenticated network peer that can open RTMP connections to a server using the built-in media relay (for example the default examples/minimal_server or librtmp2-server without custom on_publish_cb / on_play_cb authorization).

Controlled input

• Number of simultaneous play connections on the same (app, stream_name) route (up to max_connections, default 256 via FFI zero-init).
• Publisher media frame rate and pending_relay depth (up to 1024 frames / 8 MiB per publisher).

Attack path

1. Attacker opens one publisher connection and publishes to live/stream.
2. Attacker opens many player connections (e.g. 255) that all play the same route.
3. Publisher floods audio/video frames, filling Conn::pending_relay (capped at 1024 frames).
4. On each Server::process_connections() poll tick, the server drains all queued relay frames and, for each frame, iterates every playing connection and calls Conn::send_frame() (chunk encode + send_buffer append).
5. Worst case per poll: 1024 frames × 255 players ≈ 261,120 send_frame calls in the single-threaded poll loop, starving all other sessions until the batch completes.

Impact

CPU exhaustion / event-loop monopolization on single-threaded embedders. Other inbound connections cannot be accepted, read, or flushed promptly during the relay burst. This is distinct from issue #41 (publisher-side pending_relay memory) — the amplification here is the player fan-out multiplier in Server::process_connections.

Evidence

for frame in &relay_frames {
    for conn in self.connections.iter_mut() {
        // ...
        conn.send_frame(frame.frame_type, frame.timestamp, &frame.payload)?;
    }
}

No per-poll budget existed on total relay send_frame calls before this report.

Remediation

Cap total relay send_frame deliveries per process_connections pass (e.g. MAX_RELAY_SENDS_PER_POLL) and re-queue deferred frames on the publisher connection for the next poll tick. A fix is proposed on branch cursor/application-security-review-a58c.