Single-stream h265:// publish negotiates SDP but SFU never forwards RTP (subscriber stays muted)

[aphexcx]

Summary

lk room join --publish h265://host:port (single-stream, no WxH suffix) successfully connects, publishes a track, and the SFU completes SDP negotiation with subscribers — but the SFU never starts forwarding RTP. The subscriber's video track stays muted: true indefinitely and 0 bytes ever arrive at the receiver. The same exact pipeline with h264://host:port works perfectly. Confirmed against livekit-server v1.11.0 and livekit-cli v2.16.2, with Chrome 147 on Mac as subscriber, ARM64 Ubuntu 20.04 as publisher host.

Reproduction

# Bridge an H.265 RTSP source to a TCP socket as raw HEVC bytestream:
ffmpeg -fflags +genpts -rtsp_transport udp -i rtsp://camera/video1 \
  -an -c:v copy -f hevc 'tcp://127.0.0.1:9001?listen=1' &

# Publish the bytestream as a single-stream H.265 track:
LIVEKIT_URL=ws://localhost:7880 \
LIVEKIT_API_KEY=key LIVEKIT_API_SECRET=secret \
  lk room join --identity rtsp-pub --room test \
    --publish 'h265://127.0.0.1:9001' --fps 30

Subscribe via Chrome (>= 136 for native H.265 WebRTC support) using livekit-client. The track will be subscribed (TrackSubscribed fires with codec: video/H265), but no RTP arrives.

What works

• lk ... --publish 'h264://127.0.0.1:9001' (after re-encoding source to H.264) → renders perfectly, 30fps, 0 packet loss, 1.1ms decode time on Mac.
• The exact same SFU forwarding path is used for both codecs (verified in livekit/pkg/sfu/{receiver.go,downtrack.go,forwarder.go} — Forwarder.DetermineCodec handles H264 and H265 the same).

Root cause

The publisher-side classification logic in server-sdk-go at v2/publication.go:482-516 only treats a published codec as "primary" if p.MimeType() ends with the codec name (e.g. h265). Otherwise it falls through into the backup-codec branch, which finds no matching backup track for an H.265-only publish and exits silently — never unblocking RTP forwarding to the subscriber.

MimeType() returns the empty string when TrackInfo.MimeType and TrackInfo.Codecs[].MimeType are not set in the publisher's AddTrackRequest. TrackInfo.MimeType is only populated when SimulcastCodecs was supplied to AddTrackRequest.

In livekit-cli's publishReader (cmd/lk/join.go:393):

pub, err = room.LocalParticipant.PublishTrack(track, &lksdk.TrackPublicationOptions{})

TrackPublicationOptions{} is empty — no WithSimulcast(...), no codec metadata. For H.264 this works because the SDK has H.264 inference paths; for H.265 the SDK requires explicit metadata or the MimeType() check fails.

In createSimulcastVideoTrack (cmd/lk/join.go:480) the multi-stream path correctly sets:

opts = append(opts, lksdk.ReaderTrackWithSampleOptions(lksdk.WithSimulcast("simulcast", &livekit.VideoLayer{
    Quality: quality, Width: urlParts.width, Height: urlParts.height,
})))

…which is why publishing 2-3 H.265 simulcast streams works around this issue.

Diagnostic evidence

Server log on publish (note videoLayerMode: MODE_UNUSED, single layer at HIGH):

mediaTrack published  mime=video/H265  trackInfo={..., codecs: [{mimeType: video/H265,
  layers: [{quality: HIGH, ssrc: ...}], videoLayerMode: MODE_UNUSED}],
  backupCodecPolicy: PREFER_REGRESSION}

Publisher SDK log on subscriber join (note mime: ""):

INFO  handling subscribed quality update  trackID=... mime=""
        subscribedCodecs=[{codec: h265, qualities: [LOW,MED,HIGH]}]
WARN  subscriber requested backup codec but no track found  trackID=... codec=h265

The same mime: "" log + WARN appears for H.264 single-stream too, but for H.264 the SFU forwards anyway. So the WARN itself is noise; the underlying bug only blocks H.265.

Proposed fix

In publishReader (or wherever single-stream H.265 publishes flow), call WithSimulcast(...) with a single layer marked Quality: HIGH (W/H can be 0/0 if not known) so the publisher SDK populates SimulcastCodecs metadata in AddTrackRequest. Alternatively, fix at the SDK level so H.265 single-stream PublishTrack infers the codec MimeType the same way H.264 does.

Happy to send a PR if you'd like — wanted to surface the diagnosis first since it's a multi-layer issue (lk-cli + server-sdk-go publication classification).

Environment

• livekit-server v1.11.0
• livekit-cli v2.16.2 (with PR Add ability to publish h265 bytestream via tcp or unix socket #722's H.265 socket-publish feature)
• livekit-client (browser SDK) v2.5.10
• Chrome 147.0.0 on macOS (native H.265 WebRTC)
• Publisher host: ARM64 Ubuntu 20.04, Go SDK v2.16.2

[aphexcx]

Update: I prototyped a one-liner fix in livekit-cli's publishReader (adding lksdk.WithSimulcast("single", &livekit.VideoLayer{Quality: livekit.VideoQuality_HIGH}) to the existing PublishTrack call). On code review I confirmed it does not actually fix the bug, and would introduce a regression. Sharing the deeper diagnosis here so a real fix can land in the right place:

Why WithSimulcast in the CLI doesn't help:

WithSimulcast only feeds into AddTrackRequest.SimulcastCodecs via PublishSimulcastTrack (server-sdk-go/v2/localparticipant.go:286-305). The PublishTrack path (localparticipant.go:134-170) only populates SimulcastCodecs in its backup-codec branch (lines 158-170). Adding the WithSimulcast option is silently dropped on the floor by PublishTrack. Mime classification at publication.go:78-105 therefore still sees an empty TrackInfo.MimeType for single-layer H.265.

Side effects of the naive CLI fix (don't do this):

• WithSimulcast sets simulcastID at localtrack.go:98-104 and replaces generated track/stream IDs with that string at localtrack.go:119-136. A fixed ID like "single" would collide on the Cid field across a second H.265 publish in the same session.
• Non-empty RID activates the RTP RID extension write path at localtrack.go:364-383, changing RTP header shape vs the existing single-stream H.264 path.

Correct fix location: server-sdk-go, not livekit-cli.

PublishTrack (localparticipant.go:134-170) needs to populate AddTrackRequest.SimulcastCodecs[0].Codec with the track's MIME (and Layers with a single Quality_HIGH entry) when the track has a non-default video codec like H.265. The SDK already documents this dependency in the MimeType() comment at publication.go:78-105 — SimulcastCodecs in AddTrackRequest is what drives the response's codec MIME fallback. Currently only PublishSimulcastTrack honors it.

This would also fix publishFile in livekit-cli (cmd/lk/join.go:277-284) which has the same shape — it calls NewLocalFileTrack then PublishTrack with empty options, and the SDK already recognizes .h265 files at readersampleprovider.go:200-201. Any caller of PublishTrack with H.265 hits this bug; a CLI-only workaround would miss them.

Why H.264 single-stream works despite hitting the same code path: the SDK does not have H.264-specific inference here; the cause is more likely SFU-side default codec preferences (H.264 is in WebRTC's default codec list, H.265 is not), so the publisher SDK's "is this primary or backup?" classifier treats H.264 as primary even with empty MimeType. H.265 needs the explicit metadata to land on the same answer.

A meaningful test for this would live in server-sdk-go and assert that single-layer H.265 via PublishTrack emits AddTrackRequest.SimulcastCodecs[0].Codec == "video/H265". The CLI tests in cmd/lk/join_test.go are correctly scoped to URL parsing and shouldn't try to spin up an SFU.

Happy to send a PR against server-sdk-go if a maintainer can confirm the approach (specifically: should PublishTrack always populate SimulcastCodecs[0] with the track MIME + a HIGH-quality layer, or only when MIME is non-default per a codec preference list?).

[aphexcx]

Fix proposed upstream at the actual root location: livekit/server-sdk-go#901. Once that lands, single-stream H.265 publishing via lk room join --publish h265://... should work end-to-end with no CLI changes needed.