Skip to content

Skip unselected streams during decode - #7

Closed
ksindi wants to merge 1 commit into
mainfrom
feat/skip-unselected-streams
Closed

Skip unselected streams during decode#7
ksindi wants to merge 1 commit into
mainfrom
feat/skip-unselected-streams

Conversation

@ksindi

@ksindi ksindi commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Motivation

Audio-only loads of remote videos (gs://, s3://, http(s)) currently pay for the video track too: decode_media never marks unselected streams as discarded, so FFmpeg demuxes every packet of every stream, and the cloud reader's eager streaming downloads all of their bytes. For data-loading pipelines that only want audio out of large A/V masters, that is wasted CPU and bandwidth.

What changes

1. Unselected streams are discarded at the demuxer (src/decoder/mod.rs)

After stream selection, every stream whose index was not selected is marked AVDISCARD_ALL (new discard_unselected_streams, via rsmpeg's safe streams_mut() + set_discard()). Index-based demuxers — mp4/mov in particular — then skip unselected streams' samples instead of parsing them, and on seekable inputs seek past their chunks instead of reading them. Multiple selected audio streams are respected; the packet loop's drop-unmatched-packets branch stays as defense in depth for demuxers that ignore discard.

2. Cloud reader turns forward skips into stopped downloads (src/decoder/io.rs)

  • The forward-seek threshold (how far past the fetched frontier a seek must land before a new range GET is opened instead of waiting for the existing reader to stream through the gap) was hardcoded at 64 MiB. It is now configurable via AVTENSOR_CLOUD_SEEK_THRESHOLD_BYTES, parsed with the same pattern as AVTENSOR_MAX_CLOUD_OBJECT_BYTES. Default unchanged (64 MiB), so behavior is unchanged unless opted in.
  • When a seek past the threshold spawns a new range reader, existing readers that would keep streaming through the skipped gap are now truncated at the fetched frontier (closest_available_byte + 1) and stop, instead of downloading the skipped bytes to end-of-object. ReadRange.end became a shared Arc<AtomicUsize>; cloud_asset_reader checks it per chunk. The truncation decision is a pure function (reader_truncations) with unit tests.
  • Backward seeks into a truncated gap still work: after truncation the reader no longer covers the gap, so both the seek callback's "live reader covers it" check and the read callback's stall-recovery see no covering reader and spawn a fresh range reader at the requested position — the same path that already handles dead readers.

Expected impact (honest version)

  • CPU savings everywhere: unselected packets are no longer parsed/copied through the demuxer on any input path (local, bytes, http, gs/s3).
  • Real byte savings on the native http(s) path and on containers whose layout lets the demuxer seek across large spans (faststart MP4s with large chunks, non-interleaved files): FFmpeg's own protocol layer converts the skips into range requests immediately.
  • Bounded savings on finely-interleaved MP4s over gs:///s3:// by default: skips smaller than the seek threshold are still streamed through by the eager reader (existing behavior, deliberately kept). Lowering AVTENSOR_CLOUD_SEEK_THRESHOLD_BYTES trades more range GETs for fewer skipped bytes downloaded; each request has a fixed cost, so the right value depends on chunk sizes.
  • Memory unchanged: the cloud reader still allocates the full-object buffer up front (existing behavior; a lazy buffer redesign is out of scope here).

Testing

Ran locally against FFmpeg 7.1.1 (shared) + torch 2.11 on Linux x86_64:

  • make test — 74 passed, 0 failed, 3 ignored (NVDEC, no GPU)
  • cargo fmt --check, cargo clippy --no-default-features --all-targets -- -D warnings — clean
  • cargo llvm-cov line coverage 67.7% (floor 65%)

New tests: audio-only decode produces bit-identical samples to the audio stream of a full A/V decode; unselected streams carry AVDISCARD_ALL after setup while selected ones do not; reader_truncations edge cases; seek-threshold env var (default/valid/invalid). Not covered locally: end-to-end gs:///s3:// byte-savings measurements (no cloud test infra in this repo — end-to-end cloud tests live in the consuming project); the wheel build and Python-layer checks are left to CI.

decode_media now marks every stream that was not selected for decoding
with AVDISCARD_ALL after stream selection, so the demuxer skips their
packets instead of parsing them — and index-based demuxers (mp4/mov)
skip reading their chunks entirely on seekable inputs. The packet loop
still drops packets without a matching filter context, as defense in
depth for demuxers that ignore the discard flag.

To let the forward skips become bandwidth savings on gs://s3:// inputs,
the cloud reader's forward-seek threshold (previously hardcoded to
64 MiB) is now configurable via AVTENSOR_CLOUD_SEEK_THRESHOLD_BYTES,
and when a seek past the threshold spawns a new range request, readers
that would keep streaming through the skipped gap are truncated at the
fetched frontier (via a shared atomic range end) so they stop instead
of downloading bytes FFmpeg skipped over.
@ksindi ksindi closed this Aug 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant