Add AudioConverter to the Blocks API - #1667
Merged
Merged
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/meta-pytorch/torchcodec/1667
Note: Links to docs will display an error until the docs builds have been completed. ⏳ No Failures, 17 PendingAs of commit 1de931d with merge base 78ed5ad ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
meta-cla
Bot
added
the
CLA Signed
NicolasHug
force-pushed
the
audio-blocks-converter
branch
2 times, most recently
from
August 21, 2026 15:58
c39e054 to
e52daf7
Compare
NicolasHug
force-pushed
the
audio-blocks-converter
branch
2 times, most recently
from
August 21, 2026 15:58
d707771 to
f1a0ec5
Compare
NicolasHug
force-pushed
the
audio-blocks-converter
branch
from
August 21, 2026 16:45
f1a0ec5 to
1e3410f
Compare
NicolasHug
force-pushed
the
audio-blocks-converter
branch
2 times, most recently
from
August 21, 2026 16:45
9b9e7d1 to
33fa222
Compare
Pulls two helpers out of code that already exists: - get_swr_output_num_samples_bound(), the av_rescale_rnd() bound that convert_audio_av_frame_samples() computes inline. - swr_convert_to_tensor(), which runs swr_convert() straight into a fresh float32 [num_channels, N] tensor and narrows it to what was actually produced. That's the body of maybe_flush_audio_buffers(), which now calls it and drops from 25 lines to 6. The reason to land this separately is the signature: swr_convert() takes `const uint8_t **in` up to FFmpeg 6 and `const uint8_t * const *in` from 7 on, and only the former converts to both. Taking `const uint8_t* const*` compiles on 7 and 8 and fails on 4/5/6 with "would lose const qualifier", which is easy to miss when developing against FFmpeg 7. Verified with BUILD_AGAINST_ALL_FFMPEG_FROM_S3=1: all of core4 through core9 build.
Completes the audio pipeline: AudioDemuxer -> PacketDecoder -> AudioConverter, turning RawAudioSamples into normalized float32 AudioSamples, optionally resampled and remixed to another channel count. Unlike ColorConverter this block is a stream processor rather than a function of its input, because resampling is an interpolation filter: swresample holds the tail of each frame back until the next arrives. So convert() can return fewer samples than it was given, drain() is needed for the last ones, frames must be fed in order, and reset() is needed after a seek. None of that is true when sample_rate is left unset - format conversion and channel remixing are frame-local - but drain()/reset() are in the documented loop from the start so that adding sample_rate later can't silently truncate anyone's audio. We deliberately don't implement SingleStreamDecoder's pre-roll or its resampling alignment grid. The grid works by *dropping* up to isr/gcd(isr,osr)-1 input samples, which is only safe behind a pre-roll; without one it would eat the caller's own samples (up to ~1s for 44100 -> 16001). Consequence: samples decoded after a seek don't line up bit-for-bit with a whole-file decode. Decoding from the start does, which the tests pin. create_swr_context() grows an overload taking channel counts rather than an AVFrame, and the swr_convert() output-size bound moves into a shared helper.
This reverts commit c39e054.
- get_swr_output_num_samples_bound is now declared and defined by the parent commit, so remove this branch's copy. - swr_convert_to_tensor keeps the parent's `const uint8_t**` parameter, which is what compiles against FFmpeg 4-6 as well as 7+. - PacketDecoder became AudioPacketDecoder in #1666; update the converter tests.
NicolasHug
force-pushed
the
audio-blocks-converter
branch
from
August 24, 2026 09:10
fb5db8b to
4fa95eb
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add
AudioConverterwhich is a stream processor (unlike the ColorConverter, which is stateless and can work on a per-frame basis).AudioConvertercan convert sample rate and number of channels. It has the same API as the PacketDecoder (drain and reset).