Windows Arm64 native support: FFmpeg/libavif builders (hardware-verified on windows-11-arm) - #1673
Open
yeelam-gordon wants to merge 9 commits into
Open
Windows Arm64 native support: FFmpeg/libavif builders (hardware-verified on windows-11-arm)#1673yeelam-gordon wants to merge 9 commits into
yeelam-gordon wants to merge 9 commits into
Conversation
torchcodec currently ships no Windows ARM64 wheel at all (PyPI only has win_amd64, manylinux x86_64/aarch64, and macOS arm64 for 0.16.0); see meta-pytorch#1672. windows_wheel.yaml can't target win_arm64 yet because its two native build-time dependencies, a custom LGPL-only FFmpeg and a decode-only libavif, are only ever built for windows_x86_64 and published to S3 (BUILD_AGAINST_ALL_FFMPEG_FROM_S3); build_ffmpeg.bat / build_libavif.bat hardcode MSYS2's MINGW64 subsystem, which only ever targets x86_64. This adds LGPL-Windows-arm64 / libavif-Windows-arm64 jobs that build both dependencies natively using MSYS2's CLANGARM64 subsystem (native aarch64 clang toolchain) on GitHub's standard `windows-11-arm` hosted runner (GA for public repos since 2025-08-07). Neither build_ffmpeg.sh nor build_libavif.sh needed changes: both already gate x86-only assembly behind arch checks (`--disable-x86asm`/`--disable-asm` for FFmpeg; nasm only for x86_64/i686, `neon` SIMD-symbol verification for aarch64/arm64 in build_libavif.sh), so they work unmodified on a native aarch64 host. These jobs deliberately do NOT use pytorch/test-infra's windows_job.yml (unlike the existing x86_64 jobs in the same files): that reusable workflow's "Run script" step hardcodes paths from test-infra's own pre-provisioned x86_64 AMI (e.g. /c/Jenkins/Miniconda3, a custom setup-windows action) which do not exist on the generic windows-11-arm image. This mirrors the precedent in pytorch/pytorch's own win-arm64-build-test.yml, which similarly bypasses test-infra's reusable workflows in favor of an explicit bootstrap sequence for the same reason. Scope and remaining blockers (see Generated Files/implementation-status.md for the full writeup): - These jobs only produce a build artifact. Per this workflow's own header comment, uploading it to the S3 bucket consumed by BUILD_AGAINST_ALL_FFMPEG_FROM_S3 is a manual, maintainer-only step done once per FFmpeg/libavif release; this PR cannot and does not do that. - windows_wheel.yaml's arm64 wheel-build job (mirroring pytorch/vision's build_wheel_windows_arm64.yml, which already produces real win_arm64 wheels via this same test-infra build_wheels_windows.yml reusable workflow) is intentionally NOT added yet: it depends on the S3 artifacts above, and on libheif, which conda-forge does not yet publish for win-arm64 (libjpeg-turbo, libpng and libwebp are already available there). Wiring the wheel job before then would add a workflow guaranteed to fail on first run. Test Plan: python -c "import yaml; yaml.safe_load(open('.github/workflows/build_ffmpeg.yaml')); yaml.safe_load(open('.github/workflows/build_libavif.yaml'))" (manual review of the new .bat scripts against their x86_64 counterparts; no Windows ARM64 runner available in this environment to execute the build end to end) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Both new arm64 builder jobs failed on their first real dispatch on
GitHub's windows-11-arm runner (not caught by static py_compile/yaml
checks, since these are runtime toolchain-detection bugs):
1. build_libavif.sh: uname -m under MSYS2 always reports x86_64 (MSYS2's
own POSIX runtime has no native aarch64 build), even inside the
CLANGARM64 subsystem. This made the script wrongly require nasm (an
x86-only dependency the arm64 job intentionally does not install),
which then hit an unimplemented Windows case in the micromamba
toolchain-provisioning fallback and hard-failed with:
ERROR: no toolchain provisioning for MINGW64_NT-10.0-26200-ARM64-x86_64
Fixed by preferring $MSYSTEM (CLANGARM64 => arm64) over uname -m when
set; behavior on Linux/macOS/existing Windows x86_64 job is unchanged.
2. build_ffmpeg_arm64.bat: FFmpeg's ./configure defaults to probing for
"gcc", but MSYS2's CLANGARM64 subsystem's
mingw-w64-clang-aarch64-toolchain only provides clang/clang++, so it
failed with "gcc is unable to create an executable file." Fixed by
exporting CC=clang CXX=clang++ before invoking build_ffmpeg.sh, scoped
to only the arm64 job (the x86_64 job's MINGW64 toolchain does ship a
real gcc and is untouched).
Test plan: re-dispatched both workflows on a fork
(yeelam-gordon/torchcodec) via workflow_dispatch on real windows-11-arm
GitHub-hosted runners after this fix; see hackathon demo evidence for
the resulting run URLs and pass/fail outcome.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The previous commit (711ff55) tried to fix "gcc is unable to create an executable file" on the arm64 job by exporting CC=clang/CXX=clang++ as environment variables before invoking build_ffmpeg.sh. Re-dispatching on a real windows-11-arm runner after that fix proved it did not work: the job failed again with the exact same "gcc is unable to create an executable file" error. Root cause, confirmed by fetching FFmpeg n4.4.4's actual configure script: it hardcodes `cc_default="gcc"` / `cxx_default="g++"` and never reads $CC/$CXX from the environment at all -- only the explicit --cc=/--cxx= command-line flags override it (confirmed via `CC=$cc` / `CXX=$cxx` assignment happening from the internal $cc/$cxx variables, which are only set from cc_default or --cc=, never from env). Fix: build_ffmpeg.sh now detects MSYSTEM=CLANGARM64 (the subsystem used by the new native Windows Arm64 builder job) and passes --cc=clang --cxx=clang++ explicitly to ./configure. The previous, ineffective CC/CXX env-var export in build_ffmpeg_arm64.bat is removed. Every other platform (Linux, macOS, existing Windows x86_64 MINGW64 job) is unaffected, since MSYSTEM is unset/different there. Test plan: re-dispatched "Build non-GPL FFmpeg from source" on yeelam-gordon/torchcodec (fork) via workflow_dispatch on a real windows-11-arm GitHub-hosted runner. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…to clang) Second real-hardware failure after the --cc=clang fix (22993a4): the build now got past ./configure but failed compiling with clang errors inside MSVC's own UCRT headers, e.g.: C:\Program Files (x86)\Windows Kits\10\include\...\ucrt\stdlib.h:1184:28: error: expected identifier or '(' Root cause: build_ffmpeg_arm64.bat routed the build through packaging/vc_env_helper_arm64.bat, which calls `vcvarsall.bat arm64` and sets INCLUDE/LIB to MSVC's own Windows SDK/UCRT paths. MSYS2's CLANGARM64 clang (unlike gcc, which ignores %INCLUDE%) picks those up and mixes MSVC's UCRT headers with its own mingw-w64 sysroot headers, which are not compatible and fail to parse. This never affected the existing x86_64 MINGW64 job (gcc doesn't read %INCLUDE%), and never affected the new libavif-Windows-arm64 job (build_libavif_arm64.bat never called vc_env_helper in the first place -- confirmed passing on real hardware). Fix: build_ffmpeg_arm64.bat now invokes build_ffmpeg.sh directly, without vc_env_helper_arm64.bat, matching the already-working build_libavif_arm64.bat pattern. vc_env_helper_arm64.bat itself is kept (not deleted) for the future torchcodec Arm64 wheel-build job, which will need MSVC's cl.exe/link.exe on PATH to build the actual C++ extension -- its docstring now says so explicitly. Test plan: re-dispatched "Build non-GPL FFmpeg from source" on yeelam-gordon/torchcodec (fork) via workflow_dispatch on a real windows-11-arm GitHub-hosted runner. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/meta-pytorch/torchcodec/1673
Note: Links to docs will display an error until the docs builds have been completed. This comment was automatically generated by Dr. CI and updates every 15 minutes. |
|
Hi @yeelam-gordon! Thank you for your pull request and welcome to our community. Action RequiredIn order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you. ProcessIn order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA. Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with If you have received this in error or have any questions, please contact us at cla@meta.com. Thanks! |
yeelam-gordon
marked this pull request as ready for review
August 28, 2026 10:09
…workflow path filters) - Quote \ in bash cd commands so repo paths with spaces work - Add explicit guard in vc_env_helper_arm64.bat when vcvarsall.bat isn't found - Add packaging/build_ffmpeg_arm64.bat and packaging/build_libavif_arm64.bat to their workflows' pull_request path filters so future edits trigger CI
…ripts - Fail fast (|| exit /b 1) after choco install / pacman / build_*.sh in build_ffmpeg_arm64.bat and build_libavif_arm64.bat - Replace vc_env_helper_arm64.bat's manual %1-shift argument reconstruction (which drops quoting) with %* passthrough
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds native Windows Arm64 CI support for torchcodec's FFmpeg/libavif build dependencies, using MSYS2's CLANGARM64 (native aarch64 clang) toolchain.
Real hardware verification
Dispatched on GitHub's free, GA
windows-11-armhosted runners via a personal fork (not upstream) before opening this PR. Found and fixed 3 real bugs invisible to static review:uname -mmisreports architecture under any MSYS2 shell -> wrongnasmrequirement (711ff55)configurehardcodesgcc, ignores$CC/$CXX-> needs explicit--cc=clang --cxx=clang++(22993a4)vcvarsall.bat-set INCLUDE/LIB leak MSVC UCRT headers into clang's include path, breaking compile -> drop thevc_env_helper_arm64.batcall (541f218)Final state: all real-hardware jobs green.
Test plan
Real windows-11-arm CI runs linked above, on my fork, prior to opening this PR.