[ROCm] Test ROCm failures on PR#1554 - #1642
Open
akashveramd wants to merge 53 commits into
Open
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/meta-pytorch/torchcodec/1642
Note: Links to docs will display an error until the docs builds have been completed. ❌ 2 Unclassified FailuresAs of commit 0a8c51a with merge base d249ad5 ( UNCLASSIFIED FAILURES - DrCI could not classify the following jobs because the workflow did not run on the merge base. The failures may be pre-existing on trunk or introduced by this PR:
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
Co-authored-by: Cursor <cursoragent@cursor.com>
akashveramd
force-pushed
the
rocm_decoder_analysis
branch
from
August 18, 2026 23:21
35e74f6 to
61016ce
Compare
librocjpeg links against libva.so.2 and libva-drm.so.2 (VA-API, used by rocJPEG's HYBRID GPU-JPEG backend). auditwheel was pulling both into the wheel because they weren't excluded, and repair_wheel.py's check_bundling() then rejected them as unexpected. libva/libva-drm are system-provided display-stack libraries, present on any ROCm install alongside libdrm (which is already excluded). Add "libva*" to the same exclude list so auditwheel leaves them on the system rather than bundling them. Co-authored-by: Cursor <cursoragent@cursor.com>
ROCm >= 7.14 distributes the full ROCm stack (including rocJPEG) as pip wheels (_rocm_sdk_core / _rocm_sdk_devel site-packages) rather than system RPMs. The rocjpeg-devel, libva-amdgpu and mesa-amdgpu-va-drivers DNF packages therefore don't exist on the ROCm 7.14 builder image and the install was failing with "No package rocjpeg available". Check for librocjpeg.so under /opt/conda (pip-wheel install path) and /opt/rocm (classic RPM install path) before attempting dnf install. If already present, skip the install entirely. The existing dnf path is preserved for ROCm 7.2 and earlier where RPMs are the only source. Co-authored-by: Cursor <cursoragent@cursor.com>
librocjpeg links libva.so.2 as a DT_NEEDED entry and needs it at dlopen time even when only the HARDWARE backend is used. The previous fix skipped all dnf installs when librocjpeg was found via ROCm 7.14 pip wheels, which left libva absent on the test machine and caused libtorchcodec_image.so to fail to load with OSError. libva ships in AlmaLinux standard repos so install it unconditionally. Co-authored-by: Cursor <cursoragent@cursor.com>
_find_rocjpeg_lib() only searched ROCM_HOME/lib, torch's ROCM_HOME, and /opt/rocm/lib. For ROCm >= 7.14, librocjpeg lives inside the _rocm_sdk_core pip-wheel's site-packages (e.g. /opt/conda/lib/python3.11/site-packages/_rocm_sdk_core/lib/librocjpeg.so), none of which were on those search paths. The function therefore returned None, printed a WARNING, and auditwheel never bundled librocjpeg. At test time libtorchcodec_image.so had DT_NEEDED: librocjpeg.so but no copy was on the dynamic-linker search path → OSError: Could not load this library. Fix: add a site-packages-first fallback that checks _rocm_sdk_core/lib and _rocm_sdk_devel/lib in the current interpreter's site-packages, then falls back to a broad /opt/conda/** glob (mirrors install_rocjpeg.sh). Also add ldd diagnostics before pytest in the install-and-test workflow so that any future missing transitive dep is visible in the log rather than hidden behind a bare OSError, and update the stale comment that said the wheel does not bundle librocjpeg. Co-authored-by: Cursor <cursoragent@cursor.com>
…yout PyTorch ROCm wheels may bundle the ROCm runtime (libamdhip64, libhsa-runtime64 etc.) inside torch/lib/ rather than depending on a separate _rocm_sdk_core pip wheel. Add \$ORIGIN/../../torch/lib to the RPATH patched onto the bundled librocjpeg-HASH.so so it can find those deps in either layout: - torch/lib/ (torch bundles ROCm runtime, like it does for CUDA) - _rocm_sdk_core/lib/ (TheRock separate pip-wheel layout) - /opt/rocm/lib (ROCm <= 7.2 system install) Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
akashveramd
force-pushed
the
rocm_decoder_analysis
branch
from
August 19, 2026 16:50
7435c2f to
2efd33f
Compare
…nostics The bundled librocjpeg-HASH.so has DT_NEEDED entries for librocm_sysdeps_va.so.2, librocm_sysdeps_drm_amdgpu.so.1, librocprofiler-register.so.0 etc. These live in _rocm_sdk_core/lib/ and _rocm_sdk_core/lib/rocm_sysdeps/lib/ but are not on the system search path and are not loaded by torch at import time. The RPATH on the bundled lib uses $ORIGIN which ldd cannot expand without execute permission (ldd shows all $ORIGIN-based entries as "not found"). Whether $ORIGIN resolves correctly at actual dlopen time is unclear, so set LD_LIBRARY_PATH explicitly before pytest as belt-and-suspenders. Also fix the rocjpeg_so find path (torchcodec.libs/ is a sibling of torchcodec/, not a child) and add objdump RPATH output so we can verify the RPATH was patched correctly on the bundled librocjpeg. Co-authored-by: Cursor <cursoragent@cursor.com>
Bundling librocjpeg moves it out of _rocm_sdk_core/lib, breaking AMD's own relative RPATH inside it that points to librocm_sysdeps_* transitive deps. We then have to re-patch those paths ourselves, which is fragile and was causing the missing librocm_sysdeps_va.so.2 / libamdhip64.so.7 failures at runtime. The cleaner approach: exclude librocjpeg from auditwheel bundling and instead append _rocm_sdk_core/lib (ROCm >= 7.14) and /opt/rocm/lib (ROCm <= 7.2) to the RPATH of libtorchcodec_image.so itself. At runtime the dynamic linker finds librocjpeg in its original location; AMD's own RPATH on that library then correctly resolves all transitive deps without any intervention from us. _find_rocjpeg_lib() is kept: auditwheel still needs librocjpeg in LD_LIBRARY_PATH to resolve the DT_NEEDED during its analysis pass, even though we exclude it from bundling. Co-authored-by: Cursor <cursoragent@cursor.com>
…ness Co-authored-by: Cursor <cursoragent@cursor.com>
check_bundling() previously required librocjpeg to be bundled in every ROCm wheel, which is now intentionally wrong. Update it to: - Treat "not bundled" as correct for ROCm wheels - Instead verify that libtorchcodec_image.so has _rocm_sdk_core/lib in its RPATH (confirming _patch_image_so_rpath_in_wheel ran correctly) - Raise if librocjpeg IS bundled (that would be a regression) - Remove _is_rocjpeg from the bundled-lib allowlist for the same reason - Add top-level `import tempfile` (used by the new RPATH check) Co-authored-by: Cursor <cursoragent@cursor.com>
… rocjpeg - bundle_third_party_licenses: warn (not raise) if rocjpeg LICENSE is missing, since librocjpeg is no longer redistributed in the wheel - check_bundling license check: only enforce rocjpeg keyword if the license file was actually bundled (it's optional, not mandatory) - Fix stale "ROCm wheels bundle librocjpeg" comment - Fix stale _find_rocjpeg_license docstring Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
$ORIGIN/../../_rocm_sdk_core/lib resolves to python3.x/_rocm_sdk_core/lib which does not exist. The correct path is $ORIGIN/../_rocm_sdk_core/lib: libtorchcodec_image.so lives in site-packages/torchcodec/, so one ../ brings us to site-packages/ where _rocm_sdk_core/ lives. This matches auditwheel's own convention: it sets $ORIGIN/../torchcodec.libs for the same reason. Co-authored-by: Cursor <cursoragent@cursor.com>
rocJPEG's HYBRID backend calls vaInitialize() to initialize the AMD GPU video decoder. libva.so.2 alone is not enough; it also needs the AMD VA-API backend driver (mesa-amdgpu-va-drivers) which provides the radeonsi/amdgpu DRI plugin that vaInitialize() dlopen-s at runtime. Without it, vaInitialize() fails and rocJPEG returns ROCJPEG_STATUS_NOT_INITIALIZED / ROCJPEG_STATUS_NOT_IMPLEMENTED, causing all jpeg_cuda tests to fail with "Failed to initialize rocJPEG with the hybrid backend". Install mesa-amdgpu-va-drivers alongside libva in the ROCm 7.14 pip-wheel path; fall back to libva-only if the AMD graphics repo is not available (build-only runners). Co-authored-by: Cursor <cursoragent@cursor.com>
Since ROCm 7.14, mesa (including the amdgpu/radeonsi VA-API backend driver) ships inside the _rocm_sdk_core pip wheel. There is no need to install mesa-amdgpu-va-drivers via dnf. However, libva's driver discovery (vaInitialize) needs to know where to find the DRI driver plugin. Set LIBVA_DRIVERS_PATH to _rocm_sdk_core/lib/dri so the vendored librocm_sysdeps_va.so.2 can find the bundled mesa backend, fixing the ROCJPEG_STATUS_NOT_INITIALIZED / ROCJPEG_STATUS_NOT_IMPLEMENTED failures. Also add a fallback diagnostic that lists _rocm_sdk_core/lib/ if the dri subdirectory is not found, to help debug future layout changes. Reverts the mesa-amdgpu-va-drivers dnf install added in the previous commit (442ac6c) as it is not needed. Co-authored-by: Cursor <cursoragent@cursor.com>
The multi-line python block inside $() in the YAML run: | block had lines starting at column 0, which breaks the YAML scalar parser (same issue as before). Collapse to a single line. Also improve the *_drv_video.so search: use rglob instead of checking a hardcoded lib/dri/ path so the mesa VA-API backend driver is found regardless of the exact subdirectory AMD chose inside _rocm_sdk_core. The fallback diagnostic now greps the full recursive listing for dri/va/video/mesa/gallium/radeon keywords to pinpoint the layout if the driver isn't found. Co-authored-by: Cursor <cursoragent@cursor.com>
…iver libva auto-detects the DRM driver name as "radeonsi" from the kernel and looks for radeonsi_drv_video.so. AMD vendored it as librocm_sysdeps_gallium_drv_video.so, so LIBVA_DRIVER_NAME must be set to "librocm_sysdeps_gallium" to override that. Verified locally: vaInitialize returns 0 with both LIBVA_DRIVERS_PATH and LIBVA_DRIVER_NAME set. Co-authored-by: Cursor <cursoragent@cursor.com>
1. test/utils.py: assert_frames_equal crashes with TypeError when cuda_version_used_for_building_torch() returns None on ROCm. Handle None explicitly, treating it as >= CUDA 13 (use atol=3). 2. DecodeJpegRocm: ROCJPEG_OUTPUT_RGB_PLANAR via the HW (VCN) backend produces ~51% correct pixels for colour (YCbCr) JPEG sources on MI350X: the hardware returns unconverted YCbCr planes instead of RGB. Fix: set force_hybrid=true in make_plan() for colour JPEGs requesting RGB output, routing them to the HYBRID backend which handles YCbCr->RGB in software. 3. DecodeJpegRocm: rocJpegDecodeBatched writes with an internally-aligned pitch that does not match our tensor row stride, producing completely wrong output when the batch mixes images of different dimensions. Fix: replace rocJpegDecodeBatched with individual rocJpegDecode calls in decode_batched_hardware(). 4. test_decoders.py: test_cuda_jpeg_errors expects corrupt JPEG input to raise RuntimeError. rocJPEG silently "succeeds" on this corrupt JPEG (nvJPEG raises, rocJPEG does not). Skip the corrupt-JPEG assertion on ROCm; the CPU-tensor-on-GPU error check still runs. Co-authored-by: Cursor <cursoragent@cursor.com>
akashveramd
commented
Aug 19, 2026
| search_roots.append(Path(result.stdout.strip())) | ||
| except Exception: | ||
| pass | ||
| search_roots.append(Path("/opt/rocm")) |
Collaborator
Author
There was a problem hiding this comment.
Remove line 157. Double check with ROCm 7.2 by printing ROCM_HOME.
akashveramd
commented
Aug 19, 2026
| import glob, sys | ||
| # _rocm_sdk_core and _rocm_sdk_devel are the pip-wheel-based ROCm installs | ||
| hits = (glob.glob('/opt/conda/**/librocjpeg.so*', recursive=True) + | ||
| glob.glob('/opt/rocm/lib/librocjpeg.so*')) |
Collaborator
Author
There was a problem hiding this comment.
Remove /opt/rocm as the if condition is for 7.14.
… versions Replace the hardcoded python3.9..python3.13 list with a dynamic search: iterate over PATH directories and pick up every python3 / python3.X executable found there. Each one is queried via importlib.util.find_spec so no version numbers are baked into the script. Co-authored-by: Cursor <cursoragent@cursor.com>
akashveramd
commented
Aug 25, 2026
akashveramd
commented
Aug 25, 2026
akashveramd
force-pushed
the
rocm_decoder_analysis
branch
2 times, most recently
from
August 25, 2026 01:47
5daa2f5 to
e233033
Compare
akashveramd
force-pushed
the
rocm_decoder_analysis
branch
2 times, most recently
from
August 25, 2026 18:05
7aa5583 to
930cefe
Compare
rocJPEG only supports the HARDWARE backend (per AMD). Remove all HYBRID backend code (handle, lazy init, split_images_by_backend, decode_hybrid, is_hw_decodable_jpeg, force_hybrid workaround) and replace the two-handle design with a single handle_ created at construction. The decoder now calls rocJpegDecode directly for every image using the hardware backend. Also remove test infrastructure from linux_rocm.yaml that was added to work around VA-API initialization failures: LD_LIBRARY_PATH/LIBVA env var setup, ldd diagnostics, test dependency install, and the pytest runner. The install-and-test job now only verifies that the wheel installs cleanly. Co-authored-with: AI assistant Co-authored-by: Cursor <cursoragent@cursor.com>
akashveramd
force-pushed
the
rocm_decoder_analysis
branch
from
August 25, 2026 19:15
930cefe to
86646fa
Compare
Temporary diagnostic commit to confirm ROCJPEG_BACKEND_HARDWARE is being set and used. Logs on both success and failure of rocJpegCreate(HARDWARE) and at actual decode time to confirm VCN engine is invoked. This commit can be dropped once the VA-API initialization issue is resolved. Co-authored-by: Cursor <cursoragent@cursor.com>
Temporary diagnostic commit to identify why rocJpegCreate(HARDWARE) fails with ROCJPEG_STATUS_NOT_INITIALIZED. Logs ROCM_PATH, LIBVA env vars, /dev/dri/ device nodes, GPU visibility via rocm-smi and PyTorch, which libva librocjpeg is linked against, VA-API driver files under ROCM_PATH, and vainfo output both with default settings and with ROCM_PATH/lib/dri as the driver path. This commit can be dropped once the VA-API initialization issue is resolved. Co-authored-by: Cursor <cursoragent@cursor.com>
akashveramd
force-pushed
the
rocm_decoder_analysis
branch
from
August 25, 2026 19:28
86646fa to
99d3266
Compare
Set ROCJPEG_LOG_LEVEL=3 as an inline env var for the pytest invocation so that librocjpeg emits detailed logs (including VA-API initialization) to stderr during CI test runs. This helps diagnose why rocJpegCreate() returns ROCJPEG_STATUS_NOT_INITIALIZED on the HARDWARE backend. Co-authored-by: AI assistant Co-authored-by: Cursor <cursoragent@cursor.com>
List the full contents of ROCM_PATH/lib/rocm_sysdeps/lib/ so we can confirm exactly which VA-API libraries are present on the CI runner. Co-authored-by: AI assistant Co-authored-by: Cursor <cursoragent@cursor.com>
…lure Add three more diagnostic probes to narrow down why vaInitialize fails: - vainfo with LIBVA_DRIVERS_PATH pointing to rocm_sysdeps/lib (the correct location of the AMD VA-API driver, not lib/dri) - vainfo --display drm --device /dev/dri/renderD128 to test VA-API initialization directly against the DRM node - /sys/class/drm/renderD128/device/unique_id to expose the GPU UUID registered in the DRM subsystem, for comparison against the HIP UUID Co-authored-by: AI assistant Co-authored-by: Cursor <cursoragent@cursor.com>
We collected the VA-API and UUID diagnostic data we needed. The vainfo commands and the hardcoded renderD128 sysfs path were temporary probes that are no longer needed and caused failures due to the hardcoded device node not existing in all environments. Authored with an AI assistant. Co-authored-by: Cursor <cursoragent@cursor.com>
…ents Updates the CI to use ROCm 10.0 instead of 7.14. Replaces version-specific comments (ROCm >= 7.14, ROCm <= 7.2) in packaging scripts with layout-based descriptions (TheRock/pip-wheel layout vs legacy ROCm system install) since the _rocm_sdk_core layout applies regardless of the specific ROCm version. Co-authored-by: AI assistant Co-authored-by: Cursor <cursoragent@cursor.com>
There was a problem hiding this comment.
If rocjpeg is only supported for rocm10.0 onwards properly anyway, let's get of this file, since it doesn't seem like we need it
…heel layout Removes all legacy ROCm system install (pre-pip-wheel) support from the packaging scripts since we only target ROCm 10.0 which ships via the TheRock/pip-wheel layout (_rocm_sdk_core). Removes _get_rocm_search_roots() and the ROCM_HOME/ROCM_PATH search paths from _find_rocjpeg_lib() and _find_rocjpeg_license(). The RPATH patch now only injects $ORIGIN/../_rocm_sdk_core/lib. Updates _find_rocjpeg_license() to search _rocm_sdk_core/lib directly. Co-authored-by: AI assistant Co-authored-by: Cursor <cursoragent@cursor.com>
…r ROCm 10.0 install_rocjpeg.sh was only needed to install rocJPEG from DNF repos for legacy ROCm system installs. Since we now only support ROCm 10.0 with the TheRock/pip-wheel layout where rocJPEG is bundled in _rocm_sdk_core, this script is no longer needed. Removes the install_rocjpeg.sh invocations from linux_rocm.yaml and pre_build_script.sh. Co-authored-by: AI assistant Co-authored-by: Cursor <cursoragent@cursor.com>
Drop the /opt/rocm fallback in CMakeLists.txt: for the TheRock/pip-wheel layout, ROCM_HOME is set via torch.utils.cpp_extension and points to _rocm_sdk_core in site-packages. If it is not set, fail with a clear message instead of silently probing /opt/rocm. Update the rocJPEG-not-found fatal error to mention _rocm_sdk_core / _rocm_sdk_devel pip packages instead of legacy DNF packages. Remove :-/opt/rocm fallbacks from the diagnostic block in linux_rocm.yaml: ROCM_PATH is already set from _rocm_sdk_core above, so falling back to /opt/rocm (which does not exist in a pip-wheel environment) only produces misleading output. Co-authored-by: AI assistant Co-authored-by: Cursor <cursoragent@cursor.com>
site.getsitepackages() already returns the conda site-packages directory where _rocm_sdk_core is installed, so the broad /opt/conda glob was redundant. If _find_rocjpeg_lib() returns None, the caller already handles it gracefully with a warning. Co-authored-by: AI assistant Co-authored-by: Cursor <cursoragent@cursor.com>
The upstream generate_binary_build_matrix.yml only knows about ROCm 7.2 and 7.14, so delegating matrix generation to it caused the build job to produce rocm7.2 and rocm7.14 wheels while the install-and-test job tried to download a rocm10.0 artifact that was never built. Replace the upstream matrix generation with a hardcoded matrix that only includes the single ROCm 10.0 / Python 3.10 entry we actually need. Co-authored-by: AI assistant Co-authored-by: Cursor <cursoragent@cursor.com>
pytorch/test-infra#8670 updates the nightly ROCm matrix from 7.2/7.14 to 7.14/10.0. Once that lands, the upstream matrix generator will emit ROCm 10.0 entries natively, so there is no need to hardcode the matrix here. Co-authored-by: AI assistant Co-authored-by: Cursor <cursoragent@cursor.com>
The upstream generate_binary_build_matrix (pytorch/test-infra#8670) will include both 7.14 and 10.0 once merged. torchcodec only supports the TheRock/pip-wheel layout (ROCm 10.0+), so add a filter-matrix job that strips any entries with gpu_arch_version != "10.0" before passing the matrix to build_wheels_linux.yml. Co-authored-by: AI assistant Co-authored-by: Cursor <cursoragent@cursor.com>
… 10.0 Excluding by version name is more future-proof: when ROCm 11.0 is added to the upstream matrix it will pass through automatically without any code change here. Co-authored-by: AI assistant Co-authored-by: Cursor <cursoragent@cursor.com>
Using xargs -I{} to write to GITHUB_OUTPUT stripped the double quotes
from the JSON keys, making the output invalid for fromJSON(). Use a
variable assignment instead.
Co-authored-by: AI assistant
Co-authored-by: Cursor <cursoragent@cursor.com>
For the TheRock/pip-wheel layout (ROCm 10.0), libva is bundled inside _rocm_sdk_core, not provided by the system. The exclusion from auditwheel bundling is still correct since _rocm_sdk_core is a required dependency. Co-authored-by: AI assistant Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Creating this PR to-