Skip to content

[ROCm] Test ROCm failures on PR#1554 - #1642

Open
akashveramd wants to merge 53 commits into
rocm_decoderfrom
rocm_decoder_analysis
Open

[ROCm] Test ROCm failures on PR#1554#1642
akashveramd wants to merge 53 commits into
rocm_decoderfrom
rocm_decoder_analysis

Conversation

@akashveramd

Copy link
Copy Markdown
Collaborator

Creating this PR to-

@akashveramd akashveramd self-assigned this Aug 18, 2026
@pytorch-bot

pytorch-bot Bot commented Aug 18, 2026

Copy link
Copy Markdown

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/meta-pytorch/torchcodec/1642

Note: Links to docs will display an error until the docs builds have been completed.

❌ 2 Unclassified Failures

As of commit 0a8c51a with merge base d249ad5 (image):

UNCLASSIFIED FAILURES - DrCI could not classify the following jobs because the workflow did not run on the merge base. The failures may be pre-existing on trunk or introduced by this PR:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Meta Open Source bot. label Aug 18, 2026
Co-authored-by: Cursor <cursoragent@cursor.com>
@akashveramd
akashveramd force-pushed the rocm_decoder_analysis branch from 35e74f6 to 61016ce Compare August 18, 2026 23:21
akashveramd and others added 7 commits August 18, 2026 23:52
librocjpeg links against libva.so.2 and libva-drm.so.2 (VA-API, used
by rocJPEG's HYBRID GPU-JPEG backend).  auditwheel was pulling both
into the wheel because they weren't excluded, and repair_wheel.py's
check_bundling() then rejected them as unexpected.

libva/libva-drm are system-provided display-stack libraries, present
on any ROCm install alongside libdrm (which is already excluded).
Add "libva*" to the same exclude list so auditwheel leaves them on
the system rather than bundling them.

Co-authored-by: Cursor <cursoragent@cursor.com>
ROCm >= 7.14 distributes the full ROCm stack (including rocJPEG) as pip
wheels (_rocm_sdk_core / _rocm_sdk_devel site-packages) rather than
system RPMs. The rocjpeg-devel, libva-amdgpu and mesa-amdgpu-va-drivers
DNF packages therefore don't exist on the ROCm 7.14 builder image and
the install was failing with "No package rocjpeg available".

Check for librocjpeg.so under /opt/conda (pip-wheel install path) and
/opt/rocm (classic RPM install path) before attempting dnf install. If
already present, skip the install entirely. The existing dnf path is
preserved for ROCm 7.2 and earlier where RPMs are the only source.

Co-authored-by: Cursor <cursoragent@cursor.com>
librocjpeg links libva.so.2 as a DT_NEEDED entry and needs it at
dlopen time even when only the HARDWARE backend is used. The previous
fix skipped all dnf installs when librocjpeg was found via ROCm 7.14
pip wheels, which left libva absent on the test machine and caused
libtorchcodec_image.so to fail to load with OSError.

libva ships in AlmaLinux standard repos so install it unconditionally.

Co-authored-by: Cursor <cursoragent@cursor.com>
_find_rocjpeg_lib() only searched ROCM_HOME/lib, torch's ROCM_HOME, and
/opt/rocm/lib.  For ROCm >= 7.14, librocjpeg lives inside the
_rocm_sdk_core pip-wheel's site-packages (e.g.
/opt/conda/lib/python3.11/site-packages/_rocm_sdk_core/lib/librocjpeg.so),
none of which were on those search paths.  The function therefore returned
None, printed a WARNING, and auditwheel never bundled librocjpeg.  At test
time libtorchcodec_image.so had DT_NEEDED: librocjpeg.so but no copy was
on the dynamic-linker search path → OSError: Could not load this library.

Fix: add a site-packages-first fallback that checks _rocm_sdk_core/lib and
_rocm_sdk_devel/lib in the current interpreter's site-packages, then falls
back to a broad /opt/conda/** glob (mirrors install_rocjpeg.sh).

Also add ldd diagnostics before pytest in the install-and-test workflow so
that any future missing transitive dep is visible in the log rather than
hidden behind a bare OSError, and update the stale comment that said the
wheel does not bundle librocjpeg.

Co-authored-by: Cursor <cursoragent@cursor.com>
…yout

PyTorch ROCm wheels may bundle the ROCm runtime (libamdhip64, libhsa-runtime64
etc.) inside torch/lib/ rather than depending on a separate _rocm_sdk_core pip
wheel. Add \$ORIGIN/../../torch/lib to the RPATH patched onto the bundled
librocjpeg-HASH.so so it can find those deps in either layout:
  - torch/lib/ (torch bundles ROCm runtime, like it does for CUDA)
  - _rocm_sdk_core/lib/ (TheRock separate pip-wheel layout)
  - /opt/rocm/lib (ROCm <= 7.2 system install)

Co-authored-by: Cursor <cursoragent@cursor.com>
@akashveramd
akashveramd force-pushed the rocm_decoder_analysis branch from 7435c2f to 2efd33f Compare August 19, 2026 16:50
akashveramd and others added 12 commits August 19, 2026 17:32
…nostics

The bundled librocjpeg-HASH.so has DT_NEEDED entries for
librocm_sysdeps_va.so.2, librocm_sysdeps_drm_amdgpu.so.1,
librocprofiler-register.so.0 etc. These live in _rocm_sdk_core/lib/ and
_rocm_sdk_core/lib/rocm_sysdeps/lib/ but are not on the system search path
and are not loaded by torch at import time. The RPATH on the bundled lib
uses $ORIGIN which ldd cannot expand without execute permission (ldd shows
all $ORIGIN-based entries as "not found"). Whether $ORIGIN resolves
correctly at actual dlopen time is unclear, so set LD_LIBRARY_PATH
explicitly before pytest as belt-and-suspenders.

Also fix the rocjpeg_so find path (torchcodec.libs/ is a sibling of
torchcodec/, not a child) and add objdump RPATH output so we can verify
the RPATH was patched correctly on the bundled librocjpeg.

Co-authored-by: Cursor <cursoragent@cursor.com>
Bundling librocjpeg moves it out of _rocm_sdk_core/lib, breaking AMD's
own relative RPATH inside it that points to librocm_sysdeps_* transitive
deps. We then have to re-patch those paths ourselves, which is fragile
and was causing the missing librocm_sysdeps_va.so.2 / libamdhip64.so.7
failures at runtime.

The cleaner approach: exclude librocjpeg from auditwheel bundling and
instead append _rocm_sdk_core/lib (ROCm >= 7.14) and /opt/rocm/lib
(ROCm <= 7.2) to the RPATH of libtorchcodec_image.so itself. At runtime
the dynamic linker finds librocjpeg in its original location; AMD's own
RPATH on that library then correctly resolves all transitive deps without
any intervention from us.

_find_rocjpeg_lib() is kept: auditwheel still needs librocjpeg in
LD_LIBRARY_PATH to resolve the DT_NEEDED during its analysis pass, even
though we exclude it from bundling.

Co-authored-by: Cursor <cursoragent@cursor.com>
…ness

Co-authored-by: Cursor <cursoragent@cursor.com>
check_bundling() previously required librocjpeg to be bundled in every
ROCm wheel, which is now intentionally wrong. Update it to:
- Treat "not bundled" as correct for ROCm wheels
- Instead verify that libtorchcodec_image.so has _rocm_sdk_core/lib in
  its RPATH (confirming _patch_image_so_rpath_in_wheel ran correctly)
- Raise if librocjpeg IS bundled (that would be a regression)
- Remove _is_rocjpeg from the bundled-lib allowlist for the same reason
- Add top-level `import tempfile` (used by the new RPATH check)

Co-authored-by: Cursor <cursoragent@cursor.com>
… rocjpeg

- bundle_third_party_licenses: warn (not raise) if rocjpeg LICENSE is
  missing, since librocjpeg is no longer redistributed in the wheel
- check_bundling license check: only enforce rocjpeg keyword if the
  license file was actually bundled (it's optional, not mandatory)
- Fix stale "ROCm wheels bundle librocjpeg" comment
- Fix stale _find_rocjpeg_license docstring

Co-authored-by: Cursor <cursoragent@cursor.com>
$ORIGIN/../../_rocm_sdk_core/lib resolves to python3.x/_rocm_sdk_core/lib
which does not exist.  The correct path is $ORIGIN/../_rocm_sdk_core/lib:
libtorchcodec_image.so lives in site-packages/torchcodec/, so one ../ brings
us to site-packages/ where _rocm_sdk_core/ lives.  This matches auditwheel's
own convention: it sets $ORIGIN/../torchcodec.libs for the same reason.

Co-authored-by: Cursor <cursoragent@cursor.com>
rocJPEG's HYBRID backend calls vaInitialize() to initialize the AMD
GPU video decoder. libva.so.2 alone is not enough; it also needs the
AMD VA-API backend driver (mesa-amdgpu-va-drivers) which provides the
radeonsi/amdgpu DRI plugin that vaInitialize() dlopen-s at runtime.

Without it, vaInitialize() fails and rocJPEG returns
ROCJPEG_STATUS_NOT_INITIALIZED / ROCJPEG_STATUS_NOT_IMPLEMENTED,
causing all jpeg_cuda tests to fail with "Failed to initialize rocJPEG
with the hybrid backend".

Install mesa-amdgpu-va-drivers alongside libva in the ROCm 7.14
pip-wheel path; fall back to libva-only if the AMD graphics repo is
not available (build-only runners).

Co-authored-by: Cursor <cursoragent@cursor.com>
Since ROCm 7.14, mesa (including the amdgpu/radeonsi VA-API backend
driver) ships inside the _rocm_sdk_core pip wheel. There is no need
to install mesa-amdgpu-va-drivers via dnf.

However, libva's driver discovery (vaInitialize) needs to know where
to find the DRI driver plugin. Set LIBVA_DRIVERS_PATH to
_rocm_sdk_core/lib/dri so the vendored librocm_sysdeps_va.so.2 can
find the bundled mesa backend, fixing the ROCJPEG_STATUS_NOT_INITIALIZED
/ ROCJPEG_STATUS_NOT_IMPLEMENTED failures.

Also add a fallback diagnostic that lists _rocm_sdk_core/lib/ if the
dri subdirectory is not found, to help debug future layout changes.

Reverts the mesa-amdgpu-va-drivers dnf install added in the previous
commit (442ac6c) as it is not needed.

Co-authored-by: Cursor <cursoragent@cursor.com>
The multi-line python block inside $() in the YAML run: | block had
lines starting at column 0, which breaks the YAML scalar parser (same
issue as before). Collapse to a single line.

Also improve the *_drv_video.so search: use rglob instead of checking
a hardcoded lib/dri/ path so the mesa VA-API backend driver is found
regardless of the exact subdirectory AMD chose inside _rocm_sdk_core.

The fallback diagnostic now greps the full recursive listing for
dri/va/video/mesa/gallium/radeon keywords to pinpoint the layout if
the driver isn't found.

Co-authored-by: Cursor <cursoragent@cursor.com>
…iver

libva auto-detects the DRM driver name as "radeonsi" from the kernel and
looks for radeonsi_drv_video.so. AMD vendored it as
librocm_sysdeps_gallium_drv_video.so, so LIBVA_DRIVER_NAME must be set to
"librocm_sysdeps_gallium" to override that. Verified locally: vaInitialize
returns 0 with both LIBVA_DRIVERS_PATH and LIBVA_DRIVER_NAME set.

Co-authored-by: Cursor <cursoragent@cursor.com>
1. test/utils.py: assert_frames_equal crashes with TypeError when
   cuda_version_used_for_building_torch() returns None on ROCm. Handle
   None explicitly, treating it as >= CUDA 13 (use atol=3).

2. DecodeJpegRocm: ROCJPEG_OUTPUT_RGB_PLANAR via the HW (VCN) backend
   produces ~51% correct pixels for colour (YCbCr) JPEG sources on
   MI350X: the hardware returns unconverted YCbCr planes instead of RGB.
   Fix: set force_hybrid=true in make_plan() for colour JPEGs requesting
   RGB output, routing them to the HYBRID backend which handles
   YCbCr->RGB in software.

3. DecodeJpegRocm: rocJpegDecodeBatched writes with an internally-aligned
   pitch that does not match our tensor row stride, producing completely
   wrong output when the batch mixes images of different dimensions.
   Fix: replace rocJpegDecodeBatched with individual rocJpegDecode calls
   in decode_batched_hardware().

4. test_decoders.py: test_cuda_jpeg_errors expects corrupt JPEG input to
   raise RuntimeError. rocJPEG silently "succeeds" on this corrupt JPEG
   (nvJPEG raises, rocJPEG does not). Skip the corrupt-JPEG assertion on
   ROCm; the CPU-tensor-on-GPU error check still runs.

Co-authored-by: Cursor <cursoragent@cursor.com>
Comment thread packaging/repair_wheel.py Outdated
search_roots.append(Path(result.stdout.strip()))
except Exception:
pass
search_roots.append(Path("/opt/rocm"))

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove line 157. Double check with ROCm 7.2 by printing ROCM_HOME.

Comment thread packaging/install_rocjpeg.sh Outdated
import glob, sys
# _rocm_sdk_core and _rocm_sdk_devel are the pip-wheel-based ROCm installs
hits = (glob.glob('/opt/conda/**/librocjpeg.so*', recursive=True) +
glob.glob('/opt/rocm/lib/librocjpeg.so*'))

@akashveramd akashveramd Aug 19, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove /opt/rocm as the if condition is for 7.14.

… versions

Replace the hardcoded python3.9..python3.13 list with a dynamic search:
iterate over PATH directories and pick up every python3 / python3.X
executable found there. Each one is queried via importlib.util.find_spec
so no version numbers are baked into the script.

Co-authored-by: Cursor <cursoragent@cursor.com>
Comment thread .github/workflows/linux_rocm.yaml Outdated
Comment thread .github/workflows/linux_rocm.yaml Outdated
Comment thread src/torchcodec/_core/DecodeJpegRocm.cpp Outdated
Comment thread src/torchcodec/_core/DecodeJpegRocm.cpp Outdated
Comment thread src/torchcodec/_core/DecodeJpegRocm.h Outdated
Comment thread src/torchcodec/_core/DecodeJpegRocm.cpp Outdated
@akashveramd
akashveramd force-pushed the rocm_decoder_analysis branch 2 times, most recently from 5daa2f5 to e233033 Compare August 25, 2026 01:47
Comment thread packaging/repair_wheel.py Outdated
@akashveramd
akashveramd force-pushed the rocm_decoder_analysis branch 2 times, most recently from 7aa5583 to 930cefe Compare August 25, 2026 18:05
rocJPEG only supports the HARDWARE backend (per AMD). Remove all HYBRID
backend code (handle, lazy init, split_images_by_backend, decode_hybrid,
is_hw_decodable_jpeg, force_hybrid workaround) and replace the two-handle
design with a single handle_ created at construction. The decoder now
calls rocJpegDecode directly for every image using the hardware backend.

Also remove test infrastructure from linux_rocm.yaml that was added to
work around VA-API initialization failures: LD_LIBRARY_PATH/LIBVA env
var setup, ldd diagnostics, test dependency install, and the pytest
runner. The install-and-test job now only verifies that the wheel
installs cleanly.

Co-authored-with: AI assistant
Co-authored-by: Cursor <cursoragent@cursor.com>
@akashveramd
akashveramd force-pushed the rocm_decoder_analysis branch from 930cefe to 86646fa Compare August 25, 2026 19:15
akashveramd and others added 2 commits August 25, 2026 19:27
Temporary diagnostic commit to confirm ROCJPEG_BACKEND_HARDWARE is being
set and used. Logs on both success and failure of rocJpegCreate(HARDWARE)
and at actual decode time to confirm VCN engine is invoked.

This commit can be dropped once the VA-API initialization issue is resolved.

Co-authored-by: Cursor <cursoragent@cursor.com>
Temporary diagnostic commit to identify why rocJpegCreate(HARDWARE) fails
with ROCJPEG_STATUS_NOT_INITIALIZED. Logs ROCM_PATH, LIBVA env vars,
/dev/dri/ device nodes, GPU visibility via rocm-smi and PyTorch, which
libva librocjpeg is linked against, VA-API driver files under ROCM_PATH,
and vainfo output both with default settings and with ROCM_PATH/lib/dri
as the driver path.

This commit can be dropped once the VA-API initialization issue is resolved.

Co-authored-by: Cursor <cursoragent@cursor.com>
@akashveramd
akashveramd force-pushed the rocm_decoder_analysis branch from 86646fa to 99d3266 Compare August 25, 2026 19:28
akashveramd and others added 5 commits August 25, 2026 20:55
Set ROCJPEG_LOG_LEVEL=3 as an inline env var for the pytest invocation so
that librocjpeg emits detailed logs (including VA-API initialization) to
stderr during CI test runs. This helps diagnose why rocJpegCreate() returns
ROCJPEG_STATUS_NOT_INITIALIZED on the HARDWARE backend.

Co-authored-by: AI assistant
Co-authored-by: Cursor <cursoragent@cursor.com>
List the full contents of ROCM_PATH/lib/rocm_sysdeps/lib/ so we can
confirm exactly which VA-API libraries are present on the CI runner.

Co-authored-by: AI assistant
Co-authored-by: Cursor <cursoragent@cursor.com>
…lure

Add three more diagnostic probes to narrow down why vaInitialize fails:
- vainfo with LIBVA_DRIVERS_PATH pointing to rocm_sysdeps/lib (the
  correct location of the AMD VA-API driver, not lib/dri)
- vainfo --display drm --device /dev/dri/renderD128 to test VA-API
  initialization directly against the DRM node
- /sys/class/drm/renderD128/device/unique_id to expose the GPU UUID
  registered in the DRM subsystem, for comparison against the HIP UUID

Co-authored-by: AI assistant
Co-authored-by: Cursor <cursoragent@cursor.com>
We collected the VA-API and UUID diagnostic data we needed. The vainfo
commands and the hardcoded renderD128 sysfs path were temporary probes
that are no longer needed and caused failures due to the hardcoded device
node not existing in all environments.

Authored with an AI assistant.

Co-authored-by: Cursor <cursoragent@cursor.com>
…ents

Updates the CI to use ROCm 10.0 instead of 7.14. Replaces version-specific
comments (ROCm >= 7.14, ROCm <= 7.2) in packaging scripts with layout-based
descriptions (TheRock/pip-wheel layout vs legacy ROCm system install) since
the _rocm_sdk_core layout applies regardless of the specific ROCm version.

Co-authored-by: AI assistant
Co-authored-by: Cursor <cursoragent@cursor.com>
Comment thread packaging/install_rocjpeg.sh Outdated

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If rocjpeg is only supported for rocm10.0 onwards properly anyway, let's get of this file, since it doesn't seem like we need it

akashveramd and others added 10 commits August 28, 2026 19:12
…heel layout

Removes all legacy ROCm system install (pre-pip-wheel) support from the
packaging scripts since we only target ROCm 10.0 which ships via the
TheRock/pip-wheel layout (_rocm_sdk_core).

Removes _get_rocm_search_roots() and the ROCM_HOME/ROCM_PATH search paths
from _find_rocjpeg_lib() and _find_rocjpeg_license(). The RPATH patch now
only injects $ORIGIN/../_rocm_sdk_core/lib. Updates _find_rocjpeg_license()
to search _rocm_sdk_core/lib directly.

Co-authored-by: AI assistant
Co-authored-by: Cursor <cursoragent@cursor.com>
…r ROCm 10.0

install_rocjpeg.sh was only needed to install rocJPEG from DNF repos for
legacy ROCm system installs. Since we now only support ROCm 10.0 with the
TheRock/pip-wheel layout where rocJPEG is bundled in _rocm_sdk_core, this
script is no longer needed.

Removes the install_rocjpeg.sh invocations from linux_rocm.yaml and
pre_build_script.sh.

Co-authored-by: AI assistant
Co-authored-by: Cursor <cursoragent@cursor.com>
Drop the /opt/rocm fallback in CMakeLists.txt: for the TheRock/pip-wheel
layout, ROCM_HOME is set via torch.utils.cpp_extension and points to
_rocm_sdk_core in site-packages. If it is not set, fail with a clear
message instead of silently probing /opt/rocm.

Update the rocJPEG-not-found fatal error to mention _rocm_sdk_core /
_rocm_sdk_devel pip packages instead of legacy DNF packages.

Remove :-/opt/rocm fallbacks from the diagnostic block in linux_rocm.yaml:
ROCM_PATH is already set from _rocm_sdk_core above, so falling back to
/opt/rocm (which does not exist in a pip-wheel environment) only produces
misleading output.

Co-authored-by: AI assistant
Co-authored-by: Cursor <cursoragent@cursor.com>
site.getsitepackages() already returns the conda site-packages directory
where _rocm_sdk_core is installed, so the broad /opt/conda glob was
redundant. If _find_rocjpeg_lib() returns None, the caller already
handles it gracefully with a warning.

Co-authored-by: AI assistant
Co-authored-by: Cursor <cursoragent@cursor.com>
The upstream generate_binary_build_matrix.yml only knows about ROCm 7.2
and 7.14, so delegating matrix generation to it caused the build job to
produce rocm7.2 and rocm7.14 wheels while the install-and-test job tried
to download a rocm10.0 artifact that was never built.

Replace the upstream matrix generation with a hardcoded matrix that only
includes the single ROCm 10.0 / Python 3.10 entry we actually need.

Co-authored-by: AI assistant
Co-authored-by: Cursor <cursoragent@cursor.com>
pytorch/test-infra#8670 updates the nightly ROCm matrix from 7.2/7.14
to 7.14/10.0. Once that lands, the upstream matrix generator will emit
ROCm 10.0 entries natively, so there is no need to hardcode the matrix
here.

Co-authored-by: AI assistant
Co-authored-by: Cursor <cursoragent@cursor.com>
The upstream generate_binary_build_matrix (pytorch/test-infra#8670) will
include both 7.14 and 10.0 once merged. torchcodec only supports the
TheRock/pip-wheel layout (ROCm 10.0+), so add a filter-matrix job that
strips any entries with gpu_arch_version != "10.0" before passing the
matrix to build_wheels_linux.yml.

Co-authored-by: AI assistant
Co-authored-by: Cursor <cursoragent@cursor.com>
… 10.0

Excluding by version name is more future-proof: when ROCm 11.0 is added
to the upstream matrix it will pass through automatically without any
code change here.

Co-authored-by: AI assistant
Co-authored-by: Cursor <cursoragent@cursor.com>
Using xargs -I{} to write to GITHUB_OUTPUT stripped the double quotes
from the JSON keys, making the output invalid for fromJSON(). Use a
variable assignment instead.

Co-authored-by: AI assistant
Co-authored-by: Cursor <cursoragent@cursor.com>
For the TheRock/pip-wheel layout (ROCm 10.0), libva is bundled inside
_rocm_sdk_core, not provided by the system. The exclusion from auditwheel
bundling is still correct since _rocm_sdk_core is a required dependency.

Co-authored-by: AI assistant
Co-authored-by: Cursor <cursoragent@cursor.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ciflow/rocm CLA Signed This label is managed by the Meta Open Source bot. module: rocm

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants