Skip to content

Skip broken MIOpen XDLOPS conv solvers on gfx90a too, and move the workaround into the test - #1103

Merged
luraess merged 3 commits into
mainfrom
lr/hip_dnn_conv
Sep 24, 2026
Merged

luraess merged 3 commits into
mainfrom
lr/hip_dnn_conv

Conversation

@luraess

@luraess luraess commented Sep 23, 2026

Copy link
Copy Markdown
Member

hip_dnn/conv segfaults on gfx90a (MI250X). The worker dies with signal 11, so the whole testset is lost. The cause is upstream: MIOpen's CK grouped-conv XDLOPS solvers (ROCm/rocm-libraries#9088). gfx942 CI only stays green because of five MIOPEN_DEBUG_* vars in ci/cscs-mi300.yml and a backward-data skip gated on "gfx942".

This PR:

  • extends the gate in test/hip_dnn/conv.jl to gfx90a;
  • sets the five vars from the test itself, before the first convolution. It uses
    get!, so a value already in the environment still wins. A plain Pkg.test now
    works outside CI;
  • drops the vars from ci/cscs-mi300.yml, so CI exercises the test-side path.

Findings

Problem: the first case of conv.jl (3×3 input, 3×3 filter, 1 channel, NCHW), with the beverin uenv prgenv-gnu/7.2.3 (MIOpen 3.5.1). Results are identical on gfx90a (MI250X) and gfx942 (MI300A):

Direction Solver Failure Disabled by
forward ConvHipImplicitGemmGroupFwdXdlops segfault in hipLaunchKernel MIOPEN_DEBUG_GROUP_CONV_IMPLICIT_GEMM_HIP_FWD_XDLOPS=0
backward-weights ConvHipImplicitGemmGroupWrwXdlops segfault / "invalid device function" in the workspace-size query MIOPEN_DEBUG_GROUP_CONV_IMPLICIT_GEMM_HIP_WRW_XDLOPS=0
backward-data ConvHipImplicitGemmGroupBwdXdlops segfault in hipLaunchKernel MIOPEN_DEBUG_CONV_IMPLICIT_GEMM_HIP_GROUP_BWD_XDLOPS=1 (inverted; =0 does not disable it)
  • Not AMDGPU.jl: MIOpenDriver alone reproduces it, without Julia.
  • Why gate on arch rather than skip everywhere: XDLOPS solvers exist only on CDNA. Buildkite (gfx1100) runs the same MIOpen version, 3.5.1 from /opt/rocm-7.2.3, and passes all 27 conv tests, backward-data included.
  • Why keep skipping backward-data instead of setting its inverted var: =1 fixes the 2D case in MIOpenDriver. The 3D backward-data solver isn't checked.
  • Open question: AMD's packaged ROCm 6.3.0 (MIOpen 3.3.0) runs the same solvers without error on gfx90a. It is not yet known whether MIOpen 3.5.1 regressed or the Spack build is at fault.

Reproducer (inside the uenv):

MIOPEN_DEBUG_FIND_ONLY_SOLVER=ConvHipImplicitGemmGroupFwdXdlops \
  MIOpenDriver convfp16 -n 1 -c 1 -H 3 -W 3 -k 1 -y 3 -x 3 \
  -p 0 -q 0 -u 1 -v 1 -l 1 -j 1 -g 1 -F 1 -V 1 -i 1

Testing

hip_dnn suite on beverin MI250X (gfx90a), run with no MIOPEN_DEBUG_* vars set:

Julia 1.12 Julia 1.13
before 147 pass, 1 error (worker segfault)
this PR 170 pass, 4 broken 170 pass, 4 broken

The 4 broken results are the skipped backward-data calls. The gfx942 path is covered by this PR's CSCS MI300 CI run, which no longer gets the vars from the yml.

luraess and others added 3 commits September 23, 2026 21:40
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ree codegen kernel

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@luraess

luraess commented Sep 24, 2026

Copy link
Copy Markdown
Member Author

Merging this as the fix only targets MI250/MI300 hardware for which tests are passing. It also brings MI250X pipeline into main.

@luraess
luraess merged commit f1fce92 into main Sep 24, 2026
2 of 4 checks passed
@luraess
luraess deleted the lr/hip_dnn_conv branch September 24, 2026 08:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant