MinimaxM2.5-FP8-MI325x-vLLM: pin AITER FA attention backend#1594
Conversation
|
Thanks for the contribution! For vLLM & SGLang, please ensure that your recipes is similar to the official vLLM recipes and/or the SGLang cookbook If it is not, please create a PR first before we can merge your single node PR into the master branch. Let's ensure that the documentation is first class such that the entire ML community can benefit from your hard work! Thank you PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. If re-running failed jobs is attempted, PR authors are responsible for ensuring it passes. See GitHub's docs on re-running failed jobs: https://docs.github.com/en/actions/how-tos/manage-workflow-runs/re-run-workflows-and-jobs#re-running-failed-jobs-in-a-workflow As a rule of thumb, generally, PR authors should request a review & get a PR approval from the respective companies' CODEOWNERS before requesting a review from core maintainers. If additional help is needed, PR authors can reach out to core maintainers over Slack. |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
LGTM, straightforward benchmark recipe update to restore the prior ROCm attention path.
Extended reasoning...
Overview
This PR modifies a single benchmark shell script (benchmarks/single_node/minimaxm2.5_fp8_mi325x.sh) to pin the ROCm attention backend to ROCM_AITER_FA and set VLLM_ROCM_SHUFFLE_KV_CACHE_LAYOUT=1. The change mirrors the upstream vLLM recipe (vllm-project/recipes#481) and addresses a ~38% throughput regression introduced by vLLM PR #36702 flipping the default attention backend.
Security risks
None. This is a benchmark-only shell script that runs on internal benchmark hardware. No auth, crypto, permissions, data paths, or production runtime code is touched. The added env var and CLI flag are standard vLLM configuration knobs.
Level of scrutiny
Low. The PR is mechanical, well-motivated with upstream references, scoped to a single benchmark script, and follows an established pattern (similar AITER/attention-backend pinning appears elsewhere in this repo's benchmark configs). The dropped LinkedIn comment is cleanup, not a behavioral change.
Other factors
No bugs were flagged by the bug hunting system. The change is self-contained, easy to revert if performance numbers don't match expectations, and the description includes clear citations (vllm-project/vllm#43029, vllm-project/recipes#481). Safe to shadow-approve.
|
/sweep test-config --config-files .github/configs/amd-master.yaml --config-keys minimaxm2.5-fp8-mi325x-vllm |
|
@chunfangamd Kicking off a sweep. Run: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/26684429581 |
|
/sweep test-config --config-files .github/configs/amd-master.yaml --config-keys minimaxm2.5-fp8-mi325x-vllm |
|
@chunfangamd Kicking off a sweep. Run: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/26686258742 |
chunfangamd
requested review from
1am9trash,
billishyahao,
seungrokj and
yctseng0211
as code owners
|
/sweep test-config --config-files .github/configs/amd-master.yaml --config-keys minimaxm2.5-fp8-mi325x-vllm |
|
@chunfangamd Kicking off a sweep. Run: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/26690885651 |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 2c76947. Configure here.
|
|
||
| if [ -z "$JOB_ID" ]; then | ||
| echo "ERROR: salloc failed to allocate a job" | ||
| echo "ERROR: salloc failed to allocate a job" >&2 |
There was a problem hiding this comment.
set -e skips salloc error
Medium Severity
Adding set -euo pipefail makes the JOB_ID=$(… | grep …) assignment abort the script when grep exits non-zero (no matching Granted job allocation line). The if [ -z "$JOB_ID" ] block that prints ERROR: salloc failed to allocate a job never runs, so allocation failures surface as a silent early exit instead of the intended message.
Reviewed by Cursor Bugbot for commit 2c76947. Configure here.
|
/sweep test-config --config-files .github/configs/amd-master.yaml --config-keys minimaxm2.5-fp8-mi325x-vllm |
|
@chunfangamd Kicking off a sweep. Run: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/26692603804 |
|
/sweep test-config --config-files .github/configs/amd-master.yaml --config-keys minimaxm2.5-fp8-mi325x-vllm |
|
@chunfangamd Kicking off a sweep. Run: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/26694484598 |
vLLM PR #36702 (between v0.18.0 and v0.21.0) flipped the dense full-attention default on ROCm from ROCM_AITER_FA to ROCM_ATTN, causing a ~38% throughput regression for MiniMax-M2.5 FP8 on MI325X (vllm-project/vllm#43029). Align benchmarks/single_node/minimaxm2.5_fp8_mi325x.sh with the merged upstream recipe (vllm-project/recipes#481) to restore the v0.18.0 attention path on the v0.21.0 image: - export VLLM_ROCM_SHUFFLE_KV_CACHE_LAYOUT=1 (asm/hip paged-attention auto-dispatch) - pass --attention-backend ROCM_AITER_FA to vllm serve
Use `40000 + (JOB_ID % 10000)` instead of a hard-coded 8888 — a non-SLURM Docker workload on chi-mi325x-pod1-019 bound :8888 and made every sweep job scheduled there fail in sock.bind() with EADDRINUSE before vLLM ran. Also harden the benchmark_logs trap with `sudo -n` so it fails fast under a non-tty instead of hanging.
Set VLLM_ROCM_SHUFFLE_KV_CACHE_LAYOUT=1 (recipes#481 pillar 2) only at shapes where AITER's gfx942 ASM paged-attn kernel exists: TP=2 EP=1 CONC<=16, TP=8 EP=8 CONC<=64. Above those, pa_fwd_asm hits `get_heuristic_kernel: cannot get heuristic kernel!` (gqa=6, block_size=32, qTile=0) and HTTP-500s every request. Mirrors the per-shape toggle in the mi355x sibling. vllm#43029, sweep run 26692603804.
functionstackx
force-pushed
the
chun-hongxia/minimax-fp8-mi325-regression
branch
from
e4a72fd to
5471c53
Compare
2 participants
vLLM PR #36702 (between v0.18.0 and v0.21.0) flipped the dense full-attention default on ROCm from ROCM_AITER_FA to ROCM_ATTN, causing a ~38% throughput regression for MiniMax-M2.5 FP8 on MI325X (vllm-project/vllm#43029).
Align benchmarks/single_node/minimaxm2.5_fp8_mi325x.sh with the merged upstream recipe (vllm-project/recipes#481) to restore the v0.18.0 attention path on the v0.21.0 image:
Co-author: @hongxiayang @vllmellm @ukannika
Note
Medium Risk
Changes production benchmark serving flags and pins a downgraded vLLM image for one config; MI325X launcher behavior (ports, traps, cleanup) affects all jobs using that runner script.
Overview
Restores MiniMax-M2.5 FP8 on MI325X vLLM throughput after a ROCm default attention-backend change (~38% regression) by aligning
benchmarks/single_node/minimaxm2.5_fp8_mi325x.shwith the upstream MiniMax ROCm recipe.The benchmark script now pins
--attention-backend ROCM_AITER_FA, adds piecewise CUDA graphs (--compilation-configmode 3), and conditionally setsVLLM_ROCM_SHUFFLE_KV_CACHE_LAYOUT=1only for TP/EP/concurrency shapes where the AITER ASM paged-attention kernel exists (TP=2 EP=1 CONC≤16; TP=8 EP=8 CONC≤64), avoidingget_heuristic_kernelcrashes at higher concurrency.minimaxm2.5-fp8-mi325x-vllmin.github/configs/amd-master.yamldrops the container image from v0.21.0 to v0.20.2 (recipe-validated; v0.21.0 fails MoE CUDA-graph init on this model).perf-changelog.yamldocuments the rationale.runners/launch_mi325x-amds.shis hardened for CI:set -euo pipefail, per-jobPORTfrom Slurm job ID, EXIT trap forscanceland stalebenchmark_logscleanup, and safersalloc/srunquoting (no fixedPORT=8888).Reviewed by Cursor Bugbot for commit 5471c53. Bugbot is set up for automated code reviews on this repo. Configure here.