Conversation
luhenry
pushed a commit
that referenced
this pull request
Sep 19, 2026
luhenry
changed the title
Contributor
|
Add build-vllm.yml and docs/packages/vllm.yaml for vLLM 0.29.0+cpu, the VLLM_TARGET_DEVICE=cpu build of vllm-project/vllm. vLLM's CPU backend already has first-class riscv64 support upstream: a riscv64 branch in cmake/cpu_extension.cmake, RVV kernels under csrc/cpu (cpu_types_riscv*.hpp, sgl-kernels/gemm_int4.cpp, cpu_wna16.cpp), a CpuArchEnum.RISCV platform, and platform_machine == "riscv64" markers in requirements/cpu.txt that already pin torch==2.13.0 - exactly the version this registry publishes as 2.13.0+cpu. The wheel is built at the rv64gc baseline (-DVLLM_RVV_VLEN=0, upstream's documented scalar RISC-V mode) rather than letting cmake auto-detect VLEN from the runner's /proc/cpuinfo, which would bake that runner's vector extensions into a wheel shipped to every riscv64 user.
cibuildwheel's before-build cwd is not guaranteed to be the package directory when package-dir points at a subdirectory, so anchor the requirements path explicitly.
csrc/cpu/utils.cpp includes <numa.h> and cmake/cpu_extension.cmake links libnuma unconditionally (ENABLE_NUMA defaults to TRUE and is only turned off for Apple), so the CPU extension does not compile without the NUMA development headers. Upstream's docker/Dockerfile.cpu installs libnuma-dev for exactly this; the manylinux image is Rocky 10, where the same headers come from numactl-devel in appstream.
The wheel built, but importing it failed with ImportError: vllm/_C.abi3.so: undefined symbol: sbgemm_ cpu_extension.cmake turns on the OpenBLAS bf16 GEMM path whenever the installed torch wheel contains a libopenblas*.so*, and _C resolves the symbol through libtorch.so rather than linking OpenBLAS, so an OpenBLAS without sbgemm_ links cleanly and only fails at import. The riscv64 torch 2.13.0+cpu wheel ships torch/lib/libopenblas.so.0 built without BUILD_BFLOAT16: it exports sgemm_ but no sbgemm_ and no bf16 symbol at all. Patch cmake to probe the library for the symbol and fall back to blas_gemm.h's existing PyTorch reference path, whose at::native::cpublas::gemm_no_downcast_stub libtorch_cpu.so does export.
Upstream's docker/Dockerfile.cpu builds its CPU wheel with "setup.py bdist_wheel --py-limited-api=cp38"; cibuildwheel passes no such flag, so our first green run produced vllm-0.29.0+cpu-cp312-cp312-manylinux_2_39_riscv64.whl where upstream publishes cp38-abi3 - one interpreter served instead of every one from cp312 up, while this registry's torch 2.13.0+cpu covers cp312, cp313 and cp314. The wheel is genuinely stable-ABI already: CMakeExtension passes py_limited_api (cmake builds the C extensions USE_SABI 3, hence _C.abi3.so) and the PyO3 extension is declared with features=["pyo3/abi3-py38"] and py_limited_api=True. Only the wheel tag was missing, which is gotcha 11's setuptools-rust case. Tagged cp312 rather than upstream's cp38 because torch 2.13.0 has no riscv64 wheel below cp312, so cp312 is the oldest interpreter this can ever be built on and a cp38 tag would claim interpreters the wheel was not compiled against (gotcha 96).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
vllm0.29.0+cpuCompiles vLLM's CPU inference backend (
vllm._Cand the PyO3 frontend) against thetorch2.13.0+cpu wheels this registry already publishes. Upstream publishes no riscv64 wheel.Mirrors upstream's
docker/Dockerfile.cpu, vLLM's only CPU wheel build.Differs from upstream
-DVLLM_RVV_VLEN=0- auto-detection would bake the runner's own VLEN into every wheel.--py-limited-api=cp312where upstream passescp38.torch==2.13.0has no riscv64 wheel below cp312, so cp312 is the oldest interpreter this wheel can be built on and acp38tag would claim interpreters it was never compiled against.numactl-develinstalled in-container - upstream's Debian-based image'slibnuma-devunder its Rocky name.Matrix: one
cp312build producing onecp312-abi3wheel, which serves cp312, cp313 and cp314 - the interpreters this registry'storch2.13.0+cpu covers. The C extensions are builtUSE_SABI 3and the PyO3 extension declaresfeatures=["pyo3/abi3-py38"], so the wheel is genuinely stable-ABI; upstream gets the tag by passing--py-limited-apitobdist_wheel, which cibuildwheel does not do on its own. No free-threaded leg, matching upstream, which publishes nocp314twheel (andpy_limited_apiis off underPy_GIL_DISABLEDby construction).Testing: imports
vllmandvllm._Cfrom the installed wheel and reports the detected CPU architecture; upstream's own suites need model downloads. The import is not a formality here - it is what caught thesbgemm_gap below, which the build itself cannot see.License: OK
Patches
0001-requirements-do-not-require-numba-on-riscv64.patch-Upstream-Status: To upstream. Without it the wheel is buildable but not installable:requirements/cpu.txtrequires numba everywhere but s390x, and numba has no riscv64 wheel and cannot be built from sdist (llvmlite needs a conda-only patched LLVM with no linux-riscv64 subdir). numba is optional to vLLM by construction (is_numba_available()), and this is the configuration upstream already ships for s390x. riscv64-only.0002-cpu-probe-the-torch-openblas-for-sbgemm.patch-Upstream-Status: To upstream.cpu_extension.cmakedefinesVLLM_HAS_OPENBLASwhenever the installed torch wheel contains alibopenblas*.so*, andblas_gemm()then callssbgemm_. Our riscv64torch2.13.0+cpu does shiptorch/lib/libopenblas.so.0, but built withoutBUILD_BFLOAT16: it exportssgemm_and nosbgemm_at all, because OpenBLAS only has bf16 kernels for x86_64 Cooper Lake and later, ARM64 Neoverse, POWER10 and z14 and later. Since_Cdeliberately does not link OpenBLAS and expectslibtorch.soto have loaded itRTLD_GLOBAL, the extension links cleanly and fails only onimport vllm._Cwithundefined symbol: sbgemm_. The patch probes the library for the symbol and falls back to the PyTorch reference pathblas_gemm.halready carries for this case (at::native::cpublas::gemm_no_downcast_stub, whichlibtorch_cpu.sodoes export). Architecture-agnostic, and it keeps current behaviour wherever the probe cannot run, so no working platform changes.