[BUG] builtin.cmake + BUILD_SHARED_LIBS + PyTorch backend: undefined reference to torch/c10 when linking lmp

[bio-punk]

Bug summary

Title

builtin.cmake + BUILD_SHARED_LIBS=yes + PyTorch backend: linking lmp fails with undefined references from libdeepmd_cc.so to libtorch/libc10

---

Environment

• deepmd-kit: v3.1.2 (built from source, PyTorch backend)
• Backend: PyTorch only (ENABLE_PYTORCH=ON, ENABLE_TENSORFLOW=OFF, USE_PT_PYTHON_LIBS=ON)
• PyTorch: 2.10.0+cu126 (pip wheel, _GLIBCXX_USE_CXX11_ABI=True)
• LAMMPS: stable_29Aug2024
• Integration: built-in mode via source/lmp/builtin.cmake
• LAMMPS CMake flags: BUILD_SHARED_LIBS=yes, CMAKE_INSTALL_PREFIX=<same prefix as deepmd>
• OS: Linux (Apptainer container, GCC 13)(NGC pytorch-24.11-py3)

---

Steps to Reproduce

1. Build and install deepmd-kit C++ interface with PyTorch into $PREFIX (e.g. venv).
2. Patch LAMMPS cmake/CMakeLists.txt:

   include(/path/to/deepmd-kit/source/lmp/builtin.cmake)
   

DeePMD-kit Version

3.1.2

Backend and its version

pytorch2.10+cu126

How did you download the software?

Built from source

Input Files, Running Commands, Error Log, etc.

#!/bin/bash

DEEPMD_SRC=
LAMMPS_SRC=
VENV_PATH=
export CUDA_ROOT=/usr/local/cuda
export TMPDIR=/dev/shm
source $VENV_PATH/bin/activate

# 设置lammps的CMakeLists.txt，添加deepmd-kit的builtin.cmake
grep "include(${DEEPMD_SRC}/source/lmp/builtin.cmake)" ${LAMMPS_SRC}/cmake/CMakeLists.txt
if [ $? -ne 0 ]; then
    echo "include(${DEEPMD_SRC}/source/lmp/builtin.cmake)" >> ${LAMMPS_SRC}/cmake/CMakeLists.txt
    echo "Set CmakeLists.txt of lammps succeeded"
else
    echo "No need to set CmakeLists.txt of lammps"
fi

# 设置torch的cuda架构列表,使用v100的cuda架构
export TORCH_CUDA_ARCH_LIST="7.0"

TORCH_DIR=$(python -c "import torch; print(torch.utils.cmake_prefix_path)")

cd $LAMMPS_SRC
if [ -d build ]; then
    rm -rf build
    echo "Remove build directory"
fi
mkdir -p build
cd build
cmake \
    -D CMAKE_BUILD_TYPE=Debug \
    -D CMAKE_INSTALL_PREFIX=$VENV_PATH \
    -D CMAKE_PREFIX_PATH="${TORCH_DIR};${CUDA_ROOT};${VENV_PATH}"  \
    -D CMAKE_CXX_STANDARD=17 \
    -D CMAKE_CUDA_STANDARD=17 \
    -D CMAKE_CXX_COMPILER=${LAMMPS_SRC}/lib/kokkos/bin/nvcc_wrapper \
    -D CMAKE_MPI_C_COMPILER=mpicc \
    -D CMAKE_MPI_CXX_COMPILER=mpicxx \
    -D BUILD_MPI=ON \
    -D FFT=KISS \
    -D PKG_MEAM=yes \
    -D PKG_REAXFF=yes \
    -D GPU_API=CUDA \
    -D CUDA_TOOLKIT_ROOT_DIR=$CUDA_ROOT \
    -D CUDA_ARCHITECTURES=70 \
    -D GPU_ARCH=sm_70 \
    -D CUDAToolkit_ROOT=$CUDA_ROOT \
    -D CUDA_TOOLKIT_ROOT_DIR=$CUDA_ROOT \
    -D Kokkos_ENABLE_CUDA=ON \
    -D Kokkos_ARCH_VOLTA70=ON \
    -D Kokkos_ENABLE_CUDA_LAMBDA=ON \
    -D BUILD_SHARED_LIBS=yes \
${LAMMPS_SRC}/cmake

make -j10 VERBOSE=1
if [ $? -ne 0 ]; then
    echo "Make lammps failed"
    exit 1
else
    echo "Make lammps succeeded"
    make install
    echo "Install lammps at $VENV_PATH"
fi

Steps to Reproduce

Steps to Reproduce

0. Prerequisites

• Linux with CMake ≥ 3.25, MPI, CUDA toolkit
• Python venv with PyTorch installed (pip install torch, cxx11 ABI)
• deepmd-kit v3.1.2 source and LAMMPS stable_29Aug2024 source

Verify PyTorch CMake path:

source $PREFIX/bin/activate
python -c "import torch; print(torch.__version__)"
python -c "import torch; print(torch._C._GLIBCXX_USE_CXX11_ABI)"   # expect True
export TORCH_DIR=$(python -c "import torch; print(torch.utils.cmake_prefix_path)")
echo $TORCH_DIR

### Further Information, Files, and Links

PyTorch 后端 + CMake builtin + `BUILD_SHARED_LIBS=yes` 时，`builtin.cmake` 只 PUBLIC 链接 `DeePMD::deepmd_c`，而 `deepmd_c→deepmd_cc→torch` 均为 PRIVATE，导致链 `lmp` 时 `libdeepmd_cc.so` 缺少 `libc10`/`libtorch` 符号；需在 LAMMPS 侧显式 `target_link_libraries(lammps PUBLIC ${TORCH_LIBRARIES})` 或由 upstream 传递链接。

https://github.com/deepmodeling/deepmd-kit/discussions/2768
https://github.com/deepmodeling/deepmd-kit/discussions/3568
https://github.com/deepmodeling/deepmd-kit/discussions/340

[dosubot]

Hi @bio-punk! I'm Dosu and I'm helping the DeepModeling team.

You've correctly identified the root cause. In builtin.cmake, the LAMMPS target links only to DeePMD::deepmd_c:

target_link_libraries(lammps PUBLIC DeePMD::deepmd_c)

And in source/api_cc/CMakeLists.txt, deepmd_cc links to ${TORCH_LIBRARIES} with PRIVATE visibility:

if(ENABLE_PYTORCH AND "${OP_CXX_ABI_PT}" EQUAL "${OP_CXX_ABI}")
  target_link_libraries(${libname} PRIVATE "${TORCH_LIBRARIES}")

With BUILD_SHARED_LIBS=yes, the linker needs to resolve all symbols in libdeepmd_cc.so at link time, but since torch is PRIVATE it's not propagated to the LAMMPS target.

The PRIVATE linking is intentional—it treats PyTorch as an implementation detail. However, with shared libraries the dynamic linker still needs to find libtorch/libc10 at link time.

Workaround: The INSTALL_RPATH on libdeepmd_cc is set to $ORIGIN;${BACKEND_LIBRARY_PATH} [1], which should allow runtime discovery. The issue is that at link time of lmp, the linker can't resolve symbols from the transitive PRIVATE dependencies. You can work around this by adding the torch library path to your LAMMPS CMake invocation:

# Add after the find_package(DeePMD) call in builtin.cmake, or in your build script:
find_package(Torch REQUIRED)
target_link_libraries(lammps PUBLIC ${TORCH_LIBRARIES})

Or set LD_LIBRARY_PATH to include the torch lib directory and use -Wl,--allow-shlib-undefined if your linker supports it.

Regarding a proper fix: The cleanest upstream solution would be for builtin.cmake to either make torch a PUBLIC/INTERFACE dependency when BUILD_SHARED_LIBS is ON, or to explicitly find_package(Torch) and link it when the PyTorch backend is detected. PR #4728 [2] and PR #5475 [3] have improved related aspects (torch.libs discovery and CMake config installation), but they don't fully address this specific PRIVATE→shared lib propagation issue in the builtin mode.

Would you like me to search for any open PRs specifically targeting this PRIVATE linking propagation issue, or would you prefer guidance on crafting a minimal patch to builtin.cmake?

To reply, just mention @dosu.

---

Share context across your team and agents. Try Dosu.

[bio-punk]

@dosu.

RPATH is a good solution

Another way to fix this problem by

TORCH_LINK_CMAKE=${LAMMPS_SRC}/cmake/deepmd_torch_link.cmake
cat > ${TORCH_LINK_CMAKE} << 'EOF'
find_package(Torch REQUIRED)
target_link_libraries(lammps PUBLIC ${TORCH_LIBRARIES})
EOF
grep -q 'deepmd_torch_link.cmake' ${LAMMPS_SRC}/cmake/CMakeLists.txt
if [ $? -ne 0 ]; then
    echo "include(${LAMMPS_SRC}/cmake/deepmd_torch_link.cmake)" >> ${LAMMPS_SRC}/cmake/CMakeLists.txt
    echo "Set deepmd_torch_link.cmake of lammps succeeded"
else
    echo "No need to set deepmd_torch_link.cmake of lammps"
fi

[njzjz]

Fixed by the backend-plugin refactor in #5531, released in v3.2.0.

The C/C++ interface libraries are now backend-neutral: libdeepmd_c.so links to libdeepmd_cc.so, while PyTorch and c10 symbols are isolated in the runtime-loaded libdeepmd_backend_pt.so. Consequently, a shared LAMMPS build no longer needs to link Torch merely to satisfy transitive symbols from libdeepmd_cc.so.

I verified current master (b32f74c81) by building the C/C++ interface with PyTorch enabled, inspecting the dynamic dependencies, and linking against libdeepmd_c.so with --no-allow-shlib-undefined. The link succeeds without adding Torch to the consumer target.

Please upgrade from v3.1.2 to v3.2.0 or later. Closing the now-obsolete implementation PR #5535 as well.

Coding agent: Codex
Codex version: codex-cli 0.151.0
Model: gpt-5.6-sol
Reasoning effort: xhigh