Add nccl support for tensor parrallel

[zyang-dev]

Overview

Add a backend interface for collectives across ranks running in separate processes, implement it in the CUDA backend using NCCL, and use it in the RPC server when available.

The implementation is on top of PR #26610.

Results on DGX spark:
single node:

./build/bin/llama-bench -hf unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_M -ngl 99 -d 0,8192 -r 3
| model                          |       size |     params | backend    | ngl |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --------------: | -------------------: |
| qwen35 27B Q4_K - Medium       |  15.32 GiB |    27.32 B | CUDA       |  99 |           pp512 |       846.54 ± 18.78 |
| qwen35 27B Q4_K - Medium       |  15.32 GiB |    27.32 B | CUDA       |  99 |           tg128 |         13.02 ± 0.01 |
| qwen35 27B Q4_K - Medium       |  15.32 GiB |    27.32 B | CUDA       |  99 |   pp512 @ d8192 |       801.23 ± 10.67 |
| qwen35 27B Q4_K - Medium       |  15.32 GiB |    27.32 B | CUDA       |  99 |   tg128 @ d8192 |         12.64 ± 0.01 |

Two nodes with NCCL:

./build/bin/llama-bench -hf unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_M -ngl 99 -sm tensor  --rpc NODE-1-IP:50052,NODE-2-IP:50052 -dev RPC0/RPC1 -d 0,8192 -r 3
| model                          |       size |     params | backend    | ngl |     sm | dev          |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | -----: | ------------ | --------------: | -------------------: |
| qwen35 27B Q4_K - Medium       |  15.32 GiB |    27.32 B | CUDA,RPC   |  99 | tensor | RPC0/RPC1    |           pp512 |      1348.29 ± 38.62 |
| qwen35 27B Q4_K - Medium       |  15.32 GiB |    27.32 B | CUDA,RPC   |  99 | tensor | RPC0/RPC1    |           tg128 |         23.51 ± 0.19 |
| qwen35 27B Q4_K - Medium       |  15.32 GiB |    27.32 B | CUDA,RPC   |  99 | tensor | RPC0/RPC1    |   pp512 @ d8192 |      1282.35 ± 33.41 |
| qwen35 27B Q4_K - Medium       |  15.32 GiB |    27.32 B | CUDA,RPC   |  99 | tensor | RPC0/RPC1    |   tg128 @ d8192 |         22.86 ± 0.10 |

Two nodes with pairwise:

./build/bin/llama-bench -hf unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_M -ngl 99 -sm tensor  --rpc NODE-1-IP:50052,NODE-2-IP:50052 -dev RPC0/RPC1 -d 0,8192 -r 3
| model                          |       size |     params | backend    | ngl |     sm | dev          |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | -----: | ------------ | --------------: | -------------------: |
| qwen35 27B Q4_K - Medium       |  15.32 GiB |    27.32 B | CUDA,RPC   |  99 | tensor | RPC0/RPC1    |           pp512 |       961.61 ± 17.51 |
| qwen35 27B Q4_K - Medium       |  15.32 GiB |    27.32 B | CUDA,RPC   |  99 | tensor | RPC0/RPC1    |           tg128 |         22.33 ± 0.17 |
| qwen35 27B Q4_K - Medium       |  15.32 GiB |    27.32 B | CUDA,RPC   |  99 | tensor | RPC0/RPC1    |   pp512 @ d8192 |       911.15 ± 53.39 |
| qwen35 27B Q4_K - Medium       |  15.32 GiB |    27.32 B | CUDA,RPC   |  99 | tensor | RPC0/RPC1    |   tg128 @ d8192 |         22.14 ± 0.18 |

Ratio to a single node:

┌───────────────┬──────────────────────┬───────┐
│               │ pairwise (PR #26610) │ NCCL  │
├───────────────┼──────────────────────┼───────┤
│ pp512         │                1.14x │ 1.59x │
├───────────────┼──────────────────────┼───────┤
│ tg128         │                1.72x │ 1.81x │
├───────────────┼──────────────────────┼───────┤
│ pp512 @ d8192 │                1.14x │ 1.60x │
├───────────────┼──────────────────────┼───────┤
│ tg128 @ d8192 │                1.75x │ 1.81x │
└───────────────┴──────────────────────┴───────┘

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: Yes, used claude

[am17an]

Do you have any llama-bench results?

[JohannesGaessler]

Without having looked at any of the other code, changes to the ggml backend API like this should be avoided if at all possible.

[zyang-dev]

Added llama-bench results.

Removed ggml_backend_comm_allreduce_rank_t and now reuse ggml_backend_comm_allreduce_tensor for per-rank communicators.

@am17an , @JohannesGaessler, do you have any suggestions for handling ggml_backend_comm_get_unique_id_t and ggml_backend_comm_init_rank_t?

[am17an]

what's the reason why using NCCL is so much faster for PP vs TG? Maybe there is something to optimize in the generic path too

[zyang-dev]

what's the reason why using NCCL is so much faster for PP vs TG? Maybe there is something to optimize in the generic path too

@am17an,
Decode communicates far fewer tensor elements per collective than prefill. This explains why NCCL and pairwise have similar decode performance, while NCCL is significantly faster during prefill.

I tested three optimizations on the pairwise path:

1. Duplex exchange (ggml-rpc.cpp). Previously, rank 0 sent its entire payload before receiving, leaving one direction of the full-duplex link idle during each transfer phase. Changed comm_allreduce_pairwise to exchange chunks, with both ranks sending a chunk and then receiving, allowing traffic in both directions concurrently.

2. Deeper send queue (transport.cpp). Previously, rdma_send posted one 256 KiB work request and waited for completion before posting the next. Changed it to allow up to four outstanding sends, reducing gaps between transfers. Increasing the depth further to eight did not improve performance.

3. Direct registered buffers. Previously, the data path was GPU → application host buffer → RDMA host buffer → network, with the reverse copies on receive. Replaced the intermediate buffers with persistent, pinned host buffers registered directly with RDMA: GPU → registered host buffer → network. This eliminated the extra host-to-host copies while retaining GPU ↔ host transfers.

Further improvement likely requires restructuring the collective into an overlapping pipeline of GPU transfers, network communication, and reduction.

┌─────────────────────────────┬─────────────────┬────────────────┐
│                             │      pp512      │ vs single node │
├─────────────────────────────┼─────────────────┼────────────────┤
│ pairwise (PR #26610)        │  961.61 ± 17.51 │          1.14x │
├─────────────────────────────┼─────────────────┼────────────────┤
│ + duplex exchange           │ 1057.67 ± 12.13 │          1.25x │
├─────────────────────────────┼─────────────────┼────────────────┤
│ + deep send queue           │  1144.75 ± 9.24 │          1.35x │
├─────────────────────────────┼─────────────────┼────────────────┤
│ + direct registered buffers │ 1213.09 ± 36.06 │          1.43x │
├─────────────────────────────┼─────────────────┼────────────────┤
│ NCCL                        │ 1348.29 ± 38.62 │          1.59x │
└─────────────────────────────┴─────────────────┴────────────────┘

[am17an]

Ah okay nice these can be even added to the original PR I think. @ggerganov perhaps it may help in the metal speedup. Do you have a branch with these changes?

[mamama1]

errr I'm very confused, I've recently acquired a third 5060 Ti and started using tensor parallelism with NCCL enabled. Enabling NCCL doubled decode - I think prefill also increased significantly.

I admit that I don't fully understand (or barely understand) the mechanics underneath - that's why the following question came to my mind when reading this PR:

why does the PR say "Add nccl support for tensor parrallel" if that's something that apparently is already working?

thanks

[am17an]

This PR is for NCCL support over RDMA, not related to your setup. @zyang-dev perhaps change the title to reflect it better