How to Configure GPUDirect RDMA and Prove Multi-Node GPU Performance with NCCL

This article provides a detailed operational runbook for proving GPUDirect RDMA efficiency in multi-node GPU clusters. It emphasizes that a working driver and NCCL test are insufficient; validation must cover GPU topology, GPU-to-NIC affinity, PCIe peer access, IOMMU/ACS behavior, RDMA fabric health, container resource exposure, and NCCL transport selection. The method includes mapping physical topology, testing raw RDMA paths, running nccl-tests with controlled interface selection, and comparing results against a baseline (NVIDIA Network Operator 26.4.0, NCCL 2.30.7). It targets Kubernetes clusters with InfiniBand/RoCE and addresses common symptoms like low bus bandwidth or socket fallback.