↓ Skip to main content
Le Site de François
  1. Index/

NCCL (NVIDIA Collective Communications Library)

Ai Nccl Nvidia Distributed-Training Gpu Infiniband Roce Cuda

NCCL (NVIDIA Collective Communications Library) implements collective operations—all-reduce, broadcast, reduce-scatter, all-gather, and others—optimized for NVIDIA GPUs across NVLink within a node and RDMA (InfiniBand or RoCE) across nodes. Its objective in AI is to make distributed training and multi-GPU inference (tensor parallelism) scale: gradient shards must merge every step; attention and MLP partitions must exchange activations with minimal latency. Frameworks (PyTorch DDP/FSDP, vLLM tensor parallel) call NCCL (or delegate to it) rather than hand-rolling socket code.

Architecturally, NCCL selects transport based on topology discovery: NVLink peer copies inside the server, then network RDMA between servers, with CUDA streams overlapping communication and compute where possible. The CPU launches NCCL kernels and progress threads but avoids copying full tensors through host memory on the hot path. AMD stacks use RCCL instead; NCCL is NVIDIA-specific. Performance depends on NCCL environment variables, NIC count, rail-optimized topology, and whether traffic shares congested Ethernet without lossless QoS. NCCL is not a user-facing API for app developers but is mandatory plumbing for large LLM jobs.

Red Hat supports NCCL indirectly through validated RHEL + NVIDIA driver + CUDA stacks on GPU servers and OpenShift AI training/inference guides. Multi-node job manifests (MPI, PyTorch operator, Slurm on RHEL HPC images) assume NCCL-over-IB/RoCE is correctly wired. Red Hat documentation emphasizes node labeling, huge pages, and driver versions from the support matrix; NVIDIA ships NCCL releases tied to CUDA versions. llm-d and vLLM multi-node serving likewise depend on healthy NCCL when tensor parallel spans hosts.

Related

RDMA (Remote Direct Memory Access)

Ai Rdma Networking Infiniband Roce Gpu Dpu Distributed-Training
RDMA (Remote Direct Memory Access) allows a network adapter to transfer data between the memory of two machines with little CPU overhead, low latency, and often kernel bypass (userspace stacks such as verbs on InfiniBand or RoCE). Its objective in AI infrastructure is to keep GPUs fed and synchronized: distributed training exchanges gradients quickly, disaggregated inference (llm-d) moves KV cache blocks between prefill and decode nodes, and NVMe-oF storage delivers checkpoints without the host spending cycles copying every byte. DPUs and SmartNICs also use RDMA paths for storage and east-west traffic while the host CPU runs models.

CUDA (Compute Unified Device Architecture)

Ai Cuda Nvidia Gpu Programming Parallel-Computing
CUDA (Compute Unified Device Architecture) is NVIDIA’s software platform for general-purpose computing on GPUs. Its objective is to give developers a familiar C/C++ (and Fortran, Python bindings) programming model with explicit control over kernels (functions that run on the device), streams (ordered queues of work), and memory spaces (host, device, unified). CUDA sits above the GPU driver and below frameworks such as cuDNN, NCCL, and higher-level ML stacks; it is the layer that makes it practical to implement custom operators, HPC solvers, and inference engines that are not covered by a closed library. For AI, virtually every major training and inference stack ultimately depends on CUDA (or a CUDA-compatible runtime) on NVIDIA hardware.

InfiniBand

Ai Infiniband Rdma Networking Nccl Distributed-Training Hpc
InfiniBand is a high-performance network fabric designed for datacenter and HPC clusters, natively supporting RDMA (Remote Direct Memory Access) with low latency, high bandwidth, and features such as adaptive routing and congestion control at the link layer. Its objective in AI is to connect many GPU servers so distributed training (gradient all-reduce via NCCL) and multi-node inference (tensor parallel, llm-d prefill/decode KV transfer) are not limited by TCP overhead on a CPU. InfiniBand NICs (e.g. NVIDIA ConnectX) present verbs APIs; subnets are managed with an Subnet Manager and partitioned for multi-tenant isolation.