↓ Skip to main content
Le Site de François
  1. Index/

InfiniBand

Ai Infiniband Rdma Networking Nccl Distributed-Training Hpc

InfiniBand is a high-performance network fabric designed for datacenter and HPC clusters, natively supporting RDMA (Remote Direct Memory Access) with low latency, high bandwidth, and features such as adaptive routing and congestion control at the link layer. Its objective in AI is to connect many GPU servers so distributed training (gradient all-reduce via NCCL) and multi-node inference (tensor parallel, llm-d prefill/decode KV transfer) are not limited by TCP overhead on a CPU. InfiniBand NICs (e.g. NVIDIA ConnectX) present verbs APIs; subnets are managed with an Subnet Manager and partitioned for multi-tenant isolation.

Architecturally, InfiniBand differs from Ethernet in purpose-built RDMA semantics and historically simpler lossless delivery within a well-designed fabric—though RoCE brings RDMA to Ethernet for customers who want converged networks. A CPU is not in the hot path for large transfers once queues are posted; GPUs or their NIC peers move data directly. Compared with NVLink inside a server, InfiniBand is the east-west cluster plane. Bandwidth planning (HDR, NDR, XDR generations), cable/plan topology, and PKey or tenant isolation are core ops skills. Kubernetes passes IB devices to training or inference pods via device plugins and tuned CNIs on OpenShift.

Red Hat supports InfiniBand on RHEL (drivers, IPoIB where needed, RDMA core) and documents HPC/AI cluster networking for OpenShift and bare metal. Telco and AI reference architectures on Red Hat stacks assume IB or high-end RoCE for scale-out training and for llm-d-style disaggregation. Red Hat integrates the OS and orchestration layer; Mellanox/NVIDIA and switch vendors document certified leaf-spine designs. Customers run the same SELinux-hardened RHEL on GPU nodes whether the fabric is IB or RoCE.

Related

RDMA (Remote Direct Memory Access)

Ai Rdma Networking Infiniband Roce Gpu Dpu Distributed-Training
RDMA (Remote Direct Memory Access) allows a network adapter to transfer data between the memory of two machines with little CPU overhead, low latency, and often kernel bypass (userspace stacks such as verbs on InfiniBand or RoCE). Its objective in AI infrastructure is to keep GPUs fed and synchronized: distributed training exchanges gradients quickly, disaggregated inference (llm-d) moves KV cache blocks between prefill and decode nodes, and NVMe-oF storage delivers checkpoints without the host spending cycles copying every byte. DPUs and SmartNICs also use RDMA paths for storage and east-west traffic while the host CPU runs models.

RoCE (RDMA over Converged Ethernet)

Ai Roce Rdma Ethernet Networking Nccl Distributed-Training
RoCE (RDMA over Converged Ethernet) implements RDMA semantics on Ethernet (RoCEv2 uses UDP/IP), so NICs can perform remote memory access with low CPU utilization over the same physical switches many enterprises already operate. Its objective is to deliver InfiniBand-like GPU communication economics—fast NCCL all-reduces, NVMe-oF, llm-d KV moves—without maintaining a separate InfiniBand fabric. RoCE requires lossless Ethernet behavior: Priority Flow Control (PFC), Explicit Congestion Notification (ECN), buffer tuning, and often dedicated traffic classes so RDMA traffic is not dropped under burst load.

NCCL (NVIDIA Collective Communications Library)

Ai Nccl Nvidia Distributed-Training Gpu Infiniband Roce Cuda
NCCL (NVIDIA Collective Communications Library) implements collective operations—all-reduce, broadcast, reduce-scatter, all-gather, and others—optimized for NVIDIA GPUs across NVLink within a node and RDMA (InfiniBand or RoCE) across nodes. Its objective in AI is to make distributed training and multi-GPU inference (tensor parallelism) scale: gradient shards must merge every step; attention and MLP partitions must exchange activations with minimal latency. Frameworks (PyTorch DDP/FSDP, vLLM tensor parallel) call NCCL (or delegate to it) rather than hand-rolling socket code.