RoCE (RDMA over Converged Ethernet) implements RDMA semantics on Ethernet (RoCEv2 uses UDP/IP), so NICs can perform remote memory access with low CPU utilization over the same physical switches many enterprises already operate. Its objective is to deliver InfiniBand-like GPU communication economics—fast NCCL all-reduces, NVMe-oF, llm-d KV moves—without maintaining a separate InfiniBand fabric. RoCE requires lossless Ethernet behavior: Priority Flow Control (PFC), Explicit Congestion Notification (ECN), buffer tuning, and often dedicated traffic classes so RDMA traffic is not dropped under burst load.
RDMA (Remote Direct Memory Access) allows a network adapter to transfer data between the memory of two machines with little CPU overhead, low latency, and often kernel bypass (userspace stacks such as verbs on InfiniBand or RoCE). Its objective in AI infrastructure is to keep GPUs fed and synchronized: distributed training exchanges gradients quickly, disaggregated inference (llm-d) moves KV cache blocks between prefill and decode nodes, and NVMe-oF storage delivers checkpoints without the host spending cycles copying every byte. DPUs and SmartNICs also use RDMA paths for storage and east-west traffic while the host CPU runs models.
InfiniBand is a high-performance network fabric designed for datacenter and HPC clusters, natively supporting RDMA (Remote Direct Memory Access) with low latency, high bandwidth, and features such as adaptive routing and congestion control at the link layer. Its objective in AI is to connect many GPU servers so distributed training (gradient all-reduce via NCCL) and multi-node inference (tensor parallel, llm-d prefill/decode KV transfer) are not limited by TCP overhead on a CPU. InfiniBand NICs (e.g. NVIDIA ConnectX) present verbs APIs; subnets are managed with an Subnet Manager and partitioned for multi-tenant isolation.
