Distributed-Training on Le Site de François

InfiniBand

Mon, 01 Jan 0001 00:00:00 +0000

InfiniBand is a high-performance network fabric designed for datacenter and HPC clusters, natively supporting RDMA (Remote Direct Memory Access) with low latency, high bandwidth, and features such as adaptive routing and congestion control at the link layer. Its objective in AI is to connect many GPU servers so distributed training (gradient all-reduce via NCCL) and multi-node inference (tensor parallel, llm-d prefill/decode KV transfer) are not limited by TCP overhead on a CPU. InfiniBand NICs (e.g. NVIDIA ConnectX) present verbs APIs; subnets are managed with an Subnet Manager and partitioned for multi-tenant isolation.

NCCL (NVIDIA Collective Communications Library)

Mon, 01 Jan 0001 00:00:00 +0000

NCCL (NVIDIA Collective Communications Library) implements collective operations—all-reduce, broadcast, reduce-scatter, all-gather, and others—optimized for NVIDIA GPUs across NVLink within a node and RDMA (InfiniBand or RoCE) across nodes. Its objective in AI is to make distributed training and multi-GPU inference (tensor parallelism) scale: gradient shards must merge every step; attention and MLP partitions must exchange activations with minimal latency. Frameworks (PyTorch DDP/FSDP, vLLM tensor parallel) call NCCL (or delegate to it) rather than hand-rolling socket code.

RDMA (Remote Direct Memory Access)

Mon, 01 Jan 0001 00:00:00 +0000

RDMA (Remote Direct Memory Access) allows a network adapter to transfer data between the memory of two machines with little CPU overhead, low latency, and often kernel bypass (userspace stacks such as verbs on InfiniBand or RoCE). Its objective in AI infrastructure is to keep GPUs fed and synchronized: distributed training exchanges gradients quickly, disaggregated inference (llm-d) moves KV cache blocks between prefill and decode nodes, and NVMe-oF storage delivers checkpoints without the host spending cycles copying every byte. DPUs and SmartNICs also use RDMA paths for storage and east-west traffic while the host CPU runs models.

RoCE (RDMA over Converged Ethernet)

Mon, 01 Jan 0001 00:00:00 +0000

RoCE (RDMA over Converged Ethernet) implements RDMA semantics on Ethernet (RoCEv2 uses UDP/IP), so NICs can perform remote memory access with low CPU utilization over the same physical switches many enterprises already operate. Its objective is to deliver InfiniBand-like GPU communication economics—fast NCCL all-reduces, NVMe-oF, llm-d KV moves—without maintaining a separate InfiniBand fabric. RoCE requires lossless Ethernet behavior: Priority Flow Control (PFC), Explicit Congestion Notification (ECN), buffer tuning, and often dedicated traffic classes so RDMA traffic is not dropped under burst load.

Training

Mon, 01 Jan 0001 00:00:00 +0000

Training is the phase of machine learning where model parameters are adjusted to minimize a loss on a dataset. For deep learning, that means repeated forward passes (compute predictions), backward passes (propagate gradients via autodiff), and optimizer steps (update weights)—from scratch pretraining, continued pretraining, or fine-tuning (full, LoRA, or other parameter-efficient methods). The objective is model quality (accuracy, perplexity, task metrics) within a compute and time budget, not millisecond response to end users. Training jobs are batch-oriented: large minibatches, epochs over terabytes of tokens or images, checkpointing to durable storage, and experiment tracking. LLM training at scale uses distributed strategies—data parallel, tensor parallel, pipeline parallel, and expert parallel for MoE—coordinated by frameworks such as PyTorch with FSDP or DeepSpeed.