↓ Skip to main content
Le Site de François
  1. Index/

NVLink

Ai Nvlink Nvidia Gpu Interconnect Tensor-Parallelism Training

NVLink is NVIDIA’s proprietary high-speed interconnect between GPUs (and, on some platforms, between GPUs and CPUs) inside a server or across an NVLink switch system (e.g. NVL72-class racks). Its objective is to move tensors—activations, gradients, KV cache shards, or partial attention results—at much higher bandwidth and lower latency than PCIe or general Ethernet, so multi-GPU training and large-model inference (tensor parallelism) are not bottlenecked on the bus. NVLink domains define which GPUs can treat each other’s memory as peer-accessible for CUDA and NCCL without leaving the box.

Architecturally, NVLink is intra-node (or intra-tray) fabric, distinct from InfiniBand/RoCE RDMA that links servers. A CPU still initiates work, but bulk GPU–GPU copies use NVLink while the host PCIe bus carries I/O and smaller control messages. LLMs that exceed one GPU’s HBM are sharded across NVLink-connected devices; decode and prefill latency for tensor-parallel serving depend on all-reduce and attention communication over that link. MIG slices on one physical GPU do not extend NVLink as a multi-GPU pool—they partition a single card. When jobs span nodes, NCCL uses network RDMA; within the node, NVLink dominates.

Red Hat documents multi-GPU RHEL and OpenShift nodes where NVLink topology is exposed to schedulers (NUMA, GPU affinity) and partner reference architectures (NVIDIA DGX, HGX) under Red Hat support matrices. OpenShift AI capacity planning assumes NVLink-aware placement for large models served via vLLM tensor parallel or NIM multi-GPU profiles. Operators verify BIOS, firmware, and driver bundles on certified hardware so NVLink lanes train at expected width—platform validation on RHEL, not a separate Red Hat product.

Related

cuDNN (CUDA Deep Neural Network library)

Ai Cudnn Nvidia Cuda Gpu Deep-Learning Inference Training
cuDNN (CUDA Deep Neural Network library) is NVIDIA’s library of highly optimized GPU kernels for operations that dominate deep learning: convolutions, matrix multiplies used in attention, pooling, normalization (batch/layer), activations, and recurrent cells. Its objective is to deliver near-peak performance on CUDA-capable GPUs without every framework author hand-writing assembly-tuned kernels. PyTorch, TensorFlow, and many inference engines call cuDNN (directly or via cuBLAS) under the hood for training and serving. cuDNN sits between raw CUDA and application code; version alignment with the CUDA toolkit and driver is mandatory for supported deployments.

CUDA (Compute Unified Device Architecture)

Ai Cuda Nvidia Gpu Programming Parallel-Computing
CUDA (Compute Unified Device Architecture) is NVIDIA’s software platform for general-purpose computing on GPUs. Its objective is to give developers a familiar C/C++ (and Fortran, Python bindings) programming model with explicit control over kernels (functions that run on the device), streams (ordered queues of work), and memory spaces (host, device, unified). CUDA sits above the GPU driver and below frameworks such as cuDNN, NCCL, and higher-level ML stacks; it is the layer that makes it practical to implement custom operators, HPC solvers, and inference engines that are not covered by a closed library. For AI, virtually every major training and inference stack ultimately depends on CUDA (or a CUDA-compatible runtime) on NVIDIA hardware.

Confidential GPU

Security Confidential-Computing Gpu Ai Attestation Nvidia
A Confidential GPU is a GPU whose memory, computation state, and data transfers are hardware-encrypted and isolated from the host system — extending the Trusted Execution Environment (TEE) boundary that technologies like TDX and SEV-SNP provide at the CPU level to encompass the GPU accelerator as well. The primary implementation today is NVIDIA Confidential Computing on the Hopper architecture (H100 and later), which encrypts all data resident in GPU High Bandwidth Memory (HBM) using per-context keys managed by the GPU’s on-die security processor. This means that model weights, training data, activations, and intermediate computations are cryptographically protected throughout GPU processing — a host administrator, hypervisor, or co-tenant with DMA access to the PCIe bus sees only ciphertext. The GPU also participates in a dedicated attestation flow: the NVIDIA Remote Attestation Service (NRAS) produces signed evidence that a specific GPU is genuine NVIDIA hardware running in Confidential Computing mode with unmodified firmware, analogous to how Intel DCAP or AMD KDS attest CPU TEEs. This GPU attestation is verified alongside CPU attestation before secrets (model decryption keys, dataset credentials) are released to the combined CPU+GPU TEE. The technology requires no application code changes — existing TensorFlow, PyTorch, and CUDA workloads run unmodified inside the confidential boundary. The primary threat model is the same as CPU-level confidential computing (protecting data-in-use from the infrastructure operator) but applied to the specific risk of AI workloads: model intellectual property theft, training data exfiltration, and inference input/output interception during GPU computation.