↓ Skip to main content
Le Site de François
  1. Index/

cuDNN (CUDA Deep Neural Network library)

Ai Cudnn Nvidia Cuda Gpu Deep-Learning Inference Training

cuDNN (CUDA Deep Neural Network library) is NVIDIA’s library of highly optimized GPU kernels for operations that dominate deep learning: convolutions, matrix multiplies used in attention, pooling, normalization (batch/layer), activations, and recurrent cells. Its objective is to deliver near-peak performance on CUDA-capable GPUs without every framework author hand-writing assembly-tuned kernels. PyTorch, TensorFlow, and many inference engines call cuDNN (directly or via cuBLAS) under the hood for training and serving. cuDNN sits between raw CUDA and application code; version alignment with the CUDA toolkit and driver is mandatory for supported deployments.

Architecturally, cuDNN targets GPU execution only: the CPU submits operator descriptors (tensor layouts, dtypes, algorithm choices such as Winograd vs implicit GEMM for convolutions) and cuDNN selects or autotunes implementations on the device. Compared with naive CUDA loops, cuDNN exploits tensor cores, fusion opportunities, and memory layout (NCHW vs NHWC) for FP16, BF16, and FP8 where supported. It is not an alternative to NCCL (collectives across GPUs) or TensorRT (full-graph inference compilation)—it provides building blocks. On AMD hardware, ROCm uses MIOpen instead; cuDNN is NVIDIA-specific. LLM stacks increasingly lean on custom attention kernels, but cuDNN and cuBLAS still underpin many layers and legacy CV/NLP paths inside unified frameworks.

Red Hat supports cuDNN indirectly through validated RHEL stacks with NVIDIA drivers and CUDA/cuDNN versions listed in release notes for OpenShift AI, GPU workloads, and partner matrices. Containers for NIM, vLLM, and PyTorch training images pull cuDNN-bearing layers from NVIDIA or framework publishers; Red Hat documents compatible driver/CUDA combinations on certified GPU servers rather than shipping cuDNN as a separate product. Operators treat cuDNN like any CUDA dependency: pin versions in images, test upgrades on staging clusters, and align subscription support with NVIDIA’s and Red Hat’s joint hardware guidance.

Related

ROCm (Radeon Open Compute)

Ai Rocm Amd Gpu Cuda Pytorch Inference Training
ROCm (Radeon Open Compute) is AMD’s software stack for GPU compute on datacenter Instinct accelerators (and select consumer GPUs in community setups). Its objective mirrors CUDA for NVIDIA: provide kernel compilers (HIP), math libraries (rocBLAS, rocFFT), collective communication (RCCL, analogous to NCCL), and framework integrations so PyTorch and inference runtimes can execute training and inference on AMD hardware. ROCm is positioned as an open platform (Linux-first) for customers who want accelerator choice or who standardize on AMD in HPC and AI clusters.

CUDA (Compute Unified Device Architecture)

Ai Cuda Nvidia Gpu Programming Parallel-Computing
CUDA (Compute Unified Device Architecture) is NVIDIA’s software platform for general-purpose computing on GPUs. Its objective is to give developers a familiar C/C++ (and Fortran, Python bindings) programming model with explicit control over kernels (functions that run on the device), streams (ordered queues of work), and memory spaces (host, device, unified). CUDA sits above the GPU driver and below frameworks such as cuDNN, NCCL, and higher-level ML stacks; it is the layer that makes it practical to implement custom operators, HPC solvers, and inference engines that are not covered by a closed library. For AI, virtually every major training and inference stack ultimately depends on CUDA (or a CUDA-compatible runtime) on NVIDIA hardware.

MIG (Multi-Instance GPU)

Ai Mig Nvidia Gpu Kubernetes Multi-Tenancy Inference
MIG (Multi-Instance GPU) is an NVIDIA GPU partitioning mode on datacenter accelerators (e.g. A100, H100) that splits one physical card into up to seven GPU instances (GIs), each with isolated streaming multiprocessors, memory bandwidth, and HBM capacity. The objective is higher utilization in multi-tenant environments: several smaller models or dev/test workloads share one expensive GPU without time-slicing contention as severe as full-card sharing. Each MIG instance appears to the OS and CUDA as a separate GPU with fixed resources; workloads cannot oversubscribe another instance’s memory. MIG suits inference and modest training more often than massive single-job training that needs the entire GPU and NVLink domain.