↓ Skip to main content
Le Site de François
  1. Index/

MIG (Multi-Instance GPU)

Ai Mig Nvidia Gpu Kubernetes Multi-Tenancy Inference

MIG (Multi-Instance GPU) is an NVIDIA GPU partitioning mode on datacenter accelerators (e.g. A100, H100) that splits one physical card into up to seven GPU instances (GIs), each with isolated streaming multiprocessors, memory bandwidth, and HBM capacity. The objective is higher utilization in multi-tenant environments: several smaller models or dev/test workloads share one expensive GPU without time-slicing contention as severe as full-card sharing. Each MIG instance appears to the OS and CUDA as a separate GPU with fixed resources; workloads cannot oversubscribe another instance’s memory. MIG suits inference and modest training more often than massive single-job training that needs the entire GPU and NVLink domain.

Architecturally, MIG is a hardware partition, not a CPU hypervisor: the host still runs Linux and Kubernetes, but the NVIDIA driver exposes multiple minor devices. Kubernetes schedules pods with nvidia.com/mig-1g.5gb-style extended resources via the GPU Operator and device plugin. Compared with whole-GPU assignment, MIG caps per-tenant VRAM and compute—large LLMs may not fit a small profile. Compared with time-slicing, MIG provides stronger isolation and predictable performance. MIG does not replace tensor parallelism across cards; it partitions within one card. NCCL groups are typically scoped per instance, not across MIG slices on the same GPU for multi-GPU jobs.

Red Hat supports MIG on RHEL and OpenShift through documented NVIDIA driver and GPU Operator flows: enabling MIG profiles on nodes, labeling profiles in OpenShift AI, and sizing vLLM/NIM deployments per instance memory. Reference architectures describe multi-tenant inference namespaces where each tenant receives a MIG slice rather than a full H100. Operators must plan profiles at node provisioning time (reboot may be required to change geometry) and align KV cache / model size with the instance’s fixed HBM quota.

Related

NIM (NVIDIA Inference Microservices)

Ai Nim Nvidia Inference Microservices Containers Gpu Openshift-Ai
NIM (NVIDIA Inference Microservices) are container images and Helm charts that deliver ready-to-run inference endpoints for specific models (LLMs, vision, embedding, reranking, and more). The objective is to shrink time-to-production: instead of assembling CUDA drivers, frameworks, model weights, and an OpenAI-compatible server yourself, operators pull a NIM that bundles a performance-tuned engine (often TensorRT-LLM or Triton-backed paths), default model artifacts or download hooks, health checks, and a stable HTTP/gRPC API. NIMs are sized for GPU deployment and target enterprise MLOps teams that want versioned, scannable containers with predictable resource requests rather than bespoke notebooks turned into scripts.

TensorRT

Ai Tensorrt Nvidia Inference Optimization Nim Gpu Tensorrt-Llm
TensorRT is NVIDIA’s SDK for optimizing and deploying trained neural networks for inference on NVIDIA GPUs. Its objective is minimum latency and maximum throughput: it ingests a model (ONNX, TensorFlow, PyTorch export, or framework-specific parsers), applies graph optimizations (layer fusion, constant folding, kernel autotuning), selects precisions (FP32, FP16, INT8, FP8), and produces a serialized engine executed by a lightweight runtime. For LLMs, TensorRT-LLM extends this with attention-specific fusions, inflight batching, and multi-GPU serving patterns; many NIM microservices bundle TensorRT-LLM–optimized engines rather than raw PyTorch loops.

cuDNN (CUDA Deep Neural Network library)

Ai Cudnn Nvidia Cuda Gpu Deep-Learning Inference Training
cuDNN (CUDA Deep Neural Network library) is NVIDIA’s library of highly optimized GPU kernels for operations that dominate deep learning: convolutions, matrix multiplies used in attention, pooling, normalization (batch/layer), activations, and recurrent cells. Its objective is to deliver near-peak performance on CUDA-capable GPUs without every framework author hand-writing assembly-tuned kernels. PyTorch, TensorFlow, and many inference engines call cuDNN (directly or via cuBLAS) under the hood for training and serving. cuDNN sits between raw CUDA and application code; version alignment with the CUDA toolkit and driver is mandatory for supported deployments.