↓ Skip to main content
Le Site de François
  1. Index/

NIM (NVIDIA Inference Microservices)

Ai Nim Nvidia Inference Microservices Containers Gpu Openshift-Ai

NIM (NVIDIA Inference Microservices) are container images and Helm charts that deliver ready-to-run inference endpoints for specific models (LLMs, vision, embedding, reranking, and more). The objective is to shrink time-to-production: instead of assembling CUDA drivers, frameworks, model weights, and an OpenAI-compatible server yourself, operators pull a NIM that bundles a performance-tuned engine (often TensorRT-LLM or Triton-backed paths), default model artifacts or download hooks, health checks, and a stable HTTP/gRPC API. NIMs are sized for GPU deployment and target enterprise MLOps teams that want versioned, scannable containers with predictable resource requests rather than bespoke notebooks turned into scripts.

Architecturally, a NIM is not a new chip; it is a packaging and optimization layer above GPU hardware and below your application. Compared with running raw vLLM from source, a NIM trades some flexibility for NVIDIA-validated kernels, default parallelism settings, and curated model manifests. Compared with a CPU inference server, NIM still assumes accelerator memory for weights and KV state—the same fundamental limits as any LLM stack. In Kubernetes, each NIM is typically one Deployment (or Knative-style service) per model SKU, with GPU requests, liveness probes, and horizontal scaling driven by queue depth or latency SLOs. Multi-GPU and multi-node variants depend on the specific NIM profile (tensor parallel size, disaggregation options).

Red Hat supports NIM through its NVIDIA partnership and Red Hat OpenShift AI: documented flows to deploy NIM microservices on OpenShift with GPU operators, RHEL-based nodes, and enterprise support boundaries defined in joint guidance. Customers use the same platform primitives—OpenShift security (SCCs, routes, secrets), RHEL driver stacks, and GitOps—as for other AI workloads, while NVIDIA maintains the NIM image lifecycle and model optimizations. RHEL AI and OpenShift AI reference architectures position NIM alongside open engines like vLLM so teams can choose managed NVIDIA microservices where they fit, and open-source stacks where they need maximum customization—often orchestrated at cluster scale with llm-d when many replicas and smart routing matter.

Related

TensorRT

Ai Tensorrt Nvidia Inference Optimization Nim Gpu Tensorrt-Llm
TensorRT is NVIDIA’s SDK for optimizing and deploying trained neural networks for inference on NVIDIA GPUs. Its objective is minimum latency and maximum throughput: it ingests a model (ONNX, TensorFlow, PyTorch export, or framework-specific parsers), applies graph optimizations (layer fusion, constant folding, kernel autotuning), selects precisions (FP32, FP16, INT8, FP8), and produces a serialized engine executed by a lightweight runtime. For LLMs, TensorRT-LLM extends this with attention-specific fusions, inflight batching, and multi-GPU serving patterns; many NIM microservices bundle TensorRT-LLM–optimized engines rather than raw PyTorch loops.

MIG (Multi-Instance GPU)

Ai Mig Nvidia Gpu Kubernetes Multi-Tenancy Inference
MIG (Multi-Instance GPU) is an NVIDIA GPU partitioning mode on datacenter accelerators (e.g. A100, H100) that splits one physical card into up to seven GPU instances (GIs), each with isolated streaming multiprocessors, memory bandwidth, and HBM capacity. The objective is higher utilization in multi-tenant environments: several smaller models or dev/test workloads share one expensive GPU without time-slicing contention as severe as full-card sharing. Each MIG instance appears to the OS and CUDA as a separate GPU with fixed resources; workloads cannot oversubscribe another instance’s memory. MIG suits inference and modest training more often than massive single-job training that needs the entire GPU and NVLink domain.

Quantization

Ai Quantization Fp8 Int8 Inference Llm Gpu Nim Vllm
Quantization is the process of representing a model’s weights and/or activations with fewer bits than full FP32 training precision—commonly FP16, BF16, FP8, INT8, or INT4 (GPTQ, AWQ, GGUF-style formats). The objective is lower GPU memory (larger models or more concurrent sessions per card), higher throughput, and sometimes faster kernels on hardware with native low-precision units, at the cost of possible quality degradation if pushed too aggressively. Quantization can be applied post-training (calibration on a sample dataset) or during training (quantization-aware training). For inference, serving engines vLLM and NIM load quantized checkpoints and dispatch to vendor libraries (TensorRT-LLM, CUTLASS, etc.) that implement fused low-precision matmuls.