↓ Skip to main content
Le Site de François
  1. Index/

TensorRT

Ai Tensorrt Nvidia Inference Optimization Nim Gpu Tensorrt-Llm

TensorRT is NVIDIA’s SDK for optimizing and deploying trained neural networks for inference on NVIDIA GPUs. Its objective is minimum latency and maximum throughput: it ingests a model (ONNX, TensorFlow, PyTorch export, or framework-specific parsers), applies graph optimizations (layer fusion, constant folding, kernel autotuning), selects precisions (FP32, FP16, INT8, FP8), and produces a serialized engine executed by a lightweight runtime. For LLMs, TensorRT-LLM extends this with attention-specific fusions, inflight batching, and multi-GPU serving patterns; many NIM microservices bundle TensorRT-LLM–optimized engines rather than raw PyTorch loops.

Architecturally, TensorRT shifts work from interpretive framework execution on the CPU orchestrating many small GPU kernels to a compiled plan tuned for specific GPU SKUs and batch shapes. Build time can be minutes (engine compilation is shape- and profile-sensitive); runtime is fast but less flexible than vLLM’s fully dynamic Python path unless built with dynamic shape profiles. Compared with cuDNN alone (operator library), TensorRT owns the whole graph and memory scheduling. Compared with CPU inference, TensorRT still requires GPU VRAM for weights and KV cache; optimizations do not remove LLM memory laws. Rebuilding engines is required when models, GPUs, or precision policies change.

Red Hat customers deploy TensorRT inside NVIDIA NIM containers and partner images on OpenShift AI and RHEL GPU nodes—Red Hat supports the platform (Kubernetes, drivers, security), while NVIDIA maintains TensorRT releases tied to CUDA versions. Documentation describes running NIM and validated inference stacks on OpenShift with GPU operators and MIG sizing. Teams choosing TensorRT/NIM trade some openness (versus open vLLM) for vendor-tuned performance; Red Hat reference architectures often present both paths on the same OpenShift cluster for different SLAs and compliance needs.

Related

NIM (NVIDIA Inference Microservices)

Ai Nim Nvidia Inference Microservices Containers Gpu Openshift-Ai
NIM (NVIDIA Inference Microservices) are container images and Helm charts that deliver ready-to-run inference endpoints for specific models (LLMs, vision, embedding, reranking, and more). The objective is to shrink time-to-production: instead of assembling CUDA drivers, frameworks, model weights, and an OpenAI-compatible server yourself, operators pull a NIM that bundles a performance-tuned engine (often TensorRT-LLM or Triton-backed paths), default model artifacts or download hooks, health checks, and a stable HTTP/gRPC API. NIMs are sized for GPU deployment and target enterprise MLOps teams that want versioned, scannable containers with predictable resource requests rather than bespoke notebooks turned into scripts.

MIG (Multi-Instance GPU)

Ai Mig Nvidia Gpu Kubernetes Multi-Tenancy Inference
MIG (Multi-Instance GPU) is an NVIDIA GPU partitioning mode on datacenter accelerators (e.g. A100, H100) that splits one physical card into up to seven GPU instances (GIs), each with isolated streaming multiprocessors, memory bandwidth, and HBM capacity. The objective is higher utilization in multi-tenant environments: several smaller models or dev/test workloads share one expensive GPU without time-slicing contention as severe as full-card sharing. Each MIG instance appears to the OS and CUDA as a separate GPU with fixed resources; workloads cannot oversubscribe another instance’s memory. MIG suits inference and modest training more often than massive single-job training that needs the entire GPU and NVLink domain.

Quantization

Ai Quantization Fp8 Int8 Inference Llm Gpu Nim Vllm
Quantization is the process of representing a model’s weights and/or activations with fewer bits than full FP32 training precision—commonly FP16, BF16, FP8, INT8, or INT4 (GPTQ, AWQ, GGUF-style formats). The objective is lower GPU memory (larger models or more concurrent sessions per card), higher throughput, and sometimes faster kernels on hardware with native low-precision units, at the cost of possible quality degradation if pushed too aggressively. Quantization can be applied post-training (calibration on a sample dataset) or during training (quantization-aware training). For inference, serving engines vLLM and NIM load quantized checkpoints and dispatch to vendor libraries (TensorRT-LLM, CUTLASS, etc.) that implement fused low-precision matmuls.