TensorRT is NVIDIA’s SDK for optimizing and deploying trained neural networks for inference on NVIDIA GPUs. Its objective is minimum latency and maximum throughput: it ingests a model (ONNX, TensorFlow, PyTorch export, or framework-specific parsers), applies graph optimizations (layer fusion, constant folding, kernel autotuning), selects precisions (FP32, FP16, INT8, FP8), and produces a serialized engine executed by a lightweight runtime. For LLMs, TensorRT-LLM extends this with attention-specific fusions, inflight batching, and multi-GPU serving patterns; many NIM microservices bundle TensorRT-LLM–optimized engines rather than raw PyTorch loops.
Quantization is the process of representing a model’s weights and/or activations with fewer bits than full FP32 training precision—commonly FP16, BF16, FP8, INT8, or INT4 (GPTQ, AWQ, GGUF-style formats). The objective is lower GPU memory (larger models or more concurrent sessions per card), higher throughput, and sometimes faster kernels on hardware with native low-precision units, at the cost of possible quality degradation if pushed too aggressively. Quantization can be applied post-training (calibration on a sample dataset) or during training (quantization-aware training). For inference, serving engines vLLM and NIM load quantized checkpoints and dispatch to vendor libraries (TensorRT-LLM, CUTLASS, etc.) that implement fused low-precision matmuls.
NIM (NVIDIA Inference Microservices) are container images and Helm charts that deliver ready-to-run inference endpoints for specific models (LLMs, vision, embedding, reranking, and more). The objective is to shrink time-to-production: instead of assembling CUDA drivers, frameworks, model weights, and an OpenAI-compatible server yourself, operators pull a NIM that bundles a performance-tuned engine (often TensorRT-LLM or Triton-backed paths), default model artifacts or download hooks, health checks, and a stable HTTP/gRPC API. NIMs are sized for GPU deployment and target enterprise MLOps teams that want versioned, scannable containers with predictable resource requests rather than bespoke notebooks turned into scripts.
Inference is the operational phase of machine learning where a trained model is applied to new inputs to produce outputs: next tokens in an LLM, bounding boxes in vision, embeddings for search, or scores in tabular models. Its objective is reliable serving at scale—honoring latency targets (time to first token, p99 completion time), throughput (requests or tokens per second), availability, and cost per query—rather than improving weights. In generative AI, inference splits into prefill (processing the prompt in one or few forward passes) and decode (autoregressive generation of each output token), each with different bottlenecks. Production inference adds API gateways, auth, rate limiting, observability, model versioning, A/B tests, and guardrails; the model file is read-mostly while KV cache and batch state are ephemeral per session.
