TensorRT

Mon, 01 Jan 0001 00:00:00 +0000

TensorRT is NVIDIA’s SDK for optimizing and deploying trained neural networks for inference on NVIDIA GPUs. Its objective is minimum latency and maximum throughput: it ingests a model (ONNX, TensorFlow, PyTorch export, or framework-specific parsers), applies graph optimizations (layer fusion, constant folding, kernel autotuning), selects precisions (FP32, FP16, INT8, FP8), and produces a serialized engine executed by a lightweight runtime. For LLMs, TensorRT-LLM extends this with attention-specific fusions, inflight batching, and multi-GPU serving patterns; many NIM microservices bundle TensorRT-LLM–optimized engines rather than raw PyTorch loops.

Tensorrt-Llm on Le Site de François

TensorRT