Inference on Le Site de François

Context window

Mon, 01 Jan 0001 00:00:00 +0000

The context window is the maximum span of tokens—input prompt plus model-generated output—that an LLM can process in a single forward pass chain without truncating or sliding attention. It is set by model architecture (positional encoding limit, e.g. 8K, 128K, 1M+ in newer models) and by practical VRAM on the serving GPU, because the KV cache scales with total sequence length. Its objective is to bound memory and compute: longer windows enable whole documents, multi-turn chat history, and large RAG payloads in one shot, but cost more on every prefill and decode step. APIs expose this as max_tokens, context limits, or model cards; exceeding it yields errors or silent truncation.

cuDNN (CUDA Deep Neural Network library)

Mon, 01 Jan 0001 00:00:00 +0000

cuDNN (CUDA Deep Neural Network library) is NVIDIA’s library of highly optimized GPU kernels for operations that dominate deep learning: convolutions, matrix multiplies used in attention, pooling, normalization (batch/layer), activations, and recurrent cells. Its objective is to deliver near-peak performance on CUDA-capable GPUs without every framework author hand-writing assembly-tuned kernels. PyTorch, TensorFlow, and many inference engines call cuDNN (directly or via cuBLAS) under the hood for training and serving. cuDNN sits between raw CUDA and application code; version alignment with the CUDA toolkit and driver is mandatory for supported deployments.

Decode

Mon, 01 Jan 0001 00:00:00 +0000

Decode is the second stage of LLM inference: after prefill has stored keys and values for the prompt, the model generates one new token per forward pass, appends it to the sequence, extends the KV cache, and repeats until a stop condition (EOS token, max length, or API limit). Its objective is fluent continuation—answer text, code, or tool-call JSON—at acceptable inter-token latency and cluster throughput (tokens per second across many concurrent sessions). Decode drives the “typing” experience in chat UIs; prefill drives how long users wait before the first character appears.

Guardrails

Mon, 01 Jan 0001 00:00:00 +0000

Guardrails are controls wrapped around LLM inference to reduce harmful, non-compliant, or off-policy behavior without replacing the base model. Their objective is AI safety and governance in production: block or rewrite prompts that attempt prompt injection or jailbreaks, filter toxic or leaked PII in outputs, enforce topic allowlists, validate structured tool calls, and log decisions for audit. Guardrails sit on the request path (before tokens reach the model or after the model proposes a draft response), combining rule engines, classifiers, regex, and sometimes smaller models. They complement—not replace—application auth, network policy, and human review; enterprises treat them as mandatory for customer-facing and internal copilots.

Inference

Mon, 01 Jan 0001 00:00:00 +0000

Inference is the operational phase of machine learning where a trained model is applied to new inputs to produce outputs: next tokens in an LLM, bounding boxes in vision, embeddings for search, or scores in tabular models. Its objective is reliable serving at scale—honoring latency targets (time to first token, p99 completion time), throughput (requests or tokens per second), availability, and cost per query—rather than improving weights. In generative AI, inference splits into prefill (processing the prompt in one or few forward passes) and decode (autoregressive generation of each output token), each with different bottlenecks. Production inference adds API gateways, auth, rate limiting, observability, model versioning, A/B tests, and guardrails; the model file is read-mostly while KV cache and batch state are ephemeral per session.

KV cache (Key-Value Cache)

Mon, 01 Jan 0001 00:00:00 +0000

The KV cache (key-value cache) is the stored result of the attention layers for tokens already processed in a sequence. During autoregressive decode, each new token only needs a forward pass that depends on prior context; recomputing keys and values for all earlier tokens every step would be wasteful. The cache therefore holds, per layer and per sequence, the K and V tensors produced when those tokens were first seen (during prefill for the prompt, then extended one token at a time during decode). The objective is lower time per output token and lower FLOPs; the cost is GPU memory: cache size grows with batch × layers × heads × sequence_length × head_dim, and is often the limit on concurrent sessions or context length before model weights fill VRAM.

LLM (Large Language Model)

Mon, 01 Jan 0001 00:00:00 +0000

An LLM (large language model) is a deep neural network—almost always a Transformer—trained on large amounts of text (and sometimes multimodal data) to model the probability of the next token given prior context. Its objective at training time is to minimize prediction error over billions of tokens, producing weights that encode grammar, facts (with limitations), reasoning patterns, and task-following behavior after alignment or instruction tuning. At inference time the same model generates completions, answers questions, summarizes documents, or drives agents; production systems expose it through APIs (often OpenAI-compatible) backed by engines such as vLLM or NIM. LLMs power chatbots, code assistants, RAG pipelines, and enterprise copilots.

llm-d

Mon, 01 Jan 0001 00:00:00 +0000

llm-d is an open-source distributed inference serving stack for production LLM workloads on Kubernetes. Its objective is not to replace model servers such as vLLM or SGLang but to sit above them and fix cluster-scale problems: which replica should receive the next request, how to split prefill (compute-heavy) from decode (memory-bandwidth-heavy), how to share or tier KV cache state, and how to scale MoE models with wide expert parallelism. llm-d publishes “well-lit path” guides—benchmarked Helm recipes and architectures—so teams reach strong time-to-first-token and throughput without hand-rolling schedulers. The project is a CNCF sandbox effort with contributors including Red Hat, IBM, Google, and cloud partners.

MIG (Multi-Instance GPU)

Mon, 01 Jan 0001 00:00:00 +0000

MIG (Multi-Instance GPU) is an NVIDIA GPU partitioning mode on datacenter accelerators (e.g. A100, H100) that splits one physical card into up to seven GPU instances (GIs), each with isolated streaming multiprocessors, memory bandwidth, and HBM capacity. The objective is higher utilization in multi-tenant environments: several smaller models or dev/test workloads share one expensive GPU without time-slicing contention as severe as full-card sharing. Each MIG instance appears to the OS and CUDA as a separate GPU with fixed resources; workloads cannot oversubscribe another instance’s memory. MIG suits inference and modest training more often than massive single-job training that needs the entire GPU and NVLink domain.

NIM (NVIDIA Inference Microservices)

Mon, 01 Jan 0001 00:00:00 +0000

NIM (NVIDIA Inference Microservices) are container images and Helm charts that deliver ready-to-run inference endpoints for specific models (LLMs, vision, embedding, reranking, and more). The objective is to shrink time-to-production: instead of assembling CUDA drivers, frameworks, model weights, and an OpenAI-compatible server yourself, operators pull a NIM that bundles a performance-tuned engine (often TensorRT-LLM or Triton-backed paths), default model artifacts or download hooks, health checks, and a stable HTTP/gRPC API. NIMs are sized for GPU deployment and target enterprise MLOps teams that want versioned, scannable containers with predictable resource requests rather than bespoke notebooks turned into scripts.

Prefill

Mon, 01 Jan 0001 00:00:00 +0000

Prefill is the first stage of LLM inference after a user (or RAG pipeline) submits a prompt: the model runs a forward pass over all input tokens at once (or in chunked blocks for very long contexts) to compute hidden states and populate the KV cache for every layer. Its objective is to prepare context the model will attend to during generation; the user-visible metric is often time to first token (TTFT), which is dominated by prefill for long prompts. Prefill is compute-intensive (large matrix multiplies across the full sequence) compared with decode, which adds one token at a time. In chat, each new user message typically triggers a new prefill over the accumulated conversation (unless caching optimizations apply).

Quantization

Mon, 01 Jan 0001 00:00:00 +0000

Quantization is the process of representing a model’s weights and/or activations with fewer bits than full FP32 training precision—commonly FP16, BF16, FP8, INT8, or INT4 (GPTQ, AWQ, GGUF-style formats). The objective is lower GPU memory (larger models or more concurrent sessions per card), higher throughput, and sometimes faster kernels on hardware with native low-precision units, at the cost of possible quality degradation if pushed too aggressively. Quantization can be applied post-training (calibration on a sample dataset) or during training (quantization-aware training). For inference, serving engines vLLM and NIM load quantized checkpoints and dispatch to vendor libraries (TensorRT-LLM, CUTLASS, etc.) that implement fused low-precision matmuls.

RAG (Retrieval-Augmented Generation)

Mon, 01 Jan 0001 00:00:00 +0000

RAG (retrieval-augmented generation) is an architecture pattern, not a single product: before the LLM generates an answer, a retriever finds relevant chunks from a knowledge base (wikis, tickets, PDFs, databases) and injects them into the prompt as context. The objective is grounded responses—fewer hallucinations on company facts, answers that reflect documents updated yesterday, and traceability to sources—without running full fine-tuning every time content changes. A typical pipeline embeds queries and documents with an embedding model, stores vectors in a search index, retrieves top-k passages, optionally reranks them, then calls the LLM with a system prompt plus retrieved text. RAG is the dominant enterprise pattern for private AI assistants and support bots.

ROCm (Radeon Open Compute)

Mon, 01 Jan 0001 00:00:00 +0000

ROCm (Radeon Open Compute) is AMD’s software stack for GPU compute on datacenter Instinct accelerators (and select consumer GPUs in community setups). Its objective mirrors CUDA for NVIDIA: provide kernel compilers (HIP), math libraries (rocBLAS, rocFFT), collective communication (RCCL, analogous to NCCL), and framework integrations so PyTorch and inference runtimes can execute training and inference on AMD hardware. ROCm is positioned as an open platform (Linux-first) for customers who want accelerator choice or who standardize on AMD in HPC and AI clusters.

TensorRT

Mon, 01 Jan 0001 00:00:00 +0000

TensorRT is NVIDIA’s SDK for optimizing and deploying trained neural networks for inference on NVIDIA GPUs. Its objective is minimum latency and maximum throughput: it ingests a model (ONNX, TensorFlow, PyTorch export, or framework-specific parsers), applies graph optimizations (layer fusion, constant folding, kernel autotuning), selects precisions (FP32, FP16, INT8, FP8), and produces a serialized engine executed by a lightweight runtime. For LLMs, TensorRT-LLM extends this with attention-specific fusions, inflight batching, and multi-GPU serving patterns; many NIM microservices bundle TensorRT-LLM–optimized engines rather than raw PyTorch loops.

vLLM

Mon, 01 Jan 0001 00:00:00 +0000

vLLM is an open-source library and serving stack for large language model (LLM) inference. Its objective is to turn a trained model into a production service that sustains many concurrent users with low latency and high tokens per second per GPU. vLLM targets the inference phase (prefill + decode), not training: it loads weights onto accelerators, batches incoming prompts, schedules decode steps, and streams completions back to clients over HTTP/gRPC (often via an OpenAI-compatible API). It has become a de facto engine behind many private and cloud AI gateways because it ships integrations for Hugging Face models, LoRA adapters, tensor parallelism, pipeline parallelism, speculative decoding, and quantization (GPTQ, AWQ, FP8).