↓ Skip to main content
Le Site de François
  1. Index/

Decode

Ai Decode Inference Llm Kv-Cache Vllm Throughput Latency

Decode is the second stage of LLM inference: after prefill has stored keys and values for the prompt, the model generates one new token per forward pass, appends it to the sequence, extends the KV cache, and repeats until a stop condition (EOS token, max length, or API limit). Its objective is fluent continuation—answer text, code, or tool-call JSON—at acceptable inter-token latency and cluster throughput (tokens per second across many concurrent sessions). Decode drives the “typing” experience in chat UIs; prefill drives how long users wait before the first character appears.

Architecturally, decode is usually memory-bandwidth-bound on the GPU: each step touches all model weights for a small amount of new compute, while the KV cache grows with total sequence length (prompt + generated tokens). Continuous batching in vLLM mixes decode steps from many requests in one kernel launch to raise utilization. The CPU streams tokens to clients and manages batch scheduling but does not perform the heavy matmuls. Quantization and speculative decoding target decode cost; tensor parallelism adds communication per token on multi-GPU setups. llm-d disaggregated stacks run decode on pools tuned for memory bandwidth, separate from prefill workers.

Red Hat tuning guides for OpenShift AI emphasize decode for capacity planning: how many concurrent chats per GPU, effect of MIG slice size on cache headroom, and autoscaling on tokens/sec or queue depth. Guardrails may scan each decoded chunk or the full completion before returning to users. Production SLOs often specify p99 token latency separately from TTFT (prefill). Red Hat platforms do not change decode algorithms; they host vLLM, NIM, and llm-d with supported drivers and observability on RHEL/OpenShift.

Related

Prefill

Ai Prefill Inference Llm Kv-Cache Vllm Llm-D Latency
Prefill is the first stage of LLM inference after a user (or RAG pipeline) submits a prompt: the model runs a forward pass over all input tokens at once (or in chunked blocks for very long contexts) to compute hidden states and populate the KV cache for every layer. Its objective is to prepare context the model will attend to during generation; the user-visible metric is often time to first token (TTFT), which is dominated by prefill for long prompts. Prefill is compute-intensive (large matrix multiplies across the full sequence) compared with decode, which adds one token at a time. In chat, each new user message typically triggers a new prefill over the accumulated conversation (unless caching optimizations apply).

KV cache (Key-Value Cache)

Ai Kv-Cache Inference Llm Attention Vllm Memory Transformer
The KV cache (key-value cache) is the stored result of the attention layers for tokens already processed in a sequence. During autoregressive decode, each new token only needs a forward pass that depends on prior context; recomputing keys and values for all earlier tokens every step would be wasteful. The cache therefore holds, per layer and per sequence, the K and V tensors produced when those tokens were first seen (during prefill for the prompt, then extended one token at a time during decode). The objective is lower time per output token and lower FLOPs; the cost is GPU memory: cache size grows with batch × layers × heads × sequence_length × head_dim, and is often the limit on concurrent sessions or context length before model weights fill VRAM.

Context window

Ai Context-Window Llm Tokens Kv-Cache Inference Rag
The context window is the maximum span of tokens—input prompt plus model-generated output—that an LLM can process in a single forward pass chain without truncating or sliding attention. It is set by model architecture (positional encoding limit, e.g. 8K, 128K, 1M+ in newer models) and by practical VRAM on the serving GPU, because the KV cache scales with total sequence length. Its objective is to bound memory and compute: longer windows enable whole documents, multi-turn chat history, and large RAG payloads in one shot, but cost more on every prefill and decode step. APIs expose this as max_tokens, context limits, or model cards; exceeding it yields errors or silent truncation.