↓ Skip to main content
Le Site de François
  1. Index/

Prefill

Ai Prefill Inference Llm Kv-Cache Vllm Llm-D Latency

Prefill is the first stage of LLM inference after a user (or RAG pipeline) submits a prompt: the model runs a forward pass over all input tokens at once (or in chunked blocks for very long contexts) to compute hidden states and populate the KV cache for every layer. Its objective is to prepare context the model will attend to during generation; the user-visible metric is often time to first token (TTFT), which is dominated by prefill for long prompts. Prefill is compute-intensive (large matrix multiplies across the full sequence) compared with decode, which adds one token at a time. In chat, each new user message typically triggers a new prefill over the accumulated conversation (unless caching optimizations apply).

Architecturally, prefill uses GPU tensor cores heavily; the CPU tokenizes text and submits batches to vLLM or NIM. Long prompts (big RAG retrievals, large system instructions) inflate prefill FLOPs and VRAM for the KV cache linearly with sequence length. llm-d can route requests to replicas with overlapping prefix cache so prefill work is skipped or shortened. Disaggregated inference assigns prefill to dedicated “prefill workers” and ships KV tensors to decode workers over RDMA, because prefill and decode have different optimal hardware profiles. A CPU-only server cannot prefill a large LLM at production speed; prefill is why context length and batching matter as much as decode tuning.

Red Hat addresses prefill sizing in OpenShift AI and llm-d reference architectures: GPU memory for prompt + cache, network bandwidth for disaggregated KV transfer, and SLO dashboards for TTFT. Guardrails and input filters run before or around prefill, adding latency that must be budgeted. On OpenShift, horizontal scale adds replicas; llm-d improves per-cluster efficiency via cache-aware routing rather than only adding GPUs. Operators benchmark prefill separately from decode when tuning vLLM batch limits and MIG profiles.

Related

Decode

Ai Decode Inference Llm Kv-Cache Vllm Throughput Latency
Decode is the second stage of LLM inference: after prefill has stored keys and values for the prompt, the model generates one new token per forward pass, appends it to the sequence, extends the KV cache, and repeats until a stop condition (EOS token, max length, or API limit). Its objective is fluent continuation—answer text, code, or tool-call JSON—at acceptable inter-token latency and cluster throughput (tokens per second across many concurrent sessions). Decode drives the “typing” experience in chat UIs; prefill drives how long users wait before the first character appears.

KV cache (Key-Value Cache)

Ai Kv-Cache Inference Llm Attention Vllm Memory Transformer
The KV cache (key-value cache) is the stored result of the attention layers for tokens already processed in a sequence. During autoregressive decode, each new token only needs a forward pass that depends on prior context; recomputing keys and values for all earlier tokens every step would be wasteful. The cache therefore holds, per layer and per sequence, the K and V tensors produced when those tokens were first seen (during prefill for the prompt, then extended one token at a time during decode). The objective is lower time per output token and lower FLOPs; the cost is GPU memory: cache size grows with batch × layers × heads × sequence_length × head_dim, and is often the limit on concurrent sessions or context length before model weights fill VRAM.

Context window

Ai Context-Window Llm Tokens Kv-Cache Inference Rag
The context window is the maximum span of tokens—input prompt plus model-generated output—that an LLM can process in a single forward pass chain without truncating or sliding attention. It is set by model architecture (positional encoding limit, e.g. 8K, 128K, 1M+ in newer models) and by practical VRAM on the serving GPU, because the KV cache scales with total sequence length. Its objective is to bound memory and compute: longer windows enable whole documents, multi-turn chat history, and large RAG payloads in one shot, but cost more on every prefill and decode step. APIs expose this as max_tokens, context limits, or model cards; exceeding it yields errors or silent truncation.