Kv-Cache on Le Site de François

Context window

Mon, 01 Jan 0001 00:00:00 +0000

The context window is the maximum span of tokens—input prompt plus model-generated output—that an LLM can process in a single forward pass chain without truncating or sliding attention. It is set by model architecture (positional encoding limit, e.g. 8K, 128K, 1M+ in newer models) and by practical VRAM on the serving GPU, because the KV cache scales with total sequence length. Its objective is to bound memory and compute: longer windows enable whole documents, multi-turn chat history, and large RAG payloads in one shot, but cost more on every prefill and decode step. APIs expose this as max_tokens, context limits, or model cards; exceeding it yields errors or silent truncation.

Decode

Mon, 01 Jan 0001 00:00:00 +0000

Decode is the second stage of LLM inference: after prefill has stored keys and values for the prompt, the model generates one new token per forward pass, appends it to the sequence, extends the KV cache, and repeats until a stop condition (EOS token, max length, or API limit). Its objective is fluent continuation—answer text, code, or tool-call JSON—at acceptable inter-token latency and cluster throughput (tokens per second across many concurrent sessions). Decode drives the “typing” experience in chat UIs; prefill drives how long users wait before the first character appears.

KV cache (Key-Value Cache)

Mon, 01 Jan 0001 00:00:00 +0000

The KV cache (key-value cache) is the stored result of the attention layers for tokens already processed in a sequence. During autoregressive decode, each new token only needs a forward pass that depends on prior context; recomputing keys and values for all earlier tokens every step would be wasteful. The cache therefore holds, per layer and per sequence, the K and V tensors produced when those tokens were first seen (during prefill for the prompt, then extended one token at a time during decode). The objective is lower time per output token and lower FLOPs; the cost is GPU memory: cache size grows with batch × layers × heads × sequence_length × head_dim, and is often the limit on concurrent sessions or context length before model weights fill VRAM.

Prefill

Mon, 01 Jan 0001 00:00:00 +0000

Prefill is the first stage of LLM inference after a user (or RAG pipeline) submits a prompt: the model runs a forward pass over all input tokens at once (or in chunked blocks for very long contexts) to compute hidden states and populate the KV cache for every layer. Its objective is to prepare context the model will attend to during generation; the user-visible metric is often time to first token (TTFT), which is dominated by prefill for long prompts. Prefill is compute-intensive (large matrix multiplies across the full sequence) compared with decode, which adds one token at a time. In chat, each new user message typically triggers a new prefill over the accumulated conversation (unless caching optimizations apply).