Memory

The KV cache (key-value cache) is the stored result of the attention layers for tokens already processed in a sequence. During autoregressive decode, each new token only needs a forward pass that depends on prior context; recomputing keys and values for all earlier tokens every step would be wasteful. The cache therefore holds, per layer and per sequence, the K and V tensors produced when those tokens were first seen (during prefill for the prompt, then extended one token at a time during decode). The objective is lower time per output token and lower FLOPs; the cost is GPU memory: cache size grows with batch × layers × heads × sequence_length × head_dim, and is often the limit on concurrent sessions or context length before model weights fill VRAM.