↓ Skip to main content
Le Site de François
  1. Index/

KV cache (Key-Value Cache)

Ai Kv-Cache Inference Llm Attention Vllm Memory Transformer

The KV cache (key-value cache) is the stored result of the attention layers for tokens already processed in a sequence. During autoregressive decode, each new token only needs a forward pass that depends on prior context; recomputing keys and values for all earlier tokens every step would be wasteful. The cache therefore holds, per layer and per sequence, the K and V tensors produced when those tokens were first seen (during prefill for the prompt, then extended one token at a time during decode). The objective is lower time per output token and lower FLOPs; the cost is GPU memory: cache size grows with batch × layers × heads × sequence_length × head_dim, and is often the limit on concurrent sessions or context length before model weights fill VRAM.

Architecturally, the KV cache is why LLM inference is memory-bound on the decode path whereas prefill is often compute-bound. A CPU serving stack without a cache would repeat full-sequence attention for every token—unusable at scale. GPUs hold the cache in HBM next to compute units; layout matters: contiguous buffers fragment when sequences differ in length, which motivated PagedAttention in vLLM (page tables for KV blocks, analogous to OS virtual memory). Prefix caching and KV-aware routing (as in llm-d) reuse blocks across requests that share system prompts or RAG prefixes, so duplicate work is not duplicated on every replica. Disaggregated prefill/decode even transfers KV state between specialized workers over high-speed networks. None of this replaces the model weights; it is ephemeral per session (unless explicitly offloaded to CPU DRAM or disk tiers in advanced setups).

Red Hat does not implement a proprietary KV cache format; support is through the inference stacks customers run on RHEL and OpenShift AI. vLLM on OpenShift uses PagedAttention and related features in supported serving images; llm-d adds cluster-level policies that route traffic to replicas with warm prefix cache. Red Hat documentation and reference architectures explain sizing GPU memory for context length, batching, and multi-tenant serving, and how platform choices (MIG, time-slicing, network for KV transfer) affect SLOs. Understanding KV cache behavior is prerequisite to capacity planning and to tuning Red Hat–backed inference deployments with NVIDIA or other accelerators.

Related

Decode

Ai Decode Inference Llm Kv-Cache Vllm Throughput Latency
Decode is the second stage of LLM inference: after prefill has stored keys and values for the prompt, the model generates one new token per forward pass, appends it to the sequence, extends the KV cache, and repeats until a stop condition (EOS token, max length, or API limit). Its objective is fluent continuation—answer text, code, or tool-call JSON—at acceptable inter-token latency and cluster throughput (tokens per second across many concurrent sessions). Decode drives the “typing” experience in chat UIs; prefill drives how long users wait before the first character appears.

Prefill

Ai Prefill Inference Llm Kv-Cache Vllm Llm-D Latency
Prefill is the first stage of LLM inference after a user (or RAG pipeline) submits a prompt: the model runs a forward pass over all input tokens at once (or in chunked blocks for very long contexts) to compute hidden states and populate the KV cache for every layer. Its objective is to prepare context the model will attend to during generation; the user-visible metric is often time to first token (TTFT), which is dominated by prefill for long prompts. Prefill is compute-intensive (large matrix multiplies across the full sequence) compared with decode, which adds one token at a time. In chat, each new user message typically triggers a new prefill over the accumulated conversation (unless caching optimizations apply).

Context window

Ai Context-Window Llm Tokens Kv-Cache Inference Rag
The context window is the maximum span of tokens—input prompt plus model-generated output—that an LLM can process in a single forward pass chain without truncating or sliding attention. It is set by model architecture (positional encoding limit, e.g. 8K, 128K, 1M+ in newer models) and by practical VRAM on the serving GPU, because the KV cache scales with total sequence length. Its objective is to bound memory and compute: longer windows enable whole documents, multi-turn chat history, and large RAG payloads in one shot, but cost more on every prefill and decode step. APIs expose this as max_tokens, context limits, or model cards; exceeding it yields errors or silent truncation.