↓ Skip to main content
Le Site de François
  1. Index/

Context window

Ai Context-Window Llm Tokens Kv-Cache Inference Rag

The context window is the maximum span of tokens—input prompt plus model-generated output—that an LLM can process in a single forward pass chain without truncating or sliding attention. It is set by model architecture (positional encoding limit, e.g. 8K, 128K, 1M+ in newer models) and by practical VRAM on the serving GPU, because the KV cache scales with total sequence length. Its objective is to bound memory and compute: longer windows enable whole documents, multi-turn chat history, and large RAG payloads in one shot, but cost more on every prefill and decode step. APIs expose this as max_tokens, context limits, or model cards; exceeding it yields errors or silent truncation.

Architecturally, context length is the bridge between user-facing features and hardware. A CPU can assemble a 200K-token string, but the GPU must store KV states for each layer × each token; beyond context window, frameworks drop middle sections (sliding window models), summarize, or fail. RAG often fights the window by retrieving only top-k chunks; fine-tuning does not extend window unless the architecture supports it (e.g. YaRN, rope scaling). Quantization frees VRAM so more tokens fit in the same window. llm-d prefix caching helps when many requests share the same long system prompt within the window. Window size is unrelated to training corpus size—it is an inference-time limit per request.

Red Hat documents context planning on OpenShift AI and RHEL AI: matching model SKU (8K vs 128K) to use cases, GPU HBM sizing with vLLM/NIM, and MIG profiles too small for long-context models. Guardrails may reject oversize prompts before they hit the GPU. Reference architectures for RAG recommend chunking and reranking so retrieved text fits comfortably inside the window with room for the answer. Operators validate SLOs at worst-case prompt lengths, not only average chat size.

Related

Decode

Ai Decode Inference Llm Kv-Cache Vllm Throughput Latency
Decode is the second stage of LLM inference: after prefill has stored keys and values for the prompt, the model generates one new token per forward pass, appends it to the sequence, extends the KV cache, and repeats until a stop condition (EOS token, max length, or API limit). Its objective is fluent continuation—answer text, code, or tool-call JSON—at acceptable inter-token latency and cluster throughput (tokens per second across many concurrent sessions). Decode drives the “typing” experience in chat UIs; prefill drives how long users wait before the first character appears.

KV cache (Key-Value Cache)

Ai Kv-Cache Inference Llm Attention Vllm Memory Transformer
The KV cache (key-value cache) is the stored result of the attention layers for tokens already processed in a sequence. During autoregressive decode, each new token only needs a forward pass that depends on prior context; recomputing keys and values for all earlier tokens every step would be wasteful. The cache therefore holds, per layer and per sequence, the K and V tensors produced when those tokens were first seen (during prefill for the prompt, then extended one token at a time during decode). The objective is lower time per output token and lower FLOPs; the cost is GPU memory: cache size grows with batch × layers × heads × sequence_length × head_dim, and is often the limit on concurrent sessions or context length before model weights fill VRAM.

Prefill

Ai Prefill Inference Llm Kv-Cache Vllm Llm-D Latency
Prefill is the first stage of LLM inference after a user (or RAG pipeline) submits a prompt: the model runs a forward pass over all input tokens at once (or in chunked blocks for very long contexts) to compute hidden states and populate the KV cache for every layer. Its objective is to prepare context the model will attend to during generation; the user-visible metric is often time to first token (TTFT), which is dominated by prefill for long prompts. Prefill is compute-intensive (large matrix multiplies across the full sequence) compared with decode, which adds one token at a time. In chat, each new user message typically triggers a new prefill over the accumulated conversation (unless caching optimizations apply).