Llm-D

Prefill is the first stage of LLM inference after a user (or RAG pipeline) submits a prompt: the model runs a forward pass over all input tokens at once (or in chunked blocks for very long contexts) to compute hidden states and populate the KV cache for every layer. Its objective is to prepare context the model will attend to during generation; the user-visible metric is often time to first token (TTFT), which is dominated by prefill for long prompts. Prefill is compute-intensive (large matrix multiplies across the full sequence) compared with decode, which adds one token at a time. In chat, each new user message typically triggers a new prefill over the accumulated conversation (unless caching optimizations apply).

llm-d is an open-source distributed inference serving stack for production LLM workloads on Kubernetes. Its objective is not to replace model servers such as vLLM or SGLang but to sit above them and fix cluster-scale problems: which replica should receive the next request, how to split prefill (compute-heavy) from decode (memory-bandwidth-heavy), how to share or tier KV cache state, and how to scale MoE models with wide expert parallelism. llm-d publishes “well-lit path” guides—benchmarked Helm recipes and architectures—so teams reach strong time-to-first-token and throughput without hand-rolling schedulers. The project is a CNCF sandbox effort with contributors including Red Hat, IBM, Google, and cloud partners.