↓ Skip to main content
Le Site de François
  1. Index/

llm-d

Ai Llm-D Kubernetes Inference Vllm Gateway Distributed-Serving
Table of Contents

llm-d is an open-source distributed inference serving stack for production LLM workloads on Kubernetes. Its objective is not to replace model servers such as vLLM or SGLang but to sit above them and fix cluster-scale problems: which replica should receive the next request, how to split prefill (compute-heavy) from decode (memory-bandwidth-heavy), how to share or tier KV cache state, and how to scale MoE models with wide expert parallelism. llm-d publishes “well-lit path” guides—benchmarked Helm recipes and architectures—so teams reach strong time-to-first-token and throughput without hand-rolling schedulers. The project is a CNCF sandbox effort with contributors including Red Hat, IBM, Google, and cloud partners.

Compared with a single vLLM pod behind a naive round-robin load balancer, llm-d’s control plane is inference-aware. An Inference Scheduler (built on the Kubernetes Gateway API inference extension and compatible gateways) scores replicas using telemetry: prefix/KV overlap, load, optional latency prediction, and prefill/decode (P/D) topology. That is fundamentally different from generic L7 routing, which treats every pod as interchangeable. Disaggregated serving runs prefill workers and decode workers as separate pools, moving KV tensors between them (e.g. via NIXL over RDMA) so each phase uses the right GPU profile. A CPU-only cluster cannot run llm-d’s value proposition; the stack assumes accelerators under the model server and Kubernetes for placement, autoscaling, and networking.

Red Hat is a core contributor and positions llm-d as the path to scale OpenShift AI inference beyond single-replica vLLM. Documentation and blog material describe deploying llm-d on OpenShift with RHEL on GPU nodes, Gateway API–based routing, and integration with Red Hat’s AI portfolio. Because llm-d standardizes on Kubernetes primitives (Helm, CRDs, gateway policies), it aligns with how Red Hat customers already operate platforms: the same RBAC, GitOps, and multi-tenancy models apply. Teams start from upstream quickstarts at llm-d.ai and harden on OpenShift using Red Hat-supported Kubernetes, networking (including SR-IOV/RDMA where needed), and joint reference designs with hardware partners.

Additional Information
#

Related

Prefill

Ai Prefill Inference Llm Kv-Cache Vllm Llm-D Latency
Prefill is the first stage of LLM inference after a user (or RAG pipeline) submits a prompt: the model runs a forward pass over all input tokens at once (or in chunked blocks for very long contexts) to compute hidden states and populate the KV cache for every layer. Its objective is to prepare context the model will attend to during generation; the user-visible metric is often time to first token (TTFT), which is dominated by prefill for long prompts. Prefill is compute-intensive (large matrix multiplies across the full sequence) compared with decode, which adds one token at a time. In chat, each new user message typically triggers a new prefill over the accumulated conversation (unless caching optimizations apply).

vLLM

Ai Vllm Inference Llm Gpu Serving Kubernetes Pagedattention
vLLM is an open-source library and serving stack for large language model (LLM) inference. Its objective is to turn a trained model into a production service that sustains many concurrent users with low latency and high tokens per second per GPU. vLLM targets the inference phase (prefill + decode), not training: it loads weights onto accelerators, batches incoming prompts, schedules decode steps, and streams completions back to clients over HTTP/gRPC (often via an OpenAI-compatible API). It has become a de facto engine behind many private and cloud AI gateways because it ships integrations for Hugging Face models, LoRA adapters, tensor parallelism, pipeline parallelism, speculative decoding, and quantization (GPTQ, AWQ, FP8).

Decode

Ai Decode Inference Llm Kv-Cache Vllm Throughput Latency
Decode is the second stage of LLM inference: after prefill has stored keys and values for the prompt, the model generates one new token per forward pass, appends it to the sequence, extends the KV cache, and repeats until a stop condition (EOS token, max length, or API limit). Its objective is fluent continuation—answer text, code, or tool-call JSON—at acceptable inter-token latency and cluster throughput (tokens per second across many concurrent sessions). Decode drives the “typing” experience in chat UIs; prefill drives how long users wait before the first character appears.