Inference

Mon, 01 Jan 0001 00:00:00 +0000

Inference is the operational phase of machine learning where a trained model is applied to new inputs to produce outputs: next tokens in an LLM, bounding boxes in vision, embeddings for search, or scores in tabular models. Its objective is reliable serving at scale—honoring latency targets (time to first token, p99 completion time), throughput (requests or tokens per second), availability, and cost per query—rather than improving weights. In generative AI, inference splits into prefill (processing the prompt in one or few forward passes) and decode (autoregressive generation of each output token), each with different bottlenecks. Production inference adds API gateways, auth, rate limiting, observability, model versioning, A/B tests, and guardrails; the model file is read-mostly while KV cache and batch state are ephemeral per session.

vLLM

Mon, 01 Jan 0001 00:00:00 +0000

vLLM is an open-source library and serving stack for large language model (LLM) inference. Its objective is to turn a trained model into a production service that sustains many concurrent users with low latency and high tokens per second per GPU. vLLM targets the inference phase (prefill + decode), not training: it loads weights onto accelerators, batches incoming prompts, schedules decode steps, and streams completions back to clients over HTTP/gRPC (often via an OpenAI-compatible API). It has become a de facto engine behind many private and cloud AI gateways because it ships integrations for Hugging Face models, LoRA adapters, tensor parallelism, pipeline parallelism, speculative decoding, and quantization (GPTQ, AWQ, FP8).

Serving on Le Site de François

Inference

vLLM