↓ Skip to main content
Le Site de François
  1. Index/

LLM (Large Language Model)

Ai Llm Transformer Generative-Ai Inference Training Nlp

An LLM (large language model) is a deep neural network—almost always a Transformer—trained on large amounts of text (and sometimes multimodal data) to model the probability of the next token given prior context. Its objective at training time is to minimize prediction error over billions of tokens, producing weights that encode grammar, facts (with limitations), reasoning patterns, and task-following behavior after alignment or instruction tuning. At inference time the same model generates completions, answers questions, summarizes documents, or drives agents; production systems expose it through APIs (often OpenAI-compatible) backed by engines such as vLLM or NIM. LLMs power chatbots, code assistants, RAG pipelines, and enterprise copilots.

Architecturally, an LLM is not run like a typical CPU application: forward passes are dominated by matrix multiplication and attention, with memory footprint driven by model size (billions of parameters) and per-session KV cache during decode. A 7B–70B+ parameter model does not fit in host DRAM for fast serving; weights and cache live on GPU HBM, often sharded with tensor parallelism across devices linked by NVLink or NCCL over InfiniBand/RoCE. The CPU handles tokenization, request batching, networking, and orchestration. Context length (prompt + output tokens) directly affects latency and VRAM; techniques such as quantization, LoRA adapters, and prefix caching exist to reduce cost and improve throughput without retraining from scratch.

Red Hat positions LLMs as workloads on RHEL AI (curated models, InstructLab, local experimentation) and Red Hat OpenShift AI (multi-tenant serving, MLOps, integration with NVIDIA NIM and open vLLM). Documentation covers GPU node sizing, security (routes, SCCs, secrets for API keys), and hybrid patterns where training or fine-tuning happens on-cluster and inference runs behind gateways or llm-d for cache-aware routing. The platform story is Linux + Kubernetes + supported accelerators, not a proprietary model family—customers bring Hugging Face or vendor weights and operate them like any other stateful service.

Related

KV cache (Key-Value Cache)

Ai Kv-Cache Inference Llm Attention Vllm Memory Transformer
The KV cache (key-value cache) is the stored result of the attention layers for tokens already processed in a sequence. During autoregressive decode, each new token only needs a forward pass that depends on prior context; recomputing keys and values for all earlier tokens every step would be wasteful. The cache therefore holds, per layer and per sequence, the K and V tensors produced when those tokens were first seen (during prefill for the prompt, then extended one token at a time during decode). The objective is lower time per output token and lower FLOPs; the cost is GPU memory: cache size grows with batch × layers × heads × sequence_length × head_dim, and is often the limit on concurrent sessions or context length before model weights fill VRAM.

Context window

Ai Context-Window Llm Tokens Kv-Cache Inference Rag
The context window is the maximum span of tokens—input prompt plus model-generated output—that an LLM can process in a single forward pass chain without truncating or sliding attention. It is set by model architecture (positional encoding limit, e.g. 8K, 128K, 1M+ in newer models) and by practical VRAM on the serving GPU, because the KV cache scales with total sequence length. Its objective is to bound memory and compute: longer windows enable whole documents, multi-turn chat history, and large RAG payloads in one shot, but cost more on every prefill and decode step. APIs expose this as max_tokens, context limits, or model cards; exceeding it yields errors or silent truncation.

Decode

Ai Decode Inference Llm Kv-Cache Vllm Throughput Latency
Decode is the second stage of LLM inference: after prefill has stored keys and values for the prompt, the model generates one new token per forward pass, appends it to the sequence, extends the KV cache, and repeats until a stop condition (EOS token, max length, or API limit). Its objective is fluent continuation—answer text, code, or tool-call JSON—at acceptable inter-token latency and cluster throughput (tokens per second across many concurrent sessions). Decode drives the “typing” experience in chat UIs; prefill drives how long users wait before the first character appears.