The maximum number of tokens (prompt plus generated output) an LLM can attend to in one session—fixed by architecture and VRAM, and a primary limit for RAG, agents, and long documents.

NVIDIA's parallel computing platform and programming model: C/C++ extensions, libraries, and drivers that let applications launch kernels on GPUs and manage device memory, streams, and multi-GPU execution.

NVIDIA's GPU-accelerated library of primitives for deep learning—convolutions, attention, normalization, and RNN ops—used by PyTorch, TensorRT, and inference runtimes on top of CUDA.

The autoregressive inference phase that generates output tokens one at a time—reusing the KV cache and typically memory-bandwidth-bound—after prefill has processed the prompt.

NVIDIA's SDK and runtime for programming BlueField DPUs: libraries and services for networking, storage, security, and telemetry so infrastructure functions run on the NIC instead of the host CPU.

Network-attached accelerators—often SmartNICs such as NVIDIA BlueField—that run infrastructure services (virtual switching, storage, security, telemetry) on dedicated silicon so host CPUs stay available for applications and AI workloads.

Adapting a pretrained LLM to a domain or task by updating weights—fully or via Low-Rank Adaptation (LoRA) adapters—on GPU clusters, with workflows on RHEL AI and OpenShift AI.

Parallel accelerators built for throughput-oriented workloads—deep learning training and inference, scientific computing, and real-time graphics—by executing thousands of similar operations concurrently rather than optimizing for single-thread latency.

Policy and safety layers around LLM inference—input/output filtering, jailbreak detection, PII handling, and tool-use constraints—implemented in platforms such as OpenShift AI Guardrails Orchestrator.

The deployment phase where a trained model serves predictions—generating tokens, class labels, or embeddings under latency and throughput SLOs—distinct from training in compute profile, software stack, and platform sizing.

Lossless RDMA-native cluster fabric for low-latency GPU collective traffic—standard in large-scale AI training and in llm-d disaggregated inference when moving KV state between nodes.

GPU-resident attention state reused across decode steps so each new token does not recompute keys and values for prior tokens—central to LLM latency, memory footprint, and routing strategies in vLLM and llm-d.

Neural networks trained on vast text corpora to predict and generate language—served at scale via GPU inference stacks, RAG, and fine-tuning on platforms such as OpenShift AI and RHEL AI.

A Kubernetes-native distributed LLM inference stack that orchestrates vLLM (and similar servers) with KV-cache-aware routing, prefill/decode disaggregation, and production guides—co-developed by Red Hat, IBM, Google, and the open-source community.

An open protocol for connecting LLM applications to tools, data sources, and services through standardized clients and servers—enabling agent workflows beyond plain chat completion APIs.

NVIDIA A100/H100 feature that partitions one physical GPU into isolated instances with dedicated compute and memory—improving utilization for multi-tenant inference on Kubernetes without separate cards per workload.

Optimized multi-GPU and multi-node collective primitives (all-reduce, broadcast, all-gather)—the communication layer beneath distributed PyTorch training and many multi-GPU LLM inference stacks.

Containerized, API-first NVIDIA microservices that package optimized inference engines, models, and runtimes for LLMs and other AI workloads—deployed on Kubernetes with enterprise support paths via Red Hat OpenShift AI and RHEL.

NVIDIA's high-bandwidth GPU–GPU and GPU–CPU interconnect within a server or NVLink switch tray—essential for tensor-parallel LLM inference and multi-GPU training without traversing PCIe or the network.

The inference phase that processes the full input prompt in parallel—building the KV cache and dominating time-to-first-token—before autoregressive decode generates output tokens one by one.

Reducing numeric precision of model weights and activations (FP16, BF16, FP8, INT8, INT4) to cut VRAM and increase inference throughput—implemented in vLLM, TensorRT-LLM, and NIM with trade-offs in accuracy.

A pattern that grounds LLM answers in retrieved documents—embedding search, reranking, and prompt assembly—so responses stay current and cite private knowledge without retraining the base model.

Network technology that lets one host read or write another's memory with minimal CPU involvement—used for distributed training, KV transfer in disaggregated inference, storage (NVMe-oF), and DPU offload.

RDMA on lossless Ethernet—an alternative to InfiniBand for GPU collective traffic and storage, widely deployed when AI clusters share a converged datacenter fabric.

AMD's open GPU compute platform—drivers, compilers, and libraries for running PyTorch and inference workloads on Instinct accelerators as an alternative to NVIDIA CUDA on RHEL and OpenShift.

NVIDIA's SDK for optimizing and deploying trained models on GPUs—graph fusion, precision calibration, and kernel autotuning—often powering NIM and high-performance LLM inference backends.

The ML phase where model weights are learned from data via forward pass, loss, and backward pass (gradient descent)—dominated by distributed GPU compute, high-bandwidth interconnects, and long-running batch jobs rather than request-level latency.

An open-source LLM inference and serving engine that maximizes GPU utilization through PagedAttention, continuous batching, and high-throughput decoding for production chat, RAG, and API workloads on Kubernetes or bare metal.