↓ Skip to main content
Le Site de François
  1. Index/

RAG (Retrieval-Augmented Generation)

Ai Rag Llm Embedding Retrieval Vector-Search Inference
Table of Contents

RAG (retrieval-augmented generation) is an architecture pattern, not a single product: before the LLM generates an answer, a retriever finds relevant chunks from a knowledge base (wikis, tickets, PDFs, databases) and injects them into the prompt as context. The objective is grounded responses—fewer hallucinations on company facts, answers that reflect documents updated yesterday, and traceability to sources—without running full fine-tuning every time content changes. A typical pipeline embeds queries and documents with an embedding model, stores vectors in a search index, retrieves top-k passages, optionally reranks them, then calls the LLM with a system prompt plus retrieved text. RAG is the dominant enterprise pattern for private AI assistants and support bots.

Architecturally, RAG adds CPU and I/O work around GPU inference: embedding batches, index lookups, and prompt assembly happen on the host or separate services, while the LLM still runs on GPU with a longer prefill (large context from retrieved chunks). Repeated identical system prompts and document prefixes make prefix caching and KV-aware routing (llm-d, vLLM) valuable—many users ask different questions over the same knowledge base header. Compared with stuffing an entire corpus into context, RAG trades retrieval latency for bounded prompt size and lower cost. Poor chunking, stale indexes, or wrong embeddings fail at the retrieval layer even when the LLM is capable; ops concerns include PII in indexes, access control per tenant, and refresh pipelines when documents change.

Red Hat supports RAG through OpenShift AI and partner ecosystems: GPU-backed inference for the generator, optional NIM embedding microservices, storage (Ceph, object stores) for corpora, and OpenShift for deploying vector databases and ingestion jobs. RHEL AI and InstructLab address model quality and alignment; RAG addresses knowledge freshness. Reference designs describe secure multi-tenant RAG (namespace isolation, OAuth/SSO, network policies) on OpenShift with RHEL GPU nodes. Operators combine open stacks (LangChain-style orchestration, open vector DBs) with Red Hat platform primitives rather than a single bundled “RAG appliance.”

Additional Information
#

Related

Context window

Ai Context-Window Llm Tokens Kv-Cache Inference Rag
The context window is the maximum span of tokens—input prompt plus model-generated output—that an LLM can process in a single forward pass chain without truncating or sliding attention. It is set by model architecture (positional encoding limit, e.g. 8K, 128K, 1M+ in newer models) and by practical VRAM on the serving GPU, because the KV cache scales with total sequence length. Its objective is to bound memory and compute: longer windows enable whole documents, multi-turn chat history, and large RAG payloads in one shot, but cost more on every prefill and decode step. APIs expose this as max_tokens, context limits, or model cards; exceeding it yields errors or silent truncation.

Decode

Ai Decode Inference Llm Kv-Cache Vllm Throughput Latency
Decode is the second stage of LLM inference: after prefill has stored keys and values for the prompt, the model generates one new token per forward pass, appends it to the sequence, extends the KV cache, and repeats until a stop condition (EOS token, max length, or API limit). Its objective is fluent continuation—answer text, code, or tool-call JSON—at acceptable inter-token latency and cluster throughput (tokens per second across many concurrent sessions). Decode drives the “typing” experience in chat UIs; prefill drives how long users wait before the first character appears.

Guardrails

Ai Guardrails Safety Llm Inference Security Openshift-Ai Policy
Guardrails are controls wrapped around LLM inference to reduce harmful, non-compliant, or off-policy behavior without replacing the base model. Their objective is AI safety and governance in production: block or rewrite prompts that attempt prompt injection or jailbreaks, filter toxic or leaked PII in outputs, enforce topic allowlists, validate structured tool calls, and log decisions for audit. Guardrails sit on the request path (before tokens reach the model or after the model proposes a draft response), combining rule engines, classifiers, regex, and sometimes smaller models. They complement—not replace—application auth, network policy, and human review; enterprises treat them as mandatory for customer-facing and internal copilots.