↓ Skip to main content
Le Site de François
  1. Index/

Guardrails

Ai Guardrails Safety Llm Inference Security Openshift-Ai Policy

Guardrails are controls wrapped around LLM inference to reduce harmful, non-compliant, or off-policy behavior without replacing the base model. Their objective is AI safety and governance in production: block or rewrite prompts that attempt prompt injection or jailbreaks, filter toxic or leaked PII in outputs, enforce topic allowlists, validate structured tool calls, and log decisions for audit. Guardrails sit on the request path (before tokens reach the model or after the model proposes a draft response), combining rule engines, classifiers, regex, and sometimes smaller models. They complement—not replace—application auth, network policy, and human review; enterprises treat them as mandatory for customer-facing and internal copilots.

Architecturally, guardrails are CPU- and latency-sensitive orchestration around GPU inference. The heavy LLM still runs in vLLM or NIM; guardrail services may call lightweight detectors (e.g. prompt-injection classifiers) or policy APIs in parallel or in series, adding milliseconds to seconds depending on depth. Unlike training, guardrails do not update weights; unlike RAG, they do not retrieve facts—they constrain how the model may respond. False positives frustrate users; false negatives create security incidents, so teams tune thresholds per use case. MCP and agent toolchains extend the attack surface, so guardrails increasingly cover tool arguments and retrieved content, not only chat text.

Red Hat documents Guardrails Orchestrator for Red Hat OpenShift AI: integrating detectors (regex, Hugging Face models, vLLM-based judges) into serving pipelines on OpenShift, with configuration aligned to enterprise security practices (routes, secrets, SCCs). Blog and security content tie guardrails to defense-in-depth for AI agents (sandboxing, NetworkPolicy, memory-poisoning awareness). RHEL AI focuses on model quality via InstructLab; OpenShift AI adds runtime safety for served models. Red Hat positions guardrails as platform plumbing customers deploy beside open or NVIDIA runtimes, not as a single proprietary model.

Related

Context window

Ai Context-Window Llm Tokens Kv-Cache Inference Rag
The context window is the maximum span of tokens—input prompt plus model-generated output—that an LLM can process in a single forward pass chain without truncating or sliding attention. It is set by model architecture (positional encoding limit, e.g. 8K, 128K, 1M+ in newer models) and by practical VRAM on the serving GPU, because the KV cache scales with total sequence length. Its objective is to bound memory and compute: longer windows enable whole documents, multi-turn chat history, and large RAG payloads in one shot, but cost more on every prefill and decode step. APIs expose this as max_tokens, context limits, or model cards; exceeding it yields errors or silent truncation.

Decode

Ai Decode Inference Llm Kv-Cache Vllm Throughput Latency
Decode is the second stage of LLM inference: after prefill has stored keys and values for the prompt, the model generates one new token per forward pass, appends it to the sequence, extends the KV cache, and repeats until a stop condition (EOS token, max length, or API limit). Its objective is fluent continuation—answer text, code, or tool-call JSON—at acceptable inter-token latency and cluster throughput (tokens per second across many concurrent sessions). Decode drives the “typing” experience in chat UIs; prefill drives how long users wait before the first character appears.

Fine-tuning / LoRA

Ai Fine-Tuning Lora Training Llm Peft Openshift-Ai Rhel-Ai
Fine-tuning is training continued from a pretrained LLM (or other model) on a smaller, task-specific dataset so behavior matches a domain—support tone, internal jargon, classification format, or tool-use style—without pretraining from scratch. LoRA (Low-Rank Adaptation) is a parameter-efficient fine-tuning method: instead of updating all billions of weights, small low-rank matrices are inserted into attention (and sometimes MLP) layers and only those adapters are trained, drastically cutting VRAM and checkpoint size. The objective is better task accuracy or alignment at lower cost than full fine-tuning; adapters can be swapped per tenant while a frozen base model stays shared. Fine-tuning differs from RAG, which injects external facts at inference time without changing weights.