↓ Skip to main content
Le Site de François
  1. Index/

cgroups (Control Groups) v2

Linux Kernel Containers Resources Kubernetes Systemd

Control Groups (cgroups) is a Linux kernel mechanism, introduced in 2.6.24 (2008), that organises processes into a hierarchy of named groups and uses controllers to account for and limit each group’s consumption of CPU time, memory, I/O bandwidth, and process count. Every container runtime in existence — Docker, containerd, CRI-O, Podman — uses cgroups to enforce the resource limits declared in a container spec (--memory, --cpus, requests.memory, limits.cpu). Every systemd service on a modern Linux system runs in its own cgroup slice. Without cgroups, a container or service could consume all available memory, CPU, or file descriptors, starving other workloads on the same host. The current production version is cgroups v2 (also written cgroupv2, unified hierarchy), stable since kernel 4.5 and the default on all major distributions since RHEL 9, Ubuntu 21.10, and Fedora 31.

The structural improvement of cgroups v2 over v1 is its unified hierarchy: in v1, each controller (cpu, memory, blkio, pids) mounted as a separate hierarchy at its own path, making cross-controller coordination difficult and policy inconsistent. In v2, all controllers share a single tree rooted at /sys/fs/cgroup, mounted as cgroup2. A cgroup in v2 is a directory; creating a subdirectory creates a child cgroup. Processes are assigned to a cgroup by writing their PID to cgroup.procs. Controllers are enabled per-level via cgroup.subtree_control: writing +memory +cpu to a parent’s cgroup.subtree_control enables those controllers for its immediate children. The no-internal-process rule requires that a cgroup with enabled controllers for its children cannot itself contain processes — it must delegate to leaf cgroups — enforcing a clean separation between structural nodes and leaf workload nodes. The key controllers in v2 are: memory (limits total memory use via memory.max, swap via memory.swap.max, and provides real memory pressure detection via memory.pressure — a major improvement over v1’s approximate OOM killer); cpu (weight-based CPU time sharing via cpu.weight and hard limits via cpu.max in the format quota period); io (per-device weight, BPS limits, and IOPS limits via io.weight, io.max); pids (maximum process/thread count via pids.max, preventing fork bombs); and cpuset (CPU and NUMA node pinning). The freezer (freeze/thaw all processes in a group atomically, used for checkpoint-restore and live migration) and hugetlb controllers are also available.

Kubernetes interacts with cgroups v2 through the container runtime (containerd, CRI-O), which creates a cgroup hierarchy per pod and per container under the kubelet’s own cgroup. The pod’s requests and limits fields map to cgroup settings: resources.requests.cpu sets cpu.weight for the scheduler’s soft guarantee, resources.limits.cpu sets cpu.max for the hard ceiling, and resources.limits.memory sets memory.max — breaching this limit triggers the Linux OOM killer against the container’s processes. Kubernetes 1.25 reached stable support for cgroup v2 (cgroup v2 feature gate graduated), enabling the QoS-based eviction and improved memory pressure handling that v2’s memory.pressure interface makes possible. systemd is the primary userspace manager of the cgroup tree on modern systems: it creates slices, scopes, and service units as cgroup subtrees, and Kubernetes node components run within the system.slice or a dedicated kubepods.slice. In the confidential computing stack, cgroups constrain the resource usage of QEMU and virt-launcher processes for KubeVirt and Kata Containers workloads on the host, and the guest kernel maintains its own independent cgroup hierarchy for workloads running inside the VM.

Related

seccomp (Secure Computing Mode)

Linux Kernel Security Containers Syscall Kubernetes
seccomp (Secure Computing Mode) is a Linux kernel facility, activated by the seccomp(2) syscall, that restricts which system calls a process may subsequently invoke. In its original SECCOMP_SET_MODE_STRICT form (2005) it was a blunt instrument: the process could call only read, write, _exit, and sigreturn. The operationally useful form is SECCOMP_SET_MODE_FILTER, introduced in kernel 3.5 (2012), which accepts a BPF (classic BPF, predating eBPF) filter program that receives each syscall’s number and arguments and returns one of several verdicts: ALLOW (continue normally), ERRNO (return a specified error to the process), KILL_PROCESS or KILL_THREAD (terminate immediately without giving the process a chance to handle signals), TRAP (deliver SIGSYS), or TRACE (notify a ptracer). Once installed, a seccomp filter cannot be removed, and child processes created by fork() or threads created by clone() inherit it. Filters may only add restrictions, never loosen them — so a chain of filters is the intersection of all their allowlists. The filter runs entirely in the kernel, in BPF bytecode verified for safety, before the syscall implementation is entered, making it extremely low-overhead relative to the security it provides.

AppArmor (Application Armor)

Linux Kernel Security Mac Lsm Ubuntu Containers
AppArmor (Application Armor) is a Mandatory Access Control (MAC) system implemented as a major LSM (Linux Security Module), developed originally by Immunix and now maintained by Canonical. It is the default MAC system on Ubuntu, Debian, and their derivatives, and the default container confinement mechanism for containerd and Docker on those distributions. Where SELinux assigns security labels to every object on the system and enforces policy based on label interactions, AppArmor takes a fundamentally different approach: it confines programs by filesystem path. A profile for nginx lists the specific file paths that nginx is allowed to read, write, and execute, the network operations it may perform, and the Linux capabilities it may use — anything not listed is denied. No relabelling of the filesystem is required and no extended attributes are set: AppArmor’s confinement decisions are made purely from the path of the file being accessed and the identity of the confined process. This path-based model makes AppArmor profiles far simpler to read, write, and audit than SELinux policy, and eliminates the mislabelled-file failure mode that is the most common SELinux operational problem.

SELinux (Security-Enhanced Linux)

Linux Kernel Security Mac Lsm Rhel Containers
SELinux (Security-Enhanced Linux) is a Mandatory Access Control (MAC) implementation developed by the NSA and released as open source in 2000, merged into the mainline Linux kernel in 2.6 via the LSM framework in 2003. Its defining characteristic is default deny: unlike the standard Linux Discretionary Access Control model (file permission bits), where anything not explicitly forbidden is permitted, SELinux refuses all access that is not explicitly allowed by policy. Every process and every object — every file, socket, pipe, device node, and IPC object — carries a security context (also called a label) of the form user:role:type:level. The policy is a compiled set of rules, loaded at boot, that defines precisely which combinations of process context and object context may interact and how. An Apache web server process running in the httpd_t domain can read files labelled httpd_sys_content_t but is denied access to files labelled user_home_t or shadow_t, regardless of what Unix file permission bits say. If the web server is compromised, the attacker is confined to what httpd_t permits — typically a narrow, well-defined set of files and network operations — rather than having the full access of the user account running Apache.