Skip to main content
  1. Index/

Syscall (System Call)

A system call (syscall) is the formal interface through which a user-space process asks the kernel to perform a privileged operation on its behalf — opening a file, allocating memory, creating a process, establishing a network connection, sending a signal, or any other action that requires kernel mediation. User-space code runs at CPU privilege level 3 (ring 3) and cannot directly access hardware, manipulate kernel data structures, or perform I/O; the kernel runs at ring 0 with unrestricted access. A syscall is the crossing point: the process places its request in a defined register convention and issues a syscall instruction (on x86-64) that atomically switches the CPU to ring 0 and transfers control to the kernel’s syscall dispatch table. The kernel validates the request, performs the operation if permitted by standard Unix permissions and any active LSM hooks, and returns the result. From a security perspective, the syscall boundary is the complete list of what a process can ask the kernel to do — and therefore the complete list of operations that security controls like seccomp and BPF LSM can police.

Linux currently has roughly 350 syscalls on x86-64 (the exact count varies by architecture and kernel version), ranging from mundane operations (read, write, open, close, mmap) to powerful but narrow ones (ptrace, mount, kexec_load, perf_event_open, bpf) to obsolete or rarely used ones (uselib, sysfs, nfsservctl). Architecture matters: a 64-bit kernel also handles 32-bit compatibility calls through a separate dispatch table, which is why seccomp profiles must account for both x86_64 and x32 or i386 syscall numbers when running on a host that allows 32-bit binaries — a frequent source of container escape bugs when legacy compat syscalls are left unfiltered. Each syscall is identified by a number and a name; the kernel exposes its current syscall table via /proc/kallsyms and ausyscall --dump. The security relevance of the syscall boundary is that it is the only way for a process to interact with the kernel: a container that is denied network access via network namespace isolation but can call socket() and finds a namespace escape vulnerability is still constrained by whether seccomp blocks socket() outright — defence in depth at the syscall layer provides a fallback when higher-level isolation is bypassed.

Syscalls are the foundation on which the other entries in this section compose. seccomp filters the allowed set of syscalls a process may invoke. cgroups intercept at the scheduling and resource accounting layer triggered by kernel paths that syscalls enter. eBPF programs attach to kprobes and tracepoints at individual syscall entry and exit points, giving observability and policy enforcement with per-argument visibility. LSM hooks fire from within syscall implementation paths — security_file_open() fires from openat(), security_socket_connect() from connect() — allowing MAC decisions based on the full semantic context of the operation rather than just the syscall number. In confidential computing, the guest kernel’s syscall interface is what KubeVirt and Kata Containers isolate behind a VM boundary: a kernel exploit that would normally give a container an escape to the host is contained within the guest kernel, because the host’s kernel is only reachable through the hypervisor interface, not through the guest syscall table.

Related

AppArmor (Application Armor)

AppArmor (Application Armor) is a Mandatory Access Control (MAC) system implemented as a major LSM (Linux Security Module), developed originally by Immunix and now maintained by Canonical. It is the default MAC system on Ubuntu, Debian, and their derivatives, and the default container confinement mechanism for containerd and Docker on those distributions. Where SELinux assigns security labels to every object on the system and enforces policy based on label interactions, AppArmor takes a fundamentally different approach: it confines programs by filesystem path. A profile for nginx lists the specific file paths that nginx is allowed to read, write, and execute, the network operations it may perform, and the Linux capabilities it may use — anything not listed is denied. No relabelling of the filesystem is required and no extended attributes are set: AppArmor’s confinement decisions are made purely from the path of the file being accessed and the identity of the confined process. This path-based model makes AppArmor profiles far simpler to read, write, and audit than SELinux policy, and eliminates the mislabelled-file failure mode that is the most common SELinux operational problem.

SELinux (Security-Enhanced Linux)

SELinux (Security-Enhanced Linux) is a Mandatory Access Control (MAC) implementation developed by the NSA and released as open source in 2000, merged into the mainline Linux kernel in 2.6 via the LSM framework in 2003. Its defining characteristic is default deny: unlike the standard Linux Discretionary Access Control model (file permission bits), where anything not explicitly forbidden is permitted, SELinux refuses all access that is not explicitly allowed by policy. Every process and every object — every file, socket, pipe, device node, and IPC object — carries a security context (also called a label) of the form user:role:type:level. The policy is a compiled set of rules, loaded at boot, that defines precisely which combinations of process context and object context may interact and how. An Apache web server process running in the httpd_t domain can read files labelled httpd_sys_content_t but is denied access to files labelled user_home_t or shadow_t, regardless of what Unix file permission bits say. If the web server is compromised, the attacker is confined to what httpd_t permits — typically a narrow, well-defined set of files and network operations — rather than having the full access of the user account running Apache.

seccomp (Secure Computing Mode)

seccomp (Secure Computing Mode) is a Linux kernel facility, activated by the seccomp(2) syscall, that restricts which system calls a process may subsequently invoke. In its original SECCOMP_SET_MODE_STRICT form (2005) it was a blunt instrument: the process could call only read, write, _exit, and sigreturn. The operationally useful form is SECCOMP_SET_MODE_FILTER, introduced in kernel 3.5 (2012), which accepts a BPF (classic BPF, predating eBPF) filter program that receives each syscall’s number and arguments and returns one of several verdicts: ALLOW (continue normally), ERRNO (return a specified error to the process), KILL_PROCESS or KILL_THREAD (terminate immediately without giving the process a chance to handle signals), TRAP (deliver SIGSYS), or TRACE (notify a ptracer). Once installed, a seccomp filter cannot be removed, and child processes created by fork() or threads created by clone() inherit it. Filters may only add restrictions, never loosen them — so a chain of filters is the intersection of all their allowlists. The filter runs entirely in the kernel, in BPF bytecode verified for safety, before the syscall implementation is entered, making it extremely low-overhead relative to the security it provides.