↓ Skip to main content
Le Site de François
  1. Index/

Storage Encryption

Storage Security Encryption Kubernetes Ceph Odf Luks Kms

Storage encryption is not a single feature but a four-layer decision that must be made independently, because each layer addresses a different adversary and a different failure mode. Layer 1 — disk/OSD at-rest encryption protects against physical media theft: a decommissioned OSD or stolen drive is unreadable without the key. Layer 2 — cluster-internal wire encryption protects against a network-layer attacker who can observe traffic between storage nodes: OSDs, monitors, and clients on the cluster network. Layer 3 — PV/volume-level encryption protects against a storage operator or another tenant reading a workload’s data through the storage system itself — the threat model where the storage cluster is itself potentially untrusted or multi-tenant. Layer 4 — object storage server-side encryption provides per-object key management for S3-compatible workloads, enabling customer-managed keys (CMK) and per-tenant key isolation in object stores. These layers are independent and composable: enabling Layer 1 without Layer 2 protects against physical theft but not a network interceptor; enabling Layer 3 without Layer 1 protects against the storage operator but not physical media extraction. A complete encryption posture addresses all four explicitly, even if some layers are deliberately left disabled with a documented rationale.

Layer 1 — disk/OSD at-rest encryption. In Ceph and Red Hat ODF, OSD-level encryption wraps each OSD’s underlying block device with dm-crypt/LUKS (currently LUKS v1 in Ceph, for broadest compatibility). When OSD provisioning runs, the OSD service generates a unique per-OSD encryption key, passes it to the Ceph Monitors via an authenticated JSON payload over stdin, and the Monitor stores it in the cluster keyring. Encryption is not enabled by default and must be set explicitly — in ODF via the installation wizard (Security and network step → “Encryption at rest”) or in rook-ceph via spec.encryption.enable: true in the CephCluster CR; in Ceph directly via the OSD service spec with encryption: true. Keys can be managed internally (stored in the Monitor keyring) or externally via a KMS (HashiCorp Vault, Thales CipherTrust, IBM HPCS, AWS KMS) configured in rook-ceph’s ConfigMap and referenced in the cluster spec. With external KMS, OSD keys are wrapped by a Vault-managed master key and only accessible to the Ceph cluster’s service identity — the KMS becomes the trust anchor for all OSD data. The equivalent in other systems: LVM/LUKS on individual disks for traditional Linux servers; AWS EBS volume encryption with KMS; Azure Disk Encryption; GCP CMEK-encrypted Persistent Disks; Pure Storage and NetApp hardware arrays with self-encrypting drives (SEDs) managed by the controller. Important caveat: OSD encryption protects the physical media; data in DRAM on the OSD host, in OSD buffer caches, and in transit between OSDs is not covered by this layer.

Layer 2 — cluster-internal wire encryption. Ceph’s messenger v2 (msgr2) protocol, the default since RHCS 4 and ODF 4.x, supports three connection modes: crc (integrity checking, no confidentiality), secure (authenticated encryption using AES-GCM), and prefer-secure (negotiate secure if the peer supports it, fall back to crc). Enabling secure mode encrypts all Ceph protocol traffic between clients, OSDs, Monitors, and the MDS — the full cluster fabric — with AES-128-GCM. In ODF, this is enabled via rook-ceph by setting network.encryption.enabled: true in the CephCluster CR, which sets ms_cluster_mode = secure and ms_service_mode = secure in the Ceph configuration. The Ceph Object Gateway (RGW) and ODF’s S3 endpoint terminate TLS at the RGW or at the Ingress/Route in front of it; the connection from the RGW to the OSDs is protected by msgr2 secure mode. For NFS exports from Ceph (via NFS-Ganesha backed by CephFS), NFS v4.1+ Kerberos (krb5p security flavour) provides wire encryption; without Kerberos, NFS traffic is unencrypted and should be protected by IPsec or MACsec at the network layer if traversing untrusted segments. For iSCSI (via Ceph’s tcmu-runner iSCSI gateway), iSCSI over TLS or IPsec is required for wire encryption. Other storage systems: NetApp ONTAP uses CIFS/SMB3 encryption and NFS Kerberos; Pure FlashArray offers Replication Encryption over TLS; AWS EFS uses TLS for in-transit encryption enabled by mount option (-o tls).

Layer 3 — PV/volume-level encryption. This layer encrypts individual Persistent Volumes with per-PV keys, so that even the storage operator (who controls the OSD layer) cannot read a specific workload’s data without the KMS credential bound to that workload’s namespace or PVC. In ODF, this is provided by the ocs-storagecluster-ceph-rbd-encrypted StorageClass (created automatically when StorageClass encryption is enabled in the ODF wizard). When a PVC is created against this StorageClass, the Ceph CSI driver requests a new encryption key from Vault (or another configured KMS) under a path namespaced to the PVC UID, wraps the key using Vault’s transit engine, stores the wrapped key in the PVC’s volume attributes, and passes the unwrapped key to dm-crypt on the node where the PV is attached — the data on the OSD is encrypted by both the OSD-level LUKS key (Layer 1) and the PV-level key (Layer 3), in independent layers. Revoking the Vault path for a specific PVC makes that volume’s data permanently inaccessible without affecting any other PVC. The key hierarchy is visible in Vault: each PVC generates a unique key at odf/<pvc-uid>, and OSD keys appear at rook-ceph-osd-encryption-key-<device-set>. Per-namespace key isolation is achievable by configuring separate Vault namespaces or paths per OpenShift namespace, binding the StorageClass to a specific KMS path prefix. The equivalent in other platforms: AWS EBS with per-volume CMKs; Kubernetes StorageClass with encrypted: true and kmsKeyId for EBS CSI; the secrets-store-csi-driver can serve encryption key material to volume init containers; Portworx PX-Encrypt with per-namespace key management; Rook-Ceph with external KMS applies the same architecture to non-ODF Ceph.

Layer 4 — object storage server-side encryption. ODF’s object storage layer is NooBaa (for multi-cloud gateway object routing) and Ceph RGW (for S3-compatible block object storage). Ceph RGW supports three S3 server-side encryption modes mirroring AWS S3 SSE: SSE-S3 (RGW manages the keys internally, one key per object, transparent to the client — x-amz-server-side-encryption: AES256), SSE-KMS (RGW requests per-object keys from an external KMS such as Vault using the x-amz-server-side-encryption: aws:kms header and the x-amz-server-side-encryption-aws-kms-key-id header specifying the KMS key, enabling customer-managed keys with full KMS audit trails), and SSE-C (the client provides the encryption key per-request in the x-amz-server-side-encryption-customer-key header; RGW uses it to encrypt the object and discards it — the client bears full key management responsibility, and loss of the key means permanent loss of the data). NooBaa encrypts its internal data using a root master key stored in a Secret or optionally in Vault (noobaa-root-master-key-backend in the Vault path listing). AWS S3, Azure Blob Storage, and GCP Cloud Storage all implement equivalent SSE-S3/SSE-KMS/SSE-C (or CMEK) models. The KMS integration pattern is consistent across all layers: a Vault kv or transit secrets engine, an AppRole or Kubernetes auth method for the CSI driver or RGW service identity, and a Vault policy restricting each component to its own key namespace — the same Vault deployment can serve Layer 1 OSD key wrapping, Layer 3 PV key issuance, and Layer 4 SSE-KMS simultaneously, with separate policies isolating each.

Related

KMS v2 (Kubernetes KMS Provider v2)

Kubernetes Security Encryption Etcd Kms Secrets Key-Management
KMS v2 (Kubernetes KMS Provider version 2) is the stable (GA since Kubernetes 1.29, KMS v1 deprecated in 1.28 and disabled by default in 1.29) mechanism for encrypting the contents of the Kubernetes etcd datastore at rest using an external key management service. Without encryption, Kubernetes Secrets stored in etcd are base64-encoded — trivially decodable by anyone with read access to the etcd data files or a snapshot. With KMS v2 enabled, resources written to etcd — Secrets, ConfigMaps, and any other API objects selected by the EncryptionConfiguration — are encrypted before being persisted, using a unique per-object key derived locally. The envelope encryption scheme means those per-object keys are never stored in plaintext: only their encrypted form lives in etcd, and only the external KMS holds the wrapping key. The encryption configuration is declared in a file referenced by the API server’s --encryption-provider-config flag, with kms: apiVersion: v2 selecting the KMS v2 code path, and endpoint: unix:///path/to/plugin.sock pointing at the plugin’s Unix domain socket.

NBDE / Clevis / Tang (Network-Bound Disk Encryption)

Security Encryption Storage Linux Luks Boot Rhel
NBDE (Network-Bound Disk Encryption) is an approach to automatic LUKS disk unlocking that binds the volume key not to hardware state (a TPM PCR measurement) but to network presence: a LUKS-encrypted volume unlocks automatically at boot if and only if the machine can reach a designated Tang server on a trusted network. Remove the machine from that network — because it was stolen, because a data centre drive was pulled, because someone exfiltrated the hardware — and the volume key becomes unrecoverable without a fallback passphrase. The threat model is therefore complementary to TPM-based unlocking: TPM sealing asks “is this the right software stack?” and locks the key to a specific platform measurement; NBDE asks “is this machine on the trusted network?” and locks the key to network presence. Neither addresses both threat classes alone, which is why the two are routinely combined — and why RHEL formalises NBDE as a subcategory of the broader Policy-Based Decryption (PBD) framework that the Clevis pin system implements.

LUKS (Linux Unified Key Setup)

Security Encryption Storage Linux Boot
LUKS (Linux Unified Key Setup) is the standard specification for block device encryption on Linux, created by Clemens Fruhwirth in 2004. It sits above the kernel’s dm-crypt subsystem — which performs the actual AES sector-by-sector encryption via the device mapper — and adds a structured, on-disk header that decouples key management from the encryption itself. Any block device can be a LUKS container: a partition, a logical volume, a loop device; anything that sits beneath it (filesystem, swap, LVM) is encrypted transparently, with no changes required to the software using it. The managed cryptsetup tool and the libcryptsetup library provide userspace access to LUKS volumes, and are the canonical interface for all operations on them.