Rook-Ceph¶

Distributed block storage using Rook as the Kubernetes operator for Ceph. It is the component that still has your data after a node dies, and the one with the steepest learning curve. Check ceph status more often than seems necessary and never ignore a WARN.

At a glance¶

Configuration¶

Disks¶

Each node has a raw partition labelled rook-osd, created by Ignition as the last partition with whatever the root leaves, which Rook adds as an OSD; data is replicated three ways across them. Its contents are wherever Ceph last wrote them, so inserting a partition ahead of it shifts its offset and takes the OSD data with it: changing the partition table is a reinstall, not an edit.

Ignition declares it with format: none and wipe_filesystem: true, because ceph-volume reads the BlueStore signature, not the partition table, and a reinstalled node that kept the old bytes would present an OSD from a cluster that no longer exists. Ceph declines any device that already carries a filesystem — the first thing to check when an OSD refuses to appear, see No OSDs after reprovisioning.

Storage classes¶

Object storage¶

A CephObjectStore provides an S3 endpoint at http://rook-ceph-rgw-object-store.rook-ceph.svc, so backups and log chunks live somewhere other than the block pool they protect. Two RGW instances keep a node reboot from stalling a backup; the data pool is erasure coded 2+1 — 1.5x overhead rather than 3x, safe at failureDomain: host with six nodes.

Monitoring¶

Rook's Grafana dashboards are vendored in grafana-dashboards.yaml — see Monitoring → Dashboards.

CSI driver¶

From Rook v1.20 the RBD driver is deployed by the ceph-csi-operator subchart, which deploys nothing until an OperatorConfig and a Driver CR tell it what to run. Neither chart ships them, so they live in csi-driver.yaml; without them every PVC stays Pending on ExternalProvisioning.

csi-rbac.yaml is the RBD half of the ceph-csi-operator's deploy/multifile/csi-rbac.yaml at the version the rook-ceph chart pulls in; Renovate does not see it. To refresh it:

1. Fetch and rename:

   curl -sSfL https://raw.githubusercontent.com/ceph/ceph-csi-operator/vX.Y.Z/deploy/multifile/csi-rbac.yaml \
     | sed -e 's/ceph-csi-operator-system/rook-ceph/' -e 's/ceph-csi-operator-//'
   

2. Keep only the rbd documents (CephFS, NFS and NVMe-oF are not enabled).

3. Re-indent the sequences for yamllint and prefix the cluster-scoped names.

cephx keys stay on aes¶

Ceph 20 warns about the legacy aes cipher, and every key here is deliberately on it: the kernel client krbd does not implement aes256k, so CSI keys moved to it make every rbd map fail with map failed: (22) Invalid argument while Ceph reports the rotation a success. Two ciphers in one cluster trigger AUTH_EMERGENCY_CIPHERS_SET and buy no security.

Usage¶

Requesting a volume¶

Any workload can claim the default class:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-app-data
  namespace: my-app
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi

Add storageClassName: rook-ceph-block to say so explicitly. A Deployment with two replicas sharing one RWO PVC schedules one pod and leaves the other in ContainerCreating with a multi-attach error.

Requesting a bucket¶

Ask for an ObjectBucketClaim; Rook creates the bucket and writes a ConfigMap (BUCKET_NAME, BUCKET_HOST, BUCKET_PORT) and a Secret (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY) sharing the claim's name:

apiVersion: objectbucket.io/v1alpha1
kind: ObjectBucketClaim
metadata:
  name: my-app-bucket
  namespace: my-app
spec:
  generateBucketName: my-app
  storageClassName: ceph-bucket

Health check¶

Is storage ready?¶

make storage-check

Run it after the GitOps handover, before trusting anything that mounts a volume: a CephCluster reports Ready with no OSDs, and a StorageClass exists whether or not a CSI driver registered for it. The script walks the chain in the order it breaks and stops at the first missing link:

The last one asks for a volume the way a workload would and waits TIMEOUT seconds (120 by default); NAMESPACE and CLASS override where and what it asks for.

Pitfalls¶

Recovery¶

No OSDs after reprovisioning¶

A node whose rook-osd partition still carries a previous cluster's BlueStore signature gets no OSD: the CephCluster reports Ready with HEALTH_WARN, kubectl -n rook-ceph get pods -l app=rook-ceph-osd is empty, and the first PVC — data-openbao-0 on a fresh bootstrap — stays Pending.

1. Confirm the cause:

   kubectl -n rook-ceph logs job/rook-ceph-osd-prepare-<node> | tail -3
   # skipping device "nvme0n1p2" because it contains a filesystem "ceph_bluestore"
   # skipping osd.3: "..." belonging to a different ceph cluster "..."
   

2. Rebuild the node; the installer wipes the disk:

   make reinstall LIMIT=<node>
   

3. If the node cannot be rebuilt now, wipe the partition by hand over SSH. wipefs removes the signature Rook skips on; the dd removes the BlueStore label behind it. Address it by label — the partition number differs between control-plane and worker nodes:

   sudo wipefs -a /dev/disk/by-partlabel/rook-osd
   sudo dd if=/dev/zero of=/dev/disk/by-partlabel/rook-osd bs=1M count=200
   

4. Restart the operator so it recreates the osd-prepare jobs:

   kubectl -n rook-ceph rollout restart deploy/rook-ceph-operator
   

5. Confirm an OSD per node and a binding PVC:

   make storage-check