Rook-Ceph¶

Distributed block storage using Rook as the Kubernetes operator for Ceph. It is the component that still has your data after a node dies, and the one with the steepest learning curve. Check ceph status more often than seems necessary and never ignore a WARN.

At a glance¶


Namespace	`rook-ceph`
Depends on	A raw `rook-osd` partition on every worker, written at install time
If it is down	Every pod with a volume — `openbao` first, which turns a storage problem into a cluster problem
Health check	`make storage-check` — it binds a real PVC, which `ceph status` alone does not prove
UI	`rook.infra.k8s.wlkr.ch`
Files	`payload/platform/rook-ceph-operator/`: the `rook-ceph` chart, the operator itself. `payload/platform/rook-ceph-cluster/`: the `rook-ceph-cluster` chart, whose `valuesObject` holds the `CephCluster`, the object store and the storage classes. `payload/platform/rook-ceph/`: the CSI driver and its RBAC, the dashboards and the UI route

Configuration¶

Disks¶

Each worker has a raw partition labelled rook-osd, created by Ignition as the last partition with whatever the root leaves, which Rook adds as an OSD; data is replicated three ways across them. Control planes get none, and the CephCluster's OSD placement excludes them: on one shared NVMe, etcd's fsyncs and BlueStore's writes stalled each other until OSDs hit their heartbeat timeout and aborted. Its contents are wherever Ceph last wrote them, so inserting a partition ahead of it shifts its offset and takes the OSD data with it: changing the partition table is a reinstall, not an edit.

Ignition declares it with format: none and wipe_filesystem: true, because ceph-volume reads the BlueStore signature, not the partition table, and a reinstalled node that kept the old bytes would present an OSD from a cluster that no longer exists. Ceph declines any device that already carries a filesystem — the first thing to check when an OSD refuses to appear, see No OSDs after reprovisioning.

Storage classes¶

StorageClass	Provides	Notes
`rook-ceph-block`	RBD block volumes, cluster default	`ReadWriteOnce` only; `ReadWriteMany` needs CephFS, which is not enabled. Block PVs are deleted with their claim, and every Application prunes, so removing a PVC from Git removes the volume — see Backups
`ceph-bucket`	S3 buckets from the object store	Reclaim `Retain`, so deleting a claim cannot delete the bucket

Object storage¶

A CephObjectStore provides an S3 endpoint at http://rook-ceph-rgw-object-store.rook-ceph.svc, so backups and log chunks live somewhere other than the block pool they protect. The infra Gateway publishes the same endpoint as https://s3.infra.k8s.wlkr.ch for Velero, whose S3 client only signs cleanly over TLS. More than one RGW instance (gateway.instances) keeps a node reboot from stalling a backup, which only holds while they sit on separate nodes: gateway.placement carries the spread constraint, because the placement.all block belongs to the CephCluster and never reaches this CR. It is a hard one: every gateway is daemon a, so on one node the exporter reports their counters under the same labels and Prometheus drops them, and the descheduler never evicts a system-cluster-critical pod to fix that. The data pool is erasure coded (dataChunks and codingChunks): less overhead than three replicas, safe at failureDomain: host only with more workers than chunks.

Monitoring¶

In rook-ceph-cluster/application.yaml's valuesObject:

Setting	Why
`cephClusterSpec.monitoring.enabled`	Turns on the mgr `prometheus` module and lets the operator maintain a `ServiceMonitor`. Without it a degraded pool, a down OSD or a near-full cluster is invisible to Prometheus, and a full Ceph cluster stops accepting writes
`monitoring.createPrometheusRules`	Ships Ceph's own alerting rules as a `PrometheusRule`. Its CRD comes from the `prometheus-operator-crds` Application; a sync before it lands fails and is retried — see Monitoring
`cephClusterSpec.cephConfig` `mgr/dashboard/*`	Sets the dashboard's Prometheus and Grafana links through the mon config store, which the operator reapplies on every reconcile, rather than a one-off `ceph dashboard set-*` Job
`mon_cluster_log_level: info`	The Ceph default is `debug`, and the mons copy the cluster log to stderr, so Rook's probes and the mgr's pgmap ticks became most of the mon log volume in Loki. Warnings and errors are unaffected

Rook's Grafana dashboards are vendored in grafana-dashboards.yaml — see Monitoring → Dashboards.

CSI driver¶

From Rook v1.20 the RBD driver is deployed by the ceph-csi-operator subchart, which deploys nothing until an OperatorConfig and a Driver CR tell it what to run. Neither chart ships them, so they live in csi-driver.yaml; without them every PVC stays Pending on ExternalProvisioning.

Setting	Why
`csi-driver.yaml` in `rook-ceph`, with `SkipDryRunOnMissingResource`	The CRs need the ceph-csi-operator's CRDs, which arrive with the operator chart
`OperatorConfig` pointing at the chart's image-set `ConfigMap`	A chart bump moves every sidecar and the cephcsi image together
Driver name `rook-ceph.rbd.csi.ceph.com`	Derived from the namespace and must match the StorageClasses' `provisioner`. RBD is the only driver; CephFS is not enabled
Two controller plugin replicas, `RollingUpdate` without surge, a PodDisruptionBudget	A drain of the provisioner's node otherwise stalls every provision, attach, resize and snapshot until the pod reschedules. The operator already keeps the replicas on distinct nodes with required anti-affinity and runs every sidecar with leader election, so the `Driver` CR carries no spread constraint, and no field for one
`csi-rbac.yaml`	The ceph-csi-operator creates no ServiceAccounts, and the names are fixed by `CSI_SERVICE_ACCOUNT_PREFIX` (empty here): `rbd-ctrlplugin-sa` and `rbd-nodeplugin-sa`. Without them the Deployment and DaemonSet exist but create no pods

csi values on the operator chart do nothing

Helm accepts values a chart no longer reads. Driver settings go in csi-driver.yaml.

csi-rbac.yaml is the RBD half of the ceph-csi-operator's deploy/multifile/csi-rbac.yaml at the version the rook-ceph chart pulls in; Renovate does not see it. To refresh it:

Fetch and rename:

curl -sSfL https://raw.githubusercontent.com/ceph/ceph-csi-operator/vX.Y.Z/deploy/multifile/csi-rbac.yaml \
  | sed -e 's/ceph-csi-operator-system/rook-ceph/' -e 's/ceph-csi-operator-//'

Keep only the rbd documents (CephFS, NFS and NVMe-oF are not enabled).
Re-indent the sequences for yamllint and prefix the cluster-scoped names.
Cut the nodeplugin ClusterRole back: drop secrets, serviceaccounts and serviceaccounts/token, and keep get on the rook-csi-rbd-node and rook-csi-rbd-provisioner Secrets in the rbd-nodeplugin-r Role. Its token sits on every node; upstream's grants serve only the encryption KMS providers, which nothing here uses, and minting any ServiceAccount's token is cluster-admin.

cephx keys stay on aes¶

Ceph 20 warns about the legacy aes cipher, and every key here is deliberately on it: the kernel client krbd does not implement aes256k, so CSI keys moved to it make every rbd map fail with map failed: (22) Invalid argument while Ceph reports the rotation a success. Two ciphers in one cluster trigger AUTH_EMERGENCY_CIPHERS_SET and buy no security. In rook-ceph-cluster/application.yaml's valuesObject:

Setting	Why
`keyType: aes`	krbd cannot use anything else
`mon_auth_allow_insecure_key: "true"`	Its default differs between mons; one that says `false` refuses an OSD's start-up key fetch, and an OSD whose on-disk key predates a rotation then fails with `Permission denied`
`keyRotationPolicy: KeyGeneration`	Changing `keyType` alone never rotates existing keys while the policy is `Disabled`
`keyGeneration`	Rotation only triggers on an increase, so a generation can never be lowered to undo one
`healthCheck.muteHealthWarning`	The `AUTH_*` warnings would otherwise keep the Application `Degraded` while the data is healthy. Temporary: remove once RBD moves to the userspace `rbd-nbd` mounter and the keys can leave `aes`

Usage¶

Requesting a volume¶

Any workload can claim the default class:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-app-data
  namespace: my-app
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi

Add storageClassName: rook-ceph-block to say so explicitly. A Deployment with two replicas sharing one RWO PVC schedules one pod and leaves the other in ContainerCreating with a multi-attach error.

Requesting a bucket¶

Ask for an ObjectBucketClaim; Rook creates the bucket and writes a ConfigMap (BUCKET_NAME, BUCKET_HOST, BUCKET_PORT) and a Secret (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY) sharing the claim's name:

apiVersion: objectbucket.io/v1alpha1
kind: ObjectBucketClaim
metadata:
  name: my-app-bucket
  namespace: my-app
spec:
  generateBucketName: my-app
  storageClassName: ceph-bucket

Health check¶

Is storage ready?¶

make storage-check

Run it after the GitOps handover, before trusting anything that mounts a volume: a CephCluster reports Ready with no OSDs, and a StorageClass exists whether or not a CSI driver registered for it. The script walks the chain in the order it breaks and stops at the first missing link:

Check	What its absence means
`CephCluster` is `Ready`	the cluster Application has not synced yet
an OSD pod is `Running`	Rook took no disk — see No OSDs after reprovisioning
`ceph health` is not `HEALTH_ERR`	Ceph itself is unwell; `HEALTH_WARN` is allowed through
the CSI driver is registered	no `Driver` CR, so the ceph-csi-operator deployed nothing
node plugin and provisioner are up	usually a ServiceAccount the DaemonSet cannot find
a 1 GiB PVC binds and is cleaned up	the only check that proves the other five

The last one asks for a volume the way a workload would and waits TIMEOUT seconds (default in scripts/storage-check.sh); NAMESPACE and CLASS override where and what it asks for.

Pitfalls¶

Ceph warnings show as Degraded

ArgoCD maps HEALTH_WARN to Degraded, so rook-ceph-cluster reports Degraded for the whole of a warning; nothing else waits on it — see GitOps.

Rook skips tainted nodes

The operator neither creates nor updates an OSD on a node with a taint placement.all does not tolerate, PreferNoSchedule included, and logs placement settings do not match. Kured taints every node waiting for its reboot lock, so that taint is tolerated.

Recovery¶

No OSDs after reprovisioning¶

A node whose rook-osd partition still carries a previous cluster's BlueStore signature gets no OSD: the CephCluster reports Ready with HEALTH_WARN, kubectl -n rook-ceph get pods -l app=rook-ceph-osd is empty, and the first PVC — data-openbao-0 on a fresh bootstrap — stays Pending.

The old cluster's data is already gone

Once the mons that held the cluster map left with the old control plane, those OSDs cannot be re-adopted by anything. Take a backup off the disks before a rebuild if you need one.

Confirm the cause:

kubectl -n rook-ceph logs job/rook-ceph-osd-prepare-<node> | tail -3
# skipping device "nvme0n1p10" because it contains a filesystem "ceph_bluestore"
# skipping osd.3: "..." belonging to a different ceph cluster "..."

Rebuild the node, steps 3 to 7 of Rebuilding or repartitioning a node: make reinstall only arms the PXE menu, and the installer wipes the disk when the node network-boots.
If the node cannot be rebuilt now, wipe the partition by hand over SSH. wipefs removes the signature Rook skips on; the dd removes the BlueStore label behind it. Address it by label, as Ignition does, rather than by the device name in the log:
```
sudo wipefs -a /dev/disk/by-partlabel/rook-osd
sudo dd if=/dev/zero of=/dev/disk/by-partlabel/rook-osd bs=1M count=200
```

Restart the operator so it recreates the osd-prepare jobs:

kubectl -n rook-ceph rollout restart deploy/rook-ceph-operator

Confirm an OSD per node and a binding PVC:
```
make storage-check
```