# Rook-Ceph

Distributed block storage using [Rook](<https://rook.io/>) as the Kubernetes operator for [Ceph](<https://ceph.io/>). It is the component that still has your data after a node dies, and the one with the steepest learning curve. Check `ceph status` more often than seems necessary and never ignore a `WARN`.

## At a glance

|  |  |
| --- | --- |
| Namespace | `rook-ceph` |
| Depends on | A raw `rook-osd` partition on every worker, written at install time |
| If it is down | Every pod with a volume — `openbao` first, which turns a storage problem into a cluster problem |
| Health check | `make storage-check` — it binds a real PVC, which `ceph status` alone does not prove |
| UI | `rook.infra.k8s.wlkr.ch` |
| Files | `payload/platform/rook-ceph-operator/`: the `rook-ceph` chart, the operator itself. `payload/platform/rook-ceph-cluster/`: the `rook-ceph-cluster` chart, whose `valuesObject` holds the `CephCluster`, the object store and the storage classes. `payload/platform/rook-ceph/`: the CSI driver and its RBAC, the dashboards and the UI route |

## Configuration

### Disks

Each worker has a raw partition labelled `rook-osd`, created by Ignition as the **last** partition with whatever the root leaves, which Rook adds as an OSD; data is replicated three ways across them. Control planes get none, and the CephCluster's OSD placement excludes them: on one shared NVMe, etcd's fsyncs and BlueStore's writes stalled each other until OSDs hit their heartbeat timeout and aborted. Its contents are wherever Ceph last wrote them, so inserting a partition ahead of it shifts its offset and takes the OSD data with it: changing the partition table is a [reinstall](<https://homelab.wlkr.ch/operations/nodes/index.md>), not an edit.

Ignition declares it with `format: none` and `wipe_filesystem: true`, because `ceph-volume` reads the BlueStore *signature*, not the partition table, and a reinstalled node that kept the old bytes would present an OSD from a cluster that no longer exists. Ceph declines any device that already carries a filesystem — the first thing to check when an OSD refuses to appear, see [No OSDs after reprovisioning](<#no-osds-after-reprovisioning>).

### Storage classes

| StorageClass | Provides | Notes |
| --- | --- | --- |
| `rook-ceph-block` | RBD block volumes, cluster default | `ReadWriteOnce` only; `ReadWriteMany` needs CephFS, which is not enabled. Block PVs are deleted with their claim, and every Application prunes, so removing a PVC from Git removes the volume — see [Backups](<https://homelab.wlkr.ch/operations/backups/index.md>) |
| `ceph-bucket` | S3 buckets from the object store | Reclaim `Retain`, so deleting a claim cannot delete the bucket |

### Object storage

A `CephObjectStore` provides an S3 endpoint at `http://rook-ceph-rgw-object-store.rook-ceph.svc`, so backups and log chunks live somewhere other than the block pool they protect. The infra Gateway publishes the same endpoint as `https://s3.infra.k8s.wlkr.ch` for [Velero](<https://homelab.wlkr.ch/platform/velero/#configuration>), whose S3 client only signs cleanly over TLS. More than one RGW instance (`gateway.instances`) keeps a node reboot from stalling a backup, which only holds while they sit on separate nodes: `gateway.placement` carries the spread constraint, because the `placement.all` block belongs to the `CephCluster` and never reaches this CR. It is a hard one: every gateway is daemon `a`, so on one node the exporter reports their counters under the same labels and Prometheus drops them, and the descheduler never evicts a `system-cluster-critical` pod to fix that. The data pool is erasure coded (`dataChunks` and `codingChunks`): less overhead than three replicas, safe at `failureDomain: host` only with more workers than chunks.

### Monitoring

In `rook-ceph-cluster/application.yaml`'s `valuesObject`:

| Setting | Why |
| --- | --- |
| `cephClusterSpec.monitoring.enabled` | Turns on the mgr `prometheus` module and lets the operator maintain a `ServiceMonitor`. Without it a degraded pool, a down OSD or a near-full cluster is invisible to Prometheus, and a full Ceph cluster stops accepting writes |
| `monitoring.createPrometheusRules` | Ships Ceph's own alerting rules as a `PrometheusRule`. Its CRD comes from the `prometheus-operator-crds` Application; a sync before it lands fails and is retried — see [Monitoring](<https://homelab.wlkr.ch/platform/monitoring/#crds>) |
| `cephClusterSpec.cephConfig` `mgr/dashboard/*` | Sets the dashboard's Prometheus and Grafana links through the mon config store, which the operator reapplies on every reconcile, rather than a one-off `ceph dashboard set-*` Job |
| `mon_cluster_log_level: info` | The Ceph default is `debug`, and the mons copy the cluster log to stderr, so Rook's probes and the mgr's pgmap ticks became most of the mon log volume in [Loki](<https://homelab.wlkr.ch/platform/logging/index.md>). Warnings and errors are unaffected |

Rook's Grafana dashboards are vendored in `grafana-dashboards.yaml` — see [Monitoring → Dashboards](<https://homelab.wlkr.ch/platform/monitoring/#dashboards>).

### CSI driver

From Rook v1.20 the RBD driver is deployed by the ceph-csi-operator subchart, which deploys nothing until an `OperatorConfig` and a `Driver` CR tell it what to run. Neither chart ships them, so they live in `csi-driver.yaml`; without them every PVC stays `Pending` on `ExternalProvisioning`.

| Setting | Why |
| --- | --- |
| `csi-driver.yaml` in `rook-ceph`, with `SkipDryRunOnMissingResource` | The CRs need the ceph-csi-operator's CRDs, which arrive with the operator chart |
| `OperatorConfig` pointing at the chart's image-set `ConfigMap` | A chart bump moves every sidecar and the cephcsi image together |
| Driver name `rook-ceph.rbd.csi.ceph.com` | Derived from the namespace and must match the StorageClasses' `provisioner`. RBD is the only driver; CephFS is not enabled |
| Two controller plugin replicas, `RollingUpdate` without surge, a PodDisruptionBudget | A drain of the provisioner's node otherwise stalls every provision, attach, resize and snapshot until the pod reschedules. The operator already keeps the replicas on distinct nodes with required anti-affinity and runs every sidecar with leader election, so the `Driver` CR carries no spread constraint, and no field for one |
| `csi-rbac.yaml` | The ceph-csi-operator creates no ServiceAccounts, and the names are fixed by `CSI_SERVICE_ACCOUNT_PREFIX` (empty here): `rbd-ctrlplugin-sa` and `rbd-nodeplugin-sa`. Without them the Deployment and DaemonSet exist but create no pods |

> [!WARNING]
>
> **`csi` values on the operator chart do nothing**
>
> Helm accepts values a chart no longer reads. Driver settings go in `csi-driver.yaml`.

`csi-rbac.yaml` is the RBD half of the ceph-csi-operator's `deploy/multifile/csi-rbac.yaml` at the version the rook-ceph chart pulls in; Renovate does not see it. To refresh it:

1. Fetch and rename:

   ```bash
   curl -sSfL https://raw.githubusercontent.com/ceph/ceph-csi-operator/vX.Y.Z/deploy/multifile/csi-rbac.yaml \
     | sed -e 's/ceph-csi-operator-system/rook-ceph/' -e 's/ceph-csi-operator-//'
   ```
2. Keep only the rbd documents (CephFS, NFS and NVMe-oF are not enabled).
3. Re-indent the sequences for yamllint and prefix the cluster-scoped names.
4. Cut the nodeplugin ClusterRole back: drop `secrets`, `serviceaccounts` and `serviceaccounts/token`, and keep `get` on the `rook-csi-rbd-node` and `rook-csi-rbd-provisioner` Secrets in the `rbd-nodeplugin-r` Role. Its token sits on every node; upstream's grants serve only the encryption KMS providers, which nothing here uses, and minting any ServiceAccount's token is cluster-admin.

### cephx keys stay on aes

Ceph 20 warns about the legacy `aes` cipher, and every key here is deliberately on it: the kernel client krbd does not implement `aes256k`, so CSI keys moved to it make every `rbd map` fail with `map failed: (22) Invalid argument` while Ceph reports the rotation a success. Two ciphers in one cluster trigger `AUTH_EMERGENCY_CIPHERS_SET` and buy no security. In `rook-ceph-cluster/application.yaml`'s `valuesObject`:

| Setting | Why |
| --- | --- |
| `keyType: aes` | krbd cannot use anything else |
| `mon_auth_allow_insecure_key: "true"` | Its default differs between mons; one that says `false` refuses an OSD's start-up key fetch, and an OSD whose on-disk key predates a rotation then fails with `Permission denied` |
| `keyRotationPolicy: KeyGeneration` | Changing `keyType` alone never rotates existing keys while the policy is `Disabled` |
| `keyGeneration` | Rotation only triggers on an increase, so a generation can never be lowered to undo one |
| `healthCheck.muteHealthWarning` | The `AUTH_*` warnings would otherwise keep the Application `Degraded` while the data is healthy. Temporary: remove once RBD moves to the userspace `rbd-nbd` mounter and the keys can leave `aes` |

## Usage

### Requesting a volume

Any workload can claim the default class:

```yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-app-data
  namespace: my-app
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
```

Add `storageClassName: rook-ceph-block` to say so explicitly. A Deployment with two replicas sharing one RWO PVC schedules one pod and leaves the other in `ContainerCreating` with a multi-attach error.

### Requesting a bucket

Ask for an `ObjectBucketClaim`; Rook creates the bucket and writes a `ConfigMap` (`BUCKET_NAME`, `BUCKET_HOST`, `BUCKET_PORT`) and a `Secret` (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`) sharing the claim's name:

```yaml
apiVersion: objectbucket.io/v1alpha1
kind: ObjectBucketClaim
metadata:
  name: my-app-bucket
  namespace: my-app
spec:
  generateBucketName: my-app
  storageClassName: ceph-bucket
```

## Health check

### Is storage ready?

```bash
make storage-check
```

Run it after the GitOps handover, before trusting anything that mounts a volume: a `CephCluster` reports `Ready` with no OSDs, and a `StorageClass` exists whether or not a CSI driver registered for it. The script walks the chain in the order it breaks and stops at the first missing link:

| Check | What its absence means |
| --- | --- |
| `CephCluster` is `Ready` | the cluster Application has not synced yet |
| an OSD pod is `Running` | Rook took no disk — see [No OSDs after reprovisioning](<#no-osds-after-reprovisioning>) |
| `ceph health` is not `HEALTH_ERR` | Ceph itself is unwell; `HEALTH_WARN` is allowed through |
| the CSI driver is registered | no `Driver` CR, so the ceph-csi-operator deployed nothing |
| node plugin and provisioner are up | usually a ServiceAccount the DaemonSet cannot find |
| a 1 GiB PVC binds and is cleaned up | the only check that proves the other five |

The last one asks for a volume the way a workload would and waits `TIMEOUT` seconds (default in `scripts/storage-check.sh`); `NAMESPACE` and `CLASS` override where and what it asks for.

## Pitfalls

> [!WARNING]
>
> **Ceph warnings show as Degraded**
>
> ArgoCD maps `HEALTH_WARN` to Degraded, so `rook-ceph-cluster` reports Degraded for the whole of a warning; nothing else waits on it — see [GitOps](<https://homelab.wlkr.ch/architecture/gitops/index.md>).

> [!WARNING]
>
> **Rook skips tainted nodes**
>
> The operator neither creates nor updates an OSD on a node with a taint `placement.all` does not tolerate, `PreferNoSchedule` included, and logs `placement settings do not match`. [Kured](<https://homelab.wlkr.ch/platform/kured/index.md>) taints every node waiting for its reboot lock, so that taint is tolerated.

## Recovery

### No OSDs after reprovisioning

A node whose `rook-osd` partition still carries a previous cluster's BlueStore signature gets no OSD: the `CephCluster` reports `Ready` with `HEALTH_WARN`, `kubectl -n rook-ceph get pods -l app=rook-ceph-osd` is empty, and the first PVC — `data-openbao-0` on a fresh bootstrap — stays `Pending`.

> [!CAUTION]
>
> **The old cluster's data is already gone**
>
> Once the mons that held the cluster map left with the old control plane, those OSDs cannot be re-adopted by anything. Take a backup off the disks *before* a rebuild if you need one.

1. Confirm the cause:

   ```bash
   kubectl -n rook-ceph logs job/rook-ceph-osd-prepare-<node> | tail -3
   # skipping device "nvme0n1p10" because it contains a filesystem "ceph_bluestore"
   # skipping osd.3: "..." belonging to a different ceph cluster "..."
   ```
2. Rebuild the node, steps 3 to 7 of [Rebuilding or repartitioning a node](<https://homelab.wlkr.ch/operations/nodes/#rebuilding-or-repartitioning-a-node>): `make reinstall` only arms the PXE menu, and the [installer](<https://homelab.wlkr.ch/architecture/boot-process/index.md>) wipes the disk when the node network-boots.
3. If the node cannot be rebuilt now, wipe the partition by hand over SSH. `wipefs` removes the signature Rook skips on; the `dd` removes the BlueStore label behind it. Address it by label, as Ignition does, rather than by the device name in the log:

   ```bash
   sudo wipefs -a /dev/disk/by-partlabel/rook-osd
   sudo dd if=/dev/zero of=/dev/disk/by-partlabel/rook-osd bs=1M count=200
   ```
4. Restart the operator so it recreates the `osd-prepare` jobs:

   ```bash
   kubectl -n rook-ceph rollout restart deploy/rook-ceph-operator
   ```
5. Confirm an OSD per node and a binding PVC:

   ```bash
   make storage-check
   ```