Backups & Recovery¶

Every mechanism below is described by what it can get back, and by what it cannot.

Current state¶

What is not covered¶

Every automated backup lands in the same cluster's Ceph. That protects against operator error (a deleted PVC, a bad prune, a corrupted database) and not against losing the cluster, because the backups go with it. Off-site replication is the missing piece: RGW supports bucket replication and Velero a second BackupStorageLocation, and neither is configured. Until then, disaster recovery is Git, the OpenBao unseal keys, and a copy of the latest OpenBao snapshot and of the two buckets kept off-cluster (Copying a bucket off the cluster); the nightly backups protect against mistakes.

Velero's PrometheusRule reports a backup that failed (VeleroBackupFailures, VeleroBackupPartialFailures); only VeleroBackupTooOld notices one that silently stopped, since a backup hung InProgress never fails (Velero). All of them only reach whoever Alertmanager is configured to tell — see Monitoring.

Velero¶

Velero backs up Kubernetes objects and volume data nightly. Chart values and why each is set are in Velero.

The velero CLI defaults to the velero namespace; this install is in backup, so pass -n backup every time.

kubectl -n backup get backups.velero.io
kubectl -n backup get backupstoragelocation     # should be Available

velero -n backup backup create manual-$(date +%s) --include-namespaces my-app

Backups in PartiallyFailed with no volume data mean Velero found no VolumeSnapshotClass labelled velero.io/csi-volumesnapshot-class: "true" and skipped the volumes silently — check payload/platform/backup/volumesnapshotclass.yaml is applied.

etcd¶

Velero restores through the API server, which is no use with no API server left (lost quorum, a corrupted data directory). For that a CronJob in payload/platform/backup/etcd-backup.yaml snapshots etcd nightly, an hour before Velero runs.

Two settings there are easy to undo by accident: the DNS policy for a host-network pod (the node's resolv.conf cannot resolve the RGW Service), and the AWS CLI config file selecting path-style addressing (no environment variable sets it).

kubectl -n backup get cronjob etcd-backup
kubectl -n backup logs job/<most-recent-job> -c upload

Defragmentation¶

The API server compacts etcd's history every five minutes, but compaction only frees pages inside the database file; the file itself never shrinks until a member is defragmented, and once less than half of it is in use etcdDatabaseHighFragmentationRatio fires. A second CronJob in payload/platform/backup/etcd-defrag.yaml runs etcdctl defrag --cluster nightly from the same kind of host-network pod as the backup, one member after another, and prints endpoint status before and after so the log shows the size reclaimed.

kubectl -n backup get cronjob etcd-defrag
kubectl -n backup create job --from=cronjob/etcd-defrag etcd-defrag-manual
kubectl -n backup logs job/etcd-defrag-manual

Taking one by hand¶

The snapshot lands in the static pod's /var/lib/etcd, which is the node's; kubectl cp cannot fetch it, because the etcd image is distroless and has no tar.

kubectl -n kube-system exec -it etcd-<node> -- etcdctl \
  --cacert /etc/kubernetes/pki/etcd/ca.crt \
  --cert /etc/kubernetes/pki/etcd/server.crt \
  --key /etc/kubernetes/pki/etcd/server.key \
  snapshot save /var/lib/etcd/snapshot.db

ssh <node> sudo cat /var/lib/etcd/snapshot.db > ./etcd-snapshot.db
ssh <node> sudo rm /var/lib/etcd/snapshot.db

Keep the copy off the cluster, like the nightly ones below.

Copying a bucket off the cluster¶

Rook writes each bucket's keys into a Secret named after its claim (velero-bucket, etcd-backup-bucket in backup) and the RGW answers on the infra Gateway; see Requesting a bucket.

1. Export the bucket's own keys and the path-style config RGW needs, as in Re-initialising the repositories.

   export AWS_ACCESS_KEY_ID=$(kubectl -n backup get secret <claim> -o jsonpath='{.data.AWS_ACCESS_KEY_ID}' | base64 -d)
   export AWS_SECRET_ACCESS_KEY=$(kubectl -n backup get secret <claim> -o jsonpath='{.data.AWS_SECRET_ACCESS_KEY}' | base64 -d)
   export AWS_CONFIG_FILE=$(mktemp)
   printf '[default]\ns3 =\n    addressing_style = path\n' > "$AWS_CONFIG_FILE"
   

2. Mirror the bucket to a disk that is not in the cluster; sync is incremental, so a repeat copies only what the last night added.

   aws --endpoint-url https://s3.infra.k8s.wlkr.ch --region eu-central-1 s3 sync s3://<bucket> ./<bucket>
   

The copy is as plain as the bucket: Velero's resource backups are JSON with Secrets in them, so store it with the unseal keys, not beside the Git clone.

OpenBao¶

kubectl -n openbao exec -it openbao-0 -- bao operator raft snapshot save /tmp/snapshot.bao
kubectl -n openbao cp openbao-0:/tmp/snapshot.bao ./openbao-snapshot.bao

The snapshot holds all KV data plus policies, roles and mounts. It does not hold the unseal keys and is useless without them — see OpenBao.

Ceph volumes¶

PVCs are covered by Velero. Ceph's own replication is not a backup: it spreads each block across OSDs, which survives a disk or node failing and nothing else. Almost every Application runs with prune: true (kube-vip and security are the deliberate exceptions), so removing a PersistentVolumeClaim from Git deletes the volume.

Restoring¶

The commands are the upstream-documented ones for the versions pinned in payload/platform/; the etcd sequence has not been exercised on this cluster.

Velero restore¶

1. Find the backup.

   velero -n backup backup get
   

2. Restore from it, scoped to what was lost; without --include-namespaces everything in the backup is restored, and existing objects are skipped, not overwritten.

   velero -n backup restore create --from-backup velero-daily-<timestamp> \
     --include-namespaces my-app
   

3. Watch it finish and read the warnings.

   velero -n backup restore describe <restore-name>
   velero -n backup restore logs <restore-name>
   

OpenBao snapshot restore¶

Needs an initialised, unsealed cluster and a root token from Generating a root token. Afterwards the snapshot's own operator login applies, not the new cluster's. -force is what lets a snapshot from a different cluster (a fresh bao operator init) load; afterwards the pods seal and want the snapshot's keys, not the new cluster's.

1. Copy the snapshot in.

   kubectl -n openbao cp ./openbao-snapshot.bao openbao-0:/tmp/snapshot.bao
   

2. Restore it.

   kubectl -n openbao exec -it openbao-0 -- \
     env BAO_TOKEN=<root-token> bao operator raft snapshot restore -force /tmp/snapshot.bao
   

3. Unseal every replica with the snapshot's key shares and confirm. The script reads output/credentials/openbao-init.json, which now holds the new cluster's keys, so point it at the old file with KEYFILE.

   KEYFILE=<old openbao-init.json> make bao-unseal
   kubectl -n openbao exec openbao-0 -- bao status   # Sealed: false
   

4. Put the old key and operator files in place of the new ones, which unseal nothing any more; every later make bao-* reads them.

   mv <old openbao-init.json> output/credentials/openbao-init.json
   mv <old openbao-operator.json> output/credentials/openbao-operator.json
   

etcd restore¶

Every etcd member restores from the same snapshot, then all three static pods come back together. Flatcar ships no etcdutl, so run it from the etcd image the CronJob uses. Upstream: Restoring an etcd cluster.

1. Copy the snapshot, from the off-cluster copy of the etcd-backup bucket (Copying a bucket off the cluster), to every control-plane node as /var/lib/etcd-restore/snapshot.db.

2. On every control-plane node, stop the API server and etcd by moving their static pod manifests out, and wait until neither container is listed.

   sudo mkdir -p /root/manifests-stopped
   sudo mv /etc/kubernetes/manifests/{etcd,kube-apiserver}.yaml /root/manifests-stopped/
   sudo crictl ps --name 'etcd|kube-apiserver'    # empty
   

3. On every control-plane node, pull the etcd image the CronJob uses (the tag is in payload/platform/backup/etcd-backup.yaml). Only the node that last ran the CronJob has it; ctr run does not pull.

   sudo ctr -n k8s.io image pull registry.k8s.io/etcd:<tag>
   

4. On every control-plane node, restore into a new data directory with the --name, --initial-cluster and --initial-advertise-peer-urls values from that node's /etc/kubernetes/manifests/etcd.yaml, and one --initial-cluster-token that is the same on every node and differs from the old cluster's, so a member that missed the restore cannot rejoin the new one.

   sudo ctr -n k8s.io run --rm \
     --mount type=bind,src=/var/lib/etcd-restore,dst=/restore,options=rbind:rw \
     registry.k8s.io/etcd:<tag> etcd-restore \
     etcdutl snapshot restore /restore/snapshot.db --data-dir /restore/etcd \
       --name <node> --initial-cluster <name=https://ip:2380,...> \
       --initial-cluster-token <new-token> \
       --initial-advertise-peer-urls https://<node-ip>:2380
   

5. Swap the data directory in, keeping the old one and its ownership.

   sudo mv /var/lib/etcd /var/lib/etcd.bak
   sudo mv /var/lib/etcd-restore/etcd /var/lib/etcd
   sudo chown -R --reference=/var/lib/etcd.bak /var/lib/etcd
   

6. Put the manifests back on every node, then confirm.

   sudo mv /root/manifests-stopped/*.yaml /etc/kubernetes/manifests/
   kubectl get nodes
   kubectl -n argocd get applications     # ArgoCD reconverges from Git
   

Rebuild from scratch¶

1. Have the Git repository, the OpenBao unseal keys and operator login, an OpenBao snapshot and the off-cluster copy of the velero bucket; an etcd snapshot is optional and skips re-issuing certificates.
2. Move the old openbao-init.json and openbao-operator.json out of output/credentials/: make bao-init refuses while either exists, because keys from another cluster unseal nothing.
3. Provision the cluster per the Quickstart from step 1.
4. When every ExternalSecret is Degraded, make bao-init, then either restore the OpenBao snapshot as above, which puts the old files back, or re-enter the typed secrets with make bao-secrets and let the PushSecrets generate the rest anew — which loses whatever was encrypted with the old keys, Velero's volume backups among them.
5. Once the platform is Synced and Healthy, restore workloads with a Velero restore from the copied-off velero bucket.