# Backups & Recovery Every mechanism below is described by what it can get back, and by what it cannot. ## Current state | Data | Backed up | How | | --- | --- | --- | | Kubernetes objects | Nightly | [Velero](<#velero>) to the Ceph object store | | Ceph RBD volumes (PVCs) | Nightly | Velero CSI snapshot, moved into the object store | | PostgreSQL databases | Continuous WAL, nightly base backup | [CloudNativePG's Barman Cloud plugin](), one bucket per database; restores to a point in time | | Grafana dashboards | Nightly | Its PVC is covered by the above | | etcd (raw) | Nightly | [CronJob](<#etcd>) to the object store | | OpenBao secrets | Manual | [Raft snapshot](<#openbao>) | | OpenBao unseal keys | Manual, off-cluster | Printed once at `bao operator init` | | Prometheus metrics | **No** | Its PVC carries `velero.io/exclude-from-backup`; metrics live for the `retention` in `payload/platform/monitoring/application.yaml`, then are gone | | Trivy reports and vulnerability database | **No** | `trivy-system` is excluded; the operator rescans and the server downloads the database again | | Everything in `payload/` | Yes | It is in Git | ## What is not covered **Every automated backup lands in the same cluster's Ceph.** That protects against operator error (a deleted PVC, a bad `prune`, a corrupted database) and not against losing the cluster, because the backups go with it. Off-site replication is the missing piece: RGW supports bucket replication and Velero a second `BackupStorageLocation`, and neither is configured. Until then, disaster recovery is **Git, the OpenBao unseal keys, and a copy of the latest OpenBao snapshot and of the two buckets kept off-cluster** ([Copying a bucket off the cluster](<#copying-a-bucket-off-the-cluster>)); the nightly backups protect against mistakes. Velero's `PrometheusRule` reports a backup that failed (`VeleroBackupFailures`, `VeleroBackupPartialFailures`); only `VeleroBackupTooOld` notices one that silently stopped, since a backup hung `InProgress` never fails ([Velero]()). All of them only reach whoever Alertmanager is configured to tell — see [Monitoring](). ## Velero [Velero]() backs up Kubernetes objects and volume data nightly. Chart values and why each is set are in [Velero](). | Property | Value | | --- | --- | | Schedule | The `schedules` block in `payload/platform/velero/application.yaml`: nightly, an hour after etcd so the two never contend for the object store, with a TTL of two weeks so a bad backup is noticed before the last good one expires | | Scope | All namespaces except `kube-system` (rebuilt from Git, and large) and `trivy-system` (rescanned), minus `events` (they expire anyway) and any PVC labelled `velero.io/exclude-from-backup`, which is only Prometheus's: the chart's `volumeClaimTemplate.metadata.labels` carries it | | Destination | S3 bucket `velero` in the [Ceph object store]() | | Volume data | CSI snapshot, streamed into the bucket by the data mover (Kopia), snapshot deleted; the durable copy is the one in the bucket | | Encryption | Volume data with the repository password generated into OpenBao at `velero/repository`, which a restore needs; resource backups, Secrets included, are plain JSON — see [Repository password]() | The `velero` CLI defaults to the `velero` namespace; this install is in `backup`, so pass `-n backup` every time. ```bash kubectl -n backup get backups.velero.io kubectl -n backup get backupstoragelocation # should be Available velero -n backup backup create manual-$(date +%s) --include-namespaces my-app ``` Backups in `PartiallyFailed` with no volume data mean Velero found no `VolumeSnapshotClass` labelled `velero.io/csi-volumesnapshot-class: "true"` and skipped the volumes silently — check `payload/platform/backup/volumesnapshotclass.yaml` is applied. ## etcd Velero restores *through* the API server, which is no use with no API server left (lost quorum, a corrupted data directory). For that a CronJob in `payload/platform/backup/etcd-backup.yaml` snapshots etcd nightly, an hour before Velero runs. | Property | Value | | --- | --- | | Schedule | `payload/platform/backup/etcd-backup.yaml`: nightly, before Velero | | Where it runs | Any control-plane node, on the host network: etcd listens on `127.0.0.1` and its client certs are on the node. It mounts the CA certificate and the healthcheck client pair, never the directory, which also holds the CA and server keys | | Verification | `etcdutl snapshot status` before upload, so a truncated snapshot fails the job instead of replacing a good backup (`etcdctl snapshot status` was removed in etcd 3.6) | | Destination | S3 bucket `etcd-backup`, separate from Velero's because it is restored by different means; the job keeps the newest snapshots and deletes the rest, the count is in the manifest | Two settings there are easy to undo by accident: the DNS policy for a host-network pod (the node's `resolv.conf` cannot resolve the RGW Service), and the AWS CLI config file selecting path-style addressing (no environment variable sets it). ```bash kubectl -n backup get cronjob etcd-backup kubectl -n backup logs job/ -c upload ``` ### Defragmentation The API server compacts etcd's history every five minutes, but compaction only frees pages inside the database file; the file itself never shrinks until a member is defragmented, and once less than half of it is in use `etcdDatabaseHighFragmentationRatio` fires. A second CronJob in `payload/platform/backup/etcd-defrag.yaml` runs `etcdctl defrag --cluster` nightly from the same kind of host-network pod as the backup, one member after another, and prints `endpoint status` before and after so the log shows the size reclaimed. | Property | Why | | --- | --- | | Nightly | This cluster's churn crosses the alert's ratio about a day and a half after a defrag, so the alert firing means a run failed or did not help, not that the next one is due | | One member at a time | A member blocks for seconds while it rewrites its file; the other two keep quorum | | Fails when a member is unreachable | `endpoint status --cluster` exits non-zero, so a cluster already short of a member is not defragmented | ```bash kubectl -n backup get cronjob etcd-defrag kubectl -n backup create job --from=cronjob/etcd-defrag etcd-defrag-manual kubectl -n backup logs job/etcd-defrag-manual ``` ### Taking one by hand The snapshot lands in the static pod's `/var/lib/etcd`, which is the node's; `kubectl cp` cannot fetch it, because the etcd image is distroless and has no `tar`. ```bash kubectl -n kube-system exec -it etcd- -- etcdctl \ --cacert /etc/kubernetes/pki/etcd/ca.crt \ --cert /etc/kubernetes/pki/etcd/server.crt \ --key /etc/kubernetes/pki/etcd/server.key \ snapshot save /var/lib/etcd/snapshot.db ssh sudo cat /var/lib/etcd/snapshot.db > ./etcd-snapshot.db ssh sudo rm /var/lib/etcd/snapshot.db ``` Keep the copy off the cluster, like the nightly ones below. ### Copying a bucket off the cluster Rook writes each bucket's keys into a `Secret` named after its claim (`velero-bucket`, `etcd-backup-bucket` in `backup`) and the RGW answers on the infra Gateway; see [Requesting a bucket](). 1. Export the bucket's own keys and the path-style config RGW needs, as in [Re-initialising the repositories](). ```bash export AWS_ACCESS_KEY_ID=$(kubectl -n backup get secret -o jsonpath='{.data.AWS_ACCESS_KEY_ID}' | base64 -d) export AWS_SECRET_ACCESS_KEY=$(kubectl -n backup get secret -o jsonpath='{.data.AWS_SECRET_ACCESS_KEY}' | base64 -d) export AWS_CONFIG_FILE=$(mktemp) printf '[default]\ns3 =\n addressing_style = path\n' > "$AWS_CONFIG_FILE" ``` 2. Mirror the bucket to a disk that is not in the cluster; `sync` is incremental, so a repeat copies only what the last night added. ```bash aws --endpoint-url https://s3.infra.k8s.wlkr.ch --region eu-central-1 s3 sync s3:// ./ ``` The copy is as plain as the bucket: Velero's resource backups are JSON with Secrets in them, so store it with the unseal keys, not beside the Git clone. ## OpenBao ```bash kubectl -n openbao exec -it openbao-0 -- bao operator raft snapshot save /tmp/snapshot.bao kubectl -n openbao cp openbao-0:/tmp/snapshot.bao ./openbao-snapshot.bao ``` The snapshot holds all KV data plus policies, roles and mounts. It does **not** hold the unseal keys and is useless without them — see [OpenBao](). ## Ceph volumes PVCs are covered by [Velero](<#velero>). Ceph's own replication is not a backup: it spreads each block across OSDs, which survives a disk or node failing and nothing else. Almost every Application runs with `prune: true` (`kube-vip` and `security` are the deliberate exceptions), so removing a `PersistentVolumeClaim` from Git deletes the volume. ## Restoring The commands are the upstream-documented ones for the versions pinned in `payload/platform/`; the etcd sequence has not been exercised on this cluster. ### Velero restore 1. Find the backup. ```bash velero -n backup backup get ``` 2. Restore from it, scoped to what was lost; without `--include-namespaces` everything in the backup is restored, and existing objects are skipped, not overwritten. ```bash velero -n backup restore create --from-backup velero-daily- \ --include-namespaces my-app ``` 3. Watch it finish and read the warnings. ```bash velero -n backup restore describe velero -n backup restore logs ``` ### OpenBao snapshot restore Needs an initialised, unsealed cluster and a root token from [Generating a root token](). Afterwards the snapshot's own operator login applies, not the new cluster's. `-force` is what lets a snapshot from a *different* cluster (a fresh `bao operator init`) load; afterwards the pods seal and want the snapshot's keys, not the new cluster's. 1. Copy the snapshot in. ```bash kubectl -n openbao cp ./openbao-snapshot.bao openbao-0:/tmp/snapshot.bao ``` 2. Restore it. ```bash kubectl -n openbao exec -it openbao-0 -- \ env BAO_TOKEN= bao operator raft snapshot restore -force /tmp/snapshot.bao ``` 3. Unseal every replica with the snapshot's key shares and confirm. The script reads `output/credentials/openbao-init.json`, which now holds the new cluster's keys, so point it at the old file with `KEYFILE`. ```bash KEYFILE= make bao-unseal kubectl -n openbao exec openbao-0 -- bao status # Sealed: false ``` 4. Put the old key and operator files in place of the new ones, which unseal nothing any more; every later `make bao-*` reads them. ```bash mv output/credentials/openbao-init.json mv output/credentials/openbao-operator.json ``` ### etcd restore Every etcd member restores from the same snapshot, then all three static pods come back together. Flatcar ships no `etcdutl`, so run it from the etcd image the CronJob uses. Upstream: [Restoring an etcd cluster](). 1. Copy the snapshot, from the off-cluster copy of the `etcd-backup` bucket ([Copying a bucket off the cluster](<#copying-a-bucket-off-the-cluster>)), to every control-plane node as `/var/lib/etcd-restore/snapshot.db`. 2. On every control-plane node, stop the API server and etcd by moving their static pod manifests out, and wait until neither container is listed. ```bash sudo mkdir -p /root/manifests-stopped sudo mv /etc/kubernetes/manifests/{etcd,kube-apiserver}.yaml /root/manifests-stopped/ sudo crictl ps --name 'etcd|kube-apiserver' # empty ``` 3. On every control-plane node, pull the etcd image the CronJob uses (the tag is in `payload/platform/backup/etcd-backup.yaml`). Only the node that last ran the CronJob has it; `ctr run` does not pull. ```bash sudo ctr -n k8s.io image pull registry.k8s.io/etcd: ``` 4. On every control-plane node, restore into a new data directory with the `--name`, `--initial-cluster` and `--initial-advertise-peer-urls` values from that node's `/etc/kubernetes/manifests/etcd.yaml`, and one `--initial-cluster-token` that is the same on every node and differs from the old cluster's, so a member that missed the restore cannot rejoin the new one. ```bash sudo ctr -n k8s.io run --rm \ --mount type=bind,src=/var/lib/etcd-restore,dst=/restore,options=rbind:rw \ registry.k8s.io/etcd: etcd-restore \ etcdutl snapshot restore /restore/snapshot.db --data-dir /restore/etcd \ --name --initial-cluster \ --initial-cluster-token \ --initial-advertise-peer-urls https://:2380 ``` 5. Swap the data directory in, keeping the old one and its ownership. ```bash sudo mv /var/lib/etcd /var/lib/etcd.bak sudo mv /var/lib/etcd-restore/etcd /var/lib/etcd sudo chown -R --reference=/var/lib/etcd.bak /var/lib/etcd ``` 6. Put the manifests back on every node, then confirm. ```bash sudo mv /root/manifests-stopped/*.yaml /etc/kubernetes/manifests/ kubectl get nodes kubectl -n argocd get applications # ArgoCD reconverges from Git ``` ### Rebuild from scratch 1. Have the Git repository, the OpenBao unseal keys and operator login, an OpenBao snapshot and the off-cluster copy of the `velero` bucket; an etcd snapshot is optional and skips re-issuing certificates. 2. Move the old `openbao-init.json` and `openbao-operator.json` out of `output/credentials/`: `make bao-init` refuses while either exists, because keys from another cluster unseal nothing. 3. Provision the cluster per the [Quickstart]() from step 1. 4. When every `ExternalSecret` is Degraded, `make bao-init`, then either restore the OpenBao snapshot as above, which puts the old files back, or re-enter the typed secrets with `make bao-secrets` and let the `PushSecret`s generate the rest anew — which loses whatever was encrypted with the old keys, Velero's volume backups among them. 5. Once the platform is Synced and Healthy, restore workloads with a Velero restore from the copied-off `velero` bucket.