Node Lifecycle¶

Everything you do to one machine. The four procedures share a shape: drain, act, wait for Ceph, move on.

Two of these destroy data and two do not

Rebooting and adding are safe and routine. Replacing destroys one node's OSD. Rebuilding destroys the disk it runs against, and doing it to every node in turn destroys every replica of everything.

Rebooting a node¶

Kured reboots nodes on its own once an update is staged, one at a time, and refuses while Ceph or etcd is unhealthy. The procedure below is for rebooting ahead of its next check (period in payload/platform/kured/application.yaml), or for a reason nothing set a sentinel for. Flatcar is installed to disk, so the node boots without the boot server and keeps /etc/kubernetes, /var/lib/etcd and /var/lib/rook. A change made with make config is not picked up by a reboot; that takes a rebuild.

Drain it.
```
kubectl drain <node> --ignore-daemonsets --delete-emptydir-data
```
A drain that loops on Cannot evict pod as it would violate the pod's disruption budget for a <name>-db-<n> pod is a CloudNativePG cluster with its budget still on — see the contract.

Reboot it and wait for Ready.

ssh core@<node> sudo systemctl reboot
kubectl get node <node> -w

Uncordon it.
```
kubectl uncordon <node>
```
The pods that moved stay where they are; the descheduler spreads them back out on its next pass.
Wait for Ceph before touching the next node: draining a second node while the first is still backfilling can take a placement group below its minimum replica count.
```
kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph status   # HEALTH_OK
```
Unseal OpenBao if the node hosted a replica (below).

After any node reboot: unseal OpenBao¶

OpenBao comes back sealed and looks healthy while sealed, and nothing unseals it for you — see OpenBao for what that costs.

make bao-unseal                      # idempotent: unseals whatever is sealed
for pod in openbao-0 openbao-1 openbao-2; do
  kubectl -n openbao exec "$pod" -- bao status | grep Sealed   # false
done

make bao-unseal reads the key shares from output/credentials/openbao-init.json. If that file has been moved into a password manager, each pod wants the threshold of shares by hand, per OpenBao.

Adding a node¶

Compare kubernetes_version in ansible/inventory.yaml with kubectl version: a node installs the inventory's version, and kubeadm join refuses a kubelet newer than the API server, so a merged minor has to be walked onto the cluster first. Then add the host to the inventory and regenerate and serve the boot files.
```
make config && make serve
```
In the same commit, add the node name to providerRegex in payload/platform/kubelet-csr-approver/application.yaml, and let that PR merge and sync before the node boots: a node the regex does not name has its kubelet certificate denied and silently drops out of kubectl top.

Against a running cluster, make config also re-creates the bootstrap token baked into the Ignition configs for 24 hours and uploads the control-plane certificates for two, on whichever control plane answers, so the node joins on its own.
Arm the node: the generated menu defaults to the local disk, so an unarmed node never installs, per Switching back to local boot.
```
make reinstall LIMIT=<node>
```
Network-boot it within those windows: two hours for a control-plane node, 24 for a worker. bootstrap-k8s.service runs the join once and only while /etc/kubernetes/kubelet.conf is absent, so it does not interfere with a node that has joined.
If the window has passed, run make config again, then retry the join on the node.
```
ssh <node> sudo systemctl restart bootstrap-k8s
```

Replacing a failed node¶

Remove it from the cluster.

kubectl drain <node> --ignore-daemonsets --delete-emptydir-data --force
kubectl delete node <node>

Let Ceph re-replicate. With useAllNodes: true the OSD on that disk is gone for good; wait for ceph status to return to HEALTH_OK.

If it was a control-plane node, remove its etcd member; a dead member still counts toward quorum.

kubectl -n kube-system exec -it etcd-<healthy-node> -- etcdctl \
  --cacert /etc/kubernetes/pki/etcd/ca.crt \
  --cert /etc/kubernetes/pki/etcd/server.crt \
  --key /etc/kubernetes/pki/etcd/server.key \
  member list
# then: member remove <id>

Reprovision the replacement per Adding a node: the inventory and regex change, then arm, then boot.

Danger

On a cluster provisioned before the Control Plane VIP, odin is not an interchangeable control-plane node: its address is baked in as the API endpoint and as Cilium's k8sServiceHost, so losing it breaks node joins and Cilium's API connection everywhere. Check which endpoint your kubeconfig uses first.

Rebuilding or repartitioning a node¶

The disk layout in ansible/templates/butane_node_config.yaml.j2 is applied by Ignition once, on the first boot after install, and the Ignition config is embedded in the OEM partition at install time, so no change under ansible/ reaches a running node. rook-osd is the last partition and deliberately raw: move its start by a sector, which any insert or resize above it does, and every OSD on that node is gone. Only the last partition can grow without a reinstall, and rook-osd cannot shrink in place because Ceph has already written across the space.

The backups are inside the thing being wiped

Velero and the etcd snapshot CronJob both write to the Ceph object store this destroys — see Backups & Recovery. Copy anything you intend to restore from off-cluster first.

Copy what matters off the cluster: the velero and etcd-backup buckets per Copying a bucket off the cluster, and anything in a PVC not reproducible from Git.
Edit whatever needs editing under ansible/.
Regenerate both Ignition configs and serve them.
```
make config && make serve
```
Drain the node, as in Rebooting a node. For a control-plane node, remove it from the cluster too: the kubectl delete node and the etcd member remove are steps 1 and 3 of Replacing a failed node. The wiped node comes back with an empty etcd under its old peer URL, and kubeadm treats a known peer as already joined; two of three members keep quorum meanwhile.
Arm the nodes you are rebuilding. Arming is the only way in (the menu shows no prompt), a later make config keeps a node armed, and the boot server disarms it once it has the image, so the reboot at the end of the install does not start a second one — see Switching back to local boot.
```
make reinstall LIMIT=odin   # `make reinstall` alone asks, then arms every host
```
Every control plane joins the running cluster; only bootstrap_cluster renders kubeadm init, and arming refuses while it is set and the VIP answers.
Network-boot the node. The installer wipes the disk, runs flatcar-install, and reboots into the installed system, which then runs kubeadm. The node comes back with a new SSH host key; make reinstall already forgot the old one, or StrictHostKeyChecking=accept-new would refuse the first make kubeconfig.
Leave the boot server up until the node is Ready (the sysext images are fetched from it on that first boot), then stop it.
Wait for Ceph before the next node, as in Rebooting a node: the OSD is new and backfills from the other two copies, and make storage-check proves it took the disk.
Unseal OpenBao if a replica moved.

One node at a time is safe when rebuilding: etcd keeps quorum once the old member is removed and Ceph backfills, as in Replacing a failed node. Repartitioning destroys every OSD as it goes, so a rolling repartition across every node destroys all replicas of everything; step 1 is not optional for that case.