Node Lifecycle¶

Everything you do to one machine. The four procedures share a shape: drain, act, wait for Ceph, move on.

Two of these destroy data and two do not

Rebooting and adding are safe and routine. Replacing destroys one node's OSD. Rebuilding destroys the disk it runs against, and doing it to every node in turn destroys every replica of everything.

Rebooting a node¶

Kured reboots nodes on its own once an update is staged, one at a time, and refuses while Ceph or etcd is unhealthy. The procedure below is for rebooting ahead of its next 30-minute check, or for a reason nothing set a sentinel for. Flatcar is installed to disk, so the node boots without the boot server and keeps /etc/kubernetes, /var/lib/etcd and /var/lib/rook. A change made with make config is not picked up by a reboot; that takes a rebuild.

Drain it.

kubectl drain <node> --ignore-daemonsets --delete-emptydir-data

Reboot it and wait for Ready.

ssh core@<node> sudo systemctl reboot
kubectl get node <node> -w

Uncordon it.
```
kubectl uncordon <node>
```
Wait for Ceph before touching the next node: draining a second node while the first is still backfilling can take a placement group below its minimum replica count.
```
kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph status   # HEALTH_OK
```
Unseal OpenBao if the node hosted a replica (below).

After any node reboot: unseal OpenBao¶

OpenBao comes back sealed and looks healthy while sealed, and nothing unseals it for you — see OpenBao for what that costs.

make bao-unseal                      # idempotent: unseals whatever is sealed
for pod in openbao-0 openbao-1 openbao-2; do
  kubectl -n openbao exec "$pod" -- bao status | grep Sealed   # false
done

make bao-unseal reads the key shares from output/credentials/openbao-init.json. If that file has been moved into a password manager, each pod wants 3 of the 5 shares by hand, per OpenBao.

Adding a node¶

Add the host to ansible/inventory.yaml, then regenerate and serve the boot files.
```
make config && make serve
```
On a control-plane node, generate a fresh join command; the provisioning token has a 24 hour TTL and has long expired.
```
ssh core@<control-plane-node>
sudo kubeadm token create --print-join-command
```
For a new control-plane node, also upload a current certificate key, which expires after two hours.
```
sudo kubeadm init phase upload-certs --upload-certs
```
Network-boot the new node and run the printed join command on it. The bootstrap-k8s.service unit only fires when /etc/kubernetes/kubelet.conf is absent, so it does not interfere with a node that has joined.

Replacing a failed node¶

Remove it from the cluster.

kubectl drain <node> --ignore-daemonsets --delete-emptydir-data --force
kubectl delete node <node>

Let Ceph re-replicate. With useAllNodes: true the OSD on that disk is gone for good; wait for ceph status to return to HEALTH_OK.

If it was a control-plane node, remove its etcd member; a dead member still counts toward quorum.

kubectl -n kube-system exec -it etcd-<healthy-node> -- etcdctl \
  --cacert /etc/kubernetes/pki/etcd/ca.crt \
  --cert /etc/kubernetes/pki/etcd/server.crt \
  --key /etc/kubernetes/pki/etcd/server.key \
  member list
# then: member remove <id>

Reprovision the replacement per Adding a node.

Danger

On a cluster provisioned before the Control Plane VIP, odin is not an interchangeable control-plane node: its address is baked in as the API endpoint and as Cilium's k8sServiceHost, so losing it breaks node joins and Cilium's API connection everywhere. Check which endpoint your kubeconfig uses first.

Rebuilding or repartitioning a node¶

The disk layout in ansible/templates/butane_node_config.yaml.j2 is applied by Ignition once, on the first boot after install, and the Ignition config is embedded in the OEM partition at install time, so no change under ansible/ reaches a running node. rook-osd is the last partition and deliberately raw: move its start by a sector, which any insert or resize above it does, and every OSD on that node is gone. Only the last partition can grow without a reinstall, and rook-osd cannot shrink in place because Ceph has already written across the space.

The backups are inside the thing being wiped

Velero and the etcd snapshot CronJob both write to the Ceph object store this destroys — see Backups & Recovery. Copy anything you intend to restore from off-cluster first.

Copy what matters off the cluster: Velero backups, the latest etcd snapshot, and anything in a PVC not reproducible from Git.
Edit whatever needs editing under ansible/.
Regenerate both Ignition configs and serve them.
```
make config && make serve
```
Arm the nodes you are rebuilding. Arming is the only way in (the menu shows no prompt), and the boot server disarms a node once it has the image, so the reboot at the end of the install does not start a second one — see Switching back to local boot.
```
make reinstall LIMIT=odin   # `make reinstall` alone asks, then arms every host
```
Network-boot the node. The installer wipes the disk, runs flatcar-install, and reboots into the installed system, which then runs kubeadm.
Leave the boot server up until the node is Ready (the sysext images are fetched from it on that first boot), then stop it.

One node at a time is safe when rebuilding: etcd keeps quorum and Ceph backfills, as in Replacing a failed node. Repartitioning destroys every OSD as it goes, so a rolling repartition across all six nodes destroys all replicas of everything; step 1 is not optional for that case.