# Node Lifecycle

Everything you do to one machine. The four procedures share a shape: drain, act, wait for Ceph, move on.

> [!CAUTION]
>
> **Two of these destroy data and two do not**
>
> [Rebooting](<#rebooting-a-node>) and [adding](<#adding-a-node>) are safe and routine. [Replacing](<#replacing-a-failed-node>) destroys one node's OSD. [Rebuilding](<#rebuilding-or-repartitioning-a-node>) destroys the disk it runs against, and doing it to every node in turn destroys every replica of everything.

## Rebooting a node

[Kured](<https://homelab.wlkr.ch/platform/kured/index.md>) reboots nodes on its own once an update is staged, one at a time, and refuses while Ceph or etcd is unhealthy. The procedure below is for rebooting ahead of its next check (`period` in `payload/platform/kured/application.yaml`), or for a reason nothing set a sentinel for. Flatcar is installed to disk, so the node boots without the boot server and keeps `/etc/kubernetes`, `/var/lib/etcd` and `/var/lib/rook`. A change made with `make config` is *not* picked up by a reboot; that takes a [rebuild](<#rebuilding-or-repartitioning-a-node>).

1. Drain it.

   ```bash
   kubectl drain <node> --ignore-daemonsets --delete-emptydir-data
   ```

   A drain that loops on `Cannot evict pod as it would violate the pod's disruption budget` for a `<name>-db-<n>` pod is a CloudNativePG cluster with its budget still on — see [the contract](<https://homelab.wlkr.ch/platform/cloudnative-pg/#the-contract>).
2. Reboot it and wait for `Ready`.

   ```bash
   ssh core@<node> sudo systemctl reboot
   kubectl get node <node> -w
   ```
3. Uncordon it.

   ```bash
   kubectl uncordon <node>
   ```

   The pods that moved stay where they are; the [descheduler](<https://homelab.wlkr.ch/platform/descheduler/index.md>) spreads them back out on its next pass.
4. **Wait for Ceph** before touching the next node: draining a second node while the first is still backfilling can take a placement group below its minimum replica count.

   ```bash
   kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph status   # HEALTH_OK
   ```
5. **Unseal OpenBao** if the node hosted a replica (below).

## After any node reboot: unseal OpenBao

OpenBao comes back sealed and looks healthy while sealed, and nothing unseals it for you — see [OpenBao](<https://homelab.wlkr.ch/platform/openbao/index.md>) for what that costs.

```bash
make bao-unseal                      # idempotent: unseals whatever is sealed
for pod in openbao-0 openbao-1 openbao-2; do
  kubectl -n openbao exec "$pod" -- bao status | grep Sealed   # false
done
```

`make bao-unseal` reads the key shares from `output/credentials/openbao-init.json`. If that file has been moved into a password manager, each pod wants the threshold of shares by hand, per [OpenBao](<https://homelab.wlkr.ch/platform/openbao/#unsealing-after-a-restart>).

## Adding a node

1. Compare `kubernetes_version` in `ansible/inventory.yaml` with `kubectl version`: a node installs the inventory's version, and `kubeadm    join` refuses a kubelet newer than the API server, so a merged minor has to be [walked onto the cluster](<https://homelab.wlkr.ch/operations/upgrades/#upgrading-a-minor-version-deliberately>) first. Then add the host to the inventory and regenerate and serve the boot files.

   ```bash
   make config && make serve
   ```

   In the same commit, add the node name to `providerRegex` in `payload/platform/kubelet-csr-approver/application.yaml`, and let that PR merge and sync before the node boots: a node the regex does not name has its kubelet certificate denied and [silently drops out of `kubectl top`](<https://homelab.wlkr.ch/platform/metrics-server/#pitfalls>).

   Against a running cluster, `make config` also re-creates the bootstrap token baked into the Ignition configs for 24 hours and uploads the control-plane certificates for two, on [whichever control plane answers](<https://homelab.wlkr.ch/architecture/ansible/#playbooks>), so the node joins on its own.
2. Arm the node: the generated menu defaults to the local disk, so an unarmed node never installs, per [Switching back to local boot](<https://homelab.wlkr.ch/architecture/boot-process/#switching-back-to-local-boot>).

   ```bash
   make reinstall LIMIT=<node>
   ```
3. Network-boot it within those windows: two hours for a **control-plane** node, 24 for a worker. `bootstrap-k8s.service` runs the join once and only while `/etc/kubernetes/kubelet.conf` is absent, so it does not interfere with a node that has joined.
4. If the window has passed, run `make config` again, then retry the join on the node.

   ```bash
   ssh <node> sudo systemctl restart bootstrap-k8s
   ```

## Replacing a failed node

1. Remove it from the cluster.

   ```bash
   kubectl drain <node> --ignore-daemonsets --delete-emptydir-data --force
   kubectl delete node <node>
   ```
2. Let Ceph re-replicate. With `useAllNodes: true` the OSD on that disk is gone for good; wait for `ceph status` to return to `HEALTH_OK`.
3. If it was a control-plane node, remove its etcd member; a dead member still counts toward quorum.

   ```bash
   kubectl -n kube-system exec -it etcd-<healthy-node> -- etcdctl \
     --cacert /etc/kubernetes/pki/etcd/ca.crt \
     --cert /etc/kubernetes/pki/etcd/server.crt \
     --key /etc/kubernetes/pki/etcd/server.key \
     member list
   # then: member remove <id>
   ```
4. Reprovision the replacement per [Adding a node](<#adding-a-node>): the inventory and regex change, then arm, then boot.

> [!CAUTION]
>
> **Danger**
>
> On a cluster provisioned before the [Control Plane VIP](<https://homelab.wlkr.ch/operations/control-plane-vip/index.md>), `odin` is not an interchangeable control-plane node: its address is baked in as the API endpoint and as Cilium's `k8sServiceHost`, so losing it breaks node joins and Cilium's API connection everywhere. Check which endpoint your kubeconfig uses first.

## Rebuilding or repartitioning a node

The disk layout in `ansible/templates/butane_node_config.yaml.j2` is applied by Ignition once, on the first boot after install, and the Ignition config is embedded in the OEM partition at install time, so no change under `ansible/` reaches a running node. `rook-osd` is the last partition and deliberately raw: move its start by a sector, which any insert or resize above it does, and every OSD on that node is gone. Only the last partition can grow without a reinstall, and `rook-osd` cannot shrink in place because Ceph has already written across the space.

> [!CAUTION]
>
> **The backups are inside the thing being wiped**
>
> Velero and the etcd snapshot CronJob both write to the Ceph object store this destroys — see [Backups &amp; Recovery](<https://homelab.wlkr.ch/operations/backups/#what-is-not-covered>). Copy anything you intend to restore from **off-cluster** first.

1. Copy what matters off the cluster: the `velero` and `etcd-backup` buckets per [Copying a bucket off the cluster](<https://homelab.wlkr.ch/operations/backups/#copying-a-bucket-off-the-cluster>), and anything in a PVC not reproducible from Git.
2. Edit whatever needs editing under `ansible/`.
3. Regenerate both Ignition configs and serve them.

   ```bash
   make config && make serve
   ```
4. Drain the node, as in [Rebooting a node](<#rebooting-a-node>). For a **control-plane** node, remove it from the cluster too: the `kubectl delete node` and the etcd `member remove` are steps 1 and 3 of [Replacing a failed node](<#replacing-a-failed-node>). The wiped node comes back with an empty etcd under its old peer URL, and kubeadm treats a known peer as already joined; two of three members keep quorum meanwhile.
5. Arm the nodes you are rebuilding. Arming is the only way in (the menu shows no prompt), a later `make config` keeps a node armed, and the boot server disarms it once it has the image, so the reboot at the end of the install does not start a second one — see [Switching back to local boot](<https://homelab.wlkr.ch/architecture/boot-process/#switching-back-to-local-boot>).

   ```bash
   make reinstall LIMIT=odin   # `make reinstall` alone asks, then arms every host
   ```

   Every control plane joins the running cluster; only [`bootstrap_cluster`](<https://homelab.wlkr.ch/architecture/ansible/#inventory>) renders `kubeadm init`, and arming refuses while it is set and the VIP answers.
6. Network-boot the node. The installer wipes the disk, runs `flatcar-install`, and reboots into the installed system, which then runs `kubeadm`. The node comes back with a new SSH host key; `make reinstall` already forgot the old one, or `StrictHostKeyChecking=accept-new` would refuse the first `make kubeconfig`.
7. Leave the boot server up until the node is `Ready` (the sysext images are fetched from it on that first boot), then stop it.
8. Wait for Ceph before the next node, as in [Rebooting a node](<#rebooting-a-node>): the OSD is new and backfills from the other two copies, and [`make storage-check`](<https://homelab.wlkr.ch/platform/rook-ceph/#is-storage-ready>) proves it took the disk.
9. [Unseal OpenBao](<#after-any-node-reboot-unseal-openbao>) if a replica moved.

One node at a time is safe when rebuilding: etcd keeps quorum once the old member is removed and Ceph backfills, as in [Replacing a failed node](<#replacing-a-failed-node>). Repartitioning destroys every OSD as it goes, so a rolling repartition across every node destroys all replicas of everything; step 1 is not optional for that case.