Kured¶

Kured applies the updates the nodes have staged. It watches every node for a sentinel file and, when it finds one, takes a cluster-wide lock, cordons and drains the node, reboots it and uncordons it when it is back. Flatcar's own locksmithd is masked because it reboots without draining, on a cluster where every node is a storage node.

At a glance¶


Namespace	`kured`
Depends on	Monitoring for the alerts it gates on
If it is down	Nothing visible: staged OS and sysext updates simply never get applied
Health check	`kubectl -n kured logs -l app.kubernetes.io/name=kured --tail=50`
Files	`payload/platform/kured/application.yaml`

Configuration¶

An update stages on disk and signals it by touching /run/reboot-required, Kured's default sentinel. /run is tmpfs, so the marker clears itself on reboot.

Mechanism	Stages	Signals by
Flatcar OS (update-engine)	New image on the passive A/B partition	`flatcar-reboot-sentinel.timer`
Kubernetes sysext	New `.raw` under `/opt/extensions/kubernetes`	`systemd-sysupdate` drop-in

flowchart TD
    OS["update-engine<br/>OS image on the passive partition"] --> SEN
    KUBE["systemd-sysupdate<br/>kubernetes sysext"] --> SEN

    SEN["/run/reboot-required<br/>sentinel, on tmpfs"] --> TICK{"Kured checks<br/>every period"}

    TICK -->|"no sentinel"| SLEEP["Nothing to do until<br/>an update stages one"]
    TICK -->|"sentinel present"| ALERT{"Blocking alert firing?<br/>Ceph, etcd, node, API, Velero"}
    ALERT -->|"yes"| HELD["Reboot deferred while<br/>the cluster is unhealthy"]
    ALERT -->|"no"| LOCK{"Cluster lock free?"}
    LOCK -->|"another node<br/>is rebooting"| TAINT["Taint PreferNoSchedule,<br/>wait for the lock"]
    LOCK -->|"acquired"| DRAIN["Cordon and drain"]
    DRAIN -->|"drain fails"| GIVEUP["No reboot: uncordon,<br/>release the lock, retry"]
    DRAIN -->|"drained"| BOOT["systemctl reboot;<br/>new partition and<br/>sysexts take effect"]
    BOOT --> UP["Node comes back,<br/>sentinel gone with tmpfs"]
    UP --> DONE["Uncordon, drop the annotation,<br/>release the lock after a delay"]

    style SEN stroke-width:3px

The values are in payload/platform/kured/application.yaml; the reasoning:

Setting	Why
No reboot window	Kured acts as soon as it sees a sentinel; the guards below make that safe, not the clock
`concurrency` and `lockReleaseDelay`	One node down at a time. The rebooted node is uncordoned as soon as Kured's pod is back; the delay only holds the next node, long enough for the OSD to rejoin and the few minutes of backfill a reboot leaves, and as long as `CephOSDDown`'s `for`, so an OSD that did not come back is a firing alert before the next drain
`forceReboot: false`	A node that will not drain is a node worth looking at
`preferNoScheduleTaint`	A node pending reboot stops attracting pods about to be evicted again
`annotateNodes: true`	`kubectl get node -o yaml` shows why a node is cordoned
`VeleroBackupInProgress` on the alert list	Velero marks any backup it finds running at startup as Failed, and draining its node restarts it. The rule in `payload/platform/velero/application.yaml` compares attempts with outcomes per live Velero pod over a window longer than Velero's default limit on asynchronous operations; a backup that hangs drops out of the window, so it holds reboots for that long at most. A backup that starts in a Velero pod's first scrape interval has no zero sample to increase from and does not hold the gate. It routes to the `null` receiver because it fires every night
`alertFilterRegexp` with `alertFilterMatchOnly: true`	Kured asks Prometheus before taking a node down and refuses while any Ceph, etcd, node or API alert on the list fires. This is the automated "confirm Ceph has recovered" step from Rebooting a node: a second reboot during a backfill can take a placement group below its minimum replica count. Kured matches the alert name only, so Prometheus loads rules from platform namespaces only

The regex means the opposite of what it looks like

alertFilterRegexp normally lists alerts to ignore. With alertFilterMatchOnly: true these become the only alerts that block, because something is almost always firing in a homelab and blocking on any alert would mean never rebooting. An alert not on the list does not stop a reboot, so widen it if something turns out to matter — and read the flag twice before editing it. A name that no PrometheusRule defines blocks nothing and fails silently; the list once carried eight such names. Check every entry against the rules that exist, with the last command under Health check.

Usage¶

To stop Kured rebooting anything without uninstalling it, take its cluster lock by hand; it holds until the annotation is removed. Clearing the sentinel does not work, because flatcar-reboot-sentinel.timer in ansible/templates/butane_node_config.yaml.j2 re-touches it while an update is staged, and a DaemonSet cannot be scaled.

kubectl -n kured annotate ds kured weave.works/kured-node-lock='{"nodeID":"manual"}'
kubectl -n kured annotate ds kured weave.works/kured-node-lock-    # release it

To force a node to reboot on the next pass:

ssh core@<node> sudo touch /run/reboot-required

NodeRebootPending in payload/platform/monitoring/node-update-rules.yaml fires when the sentinel has been there for two days: Kured reboots one node at a time and refuses while Ceph or etcd is unhealthy, so a marker that old is a blocked node, not a queued one.

Health check¶

# Which nodes want a reboot: kured_reboot_required is the sentinel's presence, per node
kubectl -n monitoring port-forward svc/kube-prometheus-stack-prometheus 9090:9090 &
curl -s localhost:9090/api/v1/query --data-urlencode 'query=kured_reboot_required == 1' \
  | jq -r '.data.result[].metric.node'

# Which node is being rebooted right now
kubectl get nodes -o json | jq -r '.items[] | select(.metadata.annotations["weave.works/kured-reboot-in-progress"]) | .metadata.name'

# Whether a node has staged anything
ssh core@<node> 'ls -l /run/reboot-required; update_engine_client -status'

kubectl -n kured logs -l app.kubernetes.io/name=kured --tail=50

# The alert names the gate may use: every entry in alertFilterRegexp must be here
kubectl get prometheusrule -A -o json | jq -r '.items[].spec.groups[].rules[] | select(.alert) | .alert' | sort -u | grep -E 'Ceph|etcd|KubeNode|KubeAPI|VeleroBackupInProgress'