# Descheduler

The scheduler places a pod once and never looks at it again. After [Kured](<https://homelab.wlkr.ch/platform/kured/index.md>) drains a node, everything that lived there restarts on the others and stays there when the node comes back, so a cluster that has rebooted through an update ends up with one idle node and the rest loaded. The [descheduler](<https://github.com/kubernetes-sigs/descheduler>) runs on a schedule, evicts a few pods from the loaded nodes, and lets the scheduler place them where there is room.

## At a glance

|  |  |
| --- | --- |
| Namespace | `descheduler` |
| Depends on | [metrics-server](<https://homelab.wlkr.ch/platform/metrics-server/index.md>) for the node usage `LowNodeUtilization` compares |
| If it is down | Nothing visible: pods stay where the last drain left them |
| Health check | `kubectl -n descheduler get jobs` → the last run `Complete` |
| Files | `payload/platform/descheduler/application.yaml` |

## Configuration

The values are in `payload/platform/descheduler/application.yaml`; the reasoning:

| Setting | Why |
| --- | --- |
| `kind: CronJob` with a `schedule` | A Deployment would poll continuously for a cluster that changes shape once per Flatcar update. A pass every few hours drifts a rebooted cluster back into balance the same day |
| `nodeFit: true` | A pod is only evicted when another node has room for it, so an eviction never turns into a Pending pod |
| `podProtections.extraEnabled: PodsWithPVC` | Ceph volumes would follow an evicted pod, but the pods that own them run alone: a moved [CloudNativePG](<https://homelab.wlkr.ch/platform/cloudnative-pg/index.md>) primary is a failover and a cold restart of its application, a moved Loki a detach and reattach. Their PDBs are off so that Kured can drain, so this is the only guard. `emptyDir` pods stay protected at the default because their data goes with them |
| System-critical pods at the default | `evictSystemCriticalPods` stays off, so nothing with `priorityClassName: system-cluster-critical` is ever moved. Those components hold their own placement rule instead — see [Replica placement](<https://homelab.wlkr.ch/platform/#replica-placement>) |
| `maxNoOfPodsToEvictPerNode` and `evictionLimits.node` | A few pods per node per pass: one Kured reboot already moved everything on a node once, a second mass move is what this component is meant to avoid |
| `LowNodeUtilization` with `metricsUtilization.source: KubernetesMetrics` | Usage comes from metrics-server, not from requests. Requests here are still being tuned and many are far below what the pod uses, so a requests-based view would call a loaded node idle |
| `thresholds` and `targetThresholds` | A node under the low mark is a candidate to receive, one over the high mark a candidate to lose pods; nodes in between are left alone |
| `RemoveDuplicates` and `RemovePodsViolatingTopologySpreadConstraint` for both `DoNotSchedule` and `ScheduleAnyway` | After a drain, replicas of one workload sit on the same node; both plugins put them back on separate ones. Soft constraints count too, because the scheduler honoured them only until the drain |
| `RemovePodsHavingTooManyRestarts` | A pod in a restart loop for hours is more likely stuck on its node than broken; a move is cheap and the restart count resets |
| Metrics off | A CronJob pod exists for seconds; a `ServiceMonitor` would scrape an empty Service |

## Usage

Nothing to operate. To watch a pass:

```bash
kubectl -n descheduler create job --from=cronjob/descheduler manual-$(date +%s)
kubectl -n descheduler logs -l app.kubernetes.io/name=descheduler --tail=100
```

Every eviction is an event on the pod, so `kubectl get events -A --field-selector reason=Descheduled` shows what moved and why.

## Health check

```bash
kubectl -n descheduler get cronjob,jobs
kubectl -n descheduler logs -l app.kubernetes.io/name=descheduler --tail=100 | grep -i "evict\|error"
kubectl top nodes    # what LowNodeUtilization sees
```

## Pitfalls

- **It only evicts, never schedules.** Where a pod lands is the scheduler's decision; the descheduler assumes the scheduler will do better the second time, which holds after a drain and not much else.
- **It honours PodDisruptionBudgets.** An eviction that would violate a PDB is refused. The single-instance [CloudNativePG](<https://homelab.wlkr.ch/platform/cloudnative-pg/index.md>) clusters carry none, so that Kured can drain; `PodsWithPVC` is what keeps the descheduler off them.
- **One pod per over-utilized node per pass** when usage comes from metrics-server. A badly skewed cluster takes several passes, which is the point of running it on a schedule rather than once.
- **A move is a restart.** Anything without a second replica is briefly down when it is evicted; `nodeFit` guarantees a place to go, not zero downtime.