metrics-server¶
metrics-server serves the
metrics.k8s.io API that kubectl top, every HorizontalPodAutoscaler and
the descheduler read; the scheduler does not, it places on
requests. Without it kubectl top errors and an HPA sits
at <unknown>/<target> with no logs, events or clue. Prometheus is not a
substitute: it answers PromQL from its own store, while the resource metrics
API is an aggregated Kubernetes API that core controllers read directly.
At a glance¶
| Namespace | kube-system; kubelet-csr-approver in kubelet-csr-approver |
| Depends on | kubelet-csr-approver, or the kubelet certificates it verifies are never issued; cert-manager for its own serving certificate |
| If it is down | kubectl top, every HPA and the descheduler's node usage. Nothing else notices |
| Health check | kubectl top nodes → a row per node |
| Files | payload/platform/metrics-server/, payload/platform/kubelet-csr-approver/ |
Configuration¶
Verifying the kubelet instead of trusting it¶
A kubelet serves a self-signed certificate by default, so the usual shortcut
is --kubelet-insecure-tls, which makes the metrics path spoofable by anything
that can occupy a kubelet's address. Kubernetes deliberately does not
auto-approve kubernetes.io/kubelet-serving CSRs, since a compromised node
could request a certificate for any name, so doing it properly takes three
pieces:
| Piece | Where | What it does |
|---|---|---|
serverTLSBootstrap: true |
ansible/templates/kubeadm.yaml.j2 |
The kubelet requests a serving certificate from the cluster CA |
| kubelet-csr-approver | payload/platform/kubelet-csr-approver/application.yaml |
Approves those CSRs, constrained by providerRegex (only the inventory's node names; anything else is denied), providerIpPrefixes (the nodes' subnet), a one-day maxExpirationSeconds, and bypassDnsResolution because node names come from the Ansible inventory, not a resolver |
--kubelet-certificate-authority |
payload/platform/metrics-server/application.yaml |
metrics-server verifies against the cluster CA |
Setting in metrics-server/application.yaml |
Why |
|---|---|
Flags via defaultArgs, not args |
The default list already sets --kubelet-preferred-address-types, and a flag passed twice leaves the winner to parsing order. It is InternalIP because node hostnames do not resolve here |
| Two replicas and a PodDisruptionBudget | A node reboot does not take the resource metrics API down |
tls.type: cert-manager with apiService.insecureSkipTLSVerify: false |
The other half of the path: the chart's self-signed Issuer signs the serving certificate and the cainjector copies its CA into the APIService, so kube-apiserver verifies metrics-server instead of trusting whatever answers at the Service |
Health check¶
kubectl get csr | grep kubelet-serving # should be Approved,Issued
kubectl get apiservice v1beta1.metrics.k8s.io # Available True, caBundle filled
kubectl top nodes
Pending CSRs mean kubelet-csr-approver is not running or a node name does not
match the regex. Denied means the regex or the IP prefix is wrong — check the
controller's logs before widening either.
Pitfalls¶
Adding a node means editing the regex
A node missing from providerRegex has its CSR denied, keeps its self-signed certificate, and silently goes missing from kubectl top. Add it to the regex when you add it to the inventory.