Monitoring¶

Full observability stack based on kube-prometheus-stack: Prometheus, Grafana, Alertmanager, node-exporter and kube-state-metrics. node-exporter runs in its own node-exporter namespace so that only it is admitted at the privileged level (security policies). The chart's default rule set is good; its default routing swallows every alert, which is what most of this page is about.

At a glance¶

Configuration¶

CRDs¶

The monitoring.coreos.com CRDs are a separate prometheus-operator-crds Application, from the prometheus-community chart of that name, and this chart runs with crds.enabled: false. Cilium, cert-manager, External Secrets, Rook and OpenBao all render ServiceMonitors or PrometheusRules, and a missing kind fails their sync until the CRDs exist — see Bootstrap convergence. The Application has no resources finalizer, so deleting it leaves the CRDs, and every ServiceMonitor, Prometheus and Alertmanager with them, in place.

Alerting¶

Alertmanager routes to email. The chart's config is off (alertmanagerSpec.useExistingSecret) and alertmanager-config.yaml renders the whole alertmanager.yaml through ESO into alertmanagerSpec.configSecret, because only the password has a file form (smtp_auth_password_file) and the login and recipient are e-mail addresses that do not belong in Git.

The three values come from kv/monitoring/smtp (username, password, to); make bao-secrets prompts for them, or by hand:

bao kv put kv/monitoring/smtp username=... password=... to=...

For another provider, change smtp_smarthost in the template and the three values in OpenBao.

Scrape targets¶

Every ServiceMonitor and PodMonitor in every namespace, through the five *SelectorNilUsesHelmValues: false settings: the chart treats an empty selector as unset and renders release: kube-prometheus-stack in its place, which limits Prometheus to its own targets.

PrometheusRules load only from the namespaces the apps project cannot deploy to, matched on the kubernetes.io/metadata.name label the API server sets. Kured gates reboots on alert names, so a workload's rule named CephHealthError would otherwise stop OS updates cluster-wide. A rule in a new platform namespace loads only once that namespace is on the list in ruleNamespaceSelector.

Grafana¶

Dashboards¶

Where a chart can render a dashboard, it does; only charts without one get a vendored copy, which Renovate does not update — the header of each file names its upstream tag.

Usage¶

Adding a dashboard¶

Add a ConfigMap labelled grafana_dashboard: "1" next to the component it describes, in a file named grafana-dashboards.yaml and a ConfigMap named <component>-grafana-dashboards; a Secret carrying the label is not read. The sidecar watches only the platform namespaces listed in sidecar.dashboards.searchNamespace, so a workload cannot replace a dashboard; a component in a new namespace adds it there, and the apps project must not be able to deploy to it. Prefer the chart's own dashboard where it has one. A dashboard built in the UI survives restarts on the PVC but lives nowhere else: not in Git, not on a rebuilt cluster.

Growing the Prometheus volume¶

Kubernetes cannot expand a StatefulSet's volumeClaimTemplate, so raising the request in payload/platform/monitoring/application.yaml does not resize the claim that already exists. The operator handles its own half of that: it fails the StatefulSet update on the immutable field, logs recreating StatefulSet because the update operation wasn't possible, and rebuilds it orphaned so the pod survives. Only the claim is left behind.

1. Merge the size bump and wait for the Application to sync.

   kubectl -n argocd get app kube-prometheus-stack -o jsonpath='{.status.sync.status}'
   

2. Patch the claim to the same size. This is the one step nothing automates.

   SIZE=$(awk '/prometheusSpec:/{f=1} f&&/storage: /{print $2; exit}' \
     payload/platform/monitoring/application.yaml)
   kubectl -n monitoring patch pvc \
     prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0 \
     --patch "{\"spec\": {\"resources\": {\"requests\": {\"storage\": \"$SIZE\"}}}}"
   

3. The claim parks on FileSystemResizePending and the filesystem grows when the volume is next mounted, so the pod is restarted and Prometheus replays its WAL — a scrape gap of about a minute, not a seamless resize. Wait for the claim to report the new capacity.

   kubectl -n monitoring get pvc -w \
     -o custom-columns=NAME:.metadata.name,CAP:.status.capacity.storage
   

If the operator has not rebuilt the StatefulSet within a sync interval, do it by hand; --cascade=orphan keeps the pod and the claim.

kubectl -n monitoring delete statefulset \
  -l operator.prometheus.io/name=kube-prometheus-stack-prometheus --cascade=orphan

Alertmanager and Grafana take the same two steps, with their own claim names.

Resetting the Grafana admin password¶

Grafana reads the password only when it creates its database, so changing kv/monitoring/grafana-admin later does not change what it checks. Reset it to the value in OpenBao rather than a new one: grafana-admin.yaml pushes the generated password with updatePolicy: IfNotExists, so OpenBao keeps the first value forever and a rebuilt Grafana starts from it.

PASSWORD=$(kubectl -n openbao exec openbao-0 -- \
  bao kv get -mount=kv -field=password monitoring/grafana-admin)
kubectl -n monitoring exec deploy/kube-prometheus-stack-grafana -c grafana -- \
  grafana cli admin reset-admin-password "$PASSWORD"

Health check¶

kubectl -n monitoring get prometheus,alertmanager
kubectl -n monitoring get prometheus kube-prometheus-stack-prometheus \
  -o jsonpath='{.spec.serviceMonitorSelector}'   # should print {}
kubectl -n monitoring port-forward svc/kube-prometheus-stack-alertmanager 9093:9093
# then open http://localhost:9093 and check Status -> Config

Watchdog proves the pipeline as far as Alertmanager and not one step further; SMTP auth, spam heuristics and an expired password are unmonitored. Send yourself a test alert once and confirm it arrives.

Pitfalls¶