Logging¶
Loki stores logs;
Grafana Alloy collects them. Both are
queried from the existing Grafana at
monitoring.infra.k8s.wlkr.ch. Without
them a crashed pod's logs are gone with the container, and on Flatcar the
kubelet, containerd, systemd-sysupdate and update-engine logs live in
journald and are reachable only over SSH to the node that is misbehaving.
At a glance¶
| Namespace | logging |
| Depends on | Rook-Ceph object storage for chunks, Monitoring for the Grafana that queries it |
| If it is down | Logs stop being collected and are not backfilled. The audit log loses its durable copy, and the log alerts stop |
| Health check | kubectl -n logging get pods, then a {job="kubernetes-audit"} query in Grafana |
| Metrics | Loki and Alloy, each through its chart's ServiceMonitor; both sets of dashboards are listed under Monitoring |
| Files | payload/platform/loki/ and payload/platform/alloy/, one chart Application each (loki, alloy); payload/platform/logging/, the logging Application: the bucket claim in bucket.yaml, loki-rules.yaml, the Grafana datasource and the Alloy dashboards |
Not Promtail, and not the grafana/loki chart
Promtail reached end of life in March 2026; Alloy is its supported replacement. Loki comes from grafana-community/loki: the OSS chart moved to grafana-community/helm-charts in March 2026, and what stayed in grafana/loki is maintained for Grafana Enterprise Logs customers, pins a k8s-sidecar several minor releases behind and closes open-source pull requests with a pointer to the community repository.
Configuration¶
Sources¶
| Source | Path | Labels |
|---|---|---|
| Container logs | /var/log/pods/<ns>_<pod>_<uid>/<container>/*.log |
namespace, pod, container, node, app |
| Node journal | /var/log/journal |
unit, node, job="systemd-journal" |
| API server audit log | /var/log/kubernetes/audit/audit.log |
verb, audit_level, node, job="kubernetes-audit" |
| Hubble policy verdicts | /var/run/cilium/hubble/events.log |
verdict, node, job="hubble" |
The first three are on the root filesystem, so one varlog: true mount covers
them; the Hubble export is a separate read-only hostPath — see
Cilium.
The audit log exists only on the control-plane nodes; Alloy runs everywhere and
local.file_match finds nothing on the workers. Audit events keep only verb
and audit_level as labels — small closed sets — while user and resource stay
in the line where | json can reach them. See
Audit logging for what is recorded.
Alloy pipeline¶
| Stage | Why |
|---|---|
discovery.kubernetes with a spec.nodeName field selector |
Each agent discovers only its own node's pods; without it every agent watches every pod and discards all but its own |
stage.cri {} on container logs |
containerd writes <timestamp> <stream> <flags> <message>; without it the timestamp lands inside the line and Loki stamps everything at ingest time |
discovery.relabel "journal" sets job |
loki.source.journal overwrites job with its component ID after applying labels; only relabel rules run after that |
stage.json, stage.labels, stage.timestamp on audit events |
Stamps each event with its requestReceivedTimestamp rather than when Alloy read the file |
Setting in alloy/application.yaml |
Why |
|---|---|
| Control-plane toleration | Without it Alloy runs on the workers alone and neither the audit log nor the control-plane kubelets' journals reach Loki |
enableReporting: false |
Alloy otherwise sends usage statistics to stats.grafana.org; the policy audit showed the call, and telemetry is off in every component here |
runAsUser: 0 |
The log files are root-owned; a non-root Alloy does not fail, it silently collects nothing. Read-only root, no capabilities and RuntimeDefault seccomp lock the rest down |
emptyDir at /tmp |
The chart's storagePath defaults to /tmp/alloy, unwritable on a read-only root; it holds tailing positions, which did not survive a restart before either |
rbac.rules without secrets |
The chart's ClusterRole reads every Secret for remote.kubernetes.secret, which nothing here uses; KubernetesAuditSecretReadByUser would show such a read, but a grant with no purpose is better not made |
Storage¶
Loki runs as a single binary (deploymentMode: Monolithic) and keeps chunks
in the Ceph object store; a local PVC on
rook-ceph-block holds the WAL and the index being built. The bucket is the
ObjectBucketClaim in logging/bucket.yaml, with a fixed bucketName: a
generated name carries a random suffix that would have to be read back at
runtime, a fixed one keeps Loki's config static and identical after a rebuild.
Setting in loki/application.yaml |
Why |
|---|---|
singleBinary.extraArgs -config.expand-env=true and singleBinary.extraEnvFrom |
Rook writes the bucket credentials to a Secret, Loki reads them as ${AWS_ACCESS_KEY_ID} — nothing in Git |
loki.analytics.reporting_enabled: false |
Loki otherwise sends usage statistics to stats.grafana.org a few times a day; same reason as Alloy's switch above |
chunksCache and resultsCache off |
Four memcached pods in front of a Loki this size |
loki.podSecurityContext |
Adds RuntimeDefault seccomp at pod level, covering the rules sidecar too |
| Retention with the compactor enabled | Old chunks are actually deleted |
rulerConfig |
The ruler evaluates LogQL alerts and posts them to Alertmanager; without it the audit log and the journal are query-only. Rules come from local files the chart's sidecar copies out of loki_rule ConfigMaps, so storage.type is local rather than the bucket the chart would pick; rule_path is the ruler's scratch directory and must not be the rules directory |
rbac.namespaced: true and sidecar.rules.resource: configmap |
The rules are ConfigMaps in logging, so a Role there replaces the chart's cluster-wide ClusterRole, for the same reason as Alloy's rbac.rules above; the chart's binding could not be repointed because a roleRef is immutable. resource: configmap keeps the sidecar from listing Secrets |
retention_stream for the findings-history container |
The daily Trivy summary is a few lines and the only long-range record of findings, so that one stream outlives the general retention_period by far |
Alerting¶
Alerts over logs live in logging/loki-rules.yaml, a ConfigMap labelled
loki_rule in the same format as a PrometheusRule group with LogQL
expressions. The k8s-sidecar-target-directory: fake annotation puts the
file under the tenant directory the ruler reads (fake is the tenant when
auth_enabled is off). Prometheus-side rules stay with their components;
these are the events only a log carries. Loki's own health alerts (request
errors, latency, panics, the compactor) are the chart's monitoring.alerts,
the mixin Grafana maintains, not rules written here.
| Alert | Fires on | Why it is a log alert |
|---|---|---|
KubernetesAuditExecIntoPod |
A successful exec or attach |
Only the audit log records who ran what in which pod |
KubernetesAuditSecretReadByUser |
A Secret read by a person, or by a ServiceAccount outside its own namespace that is not on the allowlist of platform controllers | A stolen token reads across namespaces; the allowlist is the ServiceAccounts RBAC grants Secrets to cluster-wide, plus the runner's in arc-jobs only, where its container hook keeps each job's environment, and a ServiceAccount reading its own namespace is what its Role is for |
KubernetesAuditRbacBindingChanged |
A ClusterRole or ClusterRoleBinding written by anything but the ArgoCD controller or a kube-system controller |
Argo CD is the only intended writer; namespaced RoleBindings are left out because CloudNativePG reconciles one per database continuously |
KubernetesAuditForbiddenBurst |
More than ten 403s from one identity in ten minutes | What RBAC probing from a compromised pod looks like |
KubernetesAuditAnonymousRequest |
A successful anonymous request | The audit policy drops the health endpoints, the only legitimate anonymous paths |
KubernetesAuditEsoTokenMinted |
A successful serviceaccounts/token create for an external-secrets-vault ServiceAccount by anyone but ESO's controller, a kubelet starting a pod under that ServiceAccount included |
OpenBao records the same role and ServiceAccount whoever minted the token, so only the audit log names the minter. A stolen controller token or a legacy token Secret does not trip it: neither shows a different user on this request |
OpenBaoSecretReadOutsideEso |
A kv/data/ read from OpenBao by any token whose display name is not kubernetes-<namespace>-external-secrets-vault, ESO's login for one of its stores |
OpenBao's audit device writes to stdout; ESO is the only day-to-day reader, every Kubernetes role is bound to that ServiceAccount name, and a token created by hand is prefixed token-, so it cannot borrow the name |
TetragonSudoInContainer |
sudo executed in a container, except the smartctl and nvme calls ceph-osd makes once a day for its device health report |
Only the exec event carries the binary and its parent |
TetragonPrivilegedExec |
A setuid, setgid or file-capability binary raised privileges on exec, with the same ceph-osd exception |
The binary_properties field exists only in the exec event |
TetragonFilelessExec |
A binary executed from memory, via memfd_create or an anonymous descriptor, except runc's sealed self-copy on every exec |
Same |
TetragonDeletedBinaryExec |
A binary executed after its last link was removed | Same; see Tetragon |
TetragonUnsignedKernelModule |
A kernel module loaded without a valid signature, on a node or from a container | The signature result is an argument of the kprobe event, not a metric label |
HubblePolicyDrops |
More than three POLICY_DENIED packets in ten minutes for one source workload, destination and port, each counted once although Hubble also reports it as a policy verdict; replies, ICMP errors (a pod's port-unreachable answer to a DNS reply that came too late), pods still in the init identity and addresses with no pod behind them are left out. The namespace label is the policy that refused, so the mail names the file to edit |
A connection's retransmits pass the threshold and a name rule's single early SYN does not; the metric counts per namespace pair only and misses the first burst of a new pair |
NodeSshLogin |
An accepted SSH login on a node | Nothing routine logs in after provisioning |
NodeSshAuthFailures |
More than five failed SSH attempts on a node in ten minutes | Password authentication is off; repeats are a scan or a retried key |
To confirm the ruler loaded a rule:
Dashboards¶
grafana-dashboards.yaml carries three dashboards from the Alloy mixin
(operations/alloy-mixin/rendered/dashboards/), unmodified; the chart renders
none and Renovate does not see the copy, so re-copy them when Alloy moves a
minor version. The mixin's other dashboards cover features this Alloy does not
use. The Loki datasource in grafana-datasource.yaml declares its namespace
as monitoring, the only namespace the Grafana sidecar watches for
grafana_datasource ConfigMaps.
Usage¶
Choose Loki as the datasource in Grafana and query by label:
{namespace="rook-ceph"} |= "error"
{unit="kubelet.service", node="odin"}
{namespace="openbao"} |= "sealed"
{job="kubernetes-audit"} | json | objectRef_resource="secrets"
Health check¶
Then run a {job="kubernetes-audit"} query in Grafana; an empty result with
healthy pods means the control-plane agents are not reading the file.
Pitfalls¶
Loki settings belong on singleBinary, not global
global.extraArgs and global.extraEnvFrom look right, but the single-binary StatefulSet reads only singleBinary.extraArgs and singleBinary.extraEnvFrom. Set globally, the config is full of unexpanded ${...} and Loki cannot authenticate to RGW, with an error pointing nowhere near the cause. Likewise singleBinary.podSecurityContext does not exist and is silently inert.