Tetragon¶

Tetragon watches processes at the kernel, with eBPF, on every node. Trivy says what an image could do; Tetragon records what a process did: every exec and exit in a pod, and, where a TracingPolicy says so, file opens and credential changes. It is the only thing here that sees a shell spawned in a running container.

At a glance¶

Configuration¶

Cilium's own chart, from helm.cilium.io; Tetragon does not depend on Cilium and runs privileged on the host network by the chart's default.

Policies¶

tracingpolicies.yaml carries the cluster-scoped TracingPolicy objects, adapted from the upstream policy library. All post at most one event per minute per selector, and most watch containers only, since the host has the node journal; sensitive-file-access and the signature hook of kernel-module-load record the host too. Every hook carries a message and one of upstream's observability.* tags, so an event says what it is without a lookup here.

The allow list in the first policy is the set of things that read key material in steady state, found by listing what fired. Add a binary there only after confirming it in the export; a person running cat on a node is exactly what the policy is for. A binary path is only trusted together with the host network namespace: any image can ship /usr/local/bin/etcd, but only a privileged namespace can run it with hostNetwork. The exemption stays in the kernel rather than on pod labels, because the podSelector pitfall below would miss a pod that reads a key the moment it starts; exemptions by namespace and workload go in the alert instead.

The third policy is the runtime half of an image check: nothing in an image lives on those paths, so a binary executing from one was written after the container started. An emptyDir can be mounted anywhere, so the policy names the mounts this cluster puts binaries into rather than every emptyDir; kubectl get pods -A -o json lists the rest. A readOnlyRootFilesystem pod makes the check complete, since then a new binary has nowhere else to land. The offline half, every binary Tetragon saw executed diffed against the image's file list, needs no policy: query Loki by image and binary and resolve each path in crane export <image> - | tar t.

Four more detections from the same library need no policy, because the exec event already carries what they look for: sudo, a setuid or file-capability binary raising privileges, a fileless exec and a deleted binary. They are Loki alerts. Upstream's sshd policy is covered by NodeSshLogin from the node journal.

Usage¶

The dashboard is the summary; the queries below are what its Loki panels run.

# Everything a policy caught
{namespace="kube-system", container="export-stdout"} | json | process_kprobe_policy_name!=""

# Every exec in one namespace, with arguments
{namespace="kube-system", container="export-stdout"} | json | process_exec_process_pod_namespace="home-assistant"
  | line_format "{{.process_exec_process_binary}} {{.process_exec_process_arguments}}"

The tetra CLI in the agent pod does the same live:

kubectl -n kube-system exec ds/tetragon -c tetragon -- tetra getevents -o compact --namespaces home-assistant

Health check¶

kubectl -n kube-system get ds tetragon
kubectl get tracingpolicies -o wide     # STATE should be enabled

Enforcement¶

Every policy but one only records. kill-unprivileged-user-namespace carries policy-mode: enforce and a Sigkill, because nothing here creates user namespaces and most kernel escalation chains start with one. It stays scoped to containers: a sysctl would cover the host too, where systemd's PrivateUsers= needs them. The one exception is arc-jobs, where buildah needs one for every image build; its podSelector excludes that namespace, at the price of the first seconds of every container (see Pitfalls), and a monitor-only twin records the events there. The limits, both checked on the running nodes: Override and enforcers need CONFIG_FUNCTION_ERROR_INJECTION, which Flatcar's kernel does not set, and BPF LSM hooks need bpf in the lsm= boot argument, which Flatcar does not pass. That leaves Sigkill, which upstream warns does not stop an operation already in flight. A policy without policy-mode loads in monitor mode and its kill actions are elided, so an enforcing policy has to say so.

Pitfalls¶