# Tetragon

[Tetragon](<https://tetragon.io/>) watches processes at the kernel, with eBPF, on every node. Trivy says what an image *could* do; Tetragon records what a process *did*: every exec and exit in a pod, and, where a `TracingPolicy` says so, file opens and credential changes. It is the only thing here that sees a shell spawned in a running container.

## At a glance

|  |  |
| --- | --- |
| Namespace | `kube-system` |
| Depends on | Kernel BTF at `/sys/kernel/btf/vmlinux`, which Flatcar ships; [Logging](<https://homelab.wlkr.ch/platform/logging/index.md>) for the events, [Monitoring](<https://homelab.wlkr.ch/platform/monitoring/index.md>) for the alerts |
| If it is down | No runtime record and no runtime alerts; nothing else notices |
| Health check | `kubectl get tracingpolicies -o wide`; `TetragonPolicyNotLoaded` and `TetragonEventsLost` otherwise |
| Dashboard | [Tetragon](<https://monitoring.infra.k8s.wlkr.ch/d/tetragon-overview>): agent health, event rates, policy hits by binary, and the latest policy hits, shells and execs read from Loki |
| Files | `payload/platform/tetragon/`: the chart, the policies, `prometheusrule.yaml` and a dashboard written for this cluster, since upstream ships none |

## Configuration

Cilium's own chart, from `helm.cilium.io`; Tetragon does not depend on Cilium and runs privileged on the host network by the chart's default.

| Setting | Why |
| --- | --- |
| Export to stdout | The chart runs a sidecar that tails Tetragon's JSON file to stdout, so [Alloy](<https://homelab.wlkr.ch/platform/logging/index.md>) collects it as any container log: `{namespace="kube-system", container="export-stdout"}` |
| `exportDenyList` | Exec and exit events from the host and from the platform's own `kube-system` pods, selected by label, stay out of the export; they are the bulk of the volume and the least interesting. Any other pod in `kube-system` is exported, because a stolen write role lands its privileged pod there. Policy hits still pass, which is the point of the `event_set` field |
| `enableProcessCred` | Capabilities and `privileges_changed` on every exec, so a setuid binary or a file-capability escalation is visible without a policy |
| Two `ServiceMonitor`s | `tetragon_policy_events_total` is what the alert reads; `tetragon_tracingpolicy_loaded` and the ring-buffer counters are its health |
| Memory limit | 2.5x the measured working set of the busiest agent with the policy library loaded, per the [platform rule](<https://homelab.wlkr.ch/platform/#components>); the dashboard's first row shows the current worst node against it |

### Policies

`tracingpolicies.yaml` carries the cluster-scoped `TracingPolicy` objects, adapted from the [upstream policy library](<https://tetragon.io/docs/policy-library/observability/>). All post at most one event per minute per selector, and most watch containers only, since the host has the node journal; `sensitive-file-access` and the signature hook of `kernel-module-load` record the host too. Every hook carries a `message` and one of upstream's `observability.*` tags, so an event says what it is without a lookup here.

| Policy | Hook | Fires on | Alerted |
| --- | --- | --- | --- |
| `sensitive-file-access` | `security_file_permission` | A read of a control-plane or etcd key, the encryption config, an admin kubeconfig, etcd's data or a kubelet key, matched by the end of the path so a hostPath mounted anywhere counts, unless the reader is a control-plane binary, the kubelet, `kubeadm` or `kube-vip` on the host network, or the path is under `/run/cilium/state/`, where Cilium's route reconciler keeps its own `.wal`; the alert leaves out `etcdctl` in the etcd backup and defrag Jobs, by namespace and Job name; a read of `/etc/shadow` on the host by anything but PAM's `/sbin/unix_chkpwd`, which every login and `sudo` runs (in a container any read is PAM in the image, so only writes count there); any write, host or container, to `/etc/passwd`, `/etc/shadow`, `/etc/group`, `/etc/sudoers`, `/etc/ld.so.preload`, or under `/etc/sudoers.d/`, `/etc/cron.d/` and `/root/.ssh/` | Yes: `TetragonSensitiveFileAccess` |
| `process-creds-changed` | `commit_creds` | A container process that moved into another mount, pid, network or user namespace after its exec, except `runc`; the alert leaves out the `cilium` DaemonSet in `kube-system` and all of `rook-ceph`, whose init containers and OSD prepare Jobs `nsenter` into the host by design, and `/proc/self/exe` in the `arc-jobs` workflow pods, which is buildah re-executing itself into [its user namespace](<https://homelab.wlkr.ch/platform/github-runners/index.md>). The hook runs on every execve and fork; the selector keeps the escape signal. Upstream's `matchCapabilityChanges` is not used: it matches a change in either direction, so every entrypoint that drops root fires it | Yes: `TetragonCredentialsEscalated` |
| `exec-from-writable-path` | `security_bprm_check` | A container executing a file under `/tmp/`, `/var/tmp/`, `/dev/shm/`, `/run/`, `/var/run/`, `/shared/`, `/controller/` or `/plugins/`, except the two binaries an init container copies there and the scripts s6-overlay writes under `/run/s6*` and `/run/service/` at start | Yes: `TetragonExecFromWritablePath` |
| `library-from-writable-path` | `security_mmap_file` | The same paths mapped with `PROT_EXEC`, which is how a dropped shared library loads, except the native libraries runtimes unpack under `/tmp` on every start: Alloy's Snowflake driver (`/tmp/gosnowflake-cgo*`) and zstd-jni in Dependency-Track (`/tmp/libzstd-jni-*`) | Yes: `TetragonLibraryFromWritablePath` |
| `privileges-raise`, `privileges-raise-gid`, `privileges-raise-gid-argocd` | The `__sys_set*uid` and `__sys_set*gid` family | Any setuid or setgid to root inside a container. A record: root re-asserting root is routine (`logrotate`), and a setuid binary that actually raises privileges is `TetragonPrivilegedExec`. `runc`, which sets the ids on every container start, is dropped in the kernel. The redis-ha busybox calls setgid on every applet, so the gid hooks leave out pods labelled `app: redis-ha` in `argocd` only; one `podSelector` cannot say "this label in that namespace", hence two gid policies | No |
| `bpf-program-load` | `bpf_check` | Any BPF program load from a container; the alert leaves out `cilium` and `kube-system` | Yes: `TetragonBpfProgramLoaded` |
| `kernel-module-load` | `security_kernel_module_request`, `security_kernel_read_file`, `find_module_sections` | A module requested by name or read from a container, minus the `net-pf-2-proto-262` and `net-pf-10-proto-262` requests Go binaries and haproxy make when they probe MPTCP at start and the `net-pf-31` one Home Assistant makes when it looks for a Bluetooth adapter; the alert leaves out `rook-ceph`, whose CSI plugin loads `rbd` after a boot. The third hook records every load's signature check, host included: Flatcar signs its modules but does not enforce it (`CONFIG_MODULE_SIG` without `_FORCE`) | Yes: `TetragonKernelModuleLoaded`; `TetragonUnsignedKernelModule` from Loki |
| `dns-outside-cluster` | `ip_output` | A port 53 packet from a container to anything but the kube-dns ClusterIP or the pod range, which is a hard-coded resolver in an image, a tunnel or exfiltration. The pod range is left out because it holds each node's Cilium router address, the pod's default gateway, which Home Assistant's dhcp integration asks for PTR records at start. The hook sees TCP and UDP alike, so musl's unconnected sends count too; the alert leaves out `kube-system`, where CoreDNS forwards upstream, and `cert-manager`, whose DNS-01 propagation check asks the zone's authoritative servers | Yes: `TetragonDnsOutsideCluster` |
| `mount-in-container` | `security_sb_mount` | Any mount from a container except by `runc`, which mounts the rootfs, proc and sysfs on every start. Escapes start with a mount of the host's disk, cgroup or proc; the alert leaves out `rook-ceph`, `cilium` and `kube-system`, which mount by design | Yes: `TetragonMountInContainer` |
| `ptrace-in-container` | `security_ptrace_access_check` | An attach-mode access to another process from a container: ptrace, `process_vm_writev`, `/proc/<pid>/mem`. The read mode `ps` uses is left alone | Yes: `TetragonPtraceInContainer` |
| `kill-unprivileged-user-namespace` | `create_user_ns` | A user namespace created without `CAP_SYS_ADMIN` in a container, killed, except in `arc-jobs`. The only policy in `enforce` mode | Yes: `TetragonUserNamespaceCreated` |
| `unprivileged-user-namespace-arc-jobs` | `create_user_ns` | The same in `arc-jobs`, recorded: buildah builds the images there in a user namespace, see [GitHub runners](<https://homelab.wlkr.ch/platform/github-runners/index.md>) | No; every image build would fire it |
| `egress-outside-cluster` | `security_socket_connect` | An IPv4 `connect()`, TCP or UDP, from a container to anything outside the pod, service and site ranges | No; ACME, S3, the Trivy database and Home Assistant all do this routinely |

The allow list in the first policy is the set of things that read key material in steady state, found by listing what fired. Add a binary there only after confirming it in the export; a person running `cat` on a node is exactly what the policy is for. A binary path is only trusted together with the host network namespace: any image can ship `/usr/local/bin/etcd`, but only a privileged namespace can run it with `hostNetwork`. The exemption stays in the kernel rather than on pod labels, because the `podSelector` pitfall below would miss a pod that reads a key the moment it starts; exemptions by namespace and workload go in the alert instead.

The third policy is the runtime half of an image check: nothing in an image lives on those paths, so a binary executing from one was written after the container started. An emptyDir can be mounted anywhere, so the policy names the mounts this cluster puts binaries into rather than every emptyDir; `kubectl get pods -A -o json` lists the rest. A `readOnlyRootFilesystem` pod makes the check complete, since then a new binary has nowhere else to land. The offline half, every binary Tetragon saw executed diffed against the image's file list, needs no policy: query Loki by image and binary and resolve each path in `crane export <image> - | tar t`.

Four more detections from the same library need no policy, because the exec event already carries what they look for: `sudo`, a setuid or file-capability binary raising privileges, a fileless exec and a deleted binary. They are [Loki alerts](<https://homelab.wlkr.ch/platform/logging/#alerting>). Upstream's `sshd` policy is covered by `NodeSshLogin` from the node journal.

## Usage

The dashboard is the summary; the queries below are what its Loki panels run.

```text
# Everything a policy caught
{namespace="kube-system", container="export-stdout"} | json | process_kprobe_policy_name!=""

# Every exec in one namespace, with arguments
{namespace="kube-system", container="export-stdout"} | json | process_exec_process_pod_namespace="home-assistant"
  | line_format "{{.process_exec_process_binary}} {{.process_exec_process_arguments}}"
```

The `tetra` CLI in the agent pod does the same live:

```bash
kubectl -n kube-system exec ds/tetragon -c tetragon -- tetra getevents -o compact --namespaces home-assistant
```

## Health check

```bash
kubectl -n kube-system get ds tetragon
kubectl get tracingpolicies -o wide     # STATE should be enabled
```

### Enforcement

Every policy but one only records. `kill-unprivileged-user-namespace` carries `policy-mode: enforce` and a `Sigkill`, because nothing here creates user namespaces and most kernel escalation chains start with one. It stays scoped to containers: a sysctl would cover the host too, where systemd's `PrivateUsers=` needs them. The one exception is `arc-jobs`, where buildah needs one for every image build; its `podSelector` excludes that namespace, at the price of the first seconds of every container (see Pitfalls), and a monitor-only twin records the events there. The limits, both checked on the running nodes: `Override` and enforcers need `CONFIG_FUNCTION_ERROR_INJECTION`, which Flatcar's kernel does not set, and BPF LSM hooks need `bpf` in the `lsm=` boot argument, which Flatcar does not pass. That leaves `Sigkill`, which upstream warns does not stop an operation already in flight. A policy without `policy-mode` loads in monitor mode and its kill actions are elided, so an enforcing policy has to say so.

## Pitfalls

> [!WARNING]
>
> **A `podSelector` misses a pod's first seconds**
>
> The policy filter learns pods from the API and maps their cgroups; a process that runs the moment a container starts is not yet selected, and the test pod's `unshare` went unrecorded by the filtered policy while the unfiltered one killed it. Put the hook an alert depends on in a policy without a `podSelector`, and do exclusions in PromQL.

> [!WARNING]
>
> **A first hit is a new series, and `increase()` of one sample is zero**
>
> `tetragon_policy_events_total` has a pod label, so the first hit from a pod creates a counter series inside the alert window and `increase()` reports nothing until the second scrape moves it. Every policy alert therefore also matches a series absent one window earlier, with `unless ... offset`, and holds with `keep_firing_for`, because the agent drops a pod's series when the pod exits and a one-shot pod would otherwise flap. Copy that shape for a new alert, and test it with a pod that fires once, not with one that keeps firing.

> [!WARNING]
>
> **A policy that does not load is silent**
>
> A `TracingPolicy` naming a kernel function this kernel does not export sits in state `error` and watches nothing. `TetragonPolicyNotLoaded` fires after its `for`; check `kubectl get tracingpolicies -o wide` after every Flatcar release that moves the kernel.