# Security Policies Five cluster-wide controls: Pod Security Admission, default-deny network policy in both directions, a host firewall on the nodes, no API token on the `default` ServiceAccount, and admission policies on image references, namespace labels and the `authentik` namespace. Each is easy to turn on and hard to turn on safely, so the pattern is the same for all of them: measure first, enforce second, one namespace at a time. ## At a glance | | | | --- | --- | | Namespace | `kube-system` for the Application; it owns Namespace objects across the cluster | | Depends on | [Cilium]() to enforce the network policies | | If it is down | Nothing at the time; the labels and policies stay applied and only stop being corrected | | Health check | `kubectl get ns -L pod-security.kubernetes.io/enforce`, `kubectl get validatingadmissionpolicy` | | Pruning | Enabled, except for the Namespaces: each carries `argocd.argoproj.io/sync-options: Prune=false`, because pruning a Namespace deletes everything inside it, PVCs included. Anything else removed from Git is pruned, so the Application does not sit OutOfSync on leftovers | | Files | `payload/platform/security/`, the network policies one file per namespace under `network-policies/`, plus the CoreDNS PodDisruptionBudget — see [Components]() | ## Configuration ### Pod Security Admission A namespace without labels gets the cluster default from `ansible/templates/admission.yaml.j2`: `enforce` at `baseline`, `audit` and `warn` at `restricted`. It reaches a control-plane node only at provisioning; see [Node hardening](). It exempts `kube-system`, which Cilium needs before Argo CD can label anything, and the namespaces that need `privileged` but carry no labels. An exemption skips `audit` and `warn` as well, so the labelled namespaces below are not exempt. Each namespace in `pod-security.yaml` carries three labels: | Label | Set to | Effect | | --- | --- | --- | | `enforce` | The level the namespace demonstrably needs | Rejects pods that violate it | | `audit` | Stricter | Records violations in the [API server audit log]() | | `warn` | Stricter | Warns whoever applies the manifest | Enforcement is set to what already works, so nothing running breaks, while `warn` and `audit` show what a stricter level would catch; tightening later is an informed change. The level per namespace is in `pod-security.yaml`; the rule behind it: | Enforce | Which namespaces | Why not stricter | | --- | --- | --- | | `privileged` | Those with a component on the host: the CNI and control plane, Ceph's OSDs, node-exporter, Trivy's node-collector, the etcd jobs and Velero's node-agent, Alloy, Kured | Host networking, the host PID namespace, raw block devices or a hostPath under the kubelet, etcd or log directories, all of which `baseline` forbids. node-exporter has a namespace of its own (`namespaceOverride`) so the rest of the monitoring stack need not follow it | | `baseline` | Those with a container that starts as root or escalates: Grafana's `init-chown-data`, the runners and their job pods, and the operators not yet checked against `restricted` | Reduced when the chart offers a non-root path, one namespace at a time | | `restricted` | Everything else, and the namespaces that run no pods | — | `privileged` here means "not yet reduced", not "unexamined": `audit` and `warn` are still `baseline` or `restricted` on all of them. Namespace objects are owned by this Application so the labels stay declarative rather than drifting after `CreateNamespace=true`. `cilium-secrets` is the exception: the Cilium chart creates it, so its labels are `secretsNamespaceLabels` in `payload/platform/cilium/values.yaml`, and a second owner would fight the first. Workload namespaces label themselves in the workloads repository, and `namespace-privileged-allowed` keeps them off `privileged`. ### Network policies One file per namespace in `network-policies/`, holding that namespace's `CiliumNetworkPolicy` objects. A namespace gets two whole-namespace policies, `default-ingress` and `default-egress`, so every pod in it is default-deny in both directions, plus one policy per workload that needs more than the rest of the namespace does. Each policy's `description` is the record of who may connect and why; the rules are not repeated here. `cluster-wide.yaml` holds the `CiliumClusterwideNetworkPolicy` objects: the rules every pod gets, so a namespace file is the record of what its namespace may do *beyond* those. #### Why CiliumNetworkPolicy and not NetworkPolicy Two kinds of traffic have no pod identity a standard `NetworkPolicy` can name: Gateway traffic, which reaches a backend from the per-node Envoy, and kubelet health probes, which come from the node. A plain default-deny therefore kills ingress *and* probes, and the pods restart forever in a way that looks like an application fault. Cilium's `fromEntities` names them: `ingress` for Envoy-proxied traffic, `host` and `remote-node` for the kubelet. #### Cluster-wide rules A rule that is word-for-word the same in every namespace file is lifted into `cluster-wide.yaml`, selecting every endpoint that has a namespace and with `enableDefaultDeny` off in both directions. The selector is not `{}` because that also takes the Gateway's `reserved:ingress` endpoint and the health endpoint, which then carry a policy they have no use for. Default deny is off because a cluster-wide egress rule would otherwise put every pod in a namespace without policies into egress default-deny. DNS is the only rule lifted; the intra-namespace and node-probe rules would qualify but change what a namespace file says, so they stay. The second policy there, `init`, is for the seconds before the agent has resolved a new pod's labels. The pod then carries the reserved `init` identity, which no namespace policy selects, and the audit log shows its traffic denied all the same; a scan Job's first lookups and a starting pod's probes were the bulk of the audit log. It admits the node, DNS out and DNS in, nothing else. The third, `cnpg-backups`, selects every CloudNativePG pod by its `cnpg.io/cluster` label and admits only the object store, by the same two rules Velero's namespace has. Every database's backup sidecar needs exactly that, wherever the database runs, so one policy replaces a rule in each workload's file. #### What every policy contains | Rule | Why | | --- | --- | | Ingress from within the namespace | Replicas, sidecars and a workload's own database talk to each other on ports that change with the chart | | Ingress from `host` and `remote-node` | Kubelet probes come from the node, and so do the API server's webhook calls, the aggregation layer and `kubectl port-forward`; all of them carry the node's identity, on a control-plane node with the `kube-apiserver` label as well | | Ingress from `ingress` | Only where an HTTPRoute sends the Gateway's Envoy straight at the namespace; a namespace behind the Authentik outpost admits `authentik` instead. Kept at L4: the HTTPRoute is already the L7 filter for that traffic | | Ingress from `monitoring` on the scraped port, `GET /metrics` | The one port a ServiceMonitor or PodMonitor names, at L7, so the scraper can open nothing else | | Egress within the namespace, and to `kube-apiserver` where there is a Kubernetes client | The `kube-apiserver` entity is the endpoints behind `kubernetes.default`; no pod uses the kube-vip address. DNS is not repeated: the cluster-wide rule above sends every lookup through the proxy, which is what makes `toFQDNs` work and every lookup visible in Hubble | | Egress to external names as `toFQDNs` | A name reads as the dependency it is; an address does not. The Gateway's own addresses count as external: a pod calling `auth.k8s.wlkr.ch` is classified `world`, not `ingress` | | Egress to the backend pod as well, when the name is one the Gateway serves | The Gateway's Envoy checks the caller's egress policy a second time, against the backend pod it picked and that pod's port, and answers `403 Access denied` itself when no rule matches; audit mode never sees it. The OIDC clients of Authentik carry both rules | | HTTP rules on plaintext ports, ingress side only | A request crossing a namespace boundary is proxied once, by the receiving node. TLS and gRPC ports stay at L4; the proxy cannot read them. A machine caller gets the methods and paths a week of Hubble records showed; a browser UI behind the Authentik outpost stays at `http: [{}]`, its paths being the application's own. The path is matched with its query string: a rule anchored with `$` names the query as well, as OpenBao's `/v1/sys/metrics\?format=prometheus$` does | Host-networked pods (Cilium, kube-vip, the control plane, node-exporter, Tetragon, the CSI node plugins) have no endpoint of their own and cannot be selected; to everything else they are `host` or `remote-node`. `kube-system` therefore covers only its pod-networked workloads, each selected by label. Where a chart ships `NetworkPolicy` objects of its own (`argocd`) they are switched off in its values: Cilium unions the two kinds, so a chart rule that allows everything would silently override the default-deny. ### Host firewall Pod policies cannot select a host-networked port. A `CiliumClusterwideNetworkPolicy` with a `nodeSelector` can, and Cilium's [host firewall]() enforces it on the native device and `cilium_host`. What each port is for is in [Host Firewall Inventory 2026-10-30](). | Behaviour | Why it matters | | --- | --- | | Only packets addressed to a node IP are checked | The kube-vip address counts on the node holding it; Gateway addresses and NodePorts are translated first, so their policy is the backend pod's | | A node with no selecting policy is not denied anything on IPv4 | The switch can merge alone, and reverting the policy needs no second rollout | | No rule in a direction leaves that direction unenforced | A host policy with `ingress` only never blocks a node's own connections, and their replies pass through conntrack | | Host endpoint labels are filtered | `kubernetes.io/hostname` is not among them, so a node can only be chosen by a label such as `node-access` in the upstream example, or by the `control-plane` role label | | A pod reaching another node's IP is masqueraded to its own node | The target sees `remote-node`, so the firewall can tell the LAN from the cluster but not Prometheus from any other pod; the source pod's egress policy decides that | `network-policies/host-firewall.yaml` selects the nodes carrying `host-firewall.wlkr.ch/selected`. It denies the LAN everything but SSH, the API server, DHCP and ping, and admits the whole cluster to every port, since a narrower cluster rule could not name Prometheus and could lock out a node. ### Default ServiceAccount tokens A pod that names no `serviceAccountName` lands on `default`, which nothing configures, and gets an API token it has no use for. A workload reaching the API through `default` is already a misconfiguration, so `default-serviceaccounts.yaml` sets `automountServiceAccountToken: false` on it in every namespace with nothing to preserve, empty namespaces included. Left out: `kube-system`, where the control plane authenticates with client certificates and "very likely fine" is not the standard. `authentik-server` runs on `default` by chart default and is covered: Authentik's Kubernetes calls are worker tasks, which run under the `authentik` account. The same rule applies to a chart workload whose own ServiceAccount has no binding — Alertmanager, Cilium's Envoy and Argo CD's HAProxy switch the mount off in their values — unless the process needs the in-cluster config to start at all ([the External Secrets webhook]()) or the chart offers no switch (Rook's `rook-ceph-default` and `rook-ceph-rgw`). ### Agent RBAC `claude-agent-rbac.yaml` gives the in-cluster Claude Code agents two ClusterRoles for the `claude-reader` and `claude-writer` ServiceAccounts of the `claude-agents` namespace. Cluster-scoped RBAC cannot come from the `apps` project, and what an agent may touch is a platform decision. | Role | Bound | Grants | Leaves out | | --- | --- | --- | --- | | `claude-agent-read` | Cluster-wide, to both ServiceAccounts | `get`, `list`, `watch` on named core, workload and CRD kinds, `pods/log` included | Secrets, RBAC, webhooks, CRDs, `pods/exec`, Trivy's exposed-secret reports | | `claude-agent-write` | Nowhere by the platform: the workloads repository adds a RoleBinding in each namespace a writer session may change | Create, update, patch and delete on pods, workloads, Services, ConfigMaps, jobs, autoscalers and disruption budgets | The above, plus PersistentVolumeClaims (deleting one deletes data), NetworkPolicies and routes (they change who may reach what, which is a pull request) | - **Creating a pod reads every Secret in its namespace.** That is why the write role is bound per namespace, and the `apps` project cannot place a RoleBinding in the namespaces its [destinations]() exclude. - **`claude-writer-pod-limits`** in `admission-policies.yaml` refuses any pod or pod template from the writer that uses the host's network, PID or IPC namespace, a `hostPath` volume, a privileged container, or a ServiceAccount other than `default` or the one the workload already runs as. A namespace without a Pod Security level allows all of those, and another ServiceAccount's token is that account's RBAC. - **`pods/log` is cluster-wide**, so the reader sees whatever any component logs, platform namespaces included. Triage needs it, and a cluster-wide grant cannot leave out namespaces. Kinds are listed, never wildcarded, so a new CRD is invisible to the agents until someone adds it. Argo CD reverts a hand edit within minutes either way. ### Admission policies Trivy reports a bad image after it is running; `admission-policies.yaml` refuses or records it at admission, with the built-in `ValidatingAdmissionPolicy` rather than a policy engine. The image policies match Pods and every controller that produces them, so a bad Deployment is rejected at `kubectl apply` rather than failing quietly in its ReplicaSet. The other three close what RBAC and the `apps` project cannot express. | Policy | Action | Why | | --- | --- | --- | | `image-tag-pinned` | `Deny` | A tag other than `latest`, or a digest. Nothing running violates it, and an unpinned image is the one Renovate can neither track nor roll back | | `image-registry-allowed` | `Audit` and `Warn` | The registries the cluster pulls from today; a reference with no registry counts as `docker.io`. Recorded to the [audit log]() and shown to whoever applies, not enforced, until the record is clean | | `claude-writer-pod-limits` | `Deny` | Host access and borrowed ServiceAccounts from the agent writer, see [Agent RBAC](<#agent-rbac>) | | `namespace-privileged-allowed` | `Deny` | A Namespace may carry `enforce: privileged` only if it is in the policy's list, the `privileged` rows above. The `apps` project creates its own Namespaces and their labels, so without it a workload could opt out of Pod Security | | `authentik-foreign-objects` | `Deny` | In `authentik`, an object an Argo CD Application applied must come from `authentik` or `security`, or be the ConfigMap `authentik-blueprint-`. Argo CD cannot whitelist kinds per destination namespace, so this is what holds `apps` to its blueprint there; objects with no `argocd.argoproj.io/tracking-id` annotation, from controllers and `kubectl`, are not matched | All but one carry `failurePolicy: Ignore`: an expression that errors lets the request through rather than stopping every pod in the cluster. A real violation is still refused. `claude-writer-pod-limits` fails closed, since it only ever matches one identity. ## Usage ### Promoting the registry allowlist 1. Confirm nothing running would be refused, over the full Loki retention: ```text {job="kubernetes-audit"} |= "validation.policy.admission.k8s.io/validation_failure" |= "image-registry-allowed" ``` 2. Change `validationActions` on the binding in `admission-policies.yaml` to `["Deny"]`. ### Tightening a namespace 1. See what running pods a stricter level would reject, without changing anything: ```bash kubectl label --dry-run=server --overwrite ns cert-manager \ pod-security.kubernetes.io/enforce=restricted ``` 2. Catch what is not running right now — a nightly CronJob, say — from the `audit` label's records over the full Loki retention window, in Grafana: ```text {job="kubernetes-audit"} |= "pod-security.kubernetes.io/audit-violations" ``` 3. Raise `enforce` in `pod-security.yaml`. ### Rollout A new policy is enforced on arrival: `policyAuditMode` in `payload/platform/cilium/values.yaml` is off. Switched on, it makes every agent evaluate each verdict and log it with `AUDIT` instead of dropping, and those verdicts reach Loki the way drops do. The mode is cluster-wide, so while it is on the older policies are not enforced either, which is why a batch of new policies goes through it together. 1. Merge the values change; the agents roll on their own. `Enabled` on every node before anything else merges: ```bash kubectl -n kube-system rollout status ds/cilium kubectl -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg config get PolicyAuditMode ``` 2. Merge the policies and let them run for a week. What enforcement would have refused, by namespace pair: ```promql sum by (source_namespace, destination_namespace) (increase(hubble_policy_verdicts_total{action="audit"}[7d])) ``` and flow by flow, in Grafana: ```text {job="hubble", verdict="AUDIT"} | json | line_format "{{.flow_source_namespace}}/{{.flow_source_pod_name}} -> {{.flow_destination_namespace}}/{{.flow_destination_pod_name}} {{.flow_l4_TCP_destination_port}}{{.flow_l4_UDP_destination_port}}" ``` 3. Add a rule for every audited flow that is legitimate. Layer 7 rules are never audited: a request an HTTP rule does not match is answered `403` on the spot, which is why a new policy carries `http: [{}]` — proxied and recorded, nothing refused — until the week's requests have been read: ```text {job="hubble"} | json | flow_l7_http_method != "" | line_format "{{.flow_source_namespace}} -> {{.flow_destination_namespace}}:{{.flow_l4_TCP_destination_port}} {{.flow_l7_http_method}} {{.flow_l7_http_url}}" ``` 4. Tighten each `http: [{}]` to the methods and paths seen, set `policyAuditMode: false`, which rolls the agents again. The first week's reading is [Network Policy Audit 2026-09-27](), the reading before the flip [Network Policy Audit 2026-10-16](). From then on the verdict to watch is `DROPPED`, live or from Loki — see [Cilium]() — and the [`HubblePolicyDrops`]() alert mails each refused connection. If a single endpoint needs the same treatment later, audit mode can be set on it alone, until the agent restarts: ```bash kubectl -n kube-system exec ds/cilium -c cilium-agent -- \ cilium-dbg endpoint config PolicyAuditMode=Enabled ``` ### Host firewall rollout Audit is per host endpoint, a setting of the running agent: `policyAuditMode` would stop enforcing the pod policies, and upstream says the per-endpoint setting does not survive an agent restart, after which the node enforces. The audit week therefore runs through a node label, and every step before enforcement is reversible by removing that label. 1. Merge the switch; the agents roll, one node at a time. Every agent reports the firewall on the native device: ```bash kubectl -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg status | grep 'Host firewall' ``` 2. Reboot one worker with no policy selecting it and watch it return `Ready`, as in [Rebooting a node](). 3. Merge `payload/platform/cilium/host-firewall-audit.yaml`, a DaemonSet that puts each host endpoint back in audit mode within seconds of an agent restart, because any restart would otherwise start enforcing that node: a Renovate bump, a value change, Kured. Every node must answer `Enabled` before the policy merges: ```bash for p in $(kubectl -n kube-system get pods -l k8s-app=cilium -o name); do kubectl -n kube-system exec "$p" -c cilium-agent -- \ sh -c 'cilium-dbg endpoint config $(cilium-dbg endpoint get -l reserved:host -o "jsonpath={\$[0].id}") | grep PolicyAuditMode' done ``` 4. Merge the policy; it selects nothing. Select one worker, with its console within reach, and confirm `Disabled (Audit)` on its `reserved:host` line: ```bash kubectl label node host-firewall.wlkr.ch/selected=true kubectl -n kube-system exec -c cilium-agent -- cilium-dbg endpoint list ``` 5. Reboot that worker. It must return `Ready`, and audit mode `Enabled` within a minute; if it does not, go to [Host firewall lockout](<#host-firewall-lockout>) and stop the rollout. 6. Select the other workers, then one control plane and reboot it, then the other two. Only then start the audit week. 7. Read a week of what enforcement would refuse. Pods and nodes are admitted, so anything here came from the LAN. Add a rule for what is legitimate: ```text {job="hubble", verdict="AUDIT"} | json dst_id="flow.destination.identity", src="flow.IP.source", dst="flow.IP.destination", tcp="flow.l4.TCP.destination_port", udp="flow.l4.UDP.destination_port" | dst_id="1" | line_format "{{.src}} -> {{.dst}} {{.tcp}}{{.udp}}" ``` 8. Enforce one worker: its label sends the DaemonSet away, and restarting its agent drops audit mode, rather than switching audit off at runtime. Check that `bpf policy get` lists separate identities and not one wildcard, and that SSH and the node's pods still work. Reboot it once more. ```bash kubectl label node host-firewall.wlkr.ch/enforce=true kubectl -n kube-system delete pod -l k8s-app=cilium --field-selector spec.nodeName= kubectl -n kube-system exec -c cilium-agent -- cilium-dbg bpf policy get ``` 9. Repeat for the workers, then the control planes one at a time. Then delete `payload/platform/cilium/host-firewall-audit.yaml` in a pull request. ## Health check ```bash kubectl get ns -L pod-security.kubernetes.io/enforce kubectl get cnp -A kubectl get ccnp kubectl get validatingadmissionpolicy ``` ## Pitfalls > [!NOTE] > > **Audit mode stops at the proxy** > > `policyAuditMode` is a datapath setting. A request the Gateway's Envoy or an HTTP rule refuses is answered `403 Access denied` on the spot, and Hubble records it as a forwarded response, not an audited verdict. Look for it with `hubble observe --http-status 403`; in Loki the denied request is a `DROPPED` flow of type `REQUEST` whose source carries the caller's namespace and the `ingress` identity. The agents' `argocd` calls failed this way, `server: envoy` and `grpc-message: Access denied`, while their policy named the host but not the `argocd-server` pod; see [Agent account](). > [!NOTE] > > **A name rule audits one SYN in twenty** > > A `toFQDNs` rule admits the addresses the DNS proxy has seen for the name, and the proxy releases the answer after at most 100ms whether or not every datapath has the new address. About one connection in twenty to a CDN name is recorded `AUDIT` with a `world` destination; under enforcement that `SYN` drops and the retransmit a second later goes through. A run of drops on one source port is different: the application connected to an address it cached past the DNS TTL. > [!NOTE] > > **The token change is not retroactive** > > The mount is decided at admission, so existing pods keep their token until recreated. That makes the change safe to roll out, and means a posture scan will not agree it is fixed until things restart. > [!CAUTION] > > **A reboot can isolate a node a host policy selects** > > [Upstream]() reports a selected node on this same combination, tunnel, kube-proxy replacement and WireGuard, dropping everything after a cold boot or a runtime audit-off, until an out-of-band fix. It is open and unreproduced here, and a power cut would hit every selected node at once. That is why the rollout selects one worker first and reboots it. ## Recovery ### Host firewall lockout Work down until the node answers. An agent that cannot reach the API server hears neither a policy change nor a label removal, so step 1 alone does not free a node that is already cut off. 1. API server answering: take the node out of the policy. Argo CD does not own node labels, so nothing puts it back. Use `--all` instead of a name for every node. ```bash kubectl label node host-firewall.wlkr.ch/selected- ``` 2. SSH answering: put the node in audit mode, which needs no API server and lasts until the agent restarts. ```bash ssh sudo bash -s <<'EOF' id=$(crictl ps -q --label io.kubernetes.container.name=cilium-agent) crictl exec "$id" sh -c 'cilium-dbg endpoint config $(cilium-dbg endpoint get -l reserved:host -o "jsonpath={\$[0].id}") PolicyAuditMode=Enabled' EOF ``` 3. Nothing answering, the API server up: nothing filters a node until its agent has started after boot, so stop the agent coming back. Run this from the workstation, then power-cycle the node, then do step 1 and start the kubelet again with `sudo systemctl unmask kubelet && sudo systemctl start kubelet`. ```bash until ssh -o ConnectTimeout=2 sudo systemctl mask --now kubelet; do sleep 1; done ``` 4. Every control plane cut off: the API server needs no CNI, so it comes up before the agent does. Power-cycle all three and take the label off in that window. ```bash until kubectl label node --all host-firewall.wlkr.ch/selected-; do sleep 2; done ``` 5. Still nothing: at the node's console, add `flatcar.autologin` to the boot entry in the GRUB menu, since `core` has no password, then run the inner command of step 2. Failing that, [rebuild the node](). 6. Then [unseal OpenBao]() and revert in Git: the policy first; the switch only if the firewall itself is the fault, since that revert rolls every agent again.