Security Policies¶

Five cluster-wide controls: Pod Security Admission, default-deny network policy in both directions, a host firewall on the nodes, no API token on the default ServiceAccount, and admission policies on image references, namespace labels and the authentik namespace. Each is easy to turn on and hard to turn on safely, so the pattern is the same for all of them: measure first, enforce second, one namespace at a time.

At a glance¶

Configuration¶

Pod Security Admission¶

A namespace without labels gets the cluster default from ansible/templates/admission.yaml.j2: enforce at baseline, audit and warn at restricted. It reaches a control-plane node only at provisioning; see Node hardening. It exempts kube-system, which Cilium needs before Argo CD can label anything, and the namespaces that need privileged but carry no labels. An exemption skips audit and warn as well, so the labelled namespaces below are not exempt. Each namespace in pod-security.yaml carries three labels:

Enforcement is set to what already works, so nothing running breaks, while warn and audit show what a stricter level would catch; tightening later is an informed change.

The level per namespace is in pod-security.yaml; the rule behind it:

privileged here means "not yet reduced", not "unexamined": audit and warn are still baseline or restricted on all of them. Namespace objects are owned by this Application so the labels stay declarative rather than drifting after CreateNamespace=true. cilium-secrets is the exception: the Cilium chart creates it, so its labels are secretsNamespaceLabels in payload/platform/cilium/values.yaml, and a second owner would fight the first. Workload namespaces label themselves in the workloads repository, and namespace-privileged-allowed keeps them off privileged.

Network policies¶

One file per namespace in network-policies/, holding that namespace's CiliumNetworkPolicy objects. A namespace gets two whole-namespace policies, default-ingress and default-egress, so every pod in it is default-deny in both directions, plus one policy per workload that needs more than the rest of the namespace does. Each policy's description is the record of who may connect and why; the rules are not repeated here. cluster-wide.yaml holds the CiliumClusterwideNetworkPolicy objects: the rules every pod gets, so a namespace file is the record of what its namespace may do beyond those.

Why CiliumNetworkPolicy and not NetworkPolicy¶

Two kinds of traffic have no pod identity a standard NetworkPolicy can name: Gateway traffic, which reaches a backend from the per-node Envoy, and kubelet health probes, which come from the node. A plain default-deny therefore kills ingress and probes, and the pods restart forever in a way that looks like an application fault. Cilium's fromEntities names them: ingress for Envoy-proxied traffic, host and remote-node for the kubelet.

Cluster-wide rules¶

A rule that is word-for-word the same in every namespace file is lifted into cluster-wide.yaml, selecting every endpoint that has a namespace and with enableDefaultDeny off in both directions. The selector is not {} because that also takes the Gateway's reserved:ingress endpoint and the health endpoint, which then carry a policy they have no use for. Default deny is off because a cluster-wide egress rule would otherwise put every pod in a namespace without policies into egress default-deny. DNS is the only rule lifted; the intra-namespace and node-probe rules would qualify but change what a namespace file says, so they stay.

The second policy there, init, is for the seconds before the agent has resolved a new pod's labels. The pod then carries the reserved init identity, which no namespace policy selects, and the audit log shows its traffic denied all the same; a scan Job's first lookups and a starting pod's probes were the bulk of the audit log. It admits the node, DNS out and DNS in, nothing else.

The third, cnpg-backups, selects every CloudNativePG pod by its cnpg.io/cluster label and admits only the object store, by the same two rules Velero's namespace has. Every database's backup sidecar needs exactly that, wherever the database runs, so one policy replaces a rule in each workload's file.

What every policy contains¶

Host-networked pods (Cilium, kube-vip, the control plane, node-exporter, Tetragon, the CSI node plugins) have no endpoint of their own and cannot be selected; to everything else they are host or remote-node. kube-system therefore covers only its pod-networked workloads, each selected by label.

Where a chart ships NetworkPolicy objects of its own (argocd) they are switched off in its values: Cilium unions the two kinds, so a chart rule that allows everything would silently override the default-deny.

Host firewall¶

Pod policies cannot select a host-networked port. A CiliumClusterwideNetworkPolicy with a nodeSelector can, and Cilium's host firewall enforces it on the native device and cilium_host. What each port is for is in Host Firewall Inventory 2026-10-30.

network-policies/host-firewall.yaml selects the nodes carrying host-firewall.wlkr.ch/selected. It denies the LAN everything but SSH, the API server, DHCP and ping, and admits the whole cluster to every port, since a narrower cluster rule could not name Prometheus and could lock out a node.

Default ServiceAccount tokens¶

A pod that names no serviceAccountName lands on default, which nothing configures, and gets an API token it has no use for. A workload reaching the API through default is already a misconfiguration, so default-serviceaccounts.yaml sets automountServiceAccountToken: false on it in every namespace with nothing to preserve, empty namespaces included. Left out: kube-system, where the control plane authenticates with client certificates and "very likely fine" is not the standard. authentik-server runs on default by chart default and is covered: Authentik's Kubernetes calls are worker tasks, which run under the authentik account.

The same rule applies to a chart workload whose own ServiceAccount has no binding — Alertmanager, Cilium's Envoy and Argo CD's HAProxy switch the mount off in their values — unless the process needs the in-cluster config to start at all (the External Secrets webhook) or the chart offers no switch (Rook's rook-ceph-default and rook-ceph-rgw).

Agent RBAC¶

claude-agent-rbac.yaml gives the in-cluster Claude Code agents two ClusterRoles for the claude-reader and claude-writer ServiceAccounts of the claude-agents namespace. Cluster-scoped RBAC cannot come from the apps project, and what an agent may touch is a platform decision.

• Creating a pod reads every Secret in its namespace. That is why the write role is bound per namespace, and the apps project cannot place a RoleBinding in the namespaces its destinations exclude.
• claude-writer-pod-limits in admission-policies.yaml refuses any pod or pod template from the writer that uses the host's network, PID or IPC namespace, a hostPath volume, a privileged container, or a ServiceAccount other than default or the one the workload already runs as. A namespace without a Pod Security level allows all of those, and another ServiceAccount's token is that account's RBAC.
• pods/log is cluster-wide, so the reader sees whatever any component logs, platform namespaces included. Triage needs it, and a cluster-wide grant cannot leave out namespaces.

Kinds are listed, never wildcarded, so a new CRD is invisible to the agents until someone adds it. Argo CD reverts a hand edit within minutes either way.

Admission policies¶

Trivy reports a bad image after it is running; admission-policies.yaml refuses or records it at admission, with the built-in ValidatingAdmissionPolicy rather than a policy engine. The image policies match Pods and every controller that produces them, so a bad Deployment is rejected at kubectl apply rather than failing quietly in its ReplicaSet. The other three close what RBAC and the apps project cannot express.

All but one carry failurePolicy: Ignore: an expression that errors lets the request through rather than stopping every pod in the cluster. A real violation is still refused. claude-writer-pod-limits fails closed, since it only ever matches one identity.

Usage¶

Promoting the registry allowlist¶

1. Confirm nothing running would be refused, over the full Loki retention:

   {job="kubernetes-audit"} |= "validation.policy.admission.k8s.io/validation_failure" |= "image-registry-allowed"
   

2. Change validationActions on the binding in admission-policies.yaml to ["Deny"].

Tightening a namespace¶

1. See what running pods a stricter level would reject, without changing anything:

   kubectl label --dry-run=server --overwrite ns cert-manager \
     pod-security.kubernetes.io/enforce=restricted
   

2. Catch what is not running right now — a nightly CronJob, say — from the audit label's records over the full Loki retention window, in Grafana:

   {job="kubernetes-audit"} |= "pod-security.kubernetes.io/audit-violations"
   

3. Raise enforce in pod-security.yaml.

Rollout¶

A new policy is enforced on arrival: policyAuditMode in payload/platform/cilium/values.yaml is off. Switched on, it makes every agent evaluate each verdict and log it with AUDIT instead of dropping, and those verdicts reach Loki the way drops do. The mode is cluster-wide, so while it is on the older policies are not enforced either, which is why a batch of new policies goes through it together.

1. Merge the values change; the agents roll on their own. Enabled on every node before anything else merges:

   kubectl -n kube-system rollout status ds/cilium
   kubectl -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg config get PolicyAuditMode
   

2. Merge the policies and let them run for a week. What enforcement would have refused, by namespace pair:

   sum by (source_namespace, destination_namespace) (increase(hubble_policy_verdicts_total{action="audit"}[7d]))
   

   and flow by flow, in Grafana:

   {job="hubble", verdict="AUDIT"} | json | line_format "{{.flow_source_namespace}}/{{.flow_source_pod_name}} -> {{.flow_destination_namespace}}/{{.flow_destination_pod_name}} {{.flow_l4_TCP_destination_port}}{{.flow_l4_UDP_destination_port}}"
   

3. Add a rule for every audited flow that is legitimate. Layer 7 rules are never audited: a request an HTTP rule does not match is answered 403 on the spot, which is why a new policy carries http: [{}] — proxied and recorded, nothing refused — until the week's requests have been read:

   {job="hubble"} | json | flow_l7_http_method != "" | line_format "{{.flow_source_namespace}} -> {{.flow_destination_namespace}}:{{.flow_l4_TCP_destination_port}} {{.flow_l7_http_method}} {{.flow_l7_http_url}}"
   

4. Tighten each http: [{}] to the methods and paths seen, set policyAuditMode: false, which rolls the agents again. The first week's reading is Network Policy Audit 2026-09-27, the reading before the flip Network Policy Audit 2026-10-16. From then on the verdict to watch is DROPPED, live or from Loki — see Cilium — and the HubblePolicyDrops alert mails each refused connection.

If a single endpoint needs the same treatment later, audit mode can be set on it alone, until the agent restarts:

kubectl -n kube-system exec ds/cilium -c cilium-agent -- \
  cilium-dbg endpoint config <endpoint-id> PolicyAuditMode=Enabled

Host firewall rollout¶

Audit is per host endpoint, a setting of the running agent: policyAuditMode would stop enforcing the pod policies, and upstream says the per-endpoint setting does not survive an agent restart, after which the node enforces. The audit week therefore runs through a node label, and every step before enforcement is reversible by removing that label.

1. Merge the switch; the agents roll, one node at a time. Every agent reports the firewall on the native device:

   kubectl -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg status | grep 'Host firewall'
   

2. Reboot one worker with no policy selecting it and watch it return Ready, as in Rebooting a node.

3. Merge payload/platform/cilium/host-firewall-audit.yaml, a DaemonSet that puts each host endpoint back in audit mode within seconds of an agent restart, because any restart would otherwise start enforcing that node: a Renovate bump, a value change, Kured. Every node must answer Enabled before the policy merges:

   for p in $(kubectl -n kube-system get pods -l k8s-app=cilium -o name); do
     kubectl -n kube-system exec "$p" -c cilium-agent -- \
       sh -c 'cilium-dbg endpoint config $(cilium-dbg endpoint get -l reserved:host -o "jsonpath={\$[0].id}") | grep PolicyAuditMode'
   done
   

4. Merge the policy; it selects nothing. Select one worker, with its console within reach, and confirm Disabled (Audit) on its reserved:host line:

   kubectl label node <worker> host-firewall.wlkr.ch/selected=true
   kubectl -n kube-system exec <agent-on-worker> -c cilium-agent -- cilium-dbg endpoint list
   

5. Reboot that worker. It must return Ready, and audit mode Enabled within a minute; if it does not, go to Host firewall lockout and stop the rollout.

6. Select the other workers, then one control plane and reboot it, then the other two. Only then start the audit week.

7. Read a week of what enforcement would refuse. Pods and nodes are admitted, so anything here came from the LAN. Add a rule for what is legitimate:

   {job="hubble", verdict="AUDIT"} | json dst_id="flow.destination.identity", src="flow.IP.source", dst="flow.IP.destination", tcp="flow.l4.TCP.destination_port", udp="flow.l4.UDP.destination_port" | dst_id="1" | line_format "{{.src}} -> {{.dst}} {{.tcp}}{{.udp}}"
   

8. Enforce one worker: its label sends the DaemonSet away, and restarting its agent drops audit mode, rather than switching audit off at runtime. Check that bpf policy get lists separate identities and not one wildcard, and that SSH and the node's pods still work. Reboot it once more.

   kubectl label node <worker> host-firewall.wlkr.ch/enforce=true
   kubectl -n kube-system delete pod -l k8s-app=cilium --field-selector spec.nodeName=<worker>
   kubectl -n kube-system exec <agent-on-worker> -c cilium-agent -- cilium-dbg bpf policy get <host-endpoint-id>
   

9. Repeat for the workers, then the control planes one at a time. Then delete payload/platform/cilium/host-firewall-audit.yaml in a pull request.

Health check¶

kubectl get ns -L pod-security.kubernetes.io/enforce
kubectl get cnp -A
kubectl get ccnp
kubectl get validatingadmissionpolicy

Pitfalls¶

Recovery¶

Host firewall lockout¶

Work down until the node answers. An agent that cannot reach the API server hears neither a policy change nor a label removal, so step 1 alone does not free a node that is already cut off.

1. API server answering: take the node out of the policy. Argo CD does not own node labels, so nothing puts it back. Use --all instead of a name for every node.

   kubectl label node <node> host-firewall.wlkr.ch/selected-
   

2. SSH answering: put the node in audit mode, which needs no API server and lasts until the agent restarts.

   ssh <node> sudo bash -s <<'EOF'
   id=$(crictl ps -q --label io.kubernetes.container.name=cilium-agent)
   crictl exec "$id" sh -c 'cilium-dbg endpoint config $(cilium-dbg endpoint get -l reserved:host -o "jsonpath={\$[0].id}") PolicyAuditMode=Enabled'
   EOF
   

3. Nothing answering, the API server up: nothing filters a node until its agent has started after boot, so stop the agent coming back. Run this from the workstation, then power-cycle the node, then do step 1 and start the kubelet again with sudo systemctl unmask kubelet && sudo systemctl start kubelet.

   until ssh -o ConnectTimeout=2 <node> sudo systemctl mask --now kubelet; do sleep 1; done
   

4. Every control plane cut off: the API server needs no CNI, so it comes up before the agent does. Power-cycle all three and take the label off in that window.

   until kubectl label node --all host-firewall.wlkr.ch/selected-; do sleep 2; done
   

5. Still nothing: at the node's console, add flatcar.autologin to the boot entry in the GRUB menu, since core has no password, then run the inner command of step 2. Failing that, rebuild the node.

6. Then unseal OpenBao and revert in Git: the policy first; the switch only if the firewall itself is the fault, since that revert rolls every agent again.