# Cilium
Cilium is the CNI, and on this cluster considerably more: it replaces `kube-proxy`, terminates ingress through Gateway API, hands out LoadBalancer addresses on a network with no cloud load balancer, encrypts node-to-node traffic, and shows what is talking to what. That consolidation is the argument for it on bare metal, and the reason a broken Cilium is never a small problem.
## At a glance
| | |
| --- | --- |
| Namespace | `kube-system` |
| Depends on | The Gateway API and Prometheus operator CRDs it renders against, and a `k8sServiceHost` that answers |
| If it is down | Everything. No CNI, no service routing, no ingress, no LoadBalancer addresses |
| Health check | `kubectl -n kube-system exec ds/cilium -- cilium status --brief` |
| UI | `hubble.infra.k8s.wlkr.ch` (Hubble); the same flows in 3D across namespaces at [Flowscape]() |
| Files | `payload/platform/cilium/` |
## Configuration
| Setting | Why |
| --- | --- |
| `kubeProxyReplacement: true` | Service routing in eBPF. `proxy.disabled` in `ansible/templates/kubeadm.yaml.j2` keeps `kube-proxy` out at init and on every `kubeadm upgrade apply` |
| `k8sServiceHost` as a literal IP | With `kube-proxy` gone Cilium cannot reach the API server through a Service, so it names the [control-plane VIP]() — see [Pitfalls](<#pitfalls>) |
| Gateway API | Replaces an Ingress controller — see [Gateway API]() |
| `lb-pools.yaml` | One address each for the apps and infra Gateways, announced over L2 ARP so the LAN can find them |
| `bgp.yaml` | The same two addresses as `/32` routes over eBGP from every control-plane node to the router. The router forwards traffic it routes, from other VLANs, VPN clients and port forwards, to all three nodes at once and drops a dead one after the hold time, where L2 moves one lease. L2 stays because the addresses sit inside the flat LAN, whose hosts ARP for them and never ask the router. The ASNs and timers are in `bgp.yaml`. The router side is FRR on the UDM Pro: `make config` renders it from the inventory's `router_*` variables into `output/router/frr-bgp.conf`, so a firmware reset is recovered by pasting that file |
| `encryption.type: wireguard` | Transparent node-to-node encryption |
| Hubble, behind the [Authentik]() proxy outpost | Hubble has no authentication of its own; the HTTPRoute points at the outpost and a ReferenceGrant in `payload/platform/authentik/` permits the cross-namespace reference |
| Hubble metrics labelled by namespace | The chart's Hubble dashboards filter on source and destination namespace and show nothing without them; namespace keeps the series count to the square of the namespace count. The HTTP metric adds workloads, but only for traffic Cilium proxies at L7, which is the Gateway. |
| `hubble.export.static` | Every flow with a `DROPPED`, `ERROR` or `AUDIT` verdict, and every HTTP request a policy proxies at L7, goes to a file on the node that [Alloy]() tails into Loki, with a field mask that keeps identities, ports and the request line and drops the rest. Hubble's own buffer is minutes deep; this is the record of what a policy refused after the fact, and of the requests the L7 rules are tightened from. Other forwarded flows are not exported: thousands a second on a cluster this size, where the proxied requests are a few. |
| `hubble.redact` | Every header except `User-Agent` and `Content-Type`, and the password of a basic-auth URL, is replaced before a flow leaves the agent, for the export file and Relay alike. An L7 rule proxies the requests that carry credentials: External Secrets' `X-Vault-Token`, the bearer tokens Prometheus scrapes with, Velero's signed S3 requests, Authentik's cookies. The method, path and query the L7 rules are tightened from survive |
| `hostFirewall.enabled: true` | Filters traffic to the node's own addresses, which is every host-networked port, with `CiliumClusterwideNetworkPolicy` objects that select nodes. Attaching it denies nothing on IPv4 until a policy selects the node; IPv6 is another matter, see [Pitfalls](<#pitfalls>). Rollout and policy in [Host firewall]() |
| `policyAuditMode: false` | Policies are enforced. On, verdicts are logged and not enforced, which is how a rollout's observation week runs — see [Security Policies](). Cluster-wide, and blind to L7 rules |
| `rollOutCiliumPods` and the four `rollOutPods` | The agent, Envoy, the operator, Relay and the UI read their ConfigMap once, at startup. With a checksum of it on the pod template, a merged value rolls the pods and is live within minutes, and a value the agent refuses fails at merge time, while someone is watching, not at the next unrelated restart. The price is that every change to `cilium-config` is a rolling restart of the CNI on every node: running pods keep their networking, policy updates pause per node for the seconds its agent is down, and the [`k8sServiceHost` pitfall](<#pitfalls>) bites at merge, not later |
| `updateStrategy.rollingUpdate.maxUnavailable: 1` | The chart default restarts the agent on two nodes at once, which can be two of the three control planes that hold the L2 leases and BGP sessions; one at a time keeps every Gateway address announced throughout a rollout |
| `operator.replicas` above one | The operator owns Gateway API reconciliation, LoadBalancer IP allocation and the `cilium-secrets` sync; with one replica, a drain or crash of its node stalls all three until the reschedule. The chart's hostname anti-affinity keeps the replicas apart, so the standby is already running — see the [operator pitfall](<#pitfalls>) |
| `lb-pools.yaml` on `cilium.io/v2` | The storage version for `CiliumLoadBalancerIPPool`; `v2alpha1` is served as deprecated and warns on every sync. `CiliumL2AnnouncementPolicy` has no `v2` yet and stays |
| `cilium-agent` memory limit | It is the one process whose death takes pod networking with it, so it was sized from a week of measurement, not a day, before the [platform rule]() applied |
| `trustCRDsExist: true` | The chart otherwise refuses to render while `monitoring.coreos.com/v1` is missing: the bootstrap install, a render before `prometheus-operator-crds` has synced, and the diff preview's throwaway cluster |
## Usage
### Installation
Cilium is installed twice, in a sense. `make install-cilium` installs it by Helm, because no pod runs without a CNI and ArgoCD is a pod; ArgoCD then adopts the release. The version comes from `targetRevision` in `application.yaml` — see [Version pins]().
The bootstrap install uses the same `values.yaml`, and has to: the pods roll whenever `cilium-config` changes, so a slimmer bootstrap config would restart every agent the moment ArgoCD adopts the release. The Gateway API CRDs go in first because the operator checks for them once, at startup. The one difference is the three `serviceMonitor.enabled` flags, which `make` turns off because the Prometheus operator CRDs arrive through ArgoCD; ArgoCD adds the monitors back on adoption, rolling the agent and operator once.
## Health check
```bash
kubectl -n kube-system exec ds/cilium -- cilium status --brief
```
The BGP session on each control-plane node, `established` with two routes advertised. `bgp.yaml` selects the control planes only, so ask that node's agent rather than whichever pod `ds/cilium` resolves to:
```bash
kubectl -n kube-system exec "$(kubectl -n kube-system get pod -l k8s-app=cilium \
--field-selector spec.nodeName= -o name)" -- cilium bgp peers
```
Hubble answers *what did I just break?* during a network policy rollout — see [Security Policies](). The `hubble` CLI talks to Relay through a port-forward on its default port:
```bash
kubectl -n kube-system port-forward deploy/hubble-relay 4245:4245 &
hubble observe --verdict DROPPED --follow
```
The same verdicts, for the whole Loki retention window, in Grafana:
```text
{job="hubble", verdict="DROPPED"} | json | flow_drop_reason_desc="POLICY_DENIED"
```
## Pitfalls
> [!CAUTION]
>
> **`k8sServiceHost` pointed at an address that does not answer takes the cluster down**
>
> Every Cilium pod loses the API server at once, and cluster networking goes with it — including whatever you were using to fix the problem. The change ordering is in [Control Plane VIP]().
> [!WARNING]
>
> **An operator that starts without the API server leaves the Gateways without certificates**
>
> cilium-operator checks for the Gateway API CRDs once at startup; if that check fails it runs without its Gateway controller until the next restart, and `cilium-secrets` empties. Envoy then resets the TLS handshake on every node without a cached copy, so some clients fail and others do not, depending on which node the router hashes them to. `CiliumGatewayCertificatesMissing` fires; `kubectl -n kube-system rollout restart deploy/cilium-operator` recovers it.
> [!CAUTION]
>
> **A chart default that disappears stays in `cilium-config`**
>
> The bootstrap install's `helm` field manager still co-owns every `cilium-config` key it wrote, so ArgoCD's server-side apply cannot delete one the chart stops rendering. When a chart release dropped the `envoy.xdsMode` default, the agents kept `envoy-xds-mode: ads` while Envoy's bootstrap switched to split xDS; Envoy got no listeners, and every Gateway and L7 policy stopped forwarding. Pin such a value in `values.yaml` rather than relying on the default.
> [!WARNING]
>
> **The host firewall drops all node IPv6, with or without a policy**
>
> With `enable-ipv6: false` the datapath has no IPv6 branch, so on the native device and `cilium_host` every IPv6 packet is dropped as `DROP_UNKNOWN_L3` in both directions, before any policy or audit mode ([upstream]()). A node keeps its SLAAC address until the lifetime runs out, and a client that does not fall back to IPv4 stalls until then. The drops reach Loki as `DROPPED` flows with a reason other than `POLICY_DENIED`, so `HubblePolicyDrops` stays quiet.