Cilium¶

Cilium is the CNI, and on this cluster considerably more: it replaces kube-proxy, terminates ingress through Gateway API, hands out LoadBalancer addresses on a network with no cloud load balancer, encrypts node-to-node traffic, and shows what is talking to what. That consolidation is the argument for it on bare metal, and the reason a broken Cilium is never a small problem.

At a glance¶


Namespace	`kube-system`
Depends on	The Gateway API and Prometheus operator CRDs it renders against, and a `k8sServiceHost` that answers
If it is down	Everything. No CNI, no service routing, no ingress, no LoadBalancer addresses
Health check	`kubectl -n kube-system exec ds/cilium -- cilium status --brief`
UI	`hubble.infra.k8s.wlkr.ch` (Hubble); the same flows in 3D across namespaces at Flowscape
Files	`payload/platform/cilium/`

Configuration¶

Setting	Why
`kubeProxyReplacement: true`	Service routing in eBPF. `proxy.disabled` in `ansible/templates/kubeadm.yaml.j2` keeps `kube-proxy` out at init and on every `kubeadm upgrade apply`
`k8sServiceHost` as a literal IP	With `kube-proxy` gone Cilium cannot reach the API server through a Service, so it names the control-plane VIP — see Pitfalls
Gateway API	Replaces an Ingress controller — see Gateway API
`lb-pools.yaml`	One address each for the apps and infra Gateways, announced over L2 ARP so the LAN can find them
`bgp.yaml`	The same two addresses as `/32` routes over eBGP from every control-plane node to the router. The router forwards traffic it routes, from other VLANs, VPN clients and port forwards, to all three nodes at once and drops a dead one after the hold time, where L2 moves one lease. L2 stays because the addresses sit inside the flat LAN, whose hosts ARP for them and never ask the router. The ASNs and timers are in `bgp.yaml`. The router side is FRR on the UDM Pro: `make config` renders it from the inventory's `router_*` variables into `output/router/frr-bgp.conf`, so a firmware reset is recovered by pasting that file
`encryption.type: wireguard`	Transparent node-to-node encryption
Hubble, behind the Authentik proxy outpost	Hubble has no authentication of its own; the HTTPRoute points at the outpost and a ReferenceGrant in `payload/platform/authentik/` permits the cross-namespace reference
Hubble metrics labelled by namespace	The chart's Hubble dashboards filter on source and destination namespace and show nothing without them; namespace keeps the series count to the square of the namespace count. The HTTP metric adds workloads, but only for traffic Cilium proxies at L7, which is the Gateway.
`hubble.export.static`	Every flow with a `DROPPED`, `ERROR` or `AUDIT` verdict, and every HTTP request a policy proxies at L7, goes to a file on the node that Alloy tails into Loki, with a field mask that keeps identities, ports and the request line and drops the rest. Hubble's own buffer is minutes deep; this is the record of what a policy refused after the fact, and of the requests the L7 rules are tightened from. Other forwarded flows are not exported: thousands a second on a cluster this size, where the proxied requests are a few.
`hubble.redact`	Every header except `User-Agent` and `Content-Type`, and the password of a basic-auth URL, is replaced before a flow leaves the agent, for the export file and Relay alike. An L7 rule proxies the requests that carry credentials: External Secrets' `X-Vault-Token`, the bearer tokens Prometheus scrapes with, Velero's signed S3 requests, Authentik's cookies. The method, path and query the L7 rules are tightened from survive
`hostFirewall.enabled: true`	Filters traffic to the node's own addresses, which is every host-networked port, with `CiliumClusterwideNetworkPolicy` objects that select nodes. Attaching it denies nothing on IPv4 until a policy selects the node; IPv6 is another matter, see Pitfalls. Rollout and policy in Host firewall
`policyAuditMode: false`	Policies are enforced. On, verdicts are logged and not enforced, which is how a rollout's observation week runs — see Security Policies. Cluster-wide, and blind to L7 rules
`rollOutCiliumPods` and the four `rollOutPods`	The agent, Envoy, the operator, Relay and the UI read their ConfigMap once, at startup. With a checksum of it on the pod template, a merged value rolls the pods and is live within minutes, and a value the agent refuses fails at merge time, while someone is watching, not at the next unrelated restart. The price is that every change to `cilium-config` is a rolling restart of the CNI on every node: running pods keep their networking, policy updates pause per node for the seconds its agent is down, and the `k8sServiceHost` pitfall bites at merge, not later
`updateStrategy.rollingUpdate.maxUnavailable: 1`	The chart default restarts the agent on two nodes at once, which can be two of the three control planes that hold the L2 leases and BGP sessions; one at a time keeps every Gateway address announced throughout a rollout
`operator.replicas` above one	The operator owns Gateway API reconciliation, LoadBalancer IP allocation and the `cilium-secrets` sync; with one replica, a drain or crash of its node stalls all three until the reschedule. The chart's hostname anti-affinity keeps the replicas apart, so the standby is already running — see the operator pitfall
`lb-pools.yaml` on `cilium.io/v2`	The storage version for `CiliumLoadBalancerIPPool`; `v2alpha1` is served as deprecated and warns on every sync. `CiliumL2AnnouncementPolicy` has no `v2` yet and stays
`cilium-agent` memory limit	It is the one process whose death takes pod networking with it, so it was sized from a week of measurement, not a day, before the platform rule applied
`trustCRDsExist: true`	The chart otherwise refuses to render while `monitoring.coreos.com/v1` is missing: the bootstrap install, a render before `prometheus-operator-crds` has synced, and the diff preview's throwaway cluster

Usage¶

Installation¶

Cilium is installed twice, in a sense. make install-cilium installs it by Helm, because no pod runs without a CNI and ArgoCD is a pod; ArgoCD then adopts the release. The version comes from targetRevision in application.yaml — see Version pins.

The bootstrap install uses the same values.yaml, and has to: the pods roll whenever cilium-config changes, so a slimmer bootstrap config would restart every agent the moment ArgoCD adopts the release. The Gateway API CRDs go in first because the operator checks for them once, at startup. The one difference is the three serviceMonitor.enabled flags, which make turns off because the Prometheus operator CRDs arrive through ArgoCD; ArgoCD adds the monitors back on adoption, rolling the agent and operator once.

Health check¶

kubectl -n kube-system exec ds/cilium -- cilium status --brief

The BGP session on each control-plane node, established with two routes advertised. bgp.yaml selects the control planes only, so ask that node's agent rather than whichever pod ds/cilium resolves to:

kubectl -n kube-system exec "$(kubectl -n kube-system get pod -l k8s-app=cilium \
  --field-selector spec.nodeName=<control-plane> -o name)" -- cilium bgp peers

Hubble answers what did I just break? during a network policy rollout — see Security Policies. The hubble CLI talks to Relay through a port-forward on its default port:

kubectl -n kube-system port-forward deploy/hubble-relay 4245:4245 &
hubble observe --verdict DROPPED --follow

The same verdicts, for the whole Loki retention window, in Grafana:

{job="hubble", verdict="DROPPED"} | json | flow_drop_reason_desc="POLICY_DENIED"

Pitfalls¶

k8sServiceHost pointed at an address that does not answer takes the cluster down

Every Cilium pod loses the API server at once, and cluster networking goes with it — including whatever you were using to fix the problem. The change ordering is in Control Plane VIP.

An operator that starts without the API server leaves the Gateways without certificates

cilium-operator checks for the Gateway API CRDs once at startup; if that check fails it runs without its Gateway controller until the next restart, and cilium-secrets empties. Envoy then resets the TLS handshake on every node without a cached copy, so some clients fail and others do not, depending on which node the router hashes them to. CiliumGatewayCertificatesMissing fires; kubectl -n kube-system rollout restart deploy/cilium-operator recovers it.

A chart default that disappears stays in cilium-config

The bootstrap install's helm field manager still co-owns every cilium-config key it wrote, so ArgoCD's server-side apply cannot delete one the chart stops rendering. When a chart release dropped the envoy.xdsMode default, the agents kept envoy-xds-mode: ads while Envoy's bootstrap switched to split xDS; Envoy got no listeners, and every Gateway and L7 policy stopped forwarding. Pin such a value in values.yaml rather than relying on the default.

The host firewall drops all node IPv6, with or without a policy

With enable-ipv6: false the datapath has no IPv6 branch, so on the native device and cilium_host every IPv6 packet is dropped as DROP_UNKNOWN_L3 in both directions, before any policy or audit mode (upstream). A node keeps its SLAAC address until the lifetime runs out, and a client that does not fall back to IPv4 stalls until then. The drops reach Loki as DROPPED flows with a reason other than POLICY_DENIED, so HubblePolicyDrops stays quiet.